Using the Connect gateway
This page tells you how to use the Connect gateway to connect to a registered cluster. Before reading this guide, you should be familiar with the concepts in our overview. The guide assumes that your project administrator has already set up the gateway, and given you the necessary roles and permissions.
Before you begin
Ensure that you have the following command line tools installed:
- The latest version of the Google Cloud CLI, the command line tool for interacting with Google Cloud.
kubectl
If you are using Cloud Shell as your shell environment for interacting with Google Cloud, these tools are installed for you.
Ensure that you have initialized the gcloud CLI for use with your project.
Log in to your Google Cloud account
You can use your own Google Cloud account or a Google Cloud service account to interact with connected clusters via the gateway API.
Follow the instructions in Authorizing Google Cloud CLI tools to log in to your user account. The Connect gateway supports service account impersonation, so even if you're logged in to your own user account you can use a service account to interact with clusters, as you'll see in the following sections.
Select a registered cluster
If you don't know the name of the cluster you want to access, you can see all your current fleet's registered clusters by running the following command:
gcloud container fleet memberships list
This lists all your fleet's clusters, including their membership names and external IDs. Each cluster in a fleet has a unique membership name. For GKE clusters, the membership name generally matches the name you gave it when you created the cluster, unless the cluster's name was not unique within its project at registration.
Get the cluster's gateway kubeconfig
Use the following command to get the kubeconfig
you need to interact with your
specified cluster:
gcloud container fleet memberships get-credentials MEMBERSHIP_NAME
Replace MEMBERSHIP_NAME
with your cluster's fleet membership
name.
This command returns a special Connect gateway-specific kubeconfig
that
lets you connect to the cluster through the Connect gateway.
If you want to use a service account rather than your own Google Cloud account,
use gcloud config
to set auth/impersonate_service_account
to the service
account email address.
To fetch the cluster credential used to interact with the Connect gateway using a service account, run the following commands: Note the following:
- Google Distributed Cloud (software only) clusters on bare metal and VMware: The membership name is the same as the cluster name.
GKE on AWS: Use
gcloud container aws clusters get-credentials
.GKE on Azure: Use
gcloud container azure clusters get-credentials
.
If you want to use a service account rather than your own Google Cloud account, use gcloud config
to set auth/impersonate_service_account
to the service account email address. You can find out more about letting users impersonate a service account in Manage access to service accounts.
gcloud config set auth/impersonate_service_account SA_EMAIL_ADDRESS
gcloud container fleet memberships get-credentials MEMBERSHIP_NAME
Replace SA_EMAIL_ADDRESS
with the email address of the
service account. You can find out more about letting users impersonate a service
account in Manage access to service accounts.
Run commands against the cluster
Once you have the necessary credentials, you can run commands using kubectl
or a go-client
as you normally would for any Kubernetes cluster. Your output should look something like the following:
# Get namespaces in the Cluster.
kubectl get namespaces
NAME STATUS AGE
default Active 59d
gke-connect Active 4d
Limitations
The following kubectl
commands are not supported by the gateway
using the gcloud container fleet memberships get-credentials
command:
attach
cp
exec
port-forward
Preview support for the commands
The attach
, cp
, and exec
commands (but not port-forward
) are supported
with the beta command gcloud beta container fleet memberships get-credentials
.
This command lets you use a
preview feature of the Connect gateway.
Note the following requirements:
GKE on Google Cloud isn't supported in the preview.
Clusters must be at version 1.30 or later.
The
kubectl
client must be at version 1.29 or later. If you use version 1.29, you must set the following environment variable:export KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true
kubectl
versions 1.30 and later don't require the environment variable.To check your client version, look at the output of the
kubectl version
command. To install a newer version ofkubectl
see Install tools.
Users and service accounts with the roles/gkehub.gatewayAdmin
IAM role and the cluster-admin
ClusterRole
have the
necessary permissions to run the attach
, cp
, and exec
commands. If users
and service accounts have been granted a custom IAM role or
custom RBAC role, you might need to grant additional permissions. See the
following sections for additional information.
Grant an additional permission if needed
The gkehub.gateway.stream
IAM permission is required to run
the attach
, cp
, and exec
commands through the Connect gateway. This
permission is included in roles/gkehub.gatewayAdmin
.
For users who aren't project owners, or for users or service accounts who
haven't been granted roles/gkehub.gatewayAdmin
in the project, you must
either grant them roles/gkehub.gatewayAdmin
or create a custom role that
includes the other required roles and the gkehub.gateway.stream
permission. For information on how to create a custom role, see
Create and manage custom roles in the
IAM documentation.
Create and apply additional RBAC policies if needed
Users and service accounts with the cluster-admin
ClusterRole
have the
necessary permissions to run the attach
, cp
, and exec
commands.
At a minimum, the following rules are needed to run the commands:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: stream-role
namespace: NAMESPACE # Specify the namespace
rules:
- apiGroups: ["*"]
resources: ["pods/exec", "pods/attach"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: stream-rolebinding
namespace: NAMESPACE # Specify the namespace
roleRef:
apiGroup: "rbac.authorization.k8s.io"
kind: Role
name: stream-role
subjects:
- kind: User
name: EMAIL # Specify the user that should have stream access
namespace: NAMESPACE # Specify the namespace
kubectl exec/cp/attach Troubleshooting
The error returned from running the command is often not clear enough to debug
the issue. In this case, rerun the command with a higher verbosity logging
level, for example: kubectl exec -v 5 ...
.
Connect gateway beta track is not being used
If you get a 400 Bad Request
error, regenerate the connect gateway
kubeconfig file using the gcloud CLI beta track to see if that
fixes the error:
gcloud beta container fleet memberships get-credentials my-membership`
If you still get the 400 Bad Request
error, follow the steps in the
next section.
kubectl environment flag is not set
If the kubectl client version that you are using is 1.29, the environment
variable KUBECTL_REMOTE_COMMAND_WEBSOCKETS
needs to be enabled. If this isn't
enabled, you get a 400 Bad Request
error. To see if the environment variable
is enabled, run the following command:
echo $KUBECTL_REMOTE_COMMAND_WEBSOCKETS`
If the variable isn't set, enable it by setting the following environment variable:
export KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true
kubectl version 1.30 and later doesn't require this flag as it is enabled by default.
kubectl version lower than 1.29 won't work with this feature.
Missing IAM permissions
If you get the error: error reading from error stream...
message, this
could indicate that you haven't been granted the required IAM
permissions to run the command. This feature requires users to have the
gkehub.gateway.stream
IAM permission,
which is included by default in the roles/gkehub.gatewayAdmin
role. See the
IAM permission section for instructions.
Error message generic::failed_precondition: error encountered within the cluster
If you get the error generic::failed_precondition: error encountered within
the cluster
, check the Connect Agent logs in the cluster to identify the
underlying cause:
kubectl logs -n gke-connect -l app=gke-connect-agent --tail -1
The log to look for in the Connect Agent is
failed to create the websocket connection...
.
Missing required RBAC permissions
If the error message in the Connect Agent logs contains 403 Forbidden
,
this indicates your are missing RBAC permissions. You have a set of RBAC
permissions in the cluster to run these kubectl commands.
See the
RBAC policies section
for setting up the required RBAC permissions.
Error message generic::resource_exhausted: Gateway's active_streams quota exhausted
There is a quota limit of 10 active streams per fleet host project. This is
defined under the connectgateway.googleapis.com/active_streams
quota. See
View and manage quotas for instructions on managing
your quotas.
Error message generic::failed_precondition: connection to Agent failed/terminated
If you encounter this error immediately when running the command, there is an issue with the cluster's connection to Google. See the general troubleshooting guide for more information.
If you encounter this error after about 5 to 10 minutes of the session being active, this is an expected limitation of the feature. The connection needs to be re-established.
Troubleshooting
If you're having problems connecting to a cluster through the gateway, you or your administrator can check for the following common issues.
- The server doesn't have a resource type: You may see this error message when the command
kubectl get ns
fails. There are multiple potential reasons for this error. Run yourkubectl
commands in verbose mode to see more details, for examplekubectl get ns -v 10
. - Cannot find active connections for cluster(project: 12345, membership: my-cluster): You may see this error when Connect Agent loses connectivity or isn't installed properly (clusters outside Google Cloud only). To resolve this issue, you need to verify if
gke-connect
namespace exists on the cluster. If thegke-connect
namespace exists on the cluster, see the Connect Troubleshooting page to fix the connectivity issues. - The requested URL was not found on this server: You may see this error when the
kubeconfig
contains an incorrect server address. Make sure the Google Cloud CLI version you're using is the latest version and retry to generate the gatewaykubeconfig
. Don't manually edit thekubeconfig
file, which will cause unexpected errors. - The user identity does not have sufficient permissions to use the gateway API: You need the
roles/gkehub.gatewayAdmin
roles/gkehub.gatewayReader
orroles/gkehub.gatewayEditor
role to use the API. See Grant IAM roles to users in the gateway setup guide for more details. - The Connect agent is not authorized to send the user’s requests: The Connect agent needs to be allowed to forward requests on your behalf, which is specified using an impersonation policy on the cluster. See Configure RBAC authorization in the gateway setup guide for an example of adding a user to the
gateway-impersonate
role. - The user identity does not have sufficient RBAC permissions to perform the operation: You must have appropriate permissions on the cluster to run your chosen operations. See Configure RBAC authorization in the gateway setup guide for an example of adding a user to the appropriate
ClusterRole
. - The user identity does not have sufficient permissions to perform the operation when using Google Groups or third-party support: See Collecting GKE Identity Service logs for instructions on how to inspect logs related to identity information.
- The Connect agent is unhealthy: See the Connect Troubleshooting page to make sure your cluster is connected.
- executable gke-gcloud-auth-plugin not found or no Auth Provider found for name gcp: kubectl versions 1.26 and later may display this error due to changes to kubectl authentication starting with GKE v1.26. Install
gke-gcloud-auth-plugin
and re-rungcloud container fleet memberships get-credentials MEMBERSHIP_NAME
with the latest version of Google Cloud CLI. - Connections to the gateway fail with older versions of the Google Cloud CLI: For GKE clusters, the Connect agent is no longer required for the gateway to function so it does not get installed by default during membership registration. Earlier versions of the Google Cloud CLI (399.0.0 and below) assume the existence of the Connect agent on the cluster. Attempting to use the gateway with these earlier versions may fail on clusters registered with a newer version of the Google Cloud CLI. To resolve this, either upgrade your Google Cloud CLI client to a newer version or re-run the membership registration command with the
--install-connect-agent
flag.
What's next?
- See an example of how to use the Connect gateway as part of your DevOps automation in our Integrating with Cloud Build tutorial.