Registering a cluster

This page provides an overview of how to register your Kubernetes clusters to a Google Cloud fleet. You can find out more about logically grouping your clusters in fleets and how it helps you manage your infrastructure and applications in Introducing fleets.

Anthos clusters on VMware, Anthos clusters on bare metal, and Anthos clusters on AWS are automatically registered to your project fleet at cluster creation time, although you may very rarely need to update a cluster registration. GKE clusters on Google Cloud and attached clusters must be registered manually following the instructions in this page.

Before you begin

Check the prerequisites for your chosen cluster type. For detailed instructions, see our Prerequisites guide.

All registrations

  1. Ensure that you have the relevant command line tools installed.
  2. Enable the required APIs in your Google Cloud project.
  3. Grant the required Google Cloud IAM roles to the user registering the cluster.
  4. Grant the cluster-admin RBAC role to the user registering the cluster.

GKE clusters

GKE cluster registrations may also require the following:

Clusters outside Google Cloud

All manual registrations outside Google Cloud, such as attached clusters, also require the following:

  1. Ensure network connectivity for access to Google Cloud API and service endpoints.
  2. Set up the identity you want the Connect Agent to use to authenticate to Google Cloud

You may also have special requirements for some attached cluster types.

About fleet Workload Identity

GKE lets you configure your clusters to use Workload Identity when authenticating to Google Cloud APIs and services. GKE Workload Identity lets the workloads in your cluster authenticate to Google without requiring you to download, manually rotate, and generally manage Google Cloud service account keys. Instead, workloads authenticate using short-lived tokens generated by the cluster. All clusters with GKE Workload Identity enabled use their project's workload identity pool when issuing IDs, which allows Identity and Access Management to trust and understand Kubernetes service account credentials. You can learn more about GKE Workload Identity in Using Workload Identity.

With fleets, registered clusters can get the additional benefit of using fleet Workload Identity. Registered clusters with Workload Identity enabled on their fleet membership can use the fleet-wide fleet workload identity pool, making it simpler to set up authentication to Google APIs and other services across your fleet and across multiple projects. Fleet Workload Identity can also be used by the Connect Agent on some cluster types to authenticate to Google as part of fleet membership, and is required to use some Anthos features that work across projects, such as Anthos Service Mesh. You can learn more about using fleet Workload Identity with your registered clusters (for example, to authenticate to Google APIs) in Fleet Workload Identity.

We recommend where possible registering the cluster with fleet Workload Identity enabled. Currently fleet Workload Identity is available for GKE clusters, some attached cluster types, and Anthos clusters outside Google Cloud from Anthos 1.8 onwards. For details, see the following registration instructions.

Register a cluster

Your options for registering clusters depend on your cluster type and your particular project and application needs.

Register GKE clusters

All of these options allow you to enable fleet Workload Identity if required.

  • The simplest and quickest way to register a GKE cluster in an Anthos-enabled project to your project fleet is to register it from the Anthos clusters page in the Cloud Console. Using this approach immediately lets you view your cluster as part of your fleet in the console, and use fleet-enabled features such as Anthos Config Management

  • You can create and register a GKE cluster (with or without Anthos) by using Terraform or Config Connector.

  • You can register a GKE cluster (with or without Anthos) by using the gcloud command-line tool. Registering by using gcloud command-line tool installs the Connect Agent on your cluster, allowing you to use the Connect gateway to standardize authentication across environments with Google IDs.

To register a GKE cluster:

Console

To register a cluster:

  1. (Optional) If you want your registered cluster to use fleet Workload Identity, ensure that GKE Workload Identity is enabled on the cluster. Clusters with GKE Workload Identity automatically have fleet Workload Identity enabled when you register them from the Cloud Console.
  2. In the Cloud Console, go to the Anthos Clusters page. This page shows all your registered clusters.

    Go to the Anthos Clusters page

  3. Click Register existing cluster.

  4. Click Register next to the unregistered cluster that you want to add to your fleet.

Terraform

You can register GKE clusters with Terraform using the google-beta provider. You can find a complete reference for the gke_hub_membership resource used to configure registration in the terraform registry.

Register a GKE cluster

To register the cluster, use the following blocks in your configuration.

  1. Specify the google-beta provider as a required provider, as in the following snippet. The version should be above 3.62.0.

    terraform {
      required_providers {
        google-beta = {
          source = "hashicorp/google-beta"
          version = "3.67.0"
        }
      }
    }
    

    This provider is required if you want to use gke_hub_membership.

  2. Set the following default values for the provider:

    provider "google-beta" {
      credentials = file("SERVICE_ACCOUNT_KEY_PATH")
      project = "PROJECT_ID"
    }
    

    Replace the following:

    • SERVICE_ACCOUNT_KEY_PATH: the local filepath to the service account's private key JSON file downloaded as part of Prerequisites. This service account key is stored as a secret named creds-gcp in the gke-connect namespace. This service account key enables Terraform to access your Google Cloud account.
    • PROJECT_ID: the default project that you choose to provision your Terraform resources.
  3. Create a GKE cluster (optional)

    If you want to register an existing GKE Cluster, skip this step.

    resource "google_container_cluster" "TF_CLUSTER_RESOURCE_NAME" {
      provider = google-beta
      name               = "CLUSTER_NAME"
      location           = "ZONE"
      initial_node_count = 1
    }
    

    Replace the following:

    • TF_CLUSTER_RESOURCE_NAME: the name that you choose to uniquely identify the Terraform google_container_cluster resource created by this block.
    • CLUSTER_NAME: the name that you choose to uniquely represent the cluster you created.
    • ZONE: the default zone that you choose to provision your GKE cluster resource. For example: us-central1-a.
  4. Register a membership for the GKE cluster.

    resource "google_gke_hub_membership" "TF_MEMBERSHIP_RESOURCE_NAME" {
      provider = google-beta
      project_id = "HUB_PROJECT_ID"
      membership_id = "MEMBERSHIP_NAME"
      endpoint {
        gke_cluster {
         resource_link = "//container.googleapis.com/CLUSTER_RESOURCE_NAME"
        }
      }
    }
    

    Replace the following:

    • TF_MEMBERSHIP_RESOURCE_NAME: the name that you choose to uniquely identify the Terraform google_gke_hub_membership resource created by this block.
    • MEMBERSHIP_NAME: the membership name that you choose to uniquely represent the cluster being registered to the fleet.
    • CLUSTER_RESOURCE_NAME: the Google Cloud resource name for the GKE cluster. For example: projects/my-project/zones/us-west1-a/clusters/my-cluster. If the cluster is provisioned with Terraform, this is ${google_container_cluster.TF_CLUSTER_RESOURCE_NAME.id}.
    • HUB_PROJECT_ID: If this is not set, the cluster's membership will be created in the default project you specified earlier (PROJECT_ID). Set this field if you want to register the GKE cluster to a different project.

Enable Workload Identity on a registered GKE cluster

Enabling Workload Identity uses the same configuration as registering a cluster, with the following additional steps.

  1. Enable GKE Workload Identity by adding the following in the relevant google_container_cluster resource block. You need to do this for both new and existing clusters.

    workload_identity_config {
      identity_namespace = "PROJECT_ID.svc.id.goog"
    }
    

    Replace the following:

    • PROJECT_ID: the default project that you choose to provision your Terraform resources.
  2. Enable Fleet Workload Identity by adding the following in the relevant google_gke_hub_membership resource block.

    authority {
      issuer = "https://container.googleapis.com/v1/CLUSTER_RESOURCE_NAME"
    }
    

    Replace the following:

    • CLUSTER_RESOURCE_NAME: the Google Cloud resource for the GKE cluster. For example: projects/my-project/zones/us-west1-a/clusters/my-cluster. If the cluster is provisioned with Terraform, this is ${google_container_cluster.TF_CLUSTER_RESOURCE_NAME.id}.

Config Connector

If you have the Config Connector add-on installed, you can use Config Connector to register GKE clusters. Ensure that you have a version of Config Connector above 1.47.0. You can find a complete reference for the GKEHubMembership resource in the Config Connector Reference.

Register a GKE cluster

To register the cluster, first specify where you want to create your resources, following the instructions in the Config Connector guide. Then create a YAML file to register and (optionally) create a cluster, as follows:

Create and register a GKE cluster

  1. Create a GKE cluster:

    apiVersion: container.cnrm.cloud.google.com/v1beta1
    kind: ContainerCluster
    metadata:
      name: CLUSTER_NAME
    spec:
      location: ZONE
      initialNodeCount: 1
    

    Replace the following:

    • CLUSTER_NAME: the name that you chose to uniquely represent the cluster you created with Config Connector.
    • ZONE: the zone that you chose to provision your GKE cluster resource. For example: us-central1-a.
  2. Register a membership for the GKE cluster.

    apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
    kind: GKEHubMembership
    metadata:
      name: MEMBERSHIP_NAME
    spec:
      location: global
      endpoint:
        gkeCluster:
          resourceRef:
            name: CLUSTER_NAME
    

    Replace the following:

    • MEMBERSHIP_NAME: the membership name that you chose to uniquely represent the cluster being registered to the fleet.
    • CLUSTER_NAME: the name that you chose to uniquely represent the cluster you created with Config Connector.

Register the cluster in a different project

Add the following to the metadata field of GKEHubMembership resource.

   metadata:
     annotations:
       cnrm.cloud.google.com/project-id: HUB_PROJECT_ID

Replace the following:

  • HUB_PROJECT_ID: the different project you chose to register the GKE cluster.

Register an existing cluster

Use the following configuration if you want to register any existing cluster, regardless of how it was created. In this case you need to specify the full resource name of the cluster so that Config Connector can find it. Replace the resourceRef field of GKEHubMembership resource with:

   resourceRef:
     external: //container.googleapis.com/CLUSTER_RESOURCE_NAME

Replace the following:

  • CLUSTER_RESOURCE_NAME: the Google Cloud resource name for the GKE cluster. For example: projects/my-project/zones/us-west1-a/clusters/my-cluster.

Enable Workload Identity on a registered GKE cluster

Enabling Workload Identity uses the same configuration as registering a cluster, with the following additional steps.

  1. Enable GKE Workload Identity by adding the following in the spec field of ContainerCluster resource. You need to do this for both new and existing clusters.

    spec:
      workloadIdentityConfig:
        identityNamespace: PROJECT_ID.svc.id.goog
    

    Replace the following:

  2. Enable Fleet Workload Identity by adding the following in the spec field of GKEHubMembership resource block.

    spec:
      authority:
        issuer: https://container.googleapis.com/v1/projects/PROJECT_ID/locations/ZONE/clusters/CLUSTER_NAME
    }
    

    Replace the following:

    • PROJECT_ID: the default project you chose through namespace annotation to create your GKE cluster resource.
    • ZONE: the zone that you chose to provision your GKE cluster resource. For example: us-central1-a.
    • CLUSTER_NAME: the name that you chose to uniquely represent the cluster you created.

gcloud

Registering a cluster with the gcloud command-line tool requires the Cloud SDK version 281.0.0 or higher

Registering the cluster from the command line installs the Connect Agent on the cluster, which can either use Workload Identity or a Google Cloud service account for authentication.

Register a GKE cluster with fleet Workload Identity (recommended)

  1. Ensure that the cluster has GKE Workload Identity enabled, following the instructions in our prerequisites.

  2. Run either of the following commands to register the cluster:

    gcloud container hub memberships register MEMBERSHIP_NAME \
     --gke-uri=GKE_URI \
     --enable-workload-identity
    

    or

    gcloud container hub memberships register MEMBERSHIP_NAME \
     --gke-cluster=GKE_CLUSTER \
     --enable-workload-identity
    

    Replace the following:

    • MEMBERSHIP_NAME: the membership name that you choose to uniquely represent the cluster being registered to the fleet.
    • GKE_URI: the URI of the GKE cluster, for example: https://container.googleapis.com/v1/projects/my-gke-project/locations/us-central1-a/clusters/my-gke-cluster. You can obtain the URI by running gcloud container clusters list --uri.
    • GKE_CLUSTER: the location/name of the GKE cluster from the current project. The location can be a zone or a region, for example: us-central1-a/my-gke-cluster.

Register a GKE cluster with a Service Account

  1. Ensure that you have created a service account for use by the Connect Agent, as described in our prerequisites.

  2. Run either of the following commands:

    gcloud container hub memberships register MEMBERSHIP_NAME \
     --gke-uri=GKE_URI \
     --service-account-key-file=SERVICE_ACCOUNT_KEY_PATH
    

    or

    gcloud container hub memberships register MEMBERSHIP_NAME \
     --gke-cluster=GKE_CLUSTER \
     --service-account-key-file=SERVICE_ACCOUNT_KEY_PATH
    

    Replace the following:

    • MEMBERSHIP_NAME: the membership name that you choose to uniquely represent the cluster being registered to the fleet.
    • SERVICE_ACCOUNT_KEY_PATH: the local filepath to the service account's private key JSON file downloaded as part of Prerequisites. This service account key is stored as a secret named creds-gcp in the gke-connect namespace.
    • GKE_URI: the URI of the GKE cluster, for example: https://container.googleapis.com/v1/projects/my-gke-project/locations/us-central1-a/clusters/my-gke-cluster. You can obtain the URI by running gcloud container clusters list --uri.
    • GKE_CLUSTER: the location/name of the GKE cluster from the current project. The location can be a zone or a region, for example: us-central1-a/my-gke-cluster.

Register clusters outside Google Cloud

Registering clusters outside Google Cloud installs the Connect Agent in the cluster to manage the cluster's connection to its fleet. The agent needs an identity to authenticate to Google, which can be a Google Cloud service account or (recommended) fleet Workload Identity. You can find out much more about the Connect Agent in the Connect overview.

  • Attached clusters must be manually registered following the instructions in this section. They can be registered with fleet Workload Identity enabled if they meet the relevant requirements. Otherwise, register attached clusters with a Google Cloud service account for authentication.

  • Anthos clusters on Azure currently must be manually registered following the instructions in Register your cluster with Connect. These instructions enable fleet Workload Identity by default.

  • Anthos clusters on VMware, Anthos clusters on bare metal, and Anthos clusters on AWS are automatically registered to your project fleet at cluster creation time. You may on very rare occasions need to update a cluster's registration, for example if you have accidentally unregistered it. As of Anthos 1.8, all these cluster types automatically enable fleet Workload Identity when registered. Existing registered clusters are updated to use fleet Workload Identity when they are upgraded to Anthos 1.8. Note that for these cluster types you still need to set up a service account for registration—after your initial registration, the Connect Agent uses fleet Workload Identity to authenticate to Google.

To register or update a registration with the Connect Agent:

gcloud

Registering a cluster with the gcloud command-line tool requires the Cloud SDK version 281.0.0 or higher

These instructions apply to attached clusters, although they can also be used to update registration for Anthos clusters on AWS, Anthos clusters on VMware version 1.7 and earlier, Anthos clusters on bare metal, and any other Anthos clusters.

Run the following command:

 gcloud container hub memberships register MEMBERSHIP_NAME \
   --context=KUBECONFIG_CONTEXT \
   --kubeconfig=KUBECONFIG_PATH \
   --service-account-key-file=SERVICE_ACCOUNT_KEY_PATH

Replace the following:

  • MEMBERSHIP_NAME: the membership name that you choose and that is used to uniquely represent the cluster being registered to the fleet.
  • SERVICE_ACCOUNT_KEY_PATH: the local filepath to the service account's private key JSON file downloaded as part of Prerequisites. This service account key is stored as a secret named creds-gcp in the gke-connect namespace.
  • KUBECONFIG_CONTEXT: the cluster context of the cluster being registered as it appears in the kubeconfig file. You can get this value from the command line by running kubectl config current-context.
  • KUBECONFIG_PATH: the local filepath where your kubeconfig containing an entry for the cluster being registered is stored. This defaults to $KUBECONFIG if that environment variable is set; otherwise, this defaults to $HOME/.kube/config.

Register an attached cluster with fleet Workload Identity

To register an attached cluster with fleet Workload Identity enabled, run one of the following commands. For more information on which attached cluster types can use this feature and any additional requirements, see Attached cluster prerequisites.

kind, OpenShift clusters:

 gcloud container hub memberships register MEMBERSHIP_NAME \
   --context=KUBECONFIG_CONTEXT \
   --kubeconfig=KUBECONFIG_PATH \
   --enable-workload-identity \
   --has-private-issuer

EKS clusters:

  1. Get the OIDC provider URL for your cluster and ensure it is publicly visible. If no provider exists, follow the instructions in Create an IAM OIDC provider for your cluster then run the command again.

    aws eks describe-cluster --name MEMBERSHIP_NAME --query "cluster.identity.oidc.issuer" --output text
    
  2. Run the following command, replacing OIDC_URL with the URL returned by the previous command:

     gcloud container hub memberships register MEMBERSHIP_NAME \
      --context=KUBECONFIG_CONTEXT \
      --kubeconfig=KUBECONFIG_PATH \
      --enable-workload-identity \
      --public-issuer-url=OIDC_URL
    

gkectl

Update an Anthos clusters on VMware registration

From Anthos 1.8, you can only update an Anthos clusters on VMware registration by updating your cluster configuration file with gkectl. To reregister a cluster after it has been created, ensure the gkeConnect section of your cluster configuration file is filled in correctly, and run gkectl update cluster.

Console

Generate a registration command

You can use the Cloud Console to help generate a gcloud registration command for manually registering clusters. These instructions apply to attached clusters, as well as updating registration for Anthos clusters on AWS, Anthos clusters on VMware version 1.7 and earlier, and any other Anthos clusters.

To register a cluster:

  1. In the Google Cloud Console, go to the Anthos Clusters page. This page shows all your registered clusters.

    Go to the Anthos Clusters page

  2. Click Register existing cluster.

  3. Click Add external cluster.

  4. Enter the name of the cluster that you want to register in the Cluster name field.

  5. Optional: Add Google Cloud labels to your cluster.

  6. Click Generate registration command.

  7. In Cloud Shell or wherever you have saved your service account credentials, edit and run the gcloud command that is displayed on the page. You need to specify the following values:

    • The CLUSTER_CONTEXT is the cluster's context as it appears in the kubeconfig file. You can get this value from the command line by running kubectl config current-context.
    • The KUBECONFIG_PATH is the local filepath where your kubeconfig file is stored. This defaults to $KUBECONFIG if that environment variable is set; otherwise, it defaults to $HOME/.kube/config.
    • The LOCAL_KEY_PATH is the path to your service account key file.

    Running this command deploys the Connect Agent in your user cluster. When the Connect Agent connects to Google Cloud and your cluster is registered, a success message is displayed on the page.

  8. Click Set labels, or click Skip if you didn't set any labels.

Advanced registration options (command line only)

Download the Connect Agent manifest

To download the Connect Agent installation manifest without deploying the agent, for example if you want to examine or edit the manifest before installation, pass in the --manifest-output-file flag to the gcloud container hub memberships register command. For example:

--manifest-output-file=[MANIFEST_FILE_PATH]

where [MANIFEST_FILE_PATH] is the local filepath where you want the Connect Agent installation manifest to be stored.

Using this option will not deploy the Connect Agent into the cluster. To deploy the Connect Agent, manually apply the downloaded manifest to your cluster.

Using a proxy server

To configure a proxy server, pass in the --proxy flag to gcloud container hub memberships register command. For example:

--proxy=[URL]

where [URL] is the proxy address.

The Connect Agent supports CONNECT-based HTTP and HTTPS proxies only, and accepts IP addresses and hostnames. Be sure to specify the protocol corresponding to the proxy type in the URL. For example, to pass in an HTTPS hostname:

--proxy=https://mycorpproxy.com:443

Unless you specify otherwise, Connect Agent uses port 3128 for the proxy.

If your proxy requires authorization, ensure that you pass in your credentials, such as:

--proxy=http://user:password@10.10.10.10:8888

Installing Connect Agent in a cluster with Windows and Linux nodes

The Connect Agent has to run on a Linux node. If you are installing in a mixed cluster with both Linux and Windows nodes, you can ensure that the Connect Agent is deployed on to a Linux node by adding an appropriate node selector to the deployment definition.

Run the following command to update the deployment with the appropriate node selector:

kubectl patch deployment \
$(kubectl get deployment -o=jsonpath='{.items[*].metadata.name}' -n gke-connect) \
-p '{"spec":{"template":{"spec":{"nodeSelector":{"kubernetes.io/os":"linux"}}}}}' -n gke-connect

To validate that the update succeeded run the following command:

kubectl get deployment -o=jsonpath='{.items[].spec.template.spec.nodeSelector}' -n gke-connect

The command should return:

{"kubernetes.io/os":"linux"}

What's next