Setting up Multi-cluster Ingress

This page shows you how to configure Multi Cluster Ingress to route traffic across multiple Google Kubernetes Engine (GKE) clusters in different regions.

Multi Cluster Ingress is a cloud-hosted multi-cluster Ingress controller for GKE clusters that supports deploying shared load balancing resources across clusters and regions.

To learn more about Multi Cluster Ingress, see Multi Cluster Ingress. You can also learn about Deploying Ingress across clusters.

Requirements

Multi Cluster Ingress is supported on:

  • GKE clusters only.
  • Clusters in VPC-native (Alias IP) mode. For more information, see Creating a VPC-native cluster.
  • Clusters that have HTTP-load balancing enabled, which is enabled by default. Note that Multi Cluster Ingress only supports the external HTTP(S) load balancer.
  • GKE clusters enabled with Workload Identity.
  • Cloud SDK version 290 or later (gcloud --version). To update to the latest version, use the command gcloud components update.

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Using gcloud config

  • Set your default project ID:
    gcloud config set project PROJECT_ID
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone COMPUTE_ZONE
  • If you are working with Autopilot or regional clusters, set your default compute region:
    gcloud config set compute/region COMPUTE_REGION
  • Update gcloud to the latest version:
    gcloud components update

Deployment exercise

In this exercise, you deploy components and prepare the infrastructure required to use Multi Cluster Ingress. These steps are required to use Multi Cluster Ingress in your environment. These steps require elevated privileges and should be performed by a GKE administrator.

A summary of the steps in this exercise include:

  1. Enable the required APIs.
  2. Deploy GKE clusters to host workloads.
  3. Group the clusters into a fleet.
  4. Configure Multi Cluster Ingress administrative settings.

The following diagram shows what your environment will look like after you complete the exercise:

Cluster topology showing the relationships between regions, fleet, and project.

In the diagram, there are two GKE clusters named gke-us and gke-eu in the zones europe-west1 and us-central1. The clusters are grouped into a fleet so that the Multi Cluster Ingress controller can recognize them.

Enable APIs

The APIs that you enable for Multi Cluster Ingress depend on the Multi Cluster Ingress pricing that you use.

If Multi Cluster Ingress is the only Anthos capability that you are using, then it might be more price effective to use standalone pricing. The way your project is billed depends on whether the Anthos API (anthos.googleapis.com) is enabled or disabled.

  • If the Anthos API is enabled, then your project is billed according to the number of cluster vCPUs and Anthos pricing.
  • If the Anthos API is disabled, then your project is billed according to the number of backend Multi Cluster Ingress pods in your project.

If your project is using other Anthos on Google Cloud components or capabilities, you should use the Anthos licensing.

You can change the Multi Cluster Ingress billing model from standalone to Anthos or the other way around any time without impacting Multi Cluster Ingress resources or traffic. To change the billing model, you either enable or disable the Anthos API, as in the instructions that follow.

Anthos pricing

To enable Anthos pricing, perform the following steps:

  1. Enable the Anthos, Multi Cluster Ingress, Connect, and GKE APIs in your project:

    gcloud services enable \
        anthos.googleapis.com \
        multiclusteringress.googleapis.com \
        gkehub.googleapis.com \
        container.googleapis.com \
        --project=PROJECT_ID
    

    After anthos.googleapis.com is enabled in your project, any clusters registered to Connect are billed according to Anthos pricing.

Standalone pricing

To enable standalone pricing, perform the following steps:

  1. Confirm that the Anthos API is disabled in your project:

    gcloud services list --project=PROJECT_ID | grep anthos.googleapis.com
    

    Replace PROJECT_ID with the project ID where your GKE clusters are running.

    If the output is an empty response, the Anthos API is disabled in your project and any Multi Cluster Ingress resources are billed using standalone pricing.

  2. Enable the Multi Cluster Ingress, Connect, and GKE APIs in your project:

    gcloud services enable \
        multiclusteringress.googleapis.com \
        gkehub.googleapis.com \
        container.googleapis.com \
        --project=PROJECT_ID
    

Deploy clusters

In this section, you create two GKE clusters named gke-us and gke-eu into the europe-west1 and us-central1 zones.

We recommend that you create your clusters with Workload Identity enabled because it lets the workloads in your cluster authenticate without requiring you to download, manually rotate, or manage Google Cloud service account keys. If you create your clusters without Workload Identity enabled, you must register your clusters into the fleet manually using service account keys.

Workload Identity

  1. Create the gke-us cluster in the us-central1-b zone:

    gcloud container clusters create gke-us \
        --zone=us-central1-b \
        --enable-ip-alias \
        --workload-pool=PROJECT_ID.svc.id.goog \
        --release-channel=stable \
        --project=PROJECT_ID
    
  2. Create the gke-eu cluster in the europe-west1-b zone:

    gcloud container clusters create gke-eu \
        --zone=europe-west1-b \
        --enable-ip-alias \
        --workload-pool=PROJECT_ID.svc.id.goog \
        --release-channel=stable \
        --project=PROJECT_ID
    

Manual

  1. Create the gke-us cluster in the us-central1-b zone:

    gcloud container clusters create gke-us \
        --zone=us-central1-b \
        --enable-ip-alias \
        --release-channel=stable \
        --project=PROJECT_ID
    
  2. Create the gke-eu cluster in the europe-west1-b zone:

    gcloud container clusters create gke-eu \
        --zone=europe-west1-b \
        --enable-ip-alias \
        --release-channel=stable \
        --project=PROJECT_ID
    

Configure cluster credentials

In this section, you configure cluster credentials with memorable names. This makes it easier to switch between clusters when deploying resources across several clusters.

  1. Retrieve the credentials for your clusters:

    gcloud container clusters get-credentials gke-us \
        --zone=us-central1-b \
        --project=PROJECT_ID
    
    gcloud container clusters get-credentials gke-eu \
        --zone=europe-west1-b \
        --project=PROJECT_ID
    

    The credentials are stored locally so that you can use your kubectl client to access the cluster API servers. By default, an auto-generated name is created for the credentials.

  2. Rename the cluster contexts:

    kubectl config rename-context gke_PROJECT_ID_us-central1-b_gke-us gke-us
    kubectl config rename-context gke_PROJECT_ID_europe-west1-b_gke-eu gke-eu
    

Register clusters to a fleet

You must register your clusters to a fleet by using Connect. You can register clusters manually using a Google Cloud service account or through Workload Identity.

Workload Identity

  1. If you have Workload Identity enabled for your clusters, run the following commands to register your clusters:

    gcloud container hub memberships register gke-us \
        --gke-cluster us-central1-b/gke-us \
        --enable-workload-identity \
        --project=PROJECT_ID
    
    gcloud container hub memberships register gke-eu \
        --gke-cluster europe-west1-b/gke-eu \
        --enable-workload-identity \
        --project=PROJECT_ID
    
  2. Confirm that your clusters have successfully registered to Connect:

    gcloud container hub memberships list --project=PROJECT_ID
    

    The output is similar to the following:

    NAME                                  EXTERNAL_ID
    gke-us                                0375c958-38af-11ea-abe9-42010a800191
    gke-eu                                d3278b78-38ad-11ea-a846-42010a840114
    

Manual

If you do not have Workload Identity enabled for your clusters, perform the following steps to register your clusters:

  1. Ensure you have completed the prerequisites for registering a cluster.

  2. Create a service account and download the service account's private key JSON file.

  3. Register your clusters to the fleet using the private key JSON file that you downloaded:

    gcloud container hub memberships register gke-us \
         --gke-cluster us-central1-b/gke-us \
         --service-account-key-file=SERVICE_ACCOUNT_KEY_PATH \
         --project=PROJECT_ID
    
    gcloud container hub memberships register gke-eu \
        --gke-cluster europe-west1-b/gke-eu \
        --service-account-key-file=SERVICE_ACCOUNT_KEY_PATH \
        --project=PROJECT_ID
    

    Replace the following:

    • SERVICE_ACCOUNT_KEY_PATH: the local file path to the service account's private key JSON file that you downloaded in the registration prerequisites. This service account key is stored as a secret named creds-gcp in the gke-connect namespace.
  4. Confirm that your clusters have successfully registered to Connect:

    gcloud container hub memberships list --project=PROJECT_ID
    

    The output is similar to the following:

    NAME                                  EXTERNAL_ID
    gke-us                                0375c958-38af-11ea-abe9-42010a800191
    gke-eu                                d3278b78-38ad-11ea-a846-42010a840114
    

Specify a config cluster

The config cluster is a GKE cluster you choose to be the central point of control for Ingress across the member clusters. Unlike GKE Ingress, the Multi Cluster Ingress controller does not live in a single cluster but is a Google-managed service that watches resources in the config cluster. This GKE cluster is used as a multi-cluster API server to store resources such as MultiClusterIngress and MultiClusterService. Any member cluster can become a config cluster, but there can only be one config cluster at a time.

For more information about config clusters, see Config cluster design.

If the config cluster is down or inaccessible, then MultiClusterIngress and MultiClusterService objects cannot update across the member clusters. Load balancers and traffic can continue to function independently of the config cluster in the case of an outage.

Enabling Multi Cluster Ingress and selecting the config cluster occurs in the same step. The GKE cluster you choose as the config cluster must already be registered to a fleet.

  1. Identify the URI of the cluster you want to specify as the config cluster:

    gcloud container hub memberships list
    

    The output is similar to the following:

    NAME                                  EXTERNAL_ID
    gke-us                                0375c958-38af-11ea-abe9-42010a800191
    gke-eu                                d3278b78-38ad-11ea-a846-42010a840114
    
  2. Enable Multi Cluster Ingress and select gke-us as the config cluster:

    gcloud beta container hub ingress enable \
      --config-membership=gke-us
    

    Note that this process can take a few minutes while the controller is bootstrapping. If successful, the output is similar to this:

    Waiting for Feature to be created...done.
    Waiting for controller to start...done.
    

    If unsuccessful, the command will timeout like below:

    Waiting for controller to start...failed.
    ERROR: (gcloud.alpha.container.hub.ingress.enable) Controller did not start in 2 minutes. Please use the `describe` command to check Feature state for debugging information.
    

    If a failure occurred in the previous step, then check the feature state. It should indicate what exactly went wrong:

    gcloud beta container hub ingress describe
    

    The output is similar to the following:

    createTime: '2021-02-04T14:10:25.102919191Z'
    membershipStates:
      projects/PROJECT_ID/locations/global/memberships/CLUSTER_NAME:
        state:
          code: ERROR
          description: '...is not a VPC-native GKE Cluster.'
          updateTime: '2021-08-10T13:58:50.298191306Z'
      projects/PROJECT_ID/locations/global/memberships/CLUSTER_NAME:
        state:
          code: OK
          updateTime: '2021-08-10T13:58:08.499505813Z'
    

    To find out about more such error messages, see Troubleshooting and operations.

Shared VPC deployment

Multi Cluster Ingress can be deployed for clusters in a Shared VPC network, but all of the participating backend GKE clusters must be in the same project. Having GKE clusters in different projects using the same Cloud Load Balancing VIP is not supported.

In non-Shared VPC networks, the Multi Cluster Ingress controller manages firewall rules to allow health checks to pass from the Cloud Load Balancing to container workloads.

In a Shared VPC network, Multi Cluster Ingress is not capable of managing these firewall rules because firewalling is managed by the host project, which service project administrators do not have access to. The centralized security model of Shared VPC networks intentionally centralizes network control. In Shared VPC networks, a host project administrator must manually create the necessary firewall rules for Cloud Load Balancing traffic on behalf of Multi Cluster Ingress.

The following command shows the firewall rule that you must create if your clusters are on a Shared VPC network. The source ranges are the ranges that the Cloud Load Balancing uses to send traffic to backends in Google Cloud. This rule must exist for the operational lifetime of Multi Cluster Ingress and can only be removed if Multi Cluster Ingress or Cloud Load Balancing to GKE load balancing is no longer used.

If your clusters are on a Shared VPC network, create the firewall rule:

gcloud compute firewall-rules create FIREWALL_RULE_NAME \
    --project HOST_PROJECT \
    --network SHARED_VPC \
    --direction INGRESS \
    --allow tcp:0-65535 \
    --source-ranges 130.211.0.0/22,35.191.0.0/16

Replace the following:

  • FIREWALL_RULE_NAME: the name of the new firewall rule.
  • HOST_PROJECT: the ID of the Shared VPC host project.
  • SHARED_VPC: the name of the Shared VPC network.

Known issues

InvalidValueError for field config_membership

A known issue prevents the gcloud command-line tool from interacting with Multi Cluster Ingress. This issue was introduced in version 346.0.0 and was fixed in version 348.0.0. We do not recommend using the gcloud tool versions 346.0.0 and 347.0.0 with Multi Cluster Ingress.

Invalid value for field 'resource'

Google Cloud Armor cannot communicate with Multi Cluster Ingress config clusters running on the following GKE versions:

  • 1.18.19-gke.1400 and later
  • 1.19.10-gke.700 and later
  • 1.20.6-gke.700 and later

When you configure a Google Cloud Armor security policy, the following message appears:

Invalid value for field 'resource': '{"securityPolicy": "global/securityPolicies/"}': The given policy does not exist

To avoid this issue, upgrade your config cluster to version 1.21 or later, or use the following command to update the BackendConfig CustomResourceDefinition:

kubectl patch crd backendconfigs.cloud.google.com --type='json' -p='[{"op": "replace", "path": "/spec/versions/1/schema/openAPIV3Schema/properties/spec/properties/securityPolicy", "value":{"properties": {"name": {"type": "string"}}, "required": ["name" ],"type": "object"}}]'

What's next