Configuring Traffic Director with Shared VPC on multiple GKE clusters

This tutorial shows how to configure Traffic Director for service-to-service communication between services that are running in multiple projects on a Shared VPC network. The tutorial is intended for network administrators and Google Kubernetes Engine (GKE) platform administrators.

Enterprises typically divide their resources into multiple Google Cloud projects, which separate resources by teams, applications, or environments—for example, development, staging, and production environments. Every Google Cloud project comes with its own default Virtual Private Cloud (VPC) used by Google Cloud resources, such as Compute Engine instances and GKE clusters.

Instead of managing multiple VPCs in each project, Shared VPC lets you centralize all VPC resources—subnets, firewall rules, load balancers—in a single host project, and then share networking resources in other projects called service projects. This approach lets network administrators manage all networking resources centrally, while providing project-level partitioning for teams, applications, and environments. To allow access to the networking resources required for your projects, you use service accounts to configure Identity and Access Management (IAM) permissions for service projects.

The following diagram shows a Shared VPC host project sharing four subnets to two service projects (two subnets each), including the IP addresses for each project.

A Shared VPC host project sharing four subnets to two service projects.

In some cases, service-to-service communication is required between services that are running in multiple service projects. As long as the instances across service projects are connected to the same VPC, you can establish reachability by configuring VPC firewall rules. For services that are running in GKE clusters, you can use Traffic Director to create a service mesh between services that are running in multiple GKE clusters, in multiple projects, and in multiple regions. Traffic Director functions as a service mesh control plane that configures the Envoy proxies running as sidecar containers in each Pod that is running in the GKE clusters. The Envoy proxy sidecars intercept all requests by the application container that's running in each Pod and connect services between GKE clusters. You can also use Traffic Director for services running in Compute Engine instances.

In this tutorial, you create and set up Shared VPC with two service projects. You create two GKE clusters, one in each service project in different regions, by using Shared VPC. You configure a sample service with Envoy proxies running as sidecars. You also configure Traffic Director to configure the Envoy proxies for service-to-service communication between the GKE clusters.

The following diagram shows a Shared VPC host project that's sharing four subnets to two service projects, with two subnets for each service project.

A Shared VPC host project sharing four
subnets to two service projects, with two subnets for each service project.

Each service project contains a GKE cluster that includes a sample application Pod and a BusyBox Pod. Envoy sidecar proxies that are running in each application Pod connect over the network, across regions and clusters.

Objectives

  • Create one host project and two service projects.
  • Create a shared VPC in the host project with two subnets. There is one subnet for each of the two service projects used by the GKE clusters.
  • Create two GKE clusters, one in each service project, by using the subnets from the shared VPC.
  • Deploy a sample application by using httpbin in both clusters with the Envoy sidecar proxies. Deploy a BusyBox Pod in both clusters to test connectivity.
  • Configure Traffic Director to allow network connectivity between all services.
  • Confirm that services can communicate across GKE clusters and across service projects.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

To configure a shared VPC in Google Cloud, ensure that you have the following resources and roles on Google Cloud:

  • An organization resource, which is the root node in the Google Cloud resource hierarchy and is the hierarchical super node of projects. To create an organization resource, follow the instructions in Getting an organization resource. A shared VPC can be created only between projects in an organization resource.
  • The following IAM roles at the organization level for your user:
    • Project Creator (roles/resourcemanager.projectCreator). Create host and service projects.
    • Editor (roles/editor). Create and delete resources in Cloud projects in the organization.
    • Project Billing Manager (roles/billing.projectManager). Assign billing accounts to projects in the organization. This role doesn't let you create billing accounts, but it does let you assign existing billing accounts to projects. If you need to create a new billing account, use the Billing Account Administrator (roles/billing.admin) role.
    • Compute Admin (roles/compute.admin). Create networking resources (VPCs, subnets, firewall rules) for all projects across the organization.
    • Project IAM Admin (roles/resourcemanager.projectIamAdmin). Administer IAM policies for all projects across the organization.
    • Project Deleter (roles/resourcemanager.projectDeleter). (Optional) Delete projects in the organization.

To set up the preceding resources and roles, follow these steps:

  1. Go to the Manage resources page.

    Go to the Manage resources page

  2. Select your Google Cloud organization, and then copy the organization ID number from the ID column into a notepad or file. You use this ID number later in this tutorial.

  3. On the same line as as your organization, click , and then click Settings.

  4. On the IAM and Admin page, click IAM. Ensure that you have the proper IAM roles for your user at the organization level.

    A list of roles is listed for a specific user.

  5. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

  6. Confirm that you're logged in with the same username that has the roles that you configured in step 4:

    gcloud info --format=json | jq '.config.account'
    
  7. Get the billing account that you use for the Cloud projects:

    gcloud beta billing accounts list
    

    The output is similar to the following:

    ACCOUNT_ID            NAME                    OPEN  MASTER_ACCOUNT_ID
    1337A-1337A-1337A     My Billing Account      True
    

    Copy the billing account ID into a notepad or file. You need this number in the next step.

  8. Set environment variables for your organization ID and billing account ID:

    export ORG_ID=ORG_ID_NUMBER
    export BILLING_ID=BILLING_ACCOUNT_ID
    

    Replace the following:

    • ORG_ID_NUMBER: your organization ID from step 2
    • BILLING_ACCOUNT_ID: your billing account ID from step 7

In the next section, you create resources in your organization.

Setting up your environment

In this section, you create three projects in your Google Cloud organization—the host project where you configure a shared VPC, and two service projects that use the shared VPC resources.

Create projects and billing

  1. In Cloud Shell, create three projects in your Google Cloud organization:

    export RANDOM_PERSIST=$RANDOM
    export HOST_PROJECT="td-${ORG_ID}-${RANDOM_PERSIST}-host-prj"
    export SVC_PROJECT_1="td-${ORG_ID}-${RANDOM_PERSIST}-svc-prj1"
    export SVC_PROJECT_2="td-${ORG_ID}-${RANDOM_PERSIST}-svc-prj2"
    gcloud projects create ${HOST_PROJECT} --organization=${ORG_ID}
    gcloud projects create ${SVC_PROJECT_1} --organization=${ORG_ID}
    gcloud projects create ${SVC_PROJECT_2} --organization=${ORG_ID}
    

    The project variables that are defined in the preceding commands must represent the project IDs of the three projects.

  2. Enable billing for all three projects:

    gcloud beta billing projects link ${HOST_PROJECT} \
        --billing-account ${BILLING_ID}
    gcloud beta billing projects link ${SVC_PROJECT_1} \
        --billing-account ${BILLING_ID}
    gcloud beta billing projects link ${SVC_PROJECT_2} \
        --billing-account ${BILLING_ID}
    
  3. Enable the GKE and Traffic Director APIs on all projects:

    gcloud services enable container.googleapis.com trafficdirector.googleapis.com --project $HOST_PROJECT
    gcloud services enable container.googleapis.com trafficdirector.googleapis.com --project $SVC_PROJECT_1
    gcloud services enable container.googleapis.com trafficdirector.googleapis.com --project $SVC_PROJECT_2
    

Obtain Google Cloud service accounts

Every Cloud project comes with several default service accounts that include default permissions to various resources. Following are the three Google-managed default service accounts that you need for this tutorial:

You need the service account emails from both service projects in order to create IAM permissions in the host project. These permissions enable the service projects to consume resources—in this case, a shared VPC—from the host project. These service accounts have a standard naming convention that uses the project number.

  • In Cloud Shell, get the two service account emails from both service projects:

    export SVC_PROJECT_1_NUM=$(gcloud projects list --format=json --filter="${SVC_PROJECT_1}" | jq -r '.[].projectNumber')
    export SVC_PROJECT_2_NUM=$(gcloud projects list --format=json --filter="${SVC_PROJECT_2}" | jq -r '.[].projectNumber')
    
    export SVC_PROJECT_1_COMPUTE_SA="${SVC_PROJECT_1_NUM}-compute@developer.gserviceaccount.com"
    export SVC_PROJECT_2_COMPUTE_SA="${SVC_PROJECT_2_NUM}-compute@developer.gserviceaccount.com"
    
    export SVC_PROJECT_1_GKE_SA="service-${SVC_PROJECT_1_NUM}@container-engine-robot.iam.gserviceaccount.com"
    export SVC_PROJECT_2_GKE_SA="service-${SVC_PROJECT_2_NUM}@container-engine-robot.iam.gserviceaccount.com"
    
    export SVC_PROJECT_1_API_SA="${SVC_PROJECT_1_NUM}@cloudservices.gserviceaccount.com"
    export SVC_PROJECT_2_API_SA="${SVC_PROJECT_2_NUM}@cloudservices.gserviceaccount.com"
    

Create a VPC network and subnets

  1. In Cloud Shell, create a VPC network in the host project called shared-vpc. Create a custom mode VPC so that you can create subnets manually.

    gcloud compute networks create shared-vpc \
        --subnet-mode custom \
        --project $HOST_PROJECT
    
  2. Create two subnets in the shared-vpc resource. One subnet will be used by the GKE clusters in each of the two service projects. Create one primary range for the GKE nodes and two secondary ranges, one for GKE Pods and a second for GKE Services.

    export REGION_1=us-west1
    export ZONE_1=us-west1-a
    export REGION_2=us-central1
    export ZONE_2=us-central1-a
    
    gcloud compute networks subnets create subnet-1 \
        --project $HOST_PROJECT \
        --network shared-vpc \
        --range 10.0.4.0/22 \
        --region ${REGION_1} \
        --secondary-range subnet-1-services=10.0.32.0/20,subnet-1-pods=10.4.0.0/14
    
    gcloud compute networks subnets create subnet-2 \
        --project $HOST_PROJECT \
        --network shared-vpc \
        --range 172.16.4.0/22 \
        --region ${REGION_2} \
        --secondary-range subnet-2-services=172.16.16.0/20,subnet-2-pods=172.20.0.0/14
    
  3. Create firewall rules for all IP address ranges within the VPC network so that the GKE Pods can communicate with each other on all ports and protocols. You can restrict this rule to allow only the required ports.

    gcloud compute firewall-rules create allow-gke-svc-1-all \
        --project ${HOST_PROJECT} \
        --network shared-vpc \
        --allow tcp,udp \
        --direction INGRESS \
        --source-ranges 10.0.4.0/22,10.4.0.0/14,10.0.32.0/20
    
    gcloud compute firewall-rules create allow-gke-svc-2-all \
        --project ${HOST_PROJECT} \
        --network shared-vpc \
        --allow tcp,udp \
        --direction INGRESS \
        --source-ranges 172.16.4.0/22,172.20.0.0/14,172.16.16.0/20
    
    gcloud compute firewall-rules create allow-healthchecks \
        --project ${HOST_PROJECT} \
        --network shared-vpc \
        --allow tcp,udp \
        --direction INGRESS \
        --source-ranges 130.211.0.0/22,35.191.0.0/16
    

    The first two of the preceding firewall rules allow subnet-1 and subnet-2 networking connectivity in the shared VPC network. This connectivity allows Pod-to-Pod IP communication across clusters. The third firewall rule, allow-healthchecks, enables Cloud Load Balancing to check the health of the backend services before sending traffic to them. Cloud Load Balancing uses well-known IP address ranges for health checks that are configured as source-ranges for the IP addresses in the firewall rule.

Creating a shared VPC

In this section, you enable and configure the shared VPC in the host project. You also configure the IAM permissions that enable the service projects to use the resources from the shared VPC.

Enable the shared VPC

  1. In Cloud Shell, enable the shared VPC in the host project:

    gcloud compute shared-vpc enable ${HOST_PROJECT}
    
  2. Add both service projects to the shared VPC:

    gcloud compute shared-vpc associated-projects add \
        ${SVC_PROJECT_1} \
        --host-project ${HOST_PROJECT}
    
    gcloud compute shared-vpc associated-projects add \
        ${SVC_PROJECT_2} \
        --host-project ${HOST_PROJECT}
    

Configure IAM roles

  1. In Cloud Shell, create a working folder (WORKDIR) where you create the files associated with this section:

    mkdir -p ~/td-shared-vpc
    cd ~/td-shared-vpc
    export WORKDIR=$(pwd)
    
  2. Configure IAM permissions in the host project so that service projects can use the resources in the shared VPC.

    In this step, you configure the IAM permissions so that subnet-1 is accessible by service project 1 and subnet-2 is accessible by service project 2. You assign the Compute Network User IAM role (roles/compute.networkUser) to both the Compute Engine compute default service account and the Google Cloud API service account in each service project for each subnet.

    1. For service project 1, configure IAM permissions for subnet-1:

      export SUBNET_1_ETAG=$(gcloud beta compute networks subnets get-iam-policy subnet-1 --project ${HOST_PROJECT} --region ${REGION_1} --format=json | jq -r '.etag')
      
      cat > subnet-1-policy.yaml <<EOF
      bindings:
      - members:
        - serviceAccount:${SVC_PROJECT_1_API_SA}
        - serviceAccount:${SVC_PROJECT_1_GKE_SA}
        role: roles/compute.networkUser
      etag: ${SUBNET_1_ETAG}
      EOF
      
      gcloud beta compute networks subnets set-iam-policy subnet-1 \
      subnet-1-policy.yaml \
          --project ${HOST_PROJECT} \
          --region ${REGION_1}
      
    2. For service project 2, configure IAM permissions for subnet-2:

      export SUBNET_2_ETAG=$(gcloud beta compute networks subnets get-iam-policy subnet-2 --project ${HOST_PROJECT} --region ${REGION_2} --format=json | jq -r '.etag')
      
      cat > subnet-2-policy.yaml <<EOF
      bindings:
      - members:
        - serviceAccount:${SVC_PROJECT_2_API_SA}
        - serviceAccount:${SVC_PROJECT_2_GKE_SA}
        role: roles/compute.networkUser
      etag: ${SUBNET_2_ETAG}
      EOF
      
      gcloud beta compute networks subnets set-iam-policy subnet-2 \
      subnet-2-policy.yaml \
          --project ${HOST_PROJECT} \
          --region ${REGION_2}
      
  3. For each service project, you must grant the Kubernetes Engine Host Service Agent User IAM role (roles/container.hostServiceAgentUser) to the GKE service account in the host project:

    gcloud projects add-iam-policy-binding ${HOST_PROJECT} \
        --member serviceAccount:${SVC_PROJECT_1_GKE_SA} \
        --role roles/container.hostServiceAgentUser
    
    gcloud projects add-iam-policy-binding ${HOST_PROJECT} \
        --member serviceAccount:${SVC_PROJECT_2_GKE_SA} \
        --role roles/container.hostServiceAgentUser
    

    This role lets the GKE service account of the service project use the GKE service account of the host project to configure shared network resources.

  4. For each service project, grant the Compute Engine default service account the Compute Network Viewer IAM role (roles/compute.networkViewer) in the host project.

    gcloud projects add-iam-policy-binding ${SVC_PROJECT_1} \
        --member serviceAccount:${SVC_PROJECT_1_COMPUTE_SA} \
        --role roles/compute.networkViewer
    
    gcloud projects add-iam-policy-binding ${SVC_PROJECT_2} \
        --member serviceAccount:${SVC_PROJECT_2_COMPUTE_SA} \
        --role roles/compute.networkViewer
    

    When the Envoy sidecar proxy connects to the xDS service (Traffic Director API), the proxy uses the service account of the Compute Engine virtual machine (VM) host or of the GKE node instance. The service account must have the compute.globalForwardingRules.get project-level IAM permission. The Compute Network Viewer role is sufficient for this step.

Verify the shared VPC configuration

  1. In Cloud Shell, verify usable subnets in both service projects:

    gcloud container subnets list-usable \
        --project ${SVC_PROJECT_1} \
        --network-project ${HOST_PROJECT}
    

    The output is similar to the following:

    PROJECT                       REGION    NETWORK     SUBNET    RANGE
    td-012345678-host-project  us-west1  shared-vpc  subnet-1  10.0.4.0/22
        ┌──────────────────────┬───────────────┬─────────────────────────────┐
        │ SECONDARY_RANGE_NAME │ IP_CIDR_RANGE │            STATUS           │
        ├──────────────────────┼───────────────┼─────────────────────────────┤
        │ subnet-1-pods        │ 10.4.0.0/14   │ usable for pods or services │
        │ subnet-1-services    │ 10.0.32.0/20  │ usable for pods or services │
        └──────────────────────┴───────────────┴─────────────────────────────┘
    

    If no items are listed, wait a few moments and run the preceding command again:

    gcloud container subnets list-usable \
        --project ${SVC_PROJECT_2} \
        --network-project ${HOST_PROJECT}
    

    The output is similar to the following:

    PROJECT                       REGION       NETWORK     SUBNET    RANGE
    td-012345678-host-project  us-central1  shared-vpc  subnet-2  172.16.4.0/22
        ┌──────────────────────┬────────────────┬─────────────────────────────┐
        │ SECONDARY_RANGE_NAME │ IP_CIDR_RANGE  │            STATUS           │
        ├──────────────────────┼────────────────┼─────────────────────────────┤
        │ subnet-2-pods        │ 172.20.0.0/14  │ usable for pods or services │
        │ subnet-2-services    │ 172.16.16.0/20 │ usable for pods or services │
        └──────────────────────┴────────────────┴─────────────────────────────┘
    

In the preceding output, you can see that subnet-1 and its secondary addresses are usable by service project 1. Similarly, subnet-2 and its secondary addresses are usable by service project 2.

You can now create resources, such as GKE clusters, in the service projects that use these subnets.

Creating GKE clusters

In this section, you create two GKE clusters, one in each service project. Each cluster uses one subnet from the shared VPC. There are no clusters in the host project, which is used for shared networking resources.

  1. In Cloud Shell, create two GKE clusters, one in each service project:

    export GKE_1="gke-svc-1"
    export GKE_2="gke-svc-2"
    
    gcloud container clusters create ${GKE_1} \
        --project ${SVC_PROJECT_1} \
        --zone=${ZONE_1} \
        --enable-ip-alias \
        --scopes "https://www.googleapis.com/auth/cloud-platform" \
        --network projects/${HOST_PROJECT}/global/networks/shared-vpc \
        --subnetwork projects/${HOST_PROJECT}/regions/${REGION_1}/subnetworks/subnet-1 \
        --cluster-secondary-range-name subnet-1-pods \
        --services-secondary-range-name subnet-1-services --async
    
    gcloud container clusters create ${GKE_2} \
        --project ${SVC_PROJECT_2} \
        --zone=${ZONE_2} \
        --enable-ip-alias \
        --scopes "https://www.googleapis.com/auth/cloud-platform" \
        --network projects/${HOST_PROJECT}/global/networks/shared-vpc \
        --subnetwork projects/${HOST_PROJECT}/regions/${REGION_2}/subnetworks/subnet-2 \
        --cluster-secondary-range-name subnet-2-pods \
        --services-secondary-range-name subnet-2-services
    
  2. Confirm that both clusters are running:

    gcloud container clusters list --project ${SVC_PROJECT_1}
    gcloud container clusters list --project ${SVC_PROJECT_2}
    

    For gke_svc_1, the output is similar to the following:

    NAME       LOCATION    MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION   NUM_NODES  STATUS
    gke-svc-1  us-west1-a  1.13.10-gke.0   35.233.232.169  n1-standard-1  1.13.10-gke.0  3          RUNNING
    

    For gke-svc-2, the output is similar to the following:

    NAME       LOCATION       MASTER_VERSION  MASTER_IP      MACHINE_TYPE   NODE_VERSION   NUM_NODES  STATUS
    gke-svc-2  us-central1-a  1.13.10-gke.0   35.232.172.85  n1-standard-1  1.13.10-gke.0  3          RUNNING
    
  3. Create a new kubeconfig file for this tutorial:

    touch td-kubeconfig
    
    export KUBECONFIG=td-kubeconfig
    

    After you run this command, the kubeconfig file won't interfere with your default kubeconfig file. At the end of this tutorial, you can can delete the KUBECONFIG environment variable and reset it back to default.

  4. Create credentials in your kubeconfig file for both clusters, and store both contexts in the variables for later use:

    gcloud container clusters get-credentials ${GKE_1} \
        --project ${SVC_PROJECT_1} --zone ${ZONE_1}
    export GKE_SVC_1_CONTEXT=`kubectl config current-context`
    
    gcloud container clusters get-credentials ${GKE_2} \
        --project ${SVC_PROJECT_2} --zone ${ZONE_2}
    export GKE_SVC_2_CONTEXT=`kubectl config current-context`
    

Deploying applications

You deploy sample applications in both GKE clusters. Using the curl command-line tool, you also create a busybox deployment in both clusters to test connectivity between applications.

  1. Download the sample application and the BusyBox Kubernetes manifests:

    wget https://storage.googleapis.com/solutions-public-assets/td-shared-vpc-gke/app1.yaml
    wget https://storage.googleapis.com/solutions-public-assets/td-shared-vpc-gke/busybox.yaml
    

    The sample application listens on HTTP port 80 and responds with its hostname (in this case, the Pod name).

  2. Deploy the sample application and the BusyBox application on both GKE clusters:

    kubectl apply -f app1.yaml --cluster $GKE_SVC_1_CONTEXT
    kubectl apply -f app1.yaml --cluster $GKE_SVC_2_CONTEXT
    kubectl apply -f busybox.yaml --cluster $GKE_SVC_1_CONTEXT
    kubectl apply -f busybox.yaml --cluster $GKE_SVC_2_CONTEXT
    

Inspect deployments

  1. Confirm that both applications are running in both GKE clusters:

    kubectl get pods --cluster $GKE_SVC_1_CONTEXT
    kubectl get pods --cluster $GKE_SVC_2_CONTEXT
    

    For GKE_1, the output is similar to the following:

    NAME                      READY   STATUS    RESTARTS   AGE
    app1-f565f7c75-l867n      2/2     Running   0          93s
    app1-f565f7c75-tc652      2/2     Running   0          93s
    busybox-6dcf7d84f-2hntb   2/2     Running   0          90s
    

    For GKE_2, the output is similar to the following:

    NAME                      READY   STATUS    RESTARTS   AGE
    app1-f565f7c75-sp5vf      2/2     Running   0          99s
    app1-f565f7c75-v578c      2/2     Running   0          99s
    busybox-6dcf7d84f-vccjl   2/2     Running   0          97s
    

    In both GKE clusters, there are two Pods of the sample hostname application app1 and one Pod of the busybox running. Each Pod has two containers running.

  2. Inspect the containers running in one of the app1 Pods in one of the GKE clusters:

    export APP1_POD_1=$(kubectl --cluster ${GKE_SVC_1_CONTEXT} get pods -l run=app1 | awk 'NR==2 {print $1}')
    kubectl get pods ${APP1_POD_1} --cluster ${GKE_SVC_1_CONTEXT} -o json | jq '.spec.containers[].image'
    

    The output is similar to the following:

    "gcr.io/kubernetes-e2e-test-images/serve-hostname-amd64:1.1"
    "docker.io/istio/proxyv2:1.1.2"
    

    You can see two containers running the app1 Pod. The first container is the sample hostname application container. The second container is the Envoy sidecar proxy that routes the requests in and out of the Pod.

  3. Inspect the arguments section of the Envoy proxy container:

    kubectl get pods ${APP1_POD_1} --cluster ${GKE_SVC_1_CONTEXT} -o json | jq '.spec.containers[1].args'
    

    The output is similar to the following:

    [
      "proxy",
      "sidecar",
      "--domain",
      "$(POD_NAMESPACE).svc.cluster.local",
      "--configPath",
      "/etc/istio/proxy",
      "--binaryPath",
      "/usr/local/bin/envoy",
      "--serviceCluster",
      "$(POD_NAMESPACE)",
      "--drainDuration",
      "45s",
      "--parentShutdownDuration",
      "1m0s",
      "--discoveryAddress",
      "trafficdirector.googleapis.com:443",
      "--proxyLogLevel",
      "info",
      "--connectTimeout",
      "10s",
      "--proxyAdminPort",
      "15000",
      "--concurrency",
      "2",
      "--statusPort",
      "15020"
    ]
    +
    

    The discoveryAddress proxy setting points to the Traffic Director API. This is how the Envoy proxies are configured. You configure Envoy proxies in the following section.

Configuring Traffic Director

Traffic Director configures the Envoy sidecar proxies running inside each application Pod so that they can communicate over the network. Additionally, Traffic Director uses a configuration to allow east-west traffic between services. This configuration is similar to the configuration that you use for Google Cloud external HTTP(S) Load Balancing to allow north-south traffic from external clients to a service.

You configure the following components for Traffic Director:

  • A health check to ensure that traffic is sent only to healthy Pods.
  • A backend service pointing to backends, which in this case are the network endpoint groups (NEGs) for your services. NEGs represent the Pod IP addresses for your services running in GKE clusters.
  • A route rule, which includes creating a forwarding rule that represents an IP to be forwarded to a backend service, and a URL map that points to a specific URL path for a particular service.

The following diagram shows the control plane's logical flow and the traffic flow to Kubernetes Pods or Compute Engine VM instances.

The control plane's logical flow and the traffic flow to Kubernetes Pods or Compute Engine VM instances.

Kubernetes Pods are represented as endpoints in a NEG by the NEG controllers, while Compute Engine VM instances are managed by managed instance groups (MIGs). Backend services can have one or more backends, which can be either NEGs (in this case, Kubernetes Pods) or MIGs (in this case, Compute Engine VM instances). Traffic Director performs global load balancing based on NEG or MIG capacity, and delivers health-checked NEG or MIG endpoints to the data plane.

Traffic flow is managed using information in the global forwarding rules, target proxies, and URL maps to route traffic to the backend services.

Configure Traffic Director and Cloud Load Balancing components by performing the following steps:

  1. Get the NEG name for the sample application running in both clusters.

    export GKE_SVC_1_APP1_NEG=$(gcloud beta compute network-endpoint-groups list --project ${SVC_PROJECT_1} | grep west | awk '{print     $1}')
    export GKE_SVC_2_APP1_NEG=$(gcloud beta compute network-endpoint-groups list --project ${SVC_PROJECT_2} | grep central | awk'{print $1}')
    

    NEGs allow for container native load balancing on GKE clusters. You can use NEGs as backends behind Google Cloud load balancers.

  2. Create health checks for the backend service in both service projects:

    gcloud compute health-checks create http td-gke-health-check-svc-proj-1 \
        --project ${SVC_PROJECT_1} --use-serving-port
    
    gcloud compute health-checks create http td-gke-health-check-svc-proj-2 \
        --project ${SVC_PROJECT_2} --use-serving-port
    
  3. Create a backend service in both service projects and associate the health check that you created in the previous step with the backend services:

    gcloud compute backend-services create td-gke-service-svc-proj-1 \
        --project ${SVC_PROJECT_1} --global \
        --health-checks td-gke-health-check-svc-proj-1 \
        --load-balancing-scheme INTERNAL_SELF_MANAGED
    
    gcloud compute backend-services create td-gke-service-svc-proj-2 \
        --project ${SVC_PROJECT_2} --global \
        --health-checks td-gke-health-check-svc-proj-2 \
        --load-balancing-scheme INTERNAL_SELF_MANAGED
    
  4. Add the NEGs for the sample application in both service projects as backends to the backend services:

    gcloud compute backend-services add-backend td-gke-service-svc-proj-1 \
        --project ${SVC_PROJECT_1} --global \
        --network-endpoint-group ${GKE_SVC_1_APP1_NEG} \
        --network-endpoint-group-zone ${ZONE_1} \
        --balancing-mode RATE \
        --max-rate-per-endpoint 5
    
    gcloud compute backend-services add-backend td-gke-service-svc-proj-2 \
        --project ${SVC_PROJECT_2} --global \
        --network-endpoint-group ${GKE_SVC_2_APP1_NEG} \
        --network-endpoint-group-zone ${ZONE_2} \
        --balancing-mode RATE \
        --max-rate-per-endpoint 5
    
  5. Create a URL map in each service project:

    gcloud compute url-maps create td-gke-url-map-svc-proj-1 \
        --project ${SVC_PROJECT_1} --default-service td-gke-service-svc-proj-1
    
    gcloud compute url-maps create td-gke-url-map-svc-proj-2 \
        --project ${SVC_PROJECT_2} --default-service td-gke-service-svc-proj-2
    
  6. Create a target proxy in each service project:

    gcloud compute target-http-proxies create td-gke-proxy-svc-proj-1 \
        --project ${SVC_PROJECT_1} --url-map td-gke-url-map-svc-proj-1
    
    gcloud compute target-http-proxies create td-gke-proxy-svc-proj-2 \
        --project ${SVC_PROJECT_2} --url-map td-gke-url-map-svc-proj-2
    
  7. Reserve two internal static IP addresses to use in the forwarding rules for the two services:

    gcloud compute addresses create static-ip-svc-proj-1 \
        --project ${SVC_PROJECT_1} \
        --subnet projects/${HOST_PROJECT}/regions/${REGION_1}/subnetworks/subnet-1 \
        --region=${REGION_1} --addresses 10.0.7.250
    
    gcloud compute addresses create static-ip-svc-proj-2 \
        --project ${SVC_PROJECT_2} \
        --subnet projects/${HOST_PROJECT}/regions/${REGION_2}/subnetworks/subnet-2 \
        --region=${REGION_2} --addresses 172.16.7.250
    
  8. Create a forwarding rule in each service project:

    gcloud compute forwarding-rules create td-gke-forwarding-rule-svc-proj-1 \
        --project ${SVC_PROJECT_1} --global \
        --target-http-proxy td-gke-proxy-svc-proj-1 --ports 80 \
        --address https://www.googleapis.com/compute/v1/projects/"${SVC_PROJECT_1}"/regions/"${REGION_1}"/addresses/static-ip-svc-proj-1 \
        --network https://www.googleapis.com/compute/v1/projects/"${HOST_PROJECT}"/global/networks/shared-vpc \
        --load-balancing-scheme INTERNAL_SELF_MANAGED
    
    gcloud compute forwarding-rules create td-gke-forwarding-rule-svc-proj-2 \
        --project ${SVC_PROJECT_2} --global \
        --target-http-proxy td-gke-proxy-svc-proj-2 --ports 80 \
        --address https://www.googleapis.com/compute/v1/projects/"${SVC_PROJECT_2}"/regions/"${REGION_2}"/addresses/static-ip-svc-proj-2 \
        --network https://www.googleapis.com/compute/v1/projects/"${HOST_PROJECT}"/global/networks/shared-vpc \
        --load-balancing-scheme INTERNAL_SELF_MANAGED
    

Testing connectivity between services

You have now configured Traffic Director. The Envoy sidecar proxies running inside each application Pod are now able to connect over the network across regions and clusters. Next, you test the network connectivity between the services deployed in the two GKE clusters.

  1. Get the Pod name for the busybox Pod running in both GKE clusters. You use the Pod name for network connectivity testing:

    export BUSYBOX_SVC_PROJECT_1_POD=$(kubectl get pods -l run=client -o=jsonpath='{.items[0].metadata.name}' --cluster=${GKE_SVC_1_CONTEXT})
    export BUSYBOX_SVC_PROJECT_2_POD=$(kubectl get pods -l run=client -o=jsonpath='{.items[0].metadata.name}' --cluster=${GKE_SVC_2_CONTEXT})
    
  2. Verify the istio-proxy configs from one of the busybox Pods:

    kubectl --context ${GKE_SVC_1_CONTEXT} exec ${BUSYBOX_SVC_PROJECT_1_POD} -c istio-proxy -- curl localhost:15000/clusters | grep neg
    

    The output is similar to the following:

    cloud-internal-istio:cloud_mp_748236137853_5025068062950697453::172.20.0.16:8000::sub_zone::jf:us-central1-a_3769163129600485500_neg
    cloud-internal-istio:cloud_mp_748236137853_5025068062950697453::172.20.1.10:8000::sub_zone::jf:us-central1-a_3769163129600485500_neg
    cloud-internal-istio:cloud_mp_397592564267_7855627510184552468::10.4.2.7:8000::sub_zone::pv:us-west1-a_4416141197590276889_neg
    cloud-internal-istio:cloud_mp_397592564267_7855627510184552468::10.4.0.11:8000::sub_zone::pv:us-west1-a_4416141197590276889_neg
    

    In this output, the first field is an internal service name assigned by the Traffic Director. The second field (IP address) is the Pod IP address for the service. The last field represents the zone and the NEG ID of the Pod. You see two Pod IP addresses in the us-central1-a zone for the app1 Pods in the gke-svc-2 cluster. You also see two Pod IP addresses in the us-west1-a zone for the app1 Pods in gke-svc-1 cluster.

  3. Get the Pod names for the hostname application in GKE clusters in both service projects:

    kubectl get pods -l run=app1 --cluster=${GKE_SVC_1_CONTEXT}
    kubectl get pods -l run=app1 --cluster=${GKE_SVC_2_CONTEXT}
    

    Make a note of the names of the Pods for connectivity testing in the next section.

Test connectivity

  1. Test HTTP requests from the BusyBox Pod in service project 1 to the hostname application in service project 1:

    for i in {1..10}
    do
    kubectl exec -it ${BUSYBOX_SVC_PROJECT_1_POD}
    --cluster=${GKE_SVC_1_CONTEXT} -c busybox -- /bin/sh -c
    'wget -q -O - 10.0.7.250'; echo
    done
    

    The result is the hostname of the Pods from the deployment running in the GKE cluster in service project 1.

  2. Test HTTP requests from the BusyBox in service project 1 to the hostname application in service project 2:

    for i in {1..10}
    do
    kubectl exec -it ${BUSYBOX_SVC_PROJECT_1_POD}
    --cluster=${GKE_SVC_1_CONTEXT} -c busybox -- /bin/sh -c
    'wget -q -O - 172.16.7.250'; echo
    done
    

    The result is the hostname of the Pods from the deployment running in the GKE cluster in service project 2.

  3. Test HTTP requests from the BusyBox in service project 2 to the hostname application in service project 2:

    for i in {1..10}
    do
    kubectl exec -it ${BUSYBOX_SVC_PROJECT_2_POD}
    --cluster=${GKE_SVC_2_CONTEXT} -c busybox -- /bin/sh -c
    'wget -q -O - 172.16.7.250'; echo
    done`
    

    The result is the hostname of the Pods from the deployment running in the GKE cluster in service project 2.

  4. Test HTTP requests from the Busybox in service project 2 to the hostname application in service project 1:

    for i in {1..10}
    do
    kubectl exec -it ${BUSYBOX_SVC_PROJECT_2_POD}
    --cluster=${GKE_SVC_2_CONTEXT} -c busybox -- /bin/sh -c
    'wget -q -O - 10.0.7.250'; echo
    done
    

    The result is the hostname of the Pods from the deployment running in the GKE cluster in service project 1.

You have now configured Traffic Director on a shared VPC to route traffic between services running on GKE clusters in multiple service projects.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Reset kubeconfig

  • Reset the KUBECONFIG variable to point to your default kubeconfig file:

    unset KUBECONFIG
    

Delete all projects

The easiest way to eliminate billing is to delete the projects you created for this tutorial. Before you can delete the projects, you must disable the shared VPC. To do this, perform the following steps exactly in the following order:

  1. Remove the forwarding rules from the two service projects:

    gcloud compute forwarding-rules delete td-gke-forwarding-rule \
        --project ${SVC_PROJECT_1} --global
    
    gcloud compute forwarding-rules delete td-gke-forwarding-rule \
        --project ${SVC_PROJECT_2} --global
    
  2. Remove the NEGs from the backend services from both service projects:

    gcloud compute backend-services remove-backend td-gke-service \
        --project ${SVC_PROJECT_1} --global \
        --network-endpoint-group ${GKE_SVC_1_APP1_NEG} \
        --network-endpoint-group-zone ${ZONE_1}
    
    gcloud compute backend-services remove-backend td-gke-service \
        --project ${SVC_PROJECT_2} --global \
        --network-endpoint-group ${GKE_SVC_2_APP1_NEG} \
        --network-endpoint-group-zone ${ZONE_2}
    
  3. Remove the NEGs from both service projects:

    gcloud compute network-endpoint-groups delete ${GKE_SVC_1_APP1_NEG} --project=${SVC_PROJECT_1} --zone=${ZONE_1} --quiet
    
    gcloud compute network-endpoint-groups delete ${GKE_SVC_2_APP1_NEG} --project=${SVC_PROJECT_2} --zone=${ZONE_2} --quiet
    
  4. Delete the GKE clusters from both service projects:

    gcloud container clusters delete ${GKE_1} --project=${SVC_PROJECT_1} --zone=${ZONE_1} --quiet
    
    gcloud container clusters delete ${GKE_2} --project=${SVC_PROJECT_2} --zone=${ZONE_2} --quiet
    
  5. Detach the service projects from the shared VPC:

    gcloud compute shared-vpc associated-projects remove ${SVC_PROJECT_1} --host-project ${HOST_PROJECT}
    
    gcloud compute shared-vpc associated-projects remove ${SVC_PROJECT_2} --host-project ${HOST_PROJECT}
    
  6. Disable the shared VPC in the host project:

    gcloud compute shared-vpc disable ${HOST_PROJECT}
    
  7. Delete the three projects that you created for this tutorial:

    gcloud projects delete ${HOST_PROJECT}
    gcloud projects delete ${SVC_PROJECT_1}
    gcloud projects delete ${SVC_PROJECT_2}
    

What's next