Traffic Director setup for Compute Engine VMs with manual Envoy deployment

Before you follow the instructions in this guide, review Preparing for Traffic Director setup and make sure that you have completed the prerequisite tasks described in that document.

This guide shows you how to manually deploy a data plane that consists of Envoy sidecar proxies, configure it using Traffic Director, and verify your setup to ensure that it's functioning correctly. This process involves:

  1. Creating a test service.
  2. Deploying a simple data plane on Compute Engine using Envoy proxies.
  3. Setting up Traffic Director using Compute Engine APIs, which enable Traffic Director to configure your Envoy sidecar proxies.
  4. Log in to a VM that is running an Envoy proxy and send a request to a load-balanced backend through the Envoy proxy.

The configuration examples in this document are for demonstration purposes. For a production environment, you might need to deploy additional components, based on your environment and requirements.

Overview of the configuration process

This section provides the manual configuration process for services that run on Compute Engine VMs. The configuration process for the client VMs consists of setting up a sidecar proxy and traffic interception on a Compute Engine VM host. Then you configure load balancing using Google Cloud load balancing APIs.

This section provides information on how to obtain and inject Envoy proxies from third-party sources that are not managed by Google.

When an application sends traffic to the service configured in Traffic Director, the traffic is intercepted and redirected to the xDS API-compatible sidecar proxy and then load balanced to the backends according to the configuration in the GCP load balancing components. For more information on host networking and traffic interception, read How traffic interception and load balancing work in Traffic Director.

For each VM host that requires access to Traffic Director services, perform the following steps:

  1. Assign a service account to the VM.

  2. Set the API access scope of the VM to allow full access to the Google Cloud APIs.

    • When you create the VMs, under Identity and API access, click Allow full access to all Cloud APIs.

      Go to the VM instances page.

    • With the gcloud command line tool, specify the following:

      --scopes=https://www.googleapis.com/auth/cloud-platform.

  3. Allow outgoing connections to trafficdirector.googleapis.com (TCP, port 443) from the VM, so that the sidecar proxy can connect to the Traffic Director control plane over gRPC. Outgoing connections to port 443 are enabled by default.

  4. Deploy an xDS API-compatible sidecar proxy (such as Envoy), with a bootstrap configuration pointing to trafficdirector.googleapis.com:443 as its xDS server. Refer to this sample bootstrap configuration file as an example.

  5. Redirect IP traffic that is destined to the services to the sidecar proxy interception listener port.

    1. The sidecar proxy interception listener port is defined as TRAFFICDIRECTOR_INTERCEPTION_PORT in the proxy's bootstrap metadata configuration and is set to 15001 in the sample bootstrap configuration file.
    2. The Istio iptables script can be used to set up traffic interception.

Creating the Hello World test service

This section shows you how to create a simple test service that returns the hostname of the VM that served the request from the client. The test service is uncomplicated; it's a web server deployed across a Compute Engine managed instance group.

Creating the instance template for the Hello World test service

The instance template you create configures a sample apache2 web server using the startup-script parameter.

Console

  1. In the Cloud Console, go to the Instance Templates page.

    Go to the Instance Templates page

  2. Click Create instance template.
  3. In the fields, enter the following information:
    • Name: td-demo-hello-world-template
    • Boot disk: Debian GNU/Linux 10 (buster)
    • Service account: Compute Engine default service account
    • Access scopes: Allow full access to all Cloud APIs
  4. Click Management, Security, Disks, Networking, Sole Tenancy. 1 In the Networking tab, in the **Network tags field, add the td-http-server tag.
  5. In the Management tab, copy the following script into the Startup script field.

    #! /bin/bash
    sudo apt-get update -y
    sudo apt-get install apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>'`/bin/hostname`'</h1></body></html>' | sudo tee /var/www/html/index.html
    
  6. Click Create.

gcloud

To create the instance template:

gcloud compute instance-templates create td-demo-hello-world-template \
  --machine-type=n1-standard-1 \
  --boot-disk-size=20GB \
  --image-family=debian-10 \
  --image-project=debian-cloud \
  --scopes=https://www.googleapis.com/auth/cloud-platform \
  --tags=td-http-server \
  --metadata=startup-script="#! /bin/bash
sudo apt-get update -y
sudo apt-get install apache2 -y
sudo service apache2 restart
sudo mkdir -p /var/www/html/
echo '<!doctype html><html><body><h1>'`/bin/hostname`'</h1></body></html>' | sudo tee /var/www/html/index.html"

Creating the managed instance group for the Hello World service

In this section, you specify that the managed instance group always has two instances of the test service. This is for demonstration purposes. Traffic Director supports autoscaled managed instance groups.

Console

  1. In the Cloud Console, go to the Instance Groups page.

    Go to the Instance Groups page

  2. Click Create instance group.
  3. Make sure that New managed instance group is selected.
  4. Enter td-demo-hello-world-mig for the name for the managed instance group, and select the us-central1-a zone.
  5. Under Instance template, select td-demo-hello-world-template, which is the instance template you created.
  6. Under Autoscaling mode, select Don't autoscale.
  7. Under Number of instances, specify at least two as the number of instances that you want to create in the group.
  8. Click Create.

gcloud

Use the gcloud command line tool to create a managed instance group with the instance template you previously created.

gcloud compute instance-groups managed create td-demo-hello-world-mig \
  --zone us-central1-a \
  --size=2 \
  --template=td-demo-hello-world-template

Creating the instance template and managed instance group where Envoy is deployed

Use the instructions in this section to manually create an instance template and managed instance group for Traffic Director.

Managed instance groups create new backend VMs by using autoscaling.

This example shows how to:

  • Create a VM template with a full Envoy configuration and a sample service that serves its hostname using the HTTP protocol.
  • Configure a managed instance group using this template.

Creating the instance template

First, create the Compute Engine VM instance template. This template auto-configures the Envoy sidecar proxy and sample apache2 web service through the startup-script parameter.

Console

  1. In the Cloud Console, go to the Instance Templates page.

    Go to the Instance templates page

  2. Click Create instance template.
  3. Fill in the fields as follows:

    • Name: td-vm-template
    • Boot disk: Debian GNU/Linux 9 (stretch)
    • Service account: Compute Engine default service account
    • Access scopes: Allow full access to all Cloud APIs
  4. Under Firewall, select the boxes next to Allow HTTP traffic and Allow HTTPS traffic.

  5. Click Management, Security, Disks, Networking, Sole Tenancy.

  6. In the Management tab, copy the following script into the Startup script field.

    
    #! /bin/bash
    # Add a system user to run Envoy binaries. Login is disabled for this user
    sudo adduser --system --disabled-login envoy
    # Download and extract the Traffic Director tar.gz file
    sudo wget -P /home/envoy https://storage.googleapis.com/traffic-director/traffic-director.tar.gz
    sudo tar -xzf /home/envoy/traffic-director.tar.gz -C /home/envoy
    sudo cat << END > /home/envoy/traffic-director/sidecar.env
    ENVOY_USER=envoy
    # Exclude the proxy user from redirection so that traffic doesn't loop back
    # to the proxy
    EXCLUDE_ENVOY_USER_FROM_INTERCEPT='true'
    # Intercept all traffic by default
    SERVICE_CIDR='*'
    GCP_PROJECT_NUMBER=''
    VPC_NETWORK_NAME=''
    ENVOY_PORT='15001'
    ENVOY_ADMIN_PORT='15000'
    LOG_DIR='/var/log/envoy/'
    LOG_LEVEL='info'
    XDS_SERVER_CERT='/etc/ssl/certs/ca-certificates.crt'
    END
    sudo apt-get update -y
    sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y
    sudo curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
    sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/debian stretch stable' -y
    sudo apt-get update -y
    sudo apt-get install docker-ce apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>'`/bin/hostname`'</h1></body></html>' | sudo tee /var/www/html/index.html
    sudo /home/envoy/traffic-director/pull_envoy.sh
    sudo /home/envoy/traffic-director/run.sh start
    
    
  7. Click Create to create the template.

gcloud

  1. Create the instance template.

    
    gcloud compute instance-templates create td-vm-template \
      --scopes=https://www.googleapis.com/auth/cloud-platform \
      --tags=http-td-tag,http-server,https-server \
      --image-family=debian-9 \
      --image-project=debian-cloud \
      --metadata=startup-script="#! /bin/bash
    
    # Add a system user to run Envoy binaries. Login is disabled for this user
    sudo adduser --system --disabled-login envoy
    # Download and extract the Traffic Director tar.gz file
    sudo wget -P /home/envoy https://storage.googleapis.com/traffic-director/traffic-director.tar.gz
    sudo tar -xzf /home/envoy/traffic-director.tar.gz -C /home/envoy
    sudo cat << END > /home/envoy/traffic-director/sidecar.env
    ENVOY_USER=envoy
    # Exclude the proxy user from redirection so that traffic doesn't loop back
    # to the proxy
    EXCLUDE_ENVOY_USER_FROM_INTERCEPT='true'
    # Intercept all traffic by default
    SERVICE_CIDR='*'
    GCP_PROJECT_NUMBER=''
    VPC_NETWORK_NAME=''
    ENVOY_PORT='15001'
    ENVOY_ADMIN_PORT='15000'
    LOG_DIR='/var/log/envoy/'
    LOG_LEVEL='info'
    XDS_SERVER_CERT='/etc/ssl/certs/ca-certificates.crt'
    END
    sudo apt-get update -y
    sudo apt-get install apt-transport-https ca-certificates curl gnupg2 software-properties-common -y
    sudo curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
    sudo add-apt-repository 'deb [arch=amd64] https://download.docker.com/linux/debian stretch stable' -y
    sudo apt-get update -y
    sudo apt-get install docker-ce apache2 -y
    sudo service apache2 restart
    echo '<!doctype html><html><body><h1>'`/bin/hostname`'</h1></body></html>' | sudo tee /var/www/html/index.html
    sudo /home/envoy/traffic-director/pull_envoy.sh
    sudo /home/envoy/traffic-director/run.sh start"
    
    

Creating the managed instance group

If you don't have a managed instance group with services running, create a managed instance group, using a VM template such as the one shown in the previous section. This example uses the instance template created in the previous section to demonstrate functionality. You do not have to use the instance template.

Console

  1. Go to the Instance Groups page in the Cloud Console.

    Go to the Instance Groups page

  2. Click Create an instance group. By default, you see the page for creating a managed instance group.
  3. Enter td-vm-mig-us-central1 for the name for the managed instance group, and select the us-central1-a zone.
  4. Under Instance template, select the instance template you created.
  5. Specify 2 as the number of instances that you want to create in the group.
  6. Click Create.

gcloud

Use the gcloud command line tool to create a managed instance group with the instance template you previously created.

gcloud compute instance-groups managed create td-vm-mig-us-central1 \
    --zone us-central1-a --size=2 --template=td-vm-template

Configuring Traffic Director with Google Cloud load balancing components

The instructions in this section show you how to configure Traffic Director so that your Envoy proxies load balance outbound traffic across two backend instances. You configure the following components:

Creating the health check

Use the following instructions to create a health check. For more information, refer to Creating health checks.

Console

  1. Go to the Health checks page in the Google Cloud Console.
    Go to the Health checks page
  2. Click Create Health Check.
  3. For the name, enter td-vm-health-check.
  4. For the protocol, select HTTP.
  5. Click Create.

gcloud

  1. Create the health check.
gcloud compute health-checks create http td-vm-health-check
  1. Create the firewall rule.

    gcloud compute firewall-rules create fw-allow-health-checks \
      --action ALLOW \
      --direction INGRESS \
      --source-ranges 35.191.0.0/16,130.211.0.0/22 \
      --target-tags http-td-tag,http-server,https-server \
      --rules tcp
    

Creating the backend service

If you use the gcloud command-line tool, you must designate the backend service as a global backend service with a load balancing scheme of INTERNAL_SELF_MANAGED. Add the health check and a managed or unmanaged instance group to the backend service. Note that this example uses the managed instance group with Compute Engine VM template that runs the sample HTTP service created in Creating the managed instance group.

Console

  1. Go to the Traffic Director page in the Cloud Console.

    Go to the Traffic Director page

  2. On the Services tab, click Create Service.

  3. Click Continue.

  4. For the service name, enter td-vm-service.

  5. Select the correct VPC network.

  6. Ensure that the Backend type is Instance groups.

  7. Select the managed instance group you created.

  8. Enter the correct Port numbers.

  9. Choose Utilization or Rate as the Balancing mode. The default value is Rate.

  10. Click Done.

  11. Select the health check you created.

  12. Click Save and continue.

  13. Click Create.

gcloud

  1. Create the backend service.

    gcloud compute backend-services create td-vm-service
     --global \
     --load-balancing-scheme=INTERNAL_SELF_MANAGED \
     --health-checks td-vm-health-check
    
  2. Add the backends to the backend service.

    gcloud compute backend-services add-backend td-vm-service \
      --instance-group td-vm-mig-us-central1 \
      --instance-group-zone us-central1-a \
      --global
    

Creating the routing rule map

The routing rule map defines how Traffic Director routes traffic in your mesh.

Use these instructions to create the route rule, forwarding rule, target proxy, and internal IP address for your Traffic Director configuration.

Traffic sent to the internal IP address is intercepted by the Envoy proxy and sent to the appropriate service according to the host and path rules.

The forwarding rule is created as a global forwarding rule with the load-balancing-scheme set to INTERNAL_SELF_MANAGED.

You can set the address of your forwarding rule to 0.0.0.0. If you do, traffic is routed based on the HTTP hostname and path information configured in the URL map, regardless of the actual destination IP address of the request. In this case, the hostnames of your services, as configured in the host rules, must be unique within your service mesh configuration. That is, you cannot have two different services, with different sets of backends, that both use the same hostname.

Alternatively, you can enable routing based on the actual destination VIP of the service. If you configure the VIP of your service as an address parameter of the forwarding rule, only requests destined to this address are routed based on the HTTP parameters specified in the URL map.

This example uses 0.0.0.0 as the address parameter, meaning that routing for your service is performed based on the HTTP hostname and path parameters only.

Console

In the Cloud Console, the target proxy is combined with the forwarding rule. When you create the forwarding rule, Google Cloud automatically creates a target HTTP proxy and attaches it to the URL map.

  1. Go to the Traffic Director page in the Cloud Console.

    Go to the Traffic Director page

  2. On the Routing rule maps tab, click Create Routing Rule Map.

  3. Enter a name.

  4. Click Add Forwarding Rule.

  5. For the forwarding rule name, enter td-vm-forwarding-rule.

  6. Select your network.

  7. Select your Internal IP. Traffic sent to this IP address is intercepted by the Envoy proxy and sent to the appropriate service according to the host and path rules.

    The forwarding rule is created as a global forwarding rule with the load-balancing-scheme set to INTERNAL_SELF_MANAGED.

  8. In the Custom IP field, type 10.0.0.1. When your VM sends to this IP address, the Envoy proxy intercepts it and sends it to the appropriate backend service's endpoint according to the traffic management rules defined in the URL map.

    Each forwarding rule in a VPC network must have a unique IP address and port per VPC network. If you create more than one forwarding rule with the same IP address and port in a particular VPC network, only the first forwarding rule is valid. Others are ignored. If 10.0.0.1 is not available in your network, choose a different IP address.

  9. Make sure that the Port is set to `80.

  10. Click Save.

  11. In the Routing rules section, select Simple host and path rule.

  12. In the ** Host and path rules** section, select td-vm-service as the Service.

  13. Click Add host and path rule.

  14. In Hosts, enter hello-world.

  15. In Service, select td-vm-service.

  16. Click Save.

gcloud

  1. Create a URL map that uses the backend service.

    gcloud compute url-maps create td-vm-url-map \
       --default-service td-vm-service
    
  2. Create a URL map path matcher and a host rule to route traffic for your service based on hostname and a path. This example uses service-test as the service name and a default path matcher that matches all path requests for this host (/*).

    gcloud compute url-maps add-path-matcher td-vm-url-map \
       --default-service td-vm-service --path-matcher-name td-vm-path-matcher
    
    gcloud compute url-maps add-host-rule td-vm-url-map --hosts service-test \
       --path-matcher-name td-vm-path-matcher
       --new-hosts hello-world
    
  3. Create the target HTTP proxy.

    gcloud compute target-http-proxies create td-vm-proxy \
       --url-map td-vm-url-map
    
  4. Create the forwarding rule.

    The forwarding rule must be global and must be created with the value of load-balancing-scheme set to INTERNAL_SELF_MANAGED.

    gcloud compute forwarding-rules create td-vm-forwarding-rule \
       --global \
       --load-balancing-scheme=INTERNAL_SELF_MANAGED \
       --address=10.0.0.1 \
       --target-http-proxy=td-vm-proxy \
       --ports 80 \
       --network default
    

At this point, Traffic Director is configured to load balance traffic for the services specified in the URL map across backends in the managed instance group.

Verifying the configuration

In this final portion of the Traffic Director setup guide for Compute Engine VMs, you test that traffic sent from the client VM destined to the forwarding rule VIP is intercepted and redirected to the Envoy proxy, which then routes your request to the VMs hosting the Hello World service.

To do this, you log into the client VM, that has been configured to intercept traffic and redirect it to Envoy. You send a curl request to the VIP associated with your routing rule map. Envoy inspects the curl request, determines which service it should resolve to, and sends the request to an backend associated with that service.

  1. In the Cloud Console, go to the Instance Groups page.

    Go to the Instance Groups page

  2. Select the td-vm-mig-us-central1 instance group.
  3. Under Connect, click SSH.
  4. After you are logged in to the client VM, use the curl tool to send a request to the Hello World service through Envoy:
curl -H "Host: hello-world" http://10.0.0.1/

When you issue this command repeatedly, you should see different HTML responses containing the hostnames of backends in the Hello World managed instance group. This is because Envoy is using round robin load balancing, the default load balancing algorithm, when sending traffic to the Hello World service's backends.

What's next

Depending on how your microservices are distributed on your network, you might need to add more forwarding rules or more host and path rules to the URL map. For more information on forwarding rules and URL maps, read the following documents:

Verifying the configuration

When the configuration is complete, each Compute Engine VM that has a sidecar proxy can access services configured in Traffic Director using the HTTP protocol.

If you followed the specific examples in this guide, using the Compute Engine VM template with the demonstration HTTP server and service hostname service-test, use these steps to verify the configuration.

  1. Log in to one of the VM hosts that has a sidecar proxy installed.
  2. Execute the command curl -H 'Host: service-test' 10.0.0.1. This request returns the hostname of the managed instance group backend that served the request.

In step 2, note that you can use any IP address. For example, the command curl -I -H 'Host: service-test' 1.2.3.4 would work in Step 2.

This is because the forwarding rule has the address parameter set to 0.0.0.0, which instructs Traffic Director to match based on the host defined in the URL map. In the example configuration, the host name is service-test.