Network edge services for multi-environment deployments

Google offers a variety of network edge services that can augment the capabilities and security of services based outside of Google Cloud (on-premises and multi-cloud services), for example, in an on-premises data center or in another public cloud. These network edge services allow you to:

Example of network edge services (click to enlarge)
Example of network edge services (click to enlarge)

To bring these benefits to your private, on-premises, or multi-cloud services, you deploy an external HTTP(S) load balancer to receive traffic from the public internet. The HTTP(S) load balancer forwards traffic to a Traffic Director- configured middle proxy that routes the traffic to your on-premises environment or non-Google Cloud cloud using Cloud VPN or Cloud Interconnect.

This tutorial walks you through an end-to-end example that uses Google Cloud Armor at Google's edge to selectively allow clients to access an on-premises service privately. Clients are allowed access based on their IP address.

You complete the following tasks:

  1. Deploying Envoy as a middle proxy in a managed instance group. This Envoy proxy is automatically connected to Traffic Director.
  2. Creating a simulated private, on-premises VM. In a real example, you would probably already have an on-premises VM.
  3. Configuring Traffic Director to route all requests that reach the middle proxy to the simulated on-premises VM
  4. Creating an external HTTP(S) load balancer to receive traffic from the public Internet and forward it to the middle proxy
  5. Attaching a Google Cloud Armor security policy to the external HTTP(S) load balancer

After you complete these tasks, you can optionally explore additional edge services and advanced traffic management features.


Before setting up your middle proxy, complete the following tasks:

  • Review Preparing for Traffic Director setup and complete all prerequisite tasks.
  • Ensure that your private on-premises endpoint(s) are reachable from within your Google Cloud VPC network through Cloud VPN or Cloud Interconnect. The example used in this solutions guide just routes traffic to an endpoint within Google Cloud, so you don't need to configure hybrid connectivity to follow along. In a real-world deployment scenario, this would be required.
  • Ensure that your VPC subnet CIDR ranges do not conflict with your remote CIDR ranges. Subnet routes are prioritized over remote connectivity when IP addresses overlap.
  • For this demonstration, obtain the necessary permissions to create and update Google Cloud Armor security policies. The permissions might vary if you want to use a different service, such as Cloud CDN.

Deploying the middle proxy on Compute Engine VMs

This section describes how to deploy Envoy as a middle proxy on Compute Engine to receive traffic from the external load balancer and forward it to your remote destination.

Creating the instance template for the middle proxy

An instance template specifies configuration for VMs within a managed instance group. The following template can be used to create VM instances with an Envoy proxy connected to Traffic Director:

 gcloud compute instance-templates create td-middle-proxy \
    --service-proxy=enabled \

To customize your Envoy deployment, such as by specifying the network name, setting a log path, and/or enabling tracing, see the Automated Envoy deployment option guide.

Creating the managed instance group for the middle proxy

Create the managed instance group based on the template above. In this example, you can keep the instance group size of 1 to deploy a single instance. For production usage, enable autoscaling, for example, based on CPU utilization, to avoid creating a bottleneck if your middle proxy needs to handle a lot of traffic.

gcloud compute instance-groups managed create td-middle-proxy-us-central1-a \
    --zone us-central1-a \
    --template=td-middle-proxy \

Identify and store the internal IP address of the instance in $MIDDLE_PROXY_IP for future configurations and verification.

MIDDLE_PROXY_IP=`gcloud compute instances list --filter="name~'td-middle-proxy-us-central1-a-.*'" --zones=us-central1-a --format="value(networkInterfaces.networkIP)"`

In this example, we create the managed instance group, containing VM instances running Envoy, in us-central1-a. Later on in this guide, you create an external load balancer to handle public internet traffic from your clients. Because the external load balancer can automatically route traffic to the region that is closest to your clients, and to managed instance group within that region, you might want to consider creating multiple managed instance groups. A full list of Google Cloud's available regions and zones can be found here.

Deploying the simulated on-premises service

This section describes deploying a hybrid connectivity network endpoint group (NEG). In a production deployment, this NEG would contain an endpoint (IP:port) that resolves to your on-premises server. In this example, you just create a Compute Engine VM that is reachable on an IP:port. This VM acts as your simulated on-premises server.

Creating the simulated on-premises VM

Deploy a single VM instance to simulate a private, on-premises server.

gcloud compute instances create on-prem-vm \
  --zone us-central1-a \
  --metadata startup-script='#! /bin/bash
## Installs apache and a custom homepage
sudo su -
apt-get update
apt-get install -y apache2
cat <<EOF > /var/www/html/index.html
<html><body><h1>Hello world from on-premises!</h1></body></html>

Identify and store its internal IP address for future configurations and verification. The server on this VM listens for inbound requests on port 80.

ON_PREM_IP=`gcloud compute instances describe on-prem-vm --zone=us-central1-a --format="value(networkInterfaces.networkIP)" | sed "s/['\[\]]*//g"`

Creating the network endpoint group

Create the network endpoint group (NEG) for this demonstration setup by specifying the non-gcp-private-ip-port network endpoint type. Add the IP address and port for your simulated on-premises VM as an endpoint to this NEG. Following on the previous step, the IP address is stored in the ON_PREM_IP environment variable.

First, create the NEG:

gcloud compute network-endpoint-groups create td-on-prem-neg \
    --network-endpoint-type non-gcp-private-ip-port \
    --zone us-central1-a

Next, add the IP:port to your newly-created NEG.

gcloud compute network-endpoint-groups update td-on-prem-neg \
  --zone=us-central1-a \

Configuring Traffic Director with Google Cloud load balancing components

This section shows how to configure Traffic Director and enable your middle proxy to forward traffic to your private, on-premises service. You configure the following components:

  • A health check. This health check behaves slightly differently from the health checks configured for other NEG types, as described below.
  • A backend service. For more information on backend services, see Backend services overview.
  • A routing rule map. This includes creating a forwarding rule, a target proxy, and a URL map. For more information, read Using forwarding rules and Using URL maps.

Creating the health check

Health checks verify that your endpoints are healthy and able to receive requests. Health checking for this type of NEG relies on Envoy's distributed health checking mechanism, while other NEG types use Google Cloud's centralized health checking system.

gcloud compute health-checks create http td-on-prem-health-check

Creating the backend service

Create a backend service with the INTERNAL_SELF_MANAGED load balancing scheme to be used with Traffic Director. When you create this backend service, specify the health check that you just created.

gcloud compute backend-services create td-on-prem-backend-service \
    --global \
    --load-balancing-scheme=INTERNAL_SELF_MANAGED \

Next, add the NEG that you created earlier as the backend of this backend service.

gcloud compute backend-services add-backend td-on-prem-backend-service \
    --global \
    --network-endpoint-group=td-on-prem-neg \
    --network-endpoint-group-zone=us-central1-a \
    --balancing-mode=RATE \

Creating the routing rule map

The routing rule map defines how Traffic Director routes traffic to your backend service.

  1. Create a URL map that uses the backend service defined above.

    gcloud compute url-maps create td-hybrid-url-map \
      --default-service td-on-prem-backend-service
  2. Create a file called target_proxy.yaml with the following contents:

    name: td-hybrid-proxy
    proxyBind: true
    urlMap: global/urlMaps/td-hybrid-url-map
  3. Use the import command to create the target HTTP proxy. For more information, see Target proxies for Traffic Director.

    gcloud compute target-http-proxies import td-hybrid-proxy \
       --source target_proxy.yaml
  4. Create a forwarding rule that references this target HTTP proxy. Set the forwarding rule's IP address to Setting the rule's address to routes traffic based on the inbound port, HTTP hostname, and path information configured in the URL map. The IP address specified in the HTTP request is ignored.

    gcloud compute forwarding-rules create td-hybrid-forwarding-rule \
       --global \
       --load-balancing-scheme=INTERNAL_SELF_MANAGED \
       --address= \
       --target-http-proxy=td-hybrid-proxy \
       --ports=8080 \

Verifying that the middle proxy can route requests to the simulated on-premises service

Traffic Director is now configured to route traffic through the middle proxy to your simulated private, on-premises service. You can verify this configuration by creating a test client VM, logging into that VM and sending a request to the middle proxy that is running Envoy. Once verified, delete the test client VM.

First, get the IP address of the middle proxy. You only need this information for the verification step:

gcloud compute instances list

Write down or otherwise note the IP address of the instance in the td-middle-proxy-us-central1-a managed instance group.

Next, create a test client instance:

gcloud compute instances create test-client --zone us-central1-a

Use ssh to log in to the test client:

gcloud compute ssh test-client --zone us-central1-a

Finally, send a request to the middle proxy VM, substituting the IP address you obtained above for MIDDLE_PROXY_IP:

curl $MIDDLE_PROXY_IP:8080

You should see the following output:

Hello world from on-premises!

Exit the test client VM. Once you've exited, you can delete the VM:

gcloud compute instances delete test-client \
  --zone us-central1-a

Deploying the external HTTP load balancer

In this section, you deploy an external HTTP(S) load balancer that sends incoming traffic to the middle proxy. This is a standard external HTTP(S) load balancer setup.

Reserve an external IP address

Create a global static external IP address (external-lb-vip) to which external clients will send traffic. You retrieve this external IP address during the verification step later in this guide.

gcloud compute addresses create external-lb-vip \
    --ip-version=IPV4 \

Setting up the external HTTP load balancer

Configure the external load balancer to route internet customer traffic to your already-configured middle proxy.

  1. Create a health check that is used to determine whether the managed instance group that run the middle proxy is healthy and able to receive traffic.

       gcloud compute health-checks create tcp tcp-basic-check \
            --port 8080
  2. Create a firewall rule to allow health checking. Note that you re-use the allow-hc tag here to apply the firewall rule to the middle proxy VMs.

    gcloud compute firewall-rules create fw-allow-health-checks \
      --network default \
      --action ALLOW \
      --direction INGRESS \
      --source-ranges, \
      --target-tags allow-hc \
      --rules tcp
  3. Create a backend service.

       gcloud compute backend-services create td-middle-proxy-backend-service \
            --protocol HTTP \
            --health-checks tcp-basic-check \
  4. Add the middle proxy managed instance group as the backend to this backend service.

       gcloud compute backend-services add-backend td-middle-proxy-backend-service \
            --instance-group=td-middle-proxy-us-central1-a \
            --instance-group-zone=us-central1-a \
  5. Create a URL map to route the incoming requests to the middle proxy as the default backend service.

       gcloud compute url-maps create lb-map-http \
            --default-service td-middle-proxy-backend-service
  6. Create a target HTTP proxy so that requests to the external load balancer's forwarding rule VIP are handled according to the URL map.

     gcloud compute target-http-proxies create http-lb-proxy \
            --url-map lb-map-http
  7. Create a global forwarding rule to route incoming requests to the target HTTP proxy.

       gcloud compute forwarding-rules create http-forwarding-rule \
            --global \
            --load-balancing-scheme=EXTERNAL \
            --target-http-proxy=http-lb-proxy \

Setting the managed instance group's named port

Set a named port for the instance group to allow your middle proxy to receive HTTP traffic from the external load balancer.

gcloud compute instance-groups managed set-named-ports td-middle-proxy-us-central1-a \
    --named-ports http:8080 \
    --zone us-central1-a

Verifying the external HTTP load balancer configuration

In this step, you verify that the external load balancer is set up correctly. You should be able to send a request to the load balancer's VIP and get a response from the simulated on-premises VM.

PUBLIC_VIP=`gcloud compute addresses describe external-lb-vip \
    --format="get(address)" \

Issue a curl request to the public IP address and verify that you receive the Hello world message.


You see the following output:

Hello world from on-premises!

Enabling Google Cloud Armor

Configure Google Cloud Armor security policies to solely allow access to your service from CLIENT_IP_RANGE, which should include the public IP address of the client device you intend to test with; for example, "". These policies are applied on the backend service of the external load balancer (in this example, td-hybrid-backend-service pointing at the middle proxy). For more information on the permissions required to set these rules, see Configuring Google Cloud Armor security policies.

  1. Create the Google Cloud Armor security policy.

    gcloud compute security-policies create external-clients-policy \
        --description "policy for external clients"
  2. Update the default rule of the security policy to deny all traffic.

    gcloud compute security-policies rules update 2147483647 \
        --security-policy external-clients-policy \
        --action "deny-404"
  3. Add a higher priority rule to allow traffic from a specific IP range.

    gcloud compute security-policies rules create 1000 \
        --security-policy external-clients-policy \
        --description "allow traffic from CLIENT_IP_RANGE" \
        --src-ip-ranges "CLIENT_IP_RANGE" \
        --action "allow"
  4. Attach the Google Cloud Armor security policies to your backend service.

    gcloud compute backend-services update td-middle-proxy-backend-service \
        --security-policy external-clients-policy

Final verification

Issue a curl request to the external HTTP load balancer's public virtual IP address. You should still receive the expected response if your client device's IP address is within the allowed <var>CLIENT_IP_RANGE</var> specified above.


You see the following output:

Hello world from on-premises!

Issue the same curl request from a different client device whose IP address lies outside of CLIENT_IP_RANGE, or simply update your security policy rule to no longer include your client IP address. You should now receive a 404 Not Found error.


My on-premises service is not accessible through the global external HTTP(S) load balancer IP address

Assuming that your on-premises service is already accessible on the Google Cloud VMs running Envoys, follow these steps to troubleshoot your setup.

Make sure the GCP Envoy MIG is reported as healthy. In the Google Cloud Console, go to Network services > Load balancing and click url-map lb-map-http to view its details. You should be able to see 1/1 of the instance in td-middle-proxy-us-central1-a is healthy.

If it’s not healthy, check if a firewall rule has been configured to allow ingress health-check traffic to the Google Cloud VMs that are running Envoys:

gcloud compute firewall-rules describe fw-allow-health-check

You should see the following output:

- IPProtocol: tcp
direction: INGRESS
disabled: false
- allow-hc

What's next