Options for Compute Engine VM setup using automatic Envoy deployment
This guide provides you with information on additional options and tasks for automated Envoy deployment.
Additional instance template creation options
When you create an instance template for automated Envoy proxy deployment, you can use the following parameters to define some aspects of the deployment and the behavior of the proxies.
|Parameter||Value and Description||Required or optional|
Controls whether the service proxy and agent are installed and configured on the VM.
|Required if you want to automatically deploy and configure the service proxy. If you omit this setting, the service proxy isn't installed or configured.|
||A semicolon-separated list of ports.
The port(s) on which your application/workload is serving. The service proxy intercepts inbound traffic, then forwards it to the specified serving port(s) on localhost.
If you omit this flag, the service proxy only handles outbound traffic from your workload. Inbound traffic is not handled by the service proxy.
||A single port.
The port on which the service proxy listens. The VM intercepts traffic and redirects it to this port for handling by the service proxy.
If you omit this flag, the value defaults to
||The name of a valid VPC network.
The Google Cloud VPC network used by the service proxy's control plane to generate dynamic configuration for the service proxy.
|Required if the VM is on more than one network; otherwise, this is optional.
If you omit this flag, the value that is specified using the
Enables the service proxy to generate distributed tracing information. If set to
For more information, refer to the
If you omit this flag, tracing is not enabled.
||The file path for access logs sent to the service proxy by the control plane. All incoming and outgoing requests are recorded in this file.
For more information, refer to the File access log documentation for the Envoy proxy.
|Optional. There is no default value. If you do not specify the file path, the logs are not created.|
||Enables interception of all outgoing traffic by the service proxy and then redirect to an external host. Use
||A semicolon-separated (;) list of the IP addresses or CIDR ranges, specified inside quotation marks ("), that should be excluded from redirection. Only applies when the
||A semicolon-separated (;) list of the ports or port ranges, specified inside quotation marks ("), that should be excluded from redirection. Only applies when the
||Key/value pairs in the format
Labels that you can apply to your service proxy. These are reflected in your Envoy proxy's bootstrap metadata. The labels can be any
For example, the following
gcloud command creates an instance template
proxy-it. Instances created from this template have the Envoy proxy and
service proxy agent installed.
gcloud compute instance-templates create proxy-it \ --service-proxy enabled
In the following example, the instance template is called
proxy-it, the Envoy
proxy and service proxy agent are installed, the serving port and proxy port are
set, tracing is enabled, and a label is defined.
gcloud compute instance-templates create proxy-it \ --service-proxy enabled,serving-ports=8080,proxy-port=15001,tracing=ON \ --service-proxy-labels version=canary
The following diagram illustrates the traffic flow when you specify the serving
8080. Inbound TCP connections to port
8080 are intercepted by
iptables and redirected to the Envoy proxy, which then passes them to the
application on your VM listening on TCP port
8080. In addition:
- All outbound connections to the VIPs of services configured in Traffic Director are intercepted by iptables, which configures netfilter. Netfilter ensures that the corresponding traffic traversing the network stack is intercepted and redirected to the Envoy proxy. The traffic is then load balanced based on how you have Traffic Director configured.
- All other inbound connections are not intercepted by iptables, and are passed directly services on the VM.
- All connections to external endpoints are directly passed to external IP addresses without being intercepted.
- All UDP traffic is passed directly to the destination without being intercepted by iptables.
This is illustrated in the following diagram.
The traffic flows in the diagram are as follows:
- Inbound traffic with destination port
80, not intercepted and routed directly to service listening on the port.
- Inbound traffic with destination port
8080, intercepted and redirected to the Envoy listener port.
- Envoy forwards traffic from (2) to service 2 listening on localhost port
- Outbound traffic destined to Traffic Director forwarding rule VIP and port, intercepted and redirected to Envoy listener port.
- Envoy forwards traffic from (4) to endpoint of backend of destination Traffic Director backend service.
- Outbound traffic destined to non-Traffic Director VIP and port, not intercepted, and routed directly to external service.
If you want to specify labels to use with Traffic Director metadata
filtering or enable access logging for the Envoy proxies, use the
--service-proxy access-log parameters.
gcloud compute instance-templates create td-vm-template-auto \ --service-proxy enabled,access-log=/var/log/envoy/access.log,network=default \ --service-proxy-labels myapp=review,version=canary
The Envoy proxy can intercept the health check ports for services on the VMs. If you do this, the health check probes report on your application and on the Envoy proxies. Traffic is not directed to the VM if the Envoy proxy is not running correctly. For example:
gcloud compute instance-templates create td-vm-template-auto \ --service-proxy=enabled,serving-ports="80;8080"
Using the managed instance group update process
- Add the service proxy components to an existing managed instance group and enroll it in a Traffic Director network.
- Update the service proxy components on the VMs.
Before you perform the rolling update, do the following.
- Set connection draining on the backend service to a value of at least 30 seconds.
- Use the
--min-readyparameter, set to a value of 3 minutes or more, when you call the updater. The
--min-readyparameter makes the managed instance group wait after a VM is updated before proceeding to update the next VM. Without this, the newly-created VM has no time to fully boot up Envoy and the service proxy agent, and the update proceeds too quickly.
To perform the rolling update on the managed instance group, follow these steps.
- Create a new instance template, as described above, with the service proxy information. The original version of the instance template can be used for the update if it contains the service proxy information and your goal is to update to the most recent stable version of the proxy software.
- Perform a rolling update of the managed instance group. Make sure that you
REPLACEas the minimum action that the Updater must perform. The instance group installs the latest version of the proxy software and configures it as specified in the instance template.
You can also remove the service proxy components from a managed instance group using the update process:
Create a new instance template without specifying the flag
Perform a rolling update using the managed instance group update process.
This removes the Envoy proxy from your VMs. If that managed instance group was the only MIG attached to the backend service, you might want to remove the Traffic Director configuration that you created when you set up Traffic Director.