Anthos Service Mesh and Traffic Director are now Cloud Service Mesh. For more information, see the Cloud Service Mesh overview.

Options for Google Kubernetes Engine Pod setup using automatic Envoy injection

This guide provides you with information on additional options and tasks for the automatic Envoy sidecar injector.

Adding sidecar proxies to existing workloads

After you've installed the sidecar injector to your clusters, sidecar proxies are automatically injected to newly created pods in enabled namespaces. If you have workloads already running prior to enabling the sidecar injector, you must restart them for injection to take place.

For pods managed by Deployment, DaemonSet or StatefulSet controllers, you can run the following:

# Deployment
kubectl rollout restart deployment/DEPLOYMENT_NAME --namespace NAMESPACE

# DaemonSet
kubectl rollout restart daemonset/DAEMONSET_NAME --namespace NAMESPACE

# StatefulSet
kubectl rollout restart statefulset/STATEFULSET_NAME --namespace NAMESPACE

If you didn't use any of the above controllers to deploy your pod(s), you must delete the pod(s) individually. Afterwards, they are automatically recreated with new sidecar proxies.

kubectl delete pod POD_NAME -n NAMESPACE

Verify that a sidecar proxy container has been injected in each of your pods:

kubectl get pods -n NAMESPACE

For example, with the busybox client created above, you should see 2/2 pods running, one for the busybox application itself and one for the injected Envoy sidecar proxy:

NAME                      READY   STATUS    RESTARTS   AGE
busybox-c54f578c9-c9fk4   2/2     Running   183        7d15h

Injection overrides

By default, enabling a namespace enables sidecar proxy injection for all resident pods. Injection can also be selectively configured for different scopes to suit specific needs. For instance, overrides should be used to prevent sidecar proxy injection for proxyless gRPC services.

Note that the injection overrides only apply when the namespace is enabled, and take effect with the following priority: Pod Annotations > NeverInjectSelector > AlwaysInjectSelector > Default Policy

Enabling or disabling injection for specific individual pods

Use the following pod annotation to turn injection on or off for a specific pod in an enabled namespace:

...
metadata:
  annotations:
    td-injection: "true" / "false"

Customizing traffic interception behavior

By default, all outbound traffic from your application is intercepted and redirected to the Envoy sidecar proxy. The Envoy proxy can then handle the traffic according to the instructions received from Cloud Service Mesh. In some cases, you might want to modify this behavior to bypass the sidecar proxy.

Use the following Pod annotations to exclude traffic from interception and redirection.

Exclude from interception by outbound IP address range

You can exclude traffic from interception by IP address range.

...
metadata:
  annotations:
    cloud.google.com/excludeOutboundCIDRs: "10.0.0.1/32,169.254.169.254/32"

The cloud.google.com/excludeOutboundCIDRs Pod annotation is a comma-separated list of outbound IP address ranges in CIDR format. Egress traffic destined to these IP address ranges is not redirected to the Envoy sidecar.

Note that you need to list 169.254.169.254/32 in the Pod annotation to ensure that applications can communicate with the metadata server. If you don't specify the cloud.google.com/excludeOutboundCIDRs Pod annotation, traffic interception is configured to exclude the "169.254.169.254/32" outbound CIDR range.

Include in interception by outbound IP address range

You can include traffic in interception by IP address range.

...
metadata:
  annotations:
    cloud.google.com/includeOutboundCIDRs: "10.0.0.1/32,169.254.169.254/32"

The cloud.google.com/includeOutboundCIDRs Pod annotation is a comma-separated list of outbound IP address ranges in CIDR format. Egress traffic destined to these IP address ranges is redirected to the Envoy sidecar.

The wildcard character * can be used to redirect all outbound traffic. An empty list disables all outbound traffic. The annotation defaults to *.

Exclude from interception by outbound port number

You can exclude traffic from interception and redirection by outbound port number.

...
metadata:
  annotations:
    cloud.google.com/excludeOutboundPorts: "10001, 10002"

The cloud.google.com/excludeOutboundPorts Pod annotation is a comma-separated list of outbound ports. Egress traffic destined to these ports is excluded from interception and redirection to the Envoy sidecar.

If you don't specify the cloud.google.com/excludeOutboundPorts annotation, outbound traffic destined for any port is intercepted and redirected to the Envoy sidecar. This is equivalent to passing the cloud.google.com/excludeOutboundPorts annotation with an empty ("") list.

Include in interception by inbound port number

You can include traffic in interception by inbound port number.

...
metadata:
  annotations:
    cloud.google.com/includeInboundPorts: "10001, 10002"

The cloud.google.com/includeInboundPorts Pod annotation is a comma-separated list of inbound ports for which traffic is to be redirected to the Envoy sidecar. The wildcard character * can be used to configure redirection for all ports. An empty value disables all inbound redirection. The value defaults to an empty string ("").

Exclude from interception by inbound port number

You can exclude traffic from interception by inbound port number.

...
metadata:
  annotations:
    cloud.google.com/excludeInboundPorts: "10001, 10002"

The cloud.google.com/excludeInboundPorts Pod annotation is a comma separated list of inbound ports to be excluded from redirection to the Envoy sidecar. The annotation only applies when all inbound traffic (*) is being redirected. The value defaults to an empty string ("").

Enable managed certificates

You can enable managed workload certificates.

...
metadata:
  annotations:
    cloud.google.com/enableManagedCerts: "true"

When the Pod annotation cloud.google.com/enableManagedCerts is set to true, GKE managed workload certificates signed by Certificate Authority Service are inserted and mounted on the sidecar container. The value of the annotation defaults to false.

Configuring sidecar proxy metadata

To support additional Cloud Service Mesh features, sidecar proxies can inherit specific metadata from their encapsulating Pods. There are two ways to accomplish this. Both options append metadata and share the metadata with Cloud Service Mesh when the sidecar proxy connects to Cloud Service Mesh. The options are mutually exclusive.

The first option allows you to specify individual metadata key/value pairs. For example, include the following annotation in your Pod template specification to apply the "version": "dev" label to its injected sidecar proxies.

...
metadata:
  annotations:
    cloud.google.com/proxyMetadata: '{"version": "dev"}'

The second option appends all of the Pod's labels to the Pod's injected sidecar proxy.

...
metadata:
  annotations:
    cloud.google.com/forwardPodLabels: "true"

If you don't specify the cloud.google.com/forwardPodLabels annotation, Pod labels will not be appended to the sidecar proxy. Note that the cloud.google.com/proxyMetadata and cloud.google.com/forwardPodLabels annotations are mutually exclusive. If you set both, cloud.google.com/forwardPodLabels takes priority and cloud.google.com/proxyMetadata is ignored.

Config filtering then allows Cloud Service Mesh to share a subset of configuration only with the specific proxies that match on this "version": "dev" label.

Existing deployments must be restarted for this configuration to take effect.

Supported pod annotations

Cloud Service Mesh supports the following Pod annotations for sidecar injection. Though additional sidecar injector annotations might work, the following list represents annotations that Cloud Service Mesh supports. To avoid breakage or instability, do not create a dependency on other annotations in your production deployment.

Annotation Name	Value	Description
td-injection	Boolean, represented as a string. For example: "`true`"	Specifies whether or not an Envoy sidecar should be automatically injected into the workload.
cloud.google.com/proxyMetadata	JSON map of key-value pairs. For example: "`'{"version": "dev"}'`"	Specifies the key/value pairs in a JSON map that should be appended to Envoy metadata.
cloud.google.com/forwardPodLabels	"true" or "false"	When set to "true", all Pod labels are appended to Envoy metadata, and the "cloud.google.com/proxyMetadata" annotation is ignored. Defaults to "false".
cloud.google.com/excludeOutboundPorts	Comma-separated list of outbound ports	Egress traffic that indicates any of these destination ports are excluded from interception/redirection to the Envoy sidecar. This traffic will bypass the Envoy proxy and will not be handled according to Cloud Service Mesh configuration. Defaults to an empty string (i.e. "").
cloud.google.com/includeInboundPorts	Comma-separated list of inbound ports	A comma-separated list of inbound ports for which traffic is redirected to the Envoy sidecar. Use the wildcard character `*` to configure redirection for all ports. An empty value disables all inbound redirection. The value defaults to an empty string ("").
cloud.google.com/excludeInboundPorts	Comma-separated list of inbound ports	A comma-separated list of inbound ports for which traffic is not redirected to the Envoy sidecar. The annotation applies only when all inbound traffic (*) is being redirected. The value defaults to an empty string ("").
cloud.google.com/excludeOutboundCIDRs	Comma-separated list of outbound IP ranges in CIDR form.	Egress traffic that indicates any of these destination IPs are excluded from interception/redirection to the Envoy sidecar. This traffic will bypass the Envoy proxy and will not be handled according to Cloud Service Mesh configuration. Defaults to "169.254.169.254/32", which is the range required to communicate with the metadata server. Note that this range is required so if you specify the `excludeOutboundCIDRs` annotation, ensure that you also include "169.254.169.254/32" in addition to any other CIDRs. Ensure that there are no spaces in the comma-separated list.
cloud.google.com/includeOutboundCIDRs	Comma-separated list of outbound IP ranges in CIDR form.	Egress traffic that indicates any of these destination IPs are included in interception/redirection to the Envoy sidecar. This traffic is directed to the Envoy proxy and is handled according to Cloud Service Mesh configuration. Defaults to "169.254.169.254/32", which is the range required to communicate with the metadata server. Note that this range is required so if you specify the `includeOutboundCIDRs` annotation, ensure that you also include "169.254.169.254/32" in addition to any other CIDRs. Ensure that there are no spaces in the comma-separated list.
cloud.google.com/enableManagedCerts	Boolean, represented as a string. For example: "`true`"	When set to "`true`", GKE managed workload certificates signed by Certificate Authority Service are inserted and mounted on the sidecar container. The default value is "`false`".

Uninstalling the sidecar injector

Uninstall the sidecar injector with the following commands:

kubectl delete MutatingWebhookConfiguration td-mutating-webhook
kubectl label namespace default td-injection-