Options for Google Kubernetes Engine Pod setup with automatic Envoy injection

This guide provides you with information on additional options and tasks for the automatic Envoy sidecar injector.

Adding sidecar proxies to existing workloads

After you've installed the sidecar injector to your clusters, sidecar proxies are automatically injected to newly created pods in enabled namespaces. If you have workloads already running prior to enabling the sidecar injector, you must restart them for injection to take place.

For pods managed by Deployment, DaemonSet or StatefulSet controllers, you can run the following:

# Deployment
kubectl rollout restart deployment/DEPLOYMENT_NAME --namespace NAMESPACE

# DaemonSet
kubectl rollout restart daemonset/DAEMONSET_NAME --namespace NAMESPACE

# StatefulSet
kubectl rollout restart statefulset/STATEFULSET_NAME --namespace NAMESPACE

If you didn't use any of the above controllers to deploy your pod(s), you must delete the pod(s) individually. Afterwards, they are automatically recreated with new sidecar proxies.

kubectl delete pod POD_NAME -n NAMESPACE

Verify that a sidecar proxy container has been injected in each of your pods:

kubectl get pods -n NAMESPACE

For example, with the busybox client created above, you should see 2/2 pods running, one for the busybox application itself and one for the injected Envoy sidecar proxy:

NAME                      READY   STATUS    RESTARTS   AGE
busybox-c54f578c9-c9fk4   2/2     Running   183        7d15h

Injection overrides

By default, enabling a namespace enables sidecar proxy injection for all resident pods. Injection can also be selectively configured for different scopes to suit specific needs. For instance, overrides should be used to prevent sidecar proxy injection for proxyless gRPC services.

Note that the injection overrides only apply when the namespace is enabled, and take effect with the following priority: Pod Annotations > NeverInjectSelector > AlwaysInjectSelector > Default Policy

Enabling/disabling injection for specific individual pods

Use the following pod annotation to turn injection on or off for a specific pod in an enabled namespace:

...
metadata:
  annotations:
    sidecar.istio.io/inject: "true" / "false"

Enabling/disabling injection for specific groups of pods

The sidecar injector itself can be configured to always or never inject pods in enabled namespaces based on an array of Kubernetes label selectors. For example, use the following commands to configure the sidecar injector to not inject a sidecar proxy if the pod has the label "run=client":

kubectl edit configmap -n istio-control istio-sidecar-injector

...
config: |-
  policy: enabled
  alwaysInjectSelector:
    []

  neverInjectSelector:
    - matchLabels:
        run: client
...

Existing sidecar injector deployments must be restarted for this configuration to take effect.

Customizing traffic interception behavior

By default, all outbound traffic from your application is intercepted and redirected to the Envoy sidecar proxy. The Envoy proxy can then handle the traffic according to the instructions received from Traffic Director. In some cases, you might want to modify this behavior to bypass the sidecar proxy.

Use the following Pod annotations to exclude traffic from interception and redirection.

Exclude from interception by IP address range

...
metadata:
  annotations:
    cloud.google.com/excludeOutboundCIDRs: "10.0.0.1/32, 169.254.169.254/32"

This Pod annotation is a comma-separated list of outbound IP address ranges in CIDR format. Egress traffic destined to these IP address ranges is not redirected to the Envoy sidecar.

Note that you need to list 169.254.169.254/32 in the Pod annotation to ensure that applications can communicate with the metadata server. If you don't specify the cloud.google.com/excludeOutboundCIDRs Pod annotation, traffic interception is configured to exclude the "169.254.169.254/32" outbound CIDR range.

Exclude from interception by port number

You can also exclude traffic from interception and redirection by port.

...
metadata:
  annotations:
    cloud.google.com/excludeOutboundPorts: "10001, 10002"

This Pod annotation is a comma-separated list of outbound ports. Egress traffic destined to these ports is excluded from interception and redirection to the Envoy sidecar.

If you don't specify the cloud.google.com/excludeOutboundPorts annotation, outbound traffic destined for any port is intercepted and redirected to the Envoy sidecar. This is equivalent to passing the cloud.google.com/excludeOutboundPorts annotation with an empty ("") list.

Configuring sidecar proxy metadata

To support additional Traffic Director features, sidecar proxies can inherit specific metadata from their encapsulating Pods. There are two ways to accomplish this. Both options append metadata and share the metadata with Traffic Director when the sidecar proxy connects to Traffic Director. The options are mutually exclusive.

The first option allows you to specify individual metadata key/value pairs. For example, include the following annotation in your Pod template specification to apply the "version": "dev" label to its injected sidecar proxies.

...
metadata:
  annotations:
    cloud.google.com/proxyMetadata: '{"version": "dev"}'

The second option appends all of the Pod's labels to the Pod's injected sidecar proxy.

...
metadata:
  annotations:
    cloud.google.com/forwardPodLabels: "true"

If you don't specify the cloud.google.com/forwardPodLabels annotation, Pod labels will not be appended to the sidecar proxy. Note that the cloud.google.com/proxyMetadata and cloud.google.com/forwardPodLabels annotations are mutually exclusive. If you set both, cloud.google.com/forwardPodLabels takes priority and cloud.google.com/proxyMetadata is ignored.

Config filtering then allows Traffic Director to share a subset of configuration only with the specific proxies that match on this "version": "dev" label.

Existing deployments must be restarted for this configuration to take effect.

Supported pod annotations

Traffic Director supports the following Pod annotations for sidecar injection. Though additional sidecar injector annotations might work, the following list represents annotations that Traffic Director supports. To avoid breakage or instability, do not create a dependency on other annotations in your production deployment.

Annotation Name Value Description
sidecar.istio.io/inject Boolean, represented as a string. For example: "true" Specifies whether or not an Envoy sidecar should be automatically injected into the workload.
cloud.google.com/proxyMetadata JSON map of key-value pairs. For example: "'{"version": "dev"}'" Specifies the key/value pairs in a JSON map that should be appended to Envoy metadata.
cloud.google.com/forwardPodLabels "true" or "false" When set to "true", all Pod labels are appended to Envoy metadata, and the "cloud.google.com/proxyMetadata" annotation is ignored. Defaults to "false".
cloud.google.com/excludeOutboundPorts Comma-separated list of outbound ports Egress traffic that indicates any of these destination ports are excluded from interception/redirection to the Envoy sidecar. This traffic will bypass the Envoy proxy and will not be handled according to Traffic Director configuration. Defaults to an empty string (i.e. "").
cloud.google.com/excludeOutboundCIDRs Comma-separated list of outbound IP ranges in CIDR form. Egress traffic that indicates any of these destination IPs are excluded from interception/redirection to the Envoy sidecar. This traffic will bypass the Envoy proxy and will not be handled according to Traffic Director configuration. Defaults to "169.254.169.254/32", which is the range required to communicate with the metadata server. Note that this range is required so if you specify the `excludeOutboundCIDRs` annotation, ensure that you also include "169.254.169.254/32" in addition to any other CIDRs.

Uninstalling the sidecar injector

Uninstall the sidecar injector with the following commands:

kubectl delete -f specs/
kubectl label namespace default istio-injection-