Enabling optional features on the in-cluster control plane
This page describes how to enable optional features on an in-cluster control plane. For information on the managed control plane, see Configure managed Anthos Service Mesh.
When you install Anthos Service Mesh, the features that are
enabled by default differ by platform. You
enable optional features by including an overlay file when you install
(or upgrade) Anthos Service Mesh. An overlay file is a YAML file containing an
IstioOperator
custom resource (CR)
that you use to configure the control plane. You can override the default
configuration and enable an optional feature or disable a default
feature in an overlay file. Specify one feature per overlay file. You can
layer on more overlays, and each overlay file overrides the configuration on the
previous layers.
About the overlay files
The overlay files on this page are in the
anthos-service-mesh
package in GitHub. These files contain common customizations to the default
configuration. You can use these files as they are, or you can make additional
changes to them as needed.
When you install Anthos Service Mesh using the Google-provided
asmcli
script, you
can specify one or more overlay files with the --option
or the
--custom_overlay
options. If you don't need to make any changes to the
files in the anthos-service-mesh
repository, you can use --option
, and
the script fetches the file from GitHub for you. Otherwise, you can make
changes to the overlay file, and then use the --custom_overlay
option to
pass it to the asmcli
.
Don't include multiple CRs in one overlay file | Create separate overlay files for each CR |
---|---|
![]() |
![]() |
How to enable optional features
The following examples are simplified to show only using the custom overlays to
enable optional features. Replace OTHER_FLAGS
with the
other command line options.
The asmcli install
command provides two ways to enable an optional feature. The
method that you use depends on whether you need to make changes to the overlay
file.
Use
--option
when you don't need to make any changes to the overlay file. With--option
,asmcli
fetches the file from the GitHub repository for you, so you must have an internet connection../asmcli install \ OTHER_FLAGS \ --option OPTION_NAME
Replace
OPTION_NAME
with the option you want to enable. For a list of options refer to theanthos-service-mesh
package.Use
--custom_overlay
when you need to customize the overlay file../asmcli install \ OTHER_FLAGS \ --custom_overlay PATH_TO_FILE
Replace
PATH_TO_FILE
with the path to the overlay file you want to use.
YAML for optional features
The following sections provide the YAML to enable optional and supported features.
mTLS STRICT
mode
The global.mtls.enabled
configuration was removed from the IstioOperator
CR to avoid issues with upgrades and to provide a more flexible installation.
To enable STRICT
mTLS,
configure a
peer authentication policy instead.
Distroless proxy image
As a best practice, you should restrict the contents of a container runtime to only the necessary packages. This approach improves security and the signal-to-noise ratio of Common Vulnerabilities and Exposures (CVE) scanners. Istio provides proxy images based on distroless base images.
The following configuration enables distroless images for the entire Anthos Service Mesh. An image type change requires each pod to restart and get re-injected to take effect.
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
defaultConfig:
image:
imageType: distroless
The distroless image does not contain any binaries other than the proxy. It is therefore not possible to exec
a shell or use curl
, ping
, or other debug utilities inside the container. If you try to execute a shell you will see the following error.
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "<container-id>"
OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "sh": executable file not found in $PATH: unknown
If you need access to these tools for specific pods, you may override the imageType
by using the following pod annotation.
sidecar.istio.io/proxyImageType: debug
After changing the image type of a deployment via the annotation, the deployment should be restarted.
kubectl rollout restart deployment -n NAMESPACE DEPLOYMENT_NAME
For most types of proxy debugging, istioctl proxy-cmd
should be used which does not require a debug base image.
Use a custom overlay for custom registry
You can use a custom overlay for custom registries, such as if you need to install Anthos Service Mesh from a custom container registry. For example:
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
hub: {private_registry_url}
The following is a list of images for Anthos Service Mesh that you need to mirror to the custom container registry:
- Install-cni -
gcr.io/gke-release/asm/install-cni:1.14.6-asm.16
- Managed Data Plane -
gcr.io/gke-release/asm/mdp:1.14.6-asm.16
- Pilot -
gcr.io/gke-release/asm/pilot:1.14.6-asm.16
- Proxyv2 -
gcr.io/gke-release/asm/proxyv2:1.14.6-asm.16
Add images to a private registry
To push Anthos Service Mesh images to a private registry, complete the following steps.
-
Pull the Anthos Service Mesh images:
docker pull gcr.io/gke-release/asm/install-cni:1.14.6-asm.16 docker pull gcr.io/gke-release/asm/mdp:1.14.6-asm.16 docker pull gcr.io/gke-release/asm/pilot:1.14.6-asm.16 docker pull gcr.io/gke-release/asm/proxyv2:1.14.6-asm.16
-
Create a variable for your private registry URL:
export PRIVATE_REGISTRY_URL=PRIVATE_REGISTRY_URL
ReplacePRIVATE_REGISTRY_URL
with your private registry URL. -
Tag the images with your private registry URL:
docker tag gcr.io/gke-release/asm/install-cni:1.14.6-asm.16 \ ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/install-cni:1.14.6-asm.16 docker tag gcr.io/gke-release/asm/mdp:1.14.6-asm.16 \ ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/mdp:1.14.6-asm.16 docker tag gcr.io/gke-release/asm/pilot:1.14.6-asm.16 \ ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/pilot:1.14.6-asm.16 docker tag gcr.io/gke-release/asm/proxyv2:1.14.6-asm.16 \ ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/proxyv2:1.14.6-asm.16
- Push the tagged images to your private registry:
docker push ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/install-cni:1.14.6-asm.16 docker push ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/mdp:1.14.6-asm.16 docker push ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/pilot:1.14.6-asm.16 docker push ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/proxyv2:1.14.6-asm.16
- (Optional) If you use a
canonical service, then add the
canonical service images to your private registry.
- Pull the Anthos Service Mesh canonical service images:
docker pull gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0 docker pull gcr.io/gke-release/asm/canonical-service-controller:1.10.3-asm.16
- Tag the images with your private registry URL:
docker tag gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0 \ ${PRIVATE_REGISTRY_URL}/gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0 docker tag gcr.io/gke-release/asm/canonical-service-controller:1.10.3-asm.16 \ ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/canonical-service-controller:1.10.3-asm.16
- Push the tagged images to your private registry:
docker push ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/kube-rbac-proxy:v0.5.0 docker push ${PRIVATE_REGISTRY_URL}/gcr.io/gke-release/asm/canonical-service-controller:1.10.3-asm.16
- Pull the Anthos Service Mesh canonical service images:
If you can pull the tagged images from your private registry, then the procedure was successful.
Direct Envoy to stdout
For more information, see Enable Envoy's access logging.
Cloud Trace
Cloud Trace is available with Anthos Service Mesh installations on the following platforms:
- GKE on Google Cloud
- GKE Enterprise clusters on-premises if you install with Anthos Service Mesh certificate authority (Mesh CA)
For detailed pricing information, refer to the Cloud Trace pricing page.
The default sampling rate is 1%, but you can override the default by specifying
a tracing.sampling
value. The value must be in the range of 0.0 to 100.0 with a
precision of 0.01. For example, to trace 5 requests out of every 10,000, use
0.05.
The following example shows a sampling rate of 100% (which you would only do for demo or troubleshooting purposes).
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
meshConfig:
enableTracing: true
defaultConfig:
tracing:
sampling: 100
values:
global:
proxy:
tracer: stackdriver
Note that currently tracer configuration is part of proxy bootstrap configuration, so pod needs to restart and get re-injected to pick up the tracer update. For example, you can use the following command the restart pods belong to a deployment:
kubectl rollout restart deployment -n NAMESPACE DEPLOYMENT_NAME
Trace context propagation
Although the sidecar proxies can automatically send trace spans, they need some hints to tie together the entire trace. Applications need to propagate the appropriate HTTP headers so that when the proxies send span information, the spans can be correlated correctly into a single trace.
To do this, an application needs to collect and propagate the appropriate headers from the incoming request to any outgoing requests. The Anthos Service Mesh Stackdriver tracing configuration will accept any of the following header formats, and will propagate all of the following formats:
- B3 (
x-b3-traceid
,x-b3-spanid
,x-b3parentspanid
,x-b3-sampled
,x-b3-flags
) - W3C TraceContext (
traceparent
) - Google Cloud Trace (
x-cloud-trace-context
) - gRPC TraceBin (
grpc-trace-bin
)
This means that your applications can use any of those formats to propagate tracing context and the traces will be generated and set to Stackdriver appropriately.
Example
Here is an example HTTP-Get request with a traceparent
header in the original
request. Notice the additional trace context headers added by the proxy.
$ kubectl exec -it sleep-557747455f-n6flv -- curl "httpbin:8000/anything?freeform=" -H "accept: application/json" -H "Traceparent: 00-7543d15e09e5d61801d4f74cde1269b8-604ef051d35c5b3f-01" -vv
* Trying 10.12.3.52:8000...
* Connected to httpbin (10.12.3.52) port 8000 (#0)
> GET /anything?freeform= HTTP/1.1
> Host: httpbin:8000
> User-Agent: curl/7.80.0-DEV
> accept: application/json
> Traceparent: 00-7543d15e09e5d61801d4f74cde1269b8-604ef051d35c5b3f-01
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: envoy
< date: Wed, 10 Nov 2021 20:36:04 GMT
< content-type: application/json
< content-length: 1032
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 5
<
{
"args": {
"freeform": ""
},
"data": "",
"files": {},
"form": {},
"headers": {
"Accept": "application/json",
"Grpc-Trace-Bin": "AAB1Q9FeCeXWGAHU90zeEmm4AaDHmGRtdM7wAgE",
"Host": "httpbin:8000",
"Traceparent": "00-7543d15e09e5d61801d4f74cde1269b8-a0c798646d74cef0-01",
"User-Agent": "curl/7.80.0-DEV",
"X-B3-Sampled": "1",
"X-B3-Spanid": "a0c798646d74cef0",
"X-B3-Traceid": "7543d15e09e5d61801d4f74cde1269b8",
"X-Cloud-Trace-Context": "7543d15e09e5d61801d4f74cde1269b8/11585396123534413552;o=1",
"X-Envoy-Attempt-Count": "1",
"X-Forwarded-Client-Cert": "<REDACTED>"
},
"json": null,
"method": "GET",
"origin": "127.0.0.6",
"url": "http://httpbin:8000/anything?freeform="
}
Notice that in the returned set of request headers, the full set of trace context headers is present.
For more examples propagating the headers, see Trace context propagation.
Create a trace from client with custom ID
To create a trace from a client with a custom ID, use the curl
command to
create a request with an external client and force it to show a trace. For example:
curl $URL --header "x-client-trace-id: 105445aa7843bc8bf206b12000100000"
For more information about x-client-trace-id
, refer to the
Envoy documentation.
Egress via egress gateways
We recommend that you install an injected gateway as described in Install and upgrade gateways.
Istio Container Network Interface
How you enable the Istio Container Network Interface (CNI) depends on the environment that Anthos Service Mesh is installed on.
Choose the overlay file that matches your platform.
Enable CNI on GKE
Enable CNI on-premises
Enable Google Cloud's operations suite for off-Google Cloud
Installing Anthos Service Mesh with Istio CA outside of Google Cloud reports metrics to Prometheus by default. Use this option to enable reporting metrics to Google Cloud's operations suite instead, or both Prometheus and Stackdriver, so you can use the Anthos Service Mesh dashboards.
Only Stackdriver
Stackdriver and Prometheus
Enable an internal load balancer
We recommend that you install an
injected gateway as described in
Install and upgrade gateways to set up an internal
load balancer on GKE. When configuring the gateway Service,
you include the annotation: networking.gke.io/load-balancer-type: "Internal"
External certificate management on the ingress gateway
For information on enabling external certificate management on the ingress gateway using Envoy SDS, see Secure Gateways.