This guide explains how to do a clean installation of Anthos Service Mesh version
1.11.8-asm.4 on GKE on VMware and on bare metal. If
you have a previous version of Anthos Service Mesh installed, refer to
Upgrading Anthos Service Mesh on-premises.
The installation guide shows you how to install Anthos Service Mesh certificate
authority (Mesh CA) or the Istio certificate authority (previously known as
Citadel). This guide refers to the cluster as cluster1
, but you can repeat
these steps to set up many clusters.
About the control plane components
GKE on VMware and bare metal come with the following Istio components preinstalled:
- The Istio certificate authority (previously known as Citadel) is installed in
the
kube-system
namespace. - The Istio Ingress Gateway and other Istio components are installed in the
gke-system
namespace.
GKE on VMware and on bare metal use these components to enable ingress and to secure communication between Google-controlled components.
When you install Anthos Service Mesh, its components are installed in the
istio-system
namespace. Because the Anthos Service Mesh components are in a
different namespace, they don't conflict with the preinstalled Istio components.
Before you begin
Review the following requirements before you begin setup.
Requirements
You must have an Anthos subscription. Alternatively, a pay-as-you-go billing option is available for GKE Enterprise on Google Cloud only. For more information, see the GKE Enterprise Pricing guide.
Verify that your user cluster that you install Anthos Service Mesh on has at least 4 vCPUs, 15 GB memory, and 4 nodes.
Mesh CA Only: Your user cluster nodes need the Internet to complete Anthos Service Mesh installation successfully. Internet access through an HTTP proxy is not possible.
You must name your service ports using the following syntax:
name: protocol[-suffix]
where the square brackets indicate an optional suffix that must start with a dash. For more information, see Naming service ports.Verify that your cluster version is listed in Supported platforms.
Setting up your environment
You need the following tools on the computer where you control the installation process. Note that you can install Anthos Service Mesh only on a user cluster, not an admin cluster.
- The
curl
command-line tool. - The Google Cloud CLI.
After installing the Google Cloud CLI:
Authenticate with the Google Cloud CLI:
gcloud auth login
Update the components:
gcloud components update
Install
kubectl
:gcloud components install kubectl
Install
kpt
:curl -L https://github.com/GoogleContainerTools/kpt/releases/download/v0.39.2/kpt_linux_amd64 > kpt_0_39_2 chmod +x kpt_0_39_2 alias kpt="$(readlink -f kpt_0_39_2)"
Setting environment variables
Get the context name for the cluster by using the values under the
NAME
column in the output of this command:kubectl config get-contexts
Set the environment variables to the cluster context name, which this guide uses in many steps later:
export CTX_CLUSTER1=CLUSTER1_CONTEXT_NAME
Granting cluster admin permissions
Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:
kubectl --context="${CTX_CLUSTER1}" create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=USER_ACCOUNT
Downloading the installation file
Linux
Download the Anthos Service Mesh installation file to your current working directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.11.8-asm.4-linux-amd64.tar.gz
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.11.8-asm.4-linux-amd64.tar.gz.1.sig openssl dgst -verify /dev/stdin -signature istio-1.11.8-asm.4-linux-amd64.tar.gz.1.sig istio-1.11.8-asm.4-linux-amd64.tar.gz <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
.Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
tar xzf istio-1.11.8-asm.4-linux-amd64.tar.gz
The command creates an installation directory in your current working directory named
istio-1.11.8-asm.4
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
manifests/profiles
directory.
- Sample applications in the
Ensure that you're in the Anthos Service Mesh installation's root directory.
cd istio-1.11.8-asm.4
Mac OS
Download the Anthos Service Mesh installation file to your current working directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.11.8-asm.4-osx.tar.gz
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.11.8-asm.4-osx.tar.gz.1.sig openssl dgst -sha256 -verify /dev/stdin -signature istio-1.11.8-asm.4-osx.tar.gz.1.sig istio-1.11.8-asm.4-osx.tar.gz <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
.Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
tar xzf istio-1.11.8-asm.4-osx.tar.gz
The command creates an installation directory in your current working directory named
istio-1.11.8-asm.4
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
manifests/profiles
directory.
- Sample applications in the
Ensure that you're in the Anthos Service Mesh installation's root directory.
cd istio-1.11.8-asm.4
Windows
Download the Anthos Service Mesh installation file to your current working directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.11.8-asm.4-win.zip
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.11.8-asm.4-win.zip.1.sig openssl dgst -verify - -signature istio-1.11.8-asm.4-win.zip.1.sig istio-1.11.8-asm.4-win.zip <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
.Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:
tar xzf istio-1.11.8-asm.4-win.zip
The command creates an installation directory in your current working directory named
istio-1.11.8-asm.4
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
manifests/profiles
directory.
- Sample applications in the
Ensure that you're in the Anthos Service Mesh installation's root directory.
cd istio-1.11.8-asm.4
Installing Anthos Service Mesh with Mesh CA
This section explains how to install Anthos Service Mesh with Mesh CA and register the cluster to the fleet. We recommend that you use Mesh CA for the following reasons:
- Mesh CA is a highly reliable and scalable service that is optimized for dynamically scaled workloads on Google Cloud and on-premises.
- With Mesh CA, Google manages the security and availability of the CA backend.
- Mesh CA lets you rely on a single root of trust across clusters.
If you want to use Istio CA with your own root CA, see Configuring the Istio CA.
The Mesh CA feature depends on fleet registration. By default, your cluster is registered to a fleet. To verify your cluster's status, see Viewing registered clusters.
Enable the following APIs:
gcloud services enable \ anthos.googleapis.com \ cloudtrace.googleapis.com \ cloudresourcemanager.googleapis.com \ container.googleapis.com \ compute.googleapis.com \ gkeconnect.googleapis.com \ gkehub.googleapis.com \ iam.googleapis.com \ iamcredentials.googleapis.com \ logging.googleapis.com \ meshca.googleapis.com \ meshtelemetry.googleapis.com \ meshconfig.googleapis.com \ monitoring.googleapis.com \ stackdriver.googleapis.com \ sts.googleapis.com
Initialize the mesh configuration:
IDENTITY_PROVIDER="$(kubectl get memberships.hub.gke.io membership -o=jsonpath='{.spec.identity_provider}')" IDENTITY="$(echo "${IDENTITY_PROVIDER}" | sed 's/^https:\/\/gkehub.googleapis.com\/projects\/\(.*\)\/locations\/global\/memberships\/\(.*\)$/\1 \2/g')" read -r ENVIRON_PROJECT_ID HUB_MEMBERSHIP_ID <<EOF ${IDENTITY} EOF POST_DATA='{"workloadIdentityPools":["'${ENVIRON_PROJECT_ID}'.hub.id.goog","'${ENVIRON_PROJECT_ID}'.svc.id.goog"]}' curl --request POST \ --header "Content-Type: application/json" \ --header "Authorization: Bearer $(gcloud auth print-access-token)" \ --data "${POST_DATA}" \ https://meshconfig.googleapis.com/v1alpha1/projects/${ENVIRON_PROJECT_ID}:initialize
Configure the installation:
kpt pkg get \ https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages.git/asm@release-1.11 asm ENVIRON_PROJECT_NUMBER=$(gcloud projects describe "${ENVIRON_PROJECT_ID}" --format="value(projectNumber)") CLUSTER_NAME="${HUB_MEMBERSHIP_ID}" CLUSTER_LOCATION="global" HUB_IDP_URL="$(kubectl get memberships.hub.gke.io membership -o=jsonpath='{.spec.identity_provider}')" kpt cfg set asm gcloud.core.project ${ENVIRON_PROJECT_ID} kpt cfg set asm gcloud.container.cluster ${CLUSTER_NAME} kpt cfg set asm gcloud.compute.location ${CLUSTER_LOCATION} kpt cfg set asm anthos.servicemesh.hub gcr.io/gke-release/asm kpt cfg set asm anthos.servicemesh.rev asm-1118-4 kpt cfg set asm anthos.servicemesh.tag 1.11.8-asm.4 kpt cfg set asm gcloud.project.environProjectNumber ${ENVIRON_PROJECT_NUMBER} kpt cfg set asm anthos.servicemesh.trustDomain ${ENVIRON_PROJECT_ID}.svc.id.goog kpt cfg set asm anthos.servicemesh.idp-url "${HUB_IDP_URL}"
If needed, change to the
istio-1.11.8-asm.4
directory. Theistioctl
client is version dependent. Make sure that you use the version in theistio-1.11.8-asm.4/bin
directory.Run the following command to install Anthos Service Mesh.
bin/istioctl install \ -f asm/istio/istio-operator.yaml \ -f asm/istio/options/hub-meshca.yaml --revision=asm-1118-4
If you want to enable a supported optional feature, include
-f
and the YAML filename on the following command line, after-f cluster.yaml
. See Enabling optional features for more information.
Installing Anthos Service Mesh with Istio CA
This section explains how to generate certificates and keys for the Istio CA that Anthos Service Mesh on-prem uses to sign your workloads. If you've already Installed Anthos Service Mesh with Mesh CA, you can skip to Setting the default network.
For the best security, we highly recommend maintaining an offline root CA and using the subordinate CAs to issue certificates for each cluster. For more information, see Plug in CA Certificates. In this configuration, all workloads in the service mesh use the same root certificate authority (CA). Each Anthos Service Mesh CA uses an intermediate CA signing key and certificate, signed by the root CA. When multiple CAs exist within a mesh, this establishes a hierarchy of trust among the CAs. You can repeat these steps to provision certificates and keys for any number of certificate authorities.
Create a directory for the certificates and keys:
mkdir -p certs && \ pushd certs
Generate a root certificate and key:
make -f ../tools/certs/Makefile.selfsigned.mk root-ca
This generates these files:
- root-cert.pem: the root certificate
- root-key.pem: the root key
- root-ca.conf: the configuration for openssl to generate the root certificate
- root-cert.csr: the CSR for the root certificate
Generate an intermediate certificate and key:
make -f ../tools/certs/Makefile.selfsigned.mk cluster1-cacerts
This generates these files in a directory named
cluster1
:- ca-cert.pem: the intermediate certificates
- ca-key.pem: the intermediate key
- cert-chain.pem: the certificate chain which istiod uses
- root-cert.pem: the root certificate
If you perform these steps using an offline computer, copy the generated directory to a computer with access to the clusters.
Create a secret
cacerts
including all the input filesca-cert.pem
,ca- key.pem
,root-cert.pem
andcert-chain.pem
:kubectl --context="${CTX_CLUSTER1}" create namespace istio-system kubectl --context="${CTX_CLUSTER1}" create secret generic cacerts -n istio-system \ --from-file=cluster1/ca-cert.pem \ --from-file=cluster1/ca-key.pem \ --from-file=cluster1/root-cert.pem \ --from-file=cluster1/cert-chain.pem
Anthos Service Mesh on-prem will detect the presence of these certificates/keys and use them during the install process in later steps.
Return to the previous directory:
popd
Configuring and installing Anthos Service Mesh with Istio CA
This section explains how to install Anthos Service Mesh with the Istio CA (previously known as Citadel) certificates you previously generated. If you've already Installed Anthos Service Mesh with Mesh CA, you can skip to Setting the default network.
Create an environment variable for the project ID:
export PROJECT_ID=YOUR_PROJECT_ID
Create an environment variable for the project number:
export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)")
Create an environment variable for the mesh identifier. This can be any string, but should be in a format that is consistent among your clusters.
export MESH_ID="proj-${PROJECT_NUMBER}"
Create the configuration for the cluster's control plane, which will install Anthos Service Mesh using the
asm-multicloud
profile. If you want to enable a supported optional feature, include-f
and the YAML filename on the following command line. See Enabling optional features for more information.In the following example:
Use the
MESH_ID
defined in the previous steps.The
NETWORK_ID
can be any string that identifies a cluster's network. In this on-premises configuration, every cluster is on its own network, so each cluster should have a different value.NETWORK_ID
has the same string limitations as a Kubernetes label described in Syntax and character set.
cat <<EOF > cluster.yaml apiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: profile: asm-multicloud revision: asm-1118-4 values: global: meshID: ${MESH_ID} multiCluster: clusterName: CLUSTER_NAME network: NETWORK_ID EOF
If needed, change to the
istio-1.11.8-asm.4
directory. Theistioctl
client is version dependent. Make sure that you use the version in theistio-1.11.8-asm.4/bin
directory.Run the following command to install Anthos Service Mesh. If you want to enable a supported optional feature, include
-f
and the YAML filename on the following command line, after-f cluster.yaml
. See Enabling optional features for more information.bin/istioctl install \ --context="${CTX_CLUSTER1}" \ -f cluster.yaml
Setting the default network
Set the default network on the istio-system
namespace.
kubectl --context="${CTX_CLUSTER1}" label \ namespace istio-system topology.istio.io/network=NETWORK_ID
Configuring the validating webhook
When you install Anthos Service Mesh, you set a revision label on istiod
. You
need to set the same revision on the validating webhook.
Copy the following YAML to a file called
istiod-service.yaml
:cat <<EOF > istiod-service.yaml apiVersion: v1 kind: Service metadata: name: istiod namespace: istio-system labels: istio.io/rev: asm-1118-4 app: istiod istio: pilot release: istio spec: ports: - port: 15010 name: grpc-xds # plaintext protocol: TCP - port: 15012 name: https-dns # mTLS with k8s-signed cert protocol: TCP - port: 443 name: https-webhook # validation and injection targetPort: 15017 protocol: TCP - port: 15014 name: http-monitoring # prometheus stats protocol: TCP selector: app: istiod istio.io/rev: asm-1118-4 EOF
Configure the validating webhook so that it can locate the
istiod
service with the revision label:kubectl --context="${CTX_CLUSTER1}" apply -f istiod-service.yaml
This command creates a service entry that lets the validating webhook automatically check configurations before they are applied.
Auto mutual TLS (auto mTLS) is enabled by default. With auto mTLS, a client sidecar proxy automatically detects if the server has a sidecar. The client sidecar sends mTLS to workloads with sidecars and sends plain text traffic to workloads without sidecars.
Checking the control plane components
Check that the control plane pods in istio-system
are running:
kubectl --context="${CTX_CLUSTER1}" get pod -n istio-system
Expected output is similar to the following:
NAME READY STATUS RESTARTS AGE istio-ingressgateway-74cc894bfd-786rg 1/1 Running 0 7m19s istiod-78cdbbbdb-d7tps 1/1 Running 0 7m36s promsd-576b8db4d6-lqf64 2/2 Running 1 7m19s
Injecting sidecar proxies
Anthos Service Mesh uses sidecar proxies to enhance network security, reliability, and observability. With Anthos Service Mesh, these functions are abstracted away from the application's primary container and implemented in a common out-of-process proxy delivered as a separate container in the same Pod.
Your installation isn't complete until you enable automatic sidecar proxy injection (auto-injection) and restart the Pods for any workloads that were running on your cluster before you installed Anthos Service Mesh.
To enable auto-injection, you label your namespaces with the revision label
that was set on istiod
when you installed Anthos Service Mesh. The revision label is
used by the sidecar injector webhook to associate injected sidecars with a
particular istiod
revision. After adding the label, any existing Pods in the
namespace must be restarted for sidecars to be injected.
Before you deploy new workloads in a new namespace, make sure to configure auto-injection so that Anthos Service Mesh can monitor and secure traffic.
To enable auto-injection:
Use the following command to locate the revision label on
istiod
:kubectl --context=${CTX_CLUSTER1} \ -n istio-system get pods -l app=istiod --show-labels
The output looks similar to the following:
NAME READY STATUS RESTARTS AGE LABELS istiod-asm-1118-4-5788d57586-bljj4 1/1 Running 0 23h app=istiod,istio.io/rev=asm-1118-4,istio=istiod,pod-template-hash=5788d57586 istiod-asm-1118-4-5788d57586-vsklm 1/1 Running 1 23h app=istiod,istio.io/rev=asm-1118-4,istio=istiod,pod-template-hash=5788d57586
In the output, under the
LABELS
column, note the value of theistiod
revision label, which follows the prefixistio.io/rev=
. In this example, the value isasm-1118-4
.Apply the revision label and remove the
istio-injection
label if it exists. In the following command,NAMESPACE
is the name of the namespace where you want to enable auto-injection, andREVISION
is the revision label you noted in the previous step.kubectl --context=${CTX_CLUSTER1} \ label namespace NAMESPACE istio-injection- istio.io/rev=REVISION --overwrite
You can ignore the message
"istio-injection not found"
in the output. That means that the namespace didn't previously have theistio-injection
label, which you should expect in new installations of Anthos Service Mesh or new deployments. Because auto-injection fails if a namespace has both theistio-injection
and the revision label, allkubectl label
commands in the Anthos Service Mesh documentation include removing theistio-injection
label.If workloads were running on your cluster before you installed Anthos Service Mesh, restart the Pods to trigger re-injection.
How you restart Pods depends on your application and the environment the cluster is in. For example, in your staging environment, you might simply delete all the Pods, which causes them to restart. But in your production environment, you might have a process that implements a blue-green deployment so that you can safely restart Pods to avoid traffic interruption.
You can use
kubectl
to perform a rolling restart:kubectl --context=${CTX_CLUSTER1} \ rollout restart deployment -n NAMESPACE
Verify that your Pods are configured to point to the new version of
istiod
.kubectl --context=${CTX_CLUSTER1} \ get pods -n NAMESPACE -l istio.io/rev=REVISION
What's next?
If you plan to use multiple clusters for your on-prem service mesh, see Installing Anthos Service Mesh on-premises on multiple clusters and networks.
Otherwise, your next step is Configuring external IP addresses.