Version 1.9

Installing Anthos Service Mesh on premises with an HSM and Vault

This guide explains how to install Anthos Service Mesh on existing Anthos clusters on VMware and use Thales Luna HSM 7+ and Hashicorp Vault to securely generate Istiod's CA signing key and protect the CA signing key.

If you have a previous version of Anthos Service Mesh installed, refer to Upgrading Anthos Service Mesh Anthos clusters on VMware. This installation enables the supported features on your cluster.

Before you begin

Review the following requirements before you begin setup.

Requirements

  • You must have an Anthos subscription. Alternatively, a pay-as-you-go billing option is available for Anthos on Google Cloud only. For more information, see the Anthos Pricing guide.

  • You must contact the support team to grant you the permissions to access the images. The support team will also provide you additional assistance.

  • Verify that the cluster where you install Anthos Service Mesh has at least 4 vCPUs, 15 GB memory, and 4 nodes.

  • You must name your service ports using the following syntax: name: protocol[-suffix] where the square brackets indicate an optional suffix that must start with a dash. For more information, see Naming service ports.

  • Verify that your cluster version is listed in Supported environments. To check your cluster version, you can use the gkectl command line tool. If you don't have gkectl installed, see GKE on-prem downloads.

    gkectl version
  • You should have the Thales Luna HSM 7+ in your infrastructure. The HSM should have network service set up and reachable from the Anthos Service Mesh cluster.

  • You should have Hashicorp Vault servers in your infrastructure for storing the credentials to access the HSM. The Vault servers should be reachable from the Anthos Service Mesh cluster through the network.

Set up your environment

Install the following tools on the machine from where you control the Anthos Service Mesh installation process.

  • The curl command-line tool.
  • The Cloud SDK (the gcloud command-line tool).

After you install the pre-requisite tools, perform the following steps:

  1. Authenticate with the Cloud SDK:

    gcloud auth login
    
  2. Update the components:

    gcloud components update
    
  3. Install kubectl:

    gcloud components install kubectl
    
  4. To deploy and test your installation with the Online Boutique sample application, install kpt:

    gcloud components install kpt
    
  5. Switch context to your user cluster (if necessary):

    kubectl config use-context CLUSTER_NAME
  6. Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:

    kubectl create clusterrolebinding cluster-admin-binding \
      --clusterrole=cluster-admin \
      --user=USER_ACCOUNT

Deploy dependencies

  • Deploy Thales Luna HSM 7+ in your infrastructure for securely managing the Istiod CA signing keys. Ensure the HSM has network service and connectivity with the Anthos Service Mesh cluster.
  • Deploy Hashicorp Vault servers in your infrastructure for storing the credentials to access the HSM. Ensure the Vault servers have network connectivity with the Anthos Service Mesh cluster.
  • Have a workstation with access to the Vault server, HSM and the Kubernetes cluster where you install Anthos Service Mesh.

Set up the hardware security module (HSM)

This section lists the steps to set up the hardware security module (HSM) and generate encryption keys.

Provision the HSM credentials and slot

Follow the HSM install guide to perform the following steps.

  1. Set the HSM up as a network HSM.

  2. Provision PEM files for mTLS connections with the HSM, including client key file, client certificate file and server certificate file.

  3. Provision an HSM slot for Anthos Service Mesh and record the slot label and PIN.

Generate an HSM Key Encryption Key (KEK)

  1. On the machine with access to the HSM, download the HSM tool (available only for Linux amd-64).

    gsutil cp gs://artifacts.thalescpl-io-k8s-kms-plugin.appspot.com/binary/k8s-kms-plugin .
  2. Set environment variables before you run the HSM tool as a local server, which connects to the HSM and serves on the local Unix Domain Socket (UDS).

    export P11_TOKEN=HSM slot label
    export ChrystokiConfigurationPath=path to the Thales HSM Chrystoki configuration file
    export P11_LIB=path to the libCryptoki2.so file for accessing the Thales HSM
    export P11_PIN_FILE=path to the file that stores the HSM slot PIN
    export SOCKET=/tmp/.hsm-sock
    ./k8s-kms-plugin serve &

    Expect output similar to the following:

    The token has been initialized and is reassigned to slot 412789065
    INFO Loaded P11 PIN from file: /home/hsm/hsm-credential/pin  line="cmd/serve.go:93"
    INFO KMS Plugin Listening on : /var/.hsm-sock  line="cmd/serve.go:119"
    INFO Serving on socket: /tmp/.hsm-sock  line="cmd/serve.go:198"
  3. If you don't have a Key Encryption Key (KEK) on the HSM, follow these steps to generate one. Use k8s-kms-plugin as a client to communicate with the HSM tool server. If you want to explicitly specify a KEK ID, you can pass the --kek-id flag in the command. Otherwise, a random KEK ID will be automatically generated.

    ./k8s-kms-plugin generate-kek --kek-id KEK ID

    Record the KEK ID. You will use it in the following steps.

  4. HSM uses the imported root certificate to verify the signed Istiod CA certificate. You can follow this step to import the root certificate into HSM. Use k8s-kms-plugin as a client to talk to the HSM tool server. Import the root certificate using the following command.

    ./k8s-kms-plugin import-ca -f root cert PEM file

Set up Kubernetes for Vault authentication

  1. In the Kubernetes cluster, create a dedicated service account and RBAC policy for Vault to call the TokenReview API. Create a dedicated namespace for this purpose.

  2. Create the default service account in the vault namespace, set the permissions, and extract the associated JSON Web Token (JWT) for the Vault server to call Kubernetes TokenReview API. Note: You can use any Kubernetes service account and namespaces if they have sufficient permission.

  3. Configure your cluster suffix for the cluster you want to set up. Use a dedicated directory for the configuration.

    export CLUSTER_SUFFIX="c1"
    mkdir ${CLUSTER_SUFFIX}
    cd ${CLUSTER_SUFFIX}
    
  4. Switch to the cluster you want to manage and create the vault namespace. kubectl creates the default service account automatically.

    kubectl create ns vault
    
  5. To grant permissions, bind the service account default.vault to the auth-delegator role.

    kubectl apply -f -<< EOF
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRoleBinding
    metadata:
     name: role-tokenreview-binding
     namespace: vault
    roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: ClusterRole
     name: system:auth-delegator
    subjects:
     - kind: ServiceAccount
       name: default
       namespace: vault
    EOF
    
  6. Get the JWT for service account default.vault.

    VAULT_SA_SECRET=$(kubectl get sa -n vault  default -o=jsonpath="{.secrets[0].name}")
    VAULT_SA_JWT_TOKEN=$(kubectl get -n vault secret $VAULT_SA_SECRET -o jsonpath="{.data.token}" | base64 --decode)
    echo $VAULT_SA_JWT_TOKEN > ${CLUSTER_SUFFIX}-tokenreview_jwt
    cat ${CLUSTER_SUFFIX}-tokenreview_jwt
    
  7. Retrieve the address and the CA certificate of the Kubernetes API server. Later, configure the Vault server to call the Kubernetes API server.

  8. Get the Kubernetes public IP:Port values.

    K8S_ADDR=$(kubectl config  view -o jsonpath="{.clusters[?(@.name == '$(kubectl config current-context)')].cluster.server}")
    echo $K8S_ADDR > ${CLUSTER_SUFFIX}-k8s_addr
    cat ${CLUSTER_SUFFIX}-k8s_addr
    
  9. Get the certificate for authenticating the Kubernetes API server.

    VAULT_SA_CA_CRT=$(kubectl get -n vault secret $VAULT_SA_SECRET -o jsonpath="{.data['ca\.crt']}" | base64 --decode)
    echo $VAULT_SA_CA_CRT > "${CLUSTER_SUFFIX}-k8s_cert"
    cat "${CLUSTER_SUFFIX}-k8s_cert"
    
  10. Convert the certificate file into the PEM file format and validate it.

    sed -i 's/ CERTIFICATE/CERTIFICATE/g' ${CLUSTER_SUFFIX}-k8s_cert
    sed -i 's/ /\n/g' ${CLUSTER_SUFFIX}-k8s_cert
    sed -i 's/CERTIFICATE/ CERTIFICATE/g' ${CLUSTER_SUFFIX}-k8s_cert
    
  11. Verify that the certificate is valid.

    openssl x509 -in ${CLUSTER_SUFFIX}-k8s_cert -noout -text
    
  12. Go back to the previous directory.

    cd ..
    

Now use the three files in the cluster's directory to configure Vault in the next step.

Set up Vault to store HSM credentials

When the Vault server is serving TLS traffic with a valid certificate, configure the Vault PKI and configure Kubernetes authentication. The following is an example to setup a single root CA and per-cluster intermediate CAs.

  1. Login Vault as root.

    vault login
    
  2. Store the secret for HSM.

    SECRET_PATH=asm-$CLUSTER_SUFFIX
    vault secrets enable -path=$SECRET_PATH  kv
    vault secrets tune -max-lease-ttl=87600h $SECRET_PATH
    export HSM_SECRET_PATH="$SECRET_PATH/hsm"
    vault kv put ${HSM_SECRET_PATH} clientcert=HSM_CLIENT_CERT clientkey=HSM_CLIENT_KEY servercert=HSM_SERVER_CERT PIN=HSM_PIN
    vault kv get ${HSM_SECRET_PATH}
    
  3. Create a policy with permissions to the PKI path.

    vault policy write asm-$CLUSTER_SUFFIX-hsm-policy -<< EOF
    path "${HSM_SECRET_PATH}" {
    capabilities = ["read"]
    }
    EOF
    
  4. Add Kubernetes authentication for the cluster.

    vault auth enable --path="${HSM_SECRET_PATH}"  kubernetes
    
  5. Set the Kubernetes JWT, the Kubernetes address, and the Kubernetes CA cert into the Vault authentication path.

    vault write auth/${HSM_SECRET_PATH}/config \
    token_reviewer_jwt=cat "${CLUSTER_SUFFIX}"-tokenreview_jwt \
    kubernetes_host=cat "${CLUSTER_SUFFIX}"-k8s_addr \
    kubernetes_ca_cert=@"${CLUSTER_SUFFIX}"-k8s_cert
    
  6. Allow istiod-service-account.istio-system to authenticate as role demo, and use the created policy.

    vault write auth/${HSM_SECRET_PATH}/role/istiod \
    bound_service_account_names=istiod-service-account \
    bound_service_account_namespaces=istio-system \
    policies=asm-$CLUSTER_SUFFIX-hsm-policy \
    ttl=768h
    
  7. Go back to the parent folder.

    cd ..
    

Download the installation file

Linux

  1. Download the Anthos Service Mesh installation file to your current working directory:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.9.5-asm.2-linux-amd64.tar.gz
  2. Download the signature file and use openssl to verify the signature:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.9.5-asm.2-linux-amd64.tar.gz.1.sig
    openssl dgst -verify /dev/stdin -signature istio-1.9.5-asm.2-linux-amd64.tar.gz.1.sig istio-1.9.5-asm.2-linux-amd64.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK.

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

     tar xzf istio-1.9.5-asm.2-linux-amd64.tar.gz

    The command creates an installation directory in your current working directory named istio-1.9.5-asm.2 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  4. Ensure that you're in the Anthos Service Mesh installation's root directory.

    cd istio-1.9.5-asm.2

Mac OS

  1. Download the Anthos Service Mesh installation file to your current working directory:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.9.5-asm.2-osx.tar.gz
  2. Download the signature file and use openssl to verify the signature:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.9.5-asm.2-osx.tar.gz.1.sig
    openssl dgst -sha256 -verify /dev/stdin -signature istio-1.9.5-asm.2-osx.tar.gz.1.sig istio-1.9.5-asm.2-osx.tar.gz <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK.

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

    tar xzf istio-1.9.5-asm.2-osx.tar.gz

    The command creates an installation directory in your current working directory named istio-1.9.5-asm.2 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  4. Ensure that you're in the Anthos Service Mesh installation's root directory.

    cd istio-1.9.5-asm.2

Windows

  1. Download the Anthos Service Mesh installation file to your current working directory:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.9.5-asm.2-win.zip
  2. Download the signature file and use openssl to verify the signature:

    curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.9.5-asm.2-win.zip.1.sig
    openssl dgst -verify - -signature istio-1.9.5-asm.2-win.zip.1.sig istio-1.9.5-asm.2-win.zip <<'EOF'
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
    wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
    -----END PUBLIC KEY-----
    EOF

    The expected output is: Verified OK.

  3. Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

    tar xzf istio-1.9.5-asm.2-win.zip

    The command creates an installation directory in your current working directory named istio-1.9.5-asm.2 that contains:

    • Sample applications in the samples directory.
    • The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
    • The Anthos Service Mesh configuration profiles are in the manifests/profiles directory.
  4. Ensure that you're in the Anthos Service Mesh installation's root directory.

    cd istio-1.9.5-asm.2

Set up the resource in control plane

  1. Set the environment variables.

    export CLUSTER_SUFFIX="c1"
    export VAULT_ADDR="https://VAULT_IP:PORT"
    
  2. Copy the file containing the certificate for authenticating Vault to a file named cert. For example, the Vault server's TLS certificate, ${VAULT_CACERT}.

    cp ${VAULT_CACERT} ./cert
    
  3. Create a ConfigMap from the file in the namespace istio-system.

    kubectl create ns istio-system
    kubectl delete configmap vault-tls-cert -n istio-system
    kubectl create configmap vault-tls-cert -n istio-system --from-file=./cert
    

Configure the validating webhook

When you install Anthos Service Mesh, you set a revision label on istiod. You need to set the same revision on the validating webhook.

Copy the following YAML to a file called istiod-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: istiod
  namespace: istio-system
  labels:
    istio.io/rev: asm-195-2
    app: istiod
    istio: pilot
    release: istio
spec:
  ports:
    - port: 15010
      name: grpc-xds # plaintext
      protocol: TCP
    - port: 15012
      name: https-dns # mTLS with k8s-signed cert
      protocol: TCP
    - port: 443
      name: https-webhook # validation and injection
      targetPort: 15017
      protocol: TCP
    - port: 15014
      name: http-monitoring # prometheus stats
      protocol: TCP
  selector:
    app: istiod
    istio.io/rev: asm-195-2

Install Anthos Service Mesh

  1. Set up the environment variables to configure ASM to interact with Vault and HSM. The following command shortens the waiting time to 10 seconds, because Istiod will not be fully ready due to the pending import operation by the operator.

    export HSM_SLOT_LABEL=the HSM slot label
    export VAULT_ADDR=address of the vault server
    export HSM_SECRET_PATH=the path to the HSM secret on Vault
    export KEK_ID=the HSM slot KEK ID
    export HSM_PLUGIN_IMAGE=gcr.io/thalescpl-io-k8s-kms-plugin/hsm-plugin:asm-1.7-1
    export VAULT_CLIENT_IMAGE=gcr.io/gke-release/asm/vaultclient:latest
    export WAIT_FOR_RESOURCES_TIMEOUT=10s
  2. Run the following command to install Anthos Service Mesh using the asm-multicloud profile. If you want to enable a supported optional feature, include -f and the YAML filename on the following command line. See Enabling optional features for more information.

    istioctl install --set profile=asm-multicloud \
       --set revision=asm-195-2 \
       --set values.hsm.enabled=true \
       --set values.hsm.hsmKEKID=${KEK_ID} \
       --set values.hsm.hsmPluginImage=${HSM_PLUGIN_IMAGE} \
       --set values.hsm.hsmSlotLabel=${HSM_SLOT_LABEL} \
       --set values.hsm.vaultClientImage=${VAULT_CLIENT_IMAGE} \
       --set values.hsm.vaultAddr=${VAULT_ADDR} \
       --set values.hsm.vaultAuthPath=auth/${HSM_SECRET_PATH}/login \
       --set values.hsm.vaultAuthRole=istiod \
       --set values.hsm.vaultAuthJwtPath="/var/run/secrets/kubernetes.io/serviceaccount/token" \
       --set values.hsm.vaultSecretPath=${HSM_SECRET_PATH} \
       --set values.global.jwtPolicy="first-party-jwt"
    
  3. Configure the validating webhook so that it can locate the istiod service with the revision label:

    kubectl apply -f istiod-service.yaml
    

    This command creates a service entry that lets the validating webhook automatically check configurations before they are applied.

  4. Verify the Istiod is initialized correctly and in waiting status.

    kubectl get pod -l app=istiod -n istio-system

    Expect output like the following:

    NAME                      READY   STATUS    RESTARTS   AGE
    istiod-66ff56d76c-f9p5l   2/3     Running   2          1m27s

    See that Istiod is not fully ready because the discovery container is blocked and waiting for the certificate.

  5. Check the log of Istiod containers to make sure they are in the right state.

    kubectl logs -c hsm-plugin -l app=istiod -n istio-system

    Expect output like the following:

    The token has been initialized and is reassigned to slot 412789065
    INFO Loaded P11 PIN from file: /var/run/hsm-credential/pin  line="cmd/serve.go:93"
    INFO KMS Plugin Listening on : /var/run/hsm-socket/.sock  line="cmd/serve.go:119"
    INFO Serving on socket: /var/run/hsm-socket/.sock  line="cmd/serve.go:198"
    KEK ID: 7651a4ea-eeb7-4c1f-927b-8c871c2127aa
    kubectl logs -c discovery -l app=istiod -n istio-system

    Expect the last line of output like the following:

    ...
    2020-10-15T21:56:56.918931Z info    pkica   Wait until the CA certificate secret istio-system.istio-ca-cert can be loaded...

Sign certificate for Istiod

  1. Download the encrypted CSR from the Kubernetes secret.

    kubectl get secret istio-ca-cert-csr -n istio-system -o jsonpath={.data} > encrypted_csr.json
    
  2. Decrypt the CSR.

    kubectl get secret istio-ca-cert-csr -n istio-system -o jsonpath={.data} > encrypted_csr.json
    ./k8s-kms-plugin decrypt-csr -f encrypted_csr.json -o csr.pem
    
  3. The security administrator should take the csr.pem file and sign it using the root CA.

  4. Put your certificate chain into the root of the file named cert-chain.pem, and then run the following command.

    kubectl create secret generic istio-ca-cert --from-file=cert-chain.pem -n istio-system
    
  5. Verify Istiod loads the new cert chain successfully by inspecting at the Istiod log.

    kubectl logs ISTIOD_POD -c discovery -n istio-system | grep "CA cert\:" -A 60
    

    Expect the output to be similar to the following:

    2020-10-24T18:58:14.354254Z info    pkica   CA cert:
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

intermediate certs: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

root cert: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

Verify the installation

Use the following steps to verify that your Anthos Service Mesh installation is working correctly.

Check the control plane components

  1. Check that the control plane pods in istio-system are running:

    kubectl get pod -n istio-system

    Expected output is similar to the following:

    NAME                                      READY   STATUS      RESTARTS   AGE
    istio-ingressgateway-74cc894bfd-786rg     1/1     Running     0          7m19s
    istiod-78cdbbbdb-d7tps                    3/3     Running     0          7m36s
    promsd-576b8db4d6-lqf64                   2/2     Running     1          7m19s
  2. Trigger a TLS call to a service from the sidecar, and inspect the certificate used by the service, using bookinfo as an example.

    kubectl exec POD -c istio-proxy -- openssl s_client -alpn istio -showcerts -connect details:9080
    

    Inspect the certificates in the output, and expect to see the server-side certificate chain in the output.

Inject sidecar proxies

Anthos Service Mesh uses sidecar proxies to enhance network security, reliability, and observability. With Anthos Service Mesh, these functions are abstracted away from the application's primary container and implemented in a common out-of-process proxy delivered as a separate container in the same Pod.

Your installation isn't complete until you enable automatic sidecar proxy injection (auto-injection) and restart the Pods for any workloads that were running on your cluster before you installed Anthos Service Mesh.

To enable auto-injection, you label your namespaces with the revision label that was set on istiod when you installed Anthos Service Mesh. The revision label is used by the sidecar injector webhook to associate injected sidecars with a particular istiod revision. After adding the label, any existing Pods in the namespace must be restarted for sidecars to be injected.

Before you deploy new workloads in a new namespace, make sure to configure auto-injection so that Anthos Service Mesh can monitor and secure traffic.

To enable auto-injection:

  1. Use the following command to locate the revision label on istiod:

    kubectl -n istio-system get pods -l app=istiod --show-labels
    

    The output looks similar to the following:

    NAME                                READY   STATUS    RESTARTS   AGE   LABELS
    istiod-asm-195-2-5788d57586-bljj4   1/1     Running   0          23h   app=istiod,istio.io/rev=asm-195-2,istio=istiod,pod-template-hash=5788d57586
    istiod-asm-195-2-5788d57586-vsklm   1/1     Running   1          23h   app=istiod,istio.io/rev=asm-195-2,istio=istiod,pod-template-hash=5788d57586

    In the output, under the LABELS column, note the value of the istiod revision label, which follows the prefix istio.io/rev=. In this example, the value is asm-195-2.

  2. Apply the revision label and remove the istio-injection label if it exists. In the following command, NAMESPACE is the name of the namespace where you want to enable auto-injection, and REVISION is the revision label you noted in the previous step.

    kubectl label namespace NAMESPACE istio-injection- istio.io/rev=REVISION --overwrite
    

    You can ignore the message "istio-injection not found" in the output. That means that the namespace didn't previously have the istio-injection label, which you should expect in new installations of Anthos Service Mesh or new deployments. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.

  3. If workloads were running on your cluster before you installed Anthos Service Mesh, restart the Pods to trigger re-injection.

    How you restart Pods depends on your application and the environment the cluster is in. For example, in your staging environment, you might simply delete all the Pods, which causes them to restart. But in your production environment, you might have a process that implements a blue-green deployment so that you can safely restart Pods to avoid traffic interruption.

    You can use kubectl to perform a rolling restart:

    kubectl rollout restart deployment -n NAMESPACE
    
  4. Verify that your Pods are configured to point to the new version of istiod.

    kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION
    

Configure an external IP address

The default Anthos Service Mesh installation assumes that an external IP address is automatically allocated for LoadBalancer services. This is not true in Anthos clusters on VMware. Because of this, you need to allocate an IP address manually for the Anthos Service Mesh ingress Gateway resource.

To configure an external IP address, follow one of the sections below, depending on your cluster's load balancing mode:

Configure integrated load balancing mode

  1. Open the istio-ingressgateway Service's configuration:

    kubectl edit svc -n istio-system istio-ingressgateway
    

    The configuration for the istio-ingressgateway service opens in your shell's default text editor.

  2. In the file, add the following line under the specification (spec) block:

    loadBalancerIP: <your static external IP address>
    

    For example:

    spec:
     loadBalancerIP: 203.0.113.1
    
  3. Save the file.

Configure manual load balancing mode

To expose a service of type NodePort with a virtual IP address (VIP) on your load balancer, get the nodePort values first:

  1. View the istio-ingressgateway service's configuration in your shell:

    kubectl get svc -n istio-system istio-ingressgateway -o yaml
    

    Each of the ports for Anthos Service Mesh's gateways are displayed. The command output is similar to the following:

     ...
     ports:
     - name: status-port
       nodePort: 30391
       port: 15020
       protocol: TCP
       targetPort: 15020
     - name: http2
       nodePort: 31380
       port: 80
       protocol: TCP
       targetPort: 80
     - name: https
       nodePort: 31390
       port: 443
       protocol: TCP
       targetPort: 443
     - name: tcp
       nodePort: 31400
       port: 31400
       protocol: TCP
       targetPort: 31400
     - name: https-kiali
       nodePort: 31073
       port: 15029
       protocol: TCP
       targetPort: 15029
     - name: https-prometheus
       nodePort: 30253
       port: 15030
       protocol: TCP
       targetPort: 15030
     - name: https-grafana
       nodePort: 30050
       port: 15031
       protocol: TCP
       targetPort: 15031
     - name: https-tracing
       nodePort: 31204
       port: 15032
       protocol: TCP
       targetPort: 15032
     - name: tls
       nodePort: 30158
       port: 15443
       protocol: TCP
       targetPort: 15443
     ...
    
  2. Expose these ports through your load balancer.

    For example, the service port named http2 has port 80 and nodePort 31380. Suppose the node addresses for your user cluster are 192.168.0.10, 192.168.0.11, and 192.168.0.12, and your load balancer's VIP is 203.0.113.1.

    Configure your load balancer so that traffic sent to 203.0.113.1:80 is forwarded to 192.168.0.10:31380, 192.168.0.11:31380, or 192.168.0.12:31380. You can select the service ports that you want to expose on this given VIP.

What's next?