Anthos Service Mesh 1.8 has reached end of life and is no longer supported. See Upgrading from earlier versions.

View the latest documentation or select another available version:

Available versions

1.20 (latest)
1.19
1.18
1.17
1.16
1.15
1.14
1.13
1.12
1.11
1.10
1.9
1.8
1.7
1.6
1.5
1.4

Installing Anthos Service Mesh on premises with an HSM and Vault

This guide explains how to install Anthos Service Mesh on existing GKE on VMware and use Thales Luna HSM 7+ and Hashicorp Vault to securely generate Istiod's CA signing key and protect the CA signing key.

If you have a previous version of Anthos Service Mesh installed, refer to Upgrading Anthos Service Mesh GKE on VMware. This installation enables the supported features on your cluster.

Before you begin

Review the following requirements before you begin setup.

Requirements

You must have an Anthos subscription. Alternatively, a pay-as-you-go billing option is available for GKE Enterprise on Google Cloud only. For more information, see the GKE Enterprise Pricing guide.
You must contact the support team to grant you the permissions to access the images. The support team will also provide you additional assistance.
Verify that the cluster where you install Anthos Service Mesh has at least 4 vCPUs, 15 GB memory, and 4 nodes.
You must name your service ports using the following syntax: name: protocol[-suffix] where the square brackets indicate an optional suffix that must start with a dash. For more information, see Naming service ports.
Verify that your cluster version is listed in Supported environments. To check your cluster version, you can use the gkectl command line tool. If you don't have gkectl installed, see GKE on-prem downloads.
```
gkectl version
```
You should have the Thales Luna HSM 7+ in your infrastructure. The HSM should have network service set up and reachable from the Anthos Service Mesh cluster.
You should have Hashicorp Vault servers in your infrastructure for storing the credentials to access the HSM. The Vault servers should be reachable from the Anthos Service Mesh cluster through the network.

Set up your environment

Install the following tools on the machine from where you control the Anthos Service Mesh installation process.

The curl command-line tool.
The Google Cloud CLI.

After you install the pre-requisite tools, perform the following steps:

Authenticate with the Google Cloud CLI:
```
gcloud auth login
```
Update the components:
```
gcloud components update
```
Install kubectl:
```
gcloud components install kubectl
```
To deploy and test your installation with the Online Boutique sample application, install kpt:
```
gcloud components install kpt
```
Switch context to your user cluster (if necessary):
```
kubectl config use-context CLUSTER_NAME
```
Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:
```
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user=USER_ACCOUNT
```

Deploy dependencies

Deploy Thales Luna HSM 7+ in your infrastructure for securely managing the Istiod CA signing keys. Ensure the HSM has network service and connectivity with the Anthos Service Mesh cluster.
Deploy Hashicorp Vault servers in your infrastructure for storing the credentials to access the HSM. Ensure the Vault servers have network connectivity with the Anthos Service Mesh cluster.
Have a workstation with access to the Vault server, HSM and the Kubernetes cluster where you install Anthos Service Mesh.

Set up the hardware security module (HSM)

This section lists the steps to set up the hardware security module (HSM) and generate encryption keys.

Provision the HSM credentials and slot

Follow the HSM install guide to perform the following steps.

Set the HSM up as a network HSM.
Provision PEM files for mTLS connections with the HSM, including client key file, client certificate file and server certificate file.
Provision an HSM slot for Anthos Service Mesh and record the slot label and PIN.

Generate an HSM Key Encryption Key (KEK)

On the machine with access to the HSM, download the HSM tool (available only for Linux amd-64).
```
gsutil cp gs://artifacts.thalescpl-io-k8s-kms-plugin.appspot.com/binary/k8s-kms-plugin .
```

Set environment variables before you run the HSM tool as a local server, which connects to the HSM and serves on the local Unix Domain Socket (UDS).

export P11_TOKEN=HSM slot label
export ChrystokiConfigurationPath=path to the Thales HSM Chrystoki configuration file
export P11_LIB=path to the libCryptoki2.so file for accessing the Thales HSM
export P11_PIN_FILE=path to the file that stores the HSM slot PIN
export SOCKET=/tmp/.hsm-sock
./k8s-kms-plugin serve &

Expect output similar to the following:

The token has been initialized and is reassigned to slot 412789065
INFO Loaded P11 PIN from file: /home/hsm/hsm-credential/pin  line="cmd/serve.go:93"
INFO KMS Plugin Listening on : /var/.hsm-sock  line="cmd/serve.go:119"
INFO Serving on socket: /tmp/.hsm-sock  line="cmd/serve.go:198"

If you don't have a Key Encryption Key (KEK) on the HSM, follow these steps to generate one. Use k8s-kms-plugin as a client to communicate with the HSM tool server. If you want to explicitly specify a KEK ID, you can pass the --kek-id flag in the command. Otherwise, a random KEK ID will be automatically generated.
```
./k8s-kms-plugin generate-kek --kek-id KEK ID
```
Record the KEK ID. You will use it in the following steps.
HSM uses the imported root certificate to verify the signed Istiod CA certificate. You can follow this step to import the root certificate into HSM. Use k8s-kms-plugin as a client to talk to the HSM tool server. Import the root certificate using the following command.
```
./k8s-kms-plugin import-ca -f root cert PEM file
```

Set up Kubernetes for Vault authentication

In the Kubernetes cluster, create a dedicated service account and RBAC policy for Vault to call the TokenReview API. Create a dedicated namespace for this purpose.
Create the default service account in the vault namespace, set the permissions, and extract the associated JSON Web Token (JWT) for the Vault server to call Kubernetes TokenReview API. Note: You can use any Kubernetes service account and namespaces if they have sufficient permission.
Configure your cluster suffix for the cluster you want to set up. Use a dedicated directory for the configuration.
```
export CLUSTER_SUFFIX="c1"
mkdir ${CLUSTER_SUFFIX}
cd ${CLUSTER_SUFFIX}
```
Switch to the cluster you want to manage and create the vault namespace. kubectl creates the default service account automatically.
```
kubectl create ns vault
```

To grant permissions, bind the service account default.vault to the auth-delegator role.

kubectl apply -f -<< EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: role-tokenreview-binding
 namespace: vault
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:auth-delegator
subjects:
 - kind: ServiceAccount
   name: default
   namespace: vault
EOF

Create a secret to hold the token credential.

VAULT_SA_SECRET=default-token
kubectl apply -n vault -f - << EOF
apiVersion: v1
kind: Secret
metadata:
 name: "${VAULT_SA_SECRET}"
 annotations:
   kubernetes.io/service-account.name: default
type: kubernetes.io/service-account-token
EOF

Get the JWT for service account default.vault.

VAULT_SA_JWT_TOKEN=$(kubectl get -n vault secret $VAULT_SA_SECRET -o jsonpath="{.data.token}" | base64 --decode)
echo $VAULT_SA_JWT_TOKEN > ${CLUSTER_SUFFIX}-tokenreview_jwt
cat ${CLUSTER_SUFFIX}-tokenreview_jwt

Retrieve the address and the CA certificate of the Kubernetes API server. Later, configure the Vault server to call the Kubernetes API server.

Get the Kubernetes public IP:Port values.

K8S_ADDR=$(kubectl config  view -o jsonpath="{.clusters[?(@.name == '$(kubectl config current-context)')].cluster.server}")
echo $K8S_ADDR > ${CLUSTER_SUFFIX}-k8s_addr
cat ${CLUSTER_SUFFIX}-k8s_addr

Get the certificate for authenticating the Kubernetes API server.

VAULT_SA_CA_CRT=$(kubectl get -n vault secret $VAULT_SA_SECRET -o jsonpath="{.data['ca\.crt']}" | base64 --decode)
echo $VAULT_SA_CA_CRT > "${CLUSTER_SUFFIX}-k8s_cert"
cat "${CLUSTER_SUFFIX}-k8s_cert"

Convert the certificate file into the PEM file format and validate it.

sed -i 's/ CERTIFICATE/CERTIFICATE/g' ${CLUSTER_SUFFIX}-k8s_cert
sed -i 's/ /\n/g' ${CLUSTER_SUFFIX}-k8s_cert
sed -i 's/CERTIFICATE/ CERTIFICATE/g' ${CLUSTER_SUFFIX}-k8s_cert

Verify that the certificate is valid.

openssl x509 -in ${CLUSTER_SUFFIX}-k8s_cert -noout -text

Go back to the previous directory.
```
cd ..
```

Now use the three files in the cluster's directory to configure Vault in the next step.

Set up Vault to store HSM credentials

When the Vault server is serving TLS traffic with a valid certificate, configure the Vault PKI and configure Kubernetes authentication. The following is an example to setup a single root CA and per-cluster intermediate CAs.

Store the secret for HSM.

SECRET_PATH=asm-$CLUSTER_SUFFIX
vault secrets enable -path=$SECRET_PATH  kv
vault secrets tune -max-lease-ttl=87600h $SECRET_PATH
export HSM_SECRET_PATH="$SECRET_PATH/hsm"
vault kv put ${HSM_SECRET_PATH} clientcert=HSM_CLIENT_CERT clientkey=HSM_CLIENT_KEY servercert=HSM_SERVER_CERT PIN=HSM_PIN
vault kv get ${HSM_SECRET_PATH}

Create a policy with permissions to the PKI path.

vault policy write asm-$CLUSTER_SUFFIX-hsm-policy -<< EOF
path "${HSM_SECRET_PATH}" {
capabilities = ["read"]
}
EOF

Add Kubernetes authentication for the cluster.

vault auth enable --path="${HSM_SECRET_PATH}"  kubernetes

Set the Kubernetes JWT, the Kubernetes address, and the Kubernetes CA cert into the Vault authentication path.

vault write auth/${HSM_SECRET_PATH}/config \
token_reviewer_jwt=cat "${CLUSTER_SUFFIX}"-tokenreview_jwt \
kubernetes_host=cat "${CLUSTER_SUFFIX}"-k8s_addr \
kubernetes_ca_cert=@"${CLUSTER_SUFFIX}"-k8s_cert

Allow istiod-service-account.istio-system to authenticate as role demo, and use the created policy.

vault write auth/${HSM_SECRET_PATH}/role/istiod \
bound_service_account_names=istiod-service-account \
bound_service_account_namespaces=istio-system \
policies=asm-$CLUSTER_SUFFIX-hsm-policy \
ttl=768h

Go back to the parent folder.
```
cd ..
```

Download the installation file

Linux

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-linux-amd64.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-linux-amd64.tar.gz.1.sig
openssl dgst -verify /dev/stdin -signature istio-1.8.6-asm.8-linux-amd64.tar.gz.1.sig istio-1.8.6-asm.8-linux-amd64.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.8.6-asm.8-linux-amd64.tar.gz

The command creates an installation directory in your current working directory named istio-1.8.6-asm.8 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the install/kubernetes/operator/profiles directory.

Mac OS

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-osx.tar.gz

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-osx.tar.gz.1.sig
openssl dgst -sha256 -verify /dev/stdin -signature istio-1.8.6-asm.8-osx.tar.gz.1.sig istio-1.8.6-asm.8-osx.tar.gz <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.8.6-asm.8-osx.tar.gz

The command creates an installation directory in your current working directory named istio-1.8.6-asm.8 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the install/kubernetes/operator/profiles directory.

Windows

Download the Anthos Service Mesh installation file to your current working directory:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-win.zip

Download the signature file and use openssl to verify the signature:

curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.8.6-asm.8-win.zip.1.sig
openssl dgst -verify - -signature istio-1.8.6-asm.8-win.zip.1.sig istio-1.8.6-asm.8-win.zip <<'EOF'
-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ
wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw==
-----END PUBLIC KEY-----
EOF

The expected output is: Verified OK

Extract the contents of the file to any location on your file system. For example, to extract the contents to the current working directory:

tar xzf istio-1.8.6-asm.8-win.zip

The command creates an installation directory in your current working directory named istio-1.8.6-asm.8 that contains:

Sample applications in the samples directory.
The istioctl command-line tool that you use to install Anthos Service Mesh is in the bin directory.
The Anthos Service Mesh configuration profiles are in the install/kubernetes/operator/profiles directory.

Ensure that you're in the Anthos Service Mesh installation's root directory.
```
cd istio-1.8.6-asm.8
```
For convenience, add the tools in the /bin directory to your PATH:
```
export PATH=$PWD/bin:$PATH
```

Set up the resource in control plane

Set the environment variables.

export CLUSTER_SUFFIX="c1"
export VAULT_ADDR="https://VAULT_IP:PORT"

Copy the file containing the certificate for authenticating Vault to a file named cert. For example, the Vault server's TLS certificate, ${VAULT_CACERT}.

Note: The vault client uses this certificate to authenticate the Vault server. This certificate is different from the certificate used by Vault to authenticate Kubernetes API server.
```
cp ${VAULT_CACERT} ./cert
```

Create a ConfigMap from the file in the namespace istio-system.

kubectl create ns istio-system
kubectl delete configmap vault-tls-cert -n istio-system
kubectl create configmap vault-tls-cert -n istio-system --from-file=./cert

Configure the validating webhook

When you install Anthos Service Mesh, you set a revision label on istiod. You need to set the same revision on the validating webhook.

Copy the following YAML to a file called istiod-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: istiod
  namespace: istio-system
  labels:
    istio.io/rev: asm-186-8
    app: istiod
    istio: pilot
    release: istio
spec:
  ports:
    - port: 15010
      name: grpc-xds # plaintext
      protocol: TCP
    - port: 15012
      name: https-dns # mTLS with k8s-signed cert
      protocol: TCP
    - port: 443
      name: https-webhook # validation and injection
      targetPort: 15017
      protocol: TCP
    - port: 15014
      name: http-monitoring # prometheus stats
      protocol: TCP
  selector:
    app: istiod
    istio.io/rev: asm-186-8

Install Anthos Service Mesh

Set up the environment variables to configure ASM to interact with Vault and HSM. The following command shortens the waiting time to 10 seconds, because Istiod will not be fully ready due to the pending import operation by the operator.

export HSM_SLOT_LABEL=the HSM slot label
export VAULT_ADDR=address of the vault server
export HSM_SECRET_PATH=the path to the HSM secret on Vault
export KEK_ID=the HSM slot KEK ID
export HSM_PLUGIN_IMAGE=gcr.io/thalescpl-io-k8s-kms-plugin/hsm-plugin:asm-1.7-1
export VAULT_CLIENT_IMAGE=gcr.io/gke-release/asm/vaultclient:latest
export WAIT_FOR_RESOURCES_TIMEOUT=10s

Run the following command to install Anthos Service Mesh using the asm-multicloud profile. If you want to enable a supported optional feature, include -f and the YAML filename on the following command line. See Enabling optional features for more information.

istioctl install --set profile=asm-multicloud \
   --set revision=asm-186-8 \
   --set values.hsm.enabled=true \
   --set values.hsm.hsmKEKID=${KEK_ID} \
   --set values.hsm.hsmPluginImage=${HSM_PLUGIN_IMAGE} \
   --set values.hsm.hsmSlotLabel=${HSM_SLOT_LABEL} \
   --set values.hsm.vaultClientImage=${VAULT_CLIENT_IMAGE} \
   --set values.hsm.vaultAddr=${VAULT_ADDR} \
   --set values.hsm.vaultAuthPath=auth/${HSM_SECRET_PATH}/login \
   --set values.hsm.vaultAuthRole=istiod \
   --set values.hsm.vaultAuthJwtPath="/var/run/secrets/kubernetes.io/serviceaccount/token" \
   --set values.hsm.vaultSecretPath=${HSM_SECRET_PATH} \
   --set values.global.jwtPolicy="first-party-jwt"

Configure the validating webhook so that it can locate the istiod service with the revision label:
```
kubectl apply -f istiod-service.yaml
```
This command creates a service entry that lets the validating webhook automatically check configurations before they are applied.
Verify the Istiod is initialized correctly and in waiting status.
```
kubectl get pod -l app=istiod -n istio-system
```
Expect output like the following:
```
NAME                      READY   STATUS    RESTARTS   AGE
istiod-66ff56d76c-f9p5l   2/3     Running   2          1m27s
```
See that Istiod is not fully ready because the discovery container is blocked and waiting for the certificate.

Check the log of Istiod containers to make sure they are in the right state.

kubectl logs -c hsm-plugin -l app=istiod -n istio-system

Expect output like the following:

The token has been initialized and is reassigned to slot 412789065
INFO Loaded P11 PIN from file: /var/run/hsm-credential/pin  line="cmd/serve.go:93"
INFO KMS Plugin Listening on : /var/run/hsm-socket/.sock  line="cmd/serve.go:119"
INFO Serving on socket: /var/run/hsm-socket/.sock  line="cmd/serve.go:198"
KEK ID: 7651a4ea-eeb7-4c1f-927b-8c871c2127aa

kubectl logs -c discovery -l app=istiod -n istio-system

Expect the last line of output like the following:

...
2020-10-15T21:56:56.918931Z info    pkica   Wait until the CA certificate secret istio-system.istio-ca-cert can be loaded...

Sign certificate for Istiod

Download the encrypted CSR from the Kubernetes secret.

kubectl get secret istio-ca-cert-csr -n istio-system -o jsonpath={.data} > encrypted_csr.json

Decrypt the CSR.

kubectl get secret istio-ca-cert-csr -n istio-system -o jsonpath={.data} > encrypted_csr.json
./k8s-kms-plugin decrypt-csr -f encrypted_csr.json -o csr.pem

The security administrator should take the csr.pem file and sign it using the root CA.
Put your certificate chain into the root of the file named cert-chain.pem, and then run the following command.
```
kubectl create secret generic istio-ca-cert --from-file=cert-chain.pem -n istio-system
```

Verify Istiod loads the new cert chain successfully by inspecting at the Istiod log.

kubectl logs ISTIOD_POD -c discovery -n istio-system | grep "CA cert\:" -A 60

Expect the output to be similar to the following:

2020-10-24T18:58:14.354254Z info pkica CA cert: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

intermediate certs: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

root cert: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

Verify the installation

Use the following steps to verify that your Anthos Service Mesh installation is working correctly.

Check the control plane components

Check that the control plane pods in istio-system are running:

kubectl get pod -n istio-system

Expected output is similar to the following:

NAME                                      READY   STATUS      RESTARTS   AGE
istio-ingressgateway-74cc894bfd-786rg     1/1     Running     0          7m19s
istiod-78cdbbbdb-d7tps                    3/3     Running     0          7m36s
promsd-576b8db4d6-lqf64                   2/2     Running     1          7m19s

Trigger a TLS call to a service from the sidecar, and inspect the certificate used by the service, using bookinfo as an example.
```
kubectl exec POD -c istio-proxy -- openssl s_client -alpn istio -showcerts -connect details:9080
```
Inspect the certificates in the output, and expect to see the server-side certificate chain in the output.

Inject sidecar proxies

Anthos Service Mesh uses sidecar proxies to enhance network security, reliability, and observability. With Anthos Service Mesh, these functions are abstracted away from the application's primary container and implemented in a common out-of-process proxy delivered as a separate container in the same Pod.

Your installation isn't complete until you enable automatic sidecar proxy injection (auto-injection) and restart the Pods for any workloads that were running on your cluster before you installed Anthos Service Mesh.

To enable auto-injection, you label your namespaces with the revision label that was set on istiod when you installed Anthos Service Mesh. The revision label is used by the sidecar injector webhook to associate injected sidecars with a particular istiod revision. After adding the label, any existing Pods in the namespace must be restarted for sidecars to be injected.

Before you deploy new workloads in a new namespace, make sure to configure auto-injection so that Anthos Service Mesh can monitor and secure traffic.

To enable auto-injection:

Use the following command to locate the revision label on istiod:

kubectl -n istio-system get pods -l app=istiod --show-labels

The output looks similar to the following:

NAME                                READY   STATUS    RESTARTS   AGE   LABELS
istiod-asm-186-8-5788d57586-bljj4   1/1     Running   0          23h   app=istiod,istio.io/rev=asm-186-8,istio=istiod,pod-template-hash=5788d57586
istiod-asm-186-8-5788d57586-vsklm   1/1     Running   1          23h   app=istiod,istio.io/rev=asm-186-8,istio=istiod,pod-template-hash=5788d57586

In the output, under the LABELS column, note the value of the istiod revision label, which follows the prefix istio.io/rev=. In this example, the value is asm-186-8.

Apply the revision label and remove the istio-injection label if it exists. In the following command, NAMESPACE is the name of the namespace where you want to enable auto-injection, and REVISION is the revision label you noted in the previous step.
```
kubectl label namespace NAMESPACE istio-injection- istio.io/rev=REVISION --overwrite
```
You can ignore the message "istio-injection not found" in the output. That means that the namespace didn't previously have the istio-injection label, which you should expect in new installations of Anthos Service Mesh or new deployments. Because auto-injection fails if a namespace has both the istio-injection and the revision label, all kubectl label commands in the Anthos Service Mesh documentation include removing the istio-injection label.
If workloads were running on your cluster before you installed Anthos Service Mesh, restart the Pods to trigger re-injection.

How you restart Pods depends on your application and the environment the cluster is in. For example, in your staging environment, you might simply delete all the Pods, which causes them to restart. But in your production environment, you might have a process that implements a blue-green deployment so that you can safely restart Pods to avoid traffic interruption.

You can use kubectl to perform a rolling restart:
```
kubectl rollout restart deployment -n NAMESPACE
```
Verify that your Pods are configured to point to the new version of istiod.
```
kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION
```

Configure an external IP address

The default Anthos Service Mesh installation assumes that an external IP address is automatically allocated for LoadBalancer services. This is not true in GKE on VMware. Because of this, you need to allocate an IP address manually for the Anthos Service Mesh ingress Gateway resource.

To configure an external IP address, follow one of the sections below, depending on your cluster's load balancing mode:

Configure integrated load balancing mode

Open the istio-ingressgateway Service's configuration:
```
kubectl edit svc -n istio-system istio-ingressgateway
```
The configuration for the istio-ingressgateway service opens in your shell's default text editor.

In the file, add the following line under the specification (spec) block:

loadBalancerIP: <your static external IP address>

For example:

spec:
 loadBalancerIP: 203.0.113.1

Save the file.

Configure manual load balancing mode

To expose a service of type NodePort with a virtual IP address (VIP) on your load balancer, get the nodePort values first:

View the istio-ingressgateway service's configuration in your shell:

kubectl get svc -n istio-system istio-ingressgateway -o yaml

Each of the ports for Anthos Service Mesh's gateways are displayed. The command output is similar to the following:

 ...
 ports:
 - name: status-port
   nodePort: 30391
   port: 15020
   protocol: TCP
   targetPort: 15020
 - name: http2
   nodePort: 31380
   port: 80
   protocol: TCP
   targetPort: 80
 - name: https
   nodePort: 31390
   port: 443
   protocol: TCP
   targetPort: 443
 - name: tcp
   nodePort: 31400
   port: 31400
   protocol: TCP
   targetPort: 31400
 - name: https-kiali
   nodePort: 31073
   port: 15029
   protocol: TCP
   targetPort: 15029
 - name: https-prometheus
   nodePort: 30253
   port: 15030
   protocol: TCP
   targetPort: 15030
 - name: https-grafana
   nodePort: 30050
   port: 15031
   protocol: TCP
   targetPort: 15031
 - name: https-tracing
   nodePort: 31204
   port: 15032
   protocol: TCP
   targetPort: 15032
 - name: tls
   nodePort: 30158
   port: 15443
   protocol: TCP
   targetPort: 15443
 ...

Expose these ports through your load balancer.

For example, the service port named http2 has port 80 and nodePort 31380. Suppose the node addresses for your user cluster are 192.168.0.10, 192.168.0.11, and 192.168.0.12, and your load balancer's VIP is 203.0.113.1.

Configure your load balancer so that traffic sent to 203.0.113.1:80 is forwarded to 192.168.0.10:31380, 192.168.0.11:31380, or 192.168.0.12:31380. You can select the service ports that you want to expose on this given VIP.

What's next?

Deploy the Online Boutique sample sample application