This guide explains how to do a clean installation of Anthos Service Mesh version 1.7.8-asm.10 on GKE on VMware and use Thales Luna HSM 7+ and Hashicorp Vault to securely generate Istiod's CA signing key and protect the CA signing key.
If you have a previous version of Anthos Service Mesh installed, refer to Upgrading
Anthos Service Mesh GKE on VMware. This installation enables the supported features on your cluster for the asm-multicloud
configuration profile.
Before you begin
Review the following requirements before you begin setup.
Requirements
You must have an Anthos subscription. Alternatively, a pay-as-you-go billing option is available for GKE Enterprise on Google Cloud only. For more information, see the GKE Enterprise Pricing guide.
You must contact the support team to grant you the permissions to access the images. The support team will also provide you additional assistance.
Verify that the cluster where you install Anthos Service Mesh has at least 4 vCPUs, 15 GB memory, and 4 nodes.
You must name your service ports using the following syntax:
name: protocol[-suffix]
where the square brackets indicate an optional suffix that must start with a dash. For more information, see Naming service ports.Verify that your cluster version is listed in Supported environments. To check your cluster version, you can use the
gkectl
command line tool. If you don't havegkectl
installed, see GKE on-prem downloads.gkectl version
You should have the Thales Luna HSM 7+ in your infrastructure. The HSM should have network service set up and reachable from the Anthos Service Mesh cluster.
You should have Hashicorp Vault servers in your infrastructure for storing the credentials to access the HSM. The Vault servers should be reachable from the Anthos Service Mesh cluster through the network.
Set up your environment
Install the following tools on the machine from where you control the Anthos Service Mesh installation process.
- The
curl
command-line tool. - The Google Cloud CLI.
After you install the pre-requisite tools, perform the following steps:
Authenticate with the Google Cloud CLI:
gcloud auth login
Update the components:
gcloud components update
Install
kubectl
:gcloud components install kubectl
To deploy and test your installation with the Online Boutique sample application, install
kpt
:gcloud components install kpt
Switch context to your user cluster (if necessary):
kubectl config use-context CLUSTER_NAME
Grant cluster admin permissions to your user account (your Google Cloud login email address). You need these permissions to create the necessary role based access control (RBAC) rules for Anthos Service Mesh:
kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=USER_ACCOUNT
Deploy dependencies
- Deploy Thales Luna HSM 7+ in your infrastructure for securely managing the Istiod CA signing keys. Ensure the HSM has network service and connectivity with the Anthos Service Mesh cluster.
- Deploy Hashicorp Vault servers in your infrastructure for storing the credentials to access the HSM. Ensure the Vault servers have network connectivity with the Anthos Service Mesh cluster.
- Have a workstation with access to the Vault server, HSM and the Kubernetes cluster where you install Anthos Service Mesh.
Set up the hardware security module (HSM)
This section lists the steps to set up the hardware security module (HSM) and generate encryption keys.
Provision the HSM credentials and slot
Follow the HSM install guide to perform the following steps.
Set the HSM up as a network HSM.
Provision PEM files for mTLS connections with the HSM, including client key file, client certificate file and server certificate file.
Provision an HSM slot for Anthos Service Mesh and record the slot label and PIN.
Generate an HSM Key Encryption Key (KEK)
On the machine with access to the HSM, download the HSM tool (available only for Linux amd-64).
gsutil cp gs://artifacts.thalescpl-io-k8s-kms-plugin.appspot.com/binary/k8s-kms-plugin .
Set environment variables before you run the HSM tool as a local server, which connects to the HSM and serves on the local Unix Domain Socket (UDS).
export P11_TOKEN=HSM slot label export ChrystokiConfigurationPath=path to the Thales HSM Chrystoki configuration file export P11_LIB=path to the libCryptoki2.so file for accessing the Thales HSM export P11_PIN_FILE=path to the file that stores the HSM slot PIN export SOCKET=/tmp/.hsm-sock ./k8s-kms-plugin serve &
Expect output similar to the following:
The token has been initialized and is reassigned to slot 412789065 INFO Loaded P11 PIN from file: /home/hsm/hsm-credential/pin line="cmd/serve.go:93" INFO KMS Plugin Listening on : /var/.hsm-sock line="cmd/serve.go:119" INFO Serving on socket: /tmp/.hsm-sock line="cmd/serve.go:198"
If you don't have a Key Encryption Key (KEK) on the HSM, follow these steps to generate one. Use
k8s-kms-plugin
as a client to communicate with the HSM tool server. If you want to explicitly specify a KEK ID, you can pass the--kek-id
flag in the command. Otherwise, a random KEK ID will be automatically generated../k8s-kms-plugin generate-kek --kek-id KEK ID
Record the KEK ID. You will use it in the following steps.
HSM uses the imported root certificate to verify the signed Istiod CA certificate. You can follow this step to import the root certificate into HSM. Use
k8s-kms-plugin
as a client to talk to the HSM tool server. Import the root certificate using the following command../k8s-kms-plugin import-ca -f root cert PEM file
Set up Kubernetes for Vault authentication
In the Kubernetes cluster, create a dedicated service account and RBAC policy for Vault to call the
TokenReview
API. Create a dedicated namespace for this purpose.Create the default service account in the
vault
namespace, set the permissions, and extract the associated JSON Web Token (JWT) for the Vault server to call KubernetesTokenReview
API. Note: You can use any Kubernetes service account and namespaces if they have sufficient permission.Configure your cluster suffix for the cluster you want to set up. Use a dedicated directory for the configuration.
export CLUSTER_SUFFIX="c1" mkdir ${CLUSTER_SUFFIX} cd ${CLUSTER_SUFFIX}
Switch to the cluster you want to manage and create the
vault
namespace.kubectl
creates the default service account automatically.kubectl create ns vault
To grant permissions, bind the service account
default.vault
to theauth-delegator
role.kubectl apply -f -<< EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: role-tokenreview-binding namespace: vault roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: default namespace: vault EOF
Create a secret to hold the token credential.
VAULT_SA_SECRET=default-token kubectl apply -n vault -f - << EOF apiVersion: v1 kind: Secret metadata: name: "${VAULT_SA_SECRET}" annotations: kubernetes.io/service-account.name: default type: kubernetes.io/service-account-token EOF
Get the JWT for service account
default.vault
.VAULT_SA_JWT_TOKEN=$(kubectl get -n vault secret $VAULT_SA_SECRET -o jsonpath="{.data.token}" | base64 --decode) echo $VAULT_SA_JWT_TOKEN > ${CLUSTER_SUFFIX}-tokenreview_jwt cat ${CLUSTER_SUFFIX}-tokenreview_jwt
Retrieve the address and the CA certificate of the Kubernetes API server. Later, configure the Vault server to call the Kubernetes API server.
Get the Kubernetes public
IP:Port
values.K8S_ADDR=$(kubectl config view -o jsonpath="{.clusters[?(@.name == '$(kubectl config current-context)')].cluster.server}") echo $K8S_ADDR > ${CLUSTER_SUFFIX}-k8s_addr cat ${CLUSTER_SUFFIX}-k8s_addr
Get the certificate for authenticating the Kubernetes API server.
VAULT_SA_CA_CRT=$(kubectl get -n vault secret $VAULT_SA_SECRET -o jsonpath="{.data['ca\.crt']}" | base64 --decode) echo $VAULT_SA_CA_CRT > "${CLUSTER_SUFFIX}-k8s_cert" cat "${CLUSTER_SUFFIX}-k8s_cert"
Convert the certificate file into the PEM file format and validate it.
sed -i 's/ CERTIFICATE/CERTIFICATE/g' ${CLUSTER_SUFFIX}-k8s_cert sed -i 's/ /\n/g' ${CLUSTER_SUFFIX}-k8s_cert sed -i 's/CERTIFICATE/ CERTIFICATE/g' ${CLUSTER_SUFFIX}-k8s_cert
Verify that the certificate is valid.
openssl x509 -in ${CLUSTER_SUFFIX}-k8s_cert -noout -text
Go back to the previous directory.
cd ..
Now use the three files in the cluster's directory to configure Vault in the next step.
Set up Vault to store HSM credentials
When the Vault server is serving TLS traffic with a valid certificate, configure the Vault PKI and configure Kubernetes authentication. The following is an example to setup a single root CA and per-cluster intermediate CAs.
Login Vault as root.
vault login
Store the secret for HSM.
SECRET_PATH=asm-$CLUSTER_SUFFIX vault secrets enable -path=$SECRET_PATH kv vault secrets tune -max-lease-ttl=87600h $SECRET_PATH export HSM_SECRET_PATH="$SECRET_PATH/hsm" vault kv put ${HSM_SECRET_PATH} clientcert=HSM_CLIENT_CERT clientkey=HSM_CLIENT_KEY servercert=HSM_SERVER_CERT PIN=HSM_PIN vault kv get ${HSM_SECRET_PATH}
Create a policy with permissions to the PKI path.
vault policy write asm-$CLUSTER_SUFFIX-hsm-policy -<< EOF path "${HSM_SECRET_PATH}" { capabilities = ["read"] } EOF
Add Kubernetes authentication for the cluster.
vault auth enable --path="${HSM_SECRET_PATH}" kubernetes
Set the Kubernetes JWT, the Kubernetes address, and the Kubernetes CA cert into the Vault authentication path.
vault write auth/${HSM_SECRET_PATH}/config \ token_reviewer_jwt=cat "${CLUSTER_SUFFIX}"-tokenreview_jwt \ kubernetes_host=cat "${CLUSTER_SUFFIX}"-k8s_addr \ kubernetes_ca_cert=@"${CLUSTER_SUFFIX}"-k8s_cert
Allow
istiod-service-account.istio-system
to authenticate as roledemo
, and use the created policy.vault write auth/${HSM_SECRET_PATH}/role/istiod \ bound_service_account_names=istiod-service-account \ bound_service_account_namespaces=istio-system \ policies=asm-$CLUSTER_SUFFIX-hsm-policy \ ttl=768h
Go back to the parent folder.
cd ..
Download the installation file
-
Download the Anthos Service Mesh installation file to your current working
directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.8-asm.10-linux-amd64.tar.gz
-
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.8-asm.10-linux-amd64.tar.gz.1.sig openssl dgst -verify /dev/stdin -signature istio-1.7.8-asm.10-linux-amd64.tar.gz.1.sig istio-1.7.8-asm.10-linux-amd64.tar.gz <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
-
Extract the contents of the file to any location on your file system. For
example, to extract the contents to the current working directory:
tar xzf istio-1.7.8-asm.10-linux-amd64.tar.gz
The command creates an installation directory in your current working directory named
istio-1.7.8-asm.10
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
install/kubernetes/operator/profiles
directory.
- Sample applications in the
-
Download the Anthos Service Mesh installation file to your current working
directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.8-asm.10-osx.tar.gz
-
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.8-asm.10-osx.tar.gz.1.sig openssl dgst -sha256 -verify /dev/stdin -signature istio-1.7.8-asm.10-osx.tar.gz.1.sig istio-1.7.8-asm.10-osx.tar.gz <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
-
Extract the contents of the file to any location on your file system. For
example, to extract the contents to the current working directory:
tar xzf istio-1.7.8-asm.10-osx.tar.gz
The command creates an installation directory in your current working directory named
istio-1.7.8-asm.10
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
install/kubernetes/operator/profiles
directory.
- Sample applications in the
-
Download the Anthos Service Mesh installation file to your current working
directory:
curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.8-asm.10-win.zip
-
Download the signature file and use
openssl
to verify the signature:curl -LO https://storage.googleapis.com/gke-release/asm/istio-1.7.8-asm.10-win.zip.1.sig openssl dgst -verify - -signature istio-1.7.8-asm.10-win.zip.1.sig istio-1.7.8-asm.10-win.zip <<'EOF' -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEWZrGCUaJJr1H8a36sG4UUoXvlXvZ wQfk16sxprI2gOJ2vFFggdq3ixF2h4qNBt0kI7ciDhgpwS8t+/960IsIgw== -----END PUBLIC KEY----- EOF
The expected output is:
Verified OK
-
Extract the contents of the file to any location on your file system. For
example, to extract the contents to the current working directory:
tar xzf istio-1.7.8-asm.10-win.zip
The command creates an installation directory in your current working directory named
istio-1.7.8-asm.10
that contains:- Sample applications in the
samples
directory. - The
istioctl
command-line tool that you use to install Anthos Service Mesh is in thebin
directory. - The Anthos Service Mesh configuration profiles are in the
install/kubernetes/operator/profiles
directory.
- Sample applications in the
-
Ensure that you're in the Anthos Service Mesh installation's root directory.
cd istio-1.7.8-asm.10
-
For convenience, add the tools in the
/bin
directory to your PATH:export PATH=$PWD/bin:$PATH
Linux
Mac OS
Windows
Set up the resource in control plane
Set the environment variables.
export CLUSTER_SUFFIX="c1" export VAULT_ADDR="https://VAULT_IP:PORT"
Copy the file containing the certificate for authenticating Vault to a file named
cert
. For example, the Vault server's TLS certificate,${VAULT_CACERT}
.cp ${VAULT_CACERT} ./cert
Create a
ConfigMap
from the file in the namespaceistio-system
.kubectl create ns istio-system kubectl delete configmap vault-tls-cert -n istio-system kubectl create configmap vault-tls-cert -n istio-system --from-file=./cert
Configure the validating webhook
When you install Anthos Service Mesh, you set a revision label on istiod
. You
need to set the same revision on the validating webhook.
Copy the following YAML to a file called istiod-service.yaml
:
apiVersion: v1
kind: Service
metadata:
name: istiod
namespace: istio-system
labels:
istio.io/rev: asm-178-10
app: istiod
istio: pilot
release: istio
spec:
ports:
- port: 15010
name: grpc-xds # plaintext
protocol: TCP
- port: 15012
name: https-dns # mTLS with k8s-signed cert
protocol: TCP
- port: 443
name: https-webhook # validation and injection
targetPort: 15017
protocol: TCP
- port: 15014
name: http-monitoring # prometheus stats
protocol: TCP
selector:
app: istiod
istio.io/rev: asm-178-10
Install Anthos Service Mesh
Set up the environment variables to configure ASM to interact with Vault and HSM. The following command shortens the waiting time to 10 seconds, because Istiod will not be fully ready due to the pending import operation by the operator.
export HSM_SLOT_LABEL=the HSM slot label export VAULT_ADDR=address of the vault server export HSM_SECRET_PATH=the path to the HSM secret on Vault export KEK_ID=the HSM slot KEK ID export HSM_PLUGIN_IMAGE=gcr.io/thalescpl-io-k8s-kms-plugin/hsm-plugin:asm-1.7-1 export VAULT_CLIENT_IMAGE=gcr.io/gke-release/asm/vaultclient:latest export WAIT_FOR_RESOURCES_TIMEOUT=10s
Run the following command to install Anthos Service Mesh using the
asm-multicloud
profile. If you want to enable a supported optional feature, include-f
and the YAML filename on the following command line. See Enabling optional features for more information.istioctl install --set profile=asm-multicloud \ --set revision=asm-178-10 \ --set values.hsm.enabled=true \ --set values.hsm.hsmKEKID=${KEK_ID} \ --set values.hsm.hsmPluginImage=${HSM_PLUGIN_IMAGE} \ --set values.hsm.hsmSlotLabel=${HSM_SLOT_LABEL} \ --set values.hsm.vaultClientImage=${VAULT_CLIENT_IMAGE} \ --set values.hsm.vaultAddr=${VAULT_ADDR} \ --set values.hsm.vaultAuthPath=auth/${HSM_SECRET_PATH}/login \ --set values.hsm.vaultAuthRole=istiod \ --set values.hsm.vaultAuthJwtPath="/var/run/secrets/kubernetes.io/serviceaccount/token" \ --set values.hsm.vaultSecretPath=${HSM_SECRET_PATH} \ --set values.global.jwtPolicy="first-party-jwt"
Configure the validating webhook so that it can locate the
istiod
service with the revision label:kubectl apply -f istiod-service.yaml
This command creates a service entry that lets the validating webhook automatically check configurations before they are applied.
Verify the Istiod is initialized correctly and in waiting status.
kubectl get pod -l app=istiod -n istio-system
Expect output like the following:
NAME READY STATUS RESTARTS AGE istiod-66ff56d76c-f9p5l 2/3 Running 2 1m27s
See that Istiod is not fully ready because the
discovery
container is blocked and waiting for the certificate.Check the log of Istiod containers to make sure they are in the right state.
kubectl logs -c hsm-plugin -l app=istiod -n istio-system
Expect output like the following:
The token has been initialized and is reassigned to slot 412789065 INFO Loaded P11 PIN from file: /var/run/hsm-credential/pin line="cmd/serve.go:93" INFO KMS Plugin Listening on : /var/run/hsm-socket/.sock line="cmd/serve.go:119" INFO Serving on socket: /var/run/hsm-socket/.sock line="cmd/serve.go:198" KEK ID: 7651a4ea-eeb7-4c1f-927b-8c871c2127aa
kubectl logs -c discovery -l app=istiod -n istio-system
Expect the last line of output like the following:
... 2020-10-15T21:56:56.918931Z info pkica Wait until the CA certificate secret istio-system.istio-ca-cert can be loaded...
Sign certificate for Istiod
Download the encrypted CSR from the Kubernetes secret.
kubectl get secret istio-ca-cert-csr -n istio-system -o jsonpath={.data} > encrypted_csr.json
Decrypt the CSR.
kubectl get secret istio-ca-cert-csr -n istio-system -o jsonpath={.data} > encrypted_csr.json ./k8s-kms-plugin decrypt-csr -f encrypted_csr.json -o csr.pem
The security administrator should take the
csr.pem
file and sign it using the root CA.Put your certificate chain into the root of the file named
cert-chain.pem
, and then run the following command.kubectl create secret generic istio-ca-cert --from-file=cert-chain.pem -n istio-system
Verify Istiod loads the new cert chain successfully by inspecting at the Istiod log.
kubectl logs ISTIOD_POD -c discovery -n istio-system | grep "CA cert\:" -A 60
Expect the output to be similar to the following:
2020-10-24T18:58:14.354254Z info pkica CA cert: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----
intermediate certs: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----
root cert: -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----
Verify the installation
Use the following steps to verify that your Anthos Service Mesh installation is working correctly.
Check the control plane components
Check that the control plane pods in
istio-system
are running:kubectl get pod -n istio-system
Expected output is similar to the following:
NAME READY STATUS RESTARTS AGE istio-ingressgateway-74cc894bfd-786rg 1/1 Running 0 7m19s istiod-78cdbbbdb-d7tps 3/3 Running 0 7m36s promsd-576b8db4d6-lqf64 2/2 Running 1 7m19s
Trigger a TLS call to a service from the sidecar, and inspect the certificate used by the service, using bookinfo as an example.
kubectl exec POD -c istio-proxy -- openssl s_client -alpn istio -showcerts -connect details:9080
Inspect the certificates in the output, and expect to see the server-side certificate chain in the output.
Inject sidecar proxies
Anthos Service Mesh uses sidecar proxies to enhance network security, reliability, and
observability. With Anthos Service Mesh, these functions are abstracted away from the
application's primary container and implemented in a common out-of-process proxy
delivered as a separate container in the same Pod. To inject your Pods with
the sidecar proxy, you configure automatic sidecar proxy injection
(auto-injection) by labeling your namespaces with the same revision
label that you set on istiod
when you installed Anthos Service Mesh.
You need to enable auto-injection on any namespaces with workloads that were running on your cluster before you installed Anthos Service Mesh.
Before you deploy new workloads, make sure to configure auto-injection so that Anthos Service Mesh can monitor and secure traffic.
To enable auto-injection:
Use the following command to locate the revision label on
istiod
:kubectl -n istio-system get pods -l app=istiod --show-labels
The output looks similar to the following:
NAME READY STATUS RESTARTS AGE LABELS istiod-asm-178-10-5788d57586-bljj4 1/1 Running 0 23h app=istiod,istio.io/rev=asm-178-10,istio=istiod,pod-template-hash=5788d57586 istiod-asm-178-10-5788d57586-vsklm 1/1 Running 1 23h app=istiod,istio.io/rev=asm-178-10,istio=istiod,pod-template-hash=5788d57586
In the output, under the
LABELS
column, note the value of theistiod
revision label, which follows the prefixistio.io/rev=
. In this example, the value isasm-178-10
.Apply the revision label and remove the
istio-injection
label if it exists. In the following command,NAMESPACE
is the name of the namespace where you want to enable auto-injection, andREVISION
is the revision label you noted in the previous step.kubectl label namespace NAMESPACE istio-injection-istio.io/rev=REVISION --overwrite
You can ignore the message
"istio-injection not found"
in the output. That means that the namespace didn't previously have theistio-injection
label, which you should expect in new installations of Anthos Service Mesh or new deployments. Because auto-injection fails if a namespace has both theistio-injection
and the revision label, allkubectl label
commands in the Anthos Service Mesh documentation include removing theistio-injection
label.If workloads were running on your cluster before you installed Anthos Service Mesh, restart the Pods to trigger re-injection.
How you restart Pods depends on your application and the environment the cluster is in. For example, in your staging environment, you might simply delete all the Pods, which causes them to restart. But in your production environment, you might have a process that implements a blue-green deployment so that you can safely restart Pods to avoid traffic interruption.
You can use
kubectl
to perform a rolling restart:kubectl rollout restart deployment -n NAMESPACE
Verify that your Pods are configured to point to the new version of
istiod
.kubectl get pods -n NAMESPACE -l istio.io/rev=REVISION
Configure an external IP address
The default Anthos Service Mesh installation assumes that an external IP address is
automatically allocated for LoadBalancer
services. This is not true in
GKE on VMware. Because of this, you need to allocate an IP
address manually for the Anthos Service Mesh ingress Gateway resource.
To configure an external IP address, follow one of the sections below, depending on your cluster's load balancing mode:
Configure integrated load balancing mode
Open the
istio-ingressgateway
Service's configuration:kubectl edit svc -n istio-system istio-ingressgateway
The configuration for the
istio-ingressgateway
service opens in your shell's default text editor.In the file, add the following line under the specification (
spec
) block:loadBalancerIP: <your static external IP address>
For example:
spec: loadBalancerIP: 203.0.113.1
Save the file.
Configure manual load balancing mode
To expose a service of type NodePort
with a virtual IP address (VIP) on your load balancer, get the nodePort
values first:
View the
istio-ingressgateway
service's configuration in your shell:kubectl get svc -n istio-system istio-ingressgateway -o yaml
Each of the ports for Anthos Service Mesh's gateways are displayed. The command output is similar to the following:
... ports: - name: status-port nodePort: 30391 port: 15020 protocol: TCP targetPort: 15020 - name: http2 nodePort: 31380 port: 80 protocol: TCP targetPort: 80 - name: https nodePort: 31390 port: 443 protocol: TCP targetPort: 443 - name: tcp nodePort: 31400 port: 31400 protocol: TCP targetPort: 31400 - name: https-kiali nodePort: 31073 port: 15029 protocol: TCP targetPort: 15029 - name: https-prometheus nodePort: 30253 port: 15030 protocol: TCP targetPort: 15030 - name: https-grafana nodePort: 30050 port: 15031 protocol: TCP targetPort: 15031 - name: https-tracing nodePort: 31204 port: 15032 protocol: TCP targetPort: 15032 - name: tls nodePort: 30158 port: 15443 protocol: TCP targetPort: 15443 ...
Expose these ports through your load balancer.
For example, the service port named
http2
hasport
80 andnodePort
31380. Suppose the node addresses for your user cluster are192.168.0.10
,192.168.0.11
, and192.168.0.12
, and your load balancer's VIP is203.0.113.1
.Configure your load balancer so that traffic sent to
203.0.113.1:80
is forwarded to192.168.0.10:31380
,192.168.0.11:31380
, or192.168.0.12:31380
. You can select the service ports that you want to expose on this given VIP.