Installing and upgrading gateways

Cloud Service Mesh gives you the option to deploy and manage gateways as part of your service mesh. A gateway describes a load balancer operating at the edge of the mesh receiving incoming or outgoing HTTP/TCP connections. Gateways are Envoy proxies that provide you with fine-grained control over traffic entering and leaving the mesh. Gateways are primarily used to manage ingress traffic, but you can also configure gateways to manage other types of traffic. For example:

• Egress gateways: An egress gateway lets you configure a dedicated exit node for the traffic leaving the mesh, letting you limit which services can or should access external networks, or to enable secure control of egress traffic to add security to your mesh, for example.

• East-west gateways: A proxy for east-west traffic to allow service workloads to communicate across cluster boundaries in a multi-primary mesh on different networks. By default, this gateway will be public on the Internet.

This page describes best practices for deploying and upgrading the gateway proxies as well as examples of configuring your own istio-ingressgateway and istio-egressgateway gateway proxies. Things like traffic splitting, redirects, and retry logic are possible by applying a Gateway configuration to the gateway proxies. Then instead of adding application-layer traffic routing (L7) to the same API resource, you bind a Virtual Service to the Gateway. This lets you manage gateway traffic like any other data plane traffic in the service mesh.

You can deploy gateways in different ways and you can choose to use more than one topology within the same cluster. Refer to Gateway deployment topologies in the Istio documentation to learn more about these topologies.

Best practices for deploying gateways

The best practices for deploying gateways depend on whether you are using the managed data plane or the unmanaged data plane.

Best practices for managed data plane

1. Enable the managed data plane.
2. Add a managed revision label to a namespace.
3. Deploy and manage the control plane and gateways separately.
4. As a security best practice, we recommended that you deploy gateways in a different namespace from the control plane.
5. Use automatic sidecar injection (auto-injection) to inject the proxy configuration for the gateways similar to how you would inject the sidecar proxies for your services.

These best practices:

• Ensure your managed gateways are automatically kept up-to-date with the latest enhancements and security updates.
• Offloads management and maintenance of gateway instances to Cloud Service Mesh managed data plane.

Best practices for unmanaged data plane

1. Deploy and manage the control plane and gateways separately.
2. As a security best practice, we recommended that you deploy gateways in a different namespace from the control plane.
3. Use automatic sidecar injection (auto-injection) to inject the proxy configuration for the gateways similar to how you would inject the sidecar proxies for your services.

These best practices:

• Let your namespace administrators manage gateways without needing elevated privileges to your entire cluster.
• Let your administrators use the same deployment tooling or mechanism that they use to manage Kubernetes applications to deploy and manage gateways.
• Give administrators full control over the gateway Deployment, and also simplify operations. When a new upgrade is available or a configuration has changed, administrators update gateway Pods by simply restarting them. This makes the experience of operating a gateway Deployment the same as operating sidecar proxies for your services.

Deploy gateways

To support users with existing deployment tools, Cloud Service Mesh supports the same ways to deploy a gateway as Istio: IstioOperator, Helm, and Kubernetes YAML. Each method produces the same result. Although you can choose the method you are most familiar with, we recommend that you use the Kubernetes YAML method because it is easier to modify and you can store hydrated manifests in source control.

1. Create a namespace for the gateway if you don't already have one. Replace GATEWAY_NAMESPACE with the name of your namespace.

   kubectl create namespace GATEWAY_NAMESPACE
   

2. To enable auto-injection, you label your namespaces with the default injection labels if the default tag is set up, or with the revision label to your namespace. The label that you add also depends on whether you deployed managed Cloud Service Mesh or installed the in-cluster control plane. The label is used by the sidecar injector webhook to associate injected sidecars with a particular control plane revision.

   Select the tab below according to your installation type (either managed or in-cluster).

   Managed

   Use the following command to locate the available release channels:

   kubectl -n istio-system get controlplanerevision
   

   The output is similar to the following:

   NAME                AGE
   asm-managed         6d7h
   asm-managed-rapid   6d7h
   

   In the output, the value under the NAME column is the revision label that corresponds to the available release channel for the Cloud Service Mesh version.

   In-cluster

   For in-cluster control planes, the istiod Service and Deployment typically have a revision label similar to istio.io/rev=asm-1187-26, where asm-1187-26 identifies the Cloud Service Mesh version. The revision becomes part of the istiod Service name, for example: istiod-asm-1187-26.istio-system

   Use the following command to locate the revision label on istiod for the in-cluster control plane:

   kubectl get deploy -n istio-system -l app=istiod \
     -o=jsonpath='{.items[*].metadata.labels.istio\.io\/rev}''{"\n"}'
   

3. Enable the namespace for injection. Replace and REVISION with the value for the revision label.

   Note: You can substitute istio.io/rev with the istio-injection=enabled label if the default tag is configured. Verify the default tag exists by running istioctl tag list with the istioctl from OUTPUT_DIR.

   kubectl label namespace GATEWAY_NAMESPACE istio-injection- istio.io/rev=REVISION --overwrite
   

4. If you installed Cloud Service Mesh using asmcli, change to the directory that you specified in --output_dir, then cd to the samples directory.

   Important: To ensure you are in the correct samples directory, run the ls command to list the directory contents and confirm there is a gateways directory (which you will access in the next step) and an online-boutique directory.

   If you didn't install using asmcli, copy the configuration files for the gateways from the anthos-service-mesh repository.

5. You can deploy the example gateway configuration located in the samples/gateways/ directory as is, or modify it as needed.

   Ingress

   kubectl apply -n GATEWAY_NAMESPACE -f samples/gateways/istio-ingressgateway
   

   Egress

   kubectl apply -n GATEWAY_NAMESPACE -f samples/gateways/istio-egressgateway
   

6. After you create the deployment, verify that the new services are working correctly:

   kubectl get pod,service -n GATEWAY_NAMESPACE
   

   Verify the output is similar to the following:

   NAME                                      READY   STATUS    RESTARTS   AGE
   pod/istio-ingressgateway-856b7c77-bdb77   1/1     Running   0          3s
   
   NAME                           TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE
   service/istio-ingressgateway   LoadBalancer   10.24.5.129    34.82.157.6      80:31904/TCP   3s

Important: Your gateway resources should be managed like any other Kubernetes application. The samples in the anthos-service-mesh-packages repo are meant for guidance and a quickstart and you should customize them according to your needs.

Gateway selectors

You apply a Gateway configuration to the istio-ingressgateway and istio-egressgateway proxies to manage inbound and outbound traffic for your mesh, letting you specify which traffic you want to enter or leave the mesh. The labels on a gateway deployment's Pods are used by Gateway configuration resources, so it's important that your Gateway selector matches these labels.

For example, in the above deployments, the istio=ingressgateway label is set on the gateway Pods. To apply a Gateway configuration to these deployments, you need to select the same label:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway
spec:
  selector:
    istio: ingressgateway
...

For an example of a Gateway configuration and Virtual Service, see frontend.yaml in the Online Boutique sample application.

Upgrade gateways

In-place upgrades

For most use cases, you should upgrade your gateways following the in-place upgrade pattern. Because gateways utilize Pod injection, new gateway Pods that are created will automatically be injected with the latest configuration, which includes the version.

If you would like to change the control plane revision in use by the gateway, you can set the istio.io/rev label on the gateway Deployment, which will also trigger a rolling restart.

Managed control plane

Since Google manages the control plane upgrades for the managed control plane, you can simply restart the gateway Deployment and the new pods will automatically be injected with the latest configuration and version.

kubectl rollout restart deployment istio-ingressgateway \
  -n GATEWAY_NAMESPACE

In-cluster control plane

To apply the same pattern to your gateways when you have the in-cluster control plane, you will need to change the control plane revision in use by the gateway. Set the istio.io/rev label on the gateway Deployment which will trigger a rolling restart. The steps required depend on whether you need to update the revision label on namespace and/or the gateway pod.

• If you labeled the namespace for injection, set the istio.io/rev label on the namespace to the new revision value:

    kubectl label namespace GATEWAY_NAMESPACE \
      istio-injection- istio.io/rev=REVISION \
      --overwrite
  

• If you enabled injection only for the gateway pod, set the istio.io/rev label on the Deployment to the new revision value like the following Kubernetes YAML file:

  cat <<EOF > gateway-deployment.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: istio-ingressgateway
    namespace: GATEWAY_NAMESPACE
  spec:
    selector:
      matchLabels:
        istio: ingressgateway
    template:
      metadata:
        annotations:
          # This is required to tell Anthos Service Mesh to inject the gateway with the
          # required configuration.
          inject.istio.io/templates: gateway
        labels:
          istio: ingressgateway
          istio.io/rev: REVISION
      spec:
        containers:
        - name: istio-proxy
          image: auto # The image will automatically update each time the pod starts.
  EOF
  
  kubectl apply -f gateway-deployment.yaml
  

  Note: If the image does not automatically update with the actual Cloud Service Mesh proxy image, then the container will not be found and an error will occur. See Troubleshoot gateways for next steps.

Canary upgrades (advanced)

If you are using the in-cluster control plane and would like to more slowly control the rollout of a new control plane revision, you can follow the canary upgrade pattern. You can run multiple versions of a gateway Deployment and ensure everything works as expected with a subset of your traffic. For example, if you want to roll out a new revision, canary, create a copy of your gateway Deployment with the istio.io/rev=REVISION label set to the new revision and a new name, for example istio-ingressgateway-canary:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-ingressgateway-canary
  namespace: GATEWAY_NAMESPACE
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  template:
    metadata:
      annotations:
        inject.istio.io/templates: gateway
      labels:
        istio: ingressgateway
        istio.io/rev: REVISION # Set to the control plane revision you want to deploy
    spec:
      containers:
      - name: istio-proxy
        image: auto

Note: If the image does not automatically update with the actual Cloud Service Mesh proxy image, then the container will not be found and an error will occur. See Troubleshoot gateways for next steps.

When this Deployment is created, you will then have two versions of the gateway, both selected by the same Service:

kubectl get endpoints -o "custom-columns=NAME:.metadata.name,PODS:.subsets[*].addresses[*].targetRef.name"

NAME                   PODS
istio-ingressgateway   istio-ingressgateway-788854c955-8gv96,istio-ingressgateway-canary-b78944cbd-mq2qf

Note: Unlike application services deployed inside the mesh, you cannot use Istio traffic shifting to distribute the traffic between the gateway versions because their traffic is coming directly from external clients that Istio does not control. Instead, you can control the distribution of traffic by the number of replicas of each Deployment. If you use another load balancer in front of Istio, you may also use that to control the traffic distribution.

If you are satisfied that your applications are working as expected, run this command to transition to the new version by deleting the Deployment with the old istio.io/rev label set:

kubectl delete deploy/istio-ingressgateway -n GATEWAY_NAMESPACE

If you encountered an issue when testing your application with the new version of the gateway, run this command to transition back to the old version by deleting the Deployment with the new istio.io/rev label set:

kubectl delete deploy/istio-ingressgateway-canary -n GATEWAY_NAMESPACE

Advanced configuration

Configure gateway minimum TLS version

For Cloud Service Mesh version 1.14 and later, the default minimum TLS version for gateway servers is 1.2. You can configure the minimum TLS version using the minProtocolVersion field. For more information, see ServerTLSSettings.

Troubleshoot gateways

Failed to update gateway image from auto

When you deploy or upgrade a gateway, Cloud Service Mesh inserts auto as a placeholder in the image field. After the call to mutating webhook, Cloud Service Mesh automatically replaces this placeholder with the actual Cloud Service Mesh proxy image. If the call to mutating webhook fails, the auto placeholder remains, and the container will not be found. This is typically caused by an incorrect namespace label. Ensure you have configured the correct namespace and then deploy or upgrade the gateway again.