Secure Kubernetes Services with Istio

Autopilot

---

This tutorial is intended for Kubernetes users and administrators who are interested in using Istio service mesh to securely deploy Kubernetes Services and enable mutual TLS (mTLS) communication.

Istio and Anthos Service Mesh

Istio is not a supported Google product. We recommend running managed Anthos Service Mesh instead. For more information, see Provision Anthos Service Mesh on a GKE Autopilot cluster.

To run Istio on a GKE Autopilot cluster, you must enable the NET_ADMIN and NET_RAW Linux capabilities in your containers. These are privileged linux capabilities. The NET_ADMIN capability grants broad access to network operations, such as write access to routing tables and firewalls. An attacker could potentially use this access to perform privilege escalation attacks in your cluster. For details, see the CAP_NET_ADMIN manpage.

Anthos Service Mesh provides the following benefits:

• You can provision managed Anthos Service Mesh using the Fleet API without client-side tools like istioctl.
• Anthos Service Mesh automatically injects sidecar proxies into workloads without granting elevated privileges to your containers.
• You can view rich dashboards for your mesh and services without any extra configuration and then use these metrics to configure service level objectives (SLOs) and alerts to monitor the health of your applications.
• The managed Anthos Service Mesh control plane is upgraded automatically to ensure that you get the latest security patches and features.
• The Anthos Service Mesh managed data plane automatically upgrades the sidecar proxies in your workloads so that you don't need to restart services yourself when proxy upgrades and security patches are available.
• Anthos Service Mesh is a supported product and can be configured using standard open source Istio APIs. For more information, see supported features.

Objectives

This tutorial includes the following steps:

• Create a GKE Autopilot cluster.
• Install Istio using the istioctl command line tool.
• Deploy a sample application to test mutual TLS (mTLS) authentication.
• Configure Istio to use mTLS authentication for service-to-service communication using a PeerAuthentication custom resource.
• Verify mTLS authentication using the Kiali dashboard.

Costs

In this document, you use the following billable components of Google Cloud:

• GKE
• Managed Service for Prometheus
• Cloud Load Balancing

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

Cloud Shell is preinstalled with the software you need for this tutorial, including kubectl, the gcloud CLI, and Terraform. If you don't use Cloud Shell, you must install the gcloud CLI.

1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
2. Install the Google Cloud CLI.

3. To initialize the gcloud CLI, run the following command:

   gcloud init

   Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

4. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

5. Make sure that billing is enabled for your Google Cloud project.

6. Enable the GKE API:

   gcloud services enable container.googleapis.com

7. Install the Google Cloud CLI.

8. To initialize the gcloud CLI, run the following command:

   gcloud init

   Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

9. Create or select a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   • Create a Google Cloud project:

     gcloud projects create PROJECT_ID

     Replace PROJECT_ID with a name for the Google Cloud project you are creating.

   • Select the Google Cloud project that you created:

     gcloud config set project PROJECT_ID

     Replace PROJECT_ID with your Google Cloud project name.

10. Make sure that billing is enabled for your Google Cloud project.

11. Enable the GKE API:

    gcloud services enable container.googleapis.com

12. Grant roles to your Google Account. Run the following command once for each of the following IAM roles: roles/container.clusterAdmin

    gcloud projects add-iam-policy-binding PROJECT_ID --member="user:EMAIL_ADDRESS" --role=ROLE

    • Replace PROJECT_ID with your project ID.
    • Replace EMAIL_ADDRESS with your email address.
    • Replace ROLE with each individual role.

Prepare the environment

To set up your environment, follow these steps:

1. Set environment variables:

   export PROJECT_ID=PROJECT_ID
   gcloud config set project $PROJECT_ID
   gcloud config set compute/region us-central1
   

   Replace PROJECT_ID with your Google Cloud project ID.

2. Clone the GitHub repository:

   git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples.git
   

3. Change to the working directory:

   cd kubernetes-engine-samples/service-mesh/istio-tutorial
   

Create a GKE cluster

Enable the Linux capabilities that Istio requires: NET_RAW and NET_ADMIN. GKE Autopilot does not allow NET_ADMIN by default, but you can enable NET_ADMIN using the --workload-policies=allow-net-admin command in GKE versions 1.27 and later:

gcloud container clusters create-auto istio-cluster \
    --location="us-central1" \
    --workload-policies="allow-net-admin"

To learn more about GKE Autopilot security, see Built-in security configurations.

Install Istio

You can install Istio on a GKE cluster using Istioctl.

In this tutorial, you install Istio with the default configuration profile recommended for production deployments.

1. Install Istio:

   export ISTIO_VERSION=1.20.2
   curl -L https://istio.io/downloadIstio | TARGET_ARCH=$(uname -m) sh -
   

2. Add the istioctl command line tool to the PATH:

   cd istio-${ISTIO_VERSION}
   export PATH=$PWD/bin:$PATH
   

3. Install Istio on the cluster:

   istioctl install --set profile="default" -y
   

   This step might take several minutes.

4. Wait for the Istio Pods to be ready:

   watch kubectl get pods -n istio-system
   

   The output is similar to the following:

   NAME                                    READY   STATUS        RESTARTS   AGE
   istio-ingressgateway-5c47bff876-wjm96   1/1     Running       0          2m54s
   istiod-5fc7cb65cd-k8cp4                 1/1     Running       0          2m57s
   

   When the Istio Pods are Running, return to the command line by pressing Ctrl+C.

Deploy the sample application

In this section, you use the Bank of Anthos sample application to create a service mesh with mTLS authentication.

1. Add a namespace label that instructs Istio to enable automatic injection of Envoy sidecar proxies:

   kubectl label namespace default istio-injection=enabled
   

2. Deploy the sample application:

   cd ..
   git clone https://github.com/GoogleCloudPlatform/bank-of-anthos.git
   kubectl apply -f bank-of-anthos/extras/jwt/jwt-secret.yaml
   kubectl apply -f bank-of-anthos/kubernetes-manifests/
   

3. Wait for the application to be ready:

   watch kubectl get pods
   

   The output is similar to the following:

   NAME                                 READY   STATUS    RESTARTS   AGE
   accounts-db-0                        2/2     Running   0          2m16s
   balancereader-5c695f78f5-x4wlz       2/2     Running   0          3m8s
   contacts-557fc79c5-5d7fg             2/2     Running   0          3m7s
   frontend-7dd589c5d7-b4cgq            2/2     Running   0          3m7s
   ledger-db-0                          2/2     Running   0          3m6s
   ledgerwriter-6497f5cf9b-25c6x        2/2     Running   0          3m5s
   loadgenerator-57f6896fd6-lx5df       2/2     Running   0          3m5s
   transactionhistory-6c498965f-tl2sk   2/2     Running   0          3m4s
   userservice-95f44b65b-mlk2p          2/2     Running   0          3m4s
   

   When the Pods are Running, return to the command line by pressing Ctrl+C.

4. Review the following manifest:

   # Copyright 2020 Google LLC
   #
   # Licensed under the Apache License, Version 2.0 (the "License");
   # you may not use this file except in compliance with the License.
   # You may obtain a copy of the License at
   #
   #      http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   apiVersion: networking.istio.io/v1alpha3
   kind: Gateway
   metadata:
     name: frontend-gateway
   spec:
     selector:
       istio: ingressgateway # use Istio default gateway implementation
     servers:
     - port:
         number: 80
         name: http
         protocol: HTTP
       hosts:
       - "*"
   ---
   apiVersion: networking.istio.io/v1alpha3
   kind: VirtualService
   metadata:
     name: frontend-ingress
   spec:
     hosts:
     - "*"
     gateways:
     - frontend-gateway
     http:
     - route:
       - destination:
           host: frontend
           port:
             number: 80

   This manifest describes Istio Gateway and VirtualService resources that expose the application and use Istio as the Ingress controller.

5. Apply the manifest to your cluster:

   kubectl apply -f bank-of-anthos/extras/istio/frontend-ingress.yaml
   

Configure mTLS

Mutual TLS (mTLS) authentication is enabled by default in Istio. That means that Istio monitors server workloads that have been migrated to Istio proxies and automatically configures client proxies to establish mTLS connections with these workloads. Istio also configures client proxies not to use mTLS when connecting to workloads without sidecar proxies.

Istio can configure mTLS to work in three modes:

• PERMISSIVE: Workloads accept both mTLS and plain text traffic.
• STRICT: Workloads only accept mTLS traffic.
• DISABLE: mTLS is disabled. Use this mode if you want to use your own security solution.

You can apply mTLS configuration globally, per namespace, or per workload. In this tutorial, you apply configuration per namespace using the STRICT mTLS mode.

1. Review the following manifest:

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default
   spec:
     mtls:
         mode: STRICT

   This manifest describes a Peer Authentication Istio Custom Resource.

2. Apply the manifest to your cluster:

   kubectl apply -f peer-authentication.yaml
   

For more information about mTLS in Istio, see mutual TLS authentication.

Verify mTLS is enabled

Kiali is a web-based observability dashboard for Istio service mesh that provides a graphical view of your microservices environment, allowing you to monitor and troubleshoot your applications. You can use Kiali to verify that mTLS authentication is enabled and functioning correctly in the Istio service mesh. Kiali requires Prometheus as a telemetry data source. This tutorial uses Google Cloud Managed Service for Prometheus.

Install a query interface

1. Create a IAM service account with the roles/monitoring.viewer to allow the query interface to access metrics:

   gcloud iam service-accounts create monitoring \
       --display-name="Service account for query interface"
   gcloud projects add-iam-policy-binding PROJECT_ID \
       --member "serviceAccount:monitoring@PROJECT_ID.iam.gserviceaccount.com" \
       --role roles/monitoring.viewer
   gcloud iam service-accounts add-iam-policy-binding \
     monitoring@PROJECT_ID.iam.gserviceaccount.com \
       --role roles/iam.workloadIdentityUser \
       --member "serviceAccount:PROJECT_ID.svc.id.goog[monitoring/default]"
   

2. Create a Kubernetes namespace:

   kubectl create namespace monitoring
   

3. Annotate the default Kubernetes service account in the namespace to configure Workload Identity Federation for GKE:

   kubectl annotate serviceaccount -n monitoring default \
       iam.gke.io/gcp-service-account=monitoring@PROJECT_ID.iam.gserviceaccount.com --overwrite
   

4. Deploy the query interface workload:

   kubectl -n monitoring apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.7.1/examples/frontend.yaml
   

5. Review the following manifest:

   apiVersion: monitoring.googleapis.com/v1
   kind: PodMonitoring
   metadata:
     name: istiod
     namespace: istio-system
   spec:
     selector:
       matchLabels:
         app: istiod
     endpoints:
     - port: 15014
       path: /metrics
       timeout: 30s
       interval: 60s

   This manifest describes a PodMonitoring resource that collects Istio and Envoy Proxy metrics.

6. Apply the manifest to your cluster:

   kubectl apply -f pod-monitorings.yaml
   

7. Get a link to the sample application:

   INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo "http://$INGRESS_HOST"
   

8. Open the link to view the sample application. Sign in with the default username and password to generate traffic between the microservices.

Install Kiali

We recommend that you install Kiali using the Kiali Operator.

1. Install the Kiali Operator:

   helm repo add kiali https://kiali.org/helm-charts
   helm repo update
   helm install \
       --namespace kiali-operator \
       --create-namespace \
       kiali-operator \
       kiali/kiali-operator
   

2. Review the following manifest:

   apiVersion: kiali.io/v1alpha1
   kind: Kiali
   metadata:
     name: kiali
     namespace: istio-system
   spec:
     deployment:
       namespace: istio-system
     auth:
       strategy: anonymous
     external_services:
       custom_dashboards:
         prometheus:
           url: "http://frontend.monitoring:9090/"
           auth:
             type: none
       prometheus:
         url: "http://frontend.monitoring:9090/"
         auth:
           type: none
       tracing:
         enabled: false
       grafana:
         enabled: false

   This manifest describes an Operator custom resource that defines the Kiali server.

3. Apply the manifest to your cluster:

   kubectl apply -f kiali.yaml
   

4. Wait for the Kiali server to be ready:

   watch kubectl get pods -n istio-system
   

   The output is similar to the following:

   NAME                                    READY   STATUS    RESTARTS   AGE
   istio-ingressgateway-6845466857-92zp8   1/1     Running   0          9m11s
   istiod-6b47d84cf-4cqlt                  1/1     Running   0          12m
   

   When the Pods are Running, return to the command line by pressing Ctrl+C.

5. Set up port forwarding on the Kiali server Service to access the dashboard:

   kubectl -n istio-system port-forward svc/kiali 8080:20001
   

6. Open Web Preview. In Kiali, go to the Graph section and select the Security option in the Display drop-down. This view displays the security state of each node in the graph. Nodes with an mTLS enabled badge indicate that mTLS is enabled for that service, and nodes without the badge indicate that mTLS is not enabled.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

Delete the individual resources

If you used an existing project and you don't want to delete it, delete the individual resources.

1. Delete Kiali:

   kubectl -n istio-system delete kiali kiali
   helm uninstall --namespace kiali-operator kiali-operator
   

2. Delete the monitoring resources:

   kubectl -n monitoring delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/prometheus-engine/v0.7.1/examples/frontend.yaml
   

3. Delete the sample application:

   kubectl delete -f bank-of-anthos/extras/istio/frontend-ingress.yaml
   kubectl delete -f bank-of-anthos/kubernetes-manifests
   

4. Uninstall Istio:

   istioctl uninstall --purge -y
   

5. Delete the GKE cluster:

   gcloud container clusters delete --region us-central1 istio-cluster --quiet
   

What's next

• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.