Tutorial: Secure GKE Enterprise

---

GKE Enterprise provides a consistent platform for building and delivering secure services, with security features built in at every level that work separately and together to provide defence in depth against security issues. This tutorial introduces you to some of GKE Enterprise's powerful security features using the Anthos Sample Deployment on Google Cloud. The Anthos Sample Deployment deploys a real GKE Enterprise hands-on environment with a GKE cluster, service mesh, and a Bank of GKE Enterprise application with multiple microservices.

Note: If you're new to GKE Enterprise and the Anthos Sample Deployment, before trying this tutorial, go to Explore GKE Enterprise to install the deployment and learn about GKE Enterprise features and concepts through our sample application.

Objectives

In this tutorial, you're introduced to some of GKE Enterprise's security features through the following tasks:

• Enforce mutual TLS (mTLS) in your service mesh by using Config Sync to ensure end-to-end secure communication.

• Set up a security guardrail that ensures that pods with privileged containers are not inadvertently deployed by using Policy Controller.

Costs

Deploying the Bank of Anthos application will incur pay-as-you-go charges for GKE Enterprise on Google Cloud as listed on our Pricing page, unless you have already purchased a subscription.

You are also responsible for other Google Cloud costs incurred while running the Bank of Anthos application, such as charges for Compute Engine VMs and load balancers.

We recommend cleaning up after finishing the tutorial or exploring the deployment to avoid incurring further charges.

Before you begin

This tutorial is a follow-up to the Explore Anthos tutorial. Before starting this tutorial, follow the instructions on that page to set up your project and install the Anthos Sample Deployment.

Setting up your Cloud Shell environment

In this tutorial, you will use the Cloud Shell command line and editor to make changes to cluster configuration.

To initialize the shell environment for the tutorial, the Anthos Sample Deployment provides a script that does the following:

• Installs any missing command-line tools for interactively working with and verifying changes to the deployment:

  • kubectl
  • kubectx
  • istioctl
  • nomos

• Sets the Kubernetes context for anthos-sample-cluster1

• Clones the repository that Config Sync uses for synchronizing your configuration changes to your cluster. Changes that you commit and push to the upstream repository are synchronized to your infrastructure by Config Sync. This is the recommended best practice for applying changes to your infrastructure.

To set up your environment:

1. Ensure that you have an active Cloud Shell session. You can launch Cloud Shell by clicking Activate Cloud Shell from the Google Cloud console in your tutorial project.

2. Create a directory to work in:

   mkdir tutorial
   cd tutorial
   

3. Download the initialization script:

   curl -sLO https://github.com/GoogleCloudPlatform/anthos-sample-deployment/releases/latest/download/init-anthos-sample-deployment.env
   

4. Source the initialization script into your Cloud Shell environment:

   source init-anthos-sample-deployment.env
   

   Output:

   /google/google-cloud-sdk/bin/gcloud
   /google/google-cloud-sdk/bin/kubectl
   Your active configuration is: [cloudshell-13605]
   export PROJECT as anthos-launch-demo-1
   export KUBECONFIG as ~/.kube/anthos-launch-demo-1.anthos-trial-gcp.config
   Fetching cluster endpoint and auth data.
   kubeconfig entry generated for anthos-sample-cluster1.
   Copying gs://config-management-release/released/latest/linux_amd64/nomos...
   \ [1 files][ 40.9 MiB/ 40.9 MiB]
   Operation completed over 1 objects/40.9 MiB.
   Installed nomos into ~/bin.
   Cloned ACM config repo: ./anthos-sample-deployment-config-repo
   

5. Change the directory to the configuration repository and use it as the working directory for the remainder of this tutorial:

   cd anthos-sample-deployment-config-repo
   

Enforcing mTLS in your service mesh

In anticipation of global expansion, your CIO has mandated that all user data must be encrypted in transit to safeguard sensitive information to be in compliance with regional data privacy and encryption laws.

So is all your traffic currently secure?

1. Go to the Cloud Service Mesh page in your project where you have the Anthos Sample Deployment deployed:

   Go to the Cloud Service Mesh page

2. Click transactionhistory in the services list. As you saw in Explore GKE Enterprise, the service details page shows all the telemetry available for this service.

3. On the transactionhistory page, on the Navigation menu, select Connected Services. Here you can see both the Inbound and Outbound connections for the service. An unlocked lock icon indicates that some traffic has been observed on this port that is not using mutual TLS (mTLS).

mTLS is a security protocol that ensures that traffic is secure and trusted in both directions between two services. Each service accepts only encrypted traffic from authenticated services. As you can see, Cloud Service Mesh clearly shows that you have unencrypted traffic in your mesh. Different colors are used in Cloud Service Mesh to indicate whether the unencrypted traffic has a mix of plaintext and mTLS (orange) or only plaintext (red).

With GKE Enterprise, you're only a few steps away from being in compliance. Rather than make changes at the source code level and rebuild and redeploy your application to address this situation, you can apply the new encryption policy declaratively through configuration by using Config Sync to automatically deploy your new configuration from a central Git repository.

In this section, you'll do the following:

1. Adjust the policy configuration in your Git repository to enforce that services use encrypted communications through mTLS.

2. Rely on Config Sync to automatically pick up the policy change from the repository and adjust the Cloud Service Mesh policy.

3. Verify that the policy change occurred on your cluster that is configured to sync with the repository.

Confirm Config Sync setup

1. The nomos command is a command-line tool that lets you interact with the Config Management Operator and perform other useful Config Sync tasks from your local machine or Cloud Shell. To verify that Config Sync is properly installed and configured on your cluster, run nomos status:

   nomos status
   

   Output:

   Connecting to clusters...
   Current   Context                  Sync Status  Last Synced Token   Sync Branch   Resource Status
   -------   -------                  -----------  -----------------   -----------   ---------------
   *         anthos-sample-cluster1   SYNCED       abef0b01            master        Healthy
   

   The output confirms that Config Sync is configured to sync your cluster to the master branch of your configuration repository. The asterisk in the first column indicates that the current context is set to anthos-sample-cluster1. If you don't see this, switch the current context to anthos-sample-cluster1:

   kubectl config use-context anthos-sample-cluster1
   

   Output:

   Switched to context "anthos-sample-cluster1".
   

2. Ensure that you're on the master branch:

   git checkout master
   

   Output:

   Already on 'master'
   Your branch is up to date with 'origin/master'.
   

3. Verify your upstream configuration repository:

   git remote -v
   

   Output:

   origin  https://source.developers.google.com/.../anthos-sample-deployment-config-repo (fetch)
   origin  https://source.developers.google.com/.../anthos-sample-deployment-config-repo (push)
   

4. Ensure you're still in the anthos-sample-deployment-config-repo directory, and run the following command to check your git setup. This helper function is sourced into your environment by the initialization script, and runs git config commands to check your git config's existing user.email and user.name values. If these values are not configured, the function sets defaults at the repo level based on the currently active Google Cloud account.

   init_git
   

   Output (example):

   Configured local git user.email to [e-mail]
   Configured local git user.name to user
   

You are now ready to commit policy changes to your repository. When you push these commits to your upstream repository (origin), Config Sync ensures that these changes are applied to the cluster that you have configured it to manage.

Update a policy to encrypt all service traffic

Configuration for Cloud Service Mesh is specified declaratively by using YAML files. To encrypt all service traffic, you need to modify both the YAML that specifies the types of traffic that services can accept, and the YAML that specifies the type of traffic that services send to particular destinations.

1. The first YAML file that you need to look at is namespaces/istio-system/peer-authentication.yaml, which is a mesh-level authentication policy that specifies the types of traffic that all services in your mesh accept by default.

   cat namespaces/istio-system/peer-authentication.yaml
   

   Output:

   apiVersion: "security.istio.io/v1beta1"
   kind: "PeerAuthentication"
   metadata:
     name: "default"
     namespace: "istio-system"
   spec:
     mtls:
       mode: PERMISSIVE
   

   As you can see, the PeerAuthentication mTLS mode is PERMISSIVE, which means that services accept both plaintext HTTP and mTLS traffic.

2. Modify namespaces/istio-system/peer-authentication.yaml to allow only encrypted communication between services by setting the mTLS mode to STRICT:

   cat <<EOF> namespaces/istio-system/peer-authentication.yaml
   apiVersion: "security.istio.io/v1beta1"
   kind: "PeerAuthentication"
   metadata:
     name: "default"
     namespace: "istio-system"
   spec:
     mtls:
       mode: STRICT
   EOF
   

3. Next, look at the Destination Rule in namespaces/istio-system/destination-rule.yaml. This specifies rules for sending traffic to the specified destinations, including whether the traffic is encrypted. Notice that TLSmode is DISABLE, meaning that traffic is sent in plaintext to all matching hosts.

   cat namespaces/istio-system/destination-rule.yaml
   

   Output:

   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     annotations:
       meshsecurityinsights.googleapis.com/generated: "1561996419000000000"
     name: default
     namespace: istio-system
   spec:
     host: '*.local'
     trafficPolicy:
       tls:
         mode: DISABLE
   

4. Modify namespaces/istio-system/destination-rule.yaml to have Istio set a traffic policy that enables TLS for all matching hosts in the cluster by using TLSmode ISTIO_MUTUAL:

   cat <<EOF> namespaces/istio-system/destination-rule.yaml
   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   metadata:
     annotations:
       meshsecurityinsights.googleapis.com/generated: "1561996419000000000"
     name: default
     namespace: istio-system
   spec:
     host: '*.local'
     trafficPolicy:
       tls:
         mode: ISTIO_MUTUAL
   EOF
   

Push your changes to the repository

You are almost ready to push your configuration changes; however, we recommend a few checks before you finally commit your updates.

1. Run nomos vet to ensure that your configuration is valid:

   nomos vet
   

   No output indicates that there were no validation errors.

2. As soon as you push your changes, Config Sync picks them up and applies them to your system. To avoid unexpected results, we recommend checking that the current live state of your configuration hasn't changed since you made your edits. Use kubectl to check that the destinationrule reflects that mTLS is disabled for the cluster:

   kubectl get destinationrule default -n istio-system -o yaml
   

   Output:

   apiVersion: networking.istio.io/v1alpha3
   kind: DestinationRule
   ...
   spec:
     host: '*.local'
     trafficPolicy:
       tls:
         mode: DISABLE
   

3. Now commit and push these changes to the upstream repository. The following command uses a helper function called watchmtls that was sourced into your environment by the init script. This helper function runs a combination of nomos status and the kubectl command that you tried earlier. It watches the cluster for changes until you press Ctrl+C to quit. Monitor the display until you see that the changes are applied and synchronized on the cluster.

   git commit -am "enable mtls"
   git push origin master && watchmtls
   

   You can also see the changes reflected on the Cloud Service Mesh pages in GKE Enterprise.

   Go to the Cloud Service Mesh page

   You should see that the red unlocked lock icon has changed. The lock icon appears orange (mixed traffic) rather than green (entirely encrypted traffic) because we're looking by default at the last hour with a mix of mTLS and plaintext. If you check back after an hour, you should see a green lock that shows that you have successfully encrypted all the service traffic.

Using Policy Controller to set up guardrails

Your security team is concerned about potential root attacks that might occur when running pods with privileged containers (containers with root access). While the current configuration does not deploy any privileged containers, you want to guard against as many threat vectors as possible that could compromise performance or, even worse, customer data.

Despite the team's diligence, there is still a risk that you could find yourself vulnerable to root attacks unintentionally from future configuration updates through your continuous delivery process. You decide to set up a security guardrail to protect against this danger.

Apply guardrails

Guardrails are automated administrative controls intended to enforce policies that protect your environment. Policy Controller includes support for defining and enforcing custom rules not covered by native Kubernetes objects. Policy Controller checks, audits, and enforces guardrails that you apply that correspond to your organization's unique security, regulatory compliance, and governance requirements.

Use Policy Controller

Policy Controller is built on an open source policy engine called Gatekeeper that is used to enforce policies each time a resource in the cluster is created, updated, or deleted. These policies are defined by using constraints from the Policy Controller template library or from other Gatekeeper constraint templates.

The Anthos Sample Deployment on Google Cloud already has Policy Controller installed and also has the Policy Controller template library enabled. You can take advantage of this when implementing your guardrail by using an existing constraint for privileged containers from the library.

Apply a policy constraint for privileged containers

To address your security team's concerns, you apply the K8sPSPPrivilegedContainer constraint. This constraint denies pods from running with privileged containers.

1. Using the Cloud Shell terminal, create a new constraint.yaml file with the text from the library constraint, as follows:

   cat <<EOF> ~/tutorial/anthos-sample-deployment-config-repo/cluster/constraint.yaml
   apiVersion: constraints.gatekeeper.sh/v1beta1
   kind: K8sPSPPrivilegedContainer
   metadata:
     name: psp-privileged-container
   spec:
     match:
       kinds:
         - apiGroups: [""]
           kinds: ["Pod"]
       excludedNamespaces: ["kube-system"]
   EOF
   

2. Use nomos vet to verify that the updated configuration is valid before you apply it.

   nomos vet
   

   The command returns silently as long as there are no errors.

3. Commit and push the changes to apply the policy. You can use nomos status with the watch command to confirm that the changes are applied to your cluster. Press Ctrl+C to exit the watch command when finished.

   git add .
   git commit -m "add policy constraint for privileged containers"
   git push && watch nomos status
   

   Output:

   Connecting to clusters...
   Current   Context                  Sync Status  Last Synced Token   Sync Branch   Resource Status
   -------   -------                  -----------  -----------------   -----------   ---------------
   *         anthos-sample-cluster1   SYNCED       f2898e92            master        Healthy
   

Test your policy

After you've applied the policy, you can test it by attempting to run a pod with a privileged container.

1. In the Cloud Shell terminal, use the following command to create a new file in the tutorial directory, nginx-privileged.yaml, with the contents from this example spec:

   cat <<EOF> ~/tutorial/nginx-privileged.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: nginx-privileged-disallowed
     labels:
       app: nginx-privileged
   spec:
     containers:
     - name: nginx
       image: nginx
       securityContext:
         privileged: true
   EOF
   

2. Attempt to launch the pod with kubectl apply.

   kubectl apply -f ~/tutorial/nginx-privileged.yaml
   

   Output:

   Error from server ([denied by psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {"privileged": true}): error when creating "~/nginx-privileged.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by psp-privileged-container] Privileged container is not allowed: nginx, security
   Context: {"privileged": true}
   

   The error shows that the Gatekeeper admission controller monitoring your Kubernetes environment enforced your new policy. It prevented the pod's execution due to the presence of a privileged container in the pod's specification.

The concept of version-controlled policies that you can apply to set up guardrails with Policy Controller is a powerful one because it standardizes, unifies, and centralizes the governance of your clusters, enforcing your policies through active monitoring of your environment post-deployment.

You can find many other types of policies to use as guardrails for your environment in the Gatekeeper repository.

Exploring the deployment further

While this tutorial has shown you how to work with some GKE Enterprise security features, there's still lots more to see and do in GKE Enterprise with our deployment. Feel free to try another tutorial or continue to explore the Anthos Sample Deployment on Google Cloud yourself, before following the cleanup instructions in the next section.

Clean up

After you've finished exploring the Bank of Anthos application, you can clean up the resources that you created on Google Cloud so they don't take up quota and you aren't billed for them in the future.

• Option 1. You can delete the project. However, if you want to keep the project around, you can use Option 2 to delete the deployment.

• Option 2. If you want to keep your current project, you can use terraform destroy to delete the sample application and cluster.

Delete the project (option 1)

The easiest way to avoid billing is to delete the project you created for this tutorial.

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

1. In the Google Cloud console, go to the Manage resources page.

   Go to Manage resources

2. In the project list, select the project that you want to delete, and then click Delete.
3. In the dialog, type the project ID, and then click Shut down to delete the project.

Delete the deployment (option 2)

This approach deletes the Bank of Anthos application and the cluster, but does not delete the project. Run the following commands on your Cloud Shell:

1. Change to the directory that hosts the installation scripts:

   cd bank-of-anthos/iac/tf-anthos-gke
   

2. Delete the sample and the cluster:

   terraform destroy
   

3. Enter the project ID when prompted.

If you plan to redeploy, verify that all requirements are met as described in the Before you begin section.

What's next

There's lots more to explore in our GKE Enterprise documentation.

Try more tutorials

• Learn about service management with the Anthos Sample Deployment in Manage services with GKE Enterprise.

• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.

Learn more about GKE Enterprise

• Learn more about GKE Enterprise in our technical overview.

• Find out how to set up GKE Enterprise in a real production environment in our setup guide.

• Find out how to do more with Cloud Service Mesh in the Cloud Service Mesh documentation.

• Find out more about Policy Controller in the Policy Controller guide

• Find out more about declarative, centralized configuration and policy management in the Config Sync documentation and Policy Controller documentation