Strengthen your app's security with Cloud Service Mesh, Config Sync, and Policy Controller

---

This tutorial shows you how to improve your cluster and app's security posture. Imagine you are a platform administrator whose organization is managing the apps for their online store with Cloud Service Mesh, a suite of tools that helps you monitor and manage a reliable service mesh. You are responsible for ensuring that your mesh and apps are secure.

You can prevent misconfiguration and automatically validate your Cloud Service Mesh policies by using Policy Controller and Config Sync. Policy Controller enables the enforcement of fully programmable policies for your clusters. Policy Controller also comes with a default library of constraint templates that you can use with the Cloud Service Mesh security bundle to audit the compliance of your mesh security vulnerabilities and best practices. Config Sync continuously reconciles the state of clusters with a central set of Kubernetes declarative configuration files. Using Policy Controller and Config Sync together enables you to continuously enforce constraints on your Cloud Service Mesh policy configurations.

The following diagram shows you an overview of how Cloud Service Mesh, Policy Controller, and Config Sync work together in this tutorial to manage and protect an ingress gateway and the Online Boutique sample apps that you use in this tutorial:

Objectives

• Create a Google Kubernetes Engine (GKE) cluster and register the cluster to a fleet.
• Install Policy Controller, Config Sync, and Cloud Service Mesh on a cluster.
• Configure Config Sync to sync multiple repositories
• Apply best practices to deploy configs, apps and Istio resources with Config Sync.
• Deploy cluster configs, the Online Boutique sample apps, and an ingress gateway with Config Sync.
• Leverage the Cloud Service Mesh policy bundle of Policy Controller to enforce the following security best practices:
  • Ensure that all workloads in the mesh have automatic sidecar injection.
  • Encrypt all traffic in the mesh.
  • Guarantee that all workloads in the mesh have granular access control.

Costs

In this document, you use the following billable components of Google Cloud:

• GKE.
• GKE Enterprise. The billing for GKE Enterprise includes billing for the Cloud Service Mesh, Config Sync, and Policy Controller.

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

   Note: If you don't plan to keep the resources that you create in this procedure, create a project instead of selecting an existing project. After you finish these steps, you can delete the project, removing all resources associated with the project.

   Go to project selector

2. Make sure that billing is enabled for your Google Cloud project.

Prepare your environment

In this section, you prepare your environment so that you can install Cloud Service Mesh, Policy Controller, and Config Sync:

1. In the Google Cloud console, activate Cloud Shell.

   Activate Cloud Shell

2. Upgrade to the latest version of the Google Cloud CLI:

   gcloud components update
   

3. To store the files that you create in this tutorial, create a directory:

   mkdir ~/asm-acm-tutorial-dir
   

4. To simplify the remainder of the tutorial, create the following environment variables:

   PROJECT_ID=PROJECT_ID
   gcloud config set project $PROJECT_ID
   CLUSTER=asm-acm-tutorial
   CLUSTER_ZONE=us-east4-a
   MEMBERSHIP=asm-acm-tutorial
   PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format='get(projectNumber)')
   

   Replace PROJECT_ID with the project ID that you want to use for this tutorial.

   If you're asked to authorize Cloud Shell, click Authorize to complete the operation.

5. Enable the APIs that you need for this tutorial:

   gcloud

   gcloud services enable \
       mesh.googleapis.com \
       anthos.googleapis.com
   

   Config Connector

   This tutorial includes Config Connector resources. You can use these resources to complete the same tasks that you complete in the gcloud tab. To utilize these resources, install Config Connector and apply the resources in the way that works best for your environment.

   Use the following Services manifest:

   apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
   kind: Service
   metadata:
     annotations:
       cnrm.cloud.google.com/deletion-policy: "abandon"
       cnrm.cloud.google.com/disable-dependent-services: "false"
     name: mesh.googleapis.com
   spec:
     resourceID: mesh.googleapis.com
     projectRef:
       external: PROJECT_ID
   ---
   apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
   kind: Service
   metadata:
     annotations:
       cnrm.cloud.google.com/deletion-policy: "abandon"
       cnrm.cloud.google.com/disable-dependent-services: "false"
     name: anthos.googleapis.com
   spec:
     resourceID: anthos.googleapis.com
     projectRef:
       external: PROJECT_ID
   

   This operation can take over one minute to complete.

Set up a GKE cluster

In this section, you create a GKE cluster and then register it to a fleet. Fleets are a Google Cloud concept for logically organizing clusters and other resources, letting you use and manage multi-cluster capabilities and apply consistent policies across your systems.

The cluster that you create in this section is the cluster that you install Cloud Service Mesh, Policy Controller, and Config Sync on. It's also the cluster where you deploy the Online Boutique sample apps.

To set up your cluster, complete the following steps:

1. Create a GKE cluster:

   gcloud

   gcloud container clusters create ${CLUSTER} \
       --zone ${CLUSTER_ZONE} \
       --machine-type=e2-standard-4 \
       --num-nodes 4 \
       --workload-pool ${PROJECT_ID}.svc.id.goog \
       --labels mesh_id=proj-${PROJECT_NUMBER}
   

   Config Connector

   Use the following ContainerCluster and ContainerNodePool manifests:

   apiVersion: container.cnrm.cloud.google.com/v1beta1
   kind: ContainerNodePool
   metadata:
     annotations:
       cnrm.cloud.google.com/project-id: PROJECT_ID
     name: asm-acm-tutorial
   spec:
     clusterRef:
       name: asm-acm-tutorial
     location: us-east4-a
     nodeConfig:
       machineType: e2-standard-4
     nodeCount: 4
   ---
   apiVersion: container.cnrm.cloud.google.com/v1beta1
   kind: ContainerCluster
   metadata:
     annotations:
       cnrm.cloud.google.com/project-id: PROJECT_ID
       cnrm.cloud.google.com/remove-default-node-pool: "true"
     labels:
       mesh_id: proj-PROJECT_NUMBER
     name: asm-acm-tutorial
   spec:
     location: us-east4-a
     initialNodeCount: 1
     workloadIdentityConfig:
       workloadPool: PROJECT_ID.svc.id.goog
   

   Replace PROJECT_NUMBER by the value of the PROJECT_NUMBER environment variable retrieved earlier.

   This operation can take over five minutes to complete.

2. To ensure the successful creation of the GKE cluster, describe its status:

   gcloud container clusters list \
       --zone ${CLUSTER_ZONE} \
       --project ${PROJECT_ID}
   

   The output is similar to the following:

   NAME                LOCATION      MASTER_VERSION   MASTER_IP      MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
   asm-acm-tutorial    us-east4-a    1.23.12-gke.100  35.186.179.30  e2-standard-4  1.23.12-gke.100  3          RUNNING
   

3. Connect to the GKE cluster:

   gcloud container clusters get-credentials ${CLUSTER} \
       --zone ${CLUSTER_ZONE} \
       --project ${PROJECT_ID}
   

4. Register your cluster to a fleet:

   gcloud

   gcloud container fleet memberships register ${MEMBERSHIP} \
       --project ${PROJECT_ID} \
       --gke-cluster ${CLUSTER_ZONE}/${CLUSTER} \
       --enable-workload-identity
   

   The output is similar to the following:

   kubeconfig entry generated for asm-acm-tutorial.
   Waiting for membership to be created...done.
   Created a new membership [projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial] for the cluster [asm-acm-tutorial]
   Generating the Connect Agent manifest...
   Deploying the Connect Agent on cluster [asm-acm-tutorial] in namespace [gke-connect]...
   Deployed the Connect Agent on cluster [asm-acm-tutorial] in namespace [gke-connect].
   Finished registering the cluster [asm-acm-tutorial] with the Fleet.
   

   Config Connector

   Use the following GKEHubMembership manifest:

   apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
   kind: GKEHubMembership
   metadata:
     annotations:
       cnrm.cloud.google.com/project-id: PROJECT_ID
     name: asm-acm-tutorial
   spec:
     location: global
     authority:
       issuer: https://container.googleapis.com/v1/projects/PROJECT_ID/locations/us-east4-a/clusters/asm-acm-tutorial
     endpoint:
       gkeCluster:
         resourceRef:
           name: asm-acm-tutorial
   

5. To ensure the successful registration of the GKE cluster, describe its status:

   gcloud container fleet memberships list
   

   The output is similar to the following:

   NAME              EXTERNAL_ID                           LOCATION
   asm-acm-tutorial  0e12258c-8831-4d81-b5c0-5e7099a468cc  global
   

Explore the repositories

In the following installation section, you apply a manifest acm-config.yaml file. This manifest configures your cluster to sync from the asm-acm-tutorial folder of the sample repository. This folder contains all the configuration files that you need to complete the remainder of the tutorial.

To simplify this tutorial, you use sed commands to update the acm-config.yaml. With the acm-config.yaml file, Config Sync deploys the manifests required for each step of this tutorial. Updating a single file helps you focus on the concepts and the flow of securing your clusters, mesh, and applications without repeatedly manipulating the files and repeatedly running git commands.

To utilize Config Sync's ability to sync multiple repositories, you use the following resources:

• root-sync, as a RootSync repository, contains all the configs across your cluster including RepoSyncs, Constraints, ClusterRole, RoleBindings, and resources included in some system namespaces such as istio-system.
• ingress-gateway, as a first RepoSync, contains all the resources needed to deploy an ingress gateway and progressively secure it throughout this tutorial.
• online-boutique, as a second RepoSync, contains all the resources needed to deploy the Online Boutique apps and progressively secure them throughout this tutorial.

Install Policy Controller, Config Sync, and managed Cloud Service Mesh

Now that you have created and registered your cluster, you can install Config Sync, Policy Controller, and Cloud Service Mesh on your cluster and configure your cluster to sync from the configs of the default RootSync:

1. Enable the ConfigManagement operator, which manages Config Sync and Policy Controller:

   gcloud

   gcloud beta container fleet config-management enable
   

   Config Connector

   Use the following GKEHubFeature manifest:

   apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
   kind: GKEHubFeature
   metadata:
     name: configmanagement
   spec:
     projectRef:
       external: PROJECT_ID
     location: global
     resourceID: configmanagement
   

2. Enable Cloud Service Mesh in your fleet.

   gcloud

   gcloud container fleet mesh enable
   

   Config Connector

   Use the following GKEHubFeature manifest:

   apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
   kind: GKEHubFeature
   metadata:
     name: servicemesh
   spec:
     projectRef:
       external: PROJECT_ID
     location: global
     resourceID: servicemesh
   

3. Enable the Cloud Service Mesh automatic management to let Google apply the recommended configuration of managed Cloud Service Mesh:

   gcloud

   gcloud container fleet mesh update \
       --management automatic \
       --memberships ${MEMBERSHIP}
   

   Config Connector

   Use the following GKEHubFeatureMembership manifest:

   apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
   kind: GKEHubFeatureMembership
   metadata:
     name: servicemesh-membership
   spec:
     projectRef:
       external: PROJECT_ID
     location: global
     membershipRef:
       name: asm-acm-tutorial
     featureRef:
       name: servicemesh
     mesh:
       management: MANAGEMENT_AUTOMATIC
   

4. Enable Config Sync and Policy Controller:

   gcloud

   Save the following manifest as acm-config.yaml in the ~/asm-acm-tutorial-dir directory:

   applySpecVersion: 1
   spec:
     configSync:
       enabled: true
       policyDir: asm-acm-tutorial/root-sync/init
       secretType: none
       sourceFormat: unstructured
       syncRepo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
       syncBranch: main
     policyController:
       enabled: true
       referentialRulesEnabled: true
       templateLibraryInstalled: true
   

   To learn more about the Google Cloud CLI config fields, see gcloud apply spec fields.

   Apply the file:

   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   Config Connector

   Use the following GKEHubFeatureMembership manifest:

   apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
   kind: GKEHubFeatureMembership
   metadata:
     name: configmanagement-membership
   spec:
     projectRef:
       external: PROJECT_ID
     location: global
     membershipRef:
       name: asm-acm-tutorial
     featureRef:
       name: configmanagement
     configmanagement:
       configSync:
         sourceFormat: unstructured
         git:
           policyDir: asm-acm-tutorial/root-sync/init
           secretType: none
           syncBranch: main
           syncRepo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
       policyController:
         enabled: true
         referentialRulesEnabled: true
         templateLibraryInstalled: true
   

   Policy Controller and Config Sync are installed on your cluster. Next, Config Sync begins syncing all the configs of the default RootSync to your cluster. These configs install and configure the following key components:

   • The RepoSync objects that configure the Online Boutique apps and the ingress gateway are synced:

     apiVersion: configsync.gke.io/v1beta1
     kind: RepoSync
     metadata:
       name: repo-sync
     spec:
       override:
         enableShellInRendering: true
       sourceFormat: unstructured
       git:
         repo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
         revision: HEAD
         branch: main
         dir: asm-acm-tutorial/online-boutique/init
         auth: none

     apiVersion: configsync.gke.io/v1beta1
     kind: RepoSync
     metadata:
       name: repo-sync
     spec:
       override:
         enableShellInRendering: true
       sourceFormat: unstructured
       git:
         repo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
         revision: HEAD
         branch: main
         dir: asm-acm-tutorial/ingress-gateway/init
         auth: none

   • Since the RepoSync reconcilers need additional permissions to create Istio resources, a ClusterRole and two RoleBinding objects to grant these permissions are also applied to your cluster:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: ClusterRole
     metadata:
       labels:
         rbac.authorization.k8s.io/aggregate-to-edit: "true"
       name: custom:aggregate-to-edit:istio
     rules:
     - apiGroups:
       - "networking.istio.io"
       - "security.istio.io"
       resources:
       - "virtualservices"
       - "authorizationpolicies"
       - "gateways"
       verbs:
       - "*"

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: repo-sync
     subjects:
     - kind: ServiceAccount
       name: ns-reconciler-onlineboutique
       namespace: config-management-system
     roleRef:
       kind: ClusterRole
       name: edit
       apiGroup: rbac.authorization.k8s.io

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: repo-sync
     subjects:
     - kind: ServiceAccount
       name: ns-reconciler-asm-ingress
       namespace: config-management-system
     roleRef:
       kind: ClusterRole
       name: edit
       apiGroup: rbac.authorization.k8s.io

5. To ensure the successful installation of Policy Controller and Config Sync, check the status:

   gcloud beta container fleet config-management status
   

   The output is similar to the following:

   Name: asm-acm-tutorial
   Status: SYNCED
   Last_Synced_Token: 4b3384d
   Sync_Branch: main
   Last_Synced_Time: 2022-05-04T21:32:58Z
   Policy_Controller: INSTALLED
   

   If you see PENDING or NOT_INSTALLED in the Status or Policy_Controller rows, wait a few minutes and run gcloud beta container fleet config-management status again.

6. To ensure the successful installation of Cloud Service Mesh, describe its status:

   gcloud container fleet mesh describe
   

   The output is similar to the following:

   createTime: '2022-09-13T23:12:56.477042921Z'
   membershipSpecs:
     projects/PROJECT_NUMBER/locations/global/memberships/asm-acm-tutorial:
       mesh:
         management: MANAGEMENT_AUTOMATIC
   membershipStates:
     projects/PROJECT_NUMBER/locations/global/memberships/asm-acm-tutorial:
       servicemesh:
         controlPlaneManagement:
           details:
           - code: REVISION_READY
             details: 'Ready: asm-managed'
           state: ACTIVE
         dataPlaneManagement:
           details:
           - code: OK
             details: Service is running.
           state: ACTIVE
       state:
         code: OK
         description: |-
           Revision(s) ready for use: asm-managed.
           All Canonical Services have been reconciled successfully.
         updateTime: '2022-09-14T00:19:10.571552206Z'
   name: projects/PROJECT_ID/locations/global/features/servicemesh
   resourceState:
     state: ACTIVE
   spec: {}
   state:
     state: {}
   updateTime: '2022-09-14T00:19:14.135113118Z'
   

   If you see state.code: ERROR instead of state.code: OK, wait a few minutes and run gcloud container fleet mesh describe again. Before moving forward with the tutorial, you need to make sure that the servicemesh.controlPlaneManagement.details.code field has the REVISION_READY value.

Deploy an ingress gateway and a sample application

In this section, you deploy the Online Boutique sample application and an ingress gateway to manage ingress traffic.

1. Deploy the Online Boutique sample application and the ingress gateway.

   The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the resources you need to deploy the ingress gateway and sample app.

   sed -i "s,root-sync/init,root-sync/deployments,g" ~/asm-acm-tutorial-dir/acm-config.yaml
   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   Note this step can take a few minutes to complete.

2. View the Config Sync status for the RootSync and the two RepoSyncs:

   gcloud alpha anthos config sync repo describe
   

   The output is similar to:

   getting 3 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
   [
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "95a30c052566357afb9db3d7f6153d9c0f219c03",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/deployments@main",
       "status": "SYNCED"
     },
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "95a30c052566357afb9db3d7f6153d9c0f219c03",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/ingress-gateway/deployments@main",
       "status": "SYNCED"
     },
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "95a30c052566357afb9db3d7f6153d9c0f219c03",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/online-boutique/deployments@main",
       "status": "SYNCED"
     }
   ]
   

   If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

   To only view the information of one repository you can use the --sync-name and --sync-namespace flags. To view the managed resources in detail, add the --managed-resources flag. For more information, see View Config Sync status across multiple clusters.

3. Wait for the ingress gateway's public IP address to be provisioned:

   until kubectl -n asm-ingress get svc asm-ingressgateway -o jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done
   

4. Get the ingress gateway's public IP address:

   EXTERNAL_IP=$(kubectl get svc asm-ingressgateway -n asm-ingress -o jsonpath="{.status.loadBalancer.ingress[*].ip}")
   

5. Visit the IP address from your browser to verify that the Online Boutique app has successfully deployed:

   echo http://${EXTERNAL_IP}
   

Enforce policies to secure your mesh

In the following sections, you leverage Policy Controller to enforce policies from the Cloud Service Mesh policy bundle by creating constraints.

Enforce sidecar proxies injection

In this section you enforce policies to ensure that all workloads in the mesh have automatic sidecar injection enabled.

1. To enforce sidecar proxies injection, apply constraints.

   The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

   sed -i "s,root-sync/deployments,root-sync/enforce-sidecar-injection,g" ~/asm-acm-tutorial-dir/acm-config.yaml
   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   The preceding command deploys the following resources:

   • A K8sRequiredLabels Constraint that requires any Namespace in the mesh to contain the specific Cloud Service Mesh sidecar proxy injection label:

     apiVersion: constraints.gatekeeper.sh/v1beta1
     kind: K8sRequiredLabels
     metadata:
       name: namespace-sidecar-injection-label
     spec:
       enforcementAction: deny
       match:
         kinds:
         - apiGroups:
           - ""
           kinds:
           - Namespace
         excludedNamespaces:
         - config-management-monitoring
         - config-management-system
         - default
         - gatekeeper-system
         - gke-connect
         - istio-system
         - kube-node-lease
         - kube-public
         - kube-system
         - resource-group-system
       parameters:
         labels:
         - allowedRegex: enabled
           key: istio-injection

   • An AsmSidecarInjection Constraint that prohibits any Pod in the mesh from bypassing the Istio proxy sidecar injection:

     apiVersion: constraints.gatekeeper.sh/v1beta1
     kind: AsmSidecarInjection
     metadata:
       name: pod-sidecar-injection-annotation
     spec:
       enforcementAction: deny
       match:
         kinds:
         - apiGroups:
           - ""
           kinds:
           - Pod
         excludedNamespaces:
         - kube-system
       parameters:
         strictnessLevel: High

2. View the Config Sync status for the RootSync:

   gcloud alpha anthos config sync repo describe \
       --sync-name root-sync \
       --sync-namespace config-management-system
   

   The output is similar to:

   getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
   [
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/enforce-sidecar-injection@main",
       "status": "SYNCED"
     }
   ]
   

   If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

3. Verify the Constraints are created:

   kubectl get constraints
   

   It can take a few minutes for Policy Controller to evaluate these constraints. If you don't see values in the TOTAL-VIOLATIONS column, wait and run kubectl get constraints again.

   The output is similar to:

   NAME                                                                                       ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   podsidecarinjectionannotation.constraints.gatekeeper.sh/pod-sidecar-injection-annotation   deny                 0
   
   NAME                                                                            ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   k8srequiredlabels.constraints.gatekeeper.sh/namespace-sidecar-injection-label   deny                 0
   

   Because we properly set up our Namespaces and Pods, there are 0 TOTAL-VIOLATIONS for these Constraints.

4. To see these Constraints at work, try to create a Namespace in your cluster with neither a label nor an annotation:

   kubectl create namespace test
   

   The output is similar is the following error:

   Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [namespace-sidecar-injection-label] you must provide labels: {"istio-injection"}
   

Enforce traffic encryption

In this section you enforce policies to ensure that all traffic in the mesh is encrypted.

1. To enforce traffic encryption, apply constraints.

   The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

   sed -i "s,root-sync/enforce-sidecar-injection,root-sync/enforce-strict-mtls,g" ~/asm-acm-tutorial-dir/acm-config.yaml
   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   The preceding command deploys the following resources:

   • An AsmPeerAuthnMeshStrictMtls Constraint which enforces the mesh-level mTLS PeerAuthentication in the istio-system namespace:

     apiVersion: constraints.gatekeeper.sh/v1beta1
     kind: AsmPeerAuthnMeshStrictMtls
     metadata:
       name: mesh-level-strict-mtls
     spec:
       enforcementAction: deny
       parameters:
         rootNamespace: istio-system
         strictnessLevel: High

   • A referential constraint Config in the gatekeeper-system namespace. This referential constraint enables the AsmPeerAuthnMeshStrictMtls Constraint to reference another object in its definition (for example, searching for any PeerAuthentication in the istio-system Namespace):

     apiVersion: config.gatekeeper.sh/v1alpha1
     kind: Config
     metadata:
       name: config
     spec:
       sync:
         syncOnly:
           - group: ""
             version: "v1"
             kind: "Namespace"
           - group: "security.istio.io"
             version: "v1beta1"
             kind: "PeerAuthentication"
           - group: "security.istio.io"
             version: "v1beta1"
             kind: "AuthorizationPolicy"

   • A DestinationRuleTLSEnabled Constraint which prohibits disabling TLS for all hosts and host subsets in Istio DestinationRules:

     apiVersion: constraints.gatekeeper.sh/v1beta1
     kind: DestinationRuleTLSEnabled
     metadata:
       name: destination-rule-tls-enabled
     spec:
       enforcementAction: deny
       match:
         kinds:
         - apiGroups:
           - networking.istio.io
           kinds:
           - DestinationRule

   • An AsmPeerAuthnStrictMtls Constraint which enforces that all PeerAuthentications cannot overwrite STRICT mTLS:

     apiVersion: constraints.gatekeeper.sh/v1beta1
     kind: AsmPeerAuthnStrictMtls
     metadata:
       name: peerauthentication-strict-mtls
     spec:
       enforcementAction: deny
       match:
         kinds:
         - apiGroups:
           - security.istio.io
           kinds:
           - PeerAuthentication
       parameters:
         strictnessLevel: High

2. View the Config Sync status for the RootSync:

   gcloud alpha anthos config sync repo describe \
       --sync-name root-sync \
       --sync-namespace config-management-system
   

   The output is similar to:

   getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
   [
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/enforce-strict-mtls@main",
       "status": "SYNCED"
     }
   ]
   

   If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

3. Run the following command to get more information about the PeerAuthentication violation:

   kubectl get asmpeerauthnmeshstrictmtls.constraints.gatekeeper.sh/mesh-level-strict-mtls -ojsonpath='{.status.violations}'  | jq
   

   The output is similar to:

   [
     {
       "enforcementAction": "deny",
       "group": "constraints.gatekeeper.sh",
       "kind": "AsmPeerAuthnMeshStrictMtls",
       "message": "Root namespace <istio-system> does not have a strict mTLS PeerAuthentication",
       "name": "mesh-level-strict-mtls",
       "version": "v1beta1"
     }
   ]
   

4. Fix the issue by deploying a PeerAuthentication in the istio-system. To prevent all your services in the mesh from accepting plaintext traffic, set a mesh-wide PeerAuthentication policy with the mTLS mode set to STRICT. When you deploy the policy, the control plane automatically provisions TLS certificates so that workloads can authenticate with each other.

   The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

   sed -i "s,root-sync/enforce-strict-mtls,root-sync/fix-strict-mtls,g" ~/asm-acm-tutorial-dir/acm-config.yaml
   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   The preceding command deploys the following STRICT mTLS PeerAuthentication in the istio-system namespace. This applies mTLS STRICT to the entire mesh:

   apiVersion: security.istio.io/v1beta1
   kind: PeerAuthentication
   metadata:
     name: default
   spec:
     mtls:
       mode: STRICT

5. View the Config Sync status for the RootSync:

   gcloud alpha anthos config sync repo describe \
       --sync-name root-sync \
       --sync-namespace config-management-system
   

   The output is similar to:

   getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
   [
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/fix-strict-mtls@main",
       "status": "SYNCED"
     }
   ]
   

   If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

6. Verify the Constraints are created:

   kubectl get constraints
   

   Note this can take a few minutes to get Policy Controller evaluating these Constraints. Wait and run again this kubectl get constraints command until you get values under the TOTAL-VIOLATIONS column for each line.

   The output is similar to:

   NAME                                                                            ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   k8srequiredlabels.constraints.gatekeeper.sh/namespace-sidecar-injection-label   deny                 0
   NAME                                                                          ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmpeerauthnmeshstrictmtls.constraints.gatekeeper.sh/mesh-level-strict-mtls   deny                 0
   NAME                                                                               ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   destinationruletlsenabled.constraints.gatekeeper.sh/destination-rule-tls-enabled   deny                 0
   NAME                                                                              ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmpeerauthnstrictmtls.constraints.gatekeeper.sh/peerauthentication-strict-mtls   deny                 0
   NAME                                                                             ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmsidecarinjection.constraints.gatekeeper.sh/pod-sidecar-injection-annotation   deny                 0
   

Enforce granular access control

In this section you enforce policies to ensure that all workloads in the mesh have granular access control.

1. To enforce granular access control, apply constraints.

   The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

   sed -i "s,root-sync/fix-strict-mtls,root-sync/enforce-authorization-policies,g" ~/asm-acm-tutorial-dir/acm-config.yaml
   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   The preceding command deploys the following resources:

   • An AsmAuthzPolicyDefaultDeny Constraint which enforces the mesh-level default deny AuthorizationPolicy in the istio-system namespace:

     apiVersion: constraints.gatekeeper.sh/v1beta1
     kind: AsmAuthzPolicyDefaultDeny
     metadata:
       name: default-deny-authorization-policies
     spec:
       enforcementAction: deny
       parameters:
         rootNamespace: istio-system
         strictnessLevel: High

   • An AsmAuthzPolicyEnforceSourcePrincipals Constraint which enforces that any AuthorizationPolicies is defining granular source principals (other than "*"). Only the ingress gateway in the asm-ingress namespace is an exception to this rule in order to receive the traffic from the end-users and redirect the traffic to the Online Boutique's frontend app.

     apiVersion: constraints.gatekeeper.sh/v1beta1
     kind: AsmAuthzPolicyEnforceSourcePrincipals
     metadata:
       name: authz-source-principals-not-all
     spec:
       enforcementAction: deny
       match:
         kinds:
         - apiGroups:
           - security.istio.io
           kinds:
           - AuthorizationPolicy
         excludedNamespaces:
           - asm-ingress

2. View the Config Sync status for the RootSync:

   gcloud alpha anthos config sync repo describe \
       --sync-name root-sync \
       --sync-namespace config-management-system
   

   The output is similar to:

   getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
   [
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/enforce-authorization-policies@main",
       "status": "SYNCED"
     }
   ]
   

   If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

3. Run the following command to get more information about the associated violation:

   kubectl get asmauthzpolicydefaultdeny.constraints.gatekeeper.sh/default-deny-authorization-policies -ojsonpath='{.status.violations}'  | jq
   

   The output is similar to:

   [
     {
       "enforcementAction": "deny",
       "group": "constraints.gatekeeper.sh",
       "kind": "AsmAuthzPolicyDefaultDeny",
       "message": "Root namespace <istio-system> does not have a default deny AuthorizationPolicy",
       "name": "default-deny-authorization-policies",
       "version": "v1beta1"
     }
   ]
   

4. Fix the issue by deploying the AuthorizationPolicy in the istio-system namespace.

   The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

   sed -i "s,root-sync/enforce-authorization-policies,root-sync/fix-default-deny-authorization-policy,g" ~/asm-acm-tutorial-dir/acm-config.yaml
   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   The preceding command deploys the following deny-all AuthorizationPolicy in the istio-system namespace:

   apiVersion: security.istio.io/v1beta1
   kind: AuthorizationPolicy
   metadata:
     name: deny-all
   spec:
     {}

5. View the Config Sync status for the RootSync:

   gcloud alpha anthos config sync repo describe \
       --sync-name root-sync \
       --sync-namespace config-management-system
   

   The output is similar to:

   getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
   [
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/fix-default-deny-authorization-policy@main",
       "status": "SYNCED"
     }
   ]
   

   If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

6. Verify the Constraints are created:

   kubectl get constraints
   

   Note this can take a few minutes to get Policy Controller evaluating these Constraints. Wait and run again this kubectl get constraints command until you get values under the TOTAL-VIOLATIONS column for each line.

   The output is similar to:

   NAME                                                                             ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmsidecarinjection.constraints.gatekeeper.sh/pod-sidecar-injection-annotation   deny                 0
   NAME                                                                            ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   k8srequiredlabels.constraints.gatekeeper.sh/namespace-sidecar-injection-label   deny                 0
   NAME                                                                                      ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmauthzpolicydefaultdeny.constraints.gatekeeper.sh/default-deny-authorization-policies   deny                 0
   NAME                                                                          ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmpeerauthnmeshstrictmtls.constraints.gatekeeper.sh/mesh-level-strict-mtls   deny                 0
   NAME                                                                               ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   destinationruletlsenabled.constraints.gatekeeper.sh/destination-rule-tls-enabled   deny                 0
   NAME                                                                              ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmpeerauthnstrictmtls.constraints.gatekeeper.sh/peerauthentication-strict-mtls   deny                 0
   NAME                                                                                              ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
   asmauthzpolicyenforcesourceprincipals.constraints.gatekeeper.sh/authz-source-principals-not-all   deny                 0
   

7. Visit the Online Boutique app from your browser:

   echo http://${EXTERNAL_IP}
   

   You should receive the error: RBAC: access denied which confirms that the default deny AuthorizationPolicy is applied to the entire mesh.

8. Fix this issue by deploying more granular AuthorizationPolicies in the asm-ingress and onlineboutique namespaces.

   The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

   sed -i "s,root-sync/fix-default-deny-authorization-policy,root-sync/deploy-authorization-policies,g" ~/asm-acm-tutorial-dir/acm-config.yaml
   gcloud beta container fleet config-management apply \
       --membership ${MEMBERSHIP} \
       --config ~/asm-acm-tutorial-dir/acm-config.yaml
   

   The preceding command deploys the following resources:

   • An AuthorizationPolicy in the asm-ingress namespace:

     apiVersion: security.istio.io/v1beta1
     kind: AuthorizationPolicy
     metadata:
       name: asm-ingressgateway
     spec:
       selector:
         matchLabels:
           asm: ingressgateway
       rules:
       - to:
         - operation:
             ports:
             - "8080"

   • An AuthorizationPolicy per app in the onlineboutique namespace, here is the example for the cartservice app:

     apiVersion: security.istio.io/v1beta1
     kind: AuthorizationPolicy
     metadata:
       name: cartservice
     spec:
       selector:
         matchLabels:
           app: cartservice
       rules:
       - from:
         - source:
             principals:
             - cluster.local/ns/onlineboutique/sa/frontend
             - cluster.local/ns/onlineboutique/sa/checkoutservice
         to:
         - operation:
             paths:
             - /hipstershop.CartService/AddItem
             - /hipstershop.CartService/GetCart
             - /hipstershop.CartService/EmptyCart
             methods:
             - POST
             ports:
             - "7070"

   • A ServiceAccount per app in the asm-ingress and onlineboutique namespaces in order to have a unique identity per app evaluated as principal in the AuthorizationPolicies. Here is the example for the cartservice app:

     apiVersion: v1
     kind: ServiceAccount
     metadata:
       name: cartservice

9. View the Config Sync status for the RootSync and the two RepoSyncs:

   gcloud alpha anthos config sync repo describe
   

   The output is similar to:

   getting 3 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
   [
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/deploy-authorization-policies@main",
       "status": "SYNCED"
     },
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/ingress-gateway/authorization-policies@main",
       "status": "SYNCED"
     },
     {
       "clusters": [
         "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
       ],
       "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
       "errors": [],
       "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/online-boutique/authorization-policies@main",
       "status": "SYNCED"
     }
   ]
   

   If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

   To only view the information of one repository you can use the --sync-name and --sync-namespace flags. And to view in detail the managed resources, you can add the --managed-resources flag. For more information, see View Config Sync status across multiple clusters.

10. Visit the Online Boutique app again from your browser:

    echo http://${EXTERNAL_IP}
    

    If you wait a few minutes, you should now see the website working successfully again as expected.

View the status of GKE Enterprise security features

You can view the status of GKE Enterprise security features, including authentication and authorization policies, in the Google Cloud console.

1. In the Google Cloud console, go to the GKE Enterprise Security page.

   Go to GKE Enterprise Security

   The Policy Summary displays the status of application security, including Service access control (AuthorizationPolicies) and mTLS.

2. Click Policy Audit to view workload policy statuses for the cluster and both namespaces (asm-ingress and onlineboutique).

   The Service access control and mTLS status cards provide a high-level overview.

   

   The Workloads list shows the Service access control and mTLS status of each workload.

   

You now have secured your cluster and your mesh with Policy Controller and Config Sync.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

Caution: Deleting a project has the following effects:

• Everything in the project is deleted. If you used an existing project for the tasks in this document, when you delete it, you also delete any other work you've done in the project.
• Custom project IDs are lost. When you created this project, you might have created a custom project ID that you want to use in the future. To preserve the URLs that use the project ID, such as an appspot.com URL, delete selected resources inside the project instead of deleting the whole project.

If you plan to explore multiple architectures, tutorials, or quickstarts, reusing projects can help you avoid exceeding project quota limits.

Delete a Google Cloud project:

gcloud projects delete PROJECT_ID

Delete individual resources

To delete the individual resources:

1. Unregister your cluster from the fleet:

   gcloud

   gcloud container fleet memberships unregister ${CLUSTER} \
       --project=${PROJECT_ID} \
       --gke-cluster=${CLUSTER_ZONE}/${CLUSTER}
   

   The output is similar to the following:

   kubeconfig entry generated for asm-acm-tutorial.
   Waiting for membership to be deleted...done.
   Deleting membership CR in the cluster...done.
   Deleting namespace [gke-connect] in the cluster...done.
   

   Config Connector

   kubectl delete -f ~/asm-acm-tutorial-dir/fleet-membership.yaml
   

2. Delete your cluster:

   gcloud

   gcloud container clusters delete ${CLUSTER} \
       --zone ${CLUSTER_ZONE}
   

   Press y when prompted. This command can take over five minutes to complete.

   The output is similar to the following:

   Deleting cluster asm-acm-tutorial...done.
   Deleted [https://container.googleapis.com/v1/projects/PROJECT_ID/zones/us-east4-a/clusters/asm-acm-tutorial].
   

   Config Connector

   kubectl delete -f ~/asm-acm-tutorial-dir/container-cluster.yaml
   

3. Delete the files that you created:

   rm -r ~/asm-acm-tutorial-dir
   

What's next

• Discover Cloud Service Mesh security best practices.
• Learn more about the Cloud Service Mesh policy bundle.
• Explore the Policy Controller constraint template library.
• Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.