Strengthen your app's security with Cloud Service Mesh, Config Sync, and Policy Controller


This tutorial shows you how to improve your cluster and app's security posture. Imagine you are a platform administrator whose organization is managing the apps for their online store with Cloud Service Mesh, a suite of tools that helps you monitor and manage a reliable service mesh. You are responsible for ensuring that your mesh and apps are secure.

You can prevent misconfiguration and automatically validate your Cloud Service Mesh policies by using Policy Controller and Config Sync. Policy Controller enables the enforcement of fully programmable policies for your clusters. Policy Controller also comes with a default library of constraint templates that you can use with the Cloud Service Mesh security bundle to audit the compliance of your mesh security vulnerabilities and best practices. Config Sync continuously reconciles the state of clusters with a central set of Kubernetes declarative configuration files. Using Policy Controller and Config Sync together enables you to continuously enforce constraints on your Cloud Service Mesh policy configurations.

The following diagram shows you an overview of how Cloud Service Mesh, Policy Controller, and Config Sync work together in this tutorial to manage and protect an ingress gateway and the Online Boutique sample apps that you use in this tutorial:

A diagram showing the architecture that you create for this tutorial

Objectives

  • Create a Google Kubernetes Engine (GKE) cluster and register the cluster to a fleet.
  • Install Policy Controller, Config Sync, and Cloud Service Mesh on a cluster.
  • Configure Config Sync to sync multiple repositories
  • Apply best practices to deploy configs, apps and Istio resources with Config Sync.
  • Deploy cluster configs, the Online Boutique sample apps, and an ingress gateway with Config Sync.
  • Leverage the Cloud Service Mesh policy bundle of Policy Controller to enforce the following security best practices:
    • Ensure that all workloads in the mesh have automatic sidecar injection.
    • Encrypt all traffic in the mesh.
    • Guarantee that all workloads in the mesh have granular access control.

Costs

In this document, you use the following billable components of Google Cloud:

  • GKE.
  • GKE Enterprise. The billing for GKE Enterprise includes billing for the Cloud Service Mesh, Config Sync, and Policy Controller.

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Go to project selector

  2. Make sure that billing is enabled for your Google Cloud project.

Prepare your environment

In this section, you prepare your environment so that you can install Cloud Service Mesh, Policy Controller, and Config Sync:

  1. In the Google Cloud console, activate Cloud Shell.

    Activate Cloud Shell

  2. Upgrade to the latest version of the Google Cloud CLI:

    gcloud components update
    
  3. To store the files that you create in this tutorial, create a directory:

    mkdir ~/asm-acm-tutorial-dir
    
  4. To simplify the remainder of the tutorial, create the following environment variables:

    PROJECT_ID=PROJECT_ID
    gcloud config set project $PROJECT_ID
    CLUSTER=asm-acm-tutorial
    CLUSTER_ZONE=us-east4-a
    MEMBERSHIP=asm-acm-tutorial
    PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format='get(projectNumber)')
    

    Replace PROJECT_ID with the project ID that you want to use for this tutorial.

    If you're asked to authorize Cloud Shell, click Authorize to complete the operation.

  5. Enable the APIs that you need for this tutorial:

    gcloud

    gcloud services enable \
        mesh.googleapis.com \
        anthos.googleapis.com
    

    Config Connector

    This tutorial includes Config Connector resources. You can use these resources to complete the same tasks that you complete in the gcloud tab. To utilize these resources, install Config Connector and apply the resources in the way that works best for your environment.

    Use the following Services manifest:

    apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
    kind: Service
    metadata:
      annotations:
        cnrm.cloud.google.com/deletion-policy: "abandon"
        cnrm.cloud.google.com/disable-dependent-services: "false"
      name: mesh.googleapis.com
    spec:
      resourceID: mesh.googleapis.com
      projectRef:
        external: PROJECT_ID
    ---
    apiVersion: serviceusage.cnrm.cloud.google.com/v1beta1
    kind: Service
    metadata:
      annotations:
        cnrm.cloud.google.com/deletion-policy: "abandon"
        cnrm.cloud.google.com/disable-dependent-services: "false"
      name: anthos.googleapis.com
    spec:
      resourceID: anthos.googleapis.com
      projectRef:
        external: PROJECT_ID
    

    This operation can take over one minute to complete.

Set up a GKE cluster

In this section, you create a GKE cluster and then register it to a fleet. Fleets are a Google Cloud concept for logically organizing clusters and other resources, letting you use and manage multi-cluster capabilities and apply consistent policies across your systems.

The cluster that you create in this section is the cluster that you install Cloud Service Mesh, Policy Controller, and Config Sync on. It's also the cluster where you deploy the Online Boutique sample apps.

To set up your cluster, complete the following steps:

  1. Create a GKE cluster:

    gcloud

    gcloud container clusters create ${CLUSTER} \
        --zone ${CLUSTER_ZONE} \
        --machine-type=e2-standard-4 \
        --num-nodes 4 \
        --workload-pool ${PROJECT_ID}.svc.id.goog \
        --labels mesh_id=proj-${PROJECT_NUMBER}
    

    Config Connector

    Use the following ContainerCluster and ContainerNodePool manifests:

    apiVersion: container.cnrm.cloud.google.com/v1beta1
    kind: ContainerNodePool
    metadata:
      annotations:
        cnrm.cloud.google.com/project-id: PROJECT_ID
      name: asm-acm-tutorial
    spec:
      clusterRef:
        name: asm-acm-tutorial
      location: us-east4-a
      nodeConfig:
        machineType: e2-standard-4
      nodeCount: 4
    ---
    apiVersion: container.cnrm.cloud.google.com/v1beta1
    kind: ContainerCluster
    metadata:
      annotations:
        cnrm.cloud.google.com/project-id: PROJECT_ID
        cnrm.cloud.google.com/remove-default-node-pool: "true"
      labels:
        mesh_id: proj-PROJECT_NUMBER
      name: asm-acm-tutorial
    spec:
      location: us-east4-a
      initialNodeCount: 1
      workloadIdentityConfig:
        workloadPool: PROJECT_ID.svc.id.goog
    

    Replace PROJECT_NUMBER by the value of the PROJECT_NUMBER environment variable retrieved earlier.

    This operation can take over five minutes to complete.

  2. To ensure the successful creation of the GKE cluster, describe its status:

    gcloud container clusters list \
        --zone ${CLUSTER_ZONE} \
        --project ${PROJECT_ID}
    

    The output is similar to the following:

    NAME                LOCATION      MASTER_VERSION   MASTER_IP      MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
    asm-acm-tutorial    us-east4-a    1.23.12-gke.100  35.186.179.30  e2-standard-4  1.23.12-gke.100  3          RUNNING
    
  3. Connect to the GKE cluster:

    gcloud container clusters get-credentials ${CLUSTER} \
        --zone ${CLUSTER_ZONE} \
        --project ${PROJECT_ID}
    
  4. Register your cluster to a fleet:

    gcloud

    gcloud container fleet memberships register ${MEMBERSHIP} \
        --project ${PROJECT_ID} \
        --gke-cluster ${CLUSTER_ZONE}/${CLUSTER} \
        --enable-workload-identity
    

    The output is similar to the following:

    kubeconfig entry generated for asm-acm-tutorial.
    Waiting for membership to be created...done.
    Created a new membership [projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial] for the cluster [asm-acm-tutorial]
    Generating the Connect Agent manifest...
    Deploying the Connect Agent on cluster [asm-acm-tutorial] in namespace [gke-connect]...
    Deployed the Connect Agent on cluster [asm-acm-tutorial] in namespace [gke-connect].
    Finished registering the cluster [asm-acm-tutorial] with the Fleet.
    

    Config Connector

    Use the following GKEHubMembership manifest:

    apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
    kind: GKEHubMembership
    metadata:
      annotations:
        cnrm.cloud.google.com/project-id: PROJECT_ID
      name: asm-acm-tutorial
    spec:
      location: global
      authority:
        issuer: https://container.googleapis.com/v1/projects/PROJECT_ID/locations/us-east4-a/clusters/asm-acm-tutorial
      endpoint:
        gkeCluster:
          resourceRef:
            name: asm-acm-tutorial
    
  5. To ensure the successful registration of the GKE cluster, describe its status:

    gcloud container fleet memberships list
    

    The output is similar to the following:

    NAME              EXTERNAL_ID                           LOCATION
    asm-acm-tutorial  0e12258c-8831-4d81-b5c0-5e7099a468cc  global
    

Explore the repositories

In the following installation section, you apply a manifest acm-config.yaml file. This manifest configures your cluster to sync from the asm-acm-tutorial folder of the sample repository. This folder contains all the configuration files that you need to complete the remainder of the tutorial.

To simplify this tutorial, you use sed commands to update the acm-config.yaml. With the acm-config.yaml file, Config Sync deploys the manifests required for each step of this tutorial. Updating a single file helps you focus on the concepts and the flow of securing your clusters, mesh, and applications without repeatedly manipulating the files and repeatedly running git commands.

To utilize Config Sync's ability to sync multiple repositories, you use the following resources:

  • root-sync, as a RootSync repository, contains all the configs across your cluster including RepoSyncs, Constraints, ClusterRole, RoleBindings, and resources included in some system namespaces such as istio-system.
  • ingress-gateway, as a first RepoSync, contains all the resources needed to deploy an ingress gateway and progressively secure it throughout this tutorial.
  • online-boutique, as a second RepoSync, contains all the resources needed to deploy the Online Boutique apps and progressively secure them throughout this tutorial.

Install Policy Controller, Config Sync, and managed Cloud Service Mesh

Now that you have created and registered your cluster, you can install Config Sync, Policy Controller, and Cloud Service Mesh on your cluster and configure your cluster to sync from the configs of the default RootSync:

  1. Enable the ConfigManagement operator, which manages Config Sync and Policy Controller:

    gcloud

    gcloud beta container fleet config-management enable
    

    Config Connector

    Use the following GKEHubFeature manifest:

    apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
    kind: GKEHubFeature
    metadata:
      name: configmanagement
    spec:
      projectRef:
        external: PROJECT_ID
      location: global
      resourceID: configmanagement
    
  2. Enable Cloud Service Mesh in your fleet.

    gcloud

    gcloud container fleet mesh enable
    

    Config Connector

    Use the following GKEHubFeature manifest:

    apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
    kind: GKEHubFeature
    metadata:
      name: servicemesh
    spec:
      projectRef:
        external: PROJECT_ID
      location: global
      resourceID: servicemesh
    
  3. Enable the Cloud Service Mesh automatic management to let Google apply the recommended configuration of managed Cloud Service Mesh:

    gcloud

    gcloud container fleet mesh update \
        --management automatic \
        --memberships ${MEMBERSHIP}
    

    Config Connector

    Use the following GKEHubFeatureMembership manifest:

    apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
    kind: GKEHubFeatureMembership
    metadata:
      name: servicemesh-membership
    spec:
      projectRef:
        external: PROJECT_ID
      location: global
      membershipRef:
        name: asm-acm-tutorial
      featureRef:
        name: servicemesh
      mesh:
        management: MANAGEMENT_AUTOMATIC
    
  4. Enable Config Sync and Policy Controller:

    gcloud

    Save the following manifest as acm-config.yaml in the ~/asm-acm-tutorial-dir directory:

    applySpecVersion: 1
    spec:
      configSync:
        enabled: true
        policyDir: asm-acm-tutorial/root-sync/init
        secretType: none
        sourceFormat: unstructured
        syncRepo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
        syncBranch: main
      policyController:
        enabled: true
        referentialRulesEnabled: true
        templateLibraryInstalled: true
    

    To learn more about the Google Cloud CLI config fields, see gcloud apply spec fields.

    Apply the file:

    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    Config Connector

    Use the following GKEHubFeatureMembership manifest:

    apiVersion: gkehub.cnrm.cloud.google.com/v1beta1
    kind: GKEHubFeatureMembership
    metadata:
      name: configmanagement-membership
    spec:
      projectRef:
        external: PROJECT_ID
      location: global
      membershipRef:
        name: asm-acm-tutorial
      featureRef:
        name: configmanagement
      configmanagement:
        configSync:
          sourceFormat: unstructured
          git:
            policyDir: asm-acm-tutorial/root-sync/init
            secretType: none
            syncBranch: main
            syncRepo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
        policyController:
          enabled: true
          referentialRulesEnabled: true
          templateLibraryInstalled: true
    

    Policy Controller and Config Sync are installed on your cluster. Next, Config Sync begins syncing all the configs of the default RootSync to your cluster. These configs install and configure the following key components:

    • The RepoSync objects that configure the Online Boutique apps and the ingress gateway are synced:

      apiVersion: configsync.gke.io/v1beta1
      kind: RepoSync
      metadata:
        name: repo-sync
      spec:
        override:
          enableShellInRendering: true
        sourceFormat: unstructured
        git:
          repo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
          revision: HEAD
          branch: main
          dir: asm-acm-tutorial/online-boutique/init
          auth: none
      apiVersion: configsync.gke.io/v1beta1
      kind: RepoSync
      metadata:
        name: repo-sync
      spec:
        override:
          enableShellInRendering: true
        sourceFormat: unstructured
        git:
          repo: https://github.com/GoogleCloudPlatform/anthos-config-management-samples
          revision: HEAD
          branch: main
          dir: asm-acm-tutorial/ingress-gateway/init
          auth: none
    • Since the RepoSync reconcilers need additional permissions to create Istio resources, a ClusterRole and two RoleBinding objects to grant these permissions are also applied to your cluster:

      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        labels:
          rbac.authorization.k8s.io/aggregate-to-edit: "true"
        name: custom:aggregate-to-edit:istio
      rules:
      - apiGroups:
        - "networking.istio.io"
        - "security.istio.io"
        resources:
        - "virtualservices"
        - "authorizationpolicies"
        - "gateways"
        verbs:
        - "*"
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: repo-sync
      subjects:
      - kind: ServiceAccount
        name: ns-reconciler-onlineboutique
        namespace: config-management-system
      roleRef:
        kind: ClusterRole
        name: edit
        apiGroup: rbac.authorization.k8s.io
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: repo-sync
      subjects:
      - kind: ServiceAccount
        name: ns-reconciler-asm-ingress
        namespace: config-management-system
      roleRef:
        kind: ClusterRole
        name: edit
        apiGroup: rbac.authorization.k8s.io
  5. To ensure the successful installation of Policy Controller and Config Sync, check the status:

    gcloud beta container fleet config-management status
    

    The output is similar to the following:

    Name: asm-acm-tutorial
    Status: SYNCED
    Last_Synced_Token: 4b3384d
    Sync_Branch: main
    Last_Synced_Time: 2022-05-04T21:32:58Z
    Policy_Controller: INSTALLED
    Hierarchy_Controller: PENDING
    

    If you see PENDING or NOT_INSTALLED in the Status or Policy_Controller rows, wait a few minutes and run gcloud beta container fleet config-management status again.

  6. To ensure the successful installation of Cloud Service Mesh, describe its status:

    gcloud container fleet mesh describe
    

    The output is similar to the following:

    createTime: '2022-09-13T23:12:56.477042921Z'
    membershipSpecs:
      projects/PROJECT_NUMBER/locations/global/memberships/asm-acm-tutorial:
        mesh:
          management: MANAGEMENT_AUTOMATIC
    membershipStates:
      projects/PROJECT_NUMBER/locations/global/memberships/asm-acm-tutorial:
        servicemesh:
          controlPlaneManagement:
            details:
            - code: REVISION_READY
              details: 'Ready: asm-managed'
            state: ACTIVE
          dataPlaneManagement:
            details:
            - code: OK
              details: Service is running.
            state: ACTIVE
        state:
          code: OK
          description: |-
            Revision(s) ready for use: asm-managed.
            All Canonical Services have been reconciled successfully.
          updateTime: '2022-09-14T00:19:10.571552206Z'
    name: projects/PROJECT_ID/locations/global/features/servicemesh
    resourceState:
      state: ACTIVE
    spec: {}
    state:
      state: {}
    updateTime: '2022-09-14T00:19:14.135113118Z'
    

    If you see state.code: ERROR instead of state.code: OK, wait a few minutes and run gcloud container fleet mesh describe again. Before moving forward with the tutorial, you need to make sure that the servicemesh.controlPlaneManagement.details.code field has the REVISION_READY value.

Deploy an ingress gateway and a sample application

In this section, you deploy the Online Boutique sample application and an ingress gateway to manage ingress traffic.

  1. Deploy the Online Boutique sample application and the ingress gateway.

    The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the resources you need to deploy the ingress gateway and sample app.

    sed -i "s,root-sync/init,root-sync/deployments,g" ~/asm-acm-tutorial-dir/acm-config.yaml
    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    Note this step can take a few minutes to complete.

  2. View the Config Sync status for the RootSync and the two RepoSyncs:

    gcloud alpha anthos config sync repo describe
    

    The output is similar to:

    getting 3 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
    [
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "95a30c052566357afb9db3d7f6153d9c0f219c03",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/deployments@main",
        "status": "SYNCED"
      },
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "95a30c052566357afb9db3d7f6153d9c0f219c03",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/ingress-gateway/deployments@main",
        "status": "SYNCED"
      },
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "95a30c052566357afb9db3d7f6153d9c0f219c03",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/online-boutique/deployments@main",
        "status": "SYNCED"
      }
    ]
    

    If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

    To only view the information of one repository you can use the --sync-name and --sync-namespace flags. To view the managed resources in detail, add the --managed-resources flag. For more information, see View Config Sync status across multiple clusters.

  3. Wait for the ingress gateway's public IP address to be provisioned:

    until kubectl -n asm-ingress get svc asm-ingressgateway -o jsonpath='{.status.loadBalancer}' | grep "ingress"; do : ; done
    
  4. Get the ingress gateway's public IP address:

    EXTERNAL_IP=$(kubectl get svc asm-ingressgateway -n asm-ingress -o jsonpath="{.status.loadBalancer.ingress[*].ip}")
    
  5. Visit the IP address from your browser to verify that the Online Boutique app has successfully deployed:

    echo http://${EXTERNAL_IP}
    

Enforce policies to secure your mesh

In the following sections, you leverage Policy Controller to enforce policies from the Cloud Service Mesh policy bundle by creating constraints.

Enforce sidecar proxies injection

In this section you enforce policies to ensure that all workloads in the mesh have automatic sidecar injection enabled.

  1. To enforce sidecar proxies injection, apply constraints.

    The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

    sed -i "s,root-sync/deployments,root-sync/enforce-sidecar-injection,g" ~/asm-acm-tutorial-dir/acm-config.yaml
    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    The preceding command deploys the following resources:

    • A K8sRequiredLabels Constraint that requires any Namespace in the mesh to contain the specific Cloud Service Mesh sidecar proxy injection label:

      apiVersion: constraints.gatekeeper.sh/v1beta1
      kind: K8sRequiredLabels
      metadata:
        name: namespace-sidecar-injection-label
      spec:
        enforcementAction: deny
        match:
          kinds:
          - apiGroups:
            - ""
            kinds:
            - Namespace
          excludedNamespaces:
          - config-management-monitoring
          - config-management-system
          - default
          - gatekeeper-system
          - gke-connect
          - istio-system
          - kube-node-lease
          - kube-public
          - kube-system
          - resource-group-system
        parameters:
          labels:
          - allowedRegex: enabled
            key: istio-injection
    • An AsmSidecarInjection Constraint that prohibits any Pod in the mesh from bypassing the Istio proxy sidecar injection:

      apiVersion: constraints.gatekeeper.sh/v1beta1
      kind: AsmSidecarInjection
      metadata:
        name: pod-sidecar-injection-annotation
      spec:
        enforcementAction: deny
        match:
          kinds:
          - apiGroups:
            - ""
            kinds:
            - Pod
          excludedNamespaces:
          - kube-system
        parameters:
          strictnessLevel: High
  2. View the Config Sync status for the RootSync:

    gcloud alpha anthos config sync repo describe \
        --sync-name root-sync \
        --sync-namespace config-management-system
    

    The output is similar to:

    getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
    [
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/enforce-sidecar-injection@main",
        "status": "SYNCED"
      }
    ]
    

    If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

  3. Verify the Constraints are created:

    kubectl get constraints
    

    It can take a few minutes for Policy Controller to evaluate these constraints. If you don't see values in the TOTAL-VIOLATIONS column, wait and run kubectl get constraints again.

    The output is similar to:

    NAME                                                                                       ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    podsidecarinjectionannotation.constraints.gatekeeper.sh/pod-sidecar-injection-annotation   deny                 0
    
    NAME                                                                            ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    k8srequiredlabels.constraints.gatekeeper.sh/namespace-sidecar-injection-label   deny                 0
    

    Because we properly set up our Namespaces and Pods, there are 0 TOTAL-VIOLATIONS for these Constraints.

  4. To see these Constraints at work, try to create a Namespace in your cluster with neither a label nor an annotation:

    kubectl create namespace test
    

    The output is similar is the following error:

    Error from server (Forbidden): admission webhook "validation.gatekeeper.sh" denied the request: [namespace-sidecar-injection-label] you must provide labels: {"istio-injection"}
    

Enforce traffic encryption

In this section you enforce policies to ensure that all traffic in the mesh is encrypted.

  1. To enforce traffic encryption, apply constraints.

    The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

    sed -i "s,root-sync/enforce-sidecar-injection,root-sync/enforce-strict-mtls,g" ~/asm-acm-tutorial-dir/acm-config.yaml
    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    The preceding command deploys the following resources:

    • An AsmPeerAuthnMeshStrictMtls Constraint which enforces the mesh-level mTLS PeerAuthentication in the istio-system namespace:

      apiVersion: constraints.gatekeeper.sh/v1beta1
      kind: AsmPeerAuthnMeshStrictMtls
      metadata:
        name: mesh-level-strict-mtls
      spec:
        enforcementAction: deny
        parameters:
          rootNamespace: istio-system
          strictnessLevel: High
    • A referential constraint Config in the gatekeeper-system namespace. This referential constraint enables the AsmPeerAuthnMeshStrictMtls Constraint to reference another object in its definition (for example, searching for any PeerAuthentication in the istio-system Namespace):

      apiVersion: config.gatekeeper.sh/v1alpha1
      kind: Config
      metadata:
        name: config
      spec:
        sync:
          syncOnly:
            - group: ""
              version: "v1"
              kind: "Namespace"
            - group: "security.istio.io"
              version: "v1beta1"
              kind: "PeerAuthentication"
            - group: "security.istio.io"
              version: "v1beta1"
              kind: "AuthorizationPolicy"
    • A DestinationRuleTLSEnabled Constraint which prohibits disabling TLS for all hosts and host subsets in Istio DestinationRules:

      apiVersion: constraints.gatekeeper.sh/v1beta1
      kind: DestinationRuleTLSEnabled
      metadata:
        name: destination-rule-tls-enabled
      spec:
        enforcementAction: deny
        match:
          kinds:
          - apiGroups:
            - networking.istio.io
            kinds:
            - DestinationRule
    • An AsmPeerAuthnStrictMtls Constraint which enforces that all PeerAuthentications cannot overwrite STRICT mTLS:

      apiVersion: constraints.gatekeeper.sh/v1beta1
      kind: AsmPeerAuthnStrictMtls
      metadata:
        name: peerauthentication-strict-mtls
      spec:
        enforcementAction: deny
        match:
          kinds:
          - apiGroups:
            - security.istio.io
            kinds:
            - PeerAuthentication
        parameters:
          strictnessLevel: High
  2. View the Config Sync status for the RootSync:

    gcloud alpha anthos config sync repo describe \
        --sync-name root-sync \
        --sync-namespace config-management-system
    

    The output is similar to:

    getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
    [
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/enforce-strict-mtls@main",
        "status": "SYNCED"
      }
    ]
    

    If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

  3. Run the following command to get more information about the PeerAuthentication violation:

    kubectl get asmpeerauthnmeshstrictmtls.constraints.gatekeeper.sh/mesh-level-strict-mtls -ojsonpath='{.status.violations}'  | jq
    

    The output is similar to:

    [
      {
        "enforcementAction": "deny",
        "group": "constraints.gatekeeper.sh",
        "kind": "AsmPeerAuthnMeshStrictMtls",
        "message": "Root namespace <istio-system> does not have a strict mTLS PeerAuthentication",
        "name": "mesh-level-strict-mtls",
        "version": "v1beta1"
      }
    ]
    
  4. Fix the issue by deploying a PeerAuthentication in the istio-system. To prevent all your services in the mesh from accepting plaintext traffic, set a mesh-wide PeerAuthentication policy with the mTLS mode set to STRICT. When you deploy the policy, the control plane automatically provisions TLS certificates so that workloads can authenticate with each other.

    The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

    sed -i "s,root-sync/enforce-strict-mtls,root-sync/fix-strict-mtls,g" ~/asm-acm-tutorial-dir/acm-config.yaml
    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    The preceding command deploys the following STRICT mTLS PeerAuthentication in the istio-system namespace. This applies mTLS STRICT to the entire mesh:

    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: default
    spec:
      mtls:
        mode: STRICT
  5. View the Config Sync status for the RootSync:

    gcloud alpha anthos config sync repo describe \
        --sync-name root-sync \
        --sync-namespace config-management-system
    

    The output is similar to:

    getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
    [
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/fix-strict-mtls@main",
        "status": "SYNCED"
      }
    ]
    

    If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

  6. Verify the Constraints are created:

    kubectl get constraints
    

    Note this can take a few minutes to get Policy Controller evaluating these Constraints. Wait and run again this kubectl get constraints command until you get values under the TOTAL-VIOLATIONS column for each line.

    The output is similar to:

    NAME                                                                            ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    k8srequiredlabels.constraints.gatekeeper.sh/namespace-sidecar-injection-label   deny                 0
    NAME                                                                          ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmpeerauthnmeshstrictmtls.constraints.gatekeeper.sh/mesh-level-strict-mtls   deny                 0
    NAME                                                                               ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    destinationruletlsenabled.constraints.gatekeeper.sh/destination-rule-tls-enabled   deny                 0
    NAME                                                                              ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmpeerauthnstrictmtls.constraints.gatekeeper.sh/peerauthentication-strict-mtls   deny                 0
    NAME                                                                             ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmsidecarinjection.constraints.gatekeeper.sh/pod-sidecar-injection-annotation   deny                 0
    

Enforce granular access control

In this section you enforce policies to ensure that all workloads in the mesh have granular access control.

  1. To enforce granular access control, apply constraints.

    The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

    sed -i "s,root-sync/fix-strict-mtls,root-sync/enforce-authorization-policies,g" ~/asm-acm-tutorial-dir/acm-config.yaml
    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    The preceding command deploys the following resources:

    • An AsmAuthzPolicyDefaultDeny Constraint which enforces the mesh-level default deny AuthorizationPolicy in the istio-system namespace:

      apiVersion: constraints.gatekeeper.sh/v1beta1
      kind: AsmAuthzPolicyDefaultDeny
      metadata:
        name: default-deny-authorization-policies
      spec:
        enforcementAction: deny
        parameters:
          rootNamespace: istio-system
          strictnessLevel: High
    • An AsmAuthzPolicyEnforceSourcePrincipals Constraint which enforces that any AuthorizationPolicies is defining granular source principals (other than "*"). Only the ingress gateway in the asm-ingress namespace is an exception to this rule in order to receive the traffic from the end-users and redirect the traffic to the Online Boutique's frontend app.

      apiVersion: constraints.gatekeeper.sh/v1beta1
      kind: AsmAuthzPolicyEnforceSourcePrincipals
      metadata:
        name: authz-source-principals-not-all
      spec:
        enforcementAction: deny
        match:
          kinds:
          - apiGroups:
            - security.istio.io
            kinds:
            - AuthorizationPolicy
          excludedNamespaces:
            - asm-ingress
  2. View the Config Sync status for the RootSync:

    gcloud alpha anthos config sync repo describe \
        --sync-name root-sync \
        --sync-namespace config-management-system
    

    The output is similar to:

    getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
    [
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/enforce-authorization-policies@main",
        "status": "SYNCED"
      }
    ]
    

    If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

  3. Run the following command to get more information about the associated violation:

    kubectl get asmauthzpolicydefaultdeny.constraints.gatekeeper.sh/default-deny-authorization-policies -ojsonpath='{.status.violations}'  | jq
    

    The output is similar to:

    [
      {
        "enforcementAction": "deny",
        "group": "constraints.gatekeeper.sh",
        "kind": "AsmAuthzPolicyDefaultDeny",
        "message": "Root namespace <istio-system> does not have a default deny AuthorizationPolicy",
        "name": "default-deny-authorization-policies",
        "version": "v1beta1"
      }
    ]
    
  4. Fix the issue by deploying the AuthorizationPolicy in the istio-system namespace.

    The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

    sed -i "s,root-sync/enforce-authorization-policies,root-sync/fix-default-deny-authorization-policy,g" ~/asm-acm-tutorial-dir/acm-config.yaml
    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    The preceding command deploys the following deny-all AuthorizationPolicy in the istio-system namespace:

    apiVersion: security.istio.io/v1beta1
    kind: AuthorizationPolicy
    metadata:
      name: deny-all
    spec:
      {}
  5. View the Config Sync status for the RootSync:

    gcloud alpha anthos config sync repo describe \
        --sync-name root-sync \
        --sync-namespace config-management-system
    

    The output is similar to:

    getting 1 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
    [
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/fix-default-deny-authorization-policy@main",
        "status": "SYNCED"
      }
    ]
    

    If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

  6. Verify the Constraints are created:

    kubectl get constraints
    

    Note this can take a few minutes to get Policy Controller evaluating these Constraints. Wait and run again this kubectl get constraints command until you get values under the TOTAL-VIOLATIONS column for each line.

    The output is similar to:

    NAME                                                                             ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmsidecarinjection.constraints.gatekeeper.sh/pod-sidecar-injection-annotation   deny                 0
    NAME                                                                            ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    k8srequiredlabels.constraints.gatekeeper.sh/namespace-sidecar-injection-label   deny                 0
    NAME                                                                                      ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmauthzpolicydefaultdeny.constraints.gatekeeper.sh/default-deny-authorization-policies   deny                 0
    NAME                                                                          ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmpeerauthnmeshstrictmtls.constraints.gatekeeper.sh/mesh-level-strict-mtls   deny                 0
    NAME                                                                               ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    destinationruletlsenabled.constraints.gatekeeper.sh/destination-rule-tls-enabled   deny                 0
    NAME                                                                              ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmpeerauthnstrictmtls.constraints.gatekeeper.sh/peerauthentication-strict-mtls   deny                 0
    NAME                                                                                              ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
    asmauthzpolicyenforcesourceprincipals.constraints.gatekeeper.sh/authz-source-principals-not-all   deny                 0
    
  7. Visit the Online Boutique app from your browser:

    echo http://${EXTERNAL_IP}
    

    You should receive the error: RBAC: access denied which confirms that the default deny AuthorizationPolicy is applied to the entire mesh.

  8. Fix this issue by deploying more granular AuthorizationPolicies in the asm-ingress and onlineboutique namespaces.

    The following command uses sed to update the acm-config.yaml manifest to get Config Sync deploying the associated resources.

    sed -i "s,root-sync/fix-default-deny-authorization-policy,root-sync/deploy-authorization-policies,g" ~/asm-acm-tutorial-dir/acm-config.yaml
    gcloud beta container fleet config-management apply \
        --membership ${MEMBERSHIP} \
        --config ~/asm-acm-tutorial-dir/acm-config.yaml
    

    The preceding command deploys the following resources:

    • An AuthorizationPolicy in the asm-ingress namespace:

      apiVersion: security.istio.io/v1beta1
      kind: AuthorizationPolicy
      metadata:
        name: asm-ingressgateway
      spec:
        selector:
          matchLabels:
            asm: ingressgateway
        rules:
        - to:
          - operation:
              ports:
              - "8080"
    • An AuthorizationPolicy per app in the onlineboutique namespace, here is the example for the cartservice app:

      apiVersion: security.istio.io/v1beta1
      kind: AuthorizationPolicy
      metadata:
        name: cartservice
      spec:
        selector:
          matchLabels:
            app: cartservice
        rules:
        - from:
          - source:
              principals:
              - cluster.local/ns/onlineboutique/sa/frontend
              - cluster.local/ns/onlineboutique/sa/checkoutservice
          to:
          - operation:
              paths:
              - /hipstershop.CartService/AddItem
              - /hipstershop.CartService/GetCart
              - /hipstershop.CartService/EmptyCart
              methods:
              - POST
              ports:
              - "7070"
    • A ServiceAccount per app in the asm-ingress and onlineboutique namespaces in order to have a unique identity per app evaluated as principal in the AuthorizationPolicies. Here is the example for the cartservice app:

      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: cartservice
  9. View the Config Sync status for the RootSync and the two RepoSyncs:

    gcloud alpha anthos config sync repo describe
    

    The output is similar to:

    getting 3 RepoSync and RootSync from projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial
    [
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/root-sync/deploy-authorization-policies@main",
        "status": "SYNCED"
      },
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/ingress-gateway/authorization-policies@main",
        "status": "SYNCED"
      },
      {
        "clusters": [
          "projects/PROJECT_ID/locations/global/memberships/asm-acm-tutorial"
        ],
        "commit": "7d15d49af13c44aa531a4565b2277ddcf8b81884",
        "errors": [],
        "source": "https://github.com/GoogleCloudPlatform/anthos-config-management-samples//asm-acm-tutorial/online-boutique/authorization-policies@main",
        "status": "SYNCED"
      }
    ]
    

    If you see status: RECONCILING instead of status: SYNCED, wait a few minutes and run gcloud alpha anthos config sync repo describe again.

    To only view the information of one repository you can use the --sync-name and --sync-namespace flags. And to view in detail the managed resources, you can add the --managed-resources flag. For more information, see View Config Sync status across multiple clusters.

  10. Visit the Online Boutique app again from your browser:

    echo http://${EXTERNAL_IP}
    

    If you wait a few minutes, you should now see the website working successfully again as expected.

View the status of GKE Enterprise security features

You can view the status of GKE Enterprise security features, including authentication and authorization policies, in the Google Cloud console.

  1. In the Google Cloud console, go to the GKE Enterprise Security page.

    Go to GKE Enterprise Security

    The Policy Summary displays the status of application security, including Service access control (AuthorizationPolicies) and mTLS.

  2. Click Policy Audit to view workload policy statuses for the cluster and both namespaces (asm-ingress and onlineboutique).

    The Service access control and mTLS status cards provide a high-level overview.

    High-level overview of service access control and mTLS status

    The Workloads list shows the Service access control and mTLS status of each workload.

    Detailed list of each workload and its service access control and mTLS status

You now have secured your cluster and your mesh with Policy Controller and Config Sync.

Clean up

To avoid incurring charges to your Google Cloud account for the resources used in this tutorial, either delete the project that contains the resources, or keep the project and delete the individual resources.

Delete the project

    Delete a Google Cloud project:

    gcloud projects delete PROJECT_ID

Delete individual resources

To delete the individual resources:

  1. Unregister your cluster from the fleet:

    gcloud

    gcloud container fleet memberships unregister ${CLUSTER} \
        --project=${PROJECT_ID} \
        --gke-cluster=${CLUSTER_ZONE}/${CLUSTER}
    

    The output is similar to the following:

    kubeconfig entry generated for asm-acm-tutorial.
    Waiting for membership to be deleted...done.
    Deleting membership CR in the cluster...done.
    Deleting namespace [gke-connect] in the cluster...done.
    

    Config Connector

    kubectl delete -f ~/asm-acm-tutorial-dir/fleet-membership.yaml
    
  2. Delete your cluster:

    gcloud

    gcloud container clusters delete ${CLUSTER} \
        --zone ${CLUSTER_ZONE}
    

    Press y when prompted. This command can take over five minutes to complete.

    The output is similar to the following:

    Deleting cluster asm-acm-tutorial...done.
    Deleted [https://container.googleapis.com/v1/projects/PROJECT_ID/zones/us-east4-a/clusters/asm-acm-tutorial].
    

    Config Connector

    kubectl delete -f ~/asm-acm-tutorial-dir/container-cluster.yaml
    
  3. Delete the files that you created:

    rm -r ~/asm-acm-tutorial-dir
    

What's next