Implementing Binary Authorization using Cloud Build and GKE

This tutorial show you how to set up, configure, and use Binary Authorization for Google Kubernetes Engine (GKE). Binary authorization is the process of creating attestations on container images for the purpose of verifying that certain criteria are met before you can deploy the images to GKE.

For example, Binary Authorization can verify that an app passed its unit tests or that an app was built using a specific set of systems. For more information and use cases, see Help secure software supply chains on Google Kubernetes Engine.

This tutorial is intended for practitioners who want to better understand container vulnerability scanning and binary authorization, as well as their implementation and application in a CI/CD pipeline.

This tutorial assumes you are familiar with the following topics and technologies:

  • Continuous integration and continuous deployment
  • Common vulnerabilities and exposures (CVE) vulnerability scanning
  • GKE
  • Container Registry
  • Cloud Build
  • Cloud Key Management Service (Cloud KMS)

Objectives

  • Deploy GKE clusters for staging and production.
  • Create multiple attestors and attestations.
  • Deploy a CI/CD pipeline using Cloud Build.
  • Test deployment pipeline.
  • Develop a break-glass process.

Costs

This tutorial uses the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator. New Google Cloud users might be eligible for a free trial.

When you finish this tutorial, you can avoid continued billing by deleting the resources you created. For more information, see Cleaning up.

Before you begin

  1. Sign in to your Google Account.

    If you don't already have one, sign up for a new account.

  2. In the Cloud Console, on the project selector page, select or create a Cloud project.

    Go to the project selector page

  3. Make sure that billing is enabled for your Google Cloud project. Learn how to confirm billing is enabled for your project.

  4. Enable the Binary Authorization, Cloud Build, Cloud KMS, GKE, Container Registry, Container Analysis, Resource Manager, and Cloud Source Repositories APIs.

    Enable the APIs

  5. In the Cloud Console, activate Cloud Shell.

    Activate Cloud Shell

    At the bottom of the Cloud Console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

  6. All commands in this tutorial are run in Cloud Shell.

Architecture of the CI/CD pipeline

An important aspect of software development lifecycle (SDLC) is ensuring and enforcing that app deployments follow your organization's approved processes. One method for establishing these checks and balances is with Binary Authorization on GKE. First, Binary Authorization attaches notes to container images. Then, GKE verifies that the required notes are present before you can deploy the app.

These notes, or attestations, make statements about the image. Attestations are completely configurable, but here are a few common examples:

  • The app passed unit tests.
  • The app was verified by the Quality Assurance (QA) team.
  • The app was scanned for vulnerabilities and none were found.

The following diagram depicts a SDLC where a single attestation is applied after vulnerability scanning finishes with no known vulnerabilities.

Architecture of SDLC with a single attestation appplied

In this tutorial, you create a CI/CD pipeline using Cloud Source Repositories, Cloud Build, Container Registry, and GKE. The following diagram illustrates the CI/CD pipeline.

Architecture of CI/CD pipeline with three Google Cloud products

This CI/CD pipeline consists of the following steps:

  1. Builds a container image with app source code.

  2. Pushes the container image into Container Registry.

  3. Container Analysis, scans the container image for known security vulnerabilities or CVEs.

If the image contains no CVEs with a severity score greater than five, the image is attested as having no critical CVEs and is automatically deployed to staging. A score greater than five indicates mid-range medium to critical vulnerability, and thus is not attested or deployed.

A QA team inspects the app in the staging cluster. If it passes their requirements, they apply a manual attestation that the container image is of sufficient quality for deployment to production. The production manifests are updated and the app is deployed to the production GKE cluster.

The GKE clusters are configured to examine the container images for attestations and reject any deployments that don't have the required attestations. The staging GKE cluster only requires the vulnerability scan attestation, but the production GKE cluster requires both a vulnerability scan and QA attestation.

In this tutorial, you introduce failures into the CI/CD pipeline to test and verify this enforcement. Finally, you implement a break-glass procedure for bypassing these deployment checks in GKE in case of an emergency.

Setting up your environment

This tutorial uses the following environment variables. You can change these values to match your requirements, but all steps in the tutorial assume these environment variables exist and contain a valid value.

  1. In Cloud Shell, set the Cloud project where you deploy and manage all resources used in this tutorial:

    export PROJECT_ID="${DEVSHELL_PROJECT_ID}"
    
  2. Set the region in which you deploy these resources:

    export REGION="us-central1"
    

    The GKE cluster and the Cloud KMS keys reside in this region. In this tutorial, the region is us-central1. For more information about regions, see Geography and regions.

  3. Set the Cloud Build project number:

    export PROJECT_NUMBER="$(gcloud projects describe "${PROJECT_ID}" \
      --format='value(projectNumber)')"
    
  4. Set the Cloud Build service account email:

    export CLOUD_BUILD_SA_EMAIL="${PROJECT_NUMBER}@cloudbuild.gserviceaccount.com"
    

Creating GKE clusters

Create two GKE clusters and grant Identity and Access Management (IAM) permissions for Cloud Build to deploy apps to GKE. Creating GKE clusters can take a few minutes.

  1. In Cloud Shell, create a GKE cluster for staging:

    gcloud container clusters create "staging-cluster" \
      --project "${PROJECT_ID}" \
      --machine-type "n1-standard-1" \
      --region "${REGION}" \
      --num-nodes "1" \
      --enable-binauthz
    
  2. Create a GKE cluster for production:

    gcloud container clusters create "prod-cluster" \
      --project "${PROJECT_ID}" \
      --machine-type "n1-standard-1" \
      --region "${REGION}" \
      --num-nodes "1" \
      --enable-binauthz
    
  3. Grant the Cloud Build service account permission to deploy to GKE:

    gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
      --member "serviceAccount:${CLOUD_BUILD_SA_EMAIL}" \
      --role "roles/container.developer"
    

Creating signing keys

Create two Cloud KMS asymmetric keys for signing attestations.

  1. In Cloud Shell, create a Cloud KMS key ring named binauthz:

    gcloud kms keyrings create "binauthz" \
      --project "${PROJECT_ID}" \
      --location "${REGION}"
    
  2. Create an asymmetric Cloud KMS key named vulnz-signer which will be used to sign and verify vulnerability scan attestations:

    gcloud kms keys create "vulnz-signer" \
      --project "${PROJECT_ID}" \
      --location "${REGION}" \
      --keyring "binauthz" \
      --purpose "asymmetric-signing" \
      --default-algorithm "rsa-sign-pkcs1-4096-sha512"
    
  3. Create an asymmetric Cloud KMS key named qa-signer to sign and verify QA attestations:

    gcloud kms keys create "qa-signer" \
      --project "${PROJECT_ID}" \
      --location "${REGION}" \
      --keyring "binauthz" \
      --purpose "asymmetric-signing" \
      --default-algorithm "rsa-sign-pkcs1-4096-sha512"
    

Configuring attestations

You create the notes that are attached to container images, grant permissions to the Cloud Build service account to view notes, attach notes, and create the attestors using the keys from the previous steps.

Create the vulnerability scanner attestation

  1. In Cloud Shell, create a Container Analysis note named vulnz-note:

    curl "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/?noteId=vulnz-note" \
      --request "POST" \
      --header "Content-Type: application/json" \
      --header "Authorization: Bearer $(gcloud auth print-access-token)" \
      --header "X-Goog-User-Project: ${PROJECT_ID}" \
      --data-binary @- <<EOF
        {
          "name": "projects/${PROJECT_ID}/notes/vulnz-note",
          "attestation": {
            "hint": {
              "human_readable_name": "Vulnerability scan note"
            }
          }
        }
    EOF
    
  2. Grant the Cloud Build service account permission to view and attach the vulnz-note note to container images:

    curl "https://containeranalysis.googleapis.com/v1beta1/projects/${PROJECT_ID}/notes/vulnz-note:setIamPolicy" \
      --request POST \
      --header "Content-Type: application/json" \
      --header "Authorization: Bearer $(gcloud auth print-access-token)" \
      --header "X-Goog-User-Project: ${PROJECT_ID}" \
      --data-binary @- <<EOF
        {
          "resource": "projects/${PROJECT_ID}/notes/vulnz-note",
          "policy": {
            "bindings": [
              {
                "role": "roles/containeranalysis.notes.occurrences.viewer",
                "members": [
                  "serviceAccount:${CLOUD_BUILD_SA_EMAIL}"
                ]
              },
              {
                "role": "roles/containeranalysis.notes.attacher",
                "members": [
                  "serviceAccount:${CLOUD_BUILD_SA_EMAIL}"
                ]
              }
            ]
          }
        }
    EOF
    
  3. Create the vulnerability scan attestor:

    gcloud container binauthz attestors create "vulnz-attestor" \
      --project "${PROJECT_ID}" \
      --attestation-authority-note-project "${PROJECT_ID}" \
      --attestation-authority-note "vulnz-note" \
      --description "Vulnerability scan attestor"
    
  4. Add the public key for the attestor's signing key:

    gcloud beta container binauthz attestors public-keys add \
      --project "${PROJECT_ID}" \
      --attestor "vulnz-attestor" \
      --keyversion "1" \
      --keyversion-key "vulnz-signer" \
      --keyversion-keyring "binauthz" \
      --keyversion-location "${REGION}" \
      --keyversion-project "${PROJECT_ID}"
    
  5. Grant the Cloud Build service account permission to verify attestations made by vulnz-attestor:

    gcloud container binauthz attestors add-iam-policy-binding "vulnz-attestor" \
      --project "${PROJECT_ID}" \
      --member "serviceAccount:${CLOUD_BUILD_SA_EMAIL}" \
      --role "roles/binaryauthorization.attestorsVerifier"
    
  6. Grant the Cloud Build service account permission to sign objects using the vulnz-signer key:

    gcloud kms keys add-iam-policy-binding "vulnz-signer" \
      --project "${PROJECT_ID}" \
      --location "${REGION}" \
      --keyring "binauthz" \
      --member "serviceAccount:${CLOUD_BUILD_SA_EMAIL}" \
      --role 'roles/cloudkms.signerVerifier'
    

Create the QA attestation

  1. In Cloud Shell, create a Container Analysis note named qa-note:

    curl "https://containeranalysis.googleapis.com/v1/projects/${PROJECT_ID}/notes/?noteId=qa-note" \
      --request "POST" \
      --header "Content-Type: application/json" \
      --header "Authorization: Bearer $(gcloud auth print-access-token)" \
      --header "X-Goog-User-Project: ${PROJECT_ID}" \
      --data-binary @- <<EOF
        {
          "name": "projects/${PROJECT_ID}/notes/qa-note",
          "attestation": {
            "hint": {
              "human_readable_name": "QA note"
            }
          }
        }
    EOF
    
  2. Grant the Cloud Build service account permission to view and attach the qa-note note to container images:

    curl "https://containeranalysis.googleapis.com/v1beta1/projects/${PROJECT_ID}/notes/qa-note:setIamPolicy" \
      --request POST \
      --header "Content-Type: application/json" \
      --header "Authorization: Bearer $(gcloud auth print-access-token)" \
      --header "X-Goog-User-Project: ${PROJECT_ID}" \
      --data-binary @- <<EOF
        {
          "resource": "projects/${PROJECT_ID}/notes/qa-note",
          "policy": {
            "bindings": [
              {
                "role": "roles/containeranalysis.notes.occurrences.viewer",
                "members": [
                  "serviceAccount:${CLOUD_BUILD_SA_EMAIL}"
                ]
              },
              {
                "role": "roles/containeranalysis.notes.attacher",
                "members": [
                  "serviceAccount:${CLOUD_BUILD_SA_EMAIL}"
                ]
              }
            ]
          }
        }
    EOF
    
  3. Create the QA attestor:

    gcloud container binauthz attestors create "qa-attestor" \
      --project "${PROJECT_ID}" \
      --attestation-authority-note-project "${PROJECT_ID}" \
      --attestation-authority-note "qa-note" \
      --description "QA attestor"
    
  4. Add the public key for the attestor's signing key:

    gcloud beta container binauthz attestors public-keys add \
      --project "${PROJECT_ID}" \
      --attestor "qa-attestor" \
      --keyversion "1" \
      --keyversion-key "qa-signer" \
      --keyversion-keyring "binauthz" \
      --keyversion-location "${REGION}" \
      --keyversion-project "${PROJECT_ID}"
    
  5. Grant the Cloud Build service account permission to verify attestations made by qa-attestor:

    gcloud container binauthz attestors add-iam-policy-binding "qa-attestor" \
      --project "${PROJECT_ID}" \
      --member "serviceAccount:${CLOUD_BUILD_SA_EMAIL}" \
      --role "roles/binaryauthorization.attestorsVerifier"
    
  6. Grant your QA team permission to sign attestations:

    gcloud kms keys add-iam-policy-binding "qa-signer" \
      --project "${PROJECT_ID}" \
      --location "${REGION}" \
      --keyring "binauthz" \
      --member "group:qa-team@example.com" \
      --role 'roles/cloudkms.signerVerifier'
    

Setting the Binary Authorization policy

Even though you created the GKE clusters with --enable-binauthz, you must author a policy that instructs GKE on what attestations the binaries require to run in the cluster. Binary Authorization policies exist at the project level, but contain configuration for the cluster level.

The following policy alters the default policy in the following ways:

  • Changes the default evaluationMode to ALWAYS_DENY. Only exempted images or images with the required attestations are admitted to run in the cluster.

  • Enables globalPolicyEvaluationMode, which changes the default whitelist to only include system images provided by Google.

  • Defines the following cluster admission rules:

    • staging-cluster requires attestations from vulnz-attestor.

    • prod-cluster requires attestations from vulnz-attestor and qa-attestor.

For more information on Binary Authorization policies, see Policy YAML reference.

  1. In Cloud Shell, create a YAML file that describes the Binary Authorization policy for the Cloud project:

    cat > ./binauthz-policy.yaml <<EOF
    admissionWhitelistPatterns:
    - namePattern: docker.io/istio/*
    defaultAdmissionRule:
      enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
      evaluationMode: ALWAYS_DENY
    globalPolicyEvaluationMode: ENABLE
    clusterAdmissionRules:
      # Staging cluster
      ${REGION}.staging-cluster:
        evaluationMode: REQUIRE_ATTESTATION
        enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
        requireAttestationsBy:
        - projects/${PROJECT_ID}/attestors/vulnz-attestor
    
      # Production cluster
      ${REGION}.prod-cluster:
        evaluationMode: REQUIRE_ATTESTATION
        enforcementMode: ENFORCED_BLOCK_AND_AUDIT_LOG
        requireAttestationsBy:
        - projects/${PROJECT_ID}/attestors/vulnz-attestor
        - projects/${PROJECT_ID}/attestors/qa-attestor
    EOF
    
  2. Upload the new policy to the Cloud project:

    gcloud container binauthz policy import ./binauthz-policy.yaml \
      --project "${PROJECT_ID}"
    

Creating the vulnerability scan checker and enabling the API

Create a container image that is used as a build step in Cloud Build. This container compares the severity scores of any detected vulnerabilities against the configured threshold. If the score is within the threshold, Cloud Build creates an attestation on the container. If the score is outside of the threshold, the build fails, and no attestation is created.

  1. In Cloud Shell, clone the binary authorization tools and sample application source:

    git clone https://github.com/GoogleCloudPlatform/gke-binary-auth-tools ~/binauthz-tools
    
  2. Build and push the vulnerability scan attestor container named cloudbuild-attestor to Container Registry:

    gcloud builds submit \
      --project "${PROJECT_ID}" \
      --tag "gcr.io/${PROJECT_ID}/cloudbuild-attestor" \
      ~/binauthz-tools
    
  3. Enable the vulnerability scanner API:

    gcloud services enable containerscanning.googleapis.com
    

Setting up the Cloud Build pipeline

Create a Cloud Source Repositories repository and Cloud Build trigger for the sample app and Kubernetes manifests.

Create the hello-app Cloud Source Repositories

  1. In Cloud Shell, create a Cloud Source Repositories repository for the sample app:

    gcloud source repos create hello-app \
      --project "${PROJECT_ID}"
    
  2. Clone the repository locally:

    gcloud source repos clone hello-app ~/hello-app \
      --project "${PROJECT_ID}"
    
  3. Copy the sample code into the repository:

    cp -R ~/binauthz-tools/examples/hello-app/* ~/hello-app
    

Create the hello-app Cloud Build trigger

  1. In the Cloud Console, go to the Triggers page.

    Go to Triggers

  2. Click Create Trigger.

  3. In the Trigger Settings window, enter the following details:

    • In the Repository field, choose hello-app from the menu.
    • In the Name field, enter build-vulnz-deploy.
    • For Trigger type choose Branch.
    • In the Branch (regex) field, enter master.
    • For Build Configuration, select Cloud Build configuration file.
    • In the Cloud Build configuration file location, enter the default value /cloudbuild.yaml.
  4. Add the following Substitution Variable pairs:

    • _COMPUTE_REGION with the value us-central1 (or the region you chose in the beginning).
    • _KMS_KEYRING with the value binauthz.
    • _KMS_LOCATION with the value us-central1 (or the region you chose in the beginning).
    • _PROD_CLUSTER with the value prod-cluster.
    • _QA_ATTESTOR with the value qa-attestor.
    • _QA_KMS_KEY with the value qa-signer.
    • _QA_KMS_KEY_VERSION with the value 1.
    • _STAGING_CLUSTER with the value staging-cluster.
    • _VULNZ_ATTESTOR with the value vulnz-attestor.
    • _VULNZ_KMS_KEY with the value vulnz-signer.
    • _VULNZ_KMS_KEY_VERSION with the value 1.
  5. Click Create Trigger.

Test the Cloud Build pipeline

Test the CI/CD pipeline by committing and pushing the sample app to the Cloud Source Repositories repository. Cloud Build detects the change, builds, and deploys the app to staging-cluster. The pipeline waits for up to 10 minutes for QA verification. After the deployment is verified by the QA team, the process continues and the production Kubernetes manifests are updated and Cloud Build deploys the app to prod-cluster.

  1. In Cloud Shell, commit and push the hello-app files to the Cloud Source Repositories repository to trigger a build:

    cd ~/hello-app
    
    git add .
    git commit -m "Initial commit"
    git push origin master
    
  2. In the Cloud Console, go to the History page.

    Go to History page

  3. To watch the build progress, click the most recent run of Cloud Build.

    Build information

  4. When the deployment to staging-cluster is finished, go to the Services page.

    Go to Services page

  5. To verify that the app is working, click the Endpoints link for the app.

  6. Go to the Images page.

    Go to Images page

  7. Click hello-app.

  8. Click the image that you validated in the staging deployment.

    Name of validated image

  9. On the Digest Details page, copy the Digest value from the image details. This information is needed in the next step

    Digest value of the image

  10. To apply the manual QA attestation, replace ... with the value you copied from the image details. The DIGEST variable should be in the format sha256:hash-value`.

    The Await QA attestation build step will also output a copy-pasteable command, shown below.

    DIGEST="sha256:..." # Replace with your value
    
    gcloud beta container binauthz attestations sign-and-create \
      --project "${PROJECT_ID}" \
      --artifact-url "gcr.io/${PROJECT_ID}/hello-app@${DIGEST}" \
      --attestor "qa-attestor" \
      --attestor-project "${PROJECT_ID}" \
      --keyversion "1" \
      --keyversion-key "qa-signer" \
      --keyversion-location "${REGION}" \
      --keyversion-keyring "binauthz" \
      --keyversion-project "${PROJECT_ID}"
    
  11. To verify the app was deployed, go to the Services page.

    Go to Services page

  12. To view the app, click the endpoint link.

Deploying an unattested image

So far, the sample app didn't have any vulnerabilities. Update the app to output a different message and change the base image.

  1. In Cloud Shell, change the output from Hello World to Binary Authorization, and change the base image from distroless to debian:

    cd ~/hello-app
    sed -i "s/Hello World/Binary Authorization/g" main.go
    sed -i "s/FROM gcr\.io\/distroless\/static/FROM debian/g" Dockerfile
    
  2. Commit and push the changes:

    git add .
    git commit -m "Change message and base image"
    git push origin master
    
  3. To monitor the status of the CI/CD pipeline, in the Cloud Console, go to the History page.

    Go to History page

    This build fails because of detected CVEs in the image.

  4. To examine the identified CVEs, go to the Images page.

    Go to Images page

  5. Click hello-app.

  6. To review the identified CVEs, click the vulnerability summary for the most recent image.

    Vulnerability summary link for a recent image.

  7. In Cloud Shell, attempt to deploy the new image to production without the attestation from the vulnerability scan:

    export SHA_DIGEST="[SHA_DIGEST_VALUE]"
    
    cd ~/hello-app
    sed "s/GOOGLE_CLOUD_PROJECT/${DEVSHELL_PROJECT_ID}/g" \
        kubernetes/deployment.yaml.tpl | sed -e  \
        "s/DIGEST/${SHA_DIGEST}/g" > kubernetes/deployment.yaml
    
    gcloud container clusters get-credentials \
        --project=${DEVSHELL_PROJECT_ID} \
        --zone=${COMPUTE_ZONE} prod-cluster
    
    kubectl apply -f kubernetes
    
  8. In the Cloud Console, go to the Workloads page.

    Go to Workloads page

    Image failure status.

    The image failed to deploy because it wasn't signed by the vulnz-attestor and the qa-attestor.

Break-glass procedure

Occasionally, you need to allow changes that are outside the normal workflow. To allow image deployments without the required attestations, the pod definition is annotated with a break-glass policy flag. Enabling this flag still causes GKE to check for the required attestations, but allows the container image to be deployed and logs violations.

For more information about bypassing attestation checks, see Override a policy.

  1. In Cloud Shell, uncomment the break-glass annotation in the Kubernetes manifest:

    sed -i "31s/^#//" kubernetes/deployment.yaml
    
  2. Use kubectl to apply the changes:

    kubectl apply -f kubernetes
    
  3. To verify the change was deployed to prod-cluster go to the Workloads page in Cloud Console.

    Go to Workloads page

    The deployment error message is now gone.

  4. To verify the app was deployed, go to the Services page.

    Go to Services page

  5. To view the app, click the endpoint link.

Cleaning up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this tutorial:

Delete the project

  1. In the Cloud Console, go to the Manage resources page.

    Go to the Manage resources page

  2. In the project list, select the project that you want to delete and then click Delete .
  3. In the dialog, type the project ID and then click Shut down to delete the project.

What's next