Help secure software supply chains on Google Kubernetes Engine

This article shows you how to ensure that your software supply chain follows a known and secure path before your code is deployed in a Google Kubernetes Engine (GKE) cluster. The article reviews how Binary Authorization works, then explains how to best implement and use it with Google Cloud to ensure that your deployment pipeline can provide the most information possible to help you enforce approvals at each of your required stages.

The State of DevOps reports identified capabilities that drive software delivery performance. This article will help you with the following capabilities:

Building blocks for securing your software supply chain

When you build and deploy containerized applications, you should use immutable artifacts as the units of deployment. Immutable artifacts ensure that you can deploy and scale your software across many hosts and environments knowing that it will behave as you expect.

After you build the immutable artifacts, put them into an artifact store where they can be versioned and cataloged for further use.

At various phases of your release process, people or automated systems can add metadata that describes the result of an activity, similar to signaling a checkpoint. For example, you might add metadata to your image indicating that it has passed an integration test suite.

After your artifacts have metadata, create a deployment policy that defines which validations they must pass before they can be deployed into your infrastructure. Build a mechanism into your infrastructure to validate that a workload passes all your required checkpoints before it is admitted. This type of validation, done just before the code is set to run, ensures that no deployments can bypass the required stages of your flow.

The following diagram shows a general process for software supply chain management.

software supply chain management process

Securing supply chains in GKE

The following table maps supply chain building blocks to Google Cloud features.

Building block Google Cloud implementation
Immutable artifact Container image
Artifact store Artifact Registry
Artifact metadata Note
Artifact auditors Attestors
Artifact validations Attestations
Deployment policy Binary Authorization policy

The GKE immutable artifact you deploy is a Docker image. This image is built once and then pushed to Artifact Registry, the artifact store, where you can deploy it to any number of GKE clusters.

In Artifact Registry, you add information about images with tags, which represent a single piece of metadata. The most common use for Docker tags on images is to denote the version of the software or a release train, such as v2.0.0 or beta. Although this information is useful, you cannot use tags to add details about the artifact such as where it was built, which source code it was built from, or whether it is approved for deployment in production environments. For these details, you use attestations.

Attestors can state whether an image meets a particular criterion by adding notes about images in Container Analysis. These bits of information can be accessed throughout the lifecycle of your image. Some kinds of notes are categorized so that you can better filter information about your images. One kind of note is an attestation, which is added by attestors in your project. For example, you might have an attestor that validates that images are built without critical security vulnerabilities, or that an image was built by a particular build system.

You add an attestor to your project by uploading an associated Pretty Good Privacy (PGP) public key to create and identify your attestor. Attestors can then create attestations by signing messages that identify which specific content-addressable image digests they are attesting.

The following diagram shows an example of adding an attestation to an image for a particular digest.

adding an attestation to an image

With Binary Authorization you can configure a policy that defines which attestations must be present on your image before it can be deployed across your clusters. This policy creates a building block for ensuring that your images follow the correct path through your deployment pipeline without skipping any important steps. At each of the critical stages of your deployment pipeline, you can create an attestation stating that the image has passed that step. Attestations can be added either by automated systems or by people.

For example, you might want to ensure that your image has passed validations by your quality assurance (QA) team. After they finish testing, one of the QA team members uses a specific key to sign an attestation from the attestor qa-validated. That attestation is required for your production clusters to deploy the image.

Similarly, you might want to ensure that your image has been scanned for vulnerabilities and that no anomalies turned up. An automated system could watch for new images in your repository and add an attestation that they meet a certain vulnerability standard. After the system validates the vulnerability scan, it can sign and add an attestation from the not-vulnerable attestor by using its cryptographic key. You can encrypt this key with Cloud Key Management Service, securely store it in Cloud Storage, control access to it through Identity and Access Management and audit its use through Cloud Logging.

The following diagram shows images in Artifact Registry with attestations added by people and automated attestors.

images in Artifact Registry added by automation or people

With Binary Authorization, you create policies that define which cryptographic keys you can use to validate each of your attestations. You can also add certain images to the list of exempt images to bypass your attestation validation or to disable Binary Authorization for certain clusters.

While Google Cloud provides managed services to create and manage your software supply chain, you can also use open source implementations to implement similar processes in other environments.

The following table maps Google Cloud services to open source tools that provide similar functionality.

Google Cloud Open source
Artifact Registry Docker distribution
Metadata API Grafeas
Binary Authorization Kritis
Vulnerability scanning Clair, Anchore Engine

Continuous delivery pipeline best practices

Validating your software supply chain is most effective when you have a continuous delivery pipeline that uses automation to deploy source code from a code repository to your production infrastructure. The stages of these pipelines vary among organizations, but you can implement some high-level best practices to help ensure that your pipeline deploys your software securely.

General CI/CD pipeline guidance

Many organizations allow individual teams to choose their own continuous delivery process and tooling. While this flexibility can boost autonomy and lets teams use familiar tools, it can complicate ensuring that all releases follow best practices. You might want to centralize release pipelines for teams with an application like Google Cloud Deploy or Spinnaker, which helps to ensure that you can templatize, share, and audit methodologies in a single place.

Image building

As with any supply chain, your software is only as secure as the materials that you use to construct your runnable artifacts. When you use GKE, you must ensure that the images you construct come from secure base images.

The first step in maximizing the security of your base images is to use a minimal distribution that includes only the software and libraries that are needed to run your application. For popular distributions like Ubuntu, Debian, and CentOS, Google Cloud provides base images that are scanned for vulnerabilities, built reproducibly, and stored on Google Cloud so you can pull them directly from your environment without traversing networks. Distroless images are a set of base images that includes only the libraries and tools that are required to run applications written in various programming languages. After you've chosen a base, try to minimize what you add. You can leverage Docker's multistage builds to separate build time dependencies from runtime dependencies, which reduces the attack surface.

After the image is built, make sure that it conforms to your organization's policies. You can create Container Structure tests that validate the image's contents and metadata. You can also use container-diff to compare your existing and newly built containers.

When you have narrowed down what your application image contains, try to audit and reduce the scope of third-party images that are allowed in your clusters. You can use Binary Authorization to add the images to the list of exempt images that can bypass your attestation requirements. Ensure that the entry is as specific as possible for each of the images that you require but cannot build through your artifact creation pipeline.

You can find more information on building containers in the Best practices for building containers documentation.

Enable image vulnerability analysis

Enable Artifact Registry vulnerability scanning in your projects to get more information about potential security issues in your container images. The automated scans that run as soon as your images are pushed can help you grasp the severity of the vulnerabilities in your images. Although you must check for vulnerabilities before you deploy, you should also check your images after they are deployed, because new vulnerabilities might be found. Vulnerability scanning continuously scans any images pulled from the registry in the last 30 days. You can use Pub/Sub to get notifications whenever vulnerabilities are found in your images.

Validate your policy during deployment

To secure your software supply chain, enable image validation during deployment. The last step in a continuous delivery pipeline, deployment is a good time to ensure that your process included all stages and passed all checkpoints. When you take this approach, even if someone obtained access to credentials for your clusters, they couldn't deploy malicious images.

With Binary Authorization, you can configure a default rule for all clusters in the project and then override those rules with cluster-specific configurations. For example, you might want all of your clusters to require attestations of vulnerability scan and QA validation, but also want to ensure that developers can experiment with new technologies and images in the development cluster without enduring QA validation. For more information on Binary Authorization policies, see the Policy YAML reference documentation.

Example pipeline

Here is an example pipeline that is divided into two main phases: build and deploy.

build and deploy phases in a sample pipeline

The following table provides more detail about pipeline steps and where the attestations get added.

Step number Summary Description Attestation
icon
1 Get source code Check out the source code at the desired revision for use by the build system. None
2 Run unit tests Ensure that the code passes all the unit tests before you create an artifact.

Run unit tests.
3 Create a Docker image Use the Dockerfile in the local source code to build an image that can be pushed to Artifact Registry. None
4 Ensure Docker image conforms to policy Using Container Structure Tests, you can assert that your image has the correct contents and that commands within the image return the correct output.

Ensure Docker image conforms to policy.
5 Push image to Artifact Registry When the code has been unit tested and the contents of your image have been validated, you can push the image to Artifact Registry. The push triggers a vulnerability scan automatically. None
6 Ensure image doesn't have critical vulnerabilities Vulnerability scanning takes a few minutes to complete and then adds notes to the image's metadata about any Common Vulnerabilities and Exposures (CVEs) that show up. Wait for this data to appear, and then add an attestation if no critical vulnerabilities are found.

Ensure image doesn't have critical vulnerabilities.
7 Deploy image to staging Kick off a deployment to a pre-production environment where your change can be validated and observed in scenarios as close to production as possible.

Deploy image to staging.
8 Approve code for deployment in production If all signals show that the change meets its requirements, approve it to be deployed in production. None
9 Roll out code in production clusters Now you can set the image to update across your fleet of clusters. Each cluster checks that all attestations are complete before running your application. None

Example attestations

Here are some attestations that you might want to include in your release pipeline:

  • Vulnerability analysis results did not include critical vulnerabilities.
  • Manual QA validation has passed.
  • All software that is included in the image is licensed appropriately.
  • Artifact was created in a trusted build system.

What's next