This article shows you how to ensure that your software supply chain follows a known and secure path before your code is deployed in a Google Kubernetes Engine (GKE) cluster. The article reviews how Binary Authorization works, then explains how to best implement and use it with Google Cloud to ensure that your deployment pipeline can provide the most information possible to help you enforce approvals at each of your required stages.
The State of DevOps reports identified capabilities that drive software delivery performance. This article will help you with the following capabilities:
Building blocks for securing your software supply chain
When you build and deploy containerized applications, you should use immutable artifacts as the units of deployment. Immutable artifacts ensure that you can deploy and scale your software across many hosts and environments knowing that it will behave as you expect.
After you build the immutable artifacts, put them into an artifact store where they can be versioned and cataloged for further use.
At various phases of your release process, people or automated systems can add metadata that describes the result of an activity, similar to signaling a checkpoint. For example, you might add metadata to your image indicating that it has passed an integration test suite.
After your artifacts have metadata, create a deployment policy that defines which validations they must pass before they can be deployed into your infrastructure. Build a mechanism into your infrastructure to validate that a workload passes all your required checkpoints before it is admitted. This type of validation, done just before the code is set to run, ensures that no deployments can bypass the required stages of your flow.
The following diagram shows a general process for software supply chain management.
Securing supply chains in GKE
The following table maps supply chain building blocks to Google Cloud features.
|Building block||Google Cloud implementation|
|Immutable artifact||Container image|
|Artifact store||Artifact Registry|
|Deployment policy||Binary Authorization policy|
The GKE immutable artifact you deploy is a Docker image. This image is built once and then pushed to Artifact Registry, the artifact store, where you can deploy it to any number of GKE clusters.
In Artifact Registry, you add information about images with tags,
which represent a single piece of metadata. The
most common use for Docker tags on images is to denote the version of the
software or a release train, such as
beta. Although this
information is useful, you cannot use tags to add details
about the artifact such as where it was built, which source code it was built
from, or whether it is approved for deployment in production environments.
For these details, you use attestations.
Attestors can state whether an image meets a particular
adding notes about images in Container Analysis.
These bits of information can be accessed throughout the lifecycle of
your image. Some
of notes are categorized so that you can better filter information about your
images. One kind of note is an
attestation, which is added by
in your project. For example, you might have an attestor that validates that
images are built without critical security vulnerabilities, or that an image was
built by a particular build system.
You add an attestor to your project by uploading an associated Pretty Good Privacy (PGP) public key to create and identify your attestor. Attestors can then create attestations by signing messages that identify which specific content-addressable image digests they are attesting.
The following diagram shows an example of adding an attestation to an image for a particular digest.
With Binary Authorization you can configure a policy that defines which attestations must be present on your image before it can be deployed across your clusters. This policy creates a building block for ensuring that your images follow the correct path through your deployment pipeline without skipping any important steps. At each of the critical stages of your deployment pipeline, you can create an attestation stating that the image has passed that step. Attestations can be added either by automated systems or by people.
For example, you might want to ensure that your image has passed validations by
your quality assurance (QA) team. After they finish testing, one of the
QA team members uses a specific key to sign an attestation from the attestor
qa-validated. That attestation is required for your production clusters to
deploy the image.
Similarly, you might want to ensure that your image has been scanned for
vulnerabilities and that no anomalies turned up. An automated system could watch
for new images in your repository and add an attestation that they meet a
certain vulnerability standard. After the system validates the vulnerability
scan, it can sign and add an attestation from the
not-vulnerable attestor by
using its cryptographic key. You can encrypt this key with Cloud Key Management Service,
securely store it in Cloud Storage, control access to it
through Identity and Access Management and audit its use through Cloud Logging.
The following diagram shows images in Artifact Registry with attestations added by people and automated attestors.
With Binary Authorization, you create policies that define which cryptographic keys you can use to validate each of your attestations. You can also add certain images to the list of exempt images to bypass your attestation validation or to disable Binary Authorization for certain clusters.
While Google Cloud provides managed services to create and manage your software supply chain, you can also use open source implementations to implement similar processes in other environments.
The following table maps Google Cloud services to open source tools that provide similar functionality.
|Google Cloud||Open source|
|Artifact Registry||Docker distribution|
|Vulnerability scanning||Clair, Anchore Engine|
Continuous delivery pipeline best practices
Validating your software supply chain is most effective when you have a continuous delivery pipeline that uses automation to deploy source code from a code repository to your production infrastructure. The stages of these pipelines vary among organizations, but you can implement some high-level best practices to help ensure that your pipeline deploys your software securely.
General CI/CD pipeline guidance
Many organizations allow individual teams to choose their own continuous delivery process and tooling. While this flexibility can boost autonomy and lets teams use familiar tools, it can complicate ensuring that all releases follow best practices. You might want to centralize release pipelines for teams with an application like Google Cloud Deploy or Spinnaker, which helps to ensure that you can templatize, share, and audit methodologies in a single place.
As with any supply chain, your software is only as secure as the materials that you use to construct your runnable artifacts. When you use GKE, you must ensure that the images you construct come from secure base images.
The first step in maximizing the security of your base images is to use a minimal distribution that includes only the software and libraries that are needed to run your application. For popular distributions like Ubuntu, Debian, and CentOS, Google Cloud provides base images that are scanned for vulnerabilities, built reproducibly, and stored on Google Cloud so you can pull them directly from your environment without traversing networks. Distroless images are a set of base images that includes only the libraries and tools that are required to run applications written in various programming languages. After you've chosen a base, try to minimize what you add. You can leverage Docker's multistage builds to separate build time dependencies from runtime dependencies, which reduces the attack surface.
After the image is built, make sure that it conforms to your organization's
policies. You can create
Container Structure tests
that validate the image's contents and metadata. You can also use
to compare your existing and newly built containers.
When you have narrowed down what your application image contains, try to audit and reduce the scope of third-party images that are allowed in your clusters. You can use Binary Authorization to add the images to the list of exempt images that can bypass your attestation requirements. Ensure that the entry is as specific as possible for each of the images that you require but cannot build through your artifact creation pipeline.
You can find more information on building containers in the Best practices for building containers documentation.
Enable image vulnerability analysis
Enable Artifact Registry vulnerability scanning in your projects to get more information about potential security issues in your container images. The automated scans that run as soon as your images are pushed can help you grasp the severity of the vulnerabilities in your images. Although you must check for vulnerabilities before you deploy, you should also check your images after they are deployed, because new vulnerabilities might be found. Vulnerability scanning continuously scans any images pulled from the registry in the last 30 days. You can use Pub/Sub to get notifications whenever vulnerabilities are found in your images.
Validate your policy during deployment
To secure your software supply chain, enable image validation during deployment. The last step in a continuous delivery pipeline, deployment is a good time to ensure that your process included all stages and passed all checkpoints. When you take this approach, even if someone obtained access to credentials for your clusters, they couldn't deploy malicious images.
With Binary Authorization, you can configure a default rule for all clusters in
the project and then override those rules with cluster-specific configurations.
For example, you might want all of your clusters to require attestations of
vulnerability scan and QA validation, but also want to ensure that developers
can experiment with new technologies and images in the
without enduring QA validation. For more information on Binary Authorization
policies, see the
Policy YAML reference
Here is an example pipeline that is divided into two main phases: build and deploy.
The following table provides more detail about pipeline steps and where the attestations get added.
|1||Get source code||Check out the source code at the desired revision for use by the build system.||None|
|2||Run unit tests||Ensure that the code passes all the unit tests before you create an artifact.||
|3||Create a Docker image||Use the Dockerfile in the local source code to build an image that can be pushed to Artifact Registry.||None|
|4||Ensure Docker image conforms to policy||Using Container Structure Tests, you can assert that your image has the correct contents and that commands within the image return the correct output.||
|5||Push image to Artifact Registry||When the code has been unit tested and the contents of your image have been validated, you can push the image to Artifact Registry. The push triggers a vulnerability scan automatically.||None|
|6||Ensure image doesn't have critical vulnerabilities||Vulnerability scanning takes a few minutes to complete and then adds notes to the image's metadata about any Common Vulnerabilities and Exposures (CVEs) that show up. Wait for this data to appear, and then add an attestation if no critical vulnerabilities are found.||
|7||Deploy image to staging||Kick off a deployment to a pre-production environment where your change can be validated and observed in scenarios as close to production as possible.||
|8||Approve code for deployment in production||If all signals show that the change meets its requirements, approve it to be deployed in production.||None|
|9||Roll out code in production clusters||Now you can set the image to update across your fleet of clusters. Each cluster checks that all attestations are complete before running your application.||None|
Here are some attestations that you might want to include in your release pipeline:
- Vulnerability analysis results did not include critical vulnerabilities.
- Manual QA validation has passed.
- All software that is included in the image is licensed appropriately.
- Artifact was created in a trusted build system.
- Review the Binary Authorization tutorial.
- Learn about the integration between Binary Authorization and vulnerability scanning.
- Learn about automatically adding an attestation for build provenance container images built by Cloud Build.
- Watch the Google Cloud Next 2021 Talk: Securing the software supply chain.
- Learn how to add metadata to your Artifact Registry images.
- Explore reference architectures, diagrams, and best practices about Google Cloud. Take a look at our Cloud Architecture Center.
- Read our resources about DevOps.
- Learn more about the DevOps capabilities related to this article:
- Take the DevOps quick check to understand where you stand in comparison with the rest of the industry.