How Google Does It: Using Binary Authorization to boost supply chain security

Ever wondered how Google does security? As part of our new “How Google Does It” series, we’ll share insights, observations, and top tips about how Google approaches some of today's most pressing security topics, challenges, and concerns — straight from Google experts. In this edition, Daniel Lees, Cloud Security Architect, shares how Google verifies code provenance and implements code identity in its software supply chains using Binary Authorization.

In today’s world of dynamic environments and containerized workloads, organizations are making a lot of great investments in trusted repositories, code review, and integrating security into their software development processes. To ensure that a software build is secure when it deploys to production, we start with a simple principle: Don’t trust, verify.

That principle guides how we secure our entire software supply chain. On a practical level, to ensure that every component meets our security best practices and standards, we use Binary Authorization. This enforcement check helps us ensure that all production software and configuration is properly reviewed, checked in, built verifiably, and authorized — especially when that code can access sensitive user data.

How Binary Authorization helps us mitigate supply chain risks

In software supply chain attacks, attack points are scattered across the software development lifecycle. SolarWinds, Log4shell, and Codecov are headline-grabbing examples of attackers targeting software build systems, exploiting vulnerabilities in third-party libraries or other components used in development, compromising updates, and infiltrating continuous integration and continuous delivery (CI/CD) pipelines.

At Google, our internal cluster management system, Borg, powers everything from Google Search to Maps to YouTube, enabling us to launch more than 4 billion containers a week. Binary Authorization is crucial to helping Google run securely at the scale we require by preventing any one point of failure from leading to catastrophe.

Consider a policy that requires a production deployment to be built in a central build pipeline, checked by your vulnerability scanner, and signed off by QA engineers. If an attacker compromised the vulnerability scanner, you still have other rules that must be verified before an image can be deployed.

Binary Authorization also helps with compliance. Many organizations today need to understand what code they are deploying, and whether that code has access to sensitive information. There are a wide range of regulations that now require organizations to carefully vet their code, which can be challenging when relying on manual processes to establish provenance in CI/CD environments.

At Google, we use Binary Authorization to significantly improve operational efficiency and control. It established uniformity across production systems by verifying containerized systems before deployment, which streamlined change management and enhanced reliability.

Furthermore, Binary Authorization introduced a common language for data protection by tracking and publishing conformance data, enabling cross-team communication and reducing back-and-forth efforts. It also provided programmatic tracking of compliance requirements like Sarbanes-Oxley (SOX), automating checks to increase the scope of compliance coverage and adoption of necessary controls that previously had to be done manually.

How Binary Authorization works

Think of your production environment like the secure side of an airport. To get on a plane, you don't just “promise” you're not a threat. You must present a valid ID and a boarding pass (proof) that are verified by an agent (an enforcer) against a set of rules (the policy).

Binary Authorization is that agent for your software. It doesn't trust; it verifies. Before a container is allowed to run, it must present a valid ID and “boarding pass,” a set of cryptographic signatures (attestations), proving the container passed all required checks. This process keeps defective, vulnerable, and unauthorized code out of your production environment.

One of the key security concerns of container-based architectures is knowing what’s running and where it came from. All that velocity, performance, and flexibility creates more churn, making it difficult to ensure you only deploy authorized software. In particular, it’s important to establish provenance — the ability to trace the source of code to a trusted point of origin and ensure that all the desired validation steps were completed.

To solve these issues we can define deploy-time policies in Binary Authorization that enforce specific criteria that a service’s code and configuration must meet before it deploys to production.

For example, a policy might check:

• Was the deployment built using an approved system?
• Was it scanned for vulnerabilities and did it pass all the desired validation steps?
• Was it verified by the required trusted authorities?

Each step in your pipeline — the build, the scan, the QA check — provides a tamper-proof cryptographic signature, a stamp of approval called an attestation. An attestation is signed using a private key from a pair and verified at deploy time with its public key.

When you deploy an image, Binary Authorization checks the policy and enforces any rules it finds that govern deployment. If an image fails to pass the required checks, the deployment is blocked until it is compliant.

Since each stage of the deployment lifecycle can have its own deployment environment, Binary Authorization allows you to define and enforce policies that can prevent noncompliant images from progressing from one stage to another. It also supports creating monitoring policies that enable you to periodically check images conform with defined policies and generate log entries for any violations.

Putting Binary Authorization into action: Lessons from Google and our customers

Binary Authorization wasn't created in a vacuum; it was an evolution of Google's internal security posture. As we scaled Borg, our internal Kubernetes, we recognized the limitations of perimeter-based security. The pain points of securing a rapidly changing, containerized environment at massive scale led us to develop a system that enforced verification at deployment time, a principle that is now a cornerstone of our software supply chain security.

When we first introduced Binary Authorization, we started out with monitoring policies in dry-run mode. This allowed us to start small without impacting our production environments. Monitoring helped us see what was going on and spot existing issues like outdated software or security scans, but we learned this meant our operations team was spending more time on fixing issues rather than preventing them.

For new projects, we now typically start with enforcement right away, using monitoring policies only briefly to check all the rules are correct before enforcing the policy as much as possible.

This shift in strategy also defines how we scope our policies. We can’t completely avoid having to clean up existing projects, but careful planning is crucial; if you only set up rules for existing systems, you’ll have to invest time later trying to apply them to new systems.

Instead, we’ve learned to anticipate and structure our policies in a way that can automatically apply to new projects. One approach might be to create a policy that covers one part of your organization — not defining it to specific resources.

Another important aspect we had to consider is how it fit into existing processes. We found it’s a huge challenge to enforce these rules across multiple teams using different tools and doing things their own way. In many global organizations, we found it better to have a centralized DevOps or Platform team that acts more like consultants, providing standardized guidelines, tools, and templates for independent development teams.

When one team controls the pathway to production, they can effectively set up and enforce policies for everyone. This central model makes your policies more scalable, as the central team can create broad policies that apply to entire organizational units — automatically covering new projects the moment they're created.

To balance this centralization with developer autonomy, the central team can manage the core policy framework while empowering individual development teams to create and manage their own specific attestors for team-level checks. Organizations looking to get started with Binary Authorization should start with checking the provenance of the software to make sure that only approved software built in trusted pipelines runs in production.

One way to do it would be to have your build system add an attestation to every container and store it with an image. With Binary Authorization, you can then set up a policy for production that only software with a valid signature from the build system can run. For example, this is how customers can deploy only images built by Cloud Build using Binary Authorization.

For multi-project setups, we recommend using separate projects: a deployer project where your policy is configured; an attestor project where your attestors are stored; and an attestation project for attestations. This supports the separation of duties principle, and applying attestors easily across multiple deployer projects.

You can also mandate Binary Authorization policies adhere to requirements, such as always requiring attestations, or mandating that specific attestors need to be included.

Don’t trust, verify: How to get started

When taking on a potentially huge shift in mindset, getting started can feel complicated — but it doesn’t need to be. A phased approach allows you to gradually increase your security posture without disrupting development teams.

• Start in monitor-mode: Begin with a policy in dry-run mode. This allows you to see what would have been blocked without impacting production. Use the generated audit logs to identify non-compliant workflows and bring existing systems into compliance.
• Enforce build provenance: Your first enforcement step is simple: Ensure all production code is signed by your trusted, automated build system. This high-value first step can help prevent untracked and developer-built images from running in production.
• Layer additional checks: Once build provenance is established, incrementally add more attestations to your policy. You might require attestations that prove an image has been successfully scanned for vulnerabilities or has been approved by your QA team.

To learn more about how you can implement these principles, check out the documentation for Binary Authorization, which includes detailed how-to guides, policy examples, and integration patterns to help you get started with your first policy.

This article includes insights from the Cloud Security Podcast episode, “This Binary Legit? How Google Uses Binary Authorization and Code Provenance.”