Exploring container security: Let Google do the patching with new managed base images

Editor’s note: This is a continuation of a series of blog posts on container security at Google.

As a Google Kubernetes Engine (GKE) user, you already enjoy the choice of several operating system (OS) images for your nodes, which we maintain and update for you behind the scenes, notably Container-Optimized OS (COS) and Ubuntu. You bring your own container images for your workloads, based on your needs. Today, we're expanding our support for container images as well, with managed base images that you can use as a starting point when building your applications.

At Google Cloud, we’ve long maintained base images as part of the infrastructure that powers hosted services such as Google App Engine. With managed base images, we’ll provide base images for these common OSes, and patch them automatically. As long as the FROM field in your Dockerfile points to `$distro:latest` from Cloud Marketplace, you know that these images have been remediated with the most recently available patches from upstream. That way, you can easily keep your images up-to-date, without having to pull from an unknown repository, or having to maintain the images yourself.

Managed base images deal with the fact that containers are often short-lived, and frequently re-deployed, making it difficult to follow best practices such as ensuring that your container image is built from up-to-date and trusted sources. A container bundles binaries and libraries together as part of the container image. Rather than pushing small changes to a running container, you instead rebuild and redeploy the whole image, including the base, binaries, and libraries. With managed base images, any processes that can take place passively, for example patching, are done on an ongoing basis, so that the latest patches will be picked up the next time you deploy.

Today, managed base images are available for the following distributions:

• CentOS 7
  
• Debian 9 “Stretch”
  
• Ubuntu 16.04
  

Managed base images follow security best practices—in addition to being maintained with regular patching and testing, they can be rebuilt from scratch reproducibly—by comparing them to the original source we can verify that no flaws were introduced.

Managed base images vs. distroless images

An alternative to managed base images is distroless images. These images contain only your application and its runtime dependencies, greatly reducing the potential attack surface. A package with a newly discovered vulnerability can’t affect you if you don’t have the package in the first place! Distroless images remove package managers, shells, or other programs you might find in a standard Linux distribution, so that you’re focusing on what’s actually important: reducing the signal-to-noise ratio that vulnerability scanning usually generates, and leaving you less to maintain.

Both distroless and managed base images are good options for your containers. If you need a full Linux distribution, including features like a package manager or a shell, then managed base images are a good choice. If you want the most locked-down option, then distroless images might be a better choice. Read more about managed base images, and pull them directly from Cloud Marketplace.

Working with managed base images

If you decide to use a managed base image, you may notice that there are still some vulnerabilities when you scan these base images with Container Registry Vulnerability Scanning. That’s to be expected, and can happen for a variety of reasons:

• Upstream maintainers don’t always agree with the vulnerabilities listed in the CVE database, e.g., CVE-2005-2541 is considered a High severity vulnerability in Debian, but is considered “intended behavior,” making it a feature, not a bug.
  
• Vulnerabilities may not have an available patch, and so even though they’ve been identified, there is no current solution.
  
• Lower-severity vulnerabilities may not have been prioritized upstream. Typically, maintainers address less severe vulnerabilities at a regular cadence, so while the latest version may not contain the relevant patch, a future version will.
  

Further, please note that although you’re pulling `$distro:latest`, this isn’t pulling the latest version of a distro, e.g., Ubuntu, but rather the latest from a particular version of a distribution, e.g., Ubuntu 16.04. This  means you get security patches, but no unexpected new functionality.

CentOS support

Based on popular demand, we’re also introducing support for CentOS, with a managed base image. CentOS is a community-driven OS providing a robust base for building your containers.

The CentOS managed base image uses `yum` and `rpm` for package management, and these pull RPM files only over HTTPS connections. You can pull the CentOS base image directly from Cloud Marketplace.

Best practices for image validation

To help you further secure your images, we’ve also published a new solution on image validation best practices as part of a CI/CD process. Since containers are meant to be immutable, they are constantly being rebuilt and redeployed. By having a straightforward, consistent CI/CD process, you can validate and restrict what ends up in your environment on an ongoing basis.

In a nutshell, here are the steps we recommend you take to validate your container images:

• Use a centralized, locked down CI/CD process. Simplify your CI/CD with a small number of image repositories, and a centralized release pipeline for all production jobs.
  
• Build your container image from trusted sources. Minimize your attack surface by using a minimal patched base image, such as Google managed base images; Also be careful to only pull in packages that you need.
  
• Implement image scanning and analysis. Vulnerabilities can still be present even when you only build from trusted sources. GCR Vulnerability Scanning is an easy way for you to get started scanning your images.
  
• Enforce a deploy-time image validation policy. Tie together your CI/CD verifications using a deploy-time policy to ensure only trusted images can be deployed to your production environment. For example, you may want to only deploy images that are built by your centralized pipeline, and that pass basic vulnerability scans. Binary Authorization provides a flexible signature based deploy policy enforcement for GKE that can easily be integrated into your existing CI/CD pipeline.
  

To learn more about container image validation, check out Secure Software Supply Chains on Kubernetes Engine. For further content on container security best practices, read our container security guide.