Security patching

This document describes how Google manages security vulnerabilities and patches for Google Kubernetes Engine (GKE) and Anthos. Except where otherwise noted, Anthos includes both GKE and Anthos platforms.

Patching shared responsibility

Patching is a shared responsibility between Google and the customer. Different platforms have different shared responsibilities. See the following topics on each platform for more details:

How we discover vulnerabilities

Google has made large investments in proactive security design and hardening, but even the best designed software systems can have vulnerabilities. To find those vulnerabilities and patch them before they can be exploited, Google has made significant investments on multiple fronts.

For patching purposes, Anthos is an Operating System (OS) layer with containers running on top. The operating systems, Container-Optimized OS or Ubuntu, are hardened and contain the minimum amount of software required to run containers. Anthos features run as containers on top of the base images.

Google identifies and fixes vulnerabilities and missing patches in the following ways:

  • Container-Optimized OS: Google scans images to identify potential vulnerabilities and missing patches. The Container-Optimized OS team reviews and resolves issues.

  • Ubuntu: Canonical provides Google with OS builds that have all available security patches applied.

Google scans containers using Container Registry Container Analysis to discover vulnerabilities and missing patches in Kubernetes and Google-managed containers. If fixes are available, the scanner automatically begins the patching and release process.

In addition to automated scanning, Google discovers and patches vulnerabilities unknown to scanners in the following ways.

Google performs its own audits, penetration testing, and vulnerability discovery across all Anthos platforms. Specialized teams inside Google and trusted third-party security vendors conduct their own attack research. Google has also worked with the CNCF to provide much of the organization and technical consulting expertise for the Kubernetes security audit.

Google actively engages with the security research community through multiple vulnerability reward programs. A dedicated Google Cloud vulnerability rewards program provides significant bounties, including $133,337 for the best cloud vulnerability found each year. For GKE, there is a program that rewards security researchers if they are able to break our security controls. The program covers all GKE software dependencies.

Google collaborates with other industry and open source software partners who share vulnerabilities, security research, and patches before the public release of the vulnerability. The goal of this collaboration is to patch large pieces of Internet infrastructure before the vulnerability is announced to the public. In some cases, Google contributes vulnerabilities found to this community. For example, Google's Project Zero discovered and publicized the Spectre and Meltdown vulnerabilities. The Google Cloud Security team also regularly finds and fixes vulnerabilities in the Kernel-based Virtual Machine (KVM).

Google's security collaboration happens on many levels. Sometimes it occurs formally through programs where organizations sign up to receive pre-release notifications about software vulnerabilities for products such as Kubernetes and Envoy. Collaboration also happens informally due to our engagement with many open source projects such as the Linux kernel, container runtimes, virtualization technology, and others.

For Kubernetes, Google is an active and founding member of the Product Security Committee and wrote much of the Security Release Process. Google is a member of the Kubernetes Distributors List that receives prior notification of vulnerabilities and has been involved in the triage, patching, mitigation development, and communication of nearly every serious Kubernetes security vulnerability. Google also discovered multiple Kubernetes vulnerabilities ourselves.

While less severe vulnerabilities are discovered and patched outside of these processes, most critical security vulnerabilities are reported privately through one of the channels previously listed. Early reporting gives Google time before the vulnerability becomes public to research how it affects Anthos, develop patches or mitigations, and prepare advice and communications for customers. When possible, Google patches all clusters before the public release of the vulnerability.

How vulnerabilities are classified

The Anthos security team classifies vulnerabilities according to the Kubernetes vulnerability scoring system. Classifications consider many factors including GKE and Anthos configuration and security hardening. The following table describes vulnerability severity categories:

Severity Description
Critical A vulnerability easily exploitable in all clusters by an unauthenticated remote attacker that leads to full system compromise.
High A vulnerability easily exploitable for many clusters that leads to loss of confidentiality, integrity, or availability.
Medium A vulnerability exploitable for some clusters where loss of confidentiality, integrity, or availability is limited by common configurations, difficulty of the exploit itself, required access, or user interaction.
Low All other vulnerabilities. Exploitation is unlikely or consequences of exploitation are limited.

See the security bulletins for example vulnerabilities, fixes and mitigations, and their ratings.

How vulnerabilities are patched

Patching a vulnerability involves upgrading to a new GKE or Anthos version number. GKE and Anthos versions include versioned components for the operating system, Kubernetes components, and other containers that make up the Anthos platform. Fixing some vulnerabilities requires only a control plane upgrade, performed automatically by Google on GKE, while others require both control plane and node upgrades.

To keep clusters patched and hardened against vulnerabilities of all severities, we recommend using node auto-upgrade on GKE (on by default). On other Anthos platforms, Google recommends upgrading your Anthos components at least monthly.

Patching timelines

Google's goal is to mitigate detected vulnerabilities within a time period appropriate for the risks they represent. GKE is included within Google Cloud's FedRAMP provisional ATO which requires that known vulnerabilities to be remediated within specific timeframes according to their severity level as specified in FedRAMP RA-5d. Google Cloud's FedRAMP provisional ATO doesn't include Anthos clusters on VMware and Anthos clusters on AWS, but we aim for the same remediation timeframes on those products.

How vulnerabilities and patches are communicated

The best source for current information about vulnerabilities and security patches is in the security bulletins feeds for the following products:

These bulletins follow a common Google Cloud vulnerability numbering scheme and are linked from the main Google Cloud bulletins page and the GKE Release Notes. Each security bulletins page has an RSS feed where users can subscribe to updates.

Vulnerabilities are sometimes kept private under embargoes for a limited time. Embargoes help prevent early publication of vulnerabilities that might lead to widespread exploitation attempts before steps can be taken to address them. In embargo situations, the release notes refer to "security updates" until the embargo has been lifted. After the embargo is lifted, Google updates release notes to include the specific vulnerabilities.

The Anthos security team publishes security bulletins for High and Critical severity vulnerabilities. When customer action is required to address these High and Critical vulnerabilities, Google contacts customers by email. In addition, Google might also contact customers with support contracts through support channels.