Security Overview

Kubernetes Engine provides many ways to help secure your workloads. Protecting workloads in Kubernetes Engine involves many layers of the stack, including the contents of your container image, the container runtime, the cluster network, and access to the cluster API server.

It's best to take a layered approach to protecting your clusters and workloads. You can apply the principle of least privilege to the level of access provided to your users and your application. In each layer there may be different tradeoffs that must be made that allow the right level of flexibility and security for your organization to securely deploy and maintain their workloads. For example, some security settings may be too constraining for certain types of applications or use cases to function without significant refactoring.

This document provides an overview of each layer of your infrastructure, and shows how you can configure its security features to best suit your needs.

Authentication and authorization

You can authenticate to a Kubernetes Engine cluster in three ways:

  1. Using a Google Account
  2. Using a Google Cloud Platform (GCP) service account
  3. Using a Kubernetes service account

Once authenticated, you need to authorize these identities to create, read, update or delete Kubernetes resources. You can authorize Google accounts and GCP service accounts at the project- and cluster-level level using Cloud Identity & Access Management (Cloud IAM) if you'd like to give accounts access to all Kubernetes Engine clusters within a project.

To configure more granular access to Kubernetes resources at the cluster level or within Kubernetes namespaces, you use Role-Based Access Control (RBAC). RBAC allows you to create detailed policies that define which operations and resources you allow users and service accounts to access. With RBAC, you can control access for Google Accounts, GCP service accounts, and Kubernetes service accounts. To further simplify and streamline your authentication and authorization strategy for Kubernetes Engine, you should ensure that the legacy Attribute Based Access Control is disabled so that Kubernetes RBAC and Cloud IAM are the source of truth.

For more information:

Control plane security

In Kubernetes Engine, the Kubernetes master components are managed and maintained by Google. The master components host the software that runs the Kubernetes control plane, including the API server, scheduler, controller manager and the etcd database where your Kubernetes configuration is persisted.

By default, the master components use a public IP address. You can protect the Kubernetes API server by using master authorized networks, and private clusters, which allow you to assign a private IP address to the master and disable access on the public IP address.

You can handle cluster authentication in Kubernetes Engine by using Cloud IAM as the identity provider. However, legacy username-and-password-based authentication is enabled by default in Kubernetes Engine. For enhanced authentication security, you should ensure that you have disabled Basic Authentication by setting an empty username and password for the MasterAuth configuration. In the same configuration, you can also disable the client certificate which ensures that you have one less key to think about when locking down access to your cluster.

Another way to help secure your Kubernetes master is to ensure that you are doing credential rotation on a regular basis. When credential rotation is initiated, the SSL certificates and cluster certificate authority are rotated. This process is automated by Kubernetes Engine and also ensures that your master IP address rotates.

For more information:

Node security

Kubernetes Engine deploys your workloads on Compute Engine instances running in your GCP project. These instances are attached to your Kubernetes Engine cluster as nodes. The following sections show you how leverage the node-level security features available to you in GCP.

Container-Optimized OS

By default, Kubernetes Engine nodes use Google's Container-Optimized OS as the operating system on which to run Kubernetes and its components. Container-Optimized OS implements several advanced features for enhancing the security of Kubernetes Engine clusters, including:

  • Locked-down firewall
  • Read-only filesystem where possible
  • Limited user accounts and disabled root login

Node upgrades

A best practice is to patch your OS on a regular basis. From time to time, security issues in the container runtime, Kubernetes itself, or the node operating system might require you to upgrade your nodes more urgently. When you upgrade your node, the node's software is upgraded to their latest versions.

You can manually upgrade the nodes in your cluster, but Kubernetes Engine also allows you to enable automatic upgrades.

Securing instance metadata

Kubernetes Engine nodes run as Compute Engine instances, and as such they have access to instance metadata by default. Instance metadata is used to provide nodes with credentials and configurations used in bootstrapping and connecting to the Kubernetes master nodes. However, a Pod running on a node does not necessarily need this information, which contains sensitive data, like the node's service account key. You can lock down sensitive instance metadata paths using metadata concealment. Metadata concealment ensures that Pods running in your cluster are not able to access sensitive data by filtering requests to fields such as the kube-env.

Network security

Most workloads running in Kubernetes Engine need to communicate with other services that could be running either inside or outside of the cluster. You can use several different methods to control what traffic is allowed to flow through your clusters and their Pods.

Limiting Pod-to-Pod communication

By default, all Pods in a cluster can be reached over the network via their Pod IP address. Similarly, by default, egress traffic allows outbound connections to any address accessible in the VPC into which the cluster was deployed.

Cluster administrators and users can lock down the ingress and egress connections created to and from the Pods in a namespace by using network policies. By default, when there are no network policies defined, all ingress and egress traffic is allowed to flow into and out of all Pods. Network policies allow you to use tags to define the traffic flowing through your Pods.

Once a network policy is applied in a namespace, all traffic is dropped to and from Pods that don't match the configured labels. As part of your creation of clusters and/or namespaces, you can apply the default deny traffic to both ingress and egress of every Pod to ensure that all new workloads added to the cluster must explicitly authorize the traffic they require.

For more information:

Filtering load balanced traffic

To load balance your Kubernetes Pods with a network load balancer, you need to create a Service of type LoadBalancer that matches your Pod's labels. With the Service created, you will have an external-facing IP that maps to ports on your Kubernetes Pods. Filtering authorized traffic is achieved at the node level by kube-proxy, which filters based on IP address.

To configure this filtering, you can use the loadBalancerSourceRanges configuration of the Service object. With this configuration parameter, you can provide a list of CIDR ranges that you would like to whitelist for access to the Service. If you do not configure loadBalancerSourceRanges, all addresses are allowed to access the Service via its external IP.

For cases in which access to the Service is not required, consider using an internal load balancer. The internal load balancer also respects the loadBalancerSourceRanges when it is necessary to filter out traffic from inside of the VPC.

For more information:

Securing your workloads

Kubernetes allows users to quickly provision, scale, and update container-based workloads. This section describes tactics that administrators and users can use to limit the abilities of the running containers to affect other containers in the cluster, the hosts on which they run, and the GCP services enabled in their project.

Limiting Pod container process privileges

Limiting the privileges of containerized processes is important for the overall security of your cluster. Kubernetes Engine allows you to set security-related options via the Security Context on both Pods and containers. These settings allow you to change security settings of your processes like:

  • User and group to run as
  • Available Linux capabilities
  • Ability to escalate privileges

In order to change these settings at the cluster level rather than at the Pod or container, you need to implement a PodSecurityPolicy. Cluster administrators can use PodSecurityPolicies to ensure that all Pods in a cluster adhere to a minimum baseline policy that you define.

The default Kubernetes Engine node operating system, Container-Optimized OS, applies the default Docker AppArmor security policies to all containers started by Kubernetes. You can view the profile's template on GitHub. Among other things, the profile denies the following abilities to containers:

  • Write files directly in /proc/
  • Write to files that are not in a process ID directory (/proc/)
  • Write to files in /proc/sys other than /proc/sys/kernel/shm*
  • Mount filesystems

For more information:

Giving Pods access to GCP resources

Your containers and Pods might need access to other resources in GCP. One way for applications to get access to credentials for GCP resources is to use the Kubernetes clusters' service account credentials from metadata. However, these credentials can be reached by any Pods running on the cluster. You should create and configure a custom service account for your clusters that has the minimum IAM roles that should apply to all the Pods running in the cluster.

Another way to provide application credentials is by using a service account JSON key. You should use application-specific GCP service accounts to provide credentials so that applications have the minimal necessary permissions. Each service account is assigned only the Cloud IAM roles that are needed for its paired application to operate successfully. Keeping the service account application-specific makes it easier to revoke its access in the case of a compromise without affecting other applications. Once you have assigned your service account the correct Cloud IAM roles, you can create a JSON service account key, and then mount that into your Pod using a Kubernetes Secret.

For more information:

Audit logging

Audit logging provides a way for administrators to retain, query, process, and alert on events that occur in your Kubernetes Engine environments. Administrators can use the logged information to do forensic analysis, real-time alerting, or for cataloging how a fleet of Kubernetes Engine clusters are being used and by whom.

By default, Kubernetes Engine logs Admin Activity logs. You can optionally also log Data Access events, depending on the types of operations you are interested in inspecting.

For more information:

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine