Hardening your cluster's security

With the speed of development in Kubernetes, there are often new security features for you to use. This page guides you through implementing our current guidance for hardening your Google Kubernetes Engine cluster. For a general overview of security topics, read the Security Overview.

Many of these recommendations, as well as other common misconfigurations, can be automatically checked using Security Health Analytics.

Disable the Kubernetes web UI (Dashboard)

You should disable the Kubernetes Web UI (Dashboard) when running on GKE.

The Kubernetes Web UI (Dashboard) is backed by a highly privileged Kubernetes Service Account. The Cloud Console provides much of the same functionality, so you don't need these permissions.

To disable the Kubernetes Web UI:

gcloud container clusters update [CLUSTER_NAME] \

Disable ABAC

You should disable Attribute-Based Access Control (ABAC), and instead use Role-Based Access Control (RBAC) in GKE.

In Kubernetes, RBAC is used to grant permissions to resources at the cluster and namespace level. RBAC allows you to define roles with rules containing a set of permissions. RBAC has significant security advantages and is now stable in Kubernetes, so it’s time to disable ABAC.

If you're still relying on ABAC, first review the Prerequisites for using RBAC. If you upgraded your cluster from an older version and are using ABAC, you should update your access controls configuration:

gcloud container clusters update [CLUSTER_NAME] \

To create a new cluster with the above recommendation:

gcloud container clusters create [CLUSTER_NAME] \

Restrict Traffic Among Pods with a Network Policy

By default, all Pods in a cluster can communicate with each other. You should control Pod to Pod communication as needed for your workload.

You can use Kubernetes' Network Policies to make it much more difficult for attackers to move laterally within your cluster. You can also use the Kubernetes Network Policy API to create Pod-level firewall rules. These firewall rules determine which Pods and services can access one another inside your cluster.

To enable network policy enforcement when creating a new cluster, specify the --enable-network-policy flag:

gcloud container clusters create [CLUSTER_NAME] \
    --zone=[COMPUTE_ZONE] \

After you enable Network Policy, you have to actually define a policy. Since this is specific to your exact topology, we can’t provide a specific recommendation. The Kubernetes documentation, however, has an excellent walkthrough for a simple nginx deployment.

To learn more about Network Policies in GKE, see Setting a Cluster Network Policy.

Using NetworkPolicy and PodSecurityPolicy together

If you are using a NetworkPolicy, and you have a Pod that is subject to a PodSecurityPolicy, create an RBAC Role or ClusterRole that has permission to use the PodSecurityPolicy. Then bind the Role or ClusterRole to the Pod's service account. Granting permissions to user accounts is not sufficient in this case. For more information, see Authorizing policies.

Use Least Privilege Service Accounts for your Nodes

Each GKE node has an IAM Service Account associated with it. By default, nodes are given the Compute Engine default service account, which you can find by navigating to the IAM section of the Cloud Console. This account has broad access by default, making it useful to wide variety of applications, but it has more permissions than are required to run your Kubernetes Engine cluster. You should create and use a minimally privileged service account to run your GKE cluster instead of using the Compute Engine default service account.

GKE requires, at a minimum, the service account to have the monitoring.viewer, monitoring.metricWriter, and logging.logWriter roles. Read more about monitoring roles and logging roles.

The following commands create an IAM service account with the minimum permissions required to operate GKE:

gcloud iam service-accounts create [SA_NAME] \

gcloud projects add-iam-policy-binding [PROJECT_ID] \
    --member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \
    --role roles/logging.logWriter

gcloud projects add-iam-policy-binding [PROJECT_ID] \
    --member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \
    --role roles/monitoring.metricWriter

gcloud projects add-iam-policy-binding [PROJECT_ID] \
    --member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \
    --role roles/monitoring.viewer

If you use private images in Google Container Registry, you also need to grant access to those:

gcloud projects add-iam-policy-binding [PROJECT_ID] \
  --member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \
  --role roles/storage.objectViewer

If you want another user to be able to create new clusters or node pools with this service account, you must grant them the Service Account User role on this service account:

gcloud iam service-accounts add-iam-policy-binding \
  [SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com \
  --member=user:[USER] \

If your cluster already exists, you can now create a new node pool with this new service account:

gcloud container node-pools create [NODE_POOL] \
  --service-account=[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com \

If you need your GKE cluster to have access to other Google Cloud services, you should create an additional service account and grant your workloads access to the service account by storing its private key in a Kubernetes Secret. To learn more, refer to Authenticating to Google Cloud Platform with Service Accounts.

Reduce your Node Service Account Scopes

If you don't want to use a custom service account, an alternative recommendation is to reduce the permissions of the default node service account.

By default, your node service account has access scopes. Access scopes are the legacy method of specifying permissions for your instance. Before the existence of IAM roles, access scopes were the only mechanism for granting permissions to service accounts.

If you are not creating a separate service account for your nodes, you should limit the scopes of the node service account to reduce the possibility of a privilege escalation in an attack. This ensures that your default service account does not have permissions beyond those necessary to run your cluster. While the default scopes are limited, they may include scopes beyond the minimally required scopes needed to run your cluster.

The default scopes for the nodes in GKE are devstorage.read_only, logging.write, monitoring, service.management.readonly, servicecontrol, and trace.append. When setting scopes, these are specified as gke-default. If you are accessing private images in Google Container Registry, the minimally required scopes are only logging.write, monitoring, and devstorage.read_only.

To create a cluster with the custom scopes, use the --scopes option:

gcloud container clusters create [CLUSTER_NAME] \

If you do not specify scopes or a custom service account, gke-default is used. If you specify a custom service account, cloud-platform and userinfo.email are used.

If your gcloud configuration includes container/new_scopes_behavior true, this behavior is already enabled for all Kubernetes versions. To set the default for your environment:

gcloud config set container/new_scopes_behavior true

To learn more about permissions for all scopes, see Google scopes.

Restrict Client Authentication Methods

There are several methods of authenticating to the Kubernetes API server. In GKE, the supported methods are OAuth tokens, x509 client certificates, and static passwords. GKE manages authentication via gcloud for you using the OAuth token method, setting up the Kubernetes configuration, getting an access token, and keeping it up to date.

The other authentication methods, x509 certificates and static passwords, present a wider surface of attack for cluster compromise. With these other methods, credentials are generated for you when you create a new cluster. Unless your application is using these methods of authentication, you should disable them. You should use OAuth to authenticate to the Kubernetes API server. This is how kubectlis authenticated when you run gcloud container clusters get-credentials. You should also disable your cluster's client certificate and static password authentication methods.

The client certificate and static password can only be retrieved by a user with the container.clusters.getCredentials permission. Note that the roles/container.admin, roles/owner, and roles/editor roles all have this permission, so use those roles wisely. Read more about GKE IAM roles.

Disable Authentication with a preprovisioned Client Certificate

With certificate authentication, a client presents a certificate that the API server verifies with the specified certificate authority. In GKE, client certificates are signed by the cluster root Certificate Authority.

With ABAC, the preprovisioned Client Certificate is fully authorized on the API server. But with RBAC enabled, client certificates must be granted permissions. A cluster with RBAC enabled and ABAC disabled still has this certificate, but it is effectively useless unless specifically granted permissions using RBAC RoleBindings.

To create a cluster without generating a client certificate, use the --no-issue-client-certificate flag:

gcloud container clusters create [CLUSTER_NAME] \

Currently, there is no way to remove a client certificate from an existing cluster.

Disable Authentication with a Static Password

A static password is a username and password combination that the API server validates. In GKE, these are generated for a username "admin" by default.

To create a cluster without generating a static password, use the --no-enable-basic-auth option:

gcloud container clusters create [CLUSTER_NAME] \

To update an existing cluster and remove the static password:

gcloud container clusters update [CLUSTER_NAME] \

Protect node metadata

Some practical attacks against Kubernetes rely on access to the VM's metadata server to extract the node’s credentials.

Using a minimally privileged service account as advised above is the first step to mitigate these attacks. The next step is to prevent workloads from impersonating nodes. You should disable legacy metadata server APIs and use metadata concealment to help protect potentially sensitive system metadata from workloads running on your cluster.

To learn more, see Protecting Cluster Metadata.

Automatically upgrade your nodes

Keeping the version of Kubernetes up to date is one of the simplest things you can do to improve your security. Kubernetes frequently introduces new security features and provides security patches.

In Google Kubernetes Engine, the masters are patched and upgraded for you, but the nodes remain your responsibility. You should enable node auto-upgrade to automatically receive upgrades and security patches for your node pools.

To learn more, see Auto-upgrading nodes.

If you do not want to enable node auto-upgrade, then be sure to carefully watch the Kubernetes Engine security bulletins for information on security patches.

Restrict Pod Permissions with a Pod Security Policy (Beta)

By default, Pods in Kubernetes can operate with capabilities beyond what they require. You should constrain the Pod's capabilities to only those required for that workload.

Kubernetes offers controls for restricting your Pods to execute with only their minimum required capabilities. Pod Security Policy allows you to set smart defaults for your Pods, and enforce controls you want to enable across your fleet. The policies you define should be specific to the needs of your application. The restricted-psp.yaml example policy is a good starting point.

To learn more about Pod Security Policy, see Using PodSecurityPolicies.

If you are using both network policies and Pod security policies in your cluster, see Using network policy and Pod security policy together.

Restrict Cluster Discovery RBAC Permissions

By default, Kubernetes bootstraps clusters with a permissive set of discovery ClusterRoleBindings which give broad access to information about a cluster's APIs, including those of CustomResourceDefinitions.

Users should be aware that the system:authenticated Group included in the subjects of the system:discovery and system:basic-user ClusterRoleBindings can include anyone with valid Google credentials (Gmail users, for example), and does not represent a meaningful level of security for clusters on GKE.

Those wishing to harden to their cluster's discovery APIs should consider one or more of:

  • Configure Authorized networks to restrict access to set IP ranges
  • Set up a private cluster to restrict access to a VPC
  • Curate the subjects of the default system:discovery and system:basic-user ClusterRoleBindings, e.g., rather than the Kubernetes default of allowing access to system:(un)authenticated, consider only allowing access to the system:serviceaccounts Group plus other known Users and Groups.

What's next

Оцените, насколько информация на этой странице была вам полезна:

Оставить отзыв о...

Текущей странице
Kubernetes Engine Documentation