With the speed of development in Kubernetes, there are often new security features for you to use. This page guides you through implementing our current guidance for hardening your Google Kubernetes Engine cluster. For a general overview of security topics, read the Security Overview.
Many of these recommendations, as well as other common misconfigurations, can be automatically checked using Security Health Analytics.
Disable the Kubernetes web UI (Dashboard)
You should disable the Kubernetes Web UI (Dashboard) when running on GKE.
The Kubernetes Web UI (Dashboard) is backed by a highly privileged Kubernetes Service Account. The Cloud Console provides much of the same functionality, so you don't need these permissions.
To disable the Kubernetes Web UI:
gcloud container clusters update [CLUSTER_NAME] \
--update-addons=KubernetesDashboard=DISABLED
Disable ABAC
You should disable Attribute-Based Access Control (ABAC), and instead use Role-Based Access Control (RBAC) in GKE.
In Kubernetes, RBAC is used to grant permissions to resources at the cluster and namespace level. RBAC allows you to define roles with rules containing a set of permissions. RBAC has significant security advantages and is now stable in Kubernetes, so it’s time to disable ABAC.
If you're still relying on ABAC, first review the Prerequisites for using RBAC. If you upgraded your cluster from an older version and are using ABAC, you should update your access controls configuration:
gcloud container clusters update [CLUSTER_NAME] \
--no-enable-legacy-authorization
To create a new cluster with the above recommendation:
gcloud container clusters create [CLUSTER_NAME] \
--no-enable-legacy-authorization
Restrict Traffic Among Pods with a Network Policy
By default, all Pods in a cluster can communicate with each other. You should control Pod to Pod communication as needed for your workload.
You can use Kubernetes' Network Policies to make it much more difficult for attackers to move laterally within your cluster. You can also use the Kubernetes Network Policy API to create Pod-level firewall rules. These firewall rules determine which Pods and services can access one another inside your cluster.
To enable network policy enforcement when creating a new cluster, specify the
--enable-network-policy flag:
gcloud container clusters create [CLUSTER_NAME] \
--zone=[COMPUTE_ZONE] \
--enable-network-policy
After you enable Network Policy, you have to actually define a policy. Since this is specific to your exact topology, we can’t provide a specific recommendation. The Kubernetes documentation, however, has an excellent walkthrough for a simple nginx deployment.
To learn more about Network Policies in GKE, see Setting a Cluster Network Policy.
Using NetworkPolicy and PodSecurityPolicy together
If you are using a NetworkPolicy, and you have a Pod that is subject to a PodSecurityPolicy, create an RBAC Role or ClusterRole that has permission to use the PodSecurityPolicy. Then bind the Role or ClusterRole to the Pod's service account. Granting permissions to user accounts is not sufficient in this case. For more information, see Authorizing policies.
Use Least Privilege Service Accounts for your Nodes
Each GKE node has an IAM Service Account associated with it. By default, nodes are given the Compute Engine default service account, which you can find by navigating to the IAM section of the Cloud Console. This account has broad access by default, making it useful to wide variety of applications, but it has more permissions than are required to run your Kubernetes Engine cluster. You should create and use a minimally privileged service account to run your GKE cluster instead of using the Compute Engine default service account.
GKE requires, at a minimum, the service account to have the
monitoring.viewer, monitoring.metricWriter, and logging.logWriter roles.
Read more about monitoring roles and logging roles.
The following commands create an IAM service account with the minimum permissions required to operate GKE:
gcloud iam service-accounts create [SA_NAME] \
--display-name=[SA_NAME]
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \
--role roles/logging.logWriter
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \
--role roles/monitoring.metricWriter
gcloud projects add-iam-policy-binding [PROJECT_ID] \
--member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \
--role roles/monitoring.viewer
If you use private images in Google Container Registry, you also need to grant access to those:
gcloud projects add-iam-policy-binding [PROJECT_ID] \ --member "serviceAccount:[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com" \ --role roles/storage.objectViewer
If you want another user to be able to create new clusters or node pools with this service account, you must grant them the Service Account User role on this service account:
gcloud iam service-accounts add-iam-policy-binding \ [SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com \ --member=user:[USER] \ --role=roles/iam.serviceAccountUser
If your cluster already exists, you can now create a new node pool with this new service account:
gcloud container node-pools create [NODE_POOL] \ --service-account=[SA_NAME]@[PROJECT_ID].iam.gserviceaccount.com \ --cluster=[CLUSTER_NAME]
If you need your GKE cluster to have access to other Google Cloud services, you should create an additional service account and grant your workloads access to the service account by storing its private key in a Kubernetes Secret. To learn more, refer to Authenticating to Google Cloud Platform with Service Accounts.
Reduce your Node Service Account Scopes
If you don't want to use a custom service account, an alternative recommendation is to reduce the permissions of the default node service account.
By default, your node service account has access scopes. Access scopes are the legacy method of specifying permissions for your instance. Before the existence of IAM roles, access scopes were the only mechanism for granting permissions to service accounts.
If you are not creating a separate service account for your nodes, you should limit the scopes of the node service account to reduce the possibility of a privilege escalation in an attack. This ensures that your default service account does not have permissions beyond those necessary to run your cluster. While the default scopes are limited, they may include scopes beyond the minimally required scopes needed to run your cluster.
The default scopes for the nodes in GKE are devstorage.read_only,
logging.write, monitoring, service.management.readonly, servicecontrol,
and trace.append. When setting scopes, these are specified as gke-default.
If you are accessing private images in Google Container Registry, the minimally
required scopes are only logging.write, monitoring, and
devstorage.read_only.
To create a cluster with the custom scopes, use the --scopes option:
gcloud container clusters create [CLUSTER_NAME] \
--scopes=[CUSTOM_SCOPES]
If you do not specify scopes or a custom service account, gke-default is used.
If you specify a custom service account, cloud-platform and userinfo.email
are used.
If your gcloud configuration includes container/new_scopes_behavior true,
this behavior is already enabled for all Kubernetes versions. To set the default
for your environment:
gcloud config set container/new_scopes_behavior true
To learn more about permissions for all scopes, see Google scopes.
Restrict Client Authentication Methods
There are several methods of authenticating to the Kubernetes API
server. In GKE, the supported methods are OpenID connect tokens,
x509 client certificates, and static passwords. GKE manages
authentication via gcloud for you using the OpenID connect token method,
setting up the Kubernetes configuration, getting an access token, and keeping
it up to date.
The other authentication methods, x509 certificates and static passwords, present a wider surface of attack for cluster compromise. With these other methods, credentials are generated for you when you create a new cluster. Unless your application is using these methods of authentication, you should disable them. You should use the OpenID authentication method, and disable your cluster's client certificate and static password authentication methods.
The client certificate and static password can only be retrieved by a user
with the container.clusters.getCredentials permission. Note that the
roles/container.admin, roles/owner, and roles/editor roles all have this
permission, so use those roles wisely. Read more about GKE
IAM roles.
Disable Authentication with a Client Certificate
With certificate authentication, a client presents a certificate that the API server verifies with the specified certificate authority. In GKE, client certificates are signed by the cluster root Certificate Authority.
With ABAC, client certificates can authenticate to the API server by default. But with RBAC enabled, client certificates must be granted permissions. A cluster with RBAC enabled and ABAC disabled still has these certificates, but the certificates are effectively useless.
To create a cluster without generating a client certificate, use the
--no-issue-client-certificate flag:
gcloud container clusters create [CLUSTER_NAME] \
--no-issue-client-certificate
Currently, there is no way to remove a client certificate from an existing cluster.
Disable Authentication with a Static Password
A static password is a username and password combination that the API server validates. In GKE, these are generated for a username "admin" by default.
To create a cluster without generating a static password, use the
--no-enable-basic-auth option:
gcloud container clusters create [CLUSTER_NAME] \
--no-enable-basic-auth
To update an existing cluster and remove the static password:
gcloud container clusters update [CLUSTER_NAME] \
--no-enable-basic-auth
Protect node metadata
Some practical attacks against Kubernetes rely on access to the VM's metadata server to extract the node’s credentials.
Using a minimally privileged service account as advised above is the first step to mitigate these attacks. The next step is to prevent workloads from impersonating nodes. You should disable legacy metadata server APIs and use metadata concealment to help protect potentially sensitive system metadata from workloads running on your cluster.
To learn more, see Protecting Cluster Metadata.
Automatically upgrade your nodes
Keeping the version of Kubernetes up to date is one of the simplest things you can do to improve your security. Kubernetes frequently introduces new security features and provides security patches.
In Google Kubernetes Engine, the masters are patched and upgraded for you, but the nodes remain your responsibility. You should enable node auto-upgrade to automatically receive upgrades and security patches for your node pools.
To learn more, see Auto-upgrading nodes.
If you do not want to enable node auto-upgrade, then be sure to carefully watch the Kubernetes Engine security bulletins for information on security patches.
Restrict Pod Permissions with a Pod Security Policy (Beta)
By default, Pods in Kubernetes can operate with capabilities beyond what they require. You should constrain the Pod's capabilities to only those required for that workload.
Kubernetes offers controls for restricting your Pods to execute with only their minimum required capabilities. Pod Security Policy allows you to set smart defaults for your Pods, and enforce controls you want to enable across your fleet. The policies you define should be specific to the needs of your application. The restricted-psp.yaml example policy is a good starting point.
To learn more about Pod Security Policy, see Using PodSecurityPolicies.
If you are using both network policies and Pod security policies in your cluster, see Using network policy and Pod security policy together.
Restrict Cluster Discovery RBAC Permissions
By default, Kubernetes bootstraps clusters with a permissive set of discovery ClusterRoleBindings which give broad access to information about a cluster's APIs, including those of CustomResourceDefinitions.
Users should be aware that the system:authenticated Group included in the
subjects of the system:discovery and system:basic-user ClusterRoleBindings
can include anyone with valid Google credentials (Gmail users, for example), and does
not represent a meaningful level of security for clusters on
GKE.
Those wishing to harden to their cluster's discovery APIs should consider one or more of:
- Configure Authorized networks to restrict access to set IP ranges
- Set up a private cluster to restrict access to a VPC
- Curate the subjects of the default
system:discoveryandsystem:basic-userClusterRoleBindings, e.g., rather than the Kubernetes default of allowing access tosystem:(un)authenticated, consider only allowing access to thesystem:serviceaccountsGroup plus other known Users and Groups.
