Kubernetes Engine uses instance metadata to configure node VMs, but some of this metadata can be used to attack a cluster, and so should be concealed from workloads running on the cluster that don't need access to it.
This page explains how to use Kubernetes Engine's metadata concealment feature to protect potentially sensitive system metadata from user workloads running on your cluster.
Beginning with Kubernetes version 1.9.3, you can enable metadata
concealment to prevent user Pods from accessing certain VM metadata for your
cluster's nodes, such as Kubelet credentials and VM instance information.
Specifically, metadata concealment protects access to
kube-env (which contains
Kubelet credentials) and the VM's instance identity token.
Metadata concealment firewalls traffic from user Pods (Pods not running on
HostNetwork) to the cluster metadata server, only allowing safe queries. The
firewall prevents user Pods from using Kubelet credentials for privilege
escalation attacks, or from using VM identity for instance escalation attacks.
Before you begin
To prepare for this task, perform the following steps:
- Ensure that you have installed the Cloud SDK.
- Set your default project ID:
gcloud config set project [PROJECT_ID]
- Set your default compute zone:
gcloud config set compute/zone [COMPUTE_ZONE]
gcloud to use the
To use this feature with
gcloud, you must enable the
v1beta1 API surface for
which allows you to run
gcloud beta container clusters commands.
To configure the
gcloud command-line tool to use the
v1beta1 API, run one of
the following commands in your shell or terminal window:
gcloud config set container/use_v1_api false
Configure node service account
Because each node's service account credentials will continue to be exposed to workloads, you should ensure that you have configured a service account with the minimal permissions that you need. You will attach this service account to your nodes in below. This way, an attacker cannot circumvent Kubernetes Engine's metadata concealment by using the Compute Engine API to access the node instances directly.
Do not use a service account that has
compute.instances.get permission, the
Compute Instance Admin role, or other similar permissions, as these permissions
allow potential attackers to obtain instance metadata using the Compute Engine
API. The best practice is to restrict the permissions of a node VM by using
service account permissions, not scopes. See Compute Engine's
service accounts documentation for more information.
If you don't have a node service account, you can create one using the following commands:
export NODE_SA_NAME=gke-node-sa gcloud iam service-accounts create $NODE_SA_NAME \ --display-name "Node Service Account" export NODE_SA_EMAIL=`gcloud iam service-accounts list --format='value(email)' \ --filter='displayName:Node Service Account'`
To configure your service account with the necessary roles and permissions,
run the following commands.
PROJECT is your project ID:
export PROJECT=`gcloud config get-value project` gcloud projects add-iam-policy-binding $PROJECT \ --member serviceAccount:$NODE_SA_EMAIL \ --role roles/monitoring.metricWriter gcloud projects add-iam-policy-binding $PROJECT \ --member serviceAccount:$NODE_SA_EMAIL \ --role roles/monitoring.viewer gcloud projects add-iam-policy-binding $PROJECT \ --member serviceAccount:$NODE_SA_EMAIL \ --role roles/logging.logWriter
Creating a new cluster or node pool with metadata concealment
After creating a service account, you can create a new cluster with metadata
concealment enabled by using the
gcloud command-line tool.
To create a cluster with metadata concealment enabled, run the following command in your shell or terminal window:
gcloud beta container clusters create [CLUSTER_NAME] \ --workload-metadata-from-node=SECURE \ --service-account=$NODE_SA_EMAIL \ --cluster-version=1.9 <additional parameters and flags omitted>
[CLUSTER_NAME]is the name of the cluster to be created.
--workload-metadata-from-nodeis set to
SECURE; setting the flag to
UNSPECIFIEDdisables metadata concealment.