Protecting Cluster Metadata

Kubernetes Engine uses instance metadata to configure node VMs, but some of this metadata can be used to attack a cluster, and so should be concealed from workloads running on the cluster that don't need access to it.

This page explains how to use Kubernetes Engine's metadata concealment feature to protect only potentially sensitive system metadata from user workloads running on your cluster.

Overview

Beginning with Kubernetes version 1.9.3, you can enable metadata concealment to prevent user Pods from accessing certain VM metadata for your cluster's nodes, such as Kubelet credentials and VM instance information. Specifically, metadata concealment protects access to kube-env (which contains Kubelet credentials) and the VM's instance identity token.

Metadata concealment firewalls traffic from user Pods (Pods not running on HostNetwork) to the cluster metadata server, only allowing safe queries. The firewall prevents user Pods from using Kubelet credentials for privilege escalation attacks, or from using VM identity for instance escalation attacks.

Limitations

  • Metadata concealment only protects access to kube-env and the node's instance identity token
  • Metadata concealment does not restrict access to the node's service account
  • Metadata concealment does not restrict access to other related instance metadata

Before you begin

To prepare for this task, perform the following steps:

  • Ensure that you have installed the Cloud SDK.
  • Set your default project ID:
    gcloud config set project [PROJECT_ID]
  • Set your default compute zone:
    gcloud config set compute/zone [COMPUTE_ZONE]
  • Update all gcloud commands to the latest version:
    gcloud components update

Configure gcloud to use the v1beta1 API

To use this feature with gcloud, you must enable the v1beta1 API surface for gcloud, which allows you to run gcloud beta container clusters commands.

To configure the gcloud command-line tool to use the v1beta1 API, run one of the following commands in your shell or terminal window:

export CLOUDSDK_CONTAINER_USE_V1_API_CLIENT=false
or:
gcloud config set container/use_v1_api false

Configure node service account

Because each node's service account credentials will continue to be exposed to workloads, you should ensure that you have configured a service account with the minimal permissions it needs. Then, attach this service account to your nodes, so that an attacker cannot circumvent Kubernetes Engine's metadata concealment by using the Compute Engine API to access the node instances directly.

Do not use a service account that has compute.instances.get permission, the Compute Instance Admin role, or other similar permissions, as these permissions allow potential attackers to obtain instance metadata using the Compute Engine API. The best practice is to restrict the permissions of a node VM by using service account permissions, not access scopes. For more information, see Compute Engine's service accounts documentation.

gcloud

If you don't have a node service account, you can create one using the following commands:

export NODE_SA_NAME=gke-node-sa
gcloud iam service-accounts create $NODE_SA_NAME \
  --display-name "Node Service Account"
export NODE_SA_EMAIL=`gcloud iam service-accounts list --format='value(email)' \
  --filter='displayName:Node Service Account'`

To configure your service account with the necessary roles and permissions, run the following commands. PROJECT is your project ID:

export PROJECT=`gcloud config get-value project`

gcloud projects add-iam-policy-binding $PROJECT \
  --member serviceAccount:$NODE_SA_EMAIL \
  --role roles/monitoring.metricWriter
gcloud projects add-iam-policy-binding $PROJECT \
  --member serviceAccount:$NODE_SA_EMAIL \
  --role roles/monitoring.viewer
gcloud projects add-iam-policy-binding $PROJECT \
  --member serviceAccount:$NODE_SA_EMAIL \
  --role roles/logging.logWriter

Additionally, if your cluster pulls private images from Container Registry, add the storage.objectviewer role:

gcloud projects add-iam-policy-binding $PROJECT \
  --member serviceAccount:$NODE_SA_EMAIL \
  --role roles/storage.objectViewer

Creating a new cluster or node pool with metadata concealment

After creating a service account, you can create a new cluster with metadata concealment enabled by using the gcloud command-line tool.

gcloud

To create a cluster with metadata concealment enabled, run the following command in your shell or terminal window:

gcloud beta container clusters create [CLUSTER_NAME] \
  --workload-metadata-from-node=SECURE \
  --service-account=$NODE_SA_EMAIL \
  --cluster-version=1.9
  <additional parameters and flags omitted>

where:

  • [CLUSTER_NAME] is the name of the cluster to be created.
  • --workload-metadata-from-node is set to SECURE; setting the flag to EXPOSED or UNSPECIFIED disables metadata concealment.

Verifying identity token metadata concealed from cluster's workload

When you conceal metadata, it should not be possible to request a signature via the node's instance identity token. To verify that requests explicitly inform users of concealed metadata, you can run a curl command from

cURL

root@pod-name# curl -H "Metadata-Flavor: Google" \
'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://www.example.com'
This metadata endpoint is concealed.

Was this page helpful? Let us know how we did:

Send feedback about...

Kubernetes Engine