Protecting cluster metadata


Overview

Google Kubernetes Engine (GKE) uses instance metadata to configure node virtual machines (VMs), but some of this metadata is potentially sensitive and should be protected from workloads running on the cluster.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Ensure that you have enabled the Google Kubernetes Engine API.
  • Enable Google Kubernetes Engine API
  • Ensure that you have installed the Cloud SDK.
  • Set up default gcloud command-line tool settings for your project by using one of the following methods:
    • Use gcloud init, if you want to be walked through setting project defaults.
    • Use gcloud config, to individually set your project ID, zone, and region.

    gcloud init

    1. Run gcloud init and follow the directions:

      gcloud init

      If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

      gcloud init --console-only
    2. Follow the instructions to authorize the gcloud tool to use your Google Cloud account.
    3. Create a new configuration or select an existing one.
    4. Choose a Google Cloud project.
    5. Choose a default Compute Engine zone.
    6. Choose a default Compute Engine region.

    gcloud config

    1. Set your default project ID:
      gcloud config set project PROJECT_ID
    2. Set your default Compute Engine region (for example, us-central1):
      gcloud config set compute/region COMPUTE_REGION
    3. Set your default Compute Engine zone (for example, us-central1-c):
      gcloud config set compute/zone COMPUTE_ZONE
    4. Update gcloud to the latest version:
      gcloud components update

    By setting default locations, you can avoid errors in gcloud tool like the following: One of [--zone, --region] must be supplied: Please specify location.

Configure node service account

Each node's service account credentials continue to be exposed to workloads. By default, your nodes use the Compute Engine default service account. You should configure a minimally-privileged service account for your nodes to use instead of the Compute Engine default service account. Then, attach this service account to your nodes, so that an attacker cannot circumvent GKE metadata protections by using the Compute Engine API to access the underlying VM instances directly.

For more information, refer to Use least privilege node service accounts.

To create a minimally privileged node service account, perform the following steps:

  1. Create a new Identity and Access Management (IAM) service account and save the email address in an environment variable:

    gcloud iam service-accounts create NODE_SA_NAME \
        --display-name="DISPLAY_NAME"
    export NODE_SA_EMAIL=$(gcloud iam service-accounts list --format='value(email)' \
        --filter='displayName:DISPLAY_NAME')
    

    Replace the following:

    • NODE_SA_NAME: the name of your new node service account.
    • DISPLAY_NAME: the display name of the new service account.

    The node service account email address has the format NODE_SA_NAME@PROJECT_ID.iam.gserviceaccount.com.

  2. Configure your service account with the minimum roles and permissions to run your GKE nodes:

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:$NODE_SA_EMAIL \
        --role=roles/monitoring.metricWriter
    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:$NODE_SA_EMAIL \
        --role=roles/monitoring.viewer
    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:$NODE_SA_EMAIL \
        --role=roles/logging.logWriter
    

    Replace PROJECT_ID with your Google Cloud project ID.

    Additionally, if your cluster pulls private images from Artifact Registry, add the roles/artifactregistry.reader role:

    gcloud projects add-iam-policy-binding PROJECT_ID \
        --member=serviceAccount:$NODE_SA_EMAIL \
        --role=roles/artifactregistry.reader
    

Metadata concealment

GKE metadata concealment protects some potentially sensitive system metadata from user workloads running on your cluster.

In Kubernetes v1.9.3 and higher, you can enable metadata concealment to prevent user Pods from accessing certain VM metadata in your cluster's nodes, such as kubelet credentials and VM instance information. Specifically, metadata concealment protects access to kube-env (which contains Kubelet credentials) and the VM's instance identity token.

Metadata concealment firewalls traffic from user Pods (Pods not running on HostNetwork) to the cluster metadata server, only allowing safe queries. The firewall prevents user Pods from using kubelet credentials for privilege escalation attacks, or from using VM identity for instance escalation attacks.

You can only enable metadata concealment when creating a new cluster, or when adding a new node pool to an existing cluster.

Limitations

  • Metadata concealment only protects access to kube-env and the node's instance identity token.
  • Metadata concealment does not restrict access to the node's service account.
  • Metadata concealment does not restrict access to other related instance metadata.
  • Metadata concealment does not restrict access to other legacy metadata APIs.

Creating a new cluster or node pool with metadata concealment

After creating a service account, you can create a new cluster with metadata concealment enabled by using the gcloud command-line tool.

To create a cluster with metadata concealment enabled, run the following command:

gcloud beta container clusters create CLUSTER_NAME \
  --workload-metadata-from-node=SECURE \
  --service-account=$NODE_SA_EMAIL

Replace CLUSTER_NAME with the name of your new cluster.

The --workload-metadata-from-node flag takes the following values:

  • SECURE: enable metadata concealment.
  • EXPOSED or UNSPECIFIED: disable metadata concealment.

Verifying identity token metadata concealed from cluster's workload

When you conceal metadata, it should not be possible to request a signature via the node's instance identity token. To verify that requests explicitly inform users of concealed metadata, you can run a curl command from within a Pod:

curl -H "Metadata-Flavor: Google" \
'http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=https://www.example.com'

The output is similar to the following:

This metadata endpoint is concealed.

Disabling and transitioning from legacy metadata APIs

The v0.1 and v1beta1 Compute Engine metadata server endpoints were deprecated and shutdown on September 30, 2020.

For the shutdown schedule, refer to v0.1 and v1beta1 metadata server endpoints deprecation.

What's next