Verify GKE control plane VM integrity

This guide describes how to verify the integrity of the Compute Engine virtual machine (VM) image that Google Kubernetes Engine (GKE) uses for control plane VMs. This guide is intended for a security team that monitors the control plane logs and wants to verify the following:

  • The control plane VM booted with authentic firmware and other boot software that was cryptographically verified by secure boot and integrity monitoring.
  • The control plane VM booted from an authentic GKE OS image.

You can also perform this verification for the OS images and boot integrity of your worker nodes.

This page describes one part of a set of optional control plane features in GKE that lets you perform tasks like verifying your control plane security posture or configuring encryption and credential signing in the control plane using keys that you manage. For details, see About GKE control plane authority.

By default, Google Cloud applies various security measures to the managed control plane. This page describes optional capabilities that give you more visibility or control over the GKE control plane.

About VM integrity verification

By default, all GKE control plane instances are Shielded VMs, which are hardened VMs that use security capabilities like secure and measured boot, a virtual trusted platform module (vTPM), and UEFI firmware. All GKE nodes also enable integrity monitoring, which validates the boot sequence of each Shielded VM against a baseline "good" boot sequence. This validation returns pass or fail results for each boot sequence phase and adds those results to Cloud Logging. Integrity monitoring is enabled by default in all GKE clusters and validates the following phases:

  • Early boot sequence: from when the UEFI firmware starts until the bootloader takes control. Added to the VM logs as earlyBootReportEvent.
  • Late boot sequence: from when the bootloader takes control until the operating system kernel takes control. Added to the VM logs as lateBootReportEvent.

GKE also adds control plane VM creation logs to Logging. These logs contain metadata that identifies the machine and includes details about the VM image and the boot sequence. Google Cloud publishes a verification summary attestation (VSA) for each GKE control plane VM image in the gke-vsa repository on GitHub. The VSA uses the in-toto framework for attestations. You can validate the control plane VM logs for your clusters against the corresponding VSAs to verify that your control plane nodes booted as expected.

Performing these validations can help you to achieve the following goals:

  • Ensure that the software on the control plane is protected by secure boot and integrity monitoring, matches the intended source code, and is exactly the same as the image that other Google Cloud customers use.
  • Improve your confidence in how GKE secures the control plane.

Pricing

This feature is offered at no extra cost in GKE.

Before you begin

Before you start, make sure you have performed the following tasks:

  • Enable the Google Kubernetes Engine API.
    • Enable Google Kubernetes Engine API
  • If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.

  • Enable the Cloud Logging API.

    Enable the API

  • Ensure that you already have a GKE Autopilot mode or Standard mode cluster running version 1.29 or later.

Required roles

To get the permissions that you need to verify control plane VM integrity, ask your administrator to grant you the following IAM roles on your project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Check for failed boot sequence phases

Integrity monitoring adds a log to Logging if a control plane VM fails or successfully completes a phase of the boot sequence. To view failed boot events, run the following commands:

  1. In the Google Cloud console, go to the Logs Explorer page:

    Go to Logs Explorer

  2. In the Query field, specify the following query:

    jsonPayload.@type="type.googleapis.com/cloud_integrity.IntegrityEvent"
jsonPayload.earlyBootReportEvent.policyEvaluationPassed="false" OR jsonPayload.lateBootReportEvent.policyEvaluationPassed="false"
jsonPayload.metadata.isKubernetesControlPlaneVM="true"

    You can also check for successful boot events by replacing false with true in this query.

  3. Click Run query. If you don't see results, your control plane VMs passed all integrity monitoring checks. If you see an output, proceed to the next step to identify the corresponding cluster.

  4. In the failed boot integrity log, copy the value in the resource.labels.instance_id field.

  5. In the Query field, specify the following query:

    protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"
protoPayload.metadata.isKubernetesControlPlaneVM="true"
resource.labels.instance_id="INSTANCE_ID"
protoPayload.methodName="v1.compute.instances.insert"

    Replace INSTANCE_ID with the value of the instance_id field from the previous step.

  6. Click Run query. The value in the protoPayload.metadata.parentResource.parentResourceId field is the GKE cluster ID.

  7. Find the name of the GKE cluster:

    gcloud asset query \
    --organization=ORGANIZATION_ID \
    --statement="SELECT name FROM container_googleapis_com_Cluster WHERE resource.data.id='CLUSTER_ID';"

    Replace the following:

    • ORGANIZATION_ID: the numerical ID of your Google Cloud organization.
    • CLUSTER_ID: the value of the protoPayload.metadata.parentResource.parentResourceId field from the previous step.

    The output is similar to the following:

    # lines omitted for clarity
//container.googleapis.com/projects/PROJECT_ID/locations/LOCATION/clusters/CLUSTER_NAME

    This output has the following fields:

    • PROJECT_ID: your Google Cloud project ID.
    • LOCATION: the location of the cluster.
    • CLUSTER_NAME: the name of the cluster.

Find and inspect the control plane VM logs

The Compute Engine VM creation logs that correspond to GKE clusters are stored in the _Default log bucket. To find the creation logs for your cluster control plane VMs and retrieve this metadata, do the following:

  1. In the Google Cloud console, go to the Logs Explorer page:

    Go to Logs Explorer

  2. In the Query field, specify the following query:

    resource.type="gce_instance"
protoPayload.methodName="v1.compute.instances.insert"
protoPayload.metadata.isKubernetesControlPlaneVM="true"

  3. Click Run query. If you don't see results, check that you meet all of the requirements in the Before you begin section.

  4. In the query results, check the metadata field. The output is similar to the following:

    # fields omitted for clarity
"metadata": {
  "usedResources": {
    "attachedDisks": [
      {
        "sourceImageId": "9046093115864736653",
        "sourceImage": "https://www.googleapis.com/compute/v1/projects/1234567890/global/images/gke-1302-gke1627000-cos-113-18244-85-49-c-pre",
        "isBootDisk": true
      }
# fields omitted for clarity

    The metadata field includes the following information:

    • usedResources: the list of resources used to create the VM.
    • attachedDisks: the boot disk for the VM.
    • sourceImageId: the unique ID of the VM image.
    • sourceImage: the URL of the source VM image. The syntax of the value in this field is https://www.googleapis.com/compute/v1/projects/PROJECT_NUMBER/global/images/IMAGE_NAME, where PROJECT_NUMBER is the number of the Google Cloud-owned project that hosts the control plane VMs and IMAGE_NAME is the name of the image that was used to boot the VM.
    • isBootDisk: a boolean identifier for whether this disk was used as the boot disk for the VM.

Find and verify the VSA for control plane VM images

In this section, you find the VSA that corresponds to your control plane VM image in the gke-vsa repository on GitHub. You then use a tool named slsa-verifier provided by the Supply chain Levels for Software Artifacts (SLSA) framework to verify the VSA. You need the following data from the control plane VM creation log:

  • The VM image ID
  • The project number of the Google Cloud-owned project that hosts the VMs
  • The OS image name that was used to boot the VM

The file that corresponds to your control plane VM has the following filename format:

IMAGE_NAME:IMAGE_ID.intoto.jsonl

Replace the following:

  • IMAGE_NAME: the VM image name, which is the string after /images/ in the attachedDisks.sourceImage field in the VM audit log from the previous section. For example, gke-1302-gke1627000-cos-113-18244-85-49-c-pre.
  • IMAGE_ID: the VM image ID, which is the value of the attachedDisks.sourceImageId field in the VM audit log from the previous section. For example, 9046093115864736653.

To find and verify the VSA when you know the filename of your VSA file, perform the following steps:

  1. Open the gke-vsa GitHub repository.
  2. In the "gke-master-images" directory, find the file that corresponds to your VM image. For example, https://github.com/GoogleCloudPlatform/gke-vsa/blob/main/gke-master-images:78064567238/IMAGE_NAME:IMAGE_ID.intoto.jsonl
  3. Download the VSA file.
  4. Install the slsa-verifier tool.

  5. Save the public key for verifying the VSA to a file named vsa_signing_public_key:

    -----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEeGa6ZCZn0q6WpaUwJrSk+PPYEsca
3Xkk3UrxvbQtoZzTmq0zIYq+4QQl0YBedSyy+XcwAMaUWTouTrB05WhYtg==
-----END PUBLIC KEY-----

  6. Verify the VSA:

    slsa-verifier verify-vsa \
    --attestation-path=PATH_TO_VSA_FILE \
    --resource-uri=gce_image://gke-master-images:IMAGE_NAME \
    --subject-digest=gce_image_id:IMAGE_ID\
    --verifier-id=https://bcid.corp.google.com/verifier/bcid_package_enforcer/v0.1 \
    --verified-level=BCID_L1 \
    --verified-level=SLSA_BUILD_LEVEL_2 \
    --public-key-path=PATH_TO_PUBLIC_KEY_FILE \
    --public-key-id=keystore://76574:prod:vsa_signing_public_key

    Replace the following:

    • PATH_TO_VSA_FILE: the path to the VSA file that you downloaded.
    • IMAGE_NAME: the name of the VM image, like gke-1302-gke1627000-cos-113-18244-85-49-c-pre.
    • IMAGE_ID: the VM image ID, like 9046093115864736653.

    If the VSA passes the verification checks, the output is the following:

    Verifying VSA: PASSED
PASSED: SLSA verification passed

What's next