Authenticate workloads to other workloads over mTLS

This document describes how to set up automatic provisioning and lifecycle management of managed workload identities for Compute Engine. You configure CA pools to issue certificates using Certificate Authority Service (CA), which is a highly-available, scalable Google Cloud service that simplifies and automates the deployment, management, and security of CA services. Each VM is provisioned with X.509 credentials from the configured CA pool. These credentials can then be used to establish mTLS connections.

With managed workload identities for Compute Engine, you can implement mutually authenticated and encrypted communications between any two Compute Engine VMs. Workload applications running on the configured VMs can use the X.509 credentials for per-VM mTLS. These mTLS certificates are automatically rotated and managed for you by Certificate Authority Service.

Before you begin

  • Request access to the managed workload identity preview.

  • Configure the Google Cloud CLI.

    Install the Google Cloud CLI, then initialize it by running the following command: 

    gcloud init

  • Configure Google Cloud CLI to use the allowlisted project for billing and quota.

      gcloud config set billing/quota_project PROJECT_ID

    Replace PROJECT_ID with the ID of the project that was added to the allowlist for the managed workload identity preview.

  • Review the Managed workload identities overview documentation.

  • Enable the Compute Engine API: 

    gcloud services enable compute.googleapis.com

Required roles

To get the permissions that you need to create VMs that use managed workload identity certificates for authentication to other workloads, ask your administrator to grant you the following IAM roles on the project:

For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Overview

To use managed workload identities for your applications, you must perform the following tasks:

  1. Security Administrator:

  2. Compute Administrator:

Configure managed workload identities in Identity and Access Management

  • Follow the instructions in Configure managed workload identities authentication .

    These instructions detail how to complete the following:

    • Create a workload identity pool.
    • Create namespaces in the workload identity pool. You use the namespaces to create administrative boundaries for your managed workload identities, for example, a namespace for each of the applications owned by your organization.
    • Create a managed workload identity in a namespace in the workload identity pool. For example, you might create a namespace for an application and create managed identities within that namespace for the microservices that support that application.
    • Create a service account. Compute Engine VMs can be authorized to receive a managed workload identity based on the Google Cloud service account attached to the VM.
    • Create a workload attestation policy that allows your workload to be issued credentials for the managed workload identity. To be issues credentials for the managed workload identity, the workload must be within a specified project and have the service account attached.
    • Configure Certificate Authority Service to issue certificates for managed workload identities:
      • Configure the root CA pool
      • Configure the subordinate CAs
      • Authorize managed workload identities to request certificates from the CA pool

Get the configuration file to upload the partner metadata

Your security administrator creates a JSON file that contains the following:

This file should be named CONFIGS.json. You use this file when creating an instance template for MIGs or when creating an individual VM.

The CONFIGS.json file should be similar to the following:

  {
  "wc.compute.googleapis.com": {
     "entries": {
        "certificate-issuance-config": {
           "primary_certificate_authority_config": {
              "certificate_authority_config": {
                 "ca_pool": "projects/PROJECT_ID/locations/SUBORDINATE_CA_POOL_REGION/caPools/SUBORDINATE_CA_POOL_ID"
              }
           },
           "key_algorithm": "rsa-2048"
        },
        "trust-config": {
           "POOL_ID.global.PROJECT_NUMBER.workload.id.goog": {
               "trust_anchors": [{
                  "ca_pool": "projects/PROJECT_ID/locations/SUBORDINATE_CA_POOL_REGION/caPools/SUBORDINATE_CA_POOL_ID"
                }]
           }
     }
  }
  },
  "iam.googleapis.com": {
     "entries": {
        "workload-identity": "spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog/ns/NAMESPACE_ID/sa/MANAGED_IDENTITY_ID"
     }
  }
  }

Enable managed workload identities for a Managed Instance Group (MIG)

A managed instance group (MIG) is a group of virtual machine (VM) instances that you treat as a single entity. Each VM in a MIG is created using an instance template. To enable the VMs in the MIG to use managed workload identities, you specify the configuration in the instance template.

Create an instance template

Create an instance template with the managed workload identities feature enabled. Then use this template to create a managed instance group (MIG).

gcloud

Use the gcloud beta compute instance-templates create command to create a new instance template that enables managed workload identities.

gcloud beta compute instance-templates create INSTANCE_TEMPLATE_NAME \
    --service-account SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \
    --metadata enable-workload-certificate=true \
    --partner-metadata-from-file CONFIGS.json

You can add additional flags when creating the instance template to customize the VMs it creates, such as specifying the machine type and image, instead of using the default values.

Replace the following:

  • INSTANCE_TEMPLATE_NAME: the name for the new template.
  • SERVICE_ACCOUNT_NAME: the name of the service account that is allowed to receive the managed identity.
  • PROJECT_ID: the ID of the project where the service account was created.
  • CONFIGS.json: The configuration file containing the certificate issuance config, the trust config, and the managed workload identity.

For more information, see Create instance templates.

Create a managed instance group from the template

Create a managed instance group that uses an instance template that enables the managed workload identities. For details on how to create the instance template, see Create an instance template.

gcloud

Create a MIG using the instance template and the gcloud compute instance-groups managed create command.

gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \
    --size=SIZE \
    --template=INSTANCE_TEMPLATE_NAME \
    --zone=ZONE

Replace the following:

  • INSTANCE_GROUP_NAME: a unique ID for the managed instance group. For details on valid names, see Name resources.
  • SIZE: the size of the managed instance group
  • INSTANCE_TEMPLATE_NAME: the name of the instance template to use when creating VMs in the MIG.
  • ZONE: the zone to create the VMs in

For detailed information about creating MIGs, see Basic scenarios for creating managed instance groups (MIGs)

Enable managed workload identities for individual VMs

You can enable managed workload identities for a VM either when creating the VM or by updating the partner metadata for an existing VM.

Create VMs with managed workload identities enabled

When creating a VM, to enable the managed workload identities feature for the VM, you must do the following:

  • Specify a service account for the VM to use
  • Set the enable-workload-certificate metadata attribute to true

  • Specify the certificate issuance config and trust config information as partner metadata.

gcloud

Use the gcloud beta compute instances create command to create a new VM. Use the CONFIGS.json file supplied by your security administrator, or created by following the instructions in Create a configuration file to upload the partner metadata.

  1. Create a VM with the managed workload identities feature enabled.

    gcloud beta compute instances create INSTANCE_NAME \
   --zone=INSTANCE_ZONE \
   --service-account SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \
   --metadata enable-workload-certificate=true \
   --partner-metadata-from-file CONFIGS.json

    You can add additional lines to the command to configure the VM, such as the machine type and image, instead of using the default values. For more information, see Create and start a VM instance.

    Replace the following:

    • INSTANCE_NAME: a unique name for the VM. For details on valid instance names, see Name resources.
    • INSTANCE_ZONE: the zone to create the VM in.
    • SERVICE_ACCOUNT_NAME: the name of the service account that is allowed to receive the managed identity.
    • PROJECT_ID: the ID of the project where the service account was created.
    • CONFIGS.json: The name of the configuration file that contains the certificate issuance config, the trust config, and the managed workload identity configuration.

Enable managed workload identities on existing VMs

To enable managed workload identities for an existing VM, update the VM to configure the following:

  • If the VM doesn't already have an attached service account, then create and attach a service account to the VM.
  • Set the enable-workload-certificate metadata attribute to true.
  • Specify the certificate issuance config and trust config information as partner metadata.

  • Restart the VM.

gcloud

This task uses the CONFIGS.json file supplied by your security administrator, or created by following the instructions in Create a configuration file to upload the partner metadata.

  1. If the VM doesn't already have an attached service account, attach the service account to the VM.

  2. Update the metadata for an existing VM to enable managed workload identities.

    gcloud beta compute instances add-metadata VM_NAME \
   --zone=ZONE \
   --metadata enable-workload-certificate=true

    Replace the following:

    • VM_NAME: the name of the VM
    • ZONE: the zone where the VM is located

  3. Update the configuration for an existing VM to add the config file.

    gcloud beta compute instances update VM_NAME \
   --zone=ZONE \
   --partner-metadata-from-file CONFIGS.json

    Replace the following:

    • VM_NAME: the name of the VM
    • ZONE: the zone where the VM is located
    • CONFIGS.json: The configuration file containing the certificate issuance config, the trust config, and the managed workload identity.

  4. Stop the VM.

    gcloud beta compute instances stop VM_NAME \
   --zone=ZONE

    Replace the following:

    • VM_NAME: the name of the VM
    • ZONE: the zone where the VM is located.

  5. Start the VM.

    gcloud beta compute instances start VM_NAME \
   --zone=ZONE

    Replace the following:

    • VM_NAME: the name of the VM
    • ZONE: the zone where the VM is located.

Access workload credentials on a Linux VM

After successfully configuring workload to workload authentication using mTLS, you can access the issued credentials on your VM.

There are two ways to access the Compute Engine managed workload identity credentials and the associated trust bundle:

  • The file system on the VM
  • The Compute Engine metadata server

Access the workload credentials and trust bundle using the file system on the VM

This method puts the X.509 credentials and the trust bundle at a specific path inside the VM's file system. Applications can directly read the credentials and the trust bundle from the file system. For examples of how to retrieve the credentials, see the following examples on GitHub:

The VM must run the Compute Engine guest agent version 20231103.01 or later. Use the following command to check the version of the Compute Engine guest agent on your VM:

gcloud beta compute instances get-serial-port-output INSTANCE_NAME \
   --zone=ZONE | grep "GCE Agent Started"

If the version of the guest agent is less than 20231103.01, you can update it by following the instructions in Updating the guest environment.

To make the workload credentials and trust bundle available in the file system of a VM, complete the following steps:

  1. Install or update the Compute Engine guest agent to version 20231103.01 or later. The guest agent does the following:

    • Automatically retrieves the credentials and the trust bundle from the Compute Engine metadata server.
    • Ensures atomic writes to the file system while updating the X.509 certificate and the corresponding private key.
    • Automatically refreshes the credentials and the trust bundle, for example, when the mTLS certificates are rotated.

  2. After you install or update the Compute Engine guest agent on the guest OS, the workload refresh job creates the directory /var/run/secrets/workload-spiffe-credentials and sets the directory permissions to 0755 (rwxr-xr-x).

    The directory contains the following files created with 0644 (rw-r--r--) permissions:

    • private_key.pem: a PEM-formatted private key
    • certificates.pem: a bundle of PEM-formatted X.509 certificates that can be presented to other VMs as the client certificate chain, or used as a server certificate chain.

    • ca_certificates.pem: a bundle of PEM-formatted X.509 certificates to use as trust anchors when validating the certificates of peers.

      spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog

    • config_status: a log file containing error messages.

  3. Applications can read the certificates, private key and the trust bundle from the file system directly to establish mTLS connections.

Access the workload credentials and trust bundle using the metadata server

An application running on a Compute Engine VM can directly query the metadata server endpoints and retrieve the credentials and the trust bundle. The application is responsible for periodically checking the metadata server endpoints for new credentials and updates to the trust bundle.

The Compute Engine metadata server exposes three HTTP endpoints to enable the use of the managed workload identities feature by applications running inside the VM.

  • gce-workload-certificates/config-status: An endpoint containing any errors in the configuration values provided through the VM metadata.
  • gce-workload-certificates/workload-identities: An endpoint of identities managed by the Compute Engine control plane. This endpoint contains the X.509 certificate and the private key for the VM's trust domain.
  • gce-workload-certificates/trust-anchors: An endpoint containing a set of trusted certificates for peer X.509 certificate chain validation.

To learn more about querying the metadata for a VM instance, see About VM metadata.

To access the workload credentials and trust bundle using the metadata server, your application should do the following:

  1. Query the gce-workload-certificates/config-status endpoint. Ensure that the HTTP response code is 200 and that the response doesn't contain any partnerMetadataConfigsErrors errors. If such errors exist, update the appropriate configuration with valid values by following the steps discussed in Update certificate issuance and trust config.

    To check the value, you can run the following command on the VM:

    curl "http://metadata.google.internal/computeMetadata/v1/instance/gce-workload-certificates/config-status" -H "Metadata-Flavor: Google"

    The config-status endpoint returns a JSON response with following structure:

    {
    "partnerMetadataConfigsErrors": {
        "errors": {  // A map of errors keyed by attribute name.
            "ATTRIBUTE_NAME" : "ERROR_DETAILS",
            ...
        }
    }
}

  2. Query the gce-workload-certificates/workload-identities endpoint. Ensure that the HTTP response code is 200. The endpoint returns a JSON response with following structure:

    {
 "workloadCredentials": {  // Credentials for the VM's trust domains
   "spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog/ns/NAMESPACE_ID/sa/MANAGED_IDENTITY_ID": {
      "certificatePem" : "X.509 certificate or certificate chain",
      "privateKeyPem" : "Private for X.509 leaf certificate"
   }
 }
}

    Extract the certificatePem and the privateKeyPem. It is critical that both values are read from the same response to avoid mismatch between the private and public key in case the managed workload identities were refreshed by the Compute Engine infrastructure.

  3. Query the gce-workload-certificates/trust-anchors endpoint. Ensure that the HTTP response code is 200. The response will only contain the trust anchors for the SPIFFE trust domain, if specified. Otherwise, the query returns an error. The trust-anchors endpoint returns a JSON response with following structure:

    {
    "trustAnchors": {  // Trust bundle for the VM's trust domains
        "POOL_ID.global.PROJECT_NUMBER.workload.id.goog": {
            "trustAnchorsPem" : "Trust bundle containing the X.509
            roots certificates"
        }
    }
}

    The contents of trustAnchorsPem contains the trust bundle that can then be used to authenticate peer X.509 credentials when establishing a mTLS connection.

Refreshing the credentials and trust bundle

The Compute Engine control plane automatically rotates the managed workload identity credentials and the trust anchors periodically.

If your applications use the file system to access the workload credentials and trust bundle, the Compute Engine guest agent automatically refreshes the credentials and the trust bundle, for example, when the mTLS certificates are rotated.

If your applications query the metadata server, then the applications running on a VM must periodically query the metadata server endpoints to get the latest set of managed workload identity credentials and the trust bundle. Failure to do so can break applications due to certificate expiration or changes to the trust bundle, which can cause mTLS connection establishment to fail. Google recommends that applications query the metadata server for the managed workload identity credentials and the trust bundle every 5 minutes.

Update certificate issuance and trust config

You can modify the certificate issuance config and trust config for a VM that uses managed workload identities.

Update the instance template for a managed instance group

To update the certificate issuance config and trust config values in an instance template, you have to create a new template with the new values. Therefore, updating the certificate issuance config and trust config for existing managed instance groups (MIGs) is not supported.

Update individual Compute Engine VMs

To update the certificate issuance config and trust config, update the contents of the CONFIGS.json file and use the gcloud beta compute instances update command to apply the updates:

gcloud beta compute instances update INSTANCE_NAME \
    --partner-metadata-from-file FILENAME.json

Replace the following:

  • INSTANCE_NAME: The name of the VM that you are updating the config values for
  • FILENAME: The name of the modified config file, for example CONFIGS.json

Troubleshoot

To find methods for diagnosing and resolving common errors related to retrieving workload credentials, refer to the Troubleshoot workload to workload authentication documentation.

What's next