This document describes how to set up automatic provisioning and lifecycle management of managed workload identities for Compute Engine. You configure CA pools to issue certificates using Certificate Authority Service (CA), which is a highly-available, scalable Google Cloud service that simplifies and automates the deployment, management, and security of CA services. Each VM is provisioned with X.509 credentials from the configured CA pool. These credentials can then be used to establish mTLS connections.
With managed workload identities for Compute Engine, you can implement mutually authenticated and encrypted communications between any two Compute Engine VMs. Workload applications running on the configured VMs can use the X.509 credentials for per-VM mTLS. These mTLS certificates are automatically rotated and managed for you by Certificate Authority Service.
Before you begin
Configure the Google Cloud CLI.
Install the Google Cloud CLI, then initialize it by running the following command:
gcloud init
Configure Google Cloud CLI to use the allowlisted project for billing and quota.
gcloud config set billing/quota_project PROJECT_ID
Replace
PROJECT_ID
with the ID of the project that was added to the allowlist for the managed workload identity preview.- Review the Managed workload identities overview documentation.
-
Enable the Compute Engine API:
gcloud services enable compute.googleapis.com
Required roles
To get the permissions that you need to create VMs that use managed workload identity certificates for authentication to other workloads, ask your administrator to grant you the following IAM roles on the project:
-
Compute Instance Admin (v1) (
roles/compute.instanceAdmin.v1
) -
Service Account User (
roles/iam.serviceAccountUser
)
For more information about granting roles, see Manage access to projects, folders, and organizations.
You might also be able to get the required permissions through custom roles or other predefined roles.
Overview
To use managed workload identities for your applications, you must perform the following tasks:
Security Administrator:
- Create managed workload identities in a workload identity pool.
- Define the workload attestation policy.
- Configure Certificate Authority Service to issue certificates for managed workload identities.
- Authorize managed workload identities to request certificates from the CA pool.
- Define the trust and certificate issuance config.
Compute Administrator:
- Get the configuration file to upload the partner metadata.
- Enable managed workload identities for workloads running in Compute Engine:
- Access workload credentials on a Linux VM.
Configure managed workload identities in Identity and Access Management
Follow the instructions in Configure managed workload identities authentication .
These instructions detail how to complete the following:
- Create a workload identity pool.
- Create namespaces in the workload identity pool. You use the namespaces to create administrative boundaries for your managed workload identities, for example, a namespace for each of the applications owned by your organization.
- Create a managed workload identity in a namespace in the workload identity pool. For example, you might create a namespace for an application and create managed identities within that namespace for the microservices that support that application.
- Create a service account. Compute Engine VMs can be authorized to receive a managed workload identity based on the Google Cloud service account attached to the VM.
- Create a workload attestation policy that allows your workload to be issued credentials for the managed workload identity. To be issues credentials for the managed workload identity, the workload must be within a specified project and have the service account attached.
- Configure Certificate Authority Service to issue certificates for managed workload
identities:
- Configure the root CA pool
- Configure the subordinate CAs
- Authorize managed workload identities to request certificates from the CA pool
Get the configuration file to upload the partner metadata
Your security administrator creates a JSON file that contains the following:
- The configuration for the workload identity
- The certificate issuance config
- The trust config
This file should be named CONFIGS.json
. You use this file when creating an
instance template for MIGs or when creating an individual VM.
The CONFIGS.json
file should be similar to the following:
{ "wc.compute.googleapis.com": { "entries": { "certificate-issuance-config": { "primary_certificate_authority_config": { "certificate_authority_config": { "ca_pool": "projects/PROJECT_ID/locations/SUBORDINATE_CA_POOL_REGION/caPools/SUBORDINATE_CA_POOL_ID" } }, "key_algorithm": "rsa-2048" }, "trust-config": { "POOL_ID.global.PROJECT_NUMBER.workload.id.goog": { "trust_anchors": [{ "ca_pool": "projects/PROJECT_ID/locations/SUBORDINATE_CA_POOL_REGION/caPools/SUBORDINATE_CA_POOL_ID" }] } } } }, "iam.googleapis.com": { "entries": { "workload-identity": "spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog/ns/NAMESPACE_ID/sa/MANAGED_IDENTITY_ID" } } }
Enable managed workload identities for a Managed Instance Group (MIG)
A managed instance group (MIG) is a group of virtual machine (VM) instances that you treat as a single entity. Each VM in a MIG is created using an instance template. To enable the VMs in the MIG to use managed workload identities, you specify the configuration in the instance template.
Create an instance template
Create an instance template with the managed workload identities feature enabled. Then use this template to create a managed instance group (MIG).
gcloud
Use the gcloud beta compute instance-templates create
command to
create a new instance template that enables managed workload identities.
gcloud beta compute instance-templates create INSTANCE_TEMPLATE_NAME \ --service-account SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \ --metadata enable-workload-certificate=true \ --partner-metadata-from-file CONFIGS.json
You can add additional flags when creating the instance template to customize the VMs it creates, such as specifying the machine type and image, instead of using the default values.
Replace the following:
- INSTANCE_TEMPLATE_NAME: the name for the new template.
- SERVICE_ACCOUNT_NAME: the name of the service account that is allowed to receive the managed identity.
- PROJECT_ID: the ID of the project where the service account was created.
- CONFIGS.json: The configuration file containing the certificate issuance config, the trust config, and the managed workload identity.
For more information, see Create instance templates.
Create a managed instance group from the template
Create a managed instance group that uses an instance template that enables the managed workload identities. For details on how to create the instance template, see Create an instance template.
gcloud
Create a MIG using the instance template and the
gcloud compute instance-groups managed create
command.
gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \ --size=SIZE \ --template=INSTANCE_TEMPLATE_NAME \ --zone=ZONE
Replace the following:
- INSTANCE_GROUP_NAME: a unique ID for the managed instance group. For details on valid names, see Name resources.
- SIZE: the size of the managed instance group
- INSTANCE_TEMPLATE_NAME: the name of the instance template to use when creating VMs in the MIG.
- ZONE: the zone to create the VMs in
For detailed information about creating MIGs, see Basic scenarios for creating managed instance groups (MIGs)
Enable managed workload identities for individual VMs
You can enable managed workload identities for a VM either when creating the VM or by updating the partner metadata for an existing VM.
Create VMs with managed workload identities enabled
When creating a VM, to enable the managed workload identities feature for the VM, you must do the following:
- Specify a service account for the VM to use
- Set the
enable-workload-certificate
metadata attribute totrue
Specify the certificate issuance config and trust config information as partner metadata.
gcloud
Use the gcloud beta compute instances create
command
to create a new VM. Use the CONFIGS.json
file supplied by your security
administrator, or created by following the instructions in
Create a configuration file to upload the partner metadata.
Create a VM with the managed workload identities feature enabled.
gcloud beta compute instances create INSTANCE_NAME \ --zone=INSTANCE_ZONE \ --service-account SERVICE_ACCOUNT_NAME@PROJECT_ID.iam.gserviceaccount.com \ --metadata enable-workload-certificate=true \ --partner-metadata-from-file CONFIGS.json
You can add additional lines to the command to configure the VM, such as the machine type and image, instead of using the default values. For more information, see Create and start a VM instance.
Replace the following:
- INSTANCE_NAME: a unique name for the VM. For details on valid instance names, see Name resources.
- INSTANCE_ZONE: the zone to create the VM in.
- SERVICE_ACCOUNT_NAME: the name of the service account that is allowed to receive the managed identity.
- PROJECT_ID: the ID of the project where the service account was created.
- CONFIGS.json: The name of the configuration file that contains the certificate issuance config, the trust config, and the managed workload identity configuration.
Enable managed workload identities on existing VMs
To enable managed workload identities for an existing VM, update the VM to configure the following:
- If the VM doesn't already have an attached service account, then create and attach a service account to the VM.
- Set the
enable-workload-certificate
metadata attribute totrue
. - Specify the certificate issuance config and trust config information as partner metadata.
Restart the VM.
gcloud
This task uses the CONFIGS.json
file supplied by your security
administrator, or created by following the instructions in
Create a configuration file to upload the partner metadata.
If the VM doesn't already have an attached service account, attach the service account to the VM.
Update the metadata for an existing VM to enable managed workload identities.
gcloud beta compute instances add-metadata VM_NAME \ --zone=ZONE \ --metadata enable-workload-certificate=true
Replace the following:
- VM_NAME: the name of the VM
- ZONE: the zone where the VM is located
Update the configuration for an existing VM to add the config file.
gcloud beta compute instances update VM_NAME \ --zone=ZONE \ --partner-metadata-from-file CONFIGS.json
Replace the following:
- VM_NAME: the name of the VM
- ZONE: the zone where the VM is located
- CONFIGS.json: The configuration file containing the certificate issuance config, the trust config, and the managed workload identity.
Stop the VM.
gcloud beta compute instances stop VM_NAME \ --zone=ZONE
Replace the following:
- VM_NAME: the name of the VM
- ZONE: the zone where the VM is located.
Start the VM.
gcloud beta compute instances start VM_NAME \ --zone=ZONE
Replace the following:
- VM_NAME: the name of the VM
- ZONE: the zone where the VM is located.
Access workload credentials on a Linux VM
After successfully configuring workload to workload authentication using mTLS, you can access the issued credentials on your VM.
There are two ways to access the Compute Engine managed workload identity credentials and the associated trust bundle:
- The file system on the VM
- The Compute Engine metadata server
Access the workload credentials and trust bundle using the file system on the VM
This method puts the X.509 credentials and the trust bundle at a specific path inside the VM's file system. Applications can directly read the credentials and the trust bundle from the file system. For examples of how to retrieve the credentials, see the following examples on GitHub:
The VM must run the Compute Engine guest agent version 20231103.01 or later. Use the following command to check the version of the Compute Engine guest agent on your VM:
gcloud beta compute instances get-serial-port-output INSTANCE_NAME \ --zone=ZONE | grep "GCE Agent Started"
If the version of the guest agent is less than 20231103.01, you can update it by following the instructions in Updating the guest environment.
To make the workload credentials and trust bundle available in the file system of a VM, complete the following steps:
Install or update the Compute Engine guest agent to version 20231103.01 or later. The guest agent does the following:
- Automatically retrieves the credentials and the trust bundle from the Compute Engine metadata server.
- Ensures atomic writes to the file system while updating the X.509 certificate and the corresponding private key.
- Automatically refreshes the credentials and the trust bundle, for example, when the mTLS certificates are rotated.
After you install or update the Compute Engine guest agent on the guest OS, the workload refresh job creates the directory
/var/run/secrets/workload-spiffe-credentials
and sets the directory permissions to0755 (rwxr-xr-x)
.The directory contains the following files created with
0644 (rw-r--r--)
permissions:private_key.pem
: a PEM-formatted private keycertificates.pem
: a bundle of PEM-formatted X.509 certificates that can be presented to other VMs as the client certificate chain, or used as a server certificate chain.ca_certificates.pem
: a bundle of PEM-formatted X.509 certificates to use as trust anchors when validating the certificates of peers.spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog
config_status
: a log file containing error messages.
Applications can read the certificates, private key and the trust bundle from the file system directly to establish mTLS connections.
Access the workload credentials and trust bundle using the metadata server
An application running on a Compute Engine VM can directly query the metadata server endpoints and retrieve the credentials and the trust bundle. The application is responsible for periodically checking the metadata server endpoints for new credentials and updates to the trust bundle.
The Compute Engine metadata server exposes three HTTP endpoints to enable the use of the managed workload identities feature by applications running inside the VM.
gce-workload-certificates/config-status
: An endpoint containing any errors in the configuration values provided through the VM metadata.gce-workload-certificates/workload-identities
: An endpoint of identities managed by the Compute Engine control plane. This endpoint contains the X.509 certificate and the private key for the VM's trust domain.gce-workload-certificates/trust-anchors
: An endpoint containing a set of trusted certificates for peer X.509 certificate chain validation.
To learn more about querying the metadata for a VM instance, see About VM metadata.
To access the workload credentials and trust bundle using the metadata server, your application should do the following:
Query the
gce-workload-certificates/config-status
endpoint. Ensure that the HTTP response code is200
and that the response doesn't contain anypartnerMetadataConfigsErrors
errors. If such errors exist, update the appropriate configuration with valid values by following the steps discussed in Update certificate issuance and trust config.To check the value, you can run the following command on the VM:
curl "http://metadata.google.internal/computeMetadata/v1/instance/gce-workload-certificates/config-status" -H "Metadata-Flavor: Google"
The
config-status
endpoint returns a JSON response with following structure:{ "partnerMetadataConfigsErrors": { "errors": { // A map of errors keyed by attribute name. "ATTRIBUTE_NAME" : "ERROR_DETAILS", ... } } }
Query the
gce-workload-certificates/workload-identities
endpoint. Ensure that the HTTP response code is200
. The endpoint returns a JSON response with following structure:{ "workloadCredentials": { // Credentials for the VM's trust domains "spiffe://POOL_ID.global.PROJECT_NUMBER.workload.id.goog/ns/NAMESPACE_ID/sa/MANAGED_IDENTITY_ID": { "certificatePem" : "X.509 certificate or certificate chain", "privateKeyPem" : "Private for X.509 leaf certificate" } } }
Extract the
certificatePem
and theprivateKeyPem
. It is critical that both values are read from the same response to avoid mismatch between the private and public key in case the managed workload identities were refreshed by the Compute Engine infrastructure.Query the
gce-workload-certificates/trust-anchors
endpoint. Ensure that the HTTP response code is200
. The response will only contain the trust anchors for the SPIFFE trust domain, if specified. Otherwise, the query returns an error. Thetrust-anchors
endpoint returns a JSON response with following structure:{ "trustAnchors": { // Trust bundle for the VM's trust domains "POOL_ID.global.PROJECT_NUMBER.workload.id.goog": { "trustAnchorsPem" : "Trust bundle containing the X.509 roots certificates" } } }
The contents of
trustAnchorsPem
contains the trust bundle that can then be used to authenticate peer X.509 credentials when establishing a mTLS connection.
Refreshing the credentials and trust bundle
The Compute Engine control plane automatically rotates the managed workload identity credentials and the trust anchors periodically.
If your applications use the file system to access the workload credentials and trust bundle, the Compute Engine guest agent automatically refreshes the credentials and the trust bundle, for example, when the mTLS certificates are rotated.
If your applications query the metadata server, then the applications running on a VM must periodically query the metadata server endpoints to get the latest set of managed workload identity credentials and the trust bundle. Failure to do so can break applications due to certificate expiration or changes to the trust bundle, which can cause mTLS connection establishment to fail. Google recommends that applications query the metadata server for the managed workload identity credentials and the trust bundle every 5 minutes.
Update certificate issuance and trust config
You can modify the certificate issuance config and trust config for a VM that uses managed workload identities.
Update the instance template for a managed instance group
To update the certificate issuance config and trust config values in an instance template, you have to create a new template with the new values. Therefore, updating the certificate issuance config and trust config for existing managed instance groups (MIGs) is not supported.
Update individual Compute Engine VMs
To update the certificate issuance config and trust config, update the
contents of the CONFIGS.json
file and use the
gcloud beta compute instances update
command
to apply the updates:
gcloud beta compute instances update INSTANCE_NAME \ --partner-metadata-from-file FILENAME.json
Replace the following:
- INSTANCE_NAME: The name of the VM that you are updating the config values for
- FILENAME: The name of the modified config file, for example
CONFIGS.json
Troubleshoot
To find methods for diagnosing and resolving common errors related to retrieving workload credentials, refer to the Troubleshoot workload to workload authentication documentation.
What's next
- Learn more about the following concepts: