Using customer managed encryption keys

This topic shows how to protect container images deployed to Cloud Run services using Cloud KMS customer managed encryption keys (CMEK). You can use this feature to protect containers imported by Cloud Run with a CMEK key at rest.

You should be aware of the following:

• File metadata, such as the file path, is not encrypted.
• Metadata of the Cloud Run service, such as name or environment variables, is not encrypted using the provided key but with a Google-managed key.
• At runtime, the memory and file content are not encrypted.
• If a CMEK key is disabled, new container instances of existing Cloud Run revisions using the key will not start.
• If a CMEK key is disabled, deploying a new Cloud Run revision will fail unless a new valid key is used.

Because the CMEK key is within your control and is not controlled by Google, no one, including Google, can access the data protected by these encryption keys when the keys are disabled or destroyed.

Note: This feature protects container images once deployed to Cloud Run. The contents of the container images prior to deployment are protected separately, such as with Artifact Registry. If you use a container image within a CMEK protected Artifact Registry repository, you do not need to use the same key for Cloud Run deployment.

When you use CMEK, audit logs are generated. Refer to Understanding audit logs and error messages for details.

Autoscaling behavior affected by CMEK

The expected autoscaling for your Cloud Run service can be impacted when you use customer managed encryption keys. For example, latency for starting new instances can increase due to delays in contacting external key management systems during key operations.

The following table shows the possible changes in behavior due to use of CMEK keys:

CMEK-related operation	Autoscaling behavior
Key disabled/destroyed	New instances won't start.
External key manager cannot be contacted	If the key request can be retried, no instances are shutdown during the retries and no new instances will start. Scale-out can appear slower than expected. If the key request cannot be retried, no new instances are started, and running instances are shut down after a waiting period.
KMS quota exceeded	CMEK operations use KMS quota. If this quota is exceeded, RESOURCE_EXHAUSTED errors are logged, and new instances won't start. You can request additional quota to fix this.

Allowing Cloud Run to access a key

Important: The key must be a symmetric key created for the same region where you will deploy the Cloud Run service, for example, us-central1. Do not choose a multi-region key.

To use this feature:

1. Enable the Cloud Run Admin API

2. Use an existing Cloud KMS symmetric key or create a new symmetric key. However, in order to allow a Cloud Run service to access the key, you must add the Cloud Run service account as a Member of the key, and grant it the Cloud KMS CryptoKey Encrypter/Decrypter role:

   Console

   1. Go to the Cryptographic keys page in the Google Cloud console

   2. Click the key ring for your key to open its key list page.

   3. Select the key and in the right side permissions tab, click Add Member.

   4. In the New members textbox, copy the email of the Cloud Run service agent. It has the suffix @serverless-robot-prod.iam.gserviceaccount.com.

   5. Grant it the role Cloud KMS CryptoKey Encrypter/Decrypter.

   Command line

   Use the following gcloud kms command:

   gcloud kms keys add-iam-policy-binding 
   
   (KEY : --keyring=KEYRING --location=LOCATION) 
   
   --member=MEMBER --role='roles/cloudkms.cryptoKeyEncrypterDecrypter'

   Replace

   • KEYRING with the name of your key ring.
   • LOCATION with the name of your region.
   • MEMBER with the email of the Cloud Run service agent. It has the suffix @serverless-robot-prod.iam.gserviceaccount.com.

   You need permission to administer Cloud KMS resources in the Google Cloud project to grant the IAM role roles/cloudkms.cryptoKeyEncrypterDecrypter. Only IAM members with Owner (roles/owner) or Cloud KMS Admin (roles/cloudkms.admin) roles can grant or revoke access to Cloud KMS resources.

Enabling CMEK for a Cloud Run service:

Any configuration change leads to the creation of a new revision. Subsequent revisions will also automatically get this configuration setting unless you make explicit updates to change it.

Console

1. Go to Cloud Run

2. Click Create Service if you are configuring a new service you are deploying to. If you are configuring an existing service, click on the service, then click Edit and Deploy New Revision.

3. If you are configuring a new service, fill out the initial service settings page as desired, then click Container, Networking, Security to expand the service configuration page.

4. Click the Security tab.

   

5. Select the checkbox Use a customer-managed encryption key (CMEK) , then select the desired key from the Select a customer-managed key pulldown menu.

6. Click Create or Deploy.

Command line

To set a key on a service, use either of the following commands:

gcloud run deploy SERVICE --image IMAGE_URL --key KEY

gcloud run services update SERVICE --key KEY

Replace

• SERVICE with the name of your service.
• IMAGE_URL with a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest
• KEY with the fully qualified key name, in this format:projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME

YAML

You can download and view existing service configurations using the gcloud run services describe --format export command, which yields cleaned results in YAML format. You can then modify the fields described below and upload the modified YAML using the gcloud run services replace command. Make sure you only modify fields as documented.

1. To view and download the configuration:

   gcloud run services describe SERVICE --format export > service.yaml

2. Update the run.googleapis.com/encryption-key annotation:

   apiVersion: serving.knative.dev/v1
   kind: Service
   metadata:
     name: SERVICE
   spec:
     template:
       metadata:
         annotations:
           run.googleapis.com/encryption-key: projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME
         name: REVISION

   Replace

   • SERVICE with the name of your Cloud Run service.
   • PROJECT_NAME with the name of the project the key was created in.
   • LOCATION with the location the key was created in. Must match the Cloud Run service location.
   • RING_NAME with the name of the key ring.
   • KEY_NAME with the name of the key.
   • REVISION with a new revision name or delete it (if present). If you supply a new revision name, it must meet the following criteria:
     • Starts with SERVICE-
     • Contains only lowercase letters, numbers and -
     • Does not end with a -
     • Does not exceed 63 characters

3. Replace the service with its new configuration using the following command:

   gcloud run services replace service.yaml

Referencing keys from other projects

You can reference a key from another project, if your project's service account has been allowed to encrypt/decrypt the key.

Console

1. Go to Cloud Run

2. Click Create Service if you are configuring a new service you are deploying to. If you are configuring an existing service, click on the service, then click Edit and Deploy New Revision.

3. If you are configuring a new service, fill out the initial service settings page as desired, then click Container, Networking, Security to expand the service configuration page.

4. Click the Security tab.

   

5. In the Security tab:

   1. Select the checkbox Use a customer-managed encryption key (CMEK) .
   2. Select Don't see your key? Enter key resource ID from the Select a customer-managed key pulldown list, to display the following form:

      

   3. In the Add a secret by resource ID form, enter the secret from the other project, in the format projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME. You can alternatively copy and paste the resource ID from the other project if you have access to it, by selecting the key, clicking the Actions ellipsis at the right of the secret, and selecting Copy resource ID from the pulldown menu.
   4. Click Deploy.

6. Click Create or Deploy.

Command line

To set a key on a service, use one of the following commands:

gcloud beta run deploy SERVICE --image IMAGE_URL --key KEY

gcloud beta run services update SERVICE --key KEY

Replace

• SERVICE with the name of your service.
• IMAGE_URL with a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest
• KEY with the fully qualified key name, in this format:projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME

View security settings

Understanding audit logs and error messages

If you are responsible for monitoring audit logs, one of your tasks might be to verify CMEK operations in your Cloud Run service. In this case, you need to understand the related audit logs.

If you are responsible for addressing and fixing runtime errors for your Cloud Run service, you may need to troubleshoot CMEK-related errors logged during the operation of the Cloud Run service.

The following sections provide information needed for the above tasks.

Audit logs

KMS audit logs provide an audit trail for every operation performed with a key. For CMEK-enabled Cloud Run services, Cloud Run adds Cloud Run-specific caller context that details why the customer key was accessed by the system. The following table lists the contexts you might see in the audit logs:

Reason for key access	Description
Decrypting CMEK-encrypted layer during container clone start.	Logged whenever a new instance is started.
Encrypting a newly created data-encryption-key w/ the customer-managed-encryption-key.	Logged during deployment of a CMEK enabled service, where the CMEK key is wrapped by a KMS key.
Decrypting an existing encrypted data-encryption-key, under the same customer-managed-encryption-key, to be used to encrypt container contents.	Logged when a new instance starts up, which requires decryption of the image.
Performing an encrypt operation on dummy data to check the customer-managed-encryption-key status and access.	Logged whenever there is a validation check on the key, which is done periodically.
Performing an decrypt operation on dummy data to check the customer-managed-encryption-key status and access.	Logged whenever there is a validation check on the key, which is done periodically.

For specifics about audit log format and content, refer to the KMS audit logging page.

Error messages

Note that error messages provided by an external key manager are directly passed through to the Cloud Run logs for your service.

The following table lists the CMEK-related error messages you might see, along with descriptions and possible remedies.

Message	Description
User's service account does not have CMEK decrypter permission. Service account: %s	Remedy: Allow the service to access the key
User's KMS operation quota has been exceeded. CMEK key: %s	The CMEK-enabled service uses KMS quota for key operations. Remedy: request more KMS quota
User's CMEK key has been disabled. CMEK key: %s	The CMEK key was revoked. Remedy: Change the service's CMEK key and redeploy the service.
User's CMEK key has been destroyed. CMEK key: %s	The CMEK key was deleted. Remedy: Change the service's CMEK key and redeploy the service.
User's CMEK key has been scheduled for deletion. CMEK Key: %s	The CMEK key was scheduled for deletion. Remedy: Change the service's CMEK key and redeploy the service.

What's next

• Learn more about Permissions and roles in Cloud KMS.
• Create a key.
• Audit logs