This topic shows how to protect container images deployed to Cloud Run services using Cloud KMS customer managed encryption keys (CMEK). You can use this feature to protect containers imported by Cloud Run with a CMEK key at rest.
You should be aware of the following:
- File metadata, such as the file path, is not encrypted.
- Metadata of the Cloud Run service, such as name or environment variables, is not encrypted using the provided key but with a Google-owned and Google-managed key.
- At runtime, the memory and file content are not encrypted.
- If a CMEK key is disabled, new instances of existing Cloud Run revisions using the key won't start.
- If a CMEK key is disabled, deploying a new Cloud Run revision will fail unless a new valid key is used.
Because the CMEK key is within your control and is not controlled by Google, no one, including Google, can access the data protected by these encryption keys when the keys are disabled or destroyed.
Using CMEK generates audit logs. Refer to Understanding audit logs and error messages for details.
Cloud KMS quotas and Cloud Run
You can set your CMEK to one of the available protection levels to indicate how cryptographic operations are performed. When you use CMEK in Cloud Run, your projects can consume Cloud KMS cryptographic requests quotas. For example, CMEK-encrypted repositories can consume these quotas for each upload or download.
Encryption and decryption operations using CMEK keys affect Cloud KMS quotas in these ways:
- For software CMEK keys generated in Cloud KMS, no Cloud KMS quota is consumed.
- For hardware CMEK keys—sometimes called Cloud HSM keys—encryption and decryption operations count against Cloud HSM quotas in the project that contains the key.
- For external CMEK keys—sometimes called Cloud EKM keys—encryption and decryption operations count against Cloud EKM quotas in the project that contains the key.
For more information, see Cloud KMS quotas.
Autoscaling behavior affected by CMEK
The expected autoscaling for your Cloud Run service can be impacted when you use customer managed encryption keys. For example, latency for starting new instances can increase due to delays in contacting external key management systems during key operations.
The following table shows the possible changes in behavior due to use of CMEK keys:
CMEK-related operation | Autoscaling behavior |
---|---|
Key disabled/destroyed/revoked | New instances won't start. |
External key manager cannot be contacted | If the key request can be retried, no instances are shutdown during the retries and no new instances will start. Scale-out can appear slower than expected. If the key request cannot be retried, no new instances are started, and running instances are shut down after a waiting period. |
KMS quota exceeded | If this quota is exceeded, RESOURCE_EXHAUSTED errors are logged, and new instances won't start. You can request additional quota to fix this. |
Before you begin
Allow Cloud Run to access a key
To use CMEK for Cloud Run, perform the following steps:
Configure Artifact Registry to use CMEK.
Using the Artifact Registry Docker Quickstart as a reference create a Docker repository push an image to it.
Use an existing Cloud KMS symmetric key or create a new symmetric key.
To allow Cloud Run access to the key, grant the Cloud Run service agent the Cloud KMS CryptoKey Encrypter/Decrypter role:
Console
Go to the Cryptographic keys page
Click the key ring for your key to open its key list page.
Select the key and in the right-side Permissions tab, click Add principal.
In the New principals field, copy the Cloud Run service agent email. It has the following suffix:
PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com.
In the Select a role field, select Cloud KMS CryptoKey Encrypter/Decrypter.
Click Save
gcloud
Use the following
gcloud kms
command:gcloud kms keys add-iam-policy-binding KEY_NAME \ --keyring=KEYRING \ --location=LOCATION) \ --member serviceAccount:PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com \ --role='roles/cloudkms.cryptoKeyEncrypterDecrypter'
Replace
- KEY_NAME with the name of your key.
- KEYRING with the name of your key ring.
- LOCATION with the name of your region.
- PROJECT_NUMBER With the project number where you intend to deploy the Cloud Run service.
You need permission to administer Cloud KMS resources in the Google Cloud project to grant the IAM role
roles/cloudkms.cryptoKeyEncrypterDecrypter
. Only IAM members with Owner (roles/owner
) or Cloud KMS Admin (roles/cloudkms.admin
) roles can grant or revoke access to Cloud KMS resources.
Configure CMEK for a Cloud Run service
Any configuration change leads to the creation of a new revision. Subsequent revisions will also automatically get this configuration setting unless you make explicit updates to change it.
Console
In the Google Cloud console, go to Cloud Run:
Click Deploy container and select Service to configure a new service. If you are configuring an existing service, click the service, then click Edit and deploy new revision.
If you are configuring a new service, fill out the initial service settings page, then click Container(s), volumes, networking, security to expand the service configuration page.
Click the Security tab.
- Under Encryption:
- Select Customer-managed encryption key (CMEK).
- From the Select a customer-managed key menu, choose one of the following options:
Select Switch project if you need to use a key from a different project. You can reference a key from another project if your project's service account can access the key for encryption and decryption operations.
Select Enter key manually to enter the secret from a project, using the following format:
projects/PROJECT_NAME/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME
.- Go to the Cryptographic keys page.
- Click the key ring
- Select the chosen key ring name and click Actions.
- Select Copy resource name from the menu and paste it in the Key resource name field from the previous step.
- From the Key revocation action menu, choose one of the following options:
Prevent new container instance: no new instances start after CMEK key revocation.
Shutdown as soon as possible: no new instances start and existing instances shut down after CMEK key revocation.
Custom shutdown delay: specify the number of hours before the service shuts down.
To copy and paste the resource name from another project that you have access to:
- Under Encryption:
Click Create or Deploy.
gcloud
To set a key on a service, use either of the following commands:
gcloud run deploy SERVICE \ --image IMAGE_URL \ --key KEY \ --post-key-revocation-action-type KEY_REVOCATION_ACTION --encryption-key-shutdown-hours SHUTDOWN_HOURS
gcloud run services update SERVICE --key KEY --post-key-revocation-action-type KEY_REVOCATION_ACTION --encryption-key-shutdown-hours SHUTDOWN_HOURS
Replace
- SERVICE with the name of your service.
- IMAGE_URL with a reference to the container image, for
example,
us-docker.pkg.dev/cloudrun/container/hello:latest
. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL has the shapeLOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG
- KEY with the fully qualified key name in the following
format:
projects/PROJECT_NAME/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME
. - KEY_REVOCATION_ACTION with
shut-down
orprevent-new
, depending on your key revocation preferences. - SHUTDOWN_HOURS with the number of hours to delay before the service shuts down after revocation.
YAML
If you are creating a new service, skip this step. If you are updating an existing service, download its YAML configuration:
gcloud run services describe SERVICE --format export > service.yaml
Update the following CMEK annotations to the values that you want:
apiVersion: serving.knative.dev/v1 kind: Service metadata: name: SERVICE spec: template: metadata: annotations: run.googleapis.com/encryption-key: projects/PROJECT_NAME/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME run.googleapis.com/post-key-revocation-action-type: KEY_REVOCATION_ACTION run.googleapis.com/encryption-key-shutdown-hours: SHUTDOWN_HOURS name: REVISION
Replace
- SERVICE with the name of your Cloud Run service.
- PROJECT_NAME with the name of the project the key was created in.
- LOCATION with the location the key was created in. Must match the Cloud Run service location.
- KEYRING_NAME with the name of the key ring.
- KEY_NAME with the name of the key.
- KEY_REVOCATION_ACTION with
shut-down
orprevent-new
, depending on your key revocation preferences. - SHUTDOWN_HOURS with the number of hours to delay before the service shuts down after revocation.
- REVISION with a new revision name or delete it (if present). If you supply a new revision name, it must meet the following criteria:
- Starts with
SERVICE-
- Contains only lowercase letters, numbers and
-
- Does not end with a
-
- Does not exceed 63 characters
- Starts with
Replace the service with its new configuration using the following command:
gcloud run services replace service.yaml
View security settings
To view the current security settings for your Cloud Run service:
Console
In the Google Cloud console, go to Cloud Run:
Click the service you are interested in to open the Service details page.
Click the Revisions tab.
In the details panel at the right, the security setting is listed under the Security tab.
gcloud
Use the following command:
gcloud run services describe SERVICE
Locate the security setting in the returned configuration.
Test CMEK revocation
Run the following command to confirm the service is accessible:
curl SERVICE_URL
Replace SERVICE_URL with the service URL. You can find this in the console UI after deployment: the container URL is displayed next to the text URL:.
Wait the number of SHUTDOWN_HOURS you specified. If you did not specify re-enable your key and edit or redeploy your service with the value set to the minimum of one hour.
After waiting the SHUTDOWN_HOURS duration you set re-run the following command and confirm the service is no longer accessible:
curl SERVICE_URL
Understanding audit logs and error messages
If you are responsible for monitoring audit logs, one of your tasks might be to verify CMEK operations in your Cloud Run service. In this case, you need to understand the related audit logs.
If you are responsible for addressing and fixing runtime errors for your Cloud Run service, you might need to troubleshoot CMEK-related errors logged during the operation of the Cloud Run service.
The following sections provide information needed for the preceding tasks.
Audit logs
KMS audit logs provide an audit trail for every operation performed with a key. For CMEK-enabled Cloud Run services, Cloud Run adds Cloud Run-specific caller context that details why the customer key was accessed by the system. The following table lists the contexts you might see in the audit logs:
Reason for key access | Description |
---|---|
Decrypting CMEK-encrypted layer during container clone start. |
Logged whenever a new instance is started. |
Encrypting a newly created data-encryption-key w/ the customer-managed-encryption-key. |
Logged during deployment of a CMEK enabled service, where the CMEK key is wrapped by a KMS key. |
Decrypting an existing encrypted data-encryption-key, under the same customer-managed-encryption-key, to be used to encrypt container contents. |
Logged when a new instance starts up, which requires decryption of the image. |
Performing an encrypt operation on dummy data to check the customer-managed-encryption-key status and access. |
Logged whenever there is a validation check on the key, which is done periodically. |
Performing a decrypt operation on dummy data to check the customer-managed-encryption-key status and access. |
Logged whenever there is a validation check on the key, which is done periodically. |
For specifics about audit log format and content, refer to the KMS audit logging page.
Error messages
Note that error messages provided by an external key manager are directly passed through to the Cloud Run logs for your service.
The following table lists the CMEK-related error messages you might see, along with descriptions and possible remedies.
Message | Description |
---|---|
User's service account does not have CMEK decrypter permission. Service account: %s |
Remedy: Allow the service to access the key |
User's KMS operation quota has been exceeded. CMEK key: %s |
The CMEK-enabled service has exceeded KMS quotas. Remedy: Request more KMS quota |
User's CMEK key has been disabled. CMEK key: %s |
The CMEK key was revoked. Remedy: Change the service's CMEK key and redeploy the service. |
User's CMEK key has been destroyed. CMEK key: %s |
The CMEK key was deleted. Remedy: Change the service's CMEK key and redeploy the service. |
User's CMEK key has been scheduled for deletion. CMEK Key: %s |
The CMEK key was scheduled for deletion. Remedy: Change the service's CMEK key and redeploy the service. |
What's next
- Learn more about Permissions and roles in Cloud KMS.
- Create a key.
- Audit logs