Using customer managed encryption keys

This topic shows how to protect container images deployed to Cloud Run services using Cloud KMS customer managed encryption keys (CMEK). You can use this feature to protect containers imported by Cloud Run with a CMEK key at rest.

You should be aware of the following:

  • File metadata, such as the file path, is not encrypted.
  • Metadata of the Cloud Run service, such as name or environment variables, is not encrypted using the provided key but with a Google-managed key.
  • At runtime, the memory and file content are not encrypted.
  • If a CMEK key is disabled, new instances of existing Cloud Run revisions using the key won't start.
  • If a CMEK key is disabled, deploying a new Cloud Run revision will fail unless a new valid key is used.

Because the CMEK key is within your control and is not controlled by Google, no one, including Google, can access the data protected by these encryption keys when the keys are disabled or destroyed.

Using CMEK generates audit logs. Refer to Understanding audit logs and error messages for details.

Cloud KMS quotas and Cloud Run

You can set your CMEK to one of the available protection levels to indicate how cryptographic operations are performed. When you use CMEK in Cloud Run, your projects can consume Cloud KMS cryptographic requests quotas. For example, CMEK-encrypted repositories can consume these quotas for each upload or download.

Encryption and decryption operations using CMEK keys affect Cloud KMS quotas in these ways:

  • For software CMEK keys generated in Cloud KMS, no Cloud KMS quota is consumed.
  • For hardware CMEK keys—sometimes called Cloud HSM keys—encryption and decryption operations count against Cloud HSM quotas in the project that contains the key.
  • For external CMEK keys—sometimes called Cloud EKM keys—encryption and decryption operations count against Cloud EKM quotas in the project that contains the key.

For more information, see Cloud KMS quotas.

Autoscaling behavior affected by CMEK

The expected autoscaling for your Cloud Run service can be impacted when you use customer managed encryption keys. For example, latency for starting new instances can increase due to delays in contacting external key management systems during key operations.

The following table shows the possible changes in behavior due to use of CMEK keys:

CMEK-related operation Autoscaling behavior
Key disabled/destroyed/revoked New instances won't start.
External key manager cannot be contacted If the key request can be retried, no instances are shutdown during the retries and no new instances will start. Scale-out can appear slower than expected.
If the key request cannot be retried, no new instances are started, and running instances are shut down after a waiting period.
KMS quota exceeded If this quota is exceeded, RESOURCE_EXHAUSTED errors are logged, and new instances won't start. You can request additional quota to fix this.

Before you begin

Allowing Cloud Run to access a key

To use CMEK for Cloud Run, perform the following steps:

  1. Use an existing Cloud KMS symmetric key or create a new symmetric key.

  2. To allow Cloud Run access to the key, grant the Cloud Run service agent the Cloud KMS CryptoKey Encrypter/Decrypter role:

    Console

    1. Go to the Cryptographic keys page

    2. Click the key ring for your key to open its key list page.

    3. Select the key and in the right-side Permissions tab, click Add principal.

    4. In the New principals field, copy the Cloud Run service agent email. It has the following suffix:

      @serverless-robot-prod.iam.gserviceaccount.com.

    5. In the Select a role field, select Cloud KMS CryptoKey Encrypter/Decrypter.

    gcloud

    Use the following gcloud kms command:

    gcloud kms keys add-iam-policy-binding KEY_NAME \
    --keyring=KEYRING \
    --location=LOCATION) \
    --member=MEMBER \
    --role='roles/cloudkms.cryptoKeyEncrypterDecrypter'

    Replace

    • KEY_NAME with the name of your key.
    • KEYRING with the name of your key ring.
    • LOCATION with the name of your region.
    • MEMBER with the email of the Cloud Run service agent. It has the suffix @serverless-robot-prod.iam.gserviceaccount.com.

    You need permission to administer Cloud KMS resources in the Google Cloud project to grant the IAM role roles/cloudkms.cryptoKeyEncrypterDecrypter. Only IAM members with Owner (roles/owner) or Cloud KMS Admin (roles/cloudkms.admin) roles can grant or revoke access to Cloud KMS resources.

Configuring CMEK for a Cloud Run service

Any configuration change leads to the creation of a new revision. Subsequent revisions will also automatically get this configuration setting unless you make explicit updates to change it.

Console

  1. In the Google Cloud console, go to Cloud Run:

    Go to Cloud Run

  2. Click Create Service if you are configuring a new service you are deploying to. If you are configuring an existing service, click the service, then click Edit and deploy new revision.

  3. If you are configuring a new service, fill out the initial service settings page as desired, then click Container(s), volumes, networking, security to expand the service configuration page.

  4. Click the Security tab.

    image

    • Under Encryption:
      1. Select Customer-managed encryption key (CMEK).
      2. From the Select a customer-managed key menu, choose one of the following options:
        • Select Switch project if you need to use a key from a different project. You can reference a key from another project if your project's service account can access the key for encryption and decryption operations.

        • Select Enter key manually to enter the secret from a project, using the following format: projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME.

        To copy and paste the resource name from another project that you have access to:

        • Go to the Cryptographic keys page.
        • Click the key ring
        • Select the chosen key ring name and click Actions.
        • Select Copy resource name from the menu and paste it in the Key resource name field from the previous step.

      3. From the Key revocation action menu, choose one of the following options:
        • Prevent new container instance: no new instances start after CMEK key revocation.

        • Shutdown as soon as possible: no new instances start and existing instances shut down after CMEK key revocation.

        • Custom shutdown delay: specify the number of hours before the service shuts down.

  5. Click Create or Deploy.

gcloud

To set a key on a service, use either of the following commands:

gcloud run deploy SERVICE \
--image IMAGE_URL \
--key KEY \
--post-key-revocation-action-type KEY_REVOCATION_ACTION
--encryption-key-shutdown-hours SHUTDOWN_HOURS
gcloud run services update SERVICE --key KEY
--post-key-revocation-action-type KEY_REVOCATION_ACTION
--encryption-key-shutdown-hours SHUTDOWN_HOURS

Replace

  • SERVICE with the name of your service.
  • IMAGE_URL with a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL has the shape REGION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG
  • KEY with the fully qualified key name in the following format: projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME.
  • KEY_REVOCATION_ACTION with shut-down or prevent-new, depending on your key revocation preferences.
  • SHUTDOWN_HOURS with the number of hours to delay before the service shuts down after revocation.

YAML

You can download and view existing service configurations using the gcloud run services describe --format export command, which yields cleaned results in YAML format. You can then modify the fields described below and upload the modified YAML using the gcloud run services replace command. Make sure you only modify fields as documented.

  1. To view and download the configuration:

    gcloud run services describe SERVICE --format export > service.yaml
  2. Update the following CMEK annotations to the values that you want:

    apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: SERVICE
    spec:
      template:
        metadata:
          annotations:
            run.googleapis.com/encryption-key: projects/PROJECT_NAME/locations/LOCATION/keyRings/RING_NAME/cryptoKeys/KEY_NAME
            run.googleapis.com/post-key-revocation-action-type: KEY_REVOCATION_ACTION
            run.googleapis.com/encryption-key-shutdown-hours: SHUTDOWN_HOURS
          name: REVISION

    Replace

    • SERVICE with the name of your Cloud Run service.
    • PROJECT_NAME with the name of the project the key was created in.
    • LOCATION with the location the key was created in. Must match the Cloud Run service location.
    • RING_NAME with the name of the key ring.
    • KEY_NAME with the name of the key.
    • KEY_REVOCATION_ACTION with shut-down or prevent-new, depending on your key revocation preferences.
    • SHUTDOWN_HOURS with the number of hours to delay before the service shuts down after revocation.
    • REVISION with a new revision name or delete it (if present). If you supply a new revision name, it must meet the following criteria:
      • Starts with SERVICE-
      • Contains only lowercase letters, numbers and -
      • Does not end with a -
      • Does not exceed 63 characters
  3. Replace the service with its new configuration using the following command:

    gcloud run services replace service.yaml

View security settings

To view the current security settings for your Cloud Run service:

Console

  1. In the Google Cloud console, go to Cloud Run:

    Go to Cloud Run

  2. Click the service you are interested in to open the Service details page.

  3. Click the Revisions tab.

  4. In the details panel at the right, the security setting is listed under the Security tab.

Command line

  1. Use the following command:

    gcloud run services describe SERVICE
  2. Locate the security setting in the returned configuration.

Understanding audit logs and error messages

If you are responsible for monitoring audit logs, one of your tasks might be to verify CMEK operations in your Cloud Run service. In this case, you need to understand the related audit logs.

If you are responsible for addressing and fixing runtime errors for your Cloud Run service, you might need to troubleshoot CMEK-related errors logged during the operation of the Cloud Run service.

The following sections provide information needed for the preceding tasks.

Audit logs

KMS audit logs provide an audit trail for every operation performed with a key. For CMEK-enabled Cloud Run services, Cloud Run adds Cloud Run-specific caller context that details why the customer key was accessed by the system. The following table lists the contexts you might see in the audit logs:

Reason for key access Description
Decrypting CMEK-encrypted layer during container clone start. Logged whenever a new instance is started.
Encrypting a newly created data-encryption-key w/ the customer-managed-encryption-key. Logged during deployment of a CMEK enabled service, where the CMEK key is wrapped by a KMS key.
Decrypting an existing encrypted data-encryption-key, under the same customer-managed-encryption-key, to be used to encrypt container contents. Logged when a new instance starts up, which requires decryption of the image.
Performing an encrypt operation on dummy data to check the customer-managed-encryption-key status and access. Logged whenever there is a validation check on the key, which is done periodically.
Performing an decrypt operation on dummy data to check the customer-managed-encryption-key status and access. Logged whenever there is a validation check on the key, which is done periodically.

For specifics about audit log format and content, refer to the KMS audit logging page.

Error messages

Note that error messages provided by an external key manager are directly passed through to the Cloud Run logs for your service.

The following table lists the CMEK-related error messages you might see, along with descriptions and possible remedies.

Message Description
User's service account does not have CMEK decrypter permission. Service account: %s Remedy: Allow the service to access the key
User's KMS operation quota has been exceeded. CMEK key: %s The CMEK-enabled service has exceeded KMS quotas. Remedy: Request more KMS quota
User's CMEK key has been disabled. CMEK key: %s The CMEK key was revoked. Remedy: Change the service's CMEK key and redeploy the service.
User's CMEK key has been destroyed. CMEK key: %s The CMEK key was deleted. Remedy: Change the service's CMEK key and redeploy the service.
User's CMEK key has been scheduled for deletion. CMEK Key: %s The CMEK key was scheduled for deletion. Remedy: Change the service's CMEK key and redeploy the service.

What's next