Using customer managed encryption keys

By default, Cloud Run encrypts customer content at rest. Cloud Run handles encryption for you without any additional actions on your part. This option is called Google default encryption.

If you want to control your encryption keys, then you can use customer-managed encryption keys (CMEKs) in Cloud KMS with CMEK-integrated services including Cloud Run. Using Cloud KMS keys gives you control over their protection level, location, rotation schedule, usage and access permissions, and cryptographic boundaries. Using Cloud KMS also lets you view audit logs and control key life cycles. Instead of Google owning and managing the symmetric key encryption keys (KEKs) that protect your data, you control and manage these keys in Cloud KMS.

After you set up your resources with CMEKs, the experience of accessing your Cloud Run resources is similar to using Google default encryption. For more information about your encryption options, see Customer-managed encryption keys (CMEK).

You should be aware of the following:

  • File metadata, such as the path, is not encrypted.
  • Metadata of the Cloud Run service, such as name or environment variables, is not encrypted using the provided key but with a Google-owned and Google-managed encryption key.
  • At runtime, the memory and file content are not encrypted.
  • If a CMEK key is disabled, new instances of existing Cloud Run revisions using the key won't start.
  • If a CMEK key is disabled, deploying a new Cloud Run revision will fail unless a new valid key is used.

Cloud KMS quotas and Cloud Run

You can set your CMEK to one of the available protection levels to indicate how cryptographic operations are performed. When you use CMEK in Cloud Run, your projects can consume Cloud KMS cryptographic requests quotas. For example, CMEK-encrypted repositories can consume these quotas for each upload or download.

Encryption and decryption operations using CMEK keys affect Cloud KMS quotas in these ways:

  • For software CMEK keys generated in Cloud KMS, no Cloud KMS quota is consumed.
  • For hardware CMEK keys—sometimes called Cloud HSM keys—encryption and decryption operations count against Cloud HSM quotas in the project that contains the key.
  • For external CMEK keys—sometimes called Cloud EKM keys—encryption and decryption operations count against Cloud EKM quotas in the project that contains the key.

For more information, see Cloud KMS quotas.

Autoscaling behavior affected by CMEK

The expected autoscaling for your Cloud Run service can be impacted when you use customer managed encryption keys. For example, latency for starting new instances can increase due to delays in contacting external key management systems during key operations.

The following table shows the possible changes in behavior due to use of CMEK keys:

CMEK-related operation Autoscaling behavior
Key disabled/destroyed/revoked New instances won't start.
External key manager cannot be contacted If the key request can be retried, no instances are shutdown during the retries and no new instances will start. Scale-out can appear slower than expected.
If the key request cannot be retried, no new instances are started, and running instances are shut down after a waiting period.
KMS quota exceeded If this quota is exceeded, RESOURCE_EXHAUSTED errors are logged, and new instances won't start. You can request additional quota to fix this.

Before you begin

Allow Cloud Run to access a key

To use CMEK for Cloud Run, perform the following steps:

  1. Configure Artifact Registry to use CMEK.

  2. Using the Artifact Registry Docker Quickstart as a reference create a Docker repository push an image to it.

  3. Use an existing Cloud KMS symmetric key or create a new symmetric key.

  4. To allow Cloud Run access to the key, grant the Cloud Run service agent the Cloud KMS CryptoKey Encrypter/Decrypter role:

    Console

    1. Go to the Cryptographic keys page

    2. Click the key ring for your key to open its key list page.

    3. Select the key and in the right-side Permissions tab, click Add principal.

    4. In the New principals field, copy the Cloud Run service agent email. It has the following suffix:

      PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com.

    5. In the Select a role field, select Cloud KMS CryptoKey Encrypter/Decrypter.

    6. Click Save

    gcloud

    Use the following gcloud kms command:

    gcloud kms keys add-iam-policy-binding KEY_NAME \
--keyring=KEYRING \
--location=LOCATION) \
--member serviceAccount:PROJECT_NUMBER@serverless-robot-prod.iam.gserviceaccount.com \
--role='roles/cloudkms.cryptoKeyEncrypterDecrypter'

    Replace

    • KEY_NAME with the name of your key.
    • KEYRING with the name of your key ring.
    • LOCATION with the name of your region.
    • PROJECT_NUMBER With the project number where you intend to deploy the Cloud Run service.

    You need permission to administer Cloud KMS resources in the Google Cloud project to grant the IAM role roles/cloudkms.cryptoKeyEncrypterDecrypter. Only IAM members with Owner (roles/owner) or Cloud KMS Admin (roles/cloudkms.admin) roles can grant or revoke access to Cloud KMS resources.

Configure CMEK for a Cloud Run service

Any configuration change leads to the creation of a new revision. Subsequent revisions will also automatically get this configuration setting unless you make explicit updates to change it.

Console

  1. In the Google Cloud console, go to Cloud Run:

    Go to Cloud Run

  2. Click Deploy container and select Service to configure a new service. If you are configuring an existing service, click the service, then click Edit and deploy new revision.

  3. If you are configuring a new service, fill out the initial service settings page, then click Container(s), volumes, networking, security to expand the service configuration page.

  4. Click the Security tab.

    image

    • Under Encryption:
      1. Select Customer-managed encryption key (CMEK).
      2. From the Select a customer-managed key menu, choose one of the following options:

        • Select Switch project if you need to use a key from a different project. You can reference a key from another project if your project's service account can access the key for encryption and decryption operations.

        • Select Enter key manually to enter the secret from a project, using the following format: projects/PROJECT_NAME/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME.

        To copy and paste the resource name from another project that you have access to:

        • Go to the Cryptographic keys page.
        • Click the key ring
        • Select the chosen key ring name and click Actions.
        • Select Copy resource name from the menu and paste it in the Key resource name field from the previous step.

      3. From the Key revocation action menu, choose one of the following options:

        • Prevent new container instance: no new instances start after CMEK key revocation.

        • Shutdown as soon as possible: no new instances start and existing instances shut down after CMEK key revocation.

        • Custom shutdown delay: specify the number of hours before the service shuts down.

  5. Click Create or Deploy.

gcloud

To set a key on a service, use either of the following commands:

gcloud run deploy SERVICE \
--image IMAGE_URL \
--key KEY \
--post-key-revocation-action-type KEY_REVOCATION_ACTION
--encryption-key-shutdown-hours SHUTDOWN_HOURS
 
gcloud run services update SERVICE --key KEY
--post-key-revocation-action-type KEY_REVOCATION_ACTION
--encryption-key-shutdown-hours SHUTDOWN_HOURS

Replace

  • SERVICE with the name of your service.
  • IMAGE_URL with a reference to the container image, for example, us-docker.pkg.dev/cloudrun/container/hello:latest. If you use Artifact Registry, the repository REPO_NAME must already be created. The URL has the shape LOCATION-docker.pkg.dev/PROJECT_ID/REPO_NAME/PATH:TAG
  • KEY with the fully qualified key name in the following format: projects/PROJECT_NAME/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME.
  • KEY_REVOCATION_ACTION with shut-down or prevent-new, depending on your key revocation preferences.
  • SHUTDOWN_HOURS with the number of hours to delay before the service shuts down after revocation.

YAML

  1. If you are creating a new service, skip this step. If you are updating an existing service, download its YAML configuration:

    gcloud run services describe SERVICE --format export > service.yaml

  2. Update the following CMEK annotations to the values that you want:

    apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: SERVICE
spec:
  template:
    metadata:
      annotations:
        run.googleapis.com/encryption-key: projects/PROJECT_NAME/locations/LOCATION/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME
        run.googleapis.com/post-key-revocation-action-type: KEY_REVOCATION_ACTION
        run.googleapis.com/encryption-key-shutdown-hours: SHUTDOWN_HOURS
      name: REVISION

    Replace

    • SERVICE with the name of your Cloud Run service.
    • PROJECT_NAME with the name of the project the key was created in.
    • LOCATION with the location the key was created in. Must match the Cloud Run service location.
    • KEYRING_NAME with the name of the key ring.
    • KEY_NAME with the name of the key.
    • KEY_REVOCATION_ACTION with shut-down or prevent-new, depending on your key revocation preferences.
    • SHUTDOWN_HOURS with the number of hours to delay before the service shuts down after revocation.
    • REVISION with a new revision name or delete it (if present). If you supply a new revision name, it must meet the following criteria:
      • Starts with SERVICE-
      • Contains only lowercase letters, numbers and -
      • Does not end with a -
      • Does not exceed 63 characters

  3. Replace the service with its new configuration using the following command:

    gcloud run services replace service.yaml

View security settings

To view the current security settings for your Cloud Run service:

Console

  1. In the Google Cloud console, go to Cloud Run:

    Go to Cloud Run

  2. Click the service you are interested in to open the Service details page.

  3. Click the Revisions tab.

  4. In the details panel at the right, the security setting is listed under the Security tab.

gcloud

  1. Use the following command:

    gcloud run services describe SERVICE

  2. Locate the security setting in the returned configuration.

Test CMEK revocation

  1. Run the following command to confirm the service is accessible: 

    curl SERVICE_URL

    Replace SERVICE_URL with the service URL. You can find this in the console UI after deployment: the container URL is displayed next to the text URL:.

  2. Disable the key version

  3. Wait the number of SHUTDOWN_HOURS you specified. If you did not specify re-enable your key and edit or redeploy your service with the value set to the minimum of one hour.

  4. After waiting the SHUTDOWN_HOURS duration you set re-run the following command and confirm the service is no longer accessible:

curl SERVICE_URL

Understanding audit logs and error messages

If you are responsible for monitoring audit logs, one of your tasks might be to verify CMEK operations in your Cloud Run service. In this case, you need to understand the related audit logs.

If you are responsible for addressing and fixing runtime errors for your Cloud Run service, you might need to troubleshoot CMEK-related errors logged during the operation of the Cloud Run service.

The following sections provide information needed for the preceding tasks.

Audit logs

KMS audit logs provide an audit trail for every operation performed with a key. For CMEK-enabled Cloud Run services, Cloud Run adds Cloud Run-specific caller context that details why the customer key was accessed by the system. The following table lists the contexts you might see in the audit logs:

Reason for key access Description
Decrypting CMEK-encrypted layer during container clone start. Logged whenever a new instance is started.
Encrypting a newly created data-encryption-key w/ the customer-managed-encryption-key. Logged during deployment of a CMEK enabled service, where the CMEK key is wrapped by a KMS key.
Decrypting an existing encrypted data-encryption-key, under the same customer-managed-encryption-key, to be used to encrypt container contents. Logged when a new instance starts up, which requires decryption of the image.
Performing an encrypt operation on dummy data to check the customer-managed-encryption-key status and access. Logged whenever there is a validation check on the key, which is done periodically.
Performing a decrypt operation on dummy data to check the customer-managed-encryption-key status and access. Logged whenever there is a validation check on the key, which is done periodically.

For specifics about audit log format and content, refer to the KMS audit logging page.

Error messages

Note that error messages provided by an external key manager are directly passed through to the Cloud Run logs for your service.

The following table lists the CMEK-related error messages you might see, along with descriptions and possible remedies.

Message Description
User's service account does not have CMEK decrypter permission. Service account: %s Remedy: Allow the service to access the key
User's KMS operation quota has been exceeded. CMEK key: %s The CMEK-enabled service has exceeded KMS quotas. Remedy: Request more KMS quota
User's CMEK key has been disabled. CMEK key: %s The CMEK key was revoked. Remedy: Change the service's CMEK key and redeploy the service.
User's CMEK key has been destroyed. CMEK key: %s The CMEK key was deleted. Remedy: Change the service's CMEK key and redeploy the service.
User's CMEK key has been scheduled for deletion. CMEK Key: %s The CMEK key was scheduled for deletion. Remedy: Change the service's CMEK key and redeploy the service.

What's next