Manage the keys that protect Logging storage data

This document provides instructions for configuring customer-managed encryption keys (CMEK) for logs stored by Logging.

CMEK for Logging storage is applied to individual Logging buckets. If you want to ensure that your logs are protected with a CMEK key throughout the entire Logging product, consider also configuring CMEK for the Log Router.

Overview

By default, Cloud Logging encrypts customer content stored at rest. Data ingested by Logging is encrypted using key-encryption keys, a process known as envelope encryption. Access to your logging data requires access to those key-encryption keys, which Google manages for you without any actions on your part.

Your organization might have regulatory, compliance-related, or advanced encryption requirements that our default encryption at rest doesn't provide. To meet your organization's requirements, instead of Google managing the key encryption keys that protect your data, you can configure CMEK to control and manage your own encryption.

For specific information about CMEK, including its advantages and limitations, see Customer-managed encryption keys.

When you configure CMEK for Logging storage, you can choose the Google Cloud resource level to which the encryption applies:

  • An individual Logging bucket: You can apply CMEK to a new or existing bucket, where it encrypts the data at rest in the bucket. The encryption doesn't apply to the data as it's routed to or copied from the bucket.

  • A Google Cloud organization: If you apply CMEK at the organization level, the configuration propagates to any buckets subsequently created in the organization.

Prerequisites

To get started with enabling CMEK for Logging storage, complete the following steps:

  1. Install and initialize the Google Cloud CLI.

    This guide provides instructions using the Google Cloud CLI.

  2. Decide whether you want to enable CMEK for Logging storage at the Google Cloud organization level or the individual Logging bucket level.

    Note that you can't configure CMEK for Logging storage for buckets created in the global region. When creating or updating a bucket to include CMEK, you must use a key whose region matches the regional scope of your data.

  3. Create or identify the Cloud project in which you want to run Cloud KMS.

  4. Verify that the Google Cloud organization that contains the Cloud project in which you want to run Cloud KMS grants you an IAM role with the logging.cmekSettings.{get,update} permissions. We recommend that you have the Logs Configuration Writer role, which contains the necessary permissions.

  5. Enable the Cloud KMS API for the Cloud project that runs Cloud KMS.

  6. Create a key ring and keys for the Cloud project that runs Cloud KMS.

    To align failure domains, you must use a key whose region matches the regional scope of your data.

  7. Identify the required parameters below; in the samples on this page, the following variables are used to indicate Google Cloud resource metadata:

    • ORGANIZATION_ID is the unique numeric identifier of the Google Cloud organization for which you are enabling CMEK.
    • If you're configuring CMEK at the individual bucket level, BUCKET_ID is the unique numeric identifier of that Logging bucket.

      The samples below show a bucket that is contained by a Cloud project; thus, BUCKET_PROJECT_ID indicates the name of the Cloud project that contains the bucket.
    • KMS_PROJECT_ID is the unique alphanumeric identifier, composed of your Cloud project name and a randomly assigned number, of the Cloud project running Cloud KMS.
    • KMS_KEY_NAME is the Cloud KMS key's resource name. It is formatted like this: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEYRING/cryptoKeys/KEY

    For information about locating resource identifiers, see Identifying projects and Getting your organization ID.

Enable CMEK

After you've completed the prerequisite steps, follow these instructions to enable CMEK for your Google Cloud organization or for an individual bucket.

Determine the service account ID

Whether you're configuring CMEK at the organization or bucket level, you need to get the service-account ID associated with the Google Cloud resource for which CMEK will apply. Run the following command:

BUCKET

gcloud logging cmek-settings describe --project=BUCKET_PROJECT_ID

ORGANIZATION

gcloud alpha logging settings describe --organization=ORGANIZATION_ID

Running this command generates a service account for the specified resource, if one doesn't exist already, and returns the ID in the serviceAccountId field:

serviceAccountId: "SERVICE_ACCOUNT_ID@gcp-sa-logging.iam.gserviceaccount.com"

You need to run this provisioning process only once per resource. Running this command multiple times returns the same value for the serviceAccountId field.

Assign the Encrypter/Decrypter role

Whether you're configuring CMEK at the organization or bucket level, give the service account permission to use your Cloud KMS by assigning the Cloud KMS CryptoKey Encrypter/Decrypter role to the service account:

gcloud kms keys add-iam-policy-binding \
--project=KMS_PROJECT_ID \
--member serviceAccount:SERVICE_ACCOUNT_ID@gcp-sa-logging.iam.gserviceaccount.com \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING \
KMS_KEY

Set the variables in the command as follows:

  • Replace SERVICE_ACCOUNT_ID with the serviceAccountId value that you determined in the previous step.

  • Replace the other variables with the values you determined in the prerequisite steps:

    • Replace KMS_PROJECT_ID with the ID of the Cloud project that is running Cloud KMS.
    • Replace KMS_KEY_LOCATION with the region of the Cloud KMS key.
    • Replace KMS_KEY_RING with the Cloud KMS key ring's name.
    • Replace KMS_KEY with the Cloud KMS key's name.

Configure the Cloud KMS key

To finish enabling CMEK, add the Cloud KMS key name to your bucket or organization. Run the following command:

BUCKET

If you're creating a new bucket, use the following command:

gcloud logging buckets create --cmek-kms-key-name=KMS_KEY_NAME

If you're updating an existing bucket, use the following command:

gcloud logging buckets update --cmek-kms-key-name=KMS_KEY_NAME

ORGANIZATION

gcloud alpha logging settings update \
    --organization=ORGANIZATION_ID --kms-key-name=KMS_KEY_NAME

After the key is successfully applied, all subsequently created buckets in the organization are configured to encrypt their data at rest using this key. You can also change keys for individual buckets later. Note that you can't configure CMEK for Logging storage for buckets created in the global region. You must use a key whose region matches the regional scope of your data.

Verify key enablement

To verify that you've successfully enabled CMEK for your bucket or organization, run the following command:

BUCKET

To list the log buckets associated with a Cloud project, and to see a tabular summary of details for each bucket, run this command:

gcloud logging buckets list --project=BUCKET_PROJECT_ID

In the tabular output, you see a column CMEK. If CMEK is TRUE, then CMEK is enabled for the bucket.

To view the details for a specific bucket, including the key's details, run this command:

gcloud logging buckets describe BUCKET_ID

ORGANIZATION

gcloud alpha logging settings describe --organization=ORGANIZATION_ID

Running this command returns the kmsKeyName:

kmsKeyName: KMS_KEY_NAME
serviceAccountId: SERVICE_ACCOUNT_ID@gcp-sa-logging.iam.gserviceaccount.com

If the kmsKeyName field is populated, then CMEK is enabled for your organization.

Manage your Cloud KMS key

The following sections explain how to change, revoke access for, or disable your Cloud KMS key.

Change your Cloud KMS key

To change the Cloud KMS key associated with your bucket or organization, create a key and update the CMEK settings for the organization with the new Cloud KMS key name.

Run the following command:

BUCKET

gcloud logging buckets update --cmek-kms-key-name=NEW_KMS_KEY_NAME

ORGANIZATION

gcloud alpha logging settings update \
    --organization=ORGANIZATION_ID
    --kms-key-name=NEW_KMS_KEY_NAME

Revoke access to the Cloud KMS key

To revoke Logging's access to the Cloud KMS key at any time, remove the configured service account's IAM permission for that key.

If you remove Logging's access to a key, it can take up to one hour for the change to take effect.

Run the following command:

gcloud kms keys remove-iam-policy-binding \
--project=KMS_PROJECT_ID \
--member serviceAccount:SERVICE_ACCOUNT_ID@gcp-sa-logging.iam.gserviceaccount.com \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING \
KMS_KEY

Disable CMEK

Disabling CMEK for your organization removes CMEK protection of future operations only and returns to Google's default encryption; any previously applied configurations remain intact.

To disable CMEK for your organization, run this command:

gcloud alpha logging settings update --organization=ORGANIZATION_ID --clear-kms-key

If you want to destroy your key, see Destroying and restoring key versions.

External key considerations

When you use a Cloud EKM key, Google has no control over the availability of your externally managed key in the external key-management partner system.

For organization-level CMEK, if an externally managed key is unavailable, Cloud Logging continues retrying to access the key and buffer the incoming log data only for up to one hour. After an hour, if Cloud Logging is still unable to access the externally managed key, Cloud Logging begins dropping the data.

For bucket-level CMEK, if an externally managed key is unavailable, Cloud Logging continues ingesting data but users will not be able to access it.

For more considerations, and potential alternatives, when using external keys, see the Cloud External Key Manager documentation.

CMEK organization policies

Logging supports organization policies that can require CMEK protection and can limit which Cloud KMS CryptoKeys can be used for CMEK protection.

When Cloud Logging API is in the Deny policy list of services of constraint constraints/gcp.restrictNonCmekServices, Logging refuses to create new buckets that aren't CMEK-protected.

When constraints/gcp.restrictCmekCryptoKeyProjects is configured, Logging creates CMEK-protected resources that are protected by a CryptoKey from an allowed project, folder, or organization.

For more information about how organization policies apply to CMEK for Logging storage, see CMEK organization policies.

Limitations

The following are known limitations of CMEK for Logging storage.

Degradation due to key unavailability

A Cloud KMS key is considered available and accessible by Logging if both of the following are true:

  • The key is enabled.
  • The Logging service account has encryption and decryption permissions on the key.

If a key becomes unavailable, Logging still ingests and stores your logs; however, those logs are unavailable for query until key access is repaired.

Logging strongly recommends ensuring that any keys are properly configured and always available.

Loss of disaster recovery

If there are critical failures in Cloud Logging primary storage, then Logging mirrors the logging data to disaster-recovery files. When CMEK is enabled for a resource, such as a Google Cloud organization, logs belonging to that resource are protected by the configured CMEK key. If the CMEK key isn't accessible, the disaster-recovery files can't be written for that resource.

Loss of disaster-recovery files doesn't affect normal logging operations. However, in the event of a storage disaster, Cloud Logging might be unable to recover logs from resources whose CMEK isn't properly configured.

Support constraints

Cloud Customer Care can't read your resource's logs if its key isn't properly configured or becomes unavailable.

Degraded query performance

When a customer-managed encryption key is inaccessible, Cloud Logging continues to ingest and encrypt your data but can't perform background optimizations on it. If key access is restored, the data becomes available; however, the data will initially be stored in an unoptimized state and query performance may suffer.

CMEK can't be disabled for buckets

After CMEK is configured for a bucket, it is permanent.

If you create or update a bucket with CMEK, you can't remove CMEK from it. If you attempt to disable CMEK from a bucket, you'll receive an error message describing this constraint.

Although you can disable CMEK for an organization, it applies only to buckets configured after CMEK has been disabled; any previous configurations remain intact.

For information about deleting a bucket, see Configure and manage log buckets.

Regionality

You can't configure CMEK for Logging storage for buckets created in the global region. When creating or updating a bucket to include CMEK, specify a bucket whose region is other than global. You must use a key whose region matches the regional scope of your data.

Client library availability

Logging client libraries don't provide methods for configuring CMEK.

Quotas

For details on Logging usage limits, see Quotas and limits.

Troubleshoot configuration errors

The following sections describe how to find and mitigate common CMEK configuration errors.

As you configure CMEK for Logging storage, the Cloud project containing the Cloud KMS key is notified of related issues. As examples, updates fail if KMS_KEY_NAME is invalid, if the associated service account doesn't have the required Cloud Key Management Service CryptoKey Encrypter/Decrypter role, or if access to the key is disabled.

Identify configuration errors

To find and view the CMEK configuration errors, do the following:

  1. Navigate to the Google Cloud console:

    Go to Google Cloud console

  2. Select the Cloud project that contains the encryption key.

    You can identify the project ID by running the following command:

    BUCKET

    gcloud logging cmek-settings describe --project=BUCKET_PROJECT_ID
    

    ORGANIZATION

     gcloud alpha logging settings describe --organization=ORGANIZATION_ID
    

    Running this command returns the following:

    kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEYRING/cryptoKeys/KEY
    serviceAccountId: SERVICE_ACCOUNT_ID@gcp-sa-logging.iam.gserviceaccount.com
    

The value of the kmsKeyName field includes the project ID of the key.

  1. Select the Activity tab in the Cloud console.

  2. Check for Cloud Logging Cloud Logging CMEK Configuration Error notifications.

    Each error notification contains steps that you can take to mitigate the issue:

    Error Recommendation
    Cryptographic key permission denied The Logging service account associated with your Cloud project doesn't have sufficient IAM permissions to operate on the specified Cloud KMS key. Follow the instructions in the error or in Grant encryption/decryption permission to grant the proper IAM permission.
    Cryptographic key is disabled The specified Cloud KMS key was disabled. Follow the instructions in the error to re-enable the key.
    Cryptographic key was destroyed The specified Cloud KMS key was destroyed. Follow the instructions in the error or in Configure the Cloud KMS key to configure CMEK encryption with a different key.

Verify key usability

To verify the key's usability, run the following command to list all keys:

gcloud kms keys list \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING

This command returns information about each key in a tabular format. The first line of the output is a list of column names:

NAME PURPOSE ...

Verify that the Cloud KMS key is listed in the command's output as ENABLED, and that the purpose of the key is symmetric encryption: the PURPOSE column must contain ENCRYPT_DECRYPT and the PRIMARY_STATE column must contain ENABLED.

If necessary, create a new key.

Verify permissions configuration

Service accounts that are associated with the organization's CMEK settings must have the Cloud KMS CryptoKey Encrypter/Decrypter role for the configured key.

To list the key's IAM policy, run the following command:

gcloud kms keys get-iam-policy KMS_KEY_NAME

If necessary, add the service account that contains the Cloud KMS CryptoKey Encrypter/Decrypter role to the key.