Configure CMEK for logs routing

This page describes how to configure and manage customer-managed encryption keys (CMEK) for the Log Router to meet your organization's compliance needs.

CMEK for the Log Router is configured at the Google Cloud organization level. It applies to all Cloud projects and folders contained by the Google Cloud organization.

Overview

By default, Cloud Logging encrypts customer content stored at rest. Data ingested by Logging is encrypted using key-encryption keys, a process known as envelope encryption. Access to your logging data requires access to those key-encryption keys, which Google manages for you without any actions on your part.

Your organization might have regulatory, compliance-related, or advanced encryption requirements that our default encryption at rest doesn't provide. To meet your organization's requirements, instead of Google managing the key encryption keys that protect your data, you can configure CMEK to control and manage your own encryption.

For specific information about CMEK, including its advantages and limitations, see Customer-managed encryption keys.

For information about using CMEK on logs that are stored in Cloud Logging, see Configure CMEK for logs storage.

Prerequisites

To get started with enabling CMEK for Log Router, complete the following steps:

1. Install the Google Cloud CLI, then initialize it by running the following command:

   gcloud init

   Note: If you installed the gcloud CLI previously, make sure you have the latest version by running gcloud components update.

   This guide provides instructions using the Google Cloud CLI.

2. Identify the organization for which you want to enable CMEK.

3. Create or identify the Google Cloud project in which you want to run Cloud KMS.

   Verify that the Google Cloud organization that contains the Cloud project in which you want to run Cloud KMS grants you an IAM role with the following permissions:

   • logging.settings.get
   • logging.settings.update

   or with the following permissions:

   • logging.cmekSettings.get
   • logging.cmekSettings.update

   We recommend that you have the Logs Configuration Writer role, which contains the necessary permissions.

4. Enable the Cloud KMS API for the Cloud project that runs Cloud KMS.

5. Create a key ring and keys for the Cloud project that runs Cloud KMS.

   The Log Router lets you use a key from any region as a single organization may have data routed to multiples regions. If all, or the majority, of the data in the organization is restricted to one region, then we recommend that you use a key whose region matches the regional scope of your data.

6. Identify the required parameters below; in the samples on this page, the following variables are used to indicate Google Cloud resource metadata:

   • ORGANIZATION_ID is the unique numeric identifier of the Google Cloud organization for which you are enabling CMEK
   • KMS_PROJECT_ID is the unique alphanumeric identifier, composed of your Cloud project name and a randomly assigned number, of the Cloud project running Cloud KMS
   • KMS_KEY_NAME is the Cloud KMS key's resource name. It is formatted like this: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEYRING/cryptoKeys/KEY

   For information about locating resource identifiers, see Identifying projects and Getting your organization ID.

Enable CMEK for an organization

After you've completed the prerequisite steps, follow these instructions to enable CMEK for your Google Cloud organization.

Determine the service account ID

To configure CMEK at the organization level, you need to get the service-account ID associated with the organization for which CMEK will apply. Run the following command:

API

1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

2. Issue a GET request to obtain the service-account ID associated with the Google Cloud organization for which you want to enable CMEK:

    curl -H "Authorization: Bearer AUTH_TOKEN" \
       https://logging.googleapis.com/v2/organizations/ORGANIZATION_ID/settings
   

gcloud

gcloud logging settings describe --organization=ORGANIZATION_ID

Running this command generates a service account for the organization, if one doesn't exist already, and returns the ID in the serviceAccountId field:

serviceAccountId: "[e-mail]"

You need to run this provisioning process only once per resource. Running this command multiple times returns the same value for the serviceAccountId field.

Assign the Encrypter/Decrypter role

To use CMEK, give the service account permission to use your Cloud KMS by assigning the Cloud KMS CryptoKey Encrypter/Decrypter role to the service account:

API

Note: This task uses a separate endpoint compared to most Logging tasks.

1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials. For instructions, see API authentication.

2. Create a JSON file that contains the following information:

   {
     "policy": {
       "bindings": {
         "role": "roles/cloudkms.cryptoKeyEncrypterDecrypter",
         "members": "serviceAccount:[e-mail]"
       },
     }
   }

3. Use cURL to call the Cloud KMS API with a POST setIamPolicy request:

   curl -X POST --data-binary @JSON_FILE_NAME.json \
   -H "Authorization: Bearer OAUTH2_TOKEN" \
   -H "Content-Type: application/json" \
   "https://cloudkms.googleapis.com/v1/KEY_RESOURCE:setIamPolicy"

Caution: This request replaces any existing Identity and Access Management policy on the key resource.

Note: There might be a delay of several seconds between assigning the role and having it applied to your service account. If you grant this permission programmatically, wait 30 seconds before configuring Logging.

gcloud

gcloud kms keys add-iam-policy-binding \
--project=KMS_PROJECT_ID \
--member serviceAccount:[e-mail] \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING \
KMS_KEY

Set the variables in the command as follows:

• Replace SERVICE_ACCOUNT_ID with the serviceAccountId value that you determined in the previous step.

• Replace the other variables with the values you determined in the prerequisite steps:

  • Replace KMS_PROJECT_ID with the ID of the Cloud project that is running Cloud KMS.
  • Replace KMS_KEY_LOCATION with the region of the Cloud KMS key.
  • Replace KMS_KEY_RING with the Cloud KMS key ring's name.
  • Replace KMS_KEY with the Cloud KMS key's name.

Console

1. Open the Cloud Key Management Service Keys browser in the Google Cloud console.
   Open the Cloud KMS Keys browser

2. Click on the name of the key ring that contains the desired key.

3. Select the checkbox for the desired key.

   The Permissions tab becomes available.

4. In the Add members dialog, specify the email address of the Logging service account you are granting access.

5. In the Select a role drop down, select Cloud KMS CryptoKey Encrypter/Decrypter.

6. Click Add.

Configure the Cloud KMS key

To finish enabling CMEK, add the Cloud KMS key name to your organization. Run the following command:

API

1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

2. Create a JSON file that contains the following information:

   {
       "kms_key_name": "KMS_KEY_NAME"
   }
   

3. Issue a PATCH request to update kms_key_name to be associated with the Google Cloud organization for which you want to enable CMEK:

   curl -X PATCH \
       -H "Authorization: Bearer AUTH_TOKEN \
       -H "Content-Type: application/json" \
       --data-binary @JSON_FILE_NAME
       https://logging.googleapis.com/v2/organizations/ORGANIZATION_ID/settings?update_mask="kms_key_name"
   

gcloud

gcloud logging settings update \
    --organization=ORGANIZATION_ID --kms-key-name=KMS_KEY_NAME

Verify key enablement

To verify that you've successfully enabled CMEK for your organization, run the following command:

API

1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

2. Issue a GET request to obtain the CMEK settings associated with the Google Cloud organization for which you want to verify key enablement:

    curl -H "Authorization: Bearer AUTH_TOKEN" \
       https://logging.googleapis.com/v2/organizations/ORGANIZATION_ID/settings
   
   {
       "kmsKeyName": "KMS_KEY_NAME",
       "serviceAccountId": "[e-mail]"
   }
   

gcloud

gcloud logging settings describe --organization=ORGANIZATION_ID

Running this command returns the Cloud KMS key name:

kmsKeyName: KMS_KEY_NAME
serviceAccountId: [e-mail]

If the kmsKeyName field is populated, then CMEK is enabled for your organization.

Manage your Cloud KMS key

The following sections explain how to change, revoke access for, or disable your Cloud KMS key.

Note: If Logging loses access to the Cloud KMS key, there are data loss and user implications; to learn more, see Limitations on this page.

Change your Cloud KMS key

To change the Cloud KMS key associated with your organization, create a key and update the CMEK settings for the organization with the new Cloud KMS key name.

Run the following command:

API

1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

2. Create a JSON file that contains the following information:

   {
       "kms_key_name": "NEW_KMS_KEY_NAME"
   }
   

3. Issue a PATCH request to update kms_key_name:

   curl -X PATCH \
       -H "Authorization: Bearer AUTH_TOKEN \
       -H "Content-Type: application/json" \
       --data-binary @JSON_FILE_NAME
       https://logging.googleapis.com/v2/organizations/ORGANIZATION_ID/settings?update_mask="kms_key_name"
   

gcloud

gcloud logging settings update \
    --organization=ORGANIZATION_ID
    --kms-key-name=NEW_KMS_KEY_NAME

Revoke access to the Cloud KMS key

To revoke Logging's access to the Cloud KMS key at any time, remove the configured service account's IAM permission for that key.

If you remove Logging's access to a key, it can take up to one hour for the change to take effect.

Run the following command:

API

1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

2. Create a JSON file that contains the following information:

   revoke.json:
   {
     "policy": {
       "bindings": {
         "role": "roles/cloudkms.cryptoKeyEncrypterDecrypter",
         "members":
       },
     }
   }
   

3. Issue a POST request:

   curl -X POST --data-binary @revoke.json -H "Authorization: Bearer
   ${OAUTH_TOKEN}" -H "Content-Type: application/json"
   "https://cloudkms.googleapis.com/v1/{$KEY}:setIamPolicy"
   

gcloud

gcloud kms keys remove-iam-policy-binding \
--project=KMS_PROJECT_ID \
--member serviceAccount:[e-mail] \
--role roles/cloudkms.cryptoKeyEncrypterDecrypter \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING \
KMS_KEY

Disable CMEK for your organization

Disabling CMEK for your organization removes CMEK policy enforcement for future operations only; any previously applied configurations remain intact.

To disable CMEK for your organization, run this command:

API

1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

2. Create a JSON file that contains the following information:

   {
       "kms_key_name": ""
   }
   

3. Issue a PATCH request to update kms_key_name:

   curl -X PATCH \
       -H "Authorization: Bearer AUTH_TOKEN \
       -H "Content-Type: application/json" \
       --data-binary @JSON_FILE_NAME
       https://logging.googleapis.com/v2/organizations/ORGANIZATION_ID/settings?update_mask="kms_key_name"
   

gcloud

gcloud logging settings update --organization=ORGANIZATION_ID --clear-kms-key

If you want to destroy your key, see Destroying and restoring key versions.

Impact of Cloud KMS key rotation

The Log Router doesn't automatically rotate the encryption key for temporary disaster recovery files when the Cloud KMS key associated with the Google Cloud organization rotates. Existing recovery files continue to use the key version with which they were created. New recovery files use the current primary key version.

Route logs to supported destinations

If you use log sinks to route logs to a supported destination, consider the following:

• Cloud Storage supports CMEK for routing logs. Use the instructions on this page to properly encrypt your data. For further details, see Using customer-managed encryption keys.

  If, when routing logs data to Cloud Storage, data is lost due to key unavailability, you can retroactively copy logs in bulk to Cloud Storage. For details, see Copy log entries.

• BigQuery, by default, encrypts customer content stored at rest. For details, see Protecting data with Cloud Key Management Service keys.

• Pub/Sub, by default, encrypts customer content stored at rest. For details, see Configuring message encryption.

External key considerations

When you use a Cloud EKM key, Google has no control over the availability of your externally managed key in the external key-management partner system.

If an externally managed key is unavailable, Cloud Logging continues retrying to access the key and buffer the incoming log data only for up to one hour. After an hour, if Cloud Logging is still unable to access the externally managed key, Cloud Logging begins dropping the data.

See the Cloud External Key Manager documentation for more considerations, and potential alternatives, when using external keys.

Limitations

The following are known limitations of CMEK for the Log Router.

Organization-level configuration only

CMEK for the Log Router can currently only be configured for Google Cloud organizations. Once configured, CMEK propagates to all Cloud projects and folders in the Google Cloud organization.

Disaster recovery file unavailability

A Cloud KMS key is considered available and accessible by Logging if both of the following are true:

• The key is enabled.
• The Logging service account has encrypt and decrypt permissions on the key.

If Logging loses access to the Cloud KMS key, Logging is unable to write temporary disaster-recovery files and, for users, queries will stop functioning. Query performance may remain degraded even after key access is restored.

Routing logs to Cloud Storage might also be affected because the Log Router will be unable to write temporary files required to facilitate the routing. If an error is encountered while encrypting or decrypting data, then a notification is sent to the Cloud project that contains the Cloud KMS key.

Client library availability

Logging client libraries don't provide methods for configuring CMEK.

Quotas

For details on Logging usage limits, see Quotas and limits.

Troubleshoot configuration errors

The following sections describe how to find and mitigate common CMEK configuration errors.

As you configure CMEK, the Cloud project containing the Cloud KMS key is notified of related issues. As examples, updates fail if KMS_KEY_NAME is invalid, if the associated service account doesn't have the required Cloud Key Management Service CryptoKey Encrypter/Decrypter role, or if access to the key is disabled.

Identify configuration errors

To find and view the CMEK configuration errors, do the following:

1. Navigate to the Google Cloud console:

   Go to Google Cloud console

2. Select the Cloud project that contains the encryption key.

   You can identify the project id by running the following command:

   gcloud logging settings describe --organization=ORGANIZATION_ID
   
   

   Running this command returns the following:

    kmsKeyName: projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEYRING/cryptoKeys/KEY
    serviceAccountId: [e-mail]
    

   The value of the kmsKeyName field includes the project ID of the key.

3. Select the Activity tab in the Google Cloud console.

4. Check for Cloud Logging Cloud Logging CMEK Configuration Error notifications.

   Each error notification contains steps that you can take to mitigate the issue:

   Error	Recommendation
   Cryptographic key permission denied	The Logging service account associated with your Cloud project doesn't have sufficient IAM permissions to operate on the specified Cloud KMS key. Follow the instructions in the error or in Grant encryption/decryption permission to grant the proper IAM permission.
   Cryptographic key is disabled	The specified Cloud KMS key was disabled. Follow the instructions in the error to re-enable the key.
   Cryptographic key was destroyed	The specified Cloud KMS key was destroyed. Follow the instructions in the error or in Configure the Cloud KMS key to configure CMEK encryption with a different key.

Verify key usability

To verify the key's usability, run the following command to list all keys:

gcloud kms keys list \
--location=KMS_KEY_LOCATION \
--keyring=KMS_KEY_RING

This command returns information about each key in a tabular format. The first line of the output is a list of column names:

NAME PURPOSE ...

Verify that the Cloud KMS key is listed in the command's output as ENABLED, and that the purpose of the key is symmetric encryption: the PURPOSE column must contain ENCRYPT_DECRYPT and the PRIMARY_STATE column must contain ENABLED.

If necessary, create a new key.

Verify permissions configuration

Service accounts that are associated with the organization's CMEK settings must have the Cloud KMS CryptoKey Encrypter/Decrypter role for the configured key.

To list the key's IAM policy, run the following command:

gcloud kms keys get-iam-policy KMS_KEY_NAME

If necessary, add the service account that contains the Cloud KMS CryptoKey Encrypter/Decrypter role to the key.