Using customer-managed encryption keys (CMEK)

This page describes how to perform tasks related to customer-managed encryption keys (CMEK) for Cloud Bigtable. For more information about CMEK in general, including when and why to enable it, see the Cloud KMS documentation.

We recommend that you use the Google Cloud Console for all key management tasks. If you plan to use the gcloud command-line tool, install the Cloud SDK for Bigtable.

Creating a CMEK-enabled instance

Before you can create any CMEK-protected Bigtable resources, you need to complete the following:

  1. Create (or retrieve) a service agent.
  2. Create a symmetric CMEK key in Cloud KMS.
  3. Configure IAM for that key.

Create a Bigtable service agent

Before you create a CMEK key, you must have a Cloud Bigtable service agent, which is a type of Google-managed service account that Bigtable uses to access the key.

Console

You are not able to create a service agent in the Cloud Console. However, if you create your key in the Cloud Console, you are prompted to grant the Cloud KMS Encrypter/Decrypter role, and the service agent is created at that time if it does not yet exist.

gcloud

  1. Run the gcloud services identity create command to view the service agent that Bigtable uses to access the CMEK key on your behalf. This command creates the service account if it does not already exist, then displays it.

    gcloud beta services identity create \
        --service=bigtableadmin.googleapis.com \
        --project CBT_PROJECT
    

    Replace CBT_PROJECT with the project that contains your Bigtable resources.

    The command displays the service agent ID, which is formatted like an email address. Record the output email string, because you'll use it in a later step.

    Service identity created:
    service-xxx@gcp-sa-bigtable.iam.gserviceaccount.com
    

Create a key

Only regional keys can be used with Bigtable, and the region you select for your key must contain the zones in which you create Bigtable resources. For example, if you create a key ring in us-central1 (Iowa), then clusters in us-central1-a, us-central1-b, and us-central1-c can be protected by keys from that key ring.

Console

To create your key using the Cloud Console, follow the instructions at Creating symmetric keys.

gcloud

  1. Run the following command to enable the Cloud KMS API on your KMS_PROJECT project if you have not already:

    gcloud services enable cloudkms.googleapis.com \
        --project KMS_PROJECT
    

    Replace KMS_PROJECT with the project that contains your CMEK key.

  2. Create a Cloud KMS key ring to house your CMEK key.

    gcloud kms keyrings create KMS_KEYRING \
        --location KMS_LOCATION --project KMS_PROJECT
    

    Provide the following:

    • KMS_KEYRING: the KMS key ring that contains your keys
    • KMS_LOCATION: the region you select for your key ring
    • KMS_PROJECT: the project to contain your CMEK key ring
  3. Create your CMEK key within the key ring you just created:

    gcloud kms keys create KMS_KEY \
        --keyring KMS_KEYRING --location KMS_LOCATION \
        --purpose "encryption" --project KMS_PROJECT
    

    Provide the following:

    • KMS_KEY: the name you want to assign the key
    • KMS_KEYRING: the KMS key ring to contain your keys
    • KMS_LOCATION : the region that contains your key ring
    • KMS_PROJECT: the project to contain your CMEK key

    These commands do not output any results. To confirm that the key was successfully created or get a list of your existing keys with their primary version and status, run one of the following commands:

    To list the all key rings in a project:

    gcloud kms keyrings list --location KMS_LOCATION \
        --project KMS_PROJECT
    

    Provide the following:

    • KMS_LOCATION: the region where your key rings are located
    • KMS_PROJECT: the project that contains your key rings

    To list all keys within a key ring:

    gcloud kms keys list --keyring KMS_KEYRING \
        --location KMS_LOCATION --project KMS_PROJECT
    

    Provide the following:

    • KMS_KEYRING: the KMS key ring that contains your keys
    • KMS_LOCATION: the region where your key rings are located
    • KMS_PROJECT: the project that contains your key rings

Configure IAM settings for the key

Console

To grant an Cloud KMS role to your service agent, do the following. You are also able to grant permission at the key or key-ring level if you want lower granularity.

  1. In the Cloud Console, go to the IAM page.

    Go to the IAM page

  2. Click Add.

  3. Enter the email-formatted ID for your Bigtable service agent.

  4. Select the Cloud KMS CryptoKey Encrypter/Decrypter role.

  5. Click Save.

gcloud

  1. Grant the cloudkms.cryptoKeyEncrypterDecrypter role to your service agent:

    gcloud kms keys add-iam-policy-binding KMS_KEY \
        --keyring KMS_KEYRING\
        --location KMS_LOCATION \
        --member serviceAccount:SERVICE_ACCOUNT_EMAIL \
        --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
        --project KMS_PROJECT
    

    Provide the following:

    • KMS_KEY: the name you assigned to the key
    • KMS_KEYRING: the KMS key ring that contains the key
    • KMS_LOCATION : the region that contains the key ring
    • SERVICE_ACCOUNT_EMAIL: the email-formatted identifier for the service agent that you are granting access to
    • KMS_PROJECT: the project that contains the key

Create a CMEK-protected instance

With a CMEK key created and configured, you can now create a CMEK-protected instance. Your existing Bigtable instances that are protected by Google default encryption cannot be converted to use CMEK; you can only choose an encryption type and key at the time of creation.

Follow the steps on Creating a Bigtable instance to create your CMEK-protected instance.

Viewing the key in use

Information on key versions comes from the encryption_info field.

To view the CMEK key version information for a table, complete the following steps:

Console

  1. Go to the Bigtable instances page in the Cloud Console.

    Open the instance list

  2. Click the name of the instance that contains the table to open the Instance Details page and view the list of clusters in the instance.

  3. Next to a cluster ID, click the key name under Encryption key to view the Versions page for the cluster's key.

gcloud

For each cluster, you can confirm its CMEK configuration as follows:

    gcloud bigtable clusters describe CLUSTER_ID \
        --instance INSTANCE_ID --project CBT_PROJECT

Provide the following:

  • CLUSTER_ID: the permanent identifier for the cluster
  • INSTANCE_ID: the permanent identifier for the instance
  • CBT_PROJECT: the project that contains your Bigtable resources

The command displays output similar to the following:

    defaultStorageType: SSD
    encryptionConfig:
      kmsKeyName: projects/cloud-kms-project/locations/us-central1/keyRings/cloud-bigtable-keys/cryptoKeys/my-cmek-key
    location: projects/cloud-bigtable-project/locations/us-central1-a
    name: projects/cloud-bigtable-project/instances/cmek-test-instance/clusters/my-cluster
    serveNodes: 1
    state: READY

Disabling a key

Disabling a CMEK key version suspends access to all data protected by that key version. Destroying a key version is the permanent (after 24 hours) counterpart of this action.

Console

This is the recommended method. Using the Cloud Console lets you disable all versions of a key at once.

Follow these instructions for each key version.

gcloud

We recommend that you use the Cloud Console to disable keys. If you wish to use the gcloud command-line tool instead, run the following commands.

  1. List all versions of the CMEK key:

    gcloud kms keys versions list --key KMS_KEY \
        --keyring KMS_KEYRING --location KMS_LOCATION \
        --project KMS_PROJECT
    

    Provide the following:

    • KMS_KEY: the name of the CMEK key
    • KMS_KEYRING: the KMS key ring that contains the key
    • KMS_LOCATION: the region that contains the key ring
    • KMS_PROJECT: the project that contains the key

    The output is similar to the following:

    NAME                                  STATE
    KMS_KEY_NAME/cryptoKeyVersions/1      ENABLED
    KMS_KEY_NAME/cryptoKeyVersions/2      ENABLED
    KMS_KEY_NAME/cryptoKeyVersions/3      ENABLED
    KMS_KEY_NAME/cryptoKeyVersions/4      ENABLED
    
  2. Disable all versions:

    for $KV in 1 2 3 4;
    do
    gcloud kms keys versions disable KV --key KMS_KEY \
        --keyring KMS_KEYRING --location KMS_LOCATION \
        --project KMS_PROJECT;
    done
    

    Provide the following:

    • 1 2 3 4: the versions that you are disabling
    • KMS_KEY: the name of the CMEK key
    • KMS_KEYRING: the KMS key ring that contains the key
    • KMS_LOCATION: the region that contains the key ring
    • KMS_PROJECT: the project that contains the key

    Listing all key versions again will show their state flipped to DISABLED, though it may take up to 4 hours for Bigtable to act on this state change.

  3. [Optional] To confirm the status of a disabled CMEK after you've waited 4 hours, run the following command:

     gcloud bigtable instances tables describe TABLE_ID \
         --instance INSTANCE_ID --view ENCRYPTION \
         --project CBT_PROJECT
    

    Provide the following:

    • TABLE_ID: the permanent identifier for the table
    • INSTANCE_ID: the permanent identifier for the instance
    • CBT_PROJECT: the project that contains your Bigtable resources

    The command displays output similar to the following:

    clusterStates:
      CLUSTER:
        encryptionInfo:
        - encryptionStatus:
            code: 9
            details:
            - '@type': type.googleapis.com/google.rpc.PreconditionFailure
              violations:
              - subject: KMS_KEY_NAME/cryptoKeyVersions/<int>
                type: KEY_DISABLED
            message: KMS_KEY_NAME is not enabled, current state is: DISABLED.
          encryptionType: CUSTOMER_MANAGED_ENCRYPTION
          kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/<int>
    name: projects/CBT_PROJECT/instances/INSTANCE/tables/TABLE
    
  4. [Optional] To verify that the clusters in the instance are disabled, run the following command:

    gcloud bigtable clusters list --instances INSTANCE_ID
        --project CBT_PROJECT
    

    Provide the following:

    • INSTANCE_ID: the permanent identifier for the instance
    • CBT_PROJECT: the project that contains your Bigtable resources

    The command displays output similar to the following:

    NAME              ZONE           NODES  STORAGE    STATE
    my-cluster        us-central1-a  1      SSD        DISABLED
    my-other-cluster  us-central1-b  1      SSD        DISABLED
    

Enabling a key

If a key version has been disabled, you can re-enable it to regain access to your Bigtable resources. This option is available for 30 days from the time the key version is disabled.

Console

This is the recommended method. Using the Cloud Console lets you enable all versions of a key at once.

Follow these instructions for each key version.

gcloud

We recommend that you use the Cloud Console to enable keys. If you wish to use the gcloud command-line tool instead, run the following commands.

  1. View the versions list to identify all the versions of the key:

    gcloud kms keys versions list --key KMS_KEY \
        --keyring KMS_KEYRING --location KMS_LOCATION \
        --project KMS_PROJECT
    

    Provide the following:

    • KMS_KEY: the name of the CMEK key
    • KMS_KEYRING: the KMS key ring that contains the key
    • KMS_LOCATION: the region that contains the key ring
    • KMS_PROJECT: the project that contains the key

    The command displays output similar to the following:

    NAME                                  STATE
    KMS_KEY_NAME/cryptoKeyVersions/1      DISABLED
    KMS_KEY_NAME/cryptoKeyVersions/2      DISABLED
    KMS_KEY_NAME/cryptoKeyVersions/3      DISABLED
    KMS_KEY_NAME/cryptoKeyVersions/4      DISABLED
    

    Run the kms keys versions enable command, providing all versions that are listed. Using the example output, the command looks similar to the following:

    for $KV in 1 2 3 4;
    do
    gcloud kms keys versions enable KV --key KMS_KEY \
        --keyring KMS_KEYRING --location KMS_LOCATION \
        --project KMS_PROJECT;
    done
    

    Provide the following:

    • 1 2 3 4: the versions that you are disabling
    • KMS_KEY: the name of the CMEK key
    • KMS_KEYRING: the KMS key ring that contains the key
    • KMS_LOCATION: the region that contains the key ring
    • KMS_PROJECT: the project that contains the key

    Listing the key versions again shows their state flipped to ENABLED.

Viewing audit logs for a Cloud KMS key

Before you enable Cloud KMS Data Access audit logs, you should be familiar with Cloud Audit Logs.

Cloud KMS Data Access audit logs show you when Bigtable or any other products that are configured to use your CMEK key make encrypt/decrypt calls to Cloud KMS. Bigtable does not issue an encrypt/decrypt call on every data request, but instead maintains a poller that checks the key periodically. The polling results appear in the audit logs.

You can set up and interact with the audit logs in the Cloud Console:

  1. Make sure that logging is enabled for the Cloud KMS API in your project.

  2. Go to Cloud Logging in the Cloud Console.

    Go to Cloud Logging

  3. Limit the log entries to your Cloud KMS key by adding the following lines to the Query builder:

    resource.type="cloudkms_cryptokey"
    resource.labels.key_ring_id = KMS_KEYRING
    resource.labels.crypto_key_id = KMS_KEY
    

    Provide the following:

    • KMS_KEY: the name of the CMEK key
    • KMS_KEYRING: the KMS key ring that contains the key

    The log shows a couple log entries about every five minutes per table in each cluster. The log entries look similar to these examples:

    Info 2021-03-20 08:02:24.869 EDT Cloudkms.googleapis.com Decrypt projects/cloud-kms-project/locations/us-central1/keyRings/cloud-bigtable-keys/cryptoKeys/my-cmek-key service-123456789123@gcp-sa-bigtable.iam.gserviceaccount.com
    audit_log, method: "Decrypt", principal_email: "service-1234567891011@gcp-sa-bigtable.iam.gserviceaccount.com"
    
    Info 2021-03-20 08:02:24.913 EDT Cloudkms.googleapis.com Encrypt projects/cloud-kms-project/locations/us-central1/keyRings/cloud-bigtable-keys/cryptoKeys/my-cmek-key service-123456789123@gcp-sa-bigtable.iam.gserviceaccount.com
    audit_log, method: "Encrypt", principal_email: "service-123456789123@gcp-sa-bigtable.iam.gserviceaccount.com"
    

See Understanding audit logs for details about interpreting audit logs.

Viewing the encryption status of a table

Console

You are not able to view the encryption status of a table in the Cloud Console.

gcloud

To view the CMEK details on a table, set view to ENCRYPTION or FULL. The response includes the in-use key version and its status as seen by Bigtable.

    gcloud alpha bigtable instances tables describe TABLE_ID \
        --instance INSTANCE_ID --view ENCRYPTION \
        --project CBT_PROJECT

Provide the following:

  • TABLE_ID: the permanent identifier for the cluster
  • INSTANCE_ID: the permanent identifier for the instance that contains the backup
  • CBT_PROJECT: the project that contains your Bigtable resources

The command displays output similar to the following:

    clusterStates:
      my-cluster:
        encryptionInfo:
        - encryptionStatus: {}
          encryptionType: CUSTOMER_MANAGED_ENCRYPTION
          kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/4
      my-other-cluster:
        encryptionInfo:
        - encryptionStatus: {}
          encryptionType: CUSTOMER_MANAGED_ENCRYPTION
          kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/4
      name: projects/cloud-bigtable-project/instances/cmek-test-instance/tables/my-table

Viewing the encryption details for a backup

A backup stays pinned to its original key version. Key version rotations in Cloud KMS don't rotate the versions used to decrypt backups.

Console

  1. Go to the Bigtable instances page in the Cloud Console.

    Open the instance list

  2. Click the instance name to open the Instance Details page..

  3. In the left navigation pane, click Backups.

The key name and key version for each backup are listed under Customer managed key.

gcloud

Run the following command to view encryption info for a backup.

    gcloud alpha bigtable backups describe BACKUP_ID \
        --instance INSTANCE_ID --cluster CLUSTER_ID \
        --project CBT_PROJECT

Provide the following:

  • BACKUP_ID: the ID assigned to the backup
  • CLUSTER_ID: the permanent identifier for the cluster
  • INSTANCE_ID: the permanent identifier for the instance that contains the backup
  • CBT_PROJECT: the project that contains your Bigtable resources

The command returns output similar to the following. The kmsKeyVersion is the CMEK key version that the backup is pinned to. The status of the key version is not reported.

    encryptionInfo:
      encryptionStatus:
        code: 2
        message: Status of the associated key version is not tracked.
      encryptionType: CUSTOMER_MANAGED_ENCRYPTION
      kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/3
    endTime: '2020-09-01T00:03:26.568600Z'
    expireTime: '2020-09-02T00:03:25.436473Z'
    name: projects/cloud-bigtable-project/instances/cmek-test-instance/clusters/my-cluster/backups/my-backup
    sizeBytes: '3780'

What's next