Use customer-managed encryption keys (CMEK)
This page describes how to perform tasks related to customer-managed encryption keys (CMEK) for Bigtable. For more information about CMEK in general, including when and why to enable it, see the Cloud KMS documentation.
We recommend that you use the Google Cloud console for all key management tasks. If you plan to use the Google Cloud CLI, install the gcloud CLI for Bigtable.
Prepare your CMEK keys
Before you can create a CMEK-protected Bigtable resource, you need to complete the following steps to create a CMEK key for each region that the instance's clusters will be in:
- Create (or retrieve) a Bigtable service agent.
- Create a CMEK key.
- Configure IAM settings for that key.
Create a Bigtable service agent
Before you create a CMEK key, you must have a Bigtable service agent, which Bigtable uses to access the key.
Console
You are not able to create a service agent in the Google Cloud console. However, if you create your key in the Google Cloud console, you are prompted to grant the Cloud KMS Encrypter/Decrypter role, and the service agent is created at that time if it does not yet exist.
gcloud
Run the gcloud services identity create command to view the service agent that Bigtable uses to access the CMEK key on your behalf. This command creates the service account if it does not already exist, then displays it.
gcloud beta services identity create \ --service=bigtableadmin.googleapis.com \ --project CBT_PROJECT
Replace
CBT_PROJECT
with the project that contains your Bigtable resources.The command displays the service agent ID, which is formatted like an email address. Record the output email string, because you'll use it in a later step.
Service identity created: service-xxx@gcp-sa-bigtable.iam.gserviceaccount.com
Create a key
You can use a key created directly in Cloud KMS or an externally managed key that you make available with Cloud External Key Manager.
In the Google Cloud project where you want to manage your keys:
Create a key ring and a key using one of the following options:
- Create the key ring and key directly in Cloud KMS.
- Use an externally-managed key. Create the external key and then create an Cloud EKM key to make the key available through Cloud KMS.
The Cloud KMS key location must be the same as the Bigtable cluster that it will be used with. For example, if you create a key ring and key in
us-central1
(Iowa), then clusters inus-central1-a
,us-central1-b
, andus-central1-c
can be protected by keys from that key ring.
Configure IAM settings for the key
Console
To grant an Cloud KMS role to your service agent, do the following. You are also able to grant permission at the key or key-ring level if you want lower granularity.
In the Google Cloud console, go to the IAM page.
Click Add.
Enter the email-formatted ID for your Bigtable service agent.
Select the Cloud KMS CryptoKey Encrypter/Decrypter role.
Click Save.
gcloud
Grant the
cloudkms.cryptoKeyEncrypterDecrypter
role to your service agent:gcloud kms keys add-iam-policy-binding KMS_KEY \ --keyring KMS_KEYRING\ --location KMS_LOCATION \ --member serviceAccount:SERVICE_ACCOUNT_EMAIL \ --role roles/cloudkms.cryptoKeyEncrypterDecrypter \ --project KMS_PROJECT
Provide the following:
KMS_KEY
: the name you assigned to the keyKMS_KEYRING
: the KMS key ring that contains the keyKMS_LOCATION
: the region that contains the key ringSERVICE_ACCOUNT_EMAIL
: the email-formatted identifier for the service agent that you are granting access toKMS_PROJECT
: the project that contains the key
Create a CMEK-enabled instance
After your CMEK keys are created and configured, you can create a CMEK-protected instance. Existing Bigtable instances that are protected by Google default encryption cannot be converted to use CMEK; you can only choose an encryption type and key at the time of creation.
Follow the steps on Create an instance to create your CMEK-protected instance.
View the key in use
Information on key versions comes from the encryption_info
field.
To view the CMEK key version information for a table, complete the following steps:
Console
Go to the Bigtable instances page in the Google Cloud console.
Click the name of the instance that contains the table to open the Instance Details page and view the list of clusters in the instance.
Next to a cluster ID, click the key name under Encryption key to view the Versions page for the cluster's key.
gcloud
For each cluster, you can confirm its CMEK configuration as follows:
gcloud bigtable clusters describe CLUSTER_ID \
--instance INSTANCE_ID --project CBT_PROJECT
Provide the following:
CLUSTER_ID
: the permanent identifier for the clusterINSTANCE_ID
: the permanent identifier for the instanceCBT_PROJECT
: the project that contains your Bigtable resources
The command displays output similar to the following:
defaultStorageType: SSD
encryptionConfig:
kmsKeyName: projects/cloud-kms-project/locations/us-central1/keyRings/cloud-bigtable-keys/cryptoKeys/my-cmek-key
location: projects/cloud-bigtable-project/locations/us-central1-a
name: projects/cloud-bigtable-project/instances/cmek-test-instance/clusters/my-cluster
serveNodes: 1
state: READY
Disable a key
Disabling a CMEK key version suspends access to all data protected by that key version. Destroying a key version is the permanent (after 24 hours) counterpart of this action. Always disable all of the keys for an instance rather than disabling only the keys for one or some clusters.
Console
This is the recommended method. Using the Google Cloud console lets you disable all versions of a key at once.
Follow these instructions for each key version.
gcloud
We recommend that you use the Google Cloud console to disable keys. If you wish to use the Google Cloud CLI instead, run the following commands.
List all versions of the CMEK key:
gcloud kms keys versions list --key KMS_KEY \ --keyring KMS_KEYRING --location KMS_LOCATION \ --project KMS_PROJECT
Provide the following:
KMS_KEY
: the name of the CMEK keyKMS_KEYRING
: the KMS key ring that contains the keyKMS_LOCATION
: the region that contains the key ringKMS_PROJECT
: the project that contains the key
The output is similar to the following:
NAME STATE KMS_KEY_NAME/cryptoKeyVersions/1 ENABLED KMS_KEY_NAME/cryptoKeyVersions/2 ENABLED KMS_KEY_NAME/cryptoKeyVersions/3 ENABLED KMS_KEY_NAME/cryptoKeyVersions/4 ENABLED
Disable all versions:
for $KV in 1 2 3 4; do gcloud kms keys versions disable KV --key KMS_KEY \ --keyring KMS_KEYRING --location KMS_LOCATION \ --project KMS_PROJECT; done
Provide the following:
1 2 3 4
: the versions that you are disablingKMS_KEY
: the name of the CMEK keyKMS_KEYRING
: the KMS key ring that contains the keyKMS_LOCATION
: the region that contains the key ringKMS_PROJECT
: the project that contains the key
Listing all key versions again will show their state flipped to
DISABLED
, though it may take up to 4 hours for Bigtable to act on this state change.[Optional] To confirm the status of a disabled CMEK after you've waited 4 hours, run the following command:
gcloud bigtable instances tables describe TABLE_ID \ --instance INSTANCE_ID --view ENCRYPTION \ --project CBT_PROJECT
Provide the following:
TABLE_ID
: the permanent identifier for the tableINSTANCE_ID
: the permanent identifier for the instanceCBT_PROJECT
: the project that contains your Bigtable resources
The command displays output similar to the following:
clusterStates: CLUSTER: encryptionInfo: - encryptionStatus: code: 9 details: - '@type': type.googleapis.com/google.rpc.PreconditionFailure violations: - subject: KMS_KEY_NAME/cryptoKeyVersions/<int> type: KEY_DISABLED message: KMS_KEY_NAME is not enabled, current state is: DISABLED. encryptionType: CUSTOMER_MANAGED_ENCRYPTION kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/<int> name: projects/CBT_PROJECT/instances/INSTANCE/tables/TABLE
[Optional] To verify that the clusters in the instance are disabled, run the following command:
gcloud bigtable clusters list --instances INSTANCE_ID --project CBT_PROJECT
Provide the following:
INSTANCE_ID
: the permanent identifier for the instanceCBT_PROJECT
: the project that contains your Bigtable resources
The command displays output similar to the following:
NAME ZONE NODES STORAGE STATE my-cluster us-central1-a 1 SSD DISABLED my-other-cluster us-central1-b 1 SSD DISABLED
Enable a key
If a key version has been disabled, you can re-enable it to regain access to your Bigtable resources. This option is available for 30 days from the time the key version is disabled.
Console
This is the recommended method. Using the Google Cloud console lets you enable all versions of a key at once.
Follow these instructions for each key version.
gcloud
We recommend that you use the Google Cloud console to enable keys. If you wish to use the Google Cloud CLI instead, run the following commands.
View the versions list to identify all the versions of the key:
gcloud kms keys versions list --key KMS_KEY \ --keyring KMS_KEYRING --location KMS_LOCATION \ --project KMS_PROJECT
Provide the following:
KMS_KEY
: the name of the CMEK keyKMS_KEYRING
: the KMS key ring that contains the keyKMS_LOCATION
: the region that contains the key ringKMS_PROJECT
: the project that contains the key
The command displays output similar to the following:
NAME STATE KMS_KEY_NAME/cryptoKeyVersions/1 DISABLED KMS_KEY_NAME/cryptoKeyVersions/2 DISABLED KMS_KEY_NAME/cryptoKeyVersions/3 DISABLED KMS_KEY_NAME/cryptoKeyVersions/4 DISABLED
Run the
kms keys versions enable
command, providing all versions that are listed. Using the example output, the command looks similar to the following:for $KV in 1 2 3 4; do gcloud kms keys versions enable KV --key KMS_KEY \ --keyring KMS_KEYRING --location KMS_LOCATION \ --project KMS_PROJECT; done
Provide the following:
1 2 3 4
: the versions that you are disablingKMS_KEY
: the name of the CMEK keyKMS_KEYRING
: the KMS key ring that contains the keyKMS_LOCATION
: the region that contains the key ringKMS_PROJECT
: the project that contains the key
Listing the key versions again shows their state flipped to
ENABLED
.
View audit logs for a Cloud KMS key
Before you enable Cloud KMS Data Access audit logs, you should be familiar with Cloud Audit Logs.
Cloud KMS Data Access audit logs show you when Bigtable or any other products that are configured to use your CMEK key make encrypt/decrypt calls to Cloud KMS. Bigtable does not issue an encrypt/decrypt call on every data request, but instead maintains a poller that checks the key periodically. The polling results appear in the audit logs.
You can set up and interact with the audit logs in the Google Cloud console:
Make sure that logging is enabled for the Cloud KMS API in your project.
Go to Cloud Logging in the Google Cloud console.
Limit the log entries to your Cloud KMS key by adding the following lines to the Query builder:
resource.type="cloudkms_cryptokey" resource.labels.key_ring_id = KMS_KEYRING resource.labels.crypto_key_id = KMS_KEY
Provide the following:
KMS_KEY
: the name of the CMEK keyKMS_KEYRING
: the KMS key ring that contains the key
The log shows a couple log entries about every five minutes per table in each cluster. The log entries look similar to these examples:
Info 2021-03-20 08:02:24.869 EDT Cloudkms.googleapis.com Decrypt projects/cloud-kms-project/locations/us-central1/keyRings/cloud-bigtable-keys/cryptoKeys/my-cmek-key service-123456789123@gcp-sa-bigtable.iam.gserviceaccount.com audit_log, method: "Decrypt", principal_email: "service-1234567891011@gcp-sa-bigtable.iam.gserviceaccount.com" Info 2021-03-20 08:02:24.913 EDT Cloudkms.googleapis.com Encrypt projects/cloud-kms-project/locations/us-central1/keyRings/cloud-bigtable-keys/cryptoKeys/my-cmek-key service-123456789123@gcp-sa-bigtable.iam.gserviceaccount.com audit_log, method: "Encrypt", principal_email: "service-123456789123@gcp-sa-bigtable.iam.gserviceaccount.com"
See Understanding audit logs for details about interpreting audit logs.
View key usage
Cloud KMS lets you view the following information about the Bigtable resources that your keys protect.
- Key usage summary: Includes information about the number of protected resources and projects that use the key.
- Key usage detail: Identifies the resources that a key protects and the resources that depend on the key.
The following Bigtable resources are supported:
bigtableadmin.googleapis.com/Backup
bigtableadmin.googleapis.com/Cluster
bigtableadmin.googleapis.com/Table
For more information about viewing key usage, see View key usage information.
View the encryption status of a table
Console
You are not able to view the encryption status of a table in the Google Cloud console.
gcloud
To view the CMEK details on a table, set view
to ENCRYPTION
or FULL
.
The response includes the in-use key version and its status as seen by
Bigtable.
gcloud alpha bigtable instances tables describe TABLE_ID \
--instance INSTANCE_ID --view ENCRYPTION \
--project CBT_PROJECT
Provide the following:
TABLE_ID
: the permanent identifier for the clusterINSTANCE_ID
: the permanent identifier for the instance that contains the backupCBT_PROJECT
: the project that contains your Bigtable resources
The command displays output similar to the following:
clusterStates:
my-cluster:
encryptionInfo:
- encryptionStatus: {}
encryptionType: CUSTOMER_MANAGED_ENCRYPTION
kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/4
my-other-cluster:
encryptionInfo:
- encryptionStatus: {}
encryptionType: CUSTOMER_MANAGED_ENCRYPTION
kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/4
name: projects/cloud-bigtable-project/instances/cmek-test-instance/tables/my-table
View the encryption details for a backup
A backup stays pinned to its original key version. Key version rotations in Cloud KMS don't rotate the versions used to decrypt backups.
Console
Go to the Bigtable instances page in the Google Cloud console.
Click the instance name to open the Instance Details page.
In the left navigation pane, click Backups.
The key name and key version for each backup are listed under Customer managed key.
gcloud
Run the following command to view encryption info for a backup.
gcloud alpha bigtable backups describe BACKUP_ID \
--instance INSTANCE_ID --cluster CLUSTER_ID \
--project CBT_PROJECT
Provide the following:
BACKUP_ID
: the ID assigned to the backupCLUSTER_ID
: the permanent identifier for the clusterINSTANCE_ID
: the permanent identifier for the instance that contains the backupCBT_PROJECT
: the project that contains your Bigtable resources
The command returns output similar to the following. The kmsKeyVersion
is the CMEK key version that the backup is pinned to. The status of the
key version is not reported.
encryptionInfo:
encryptionStatus:
code: 2
message: Status of the associated key version is not tracked.
encryptionType: CUSTOMER_MANAGED_ENCRYPTION
kmsKeyVersion: KMS_KEY_NAME/cryptoKeyVersions/3
endTime: '2020-09-01T00:03:26.568600Z'
expireTime: '2020-09-02T00:03:25.436473Z'
name: projects/cloud-bigtable-project/instances/cmek-test-instance/clusters/my-cluster/backups/my-backup
sizeBytes: '3780'