Customer-managed encryption keys (CMEK)

By default, all the data at rest in Cloud Bigtable is encrypted using Google's default encryption. Bigtable handles and manages this encryption for you without any additional action on your part.

If you have specific compliance or regulatory requirements related to the keys that protect your data, you can use customer-managed encryption keys (CMEK) for Bigtable. Instead of Google managing the encryption keys that protect your data, your Bigtable instance is protected using a key that you control and manage in Cloud Key Management Service (Cloud KMS).

This page describes CMEK for Bigtable. For more information about CMEK in general, including when and why to enable it, see the Cloud KMS documentation. For instructions on performing CMEK-related tasks with Bigtable, see Using CMEK.

Features

  • Security: CMEK provides the same level of security as Google's default encryption but provides more administrative control.

  • Data access control: Administrators can rotate, manage access to, and disable or destroy the key used to protect data at rest in Bigtable.

  • Auditability: All actions on your CMEK keys are logged and viewable in Cloud Logging.

  • Comparable performance: Bigtable CMEK-protected instances offer comparable performance to Bigtable instances that use Google default encryption.

  • Flexibility: You can use the same CMEK key in multiple projects or instances or you can use separate keys, depending on your business needs.

Pricing

Cloud KMS charges for the cost of the key and any cryptographic operations performed using that key. See Cloud KMS pricing for details.

You are billed for the operation costs when Bigtable asks the Cloud KMS key to perform an encryption or decryption operation. Each encryption or decryption request is sent from every table on every cluster in the instance. Because Bigtable uses envelope encryption, these costs per table are generally low, given the small number of expected cryptographic operations. If you store many tables in a CMEK-protected instance, your costs are higher.

There are no additional Bigtable costs for using CMEK-enabled instances.

What is protected with CMEK

In a CMEK-protected instance, Bigtable uses your CMEK key to protect your data at rest. This includes data in all tables in the cluster. Data stored on both HDD and SSD storage is protected.

Some data is protected by Google default encryption at rest and not by the CMEK key:

  • A subset of row keys that mark range boundaries and are used for routing
  • Debugging data including core dumps and operational logs
  • Data in transit or in memory
  • A subset of timestamp values used for garbage collection

Bigtable uses envelope encryption for data at rest. The CMEK key is used as a key encryption key (KEK) to encrypt other keys used by Bigtable. When you rotate the CMEK key, Bigtable only needs to re-encrypt the intermediate keys.

Enabling CMEK

At a high level, to use CMEK with Bigtable, you follow these steps:

  1. Create and configure a CMEK key.
  2. Create a new Bigtable instance that uses the CMEK key.
  3. Schedule a key rotation.

Applications that use Bigtable do not need to specify keys or encryption configs when reading, writing, or deleting data. Bigtable is able to access the key on your behalf after you grant the Cloud KMS Encrypter/Decrypter role to a Bigtable service agent, which is a type of Google-managed service account.

For detailed instructions, see Using CMEK.

You can use the following when you work with CMEK for Bigtable.

You can also access the Admin API directly, but we strongly recommend that you do so only if you cannot use a Bigtable client library that makes CMEK calls to the API.

Key management

Key management operations are performed using Cloud KMS. Bigtable cannot detect or act upon any key changes until they are propagated by Cloud KMS. Some operations, such as disabling or destroying a key, can take up to 3 hours to propagate; changes to a key's permissions usually propagate more quickly.

After you create at least one table in a CMEK-protected instance, Bigtable calls each table in each cluster every 5 minutes to make sure that the key is still valid.

If Bigtable detects that your key has been disabled or destroyed, an operation to make your instance inaccessible begins immediately. Any subsequent data operations that are sent to a cluster that has been disabled return a FAILED_PRECONDITION error: Request failed due to a problematic CMEK state.

Bigtable does not disable an instance until all clusters in the instance have identified a disabled or destroyed key. After the first cluster reports a key is disabled or destroyed and until the instance is disabled, some data requests might succeed and others return an error. This state might last for up to several hours.

If Bigtable's calls to Cloud KMS detect that a formerly disabled key has been re-enabled, Cloud KMS restores access to the Bigtable cluster automatically.

If a Cloud KMS key has been deleted, then any Bigtable instance that was encrypted with that key becomes permanently inaccessible.

How an unavailable key status is handled

In rare scenarios, such as during periods when Cloud KMS is unavailable, Bigtable may be unable to retrieve the status of your key from Cloud KMS.

If your Bigtable instance is protected by a key that is enabled at the time at which Bigtable is first unable to communicate with Cloud KMS, Bigtable continues to support full instance operations on a best-effort basis using cached keys derived from the Cloud KMS key for a period of up to 1 hour, to minimize the impact of any such incident on your workload.

After an hour, if Bigtable is still unable to connect with Cloud KMS, Bigtable begins taking the instance offline as a protective measure. The data in your Bigtable instance remains inaccessible until your instance can reconnect with Cloud KMS and Cloud KMS responds that the key is active.

Conversely, if your Bigtable instance is protected by a key that was disabled prior to the point at which Bigtable is first unable to communicate with Cloud KMS, your instance remains inaccessible until it is able to reconnect to Cloud KMS and you have re-enabled your key.

Backups

Like other data in an instance, backups are protected by the instance's CMEK key. New tables that are restored from a backup are protected by the CMEK key for the instance where they are restored. For further details about how CMEK affects backups and restore operations, see the Backups overview. To learn how to create or restore from a backup, see Managing backups.

Logging

You can audit the requests that Bigtable sends to Cloud KMS on your behalf in Cloud Logging, if you have enabled Cloud KMS audit logs for the Cloud KMS API in your project. You can expect a few log entries every 5 minutes or so per table in each cluster.

Limitations

  • CMEK for Bigtable is not supported for instances that have clusters in more than one region.

    • When you create a Cloud KMS key ring, be sure to select the region that corresponds with your planned Bigtable zonal configuration.
    • Bigtable is only compatible with regional Cloud KMS key rings. Dual-region, multi-region, and global key rings are not supported.
    • All clusters in a Bigtable instance must be configured with the same CMEK key.
  • CMEK can only be configured at the cluster level. You cannot configure CMEK on backups, tables, or app profiles.

  • The encryption configuration of a Bigtable resource (an instance, cluster, table or backup) is immutable.

    • Non-CMEK instances cannot be converted to use CMEK.
    • CMEK instances cannot be converted to use Google default encryption.
    • Instances created with a CMEK key cannot be reconfigured to use a different key.
    • New clusters added to an instance must be configured with the same CMEK key (or lack of key) as the sibling clusters.
  • CMEK-protected Bigtable resources (instances, clusters, tables, or backups) tied to a key that has been made inaccessible as the result of a user-triggered action (such as disabling or destroying a key, or by revoking the Encrypter/Decrypter role) for more than 30 consecutive days are automatically deleted.

  • If you re-enable a disabled CMEK key to restore access to Bigtable instances protected by that key, some Data API requests might time out while your data is brought back online.

  • For up to five minutes after a table is created in a CMEK-protected instance, its key version and key status might be reported as unknown. However, any data written to the table is still protected with your CMEK key during that time.

  • Disabling or deleting only one version instead of all versions of a key that Bigtable uses can result in unpredictable behavior. Always disable or delete all versions of a CMEK key.

What's next