Customer-managed encryption keys (CMEK)

By default, all the data at rest in Firestore in Datastore mode is encrypted using Google's default encryption. Datastore mode handles and manages this encryption for you without any additional action on your part.

If you have specific compliance or regulatory requirements related to the keys that protect your data, you can use customer-managed encryption keys (CMEK) for Datastore mode. Instead of Google managing the encryption keys that protect your data, your Datastore mode database is protected using a key that you control and manage in Cloud Key Management Service (Cloud KMS).

This page describes CMEK for Datastore mode. For more information about CMEK in general, including when and why to enable it, see the following Cloud KMS documentation:

• Customer-managed encryption keys (CMEK)
• Best practices for using CMEKs

For instructions on performing CMEK-related tasks with Datastore mode, see Use CMEK.

Note: For information about access to this release, see the access request form.

Features

• Data control: CMEK lets you manage the KMS key. You can rotate, disable, and destroy the key used to encrypt the data at rest in your Datastore mode database.
• Performance: CMEK does not impact the Firestore SLA.
• Auditability: If you enable audit logging for Cloud KMS, all the operations on the key are logged and viewable in Cloud Logging.
• Organization policy constraints: You can use CMEK organization policy constraints to specify encryption compliance requirements for Datastore mode databases in your organization.

Pricing

Cloud KMS charges for the cost of the key and any cryptographic operations performed using that key. For more information, see Cloud KMS pricing.

You are billed for the operation costs when Datastore mode asks the Cloud KMS key to perform an encryption or decryption operation. Encryption/decryption operation by the customer-managed key occurs every 5 minutes and is not synchronized with database requests. Costs are generally low, given the expected number of cryptographic operations generated by Datastore mode. Costs for Cloud Audit Logs are an additional expense, but is also expected to be generally low, given the expected number of cryptographic operations.

There are no additional Datastore mode costs for using CMEK-protected database and the Datastore mode pricing continues to apply.

If you revoke your key to a database, storage cost will be charged based on the size of the last day that the key was available. You will continue to incur storage costs at that database size until the database is deleted or the key becomes available again.

What is protected with CMEK

When you create a Datastore mode CMEK-protected database, your Cloud KMS key is used to protect data at rest. This includes data that you store on a disk or a flash drive, including indexes and backups. Some exceptions apply. The following data types are encrypted with Google default encryption and not by the CMEK key:

• Data in transit or in memory
• Database metadata

How an unavailable key status is handled

Encrypt and decrypt operations are not issued on every data request. Instead, the Firestore system polls Cloud KMS every 5 minutes to check if the key is still available and then performs encrypt and decrypt operations if the key is available.

If the system detects that the key is unavailable, within 10 minutes any subsequent calls to the Firestore database, including reads, writes, and queries, return a FAILED_PRECONDITION error with the The customer-managed encryption key required by the requested resource is not accessible message.

If the database has time-to-live (TTL) policies, and if any expiration times get exceeded while the key is unavailable, data deletion by TTL will be delayed until the key gets reinstated. If the database has long-running operations in progress, they will be affected as follows:

• Data import or export operations will stop making progress, and be marked as Failed. The failed operations will not be retried if the key gets reinstated.
• Index build operations, and operations enabling new TTL policies will stop making progress. The stopped operations will be retried if the key gets reinstated.

Keys are considered unavailable in any situation that intentionally disallows Firestore from accessing the key. This includes:

• Disabling or destroying the in-use key version. Please be careful when destroying a key version, because this can cause unrecoverable data loss.
• Removing permission to access the key from the Firestore service account.

If the key is reinstated, the polling operation detects that the key is available again. Access is re-enabled, usually within minutes, but it can take up to a few hours in rare cases. Note that some operations on Cloud KMS keys, such as disabling or destroying a key, can take up to 3 hours to propagate. Firestore doesn't detect any changes until after they take effect in Cloud KMS.

Reinstatement of a key involves the following, depending on the situation:

• Re-enabling a disabled key version.
• Restoring a destroyed key version. Before being permanently destroyed, a key version is scheduled for destruction. You can only restore a key during the period when a key version is scheduled for destruction. You cannot restore a key that has already been permanently destroyed.
• Re-granting the Firestore service agent permission to access the key.

Warning: Don't let your key become unavailable for longer than seven days. CMEK-protected databases with keys unavailable for more than 7 days may be deleted. In the event of a key being unavailable, to preserve data beyond the seven days limit, we recommend that you enable backups for your Datastore mode CMEK database with the required retention period. Before you revoke the key, ensure a backup has been created, as a valid key is required for backup creation. Additional charges apply for backups. For backup pricing details, see Datastore mode pricing.

Key rotation considerations

When you rotate the CMEK key, Datastore mode re-encrypts the database with the latest primary version of the CMEK key. During the re-encryption process, keep both the old and the new key version available. Once re-encryption finishes, disabling or deleting the old versions of the CMEK key won't disable access to the database since it's encrypted with the new primary key version.

You can also view the key versions that are being used to protect a database. For more information, see View the key in use.

External key considerations

When you use a Cloud EKM key, Google has no control over the availability of your externally-managed key in the external key management partner system.

If an externally-managed key is unavailable, Datastore mode continues to support full database operations using a cached version of the key, for up to one hour.

After an hour, if Datastore mode is still unable to connect with Cloud KMS, Datastore mode begins taking the database offline as a protective measure. Calls to the database will fail with a FAILED_PRECONDITION error that includes additional details.

See the Cloud External Key Manager documentation for more considerations when using external keys.

Caution: If a database remains disabled for more than 7 consecutive days, Datastore mode might automatically delete it. To avoid permanent data loss, don't leave external keys in an inaccessible state for an extended time.

Warning: If an external key is deleted and cannot be recovered, any Datastore mode database encrypted with that key becomes permanently inaccessible.

Backup and restore

A backup uses the same encryption mechanism as the database from which you created it. When a CMEK-protected Datastore mode database creates a backup, it encrypts the backup with the primary key version used at the time of backup creation.

Datastore mode creates the first backup of a CMEK database after 24 hours pass from the moment when you enable backup schedules.

Note: Once the backup is created, you can't modify its key and key version, even if you rotate the Cloud KMS key.

For more information about Datastore mode backups, see Back up and restore data.

A database restored from a backup uses the same encryption mechanism as the backup by default. When you restore a database, you can specify a different encryption type in one of the following ways:

• Restore to a CMEK database with a newly specified key.
• Restore to a non-CMEK database that uses Google's default encryption.
• Restore to a database that uses the same encryption as the backup.

For more information about restoring a Datastore mode database from a backup, see Restore data from a database backup. For more information about restoring a CMEK-protected Datastore mode database from a backup, see Restore a CMEK-protected database.

Key tracking

You can use key tracking to view resources, for example, Datastore mode databases, that a key protects. For more information about key tracking, see View key usage.

CMEK and key availability

When keys are unavailable or disabled, be aware of the following behaviors that can occur in CMEK-enabled databases:

• You can change Datastore mode point-in-time recovery (PITR) settings on a CMEK-enabled database even if the key is unavailable because PITR settings are database metadata, which isn't encrypted by CMEK.
• You can delete a CMEK database that has unavailable keys.
• When you create a CMEK-enabled database, disabled keys don't show on the list of available keys in the Google Cloud console. If you manually input a disabled key, the database creation process will fail with a FAILED_PRECONDITION error 400.

Limitations

• You can't change a key for a CMEK-protected database. You can rotate, enable, and disable keys.
• CMEK-protected databases support Key Visualizer only for entity and document data, not for index data.
• You can't enable CMEK on existing databases. You can enable CMEK only on new databases, and you must enable it when you create the database. To migrate data in an existing non-CMEK database to a CMEK-protected database, export your data and then import data to a new CMEK-protected database. You can also restore data from a non-CMEK database to a CMEK database.
• Firestore supports a limited number of CMEK-protected databases.

What's next

• Learn how to use CMEK with Firestore in Datastore mode.