Use customer-managed encryption keys (CMEK)

Use customer-managed encryption keys (CMEK)

Preview

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

For information about access to this release, see the access request page.

This page describes how to perform tasks related to customer-managed encryption keys (CMEK) for Firestore. For more information about CMEK in general, including when and why to enable it, see the Cloud KMS documentation.

Prepare your CMEK keys

Before you can create a CMEK-protected Firestore database, you must complete the following steps:

1. Create (or retrieve) a Firestore service agent.
2. Create a CMEK key.
3. Configure IAM settings for that key.

Complete these steps for each project that will contain CMEK-protected Firestore databases. If you later create a new CMEK key, you must configure IAM settings for that key.

Note: The gcloud commands on this page show placeholders for two Google Cloud projects: the project that contains Firestore databases, FIRESTORE_PROJECT, and the project that contains your CMEK key(s), KMS_PROJECT. However, you can use the same project for your Firestore databases and CMEK keys.

Create a Firestore service agent

Before you create a CMEK key, you must have a Firestore service agent, which is a type of Google-managed service account that Firestore uses to access the key.

Run the services identity create command to create the service agent that Firestore uses to access the CMEK key on your behalf. This command creates the service account if it does not already exist, then displays it.

gcloud beta services identity create \
    --service=firestore.googleapis.com \
    --project FIRESTORE_PROJECT

Replace FIRESTORE_PROJECT with the project you plan to use for your Firestore databases.

The command displays the service agent ID, which is formatted like an email address. Record the output email string, because you'll use it in a later step.

Service identity created:
[e-mail]

Create a key

You can use a key created directly in Cloud KMS or an externally managed key that you make available with Cloud External Key Manager.

The Cloud KMS key location must be the same as the location of the Firestore database that it will be used with.

• For regional database locations, use the same location name for key ring, key, and database because the location names have a one-to-one mapping.

  For example, if you want to create a CMEK-protected database in us-west1, create a key ring and key in us-west1.

• For multi-region database locations, use the location name of the KMS multi-region location:

  • Use the Cloud KMS us multi-region location for the Firestore nam5 multi-region location.
  • Use the Cloud KMS europe multi-region location for the Firestore eur3 multi-region location.

In the Google Cloud project where you want to manage your keys, complete the following:

1. Enable the Cloud KMS API.

2. Create a key ring and a key using one of the following options:

   • Create the key ring and key directly in Cloud KMS.
   • Use an externally managed key. Create the external key and then create an Cloud EKM key to make the key available through Cloud KMS.

Configure IAM settings for the key

Console

To grant an Cloud KMS role to your service agent, do the following. You are also able to grant permission at the key or key-ring level if you want lower granularity.

1. In the Google Cloud console, go to the IAM page.

   Go to the IAM page

2. Click Add.

3. Enter the email-formatted ID for your Firestore service agent.

4. Select the Cloud KMS CryptoKey Encrypter/Decrypter role.

5. Click Save.

gcloud

1. Grant the cloudkms.cryptoKeyEncrypterDecrypter role to your service agent:

   gcloud kms keys add-iam-policy-binding KMS_KEY \
       --keyring KMS_KEYRING\
       --location KMS_LOCATION \
       --member serviceAccount:SERVICE_AGENT_EMAIL \
       --role roles/cloudkms.cryptoKeyEncrypterDecrypter \
       --project KMS_PROJECT
   

   Provide the following:

   • KMS_KEY: the name you assigned to the key
   • KMS_KEYRING: the KMS key ring that contains the key
   • KMS_LOCATION : the region that contains the key ring
   • SERVICE_AGENT_EMAIL: the email-formatted identifier for the service agent that you are granting access to
   • KMS_PROJECT: the project that contains the key

   The terminal should display a response similar to the following:

   Updated IAM policy for key KMS_KEY.
   bindings:
   - members:
     - serviceAccount:
       service-{project-number}@gcp-sa-firestore.iam.gserviceaccount.com
   role: roles/cloudkms.cryptoKeyEncrypterDecrypter
   

Create a CMEK-enabled database

After your CMEK keys are created and configured, you can create a CMEK-protected database. Existing Firestore database that are protected by Google default encryption cannot be converted to use CMEK; you can only choose an encryption type and key at the time of creation.

gcloud

gcloud alpha firestore databases create --location=FIRESTORE_DATABASE_LOCATION \
      --database=DATABASE_ID \
      --kms-key-name=KMS_KEY_NAME \
      --project=FIRESTORE_PROJECT

Provide the following:

• FIRESTORE_DATABASE_LOCATION: the Firestore location for the database
• DATABASE_ID: an ID for the database

• KMS_KEY_NAME: the name you assigned to the key. Use the full resource name for the key in the format of:

  projects/KMS_PROJECT/locations/KMS_LOCATION/keyRings/KMS_KEYRING_ID/cryptoKeys/KMS_KEY_ID

• FIRESTORE_PROJECT: the project to use for your Firestore database

REST API

HTTP request:

POST https://firestore.googleapis.com/v1/projects/{FIRESOTRE_PROJECT}/databases

In the request body configure CMEK in the cmek_config.kms_key_name field.

Set to the full resource ID of a Cloud KMS key. Only a key in the same location as this database is allowed.

This value should be the Cloud KMS key resource ID in the format of projects/{KMS_PROJECT}/locations/{KMS_LOCATION}/keyRings/{KMS_KEYRING_ID}/cryptoKeys/{KMS_KEY_ID}

For more details for other fields, see the database create page.

Example request:

curl -X POST 'https://firestore.googleapis.com/v1/projects/FIRESTORE_PROJECT/databases?databaseId={DATABASE_ID}' \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-type: application/json" \
-d '{
  "type":"FIRESTORE_NATIVE",
  "locationId":"{FIRESTORE_DATABASE_LOCATION}",
  "cmekConfig": {
    "kmsKeyName":"projects/KMS_PROJECT/locations/KMS_LOCATION/keyRings/KMS_KEYRING_ID/cryptoKeys/KMS_KEY_ID"
  }
}'

Terraform

To create a CMEK-enabled database, use the google_firestore_database resource. For more information and examples, see google_firestore_database.

resource "google_firestore_database" "database" {
  project     = "FIRESTORE_PROJECT"
  name        = "DATABASE_ID"
  location_id = "FIRESTORE_DATABASE_LOCATION"
  type        = "DATABASE_TYPE"

  cmek_config {
    kms_key_name = "KMS_KEY_NAME"
  }

}

Provide the following:

• FIRESTORE_PROJECT: the project to use for your Firestore database
• DATABASE_ID: an ID for the database
• FIRESTORE_DATABASE_LOCATION: the Firestore location for the database
• DATABASE_TYPE: either FIRESTORE_NATIVE for Native mode or DATASTORE_MODE for Datastore mode.

• KMS_KEY_NAME: the name you assigned to the key. Use the full resource name for the key in the format of:

  projects/KMS_PROJECT/locations/KMS_LOCATION/keyRings/KMS_KEYRING_ID/cryptoKeys/KMS_KEY_ID

Access a CMEK-protected database

All the read, write, and query operations sent to a CMEK-protected database should function the same as with a Google default encrypted database. You don't, for example, need to provide a key for each request.

View the key in use

gcloud

You can use the databases describe gcloud CLI command to confirm database CMEK configuration:

gcloud firestore databases describe --database=DATABASE_ID --project=FIRESTORE_PROJECT

You should see CMEK information in the cmekConfig field in the response similar to the following:

cmekConfig:
    activeKeyVersion:
    - projects/PROJECT_ID/locations/us/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME/cryptoKeyVersions/1
    kmsKeyName: projects/PROJECT_ID/locations/us/keyRings/KEYRING_NAME/cryptoKeys/KEY_NAME
  locationId: nam5
  name: projects/PROJECT_ID/databases/DATABASE_ID

The response includes the following information:

• kmsKeyName: the full key resource name of the key that's used to encrypt your CMEK-protected database.
• activeKeyVersion: a list of all key versions currently in use by the CMEK-protected database. During key rotation, there can be multiple active key versions.

REST API

HTTP request:

GET https://firestore.googleapis.com/v1/{name=projects/FIRESTORE_PROJECT/databases/DATABASE_ID}

In the request body configure CMEK in the cmek_config.kms_key_name field. Set to the full resource ID of a Cloud KMS key. Only a key in the same location as this database is allowed.

This value should be the Cloud KMS key resource ID in the format of projects/{KMS_PROJECT}/locations/{KMS_LOCATION}/keyRings/{KMS_KEYRING_ID}/cryptoKeys/{KMS_KEY_ID}

For more details for other fields, see the database create page.

Example request and response:

curl 'https://firestore.googleapis.com/v1/projects/FIRESTORE_PROJECT/databases/{DATABASE_ID}' \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-type: application/json"

—----------------------------------------- Response —--------------------------------------------
{
  "name": "projects/FIRESTORE_PROJECT/databases/{DATABASE_ID}",
  "locationId": "{FIRESTORE_DATABASE_LOCATION}",
  "type": "FIRESTORE_NATIVE",
  "cmekConfig": {
    "kmsKeyName": "projects/{KMS_PROJECT}/locations/{KMS_LOCATION}/keyRings/{KMS_KEYRING_ID}/cryptoKeys/{KMS_KEY_ID}",
    "activeKeyVersion": [
      "projects/{KMS_PROJECT}/locations/{KMS_LOCATION}/keyRings/{KMS_KEYRING_ID}/cryptoKeys/{KMS_KEY_ID}/cryptoKeyVersions/1"
    ]
  },
  ……
}

Disable a key

To disable a key associated with a database, complete the following:

1. View the key versions in use for a database
2. Disable those key versions
3. Wait for the change to take effect and check if the data is no longer accessible. Changes typically take effect within minutes, but can take up to 3 hours.

When a key used by a database is disabled, expect to receive a FAILED_PRECONDITION exception with additional details in the error message, for example:

{
  "error": {
    "code": 400,
    "message": "The customer-managed encryption key required by the requested resource is not accessible. Error reason:  generic::permission_denied: Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/FIRESTORE_PROJECT/locations/{KMS_LOCATION}/keyRings/{KMS_KEYRING_ID}/cryptoKeys/{KMS_KEY_ID}' (or it may not exist).",
    "status": "FAILED_PRECONDITION",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.DebugInfo",
        "detail": "The customer-managed encryption key required by the requested resource is not accessible. Error reason:  generic::permission_denied: Permission 'cloudkms.cryptoKeyVersions.useToEncrypt' denied on resource 'projects/FIRESTORE_PROJECT/locations/{KMS_LOCATION}/keyRings/{KMS_KEYRING_ID}/cryptoKeys/{KMS_KEY_ID}' (or it may not exist)"
      }
    ]
  }
}

Enable a key

To re-enable a key associated with a database, complete the following:

1. View the key versions in use for a database
2. Enable those key versions
3. Wait for the change to take effect and check if the data is no longer accessible. Changes typically take effect within minutes, but can take up to 3 hours.

View audit logs for a Cloud KMS key

Before you enable Cloud KMS Data Access audit logs, you should be familiar with Cloud Audit Logs.

Cloud KMS Data Access audit logs show you when Firestore or any other products that are configured to use your CMEK key make encrypt/decrypt calls to Cloud KMS. Firestore does not issue an encrypt/decrypt call on every data request, but instead maintains a poller that checks the key periodically. The polling results appear in the audit logs.

You can set up and interact with the audit logs in the Google Cloud console:

1. Make sure that logging is enabled for the Cloud KMS API in your project.

2. Go to Cloud Logging in the Google Cloud console.

   Go to Cloud Logging

3. Limit the log entries to your Cloud KMS key by adding the following lines to the Query builder:

   resource.type="cloudkms_cryptokey"
   resource.labels.key_ring_id = KMS_KEYRING
   resource.labels.crypto_key_id = KMS_KEY
   resource.labels.location=KMS_LOCATION
   

   Provide the following:

   • KMS_KEY: the name of the CMEK key
   • KMS_KEYRING: the KMS key ring that contains the key
   • KMS_LOCATION: the location of the key and key ring

   The log shows a couple log entries about every five minutes per database. The log entries look similar to these examples:

   Info 2021-03-20 08:02:24.869 EDT Cloudkms.googleapis.com Decrypt projects/cloud-kms-project/locations/us-central1/keyRings/firestore-keys/cryptoKeys/my-cmek-key [e-mail]
   audit_log, method: "Decrypt", principal_email: "[e-mail]"
   
   Info 2021-03-20 08:02:24.913 EDT Cloudkms.googleapis.com Encrypt projects/cloud-kms-project/locations/us-central1/keyRings/firestore-keys/cryptoKeys/my-cmek-key [e-mail]
   audit_log, method: "Encrypt", principal_email: "[e-mail]"
   

See Understanding audit logs for details about interpreting audit logs.

Configure a CMEK organization policy

To specify encryption compliance requirements for Firestore databases in your organization, use a CMEK organization policy constraint.

Require CMEK protection

Configure constraints/gcp.restrictNonCmekServices to require CMEK for Firestore database creation. Set the constraint to deny and add firestore.googleapis.com to the deny list, for example:

 gcloud resource-manager org-policies deny gcp.restrictNonCmekServices  is:firestore.googleapis.com --project=FIRESTORE_PROJECT

Replace FIRESTORE_PROJECT with the project to restrict.

To learn more about configuring organization policies, see Creating and editing policies.

After the policy takes effect, you receive a FAILED_PRECONDITION exception and error message if you try to create a non-CMEK database under the affected project. For example, an exception looks like:

{
  "error": {
    "code": 400,
    "message": "Constraint 'constraints/gcp.restrictNonCmekServices' violated for 'projects/FIRESTORE_PROJECT' attempting to perform the operation 'google.firestore.admin.v1.FirestoreAdmin.CreateDatabase' with violated value 'firestore.googleapis.com'. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.",
    "status": "FAILED_PRECONDITION",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
        "violations": [
          {
            "type": "constraints/gcp.restrictNonCmekServices",
            "subject": "orgpolicy:projects/FIRESTORE_PROJECT",
            "description": "Constraint 'constraints/gcp.restrictNonCmekServices' violated for 'projects/FIRESTORE_PROJECT' attempting to perform the operation 'google.firestore.admin.v1.FirestoreAdmin.CreateDatabase' with violated value 'firestore.googleapis.com'. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information."
          }
        ]

Limit the use of keys for CMEK

To limit which Cloud KMS keys are used for CMEK protection, configure the constraints/gcp.restrictCmekCryptoKeyProjects constraint.

As a list constraint, the accepted values are resource hierarchy indicators (for example, projects/PROJECT_ID, under:folders/FOLDER_ID, and under:organizations/ORGANIZATION_ID). Use this constraint by configuring a list of resource hierarchy indicators and setting the constraint to Allow. This configuration restricts supported services so that CMEK keys can be chosen only from the listed projects, folders, and organizations. Requests to create CMEK-protected resources in configured services don't succeed without a Firestore key from one of the allowed resources.

The following example allows only keys from the ALLOWED_KEY_PROJECT_ID for CMEK-protected databases in the specified project:

gcloud resource-manager org-policies allow gcp.restrictCmekCryptoKeyProjects \
under:projects/ALLOWED_KEY_PROJECT_ID \
--project=FIRESTORE_PROJECT

After the policy takes effect, you receive a FAILED_PRECONDITION exception and error message if you violate the constraint. An exception looks like the following:

{
  "error": {
    "code": 400,
    "message": "Constraint 'constraints/gcp.restrictCmekCryptoKeyProjects' violated for 'projects/FIRESTORE_PROJECT' attempting to perform the operation 'google.firestore.admin.v1.FirestoreAdmin.CreateDatabase' with violated value 'projects/{NOT_ALLOWED_KEY_PROJECT}'. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information.",
    "status": "FAILED_PRECONDITION",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.PreconditionFailure",
        "violations": [
          {
            "type": "constraints/gcp.restrictCmekCryptoKeyProjects",
            "subject": "orgpolicy:projects/FIRESTORE_PROJECT",
            "description": "Constraint 'constraints/gcp.restrictCmekCryptoKeyProjects' violated for 'projects/FIRESTORE_PROJECT' attempting to perform the operation 'google.firestore.admin.v1.FirestoreAdmin.CreateDatabase' with violated value 'projects/{NOT_ALLOWED_KEY_PROJECT}'. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information."
          }
        ]
      }
    ]
  }
}

What's next

• Read the overview of Firestore and CMEK