Vertex AI Search and Conversation has been renamed to Vertex AI Agent Builder. We are in the process of updating content to reflect the new branding.

Customer-managed encryption keys

This page describes how to use your own encryption key to protect your data stores in the US and EU multi-regions.

By default, Vertex AI Agent Builder encrypts your content stored at rest. Vertex AI Agent Builder handles and manages this default encryption for you without any additional actions on your part.

However, if you have specific compliance or regulatory requirements related to the keys that protect your data, you can use customer-managed encryption keys (CMEK) to protect your resources. In this case, you'll use Cloud KMS keys and follow the procedure on this page. The key is associated with a specific location: the US multi-region or the EU multi-region.

Cloud KMS keys are used to encrypt and decrypt the data in your data stores and apps. For general information about Cloud KMS, see the Cloud Key Management Service documentation.

Limitations of Cloud KMS in Vertex AI Agent Builder

The following limitations apply to CMEK (Cloud KMS) keys in Vertex AI Agent Builder:

Keys cannot be changed (or rotated).
After a key has been registered, it cannot be deregistered or removed from a data store.
You must use US or EU multi-region data stores and apps (not global ones). For more information about multi-regions and data residency, including limits associated with using non-global locations, see Vertex AI Search locations or Vertex AI Agents locations.
If you need to register more than one key for a project, contact your Google account team to request a quota increase for CMEK configurations, providing a justification for why you need more than one key.
Data stores created before a key is registered to the project cannot be protected by the key.
For Vertex AI Search, Enterprise edition is required. For information about Enterprise edition, see About advanced features.

Before you begin

Make sure you satisfy the following prerequisites:

Contact your Google account representative and ask to be added to the allowlist for customer-managed encryption keys in Vertex AI Agent Builder.
A symmetric Cloud KMS key with the rotation period set to Never (Manual rotation). See Create a key ring and Create a key in the Cloud KMS documentation.
The CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) on the key has been granted to the Discovery Engine service agent. For general instructions on how to add a role to a service agent, see Grant or revoke a single role.
The CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) on the key has been granted to the Cloud Storage service agent. If this role is not granted, data import for CMEK-protected data stores will fail because Discovery Engine is not able to make the CMEK-protected, temporary bucket and directory that is required for importing.
Do not create any data stores or apps that you want managed by your key until after you have completed the key registration instructions on this page.
Enterprise edition features are turned on for the app. See Turn Enterprise edition on or off.

Register your Cloud KMS key

To register your own managed key for Vertex AI Agent Builder, follow these steps:

Call the UpdateCmekConfig method with the Cloud KMS key that you want to register.

curl -X PATCH \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-d '{"kms_key":"projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY_NAME"}' \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/cmekConfigs/CMEK_CONFIG_ID?set_default=SET_DEFAULT"

KMS_PROJECT_ID: The ID of your project that contains the key. The project number won't work.
LOCATION: The multi-region of your data store: us or eu.
KEY_RING: The name of the key ring that holds the key.
KEY_NAME: The name of the key.
PROJECT_ID: The ID of your project that contains the data store.
CMEK_CONFIG_ID: The ID of the CmekConfig resource.
SET_DEFAULT: Set to true to use the key as the default key for subsequent data stores created in the multi-region.

An example curl call and response looks like this:

$ curl -X PATCH
-H "Authorization: Bearer $(gcloud auth print-access-token)"
-H "Content-Type: application/json
-d '{"kms_key":"projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"}'
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1?set_default=true"
 
{
 "name": "projects/my-ai-app-project-123/locations/us/operations/update-cmek-config-56789",
 "metadata": {
  "@type": "type.googleapis.com/google.cloud.discoveryengine.v1alpha.UpdateCmekConfigMetadata"
 }
}

Optional: Record the name value returned by the method and follow the instructions in Get details about a long-running operation to see when the operation is complete.

It typically takes a few minutes to register a key.

After the operation completes, new data stores in that multi-region are protected by the key. For general information about creating data stores, see Create a search data store.

View Cloud KMS keys

To view a registered key for Vertex AI Agent Builder, do one of the following:

If you have the CmekConfig resource name, call the GetCmekConfig method:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/cmekConfigs/CMEK_CONFIG_ID"

LOCATION: The multi-region of your data store: us or eu.
PROJECT_ID: The ID of your project that contains the data
CMEK_CONFIG_ID: The ID of the CmekConfig resource.

An example curl call and response looks like this:

$ curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1"
 
{
  "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1",
  "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
  "state": "ACTIVE"
  "is_default": true
}

If you don't have the CmekConfig resource name, call the ListCmekConfigs method:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/cmekConfigs"

LOCATION: The multi-region of your data store: us or eu.
PROJECT_ID: The ID of your project that contains the data

An example curl call and response looks like this:

$ curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/cmekConfigs"
 
{
  "cmek_configs": [
    {
      "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-1",
      "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
      "state": "ACTIVE"
      "is_default": true
    }
    {
      "name": "projects/my-ai-app-project-123/locations/us/cmekConfigs/cmek-config-2",
      "kms_key": "projects/key-project-456/locations/us/keyRings/my-key-ring/cryptoKeys/my-key-2"
      "state": "ACTIVE"
    }
  ]
}

Optional: Verify that a data store is protected by a key

Data stores that were created before your key was registered are not protected by the key. If you want to confirm that a particular data store is protected by your key, follow these steps:

Run the following curl command on the data store:

curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-H "x-goog-user-project: PROJECT_ID" \
"https://LOCATION-discoveryengine.googleapis.com/v1alpha/projects/PROJECT_ID/locations/LOCATION/collections/default_collection/dataStores/DATA_STORE_ID"

LOCATION: The multi-region of your data store: us or eu.
PROJECT_ID: The ID of your project that contains the data store.
DATA_STORE_ID: The ID of the data store.

An example curl call looks like this:

curl -X GET
-H "Authorization: Bearer $(gcloud auth print-access-token)"
-H "Content-Type: application/json"
-H "x-goog-user-project: my-ai-app-project-123"
"https://us-discoveryengine.googleapis.com/v1alpha/projects/my-ai-app-project-123/locations/us/collections/default_collection/dataStores/my-data-store-1"

Review the output from the command: If the cmekConfig field is in the output and the kmsKey field shows the key that you registered, then the data store is protected by the key.

An example response looks like this:

{
 "name": "projects/969795412903/locations/us/collections/default_collection/dataStores/my-data-store-1",
 "displayName": "my-data-store-1",
 "industryVertical": "GENERIC",
 "createTime": "2023-09-05T21:20:21.520552Z",
 "solutionTypes": [
   "SOLUTION_TYPE_SEARCH"
 ],
 "defaultSchemaId": "default_schema",
 "cmekConfig": {
   "name": "projects/969795412903/locations/us/collections/default_collection/dataStores/my-data-store-1/cmekConfigs/cmek-config-1",
   "kmsKey": "projects/my-ai-app-project-123/locations/us/keyRings/my-key-ring/cryptoKeys/my-key"
 }
}

If a key is disabled or revoked

If a key is disabled or permissions for the key are revoked, the data store stops ingesting data and stops serving data within 15 minutes. However, re-enabling a key or restoring permissions takes a long time. It can take up to 24 hours before the data store can resume serving data.

Therefore, do not disable a key unless necessary. Disabling and enabling a key on a data store is a time-consuming operation. For example, repeatedly switching a key between disabled and enabled means it will take a long time for the data store to reach a protected state. Disabling a key and re-enabling it immediately afterward could result in days of downtime because the key is first disabled from the data store and subsequently re-enabled.