This page describes how to encrypt data stored in AML AI instances with customer-managed encryption keys (CMEK).
Overview
All Customer Data in an AML AI instance is encrypted at rest using a CMEK key. You manage the key within Cloud Key Management Service (Cloud KMS), and you control access to the key using Identity and Access Management. If you temporarily disable or permanently destroy the CMEK key, data encrypted with that key cannot be accessed.
AML AI only supports CMEK by using Cloud KMS. It doesn't support Google Default Encryption.
CMEK gives you control over more aspects of the lifecycle and management of your keys, but also incurs additional costs for the Cloud KMS service.
Cloud KMS can run in the same Google Cloud project as AML AI or in a separate project where you centrally manage keys for multiple projects.
Encryption configuration is set up when you create an instance. Once an instance has been created, you cannot assign a different Cloud KMS key. You can still rotate the key.
For more information about CMEK in general, see the Cloud KMS documentation.
Protection levels
Cloud KMS lets you to choose from a variety of different protection levels, including:
- Software keys
- Hardware Security Modules (HSMs) using Cloud HSM
Read how to configure CMEK in AML AI. Not all protection levels are available in all regions. Note that AML AI does not support Customer Supplied Encryption Keys (CSEK) or Cloud External Key Manager.
Customer Data
All Customer Data handled by AML AI is encrypted at rest using the CMEK key specified in the corresponding parent Instance resource. This includes all Customer Data associated with AML AI resources, such as datasets, engine configs, models, and more. All temporary and persistent storage of Customer Data, including copies of inputs and outputs, generated ML features, model hyper parameters, model weights, and prediction results, are encrypted using the CMEK key of the corresponding instance.
See the service specific terms for the definition of customer data, which may not include resource identifiers, attributes, or other data labels.
Encrypting input and output data
The AML AI encryption configuration in an instance is used only for AML AI resources and their data. AML AI doesn't manage the encryption of input or output data in your Google Cloud project. If you want this data to be encrypted using CMEK, then you must set up a Cloud KMS key to match your chosen key protection level configured on the BigQuery dataset. You can also reuse the same key used by AML AI.
Read more about encryption in BigQuery.
Key rotation
Periodically and automatically rotating keys is a recommended security practice. With CMEK, key rotation is controlled by you. When you rotate a key, data encrypted with previous key versions is not automatically re-encrypted with the new key version.
A single AML AI resource may internally be stored as multiple units. If, during the lifetime of an AML AI resource, the key version is rotated, not all units may be encrypted with the same key version.
If you rotate a key, there is no way in AML AI to force a re-encryption or to determine if older key versions are safe to delete.
Read more about key rotation with cloud KMS.
Creating a key and granting permissions
The following instructions explain how to create a key for an instance and grant permissions to encrypt and decrypt instance data with the key. You can use a key created directly in Cloud KMS or an externally-managed key that you make available with Cloud External Key Manager.
In the Google Cloud project where you want to manage your keys:
Create a key ring using the
projects.locations.keyRings.create
method. The Cloud KMS key ring location must match the location of the instance that you encrypt.REST
Before using any of the request data, make the following replacements:
KMS_PROJECT_ID
: the Google Cloud project ID for the project containing the key ringLOCATION
: the location of the key ring; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
KEY_RING_ID
: a user-defined identifier for the key ring
To send your request, choose one of these options:
curl
Execute the following command:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d "" \
"https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings?key_ring_id=KEY_RING_ID"PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-Uri "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings?key_ring_id=KEY_RING_ID" | Select-Object -Expand ContentYou should receive a JSON response similar to the following:
{ "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID", "createTime": "2023-03-14T15:52:55.358979323Z" }
gcloud
Before using any of the command data below, make the following replacements:
KMS_PROJECT_ID
: the Google Cloud project ID for the project containing the key ringLOCATION
: the location of the key ring; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
KEY_RING_ID
: a user-defined identifier for the key ring
Execute the following command:
Linux, macOS, or Cloud Shell
gcloud kms keyrings create KEY_RING_ID \ --project KMS_PROJECT_ID --location LOCATION
Windows (PowerShell)
gcloud kms keyrings create KEY_RING_ID ` --project KMS_PROJECT_ID --location LOCATION
Windows (cmd.exe)
gcloud kms keyrings create KEY_RING_ID ^ --project KMS_PROJECT_ID --location LOCATION
$
Create a key using the
projects.locations.keyRings.cryptoKeys
method.REST
Before using any of the request data, make the following replacements:
KMS_PROJECT_ID
: the Google Cloud project ID for the project containing the key ringLOCATION
: the location of the key ring; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
KEY_RING_ID
: the user-defined identifier for the key ringKEY_ID
: a user-defined identifier for the key
Request JSON body:
{ "purpose": "ENCRYPT_DECRYPT" }
To send your request, choose one of these options:
curl
Save the request body in a file named
request.json
. Run the following command in the terminal to create or overwrite this file in the current directory:cat > request.json << 'EOF' { "purpose": "ENCRYPT_DECRYPT" } EOF
Then execute the following command to send your REST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys?crypto_key_id=KEY_ID"PowerShell
Save the request body in a file named
request.json
. Run the following command in the terminal to create or overwrite this file in the current directory:@' { "purpose": "ENCRYPT_DECRYPT" } '@ | Out-File -FilePath request.json -Encoding utf8
Then execute the following command to send your REST request:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys?crypto_key_id=KEY_ID" | Select-Object -Expand ContentYou should receive a JSON response similar to the following:
{ "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID", "primary": { "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID/cryptoKeyVersions/1", "state": "ENABLED", "createTime": "2023-03-14T15:52:55.358979323Z", "protectionLevel": "SOFTWARE", "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION", "generateTime": "2023-03-14T15:52:55.358979323Z" }, "purpose": "ENCRYPT_DECRYPT", "createTime": "2023-03-14T15:52:55.358979323Z", "versionTemplate": { "protectionLevel": "SOFTWARE", "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION" }, "destroyScheduledDuration": "86400s" }
gcloud
Before using any of the command data below, make the following replacements:
KMS_PROJECT_ID
: the Google Cloud project ID for the project containing the key ringLOCATION
: the location of the key ring; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
KEY_RING_ID
: the user-defined identifier for the key ringKEY_ID
: a user-defined identifier for the key
Execute the following command:
Linux, macOS, or Cloud Shell
gcloud kms keys create KEY_ID \ --keyring KEY_RING_ID \ --project KMS_PROJECT_ID \ --location LOCATION \ --purpose "encryption"
Windows (PowerShell)
gcloud kms keys create KEY_ID ` --keyring KEY_RING_ID ` --project KMS_PROJECT_ID ` --location LOCATION ` --purpose "encryption"
Windows (cmd.exe)
gcloud kms keys create KEY_ID ^ --keyring KEY_RING_ID ^ --project KMS_PROJECT_ID ^ --location LOCATION ^ --purpose "encryption"
$
If you have not created an AML AI instance in the AML AI project, then the AML AI service account doesn't exist yet. Create the service account.
Before using any of the command data below, make the following replacements:
PROJECT_ID
: the Google Cloud project ID for the project where AML AI is running
Execute the following command:
Linux, macOS, or Cloud Shell
gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID
Windows (PowerShell)
gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID
Windows (cmd.exe)
gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID
You should receive a response similar to the following:
Service identity created: service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com
Grant the CryptoKey Encrypter/Decrypter IAM role (
roles/cloudkms.cryptoKeyEncrypterDecrypter
) to the AML AI service account. Grant this permission on the key you created.Before using any of the command data below, make the following replacements:
PROJECT_ID
: the Google Cloud project ID for the project where AML AI is runningKEY_ID
: the user-defined identifier for the keyLOCATION
: the location of the key ring; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
KEY_RING_ID
: a user-defined identifier for the key ringPROJECT_NUMBER
: the Google Cloud project number for the project where AML AI is running
Execute the following command:
Linux, macOS, or Cloud Shell
gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID \ --location LOCATION --keyring=KEY_RING_ID \ --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com \ --role roles/cloudkms.cryptoKeyEncrypterDecrypter
Windows (PowerShell)
gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID ` --location LOCATION --keyring=KEY_RING_ID ` --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com ` --role roles/cloudkms.cryptoKeyEncrypterDecrypter
Windows (cmd.exe)
gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID ^ --location LOCATION --keyring=KEY_RING_ID ^ --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com ^ --role roles/cloudkms.cryptoKeyEncrypterDecrypter
You should receive a response similar to the following:
Updated IAM policy for key KEY_ID. bindings: - members: - serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com role: roles/cloudkms.cryptoKeyEncrypterDecrypter etag: BwYCq0Sq4Ho= version: 1
For more information about this command see the gcloud kms keys add-iam-policy-binding documentation.
You can now create an instance and specify the key to use for encryption.
Removing access
There are several ways to remove access to the key from the CMEK-encrypted instance:
- Revoke the Cloud KMS CryptoKey Encrypter/Decrypter role from the AML AI service account using the Google Cloud console or the gcloud CLI
- Temporarily disable the CMEK key
- Permanently destroy the CMEK key
We recommend that you revoke the permissions from the AML AI service account before disabling or destroying a key. Changes to permissions are propagated within seconds, so you can observe the impacts of disabling or destroying a key.
When you disable or destroy the encryption key for an instance, you lose the ability to use or retrieve customer data associated with the instance. All customer data stored in the instance becomes inaccessible, including models, engine configs, backtest results, and prediction results. Users with any AML AI viewer role can still view fields such as the instance name or the other resource fields returned by retrieving AML AI resources.
Any operations which use or export customer data, for example exporting
backtestResults
metadata, will fail.
Users with the AML AI Administrator role or Owner role can delete the instance.
CMEK organization policies
AML AI does not support
CMEK organization policies. However,
AML AI always requires the use of CMEK, regardless of the
constraints/gcp.restrictNonCmekServices
organization policy.
Interaction with VPC-SC
If you have configured AML AI inside a VPC-SC perimeter, the CMEK key must still be accessible to the service account. If the key is not inside the same VPC-SC perimeter, there are multiple ways to achieve this, including:
- Use an egress rule to allowlist the resource
- Use VPC perimeter peering
What's next?
- Create an instance
- Learn more about CMEK