Encrypt data using customer-managed encryption keys (CMEK)

This page describes how to encrypt data stored in AML AI instances with customer-managed encryption keys (CMEK).

Overview

All Customer Data in an AML AI instance is encrypted at rest using a CMEK key. You manage the key within Cloud Key Management Service, and you control access to the key using Identity and Access Management. If you temporarily disable or permanently destroy the CMEK key, data encrypted with that key cannot be accessed.

AML AI supports CMEK only by using Cloud KMS. It doesn't support Google Default Encryption.

CMEK gives you control over more aspects of the lifecycle and management of your keys, but also incurs additional costs for the Cloud KMS service.

Cloud KMS can run in the same Google Cloud project as AML AI or in a separate project where you centrally manage keys for multiple projects.

Encryption configuration is set up when you create an instance. Once an instance has been created, you cannot assign a different Cloud KMS key. You can still rotate the key.

For more information about CMEK in general, see the Cloud KMS documentation.

Protection levels

Cloud KMS lets you to choose from a variety of different protection levels, including:

  • Software keys
  • Hardware Security Modules (HSMs) using Cloud HSM

Read how to configure CMEK in AML AI. Not all protection levels are available in all regions. Note that AML AI does not support Customer Supplied Encryption Keys (CSEK) or Cloud External Key Manager.

Customer Data

All Customer Data handled by AML AI is encrypted at rest using the CMEK key specified in the corresponding parent Instance resource. This includes all Customer Data associated with AML AI resources, such as datasets, engine configs, models, and more. All temporary and persistent storage of Customer Data, including copies of inputs and outputs, generated ML features, model hyper parameters, model weights, and prediction results, are encrypted using the CMEK key of the corresponding instance.

See the service specific terms for the definition of customer data, which may not include resource identifiers, attributes, or other data labels.

Encrypting input and output data

The AML AI encryption configuration in an instance is used only for AML AI resources and their data. AML AI doesn't manage the encryption of input or output data in your Google Cloud project. If you want this data to be encrypted using CMEK, then you must set up a Cloud KMS key to match your chosen key protection level configured on the BigQuery dataset. You can also reuse the same key used by AML AI.

Read more about encryption in BigQuery.

Key rotation

Periodically and automatically rotating keys is a recommended security practice. With CMEK, key rotation is controlled by you. When you rotate a key, data encrypted with previous key versions is not automatically re-encrypted with the new key version.

A single AML AI resource may internally be stored as multiple units. If, during the lifetime of an AML AI resource, the key version is rotated, not all units may be encrypted with the same key version.

If you rotate a key, there is no way in AML AI to force a re-encryption or to determine if older key versions are safe to delete.

Read more about key rotation with cloud KMS.

Creating a key and granting permissions

The following instructions explain how to create a key for an instance and grant permissions to encrypt and decrypt instance data with the key. You can use a key created directly in Cloud KMS or an externally-managed key that you make available with Cloud External Key Manager.

  1. In the Google Cloud project where you want to manage your keys:

    1. Enable the Cloud KMS API.

    2. Create a key ring using the projects.locations.keyRings.create method. The Cloud KMS key ring location must match the location of the instance that you encrypt.

      REST

      Before using any of the request data, make the following replacements:

      • KMS_PROJECT_ID: the Google Cloud project ID for the project containing the key ring
      • LOCATION: the location of the key ring; use one of the supported regions
        Show locations
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: a user-defined identifier for the key ring

      To send your request, choose one of these options:

      curl

      Execute the following command: 

      curl -X POST \
           -H "Authorization: Bearer $(gcloud auth print-access-token)" \
           -H "Content-Type: application/json; charset=utf-8" \
           -d "" \
           "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings?key_ring_id=KEY_RING_ID"

      PowerShell

      Execute the following command: 

      $cred = gcloud auth print-access-token
      $headers = @{ "Authorization" = "Bearer $cred" }

      Invoke-WebRequest `
          -Method POST `
          -Headers $headers `
          -Uri "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings?key_ring_id=KEY_RING_ID" | Select-Object -Expand Content

      You should receive a JSON response similar to the following:

      {
  "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID",
  "createTime": "2023-03-14T15:52:55.358979323Z"
}

      gcloud

      Before using any of the command data below, make the following replacements:

      • LOCATION: the location of the key ring; use one of the supported regions
        Show locations
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: a user-defined identifier for the key ring

      Execute the following command:

      Linux, macOS, or Cloud Shell

      gcloud kms keyrings create KEY_RING_ID \
  --location LOCATION

      Windows (PowerShell)

      gcloud kms keyrings create KEY_RING_ID `
  --location LOCATION

      Windows (cmd.exe)

      gcloud kms keyrings create KEY_RING_ID ^
  --location LOCATION
      You should receive an empty response: 
      $

    3. Create a key using the projects.locations.keyRings.cryptoKeys method.

      REST

      Before using any of the request data, make the following replacements:

      • KMS_PROJECT_ID: the Google Cloud project ID for the project containing the key ring
      • LOCATION: the location of the key ring; use one of the supported regions
        Show locations
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: the user-defined identifier for the key ring
      • KEY_ID: a user-defined identifier for the key

      Request JSON body: 

      {
  "purpose": "ENCRYPT_DECRYPT"
}

      To send your request, choose one of these options:

      curl

      Save the request body in a file named request.json. Run the following command in the terminal to create or overwrite this file in the current directory: 

      cat > request.json << 'EOF'
{
  "purpose": "ENCRYPT_DECRYPT"
}
EOF

      Then execute the following command to send your REST request: 

      curl -X POST \
           -H "Authorization: Bearer $(gcloud auth print-access-token)" \
           -H "Content-Type: application/json; charset=utf-8" \
           -d @request.json \
           "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys?crypto_key_id=KEY_ID"

      PowerShell

      Save the request body in a file named request.json. Run the following command in the terminal to create or overwrite this file in the current directory: 

      @'
{
  "purpose": "ENCRYPT_DECRYPT"
}
'@  | Out-File -FilePath request.json -Encoding utf8

      Then execute the following command to send your REST request: 

      $cred = gcloud auth print-access-token
      $headers = @{ "Authorization" = "Bearer $cred" }

      Invoke-WebRequest `
          -Method POST `
          -Headers $headers `
          -ContentType: "application/json; charset=utf-8" `
          -InFile request.json `
          -Uri "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys?crypto_key_id=KEY_ID" | Select-Object -Expand Content

      You should receive a JSON response similar to the following:

      {
  "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID",
  "primary": {
    "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID/cryptoKeyVersions/1",
    "state": "ENABLED",
    "createTime": CREATE_TIME,
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION",
    "generateTime": GENERATE_TIME
  },
  "purpose": "ENCRYPT_DECRYPT",
  "createTime": CREATE_TIME,
  "versionTemplate": {
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION"
  },
  "destroyScheduledDuration": "86400s"
}

      gcloud

      Before using any of the command data below, make the following replacements:

      • LOCATION: the location of the key ring; use one of the supported regions
        Show locations
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: the user-defined identifier for the key ring
      • KEY_ID: a user-defined identifier for the key

      Execute the following command:

      Linux, macOS, or Cloud Shell

      gcloud kms keys create KEY_ID \
  --keyring KEY_RING_ID \
  --location LOCATION \
  --purpose "encryption"

      Windows (PowerShell)

      gcloud kms keys create KEY_ID `
  --keyring KEY_RING_ID `
  --location LOCATION `
  --purpose "encryption"

      Windows (cmd.exe)

      gcloud kms keys create KEY_ID ^
  --keyring KEY_RING_ID ^
  --location LOCATION ^
  --purpose "encryption"
      You should receive an empty response: 
      $

  2. If you have not created an AML AI instance in the AML AI project, then the AML AI service account doesn't exist yet. Create the service account.

    Before using any of the command data below, make the following replacements:

    • PROJECT_ID: the Google Cloud project ID for the project where AML AI is running

    Execute the following command:

    Linux, macOS, or Cloud Shell

    gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID

    Windows (PowerShell)

    gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID

    Windows (cmd.exe)

    gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID

    You should receive a response similar to the following:

    Service identity created: service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com

  3. Grant the CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) to the AML AI service account. Grant this permission on the key you created.

    Before using any of the command data below, make the following replacements:

    • PROJECT_ID: the Google Cloud project ID for the project where AML AI is running
    • KEY_ID: the user-defined identifier for the key
    • LOCATION: the location of the key ring; use one of the supported regions
      Show locations
      • us-central1
      • us-east1
      • asia-south1
      • europe-west1
      • europe-west2
      • europe-west4
      • northamerica-northeast1
      • southamerica-east1
    • KEY_RING_ID: a user-defined identifier for the key ring
    • PROJECT_NUMBER: the Google Cloud project number for the project where AML AI is running

    Execute the following command:

    Linux, macOS, or Cloud Shell

    gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID \
  --location LOCATION --keyring=KEY_RING_ID \
  --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

    Windows (PowerShell)

    gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID `
  --location LOCATION --keyring=KEY_RING_ID `
  --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com `
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

    Windows (cmd.exe)

    gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID ^
  --location LOCATION --keyring=KEY_RING_ID ^
  --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com ^
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

    You should receive a response similar to the following:

    Updated IAM policy for key KEY_ID.
bindings:
- members:
  - serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
etag: BwYCq0Sq4Ho=
version: 1

    For more information about this command see the gcloud kms keys add-iam-policy-binding documentation.

You can now create an instance and specify the key to use for encryption.

Removing access

There are several ways to remove access to the key from the CMEK-encrypted instance:

We recommend that you revoke the permissions from the AML AI service account before disabling or destroying a key. Changes to permissions are propagated within seconds, so you can observe the impacts of disabling or destroying a key.

When you disable or destroy the encryption key for an instance, you lose the ability to use or retrieve customer data associated with the instance. All customer data stored in the instance becomes inaccessible, including models, engine configs, backtest results, and prediction results. Users with any AML AI viewer role can still view fields such as the instance name or the other resource fields returned by retrieving AML AI resources.

Any operations which use or export customer data, for example exporting backtestResults metadata, will fail.

Users with the AML AI Administrator role or Owner role can delete the instance.

CMEK organization policies

AML AI does not support CMEK organization policies. However, AML AI always requires the use of CMEK, regardless of the constraints/gcp.restrictNonCmekServices organization policy.

Interaction with VPC-SC

If you have configured AML AI inside a VPC-SC perimeter, the CMEK key must still be accessible to the service account. If the key is not inside the same VPC-SC perimeter, there are multiple ways to achieve this, including:

  • Use an egress rule to allowlist the resource
  • Use VPC perimeter peering

What's next?