Encrypt data using customer-managed encryption keys (CMEK)

This page describes how to encrypt data stored in AML AI instances with customer-managed encryption keys (CMEK).

Overview

All Customer Data in an AML AI Instance is encrypted using a CMEK key when stored at rest. You manage the key within Cloud Key Management Service, and you control access to the key using Identity and Access Management. If you temporarily disable or permanently destroy the CMEK key, data encrypted with that key cannot be accessed.

CMEK gives you control over more aspects of the lifecycle and management of your keys, but also incurs additional costs for the Cloud KMS service.

Cloud KMS can run in the same Google Cloud project as AML AI or in a separate project where you centrally manage keys for multiple projects.

You assign a CMEK key when you create an instance. All instances must be configured using CMEK and can't be changed to use Google default encryption.

For more information about CMEK in general, including when and why to enable it, see the Cloud KMS documentation.

Customer Data

All Customer Data handled by AML AI is encrypted at rest using the CMEK key specified in the corresponding parent Instance resource. This includes all Customer Data associated with AML AI resources, such as datasets, engine configs, models, and more. All temporary and persistent storage of Customer Data, including copies of inputs and outputs, generated ML features, logs, model hyper parameters, model weights, and prediction results, are encrypted using the CMEK key of the corresponding Instance.

AML AI doesn't manage the encryption of input or output data in your Google Cloud project. If you want this data to be encrypted, then you must set up a separate CMEK key in your Google Cloud project. The specified KMS key in an instance is used for AML AI resources and their data.

Creating a key and granting permissions

The following instructions explain how to create a key for an instance and grant permissions to encrypt and decrypt instance data with the key. You can use a key created directly in Cloud KMS or an externally-managed key that you make available with Cloud External Key Manager.

  1. In the Google Cloud project where you want to manage your keys:

    1. Enable the Cloud KMS API.

    2. Create a key ring using the projects.locations.keyRings.create method. The Cloud KMS key ring location must match the location of the instance that you encrypt.

      REST

      Before using any of the request data, make the following replacements:

      • KMS_PROJECT_ID: the Google Cloud project ID for the project containing the key ring
      • LOCATION: the location of the key ring; use one of the supported regions:
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: a user-defined identifier for the key ring

      To send your request, choose one of these options:

      curl

      Execute the following command: 

      curl -X POST \
           -H "Authorization: Bearer $(gcloud auth print-access-token)" \
           -H "Content-Type: application/json; charset=utf-8" \
           -d "" \
           "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings?key_ring_id=KEY_RING_ID"

      PowerShell

      Execute the following command: 

      $cred = gcloud auth print-access-token
      $headers = @{ "Authorization" = "Bearer $cred" }

      Invoke-WebRequest `
          -Method POST `
          -Headers $headers `
          -Uri "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings?key_ring_id=KEY_RING_ID" | Select-Object -Expand Content

      You should receive a JSON response similar to the following:

      {
  "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID",
  "createTime": CREATE_TIME
}

      gcloud

      Before using any of the command data below, make the following replacements:

      • LOCATION: the location of the key ring; use one of the supported regions:
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: a user-defined identifier for the key ring

      Execute the following command:

      Linux, macOS, or Cloud Shell

      gcloud kms keyrings create KEY_RING_ID \
  --location LOCATION

      Windows (PowerShell)

      gcloud kms keyrings create KEY_RING_ID `
  --location LOCATION

      Windows (cmd.exe)

      gcloud kms keyrings create KEY_RING_ID ^
  --location LOCATION
      You should receive an empty response: 
      $

    3. Create a key using the projects.locations.keyRings.cryptoKeys method.

      REST

      Before using any of the request data, make the following replacements:

      • KMS_PROJECT_ID: the Google Cloud project ID for the project containing the key ring
      • LOCATION: the location of the key ring; use one of the supported regions:
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: the user-defined identifier for the key ring
      • KEY_ID: a user-defined identifier for the key

      Request JSON body: 

      {
  "purpose": "ENCRYPT_DECRYPT"
}

      To send your request, choose one of these options:

      curl

      Save the request body in a file named request.json. Run the following command in the terminal to create or overwrite this file in the current directory: 

      cat > request.json << 'EOF'
{
  "purpose": "ENCRYPT_DECRYPT"
}
EOF

      Then execute the following command to send your REST request: 

      curl -X POST \
           -H "Authorization: Bearer $(gcloud auth print-access-token)" \
           -H "Content-Type: application/json; charset=utf-8" \
           -d @request.json \
           "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys?crypto_key_id=KEY_ID"

      PowerShell

      Save the request body in a file named request.json. Run the following command in the terminal to create or overwrite this file in the current directory: 

      @'
{
  "purpose": "ENCRYPT_DECRYPT"
}
'@  | Out-File -FilePath request.json -Encoding utf8

      Then execute the following command to send your REST request: 

      $cred = gcloud auth print-access-token
      $headers = @{ "Authorization" = "Bearer $cred" }

      Invoke-WebRequest `
          -Method POST `
          -Headers $headers `
          -ContentType: "application/json; charset=utf-8" `
          -InFile request.json `
          -Uri "https://cloudkms.googleapis.com/v1/projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys?crypto_key_id=KEY_ID" | Select-Object -Expand Content

      You should receive a JSON response similar to the following:

      {
  "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID",
  "primary": {
    "name": "projects/KMS_PROJECT_ID/locations/LOCATION/keyRings/KEY_RING_ID/cryptoKeys/KEY_ID/cryptoKeyVersions/1",
    "state": "ENABLED",
    "createTime": CREATE_TIME,
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION",
    "generateTime": GENERATE_TIME
  },
  "purpose": "ENCRYPT_DECRYPT",
  "createTime": CREATE_TIME,
  "versionTemplate": {
    "protectionLevel": "SOFTWARE",
    "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION"
  },
  "destroyScheduledDuration": "86400s"
}

      gcloud

      Before using any of the command data below, make the following replacements:

      • LOCATION: the location of the key ring; use one of the supported regions:
        • us-central1
        • us-east1
        • asia-south1
        • europe-west1
        • europe-west2
        • europe-west4
        • northamerica-northeast1
        • southamerica-east1
      • KEY_RING_ID: the user-defined identifier for the key ring
      • KEY_ID: a user-defined identifier for the key

      Execute the following command:

      Linux, macOS, or Cloud Shell

      gcloud kms keys create KEY_ID \
  --keyring KEY_RING_ID \
  --location LOCATION \
  --purpose "encryption"

      Windows (PowerShell)

      gcloud kms keys create KEY_ID `
  --keyring KEY_RING_ID `
  --location LOCATION `
  --purpose "encryption"

      Windows (cmd.exe)

      gcloud kms keys create KEY_ID ^
  --keyring KEY_RING_ID ^
  --location LOCATION ^
  --purpose "encryption"
      You should receive an empty response: 
      $

  2. If you have not created an AML AI instance in the AML AI project, then the AML AI service account doesn't exist yet. Create the service account.

    Before using any of the command data below, make the following replacements:

    • PROJECT_ID: the Google Cloud project ID for the project where AML AI is running

    Execute the following command:

    Linux, macOS, or Cloud Shell

    gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID

    Windows (PowerShell)

    gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID

    Windows (cmd.exe)

    gcloud beta services identity create --service=financialservices.googleapis.com --project=PROJECT_ID

    You should receive a response similar to the following:

    Service identity created: service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com

  3. Grant the CryptoKey Encrypter/Decrypter IAM role (roles/cloudkms.cryptoKeyEncrypterDecrypter) to the AML AI service account. Grant this permission on the key you created.

    Before using any of the command data below, make the following replacements:

    • PROJECT_ID: the Google Cloud project ID for the project where AML AI is running
    • KEY_ID: the user-defined identifier for the key
    • LOCATION: the location of the key ring; use one of the supported regions:
      • us-central1
      • us-east1
      • asia-south1
      • europe-west1
      • europe-west2
      • europe-west4
      • northamerica-northeast1
      • southamerica-east1
    • KEY_RING_ID: a user-defined identifier for the key ring
    • PROJECT_NUMBER: the Google Cloud project number for the project where AML AI is running

    Execute the following command:

    Linux, macOS, or Cloud Shell

    gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID \
  --location LOCATION --keyring=KEY_RING_ID \
  --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com \
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

    Windows (PowerShell)

    gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID `
  --location LOCATION --keyring=KEY_RING_ID `
  --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com `
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

    Windows (cmd.exe)

    gcloud kms keys add-iam-policy-binding KEY_ID --project=PROJECT_ID ^
  --location LOCATION --keyring=KEY_RING_ID ^
  --member serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com ^
  --role roles/cloudkms.cryptoKeyEncrypterDecrypter

    You should receive a response similar to the following:

    Updated IAM policy for key KEY_ID.
bindings:
- members:
  - serviceAccount:service-PROJECT_NUMBER@gcp-sa-financialservices.iam.gserviceaccount.com
  role: roles/cloudkms.cryptoKeyEncrypterDecrypter
etag: BwYCq0Sq4Ho=
version: 1

    For more information about this command see the gcloud kms keys add-iam-policy-binding documentation.

You can now create an instance and specify the key to use for encryption.

Removing access

There are several ways to remove access to a CMEK-encrypted instance:

We recommend that you revoke the permissions from the AML AI service account before disabling or destroying a key. Changes to permissions are propagated within seconds, so you can observe the impacts of disabling or destroying a key.

When you disable or destroy the encryption key for an instance, you lose the ability to view or retrieve instance data. All data stored in the instance becomes inaccessible, including API resources such as datasets, models, engine configs, backtest results, and prediction results. Users with the AML AI viewer role can still view instance metadata such as the instance name.

Users with the AML AI Administrator role or Owner role can delete the instance.

CMEK organization policies

AML AI does not support CMEK organization policies. However, AML AI always requires the use of CMEK, regardless of the constraints/gcp.restrictNonCmekServices organization policy.

What's next?