By default, Google Cloud automatically encrypts data when it is at rest using encryption keys managed by Google. If you have specific compliance or regulatory requirements related to the keys that protect your data, you can use customer-managed encryption keys (CMEK) for your training jobs.
You can read more about the specific benefits of using CMEK with AI Platform Training in the following section of this guide. For more information about CMEK in general, including when and why to enable it, see the Cloud Key Management Service documentation.
This guide describes some benefits of using CMEK for AI Platform Training and walks through how to configure a training job to use CMEK.
Understanding CMEK for AI Platform Training
The following sections describe basic information about CMEK for AI Platform Training that you must understand before configuring CMEK for your training jobs.
Benefits of CMEK
In general, CMEK is most useful if you need full control over the keys used to encrypt your data. With CMEK, you can manage your keys within Cloud KMS. For example, you can rotate or disable a key or you can set up a rotation schedule using the Cloud KMS API. For more information about CMEK in general, including when and why to enable it, see the Cloud KMS documentation.
When you run a training job, your code runs on one or more virtual machine (VM) instances managed by AI Platform Training. When you enable CMEK for AI Platform Training, the key that you designate, rather than a key managed by Google, is used to encrypt data on the boot disks of these VMs. The CMEK key encrypts the following kinds of data:
- The copy of your training code on the VMs.
- Any data that gets loaded by your training code.
- Any temporary data that gets saved to the local disk by your training code.
The CMEK key does not encrypt metadata associated with your training job, like the job's name and region. Metadata associated with training jobs is always encrypted using Google's default encryption mechanism.
Using CMEK with other Google Cloud products
Configuring CMEK for AI Platform Training does not automatically configure CMEK for other Google Cloud products that you use together with AI Platform Training. To use CMEK to encrypt data in other Google Cloud products, additional configuration is required. For example:
Cloud Storage: When you run a non-custom container job, AI Platform Training loads your training application from a Cloud Storage bucket. In addition, most training jobs also export trained model artifacts (for example, a TensorFlow SaveModel directory) to a Cloud Storage bucket as part of their output.
To ensure that your data in Cloud Storage is encrypted with CMEK, read the Cloud Storage guide to using customer-managed encryption keys. You can set your encryption key as the default key for the Cloud Storage bucket(s) that you use with AI Platform Training, or you can use it to encrypt specific objects.
Container Registry: When you run a custom container job, AI Platform Training loads your container image from Container Registry.
To ensure that your container image is encrypted with CMEK, read the Container Registry guide to using customer-managed encryption keys.
Cloud Logging: When you run a training job, AI Platform Training saves logs to Logging. These logs are not encrypted with CMEK. However, if you use the Logs Router, then you can configure CMEK for certain temporary files that the Logs Router creates.
Limitations
You cannot use CMEK with AI Platform Training in the following situations:
- You cannot use CMEK with training jobs that use TPUs.
- If you started using the AI Platform Training and Prediction API in 2017 or earlier and submit a training job with CMEK, you might encounter an error describing an incompatible legacy authentication mechanism. In this case, create a new Google Cloud project or select a different project to run your AI Platform Training job.
Configuring CMEK for your training job
The following sections describe how to create a keyring and key in Cloud Key Management Service, grant AI Platform Training encrypter and decrypter permissions for your key, and create a training job that uses CMEK.
Before you begin
This guide assumes that you use two separate Google Cloud projects to configure CMEK for training:
- A project for managing your encryption key (referred to as the "Cloud KMS project").
- A project for accessing AI Platform Training, managing training data or output in Cloud Storage, and interacting with any other Google Cloud products that you need for your use case (referred to as the "AI Platform project").
This recommended setup supports a separation of duties.
Alternatively, you can use a single Google Cloud project for the whole guide. To do so, use the same project for all of the following tasks that refer to the Cloud KMS project and the tasks that refer to the AI Platform project.
Setting up the Cloud KMS project
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Cloud KMS API.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the Cloud KMS API.
Setting up the AI Platform project
- Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the AI Platform Training & Prediction API.
-
In the Google Cloud console, on the project selector page, select or create a Google Cloud project.
-
Make sure that billing is enabled for your Google Cloud project.
-
Enable the AI Platform Training & Prediction API.
Setting up the Google Cloud CLI
The gcloud CLI is required for some steps in this guide and optional for others.Install the Google Cloud CLI, then initialize it by running the following command:
gcloud init
Creating a key ring and key
Follow the Cloud KMS guide to creating symmetric keys to create a key ring and a key. When you create your key ring, specify a region that supports AI Platform Training as the key ring's location. AI Platform Training only supports CMEK when your training job and key use the same region. You must not specify a dual-regional, multi-regional, or global location for your key ring.
Make sure to create your key ring and key in your Cloud KMS project.
Granting AI Platform Training permissions
To use CMEK for your training job, you must grant AI Platform Training permission to encrypt and decrypt data using your key. AI Platform Training uses a service agent to run your training jobs. This service agent is identified by an email address with the following format:
service-AI_PLATFORM_PROJECT_NUMBER@cloud-ml.google.com.iam.gserviceaccount.com
To find the appropriate service agent for your AI Platform project, go
to the IAM page in the Google Cloud console and find the member that matches
this email address format, with the project
number for
your AI Platform project replacing the
AI_PLATFORM_PROJECT_NUMBER variable. The service agent also has the
name Google Cloud ML Engine Service Agent
.
Make note of the email address for this service agent, and use it in the following steps to grant it permission to encrypt and decrypt data using your key. You can grant permission by using the Google Cloud console or by using the Google Cloud CLI:
Google Cloud console
In the Google Cloud console, go to the Cryptographic Keys page and select your Cloud KMS project.
Click on the name of the key ring that you created in a preceding section of this guide to go to the Key ring details page.
Select the checkbox for the key that you created in a preceding section of this guide. If an info panel labeled with the name of your key is not already open, click Show info panel.
In the info panel, click
Add member to open the Add members to "KEY_NAME" dialog. In this dialog, do the following:- In the New members box, enter the service account email address that
you made a note of in the preceding section:
service-AI_PLATFORM_PROJECT_NUMBER@cloud-ml.google.com.iam.gserviceaccount.com
In the Select a role drop-down list, click Cloud KMS and then select the Cloud KMS CryptoKey Encrypter/Decrypter role.
Click Save.
- In the New members box, enter the service account email address that
you made a note of in the preceding section:
gcloud
Run the following command:
gcloud kms keys add-iam-policy-binding KEY_NAME \
--keyring=KEY_RING_NAME \
--location=REGION \
--project=KMS_PROJECT_ID \
--member=serviceAccount:service-AI_PLATFORM_PROJECT_NUMBER@cloud-ml.google.com.iam.gserviceaccount.com \
--role=roles/cloudkms.cryptoKeyEncrypterDecrypter
In this command, replace the following placeholders:
- KEY_NAME: The name of the key that you created in a preceding section of this guide.
- KEY_RING_NAME: The key ring that you created in a preceding section of this guide.
- REGION: The region where you created your key ring.
- KMS_PROJECT_ID: The ID of your Cloud KMS project.
- AI_PLATFORM_PROJECT_NUMBER: The project number of your AI Platform project, which you noted in the preceding section as part of a service account email address.
Creating a training job with CMEK
Now that you have granted AI Platform Training permission to encrypt and decrypt data
using your key, you can create a training job that encrypts temporary data using
this key. When you create a training job, specify the encryptionConfig
field in your
job's trainingInput
field.
The following example shows how to do this using the gcloud CLI. You cannot currently create a training job with CMEK in the Google Cloud console.
The example assumes that you have a training
application on your local filesystem in a
directory named trainer
with a module named task
.
Create a
config.yaml
configuration file that contains the following:trainingInput: encryptionConfig: kmsKeyName: projects/KMS_PROJECT_ID/locations/REGION/keyRings/KEY_RING_NAME/cryptoKeys/KEY_NAME
In the configuration file, replace the following placeholders:
- KMS_PROJECT_ID: The ID of your Cloud KMS project.
- KEY_RING_NAME: The name of your key ring.
- KEY_NAME: The name of your key.
To create the training job, run the following command:
gcloud ai-platform jobs submit training JOB_ID \ --config=config.yaml \ --job-dir=JOB_DIR \ --module-name=trainer.task \ --package-path=trainer \ --python-version=3.7 \ --region=REGION \ --runtime-version=2.11 \ --scale-tier=basic
In this command, replace the following placeholders:
- JOB_ID: A name that you choose for the job.
JOB_DIR: A path to a Cloud Storage directory where your training application be uploaded to. Your training job can also use this directory during training; for example, to save output. For convenience, choose a directory in a Cloud Storage bucket in your AI Platform project.
REGION: The region where you created your key ring and where you plan to create the training job.
The command produces the following output if successful:
Job [JOB_ID] submitted successfully. Your job is still active. You may view the status of your job with the command $ gcloud ai-platform jobs describe JOB_ID or continue streaming the logs with the command $ gcloud ai-platform jobs stream-logs JOB_ID jobId: JOB_ID state: QUEUED
You can now monitor your training job.
Besides the region
and encryptionConfig
fields, you can customize your
training job configuration to meet your needs. You
can even use a custom container for
training.
What's next
- Learn more about CMEK on Google Cloud.
- Learn how to use CMEK with other Google Cloud products.
- To learn how to configure additional options besides CMEK, read the guide to running a training job.