Using VPC Service Controls with AI Platform Training

VPC Service Controls can help you mitigate the risk of data exfiltration from your AI Platform Training jobs. When you run a training job from a project inside a service perimeter, VPC Service Controls ensures that your data does not leave the perimeter. This includes training data that your job accesses and artifacts that your job creates.

Creating a service perimeter

Follow the VPC Service Controls guide to creating a service perimeter. When specifying which services you want to restrict, make sure to add all of the following services:

  • AI Platform Training and Prediction API (ml.googleapis.com)
  • Pub/Sub API (pubsub.googleapis.com)
  • Cloud Storage API (storage.googleapis.com)
  • Google Kubernetes Engine API (container.googleapis.com)
  • Container Registry API (containerregistry.googleapis.com)
  • Cloud Logging API (logging.googleapis.com)

Your VPC Service Controls perimeter must restrict these services in order to fully protect your AI Platform Training jobs.

Limitations

After you have created a service perimeter and added your Google Cloud project to it, you can run training jobs without any additional configuration. However, the following limitations apply:

  • If you submit a training job in the first few minutes after creating a service perimeter, then the job might fail. Wait approximately 15 minutes for the VPC Service Controls restrictions to propagate to all the relevant Google Cloud services, and then try again.
  • You cannot perform training with TPUs.
  • When ml.googleapis.com is protected, your training job does not have access to resources outside the perimeter. Your training code can access data in Cloud Storage and other Google Cloud services supported by VPC Service Controls in projects within the perimeter, but if your code sends requests to services outside the perimeter, those requests will fail.
  • Without additional configuration, you cannot use the Google Cloud Console to manage the training jobs of a project inside a service perimeter or to view logs. Learn about accessing resources protected by a service perimeter in the Cloud Console.

AI Platform Prediction

When you create a service perimeter that protects the AI Platform Training and Prediction API, VPC Service Controls protects AI Platform Training but not AI Platform Prediction. However, AI Platform does disable certain AI Platform Prediction functionality to help protect your data from exfiltration. When you protect the API with a service perimeter, performing any of the following actions causes an error:

To fully protect your data from exfiltration, you must also make sure that your Google Cloud project does not contain any models or model versions before you create your service perimeter. This prevents online prediction, which is not protected.

You can use a separate Google Cloud project outside of your service perimeter for any AI Platform Prediction actions.

What's next