Developers & Practitioners
Access larger dataset faster and easier to accelerate your ML models training in Vertex AI
Vertex AI Training delivers a serverless approach to simplify the ML model training experience for customers. As such, training data does not persist on the compute clusters by design. In the past, customers had only Cloud Storage (GCS) or BigQuery (BQ) as storage options. Now, you can also use NFS shares, such as Filestore, for training jobs and access data in the NFS share as you would files in a local file system.
Built-in NFS support for custom training jobs provides the following benefits:
Delivers an easy way to store and access large datasets for Vertex AI Training with less of the cumbersome work involving moving training data around.
Training jobs execute faster by eliminating the data download steps.
Data streams over the network with higher throughput compared to using alternative storage solutions.
This article demonstrates how to create a Filestore instance and how to use the data that’s stored in the instance to train a model with your custom training code.
Create a Filestore instance and copy data
First let’s create a Filestore instance as our NFS file server.
In the Cloud Console, go to the Filestore Instances page and click Create instance.
Configure the instance based on your needs, noting the following:
For this tutorial, we used the "default" VPC network for simplicity. You may choose any network you want, but save the network name as we will need it later.
Ensure that you are using "private service access" as the connection mode.
For in depth instructions, see Creating instances.
Your new instance will show on the dashboard page. Click on the name of the instance to view the details of the instance.
Save the NFS mount point information, which is in the form of
SERVER:PATH. We will use it later.
Copy data to your instance by following the instructions from the official guide.
Set up VPC Network Peering
Since we chose “private service access” mode for our Filestore instance as mentioned above, we already have VPC peering established between our network and Google services. If you’re using a third party NFS solution, you may need to set up the peering yourself as instructed in Set up VPC Network Peering.
Create a Custom Job accessing NFS
Once you have the NFS share and VPC peering set up, you are ready to use it with your custom training jobs. In this section, we will use the gcloud CLI to create a custom training job that can access the files in your NFS share.
To be specific, the process can be simplified into following general steps:
Decide a mount point directory under the path
/mnt/nfs/. Your NFS share will be mounted to this directory when you submit jobs.
In your custom code, you can access your NFS file share via the local path to your mount point directory.
Specify the “nfsMount” field and
networkfields in your training job request and submit it.
For example, we make
my_mount the “Mount Point'' folder. Then in our custom code, we can specify
/mnt/nfs/my_mount to get the data stored in our Filestore instance:
with open('/mnt/nfs/my_mount/data.csv', 'r') as f:
lines = f.readlines()
We may also write to the Filestore instance via that local path:
with open('/mnt/nfs/my_mount/epoch3.log', 'a') as f:
gcloud ai custom-jobs create \
- server: 10.76.0.10
Then we can check the status of your training job and see how it successfully reads/writes the data from your NFS file shares.
In this article, we used Filestore to demonstrate how to access files in an NFS share by mounting it to Vertex AI. We created a Filestore instance and VPC peering connections, and then submitted a job that can directly read from Filestore as a local directory.
By leveraging the performance and throughput benefits of streaming data from NFS shares such as Filestore, it simplifies and accelerates the process to run training jobs on Vertex AI, which empowers users to train even better models with more data.
Recommended for you
What is Vertex AI? Developer advocates share more
Vertex AI unifies Google Cloud’s existing ML offerings into a single environment for efficiently building and managing the lifecycle of ML projects. It provides tools for every step of the machine learning workflow across different model types, for varying levels of machine learning expertise.
Mount an NFS share for custom training
You can configure your custom training jobs to mount Network File System (NFS) shares to the container where your code is running. This allows your jobs to access remote files as though they are local with high throughput and low latency. This guide shows how to mount an NFS share when running a custom training job.
Use a private IP for custom training
This guide shows how to run custom training jobs in your network after you have already set up VPC Network Peering to peer your network with a Vertex AI CustomJob, HyperparameterTuningJob, or custom TrainingPipeline resource.