Stay organized with collections
Save and categorize content based on your preferences.
There are multiple use cases where storage services can
be used in artificial intelligence (AI) and machine learning (ML) workloads.
Storage use cases
Use cases where storage services might be used include the following:
High durability, medium I/O operations
Low durability, high I/O operations
Other operations
Loading model binaries for training
Loading model variables for inference
Storing model checkpoints while trainings
Storing scratch or temporary data
Loading VM images
Loading data for training
Loading model weights
Logging data
Storage recommendations
To ensure optimization of ML system performance, a combination of storage
services from both our first party and third party catalog might be appropriate.
The following storage services are recommended for each use case as follows:
Through integration with Cloud Storage FUSE, Cloud Storage buckets can
be mounted as a file system
Supports large scale (TBs to EBs) training data for GPU and TPU clusters
Supports high-throughput (up to 1.2TB/s bandwidth or greater)
training and inference.
To gain this throughput you need to tune Cloud Storage FUSE,
use a Cloud Storage FUSE File Cache, and plan for Cloud Storage
bandwidth.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Hard to understand","hardToUnderstand","thumb-down"],["Incorrect information or sample code","incorrectInformationOrSampleCode","thumb-down"],["Missing the information/samples I need","missingTheInformationSamplesINeed","thumb-down"],["Other","otherDown","thumb-down"]],["Last updated 2025-01-30 UTC."],[],[]]
