GPUs with Dataflow
Dataflow GPUs bring the accelerated benefits directly to your stream or batch data processing pipeline. Use Dataflow to simplify the process of getting data to the GPU and to take advantage of data locality. At the same time, get all the benefits of the fully managed Dataflow system: host provisioning, autoscaling, fault tolerance, and more.
Run a Dataflow job with GPUs
Follow this tutorial to see how to build a custom container image and run a Dataflow pipeline with GPUs.
Run a simple PyTorch pipeline with GPUs
Follow this tutorial to build a PyTorch pipeline and run it on Dataflow with GPUs.
Run a pipeline with NVIDIA L4 GPUs
The L4 GPU type is useful for running machine learning inference pipelines.
Run a simple TensorFlow pipeline with GPUs
Follow this tutorial to build a TensorFlow pipeline and run it on Dataflow with GPUs.
Resources
Developer best practices
See an example of a developer workflow for building pipelines using GPUs.
Right fitting
Use right fitting with your batch jobs to customize worker resources and reduce costs.
Troubleshoot jobs that use GPUs
If you run into problems running your Dataflow job with GPUs, follow these troubleshooting steps.