Dataflow HPC highly parallel workloads
When you're working with high-volume grid-computing, use Dataflow to run HPC highly parallel workloads in a fully managed system. With Dataflow, you can run your highly parallel workloads in a single pipeline, improving efficiency and making your workflow easier to manage. The data stays in one system both for pre- and post-processing and for task processing. Dataflow automatically manages performance, scalability, availability, and security needs.
Custom container tutorial
Follow this tutorial to see an end-to-end example of an HPC highly parallel pipeline that uses custom containers with C++ libraries.
HPC highly parallel best practices
Learn about best practices to consider when designing your HPC highly parallel pipeline.
Resources
Use GPUs
Using GPUs in Dataflow jobs can accelerate image processing and machine learning processing tasks.
Case study
HSBC used a Dataflow HPC highly parallel workflow to increase calculation capacity and speed while lowering costs.
View the examples on GitHub
The Dataflow HPC highly parallel pipeline example and the corresponding source code are available on GitHub.