Try Gemini 1.5 Pro, our most advanced multimodal model in Vertex AI, and see what you can build with a 1M token context window. Try Gemini 1.5 Pro, our most advanced multimodal model in Vertex AI, and see what you can build with a 1M token context window.

Dataproc Serverless components

The Dataproc Serverless components let you run Apache Spark batch workloads from a pipeline within Vertex AI Pipelines. Dataproc Serverless runs the batch workloads on a managed compute infrastructure, autoscaling resources as needed.

Learn more about Dataproc Serverless and supported Spark workloads.

In Dataproc Serverless, a Batch resource represents a batch workload. The Google Cloud Pipeline Components SDK includes the following operators to create Batch resources and monitor their execution:

API reference

For component reference, see the google_cloud_pipeline_components SDK reference page.

For Dataproc Serverless resource reference, see the following API reference page:

Batch resource reference

Tutorials

Get started with Dataproc Serverless pipeline components

Version history and release notes

To learn more about the version history and changes to the Google Cloud Pipeline Components SDK, see the Google Cloud Pipeline Components SDK Release Notes.

Technical support contacts

If you have any questions, reach out to kfp-dataproc-components@google.com.