Concepts

• Apache Beam programming model

  An overview of Apache Beam's unified programming model for batch and streaming data processing. Learn about the programming model basics, including pipelines, PCollections, transforms, and pipeline I/O.

• Streaming pipelines

  An overview of how Dataflow processes streaming data.

• Dataflow templates

  An overview of how to use Dataflow templates to stage your pipelines on Google Cloud and run them using the Google Cloud console, the Google Cloud CLI, or REST API calls.

• Security and permissions

  An overview of how Dataflow handles security and permissions for your data and your pipeline's managed cloud resources.

• Access control with IAM

  An overview of how Dataflow controls your project's users' access to Dataflow-specific resources.

• SDK and Worker Dependencies

  An overview of dependency and worker package information for Apache Beam and Dataflow SDKs.

• Regional endpoints

  An overview of Dataflow's regional endpoints, that allow you to specify a region for deploying your Dataflow jobs.

• Streaming with Pub/Sub

  An overview of Dataflow's integration with Pub/Sub.

• GPU support

  An overview of how GPUs work with Dataflow.

• Execution details

  An overview of the Execution details tab in the web-based monitoring user interface.

• Audit logging

  An overview of the audit logs created by Dataflow as part of Cloud Audit Logs.