Join our session at Google Cloud Next ’23 to learn about the latest Dataflow innovations.
Jump to
Dataflow

Dataflow

Unified stream and batch data processing that's serverless, fast, and cost-effective.

New customers get $300 in free credits to spend on Dataflow.

  • Real-time insights and activation with data streaming and machine learning

  • Fully managed data processing service

  • Automated provisioning and management of processing resources

  • Horizontal and vertical autoscaling of worker resources to maximize resource utilization

  • OSS community-driven innovation with Apache Beam SDK

Benefits

Streaming data analytics with speed

Dataflow enables fast, simplified streaming data pipeline development with lower data latency.

Simplify operations and management

Allow teams to focus on programming instead of managing server clusters as Dataflow’s serverless approach removes operational overhead from data engineering workloads.

Reduce total cost of ownership

Resource autoscaling paired with cost-optimized batch processing capabilities means Dataflow offers virtually limitless capacity to manage your seasonal and spiky workloads without overspending.

Key features

Key features

Ready-to-use real-time AI

Enabled through out-of-the box ML features including NVIDIA GPU and ready-to-use patterns, Dataflow’s real-time AI capabilities allow for real-time reactions with near-human intelligence to large torrents of events.

Customers can build intelligent solutions ranging from predictive analytics and anomaly detection to real-time personalization and other advanced analytics use cases.

Train, deploy, and manage complete machine learning (ML) pipelines, including local and remote inference with batch and streaming pipelines. 

Autoscaling of resources and dynamic work rebalancing

Minimize pipeline latency, maximize resource utilization, and reduce processing cost per data record with data-aware resource autoscaling. Data inputs are partitioned automatically and constantly rebalanced to even out worker resource utilization and reduce the effect of “hot keys” on pipeline performance.

Monitoring and observability

Observe the data at each step of a Dataflow pipeline. Diagnose problems and troubleshoot effectively with samples of actual data. Compare different runs of the job to identify problems easily.

View all features

What's new

What's new