Choose Workflows or Cloud Composer for service orchestration

Both Workflows and Cloud Composer can be used for service orchestration to combine services to implement application functionality or perform data processing. Although they are conceptually similar, each is designed for a different set of use cases. This page helps you choose the right product for your use case.

Key differences

The core difference between Workflows and Cloud Composer is what type of architecture each product is designed to support.

Workflows orchestrates multiple HTTP-based services into a durable and stateful workflow. It has low latency and can handle a high number of executions. It's also completely serverless.

Workflows is great for chaining microservices together, automating infrastructure tasks like starting or stopping a VM, and integrating with external systems. Workflows connectors also support simple sequences of operations in Google Cloud services such as Cloud Storage and BigQuery.

Cloud Composer is designed to orchestrate data driven workflows (particularly ETL/ELT). It's built on the Apache Airflow project, but Cloud Composer is fully managed. Cloud Composer supports your pipelines wherever they are, including on-premises or across multiple cloud platforms. All logic in Cloud Composer, including tasks and scheduling, is expressed in Python as Directed Acyclic Graph (DAG) definition files.

Cloud Composer is best for batch workloads that can handle a few seconds of latency between task executions. You can use Cloud Composer to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline. You can use pre-existing operators to communicate with various services, and there are over 150 operators for Google Cloud alone.

Detailed feature comparison

Feature Workflows Cloud Composer
Syntax Workflows syntax in YAML or JSON format Python
State model Imperative flow control Declarative DAG with automatic dependency resolution
Integrations HTTP requests and connectors Airflow Operators and Sensors
Passing data between steps 64KB for variables 48KB1 for XCom
Execution triggers and scheduling gcloud tool, Cloud Console, Workflows API, Workflows client libraries, Cloud Scheduler Cron-like schedules in the DAG definition file, Airflow Sensors
Asynchronous patterns
  • Polling
  • Waiting for long-running Google Cloud operations
Polling
Parallel execution Available via experimental.executions.map Automatic based on dependencies
Execution latency Milliseconds Seconds
Based on open source No Yes (Apache Airflow)
Scaling model Serverless (scales up to demand and down to zero) Provisioned
Billing model Usage-based (per step executed) Based on provisioned capacity
Data processing features No Backfills, ability to re-run DAGs

  1. Source code for airflow.models.xcom. Apache Airflow documentation. August 2, 2021.