This page provides an overview of Cloud Composer, the Google Cloud solution for creating and deploying workflows with Apache Airflow.
Why use Cloud Composer?
Cloud Composer is a fully managed workflow orchestration service, enabling you to create workflows that span across clouds and on-premises data centers. Built on the popular Apache Airflow open source project and operated using the Python programming language, Cloud Composer is free from lock-in and easy to use. By using Cloud Composer instead of a local instance of Apache Airflow, users can benefit from the best of Airflow with no installation or management overhead.
Workflows, DAGs, and tasks
In data analytics, a workflow represents a series of tasks for ingesting, transforming, analyzing, or utilizing data. In Airflow, workflows are created using DAGs, or "Directed Acyclic Graphs".
A DAG is a collection of tasks that you want to schedule and run, organized in a way that reflects their relationships and dependencies. DAGs are created in Python scripts, which define the DAG structure (tasks and their dependencies) using code.
Each task in a DAG can represent almost anything—for example, one task might perform any of the following functions:
- Preparing data for ingestion
- Monitoring an API
- Sending an email
- Running a pipeline
A DAG shouldn't be concerned with the function of each constituent task—its purpose is to ensure that each task is executed at the right time, in the right order, or with the right issue handling.
For more information on DAGs and tasks, see the Apache Airflow documentation.
To run workflows first need to create an environment. Airflow depends on many micro-services to run, so Cloud Composer provisions Google Cloud components to run your workflows. These components are collectively known as a Cloud Composer environment.
Environments are self-contained Airflow deployments based on Google Kubernetes Engine, and they work with other Google Cloud services using connectors built into Airflow. You can create one or more environments in a single Google Cloud project. You can create Cloud Composer environments in any supported region.
For an in-depth look at the components of an environment, see Cloud Composer environment architecture.