How-to guides

Authentication and access control

• Authentication

  Setting up authentication for a Cloud Composer environment

• Access control

  Controlling access to a Cloud Composer environment

Creating and configuring environments

• Enabling and disabling Cloud Composer

  Enabling the Cloud Composer API

• Configuring shared VPC

  Configuring a host project and networking resources for Cloud Composer

• Configuring private IP

  Enabling private IP in a Cloud Composer environment

• Configuring large-scale networks for Cloud Composer environments

  Organizing large-scale networks for Cloud Composer environments

• Configure email notifications

  Configuring email notifications and SMTP services for Cloud Composer environments

• Specify maintenance windows

  Specify periods of time when maintenance can happen for your environments.

• Creating environments

  Creating a new Cloud Composer environment

• Setting environment variables

  Requirements for setting environment variables

• Managing connections

  Using Airflow connections to access resources in a Google Cloud project

• Using the KubernetesPodOperator

  Launching GKE pods from Cloud Composer into an environment's GKE cluster

Accessing environments

• Airflow command-line interface

  Using the Apache Airflow command-line (CLI) interface with a Cloud Composer environment

• Airflow web interface

  Connecting to the Apache Airflow web interface for a Cloud Composer environment

• Airflow database

  Connecting to the Airflow database and running SQL queries.

Managing workflows

• Writing DAGs

  Writing DAGs (workflows) for a Cloud Composer environment

• Adding and updating DAGs

  Managing DAGs (workflows) in a Cloud Composer environment

• Grouping tasks inside DAGs

  Grouping tasks inside DAGs in a Cloud Composer environment

• Import operators from backport provider packages

  Use new versions of operators in your Airflow 1.10.* environments

• Triggering DAGs

  Triggering DAGs on a schedule and manually

• Triggering DAGs with Cloud Functions

  Triggering DAGs in response to events by using Cloud Functions

• Installing custom plugins

  Installing Plugins in a Cloud Composer environment

• Installing Python dependencies

  Installing Python Package Index dependencies in a Cloud Composer environment

• Testing DAGs

  Testing DAGs locally

• Troubleshooting DAGs

  Troubleshooting common workflow issues

Monitoring environments

• Viewing Airflow logs

  Viewing Airflow task logs for a Cloud Composer environment

• Viewing audit logs

  Viewing audit logs for a Cloud Composer environment

• Monitoring environments in Stackdriver

  Using Stackdriver to monitor a Cloud Composer environment

Updating and upgrading environments

• Updating and deleting environments

  Updating or deleting a Cloud Composer environment

• Upgrading environments

  Upgrade an existing Cloud Composer environment to a new version

• Clean up the Airlfow database

  Remove old entries from the Airflow database to reduce its size.

• Troubleshoot PyPI package installation

  Troubleshoot PyPI package dependency conflicts in Cloud Composer environments.

• Migrate environments to Airflow 2.0

  Transfer DAGs, data and configuration from your existing Airflow 1.10.* environments to Airflow 2 environments