Run a local Airflow environment with Composer Local Development CLI tool

Cloud Composer 1 | Cloud Composer 2

This section describes how to create, configure, and run a local Airflow environment using Composer Local Development CLI tool.

About Composer Local Development CLI tool

Composer Local Development CLI tool streamlines Apache Airflow DAG development for Cloud Composer 2 by running an Airflow environment locally. This local Airflow environment uses an image of a specific Cloud Composer version.

You can create a local Airflow environment based on an existing Cloud Composer environment. In this case, the local Airflow environment takes the list of installed PyPI packages and environment variable names from your Cloud Composer environment.

You can use this local Airflow environment for testing and development purposes, such as to test new DAG code, PyPI packages, or Airflow configuration options.

Warning: Use local Airflow environments only for testing and development purposes. Cloud Composer does not support using locally deployed Cloud Composer images for production purposes.

Before you begin

• Composer Local Development CLI tool supports only Cloud Composer 2 images. You can use any version of Cloud Composer 2 with Composer Local Development CLI tool.

• Composer Local Development CLI tool creates local Airflow environments in a directory where you run the composer-dev create command. To access your local Airflow environment later, run the tool commands in the path where you initially created the local environment. All data for the local environment is stored in a subdirectory at the path where you created the local environment: ./composer/<local_environment_name>.

• Your computer must have enough disk space to store Cloud Composer images. Composer Local Development CLI tool stores one image file for each Cloud Composer version. For example, if you have two local Airflow environments with different Cloud Composer versions, Composer Local Development CLI tool stores two Cloud Composer images.

• Composer Local Development CLI tool uses colorized output. You can disable colorized output with the NO_COLOR=1 variable: NO_COLOR=1 composer-dev <other commands>.

• If you have only one local environment, you can omit the local environment's name from all composer-dev commands, except the run-airflow-cmd.

• Install Composer Local Development CLI tool dependencies:

  • Python versions 3.7 - 3.10 with pip
  • Google Cloud CLI

• Install Docker. Docker must be installed and running in the local system. To verify that Docker is running, you can run any Docker CLI command, such as docker ps.

Configure credentials

If not already done, get new user credentials to use for Application Default Credentials:

gcloud auth application-default login

Login in gcloud CLI using your Google account:

gcloud auth login

All API calls done by the Composer Local Development CLI tool and DAGs are executed from the account that you use in gcloud CLI. For example, if a DAG in your local Airflow environment reads contents of a Cloud Storage bucket, then this account must have permissions to access the bucket. This is different from Cloud Composer environments, where an environment's service account makes the calls.

Install Composer Local Development CLI tool

Clone the Composer Local Development CLI repository:

git clone https://github.com/GoogleCloudPlatform/composer-local-dev.git

In the top-level directory of the cloned repository, run:

pip install .

Depending on your pip configuration, the path at which the tool is installed might not be in PATH variable. If this is the case, pip displays a warning message. You can use information from this warning message to add this directory to the PATH variable in your operating system.

Create a local Airflow environment with a specific Cloud Composer version

To list available versions of Cloud Composer, run:

composer-dev list-available-versions --include-past-releases --limit 10

To create a local Airflow environment with default parameters, run:

composer-dev create \
  --from-image-version IMAGE_VERSION \
  LOCAL_ENVIRONMENT_NAME

Other parameters:

composer-dev create \
  --from-image-version IMAGE_VERSION \
  --project PROJECT_ID \
  --port WEB_SERVER_PORT \
  --dags-path LOCAL_DAGS_PATH \
  LOCAL_ENVIRONMENT_NAME

Replace:

• IMAGE_VERSION with the name of the Cloud Composer image.
• PROJECT_ID with the Project ID.
• WEB_SERVER_PORT with the port that Airflow web server must listen at.
• LOCAL_DAGS_PATH with the path to a local directory where the DAG files are located.
• LOCAL_ENVIRONMENT_NAME with the name of this local Airflow environment.

Example:

composer-dev create \
  --from-image-version composer-2.6.6-airflow-2.6.3 \
  example-local-environment

Create a local Airflow environment from a Cloud Composer environment

Caution: Composer Local Development CLI tool does not produce a local copy of your Cloud Composer environment.

Only the following information is taken from a Cloud Composer environment:

• Image version (versions of Cloud Composer and Airflow used in your environment).
• List of custom PyPI packages installed in your environment.

• Commented list of names of environment variables set in your environment.

  Important: Cloud Composer does not copy the values of environment variables. You can manually uncomment environment variables in the configuration file and set their values, as required.

Other information and configuration parameters from the environment, such as DAG files, DAG run history, Airflow variables, and connections, are not copied from your Cloud Composer environment.

To create a local Airflow environment from an existing Cloud Composer environment:

composer-dev create LOCAL_ENVIRONMENT_NAME \
    --from-source-environment ENVIRONMENT_NAME \
    --location LOCATION \
    --project PROJECT_ID \
    --port WEB_SERVER_PORT \
    --dags-path LOCAL_DAGS_PATH

Replace:

• LOCAL_ENVIRONMENT_NAME with a name for the local Airflow environment.
• ENVIRONMENT_NAME with the name of the Cloud Composer environment.
• LOCATION with the region where the Cloud Composer environment is located.
• PROJECT_ID with the Project ID.
• WEB_SERVER_PORT with a port for the local Airflow web server.
• LOCAL_DAGS_PATH with a path to a local directory where the DAGs are located.

Example:

composer-dev create example-local-environment \
  --from-source-environment example-environment \
  --location us-central1 \
  --project example-project \
  --port 8081 \
  --dags-path example_directory/dags

Start a local Airflow environment

To start a local Airflow environment, run:

composer-dev start LOCAL_ENVIRONMENT_NAME

Replace:

• LOCAL_ENVIRONMENT_NAME with the name of a local Airflow environment.

Stop or restart a local Airflow environment

When you restart a local Airflow environment, Composer Local Development CLI tool restarts the Docker container where the environment runs. All Airflow components are stopped and started again. As a result, all DAG runs that are executed during a restart are marked as failed .

To restart or start a stopped local Airflow environment, run:

composer-dev restart LOCAL_ENVIRONMENT_NAME

Replace:

• LOCAL_ENVIRONMENT_NAME with the name of a local Airflow environment.

To stop a local Airflow environment, run:

composer-dev stop LOCAL_ENVIRONMENT_NAME

Note: The stop command does not delete the local Airflow environment.

Add and update DAGs

Dags are stored in the directory that you specified in the --dags-path parameter when you created your local Airflow environment. By default, this directory is ./composer/<local_environment_name>/dags. You can get the directory used by your environment with the describe command.

To add and update DAGs, change files in this directory. You do not need to restart your local Airflow environment.

View local Airflow environment logs

You can view recent logs from a Docker container that runs your local Airflow environment. In this way, you can monitor container-related events and check Airflow logs for errors such as dependency conflicts caused by PyPI packages installation.

Note: Composer Local Development CLI tool does not write DAG run and task logs to files. You can view these logs in the Airflow UI of your local Airflow environment.

To view logs from a Docker container that runs your local Airflow environment, run:

composer-dev logs LOCAL_ENVIRONMENT_NAME --max-lines 10

To follow the log stream, omit the --max-lines argument:

composer-dev logs LOCAL_ENVIRONMENT_NAME

Run an Airflow CLI command

You can run Airflow CLI commands in your local Airflow environment.

Note: The command syntax is different from Google Cloud CLI syntax for Airflow CLI commands.

To run an Airflow CLI command:

composer-dev run-airflow-cmd LOCAL_ENVIRONMENT_NAME \
  SUBCOMMAND SUBCOMMAND_ARGUMENTS

Example:

composer-dev run-airflow-cmd example-local-environment dags list -o table

Configure local Airflow environments

Composer Local Development CLI tool stores configuration parameters for a local Airflow environment, such as environment variables and PyPI package requirements in the local environment's directory (./composer/<local_environment_name>).

Configuration is applied when a local Airflow environment is started. For example, if you add conflicting PyPI package requirements, then Composer Local Development CLI tool reports errors when you start the local environment.

Important: Composer Local Development CLI tool does not store the airflow.cfg file in this directory. This file is located in the Docker image of the environment. You can change Airflow configuration options by configuring environment variables.

Airflow connections are stored in the database of the local Airflow environment. You can configure them by running an Airflow CLI command or by storing the connection parameters in environment variables. For more information about ways to create and configure connections, see Managing connections in the Airflow documentation.

Get a list and status of local Airflow environments

To list all available local Airflow environments and display their status:

composer-dev list

To describe a specific environment, and get details such as image version, DAGs path, and web server URL of an environment:

composer-dev describe LOCAL_ENVIRONMENT_NAME

Replace:

• LOCAL_ENVIRONMENT_NAME with the name of the local Airflow environment.

List images used by local Airflow environments

To list all images used by Composer Local Development CLI tool, run:

docker images --filter=reference='*/cloud-airflow-releaser/*/*'

Install plugins and change data

Plugins and data for a local Airflow environment are taken from the local environment's directory: ./composer/<local_environment_name>/data and ./composer/<local_environment_name>/plugins).

To change the contents of /data and /plugins directories, add or remove files in these directories. Docker automatically propagates file changes to your local Airflow environment.

Composer Local Development CLI tool does not support specifying a different directory for data and plugins.

Configure environment variables

To configure environment variables, edit the variables.env file in the environment directory: ./composer/<local_environment_name>/variables.env.

Caution: As a safety measure, values of environment variables are not copied from your Cloud Composer environment, and the list of variable names is commented. For example, you might want to specify different values so that your DAGs do not interact with your production environments, or to omit setting some of the variables.

The variables.env file must contain key-value definitions, one line for each environment variable. To change Airflow configuration options, use the AIRFLOW__SECTION__KEY format. For more information about the available environment variables, see Airflow configuration reference.

EXAMPLE_VARIABLE=True
ANOTHER_VARIABLE=test
AIRFLOW__WEBSERVER__DAG_DEFAULT_VIEW=graph

To apply the changes, restart your local Airflow environment.

Install or remove PyPI packages

To install or remove PyPI packages, modify the requirements.txt file in the environment directory: ./composer/<local_environment_name>/requirements.txt.

Requirements must follow the format specified in PEP-508 where each requirement is specified in lowercase and consists of the package name with optional extras and version specifiers.

To apply the changes, restart your local Airflow environment.

Switch to a different Cloud Composer image

You can use any Cloud Composer 2 image with Composer Local Development CLI tool and switch between the images. This approach is different from upgrading your Cloud Composer environment, because configuration parameters of your local Airflow environment are applied when it starts.

For example, after a new Cloud Composer version is released, you can switch your environment to use the new version, and keep existing local Airflow environment configuration. As another example, you can switch between different Airflow versions within a specific Cloud Composer version.

To change the environment's image used by your local Airflow environment:

1. Edit the local environment configuration file: ./composer/<local_environment_name>/config.json.

2. Change the value of the composer_image_version parameter. To view available values, you can list available Cloud Composer versions.

3. To apply the changes, restart your local Airflow environment.

Delete a local Airflow environment

Caution: Make sure that you saved all required data from the environment, such as logs and configuration.

To delete a a local Airflow environment, run the following command:

composer-dev remove LOCAL_ENVIRONMENT_NAME

If the environment is running, add the --force flag to force its removal.

Delete Docker images

To delete all images downloaded by Composer Local Development CLI tool, run:

docker rmi $(docker images --filter=reference='*/cloud-airflow-releaser/*/*' -q)

Troubleshooting

This section provides solutions to common issues.

Unable to start a local environment on MacOS

If you installed the composer-dev package to a directory where Docker cannot access it, then your local environment might not start.

For example, if Python is installed in the /opt directory, such as when you install it with default Homebrew configuration on MacOS, then the composer-dev package is also installed in the /opt directory. Because Docker complies with Apple's sandbox rules, the /opt directory isn't available by default. In addition, you cannot add it through the UI (Settings > Resources > File sharing).

In this case, Composer Local Development CLI tool generates an error message that is similar to the following example:

Failed to create container with an error: 400 Client Error for ...
Bad Request ("invalid mount config for type "bind": bind source path does not exist:
/opt/homebrew/lib/python3.9/site-packages/composer_local_dev/docker_files/entrypoint.sh

Possible reason is that composer-dev was installed in the path that is
not available to Docker. See...")

You can use one of the following solutions:

• Install Python or the composer-dev package to a different directory, so that Docker can access the package.
• Manually edit the ~/Library/Group\ Containers/group.com.docker/settings.json file and add /opt to filesharingDirectories.