Installing Custom Plugins

Apache Airflow's plugin manager allows you to write custom in-house Apache Airflow operators, hooks, sensors, or interfaces. For more information, see custom plugins in the Airflow documentation. This page describes how to install custom plugins in your Cloud Composer environment.

When you create an environment, Cloud Composer creates a Cloud Storage bucket and associates the bucket with your environment. You use the plugins folder in the associated bucket to install custom plugins. To determine the Cloud Storage bucket attached to your environment, see Adding and Updating DAGs.

Before you begin

• The following permissions are required to add and update plugins in the Cloud Storage bucket for the Cloud Composer environment:
  • storage.objectAdmin to upload files.
  • composer.environments.get to look up the DAG destination bucket. This permission is not required when using the Cloud Storage API or gsutil.

• This installation method applies only to Airflow plugins. For example, you cannot use this method to install common Python modules or other libraries.

• Plugins must conform to the Airflow plugins template guidelines.

Installing a plugin

To install a custom plugin into your Cloud Composer environment, copy the plugin code to the plugins folder in the Cloud Storage bucket attached to your Cloud Composer environment.

To install a plugin, choose an option:

• Use the Cloud Storage browser in the GCP Console.
• Use the following gcloud command:

gcloud composer environments storage plugins import --environment ENVIRONMENT_NAME \
    --location LOCATION \
    --source PATH_TO_LOCAL_FILE \
    --destination PATH_IN_SUBFOLDER

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• PATH_TO_LOCAL_FILE is the path of the file to upload.
• --destination is an option to upload a plugin to a subfolder in the plugins folder. PATH_IN_SUBFOLDER is the subfolder path such as foo/bar.

Viewing a list of plugins

To view a list of plugins, choose an option:

• Use the Cloud Storage browser in the GCP Console.
• Use the following gcloud command, replacing VARIABLES with the appropriate values:

gcloud composer environments storage plugins list --environment ENVIRONMENT_NAME \
    --location LOCATION

Deleting a plugin

To delete a plugin from your Cloud Composer environment, delete the plugin code from the plugins folder in the Cloud Storage bucket attached to your Cloud Composer environment.

To delete a plugin, choose an option:

• Use the Cloud Storage browser in the GCP Console.
• Use the following gcloud command, replacing VARIABLES with the appropriate values:

gcloud composer environments storage plugins delete --environment ENVIRONMENT_NAME \
    --location LOCATION PLUGIN_TO_DELETE

Downloading plugins

To download plugins, choose an option:

• Use the Cloud Storage browser in the GCP Console.
• Use the following gcloud command:

gcloud composer environments storage plugins export --environment ENVIRONMENT_NAME \
    --location LOCATION \
    --destination PATH_TO_LOCAL_DESTINATION \
    --source PATH_IN_FOLDER

where:

• ENVIRONMENT_NAME is the name of the environment.
• LOCATION is the Compute Engine region where the environment is located.
• PATH_TO_LOCAL_DESTINATION is the destination for downloaded file.
• --source is an option to download only one plugin. PATH_IN_FOLDER is the folder path such as /foo.

What's next

• Apache Airflow plugins
• Data Stored in Cloud Storage