Apache Airflow's plugin manager allows you to write custom in-house Apache Airflow operators, hooks, sensors, or interfaces. For more information, see custom plugins in the Airflow documentation. This page describes how to install custom plugins in your Cloud Composer environment.
When you create an environment, Cloud Composer creates a Cloud Storage
bucket and associates the bucket with your
environment. You use the plugins
folder
in the associated bucket to install custom plugins.
To determine the Cloud Storage bucket attached to your environment,
see Adding and Updating DAGs.
Before you begin
- The following permissions are required to add and update plugins in
the Cloud Storage bucket for the Cloud Composer environment:
storage.objectAdmin
to upload files.composer.environments.get
to look up the DAG destination bucket. This permission is not required when using the Cloud Storage API orgsutil
.
This installation method applies only to Airflow plugins. For example, you cannot use this method to install common Python modules or other libraries.
Plugins must conform to the Airflow plugins template guidelines.
Installing a plugin
To install a custom plugin into your Cloud Composer environment,
copy the plugin code to the plugins
folder in the Cloud Storage bucket
attached to your Cloud Composer environment.
To install a plugin, choose an option:
- Use the Cloud Storage browser in the Cloud Console.
- Use the following
gcloud
command:
gcloud composer environments storage plugins import --environment ENVIRONMENT_NAME \ --location LOCATION \ --source PATH_TO_LOCAL_FILE \ --destination PATH_IN_SUBFOLDER
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.PATH_TO_LOCAL_FILE
is the path of the file to upload.--destination
is an option to upload a plugin to a subfolder in theplugins
folder.PATH_IN_SUBFOLDER
is the subfolder path such asfoo/bar
.
Viewing a list of plugins
To view a list of plugins, choose an option:
- Use the Cloud Storage browser in the Cloud Console.
- Use the following
gcloud
command, replacing VARIABLES with the appropriate values:
gcloud composer environments storage plugins list --environment ENVIRONMENT_NAME \ --location LOCATION
Deleting a plugin
To delete a plugin from your Cloud Composer environment,
delete the plugin code from the plugins
folder in the Cloud Storage bucket
attached to your Cloud Composer environment.
To delete a plugin, choose an option:
- Use the Cloud Storage browser in the Cloud Console.
- Use the following
gcloud
command, replacing VARIABLES with the appropriate values:
gcloud composer environments storage plugins delete --environment ENVIRONMENT_NAME \ --location LOCATION PLUGIN_TO_DELETE
Downloading plugins
To download plugins, choose an option:
- Use the Cloud Storage browser in the Cloud Console.
- Use the following
gcloud
command:
gcloud composer environments storage plugins export --environment ENVIRONMENT_NAME \ --location LOCATION \ --destination PATH_TO_LOCAL_DESTINATION \ --source PATH_IN_FOLDER
where:
ENVIRONMENT_NAME
is the name of the environment.LOCATION
is the Compute Engine region where the environment is located.PATH_TO_LOCAL_DESTINATION
is the destination for downloaded file.--source
is an option to download only one plugin.PATH_IN_FOLDER
is the folder path such as /foo.