Install custom plugins

Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3

Preview — Cloud Composer 3

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

This page describes how to install custom plugins in your Cloud Composer environment.

Apache Airflow's plugin manager allows you to write custom in-house Apache Airflow operators, hooks, sensors, or interfaces. For more information, see custom plugins in the Airflow documentation.

About custom plugins

When you create an environment, Cloud Composer creates a Cloud Storage bucket and associates this bucket with your environment. You use the plugins folder in your environment's bucket to install custom plugins.

Before you begin

• Your account must have a role that allows viewing and modifying your environment's bucket contents.
• This installation method applies only to Airflow plugins. For example, you cannot use this method to install common Python modules or other libraries.
• Plugins must conform to the Airflow plugins template guidelines.

Install a plugin

To install a custom plugin into your Cloud Composer environment, copy the plugin code to the plugins folder in your environment's bucket.

Note: After the plugins are loaded, they are never reloaded, except the UI plugins in the web server. You might need to restart the web server or the scheduler processes manually. Otherwise, a scheduler only restarts after the [scheduler]num_runs, which is set to 5000 by default.

To install a plugin:

Console

1. In the Google Cloud console, go to the Environments page.

   Go to Environments

2. Find your environment and follow the DAGs link.

3. Go one folder level up, then navigate to the plugins/ folder.

4. Upload your plugin files. For more information about uploading objects, see Uploading objects.

gcloud

Use the following gcloud command:

gcloud composer environments storage plugins import \
    --environment ENVIRONMENT_NAME \
    --location LOCATION \
    --source PATH_TO_LOCAL_FILE \
    --destination PATH_IN_SUBFOLDER

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.
• PATH_TO_LOCAL_FILE with the path of the file to upload.
• (Optional) PATH_IN_SUBFOLDER with the subfolder path. Use the --destination argument to upload a plugin to a subfolder in the plugins folder. Otherwise, omit the --destination argument.

View the list of plugins

Console

1. In the Google Cloud console, go to the Environments page.

   Go to Environments

2. Find your environment and follow the DAGs link.

3. Go one folder level up, then navigate to the plugins/ folder.

4. View the plugin files.

gcloud

Use the following gcloud command:

gcloud composer environments storage plugins list \
    --environment ENVIRONMENT_NAME \
    --location LOCATION

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.

Delete a plugin

To delete a plugin from your Cloud Composer environment, delete the plugin code from the plugins folder in the environment's bucket.

Note: You can disable web server plugins without deleting them.

To delete a plugin:

Console

1. In the Google Cloud console, go to the Environments page.

   Go to Environments

2. Find your environment and follow the DAGs link.

3. Go one folder level up, then navigate to the plugins/ folder.

4. Delete the plugin files. For more information about deleting objects, see Deleting objects.

gcloud

Use the following gcloud command:

gcloud composer environments storage plugins delete \
    --environment ENVIRONMENT_NAME \
    --location LOCATION \
    PLUGIN_TO_DELETE

Download plugins

To download plugins, choose an option:

Console

1. In the Google Cloud console, go to the Environments page.

   Go to Environments

2. Find your environment and follow the DAGs link.

3. Go one folder level up, then navigate to the plugins/ folder.

4. Download the plugin files. For more information about deleting objects, see Downloading objects.

gcloud

Use the following gcloud command:

gcloud composer environments storage plugins export \
  --environment ENVIRONMENT_NAME \
  --location LOCATION \
  --destination PATH_TO_LOCAL_DESTINATION \
  --source PATH_IN_FOLDER

Replace:

• ENVIRONMENT_NAME with the name of the environment.
• LOCATION with the region where the environment is located.
• PATH_TO_LOCAL_DESTINATION with the destination for downloaded file.
• (Optional) --source is an option to download only one plugin. PATH_IN_FOLDER is the folder path.

Disable web server plugins

In Cloud Composer 3, if you experience problems with Airflow UI, you can disable web server plugins without having to remove them from the environment's bucket.

Note: You can also disable web server plugins when you create an environment.

gcloud

The following Google Cloud CLI arguments disable and enable web server plugins:

• --no-support-web-server-plugins: disable web server plugins
• --support-web-server-plugins: enable web server plugins

To disable web server plugins, run the following Google Cloud CLI command:

gcloud beta composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --no-support-web-server-plugins

To enable web server plugins, run the following Google Cloud CLI command:

gcloud beta composer environments update ENVIRONMENT_NAME \
  --location LOCATION \
  --support-web-server-plugins

Replace the following:

• ENVIRONMENT_NAME: the name of the environment
• LOCATION: the region where the environment is located

API

1. Create an environments.patch API request.

2. In this request:

   1. In the updateMask parameter, specify the config.software_config.web_server_plugins_mode mask.

   2. In the request body, in the web_server_plugins_mode field:

      • Specify PLUGINS_DISABLED to disable web server plugins.
      • Specify PLUGINS_ENABLED to enable web server plugins.

Example (disable plugins):

// PATCH https://composer.googleapis.com/v1beta1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.software_config.web_server_plugins_mode

"config": {
  "softwareConfig": {
    "web_server_plugins_mode": "PLUGINS_DISABLED"
   }
}

Example (enable plugins):

// PATCH https://composer.googleapis.com/v1beta1/projects/example-project/
// locations/us-central1/environments/example-environment?updateMask=
// config.software_config.web_server_plugins_mode

"config": {
  "softwareConfig": {
    "web_server_plugins_mode": "PLUGINS_ENABLED"
   }
}

Troubleshoot plugin issues

Plugins cause problems with Airflow UI

You can disable web server plugins without deleting them. For more information, see Disable web server plugins.

A newly-uploaded plugin is not visible in Airflow UI

If you use Airflow UI Access Control then the newly uploaded plugin might not be visible in Airflow UI. To address this issue, ask Airflow UI Administrator to configure access to the newly uploaded plugin or assign yourself the Admin role in Airflow UI.

What's next

• Data stored in Cloud Storage
• Manage DAGs