Installing Python Dependencies

If your DAGs require Python dependencies obtainable from the Python Package Index, you can install them in your environment either through the Google Cloud Platform Console or by making a partial update request to the environment using the gcloud command-line tool.

The dependencies are installed with the existing Python dependencies included in the base environment. If your Python dependencies can't be found in the Python Package Index, consider using the plugins feature.

Before you begin

Requirements must follow the format specified in PEP-508 where each requirement is specified in lowercase and consists of the package name with optional extras and version specifiers.

Adding, updating, or deleting Python dependencies

To add, update, or delete the Python dependencies for your environment:

Console

Specify the package name and version specifiers as shown:

  • "pi-python-client", "===1.1.post1"
  • "go-api-python-client", "===1.0.0.dev187"

For a package without the version specifier, use an empty string for the value, such as "glob2", " ".

To access an environment's Python dependencies, navigate to the PyPi dependencies page using the following steps:

  1. Open the Environments page in the Google Cloud Platform Console.

    Open the Environments page

  2. Click the Name of the environment you want to install, update, or delete Python dependencies for.

  3. Select the PyPi dependencies tab.

  4. Click the Edit button.

  5. To add a new dependency:

    1. Click the Add dependency button.

    2. Enter the name and version of your library in the Name and Version fields.

  6. To update an existing dependency:

    1. Select the Name and/or Version field of the library you want to update.

    2. Enter a new value.

  7. To delete a dependency:

    1. Hover over the name of the dependency to delete.

    2. Click the trash can icon that appears.

gcloud

Pass a requirements.txt file to the gcloud command-line tool. Format the file with each requirement specifier on a separate line.

Sample requirements.txt file:

scipy>=0.13.3
scikit-learn
nltk[machine_learning]

Pass the requirements.txt file to the environments.set-python-dependencies command to set your installation dependencies.

gcloud beta composer environments update --update-pypi-packages-from-file
ENVIRONMENT_NAME  --location LOCATION requirements.txt

The command terminates when the operation is finished. To avoid waiting, use the --async flag.

If a dependency conflict causes the update to fail, your environment continues running with its existing dependencies. If the operation succeeds, you can begin using the newly installed Python dependencies in your DAGs.

Send feedback about...

Cloud Composer