Troubleshooting PyPI package installation

Cloud Composer 1 | Cloud Composer 2

This page explains how to troubleshoot dependency conflicts when installing custom PyPI packages.

The most common problem with PyPI packages that you might encounter in Cloud Composer are dependency conflicts.

When you specify a new custom PyPI package to install, this package or its version might cause dependency conflicts with other custom PyPI packages or preinstalled packages in your environment. In this case, the environment update operation fails with an error.

Cloud Build and in-cluster build errors

If a problem with packages occurs, you get the error message about it in two ways, depending on the way your environment is deployed:

  • Cloud Build. An error message and a link to the Cloud Build log. For example:

    UPDATE operation on this environment failed 25 minutes ago with the
    following error message:
    Failed to install PyPI packages.  Check the Cloud Build log at ...
    
  • In-cluster builds. An error message and the location of build logs. For example:

    UPDATE operation on this environment failed 17 minutes ago with the
    following error message:
    Failed to install PyPI packages.  Check the in-cluster build logs for
    details. They can be found in the Environment logs under the build-log-*
    log name.
    

See the detailed error message

When a package installation fails, pip reports the detailed error message. You can find this error message in the build logs.

Locate the pip error in Cloud Build logs

You can follow the link from the Cloud Build error message, or locate build logs:

  1. Locate build logs:

    1. In Google Cloud console, go to the Build history page.

      Go to Build history

    2. Select the failed build to see its logs.

  2. In the build log, find the error message from pip. For example:

    ERROR: apache-airflow-backport-providers-google 2021.2.5 has requirement
    google-cloud-logging<3.0.0,>=2.1.1, but you'll have google-cloud-logging
    1.15.0 which is incompatible.
    

Locate the pip error in in-cluster build logs

  1. Locate build logs:

    1. In Google Cloud console, go to the Environments page.

      Go to Environments

    2. Select your environment.

    3. Open the Logs tab.

    4. Select All logs > Composer logs > Builds > Worker & Scheduler image.

    5. In the Severity drop-down list, select Info.

  2. In the displayed logs:

    1. Locate the installer.sh error message. For example:

      The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION
      fail' returned a non-zero code: 1
      
    2. The preceding info messages have the detailed pip error. For example:

      apache-airflow-backport-providers-google 2021.2.5 has requirement
      google-cloud-logging<3.0.0,>=2.1.1, but you have
      google-cloud-logging 1.15.0.
      

Conflicts with preinstalled PyPI packages

Some package conflicts occur between custom PyPI packages that you install and the preinstalled packages.

You can see the full list of preinstalled packages for your version of Cloud Composer on the Cloud Composer versions page.

To solve this problem, you can:

  • Install a different version of the custom PyPI package.

  • Install a different version of the preinstalled package. To do so, install a custom PyPI package with the name of the preinstalled package and specify the required version. We do not recommend downgrading preinstalled packages.

  • See if later Cloud Composer versions use a different version of the preinstalled package. You can check for potential PyPI package conflicts before you upgrade your environment to a later version of Cloud Composer.

What's next