Cloud Composer 1 is in the post-maintenance mode. Google does not release any further updates to Cloud Composer 1, including new versions of Airflow, bugfixes, and security updates. We recommend planning migration to Cloud Composer 2.

Troubleshooting PyPI package installation

Cloud Composer 1 | Cloud Composer 2

This page explains how to troubleshoot dependency conflicts when installing custom PyPI packages.

The most common problem with PyPI packages that you might encounter in Cloud Composer are dependency conflicts.

When you specify a new custom PyPI package to install, this package or its version might cause dependency conflicts with other custom PyPI packages or preinstalled packages in your environment. In this case, the environment update operation fails with an error.

Cloud Build and in-cluster build errors

If a problem with packages occurs, you get the error message about it in two ways, depending on the way your environment is deployed:

Cloud Build. An error message and a link to the Cloud Build log. For example:

UPDATE operation on this environment failed 25 minutes ago with the
following error message:
Failed to install PyPI packages.  Check the Cloud Build log at ...

In-cluster builds. An error message and the location of build logs. For example:

UPDATE operation on this environment failed 17 minutes ago with the
following error message:
Failed to install PyPI packages.  Check the in-cluster build logs for
details. They can be found in the Environment logs under the build-log-*
log name.

See the detailed error message

When a package installation fails, pip reports the detailed error message. You can find this error message in the build logs.

Locate the `pip` error in Cloud Build logs

You can follow the link from the Cloud Build error message, or locate build logs:

Locate build logs:
1. In Google Cloud console, go to the Build history page.
  
  Go to Build history
2. Select the failed build to see its logs.

In the build log, find the error message from pip. For example:

ERROR: apache-airflow-backport-providers-google 2021.2.5 has requirement
google-cloud-logging<3.0.0,>=2.1.1, but you'll have google-cloud-logging
1.15.0 which is incompatible.

Locate the `pip` error in in-cluster build logs

Locate build logs:
1. In Google Cloud console, go to the Environments page.
  
  Go to Environments
2. Select your environment.
3. Open the Logs tab.
4. Select All logs > Composer logs > Builds > Worker & Scheduler image.
5. In the Severity drop-down list, select Info.

In the displayed logs:

Locate the installer.sh error message. For example:

The command '/bin/sh -c bash installer.sh $COMPOSER_PYTHON_VERSION
fail' returned a non-zero code: 1

The preceding info messages have the detailed pip error. For example:

apache-airflow-backport-providers-google 2021.2.5 has requirement
google-cloud-logging<3.0.0,>=2.1.1, but you have
google-cloud-logging 1.15.0.

Conflicts with preinstalled PyPI packages

Some package conflicts occur between custom PyPI packages that you install and the preinstalled packages.

You can see the full list of preinstalled packages for your version of Cloud Composer on the Cloud Composer versions page.

To solve this problem, you can:

Install a different version of the custom PyPI package.
Install a different version of the preinstalled package. To do so, install a custom PyPI package with the name of the preinstalled package and specify the required version. We do not recommend downgrading preinstalled packages.
See if later Cloud Composer versions use a different version of the preinstalled package. You can check for potential PyPI package conflicts before you upgrade your environment to a later version of Cloud Composer.
Use PythonVirtualenvOperator to isolate code snippets that require conflicting packages.

Metadata server unavailable when installing PyPI packages

If your environment uses a custom pip.conf file, then you might face a problem when no new PyPI packages can be installed in your environment. In Cloud Build logs related to the package installation error, you can see the following warning message:

WARNING: Compute Engine Metadata server unavailable on attempt 3 of 3. Reason:
timed out
WARNING: Authentication failed using Compute Engine authentication due to
unavailable metadata server.

This problem is caused by a pip.conf file that does not allow access to the default package index at http://pypi.org/simple. For example, if your pip.conf file only permits installing packages from a custom Artifact Registry repository because of the redefined index-url parameter, installing packages from the default package index is not available.

To solve this problem, make sure that http://pypi.org/simple package index is added to your pip.conf file. For example, if your primary package index is defined in the index-url parameter, add the http://pypi.org/simple index in the extra-index-url parameter.