Troubleshooting environment updates and upgrades

This page provides troubleshooting information for problems that you might encounter while updating or upgrading Cloud Composer environments.

For troubleshooting information related to creating environments, see Troubleshooting environment creation.

When Cloud Composer environments are updated, the majority of issues happen because of the following reasons:

Insufficient permissions to update or upgrade an environment

The service account that updates or upgrades an environment or a user account that triggers the operation must have at least the Environment and Storage Object Administrator role, as described in Cloud Composer access control

If Cloud Composer cannot update or upgrade an environment because of insufficient permissions, it outputs the following error message:

ERROR: (gcloud.composer.environments.update) PERMISSION_DENIED: The caller does not have permission

Solution: Assign at least the Environment and Storage Object Administrator role to the user or service account.

The service account of the environment's GKE cluster has insufficient permissions

When creating a Cloud Composer environment, you can specify a service account that runs the environment's GKE nodes. If you do not specify any service account, then Google Compute Engine service account is used.

If the service account that runs Cloud Composer data plane (GKE nodes) does not have the required permissions to run update and upgrade operations, Cloud Composer outputs the following error:

    UPDATE operation on this environment failed 3 minutes ago with the
    following error message:
    Composer Backend timed out. Currently running tasks are [stage:
    CP_COMPOSER_AGENT_RUNNING
    description: "No agent response published."
    response_timestamp {
      seconds: 1618203503
      nanos: 291000000
    }
    ].

Solution: Make sure that the Composer data plane service account has the following permissions:

  • For Public IP Cloud Composer configuration, the Project Editor or the Composer Worker role.

  • For Private IP Cloud Composer configuration, the Project Editor or the Composer Worker role, and the iam.serviceAccountUser role.

The size of the Airflow database is too big to perform the operation

Cloud Composer upgrade operation might not succeed because the size of the Airflow database is too big for upgrade operations to succeed.

If the size of the Airflow database is more than 16 GB, Cloud Composer outputs the following error:

Airflow database uses more than 16 GB. Please clean the database before upgrading.

Solution: Perform the Airflow database cleanup, as described in Airflow database maintenance.

What's next