Troubleshooting environment creation

This page provides troubleshooting information for problems that you might encounter while creating Cloud Composer environments.

For troubleshooting information related to updating and upgrading environments, see Troubleshooting environment updates and upgrades.

When Cloud Composer environments are created, the majority of issues happen because of the following reasons:

  • Service account permission problems.

  • Network-related issues. For example, invalid VPC configuration, IP address conflicts, or network IP ranges that are too narrow.

  • Quota-related issues.

Insufficient permissions to create an environment

The service account that creates an environment or a user account that triggers the operation must have the following permissions:

If Cloud Composer cannot create an environment because of insufficient permissions, it outputs the following error messages:

ERROR: (gcloud.composer.environments.create) PERMISSION_DENIED: The caller
does not have permission

or

ERROR: (gcloud.composer.environments.create) PERMISSION_DENIED: User not
authorized to act as service account <service-account-name>.
The user must be granted iam.serviceAccounts.actAs permission, included in
Owner, Editor, Service Account User role. See https://cloud.google.com/iam/docs
/understanding-service-accounts for additional details.

Solution: Assign Composer Administrator and iam.serviceAccounts.actAs permissions to the user or service account.

The service account of the environment's GKE cluster has insufficient permissions

When creating a Cloud Composer environment, you can specify a service account that runs the environment's GKE nodes. If you do not specify any service account, then Google Compute Engine service account is used.

If the service account that runs Cloud Composer data plane (GKE nodes) does not have the required permissions, Cloud Composer outputs the following error:

Errors in: [Web server]; Error messages:
  Creation of airflow web server version failed. This may be an intermittent
  issue of the App Engine service. You may retry the operation later.
{"ResourceType":"appengine.v1.version","ResourceErrorCode":"504","ResourceError
Message":"Your deployment has failed to become healthy in the allotted time
and therefore was rolled back. If you believe this was an error, try adjusting
the 'app_start_timeout_sec' setting in the 'readiness_check' section."}

Solution: Make sure that the Composer data plane service account has the following permissions:

  • For Public IP Cloud Composer configuration, the Project Editor or the Composer Worker role.

  • For Private IP Cloud Composer configuration, the Project Editor or the Composer Worker role, and the iam.serviceAccountUser role.

A VPC network selected for the environment does not exist

You can specify a VPC network and a subnet for your Cloud Composer environment when you create it. If you do not specify a VPC network, then the Cloud Composer service selects the default VPC and the default subnet for the environment's region and zone.

If the specified VPC network and subnet do not exist, then Cloud Composer outputs the following error:

Errors in: [GKE cluster]; Error messages:
        {"ResourceType":"gcp-types/container-v1:projects.locations.clusters","R
        esourceErrorCode":"400","ResourceErrorMessage":{"code":400,"message":"P
        roject \"<your composer project>\" has no network named \"non-existing-
        vpc\".","status":"INVALID_ARGUMENT","statusMessage":"Bad
        Request","requestPath":"https://container.googleapis.com/
        v1/projects/<your composer
        project>/locations/<zone>/clusters","httpMethod":"POST"}}

Solution: Before creating an environment, make sure that the VPC network and the subnet for your new environment exist.

Quota issues encountered when creating environments in large-scale networks

When creating Cloud Composer environments in large-scale networks, you might encounter the following quota limitations:

  • The maximum number of VPC peerings per single VPC network is reached.
  • The maximum number of primary and secondary subnet IP ranges is reached.
  • The maximum number of forwarding rules in the peering group for Internal TCP/UDP Load Balancing is reached.

Solution: Apply the approach recommended for Cloud Composer in large-scale networks.

What's next