Known issues with Cloud Composer

This page lists known issues with Cloud Composer. Some fixes for these issues are in-progress, and will be available in future versions. Some issues affect older versions, and can be fixed by upgrading your environment.

Non-RFC 1918 address ranges are partially supported for Pods and Services

Cloud Composer depends on GKE to deliver support for non-RFC 1918 addresses for Pods and Services. Currently, only the following list of Non-RFC 1918 ranges is supported in Cloud Composer:

  • 100.64.0.0/10
  • 192.0.0.0/24
  • 192.0.2.0/24
  • 192.88.99.0/24
  • 198.18.0.0/15
  • 198.51.100.0/24
  • 203.0.113.0/24
  • 240.0.0.0/4

Airflow UI does not show tasks logs when DAG Serialization is on in Composer 1.10.2 and Composer 1.10.3

Enabling DAG serialization in environments using Composer versions 1.10.2 and 1.10.3 prevents logs from showing in the Airflow web server. Upgrade to version 1.10.4 (or later) to fix this issue.

GKE Workload Identity is not supported

You cannot turn on Workload Identity for Cloud Composer GKE clusters. As a result, you may see the WORKLOAD_IDENTITY_DISABLED finding in the Security Command Center.

GKE Pod Security Policies are not supported

You cannot turn on GKE Pod Security Policies for Cloud Composer GKE clusters. As a result, you may see the POD_SECURITY_POLICY_DISABLED finding in the Security Command Center.

Environment labels added during an update are not fully propagated to Cloud Composer dependencies

Updated labels are not applied to Cloud Storage buckets, Pub/Sub topics nor the Compute Engine VM. As a workaround, those labels can be applied by hand to the aforementioned resources.

GKE upgrades in the context of CVE-2020-14386 problem

We are working on addressing the vulnerability for all Cloud Composer environments. As part of the fix, all existing Cloud Composer's GKE clusters will get updated to a newer version.

Customers that decide to address the vulnerability immediately, can upgrade Composer GKE Cluster by following these instructions with the following considerations:

Step 1. If you're running a Cloud Composer version earlier than 1.7.2 then upgrade to a newer version of Cloud Composer. If you already have version 1.7.2 or later, please, go to the next point.

Step 2. Upgrade GKE cluster (master and nodes) to the latest 1.15 patch version containing the fix for this vulnerability.

Airflow tasks logs are unavailable in the Airflow web server after upgrading from Airflow 1.9.0 to Airflow 1.10.x

Airflow 1.10.x introduced backwards-incompatible changes to the naming convention for log files. Zone information is now added to the log names for Airflow tasks.

Airflow 1.9.0 stores and expects the log names to be in the following format: BUCKET/logs/DAG/2020-03-30T10:29:06/1.log Airflow 1.10.x stores and expects the log names to be in the following format: BUCKET/logs/DAG/2020-03-30T10:29:06+00:00/1.log

As a result, if you upgrade from Airflow 1.9.0 to Airflow 1.10.x and would like to read the log for a task executed with Airflow 1.9.0, the Airflow Web server will show the following error message: Unable to read remote log from BUCKET/logs/DAG/2020-03-30T10:29:06+00:00/1.log

Workaround: Rename the logs generated by Airflow 1.9.0 in the Cloud Storage bucket using the format: BUCKET/logs/DAG/2020-03-30T10:29:06+00:00/1.log

Airflow RBAC UI is not supported

Cloud Composer doesn't support Airflow RBAC UI. Setting rbac=TRUE in the webserver section of the Airflow configuration is not supported and can cause stability issues in your Cloud Composer environment.

Cannot create Cloud Composer environments with the organization policy constraints/compute.disableSerialPortLogging enforced

Cloud Composer environment creation will fail if constraints/compute.disableSerialPortLogging is enforced on the target project.

Diagnosis

To determine if you're impacted by this issue, follow this procedure:

Go the GKE menu in Cloud Console. Visit the GKE menu

Then, select your newly created cluster. Check for the following error:

Not all instances running in IGM after 123.45s.
Expect <number of desired instances in IGM>. Current errors:

Constraint constraints/compute.disableSerialPortLogging violated for
project <target project number>.

Workarounds:

  1. Disable the organization policy on the project where the Cloud Composer environment will be created.

    An organization policy can always be disabled at the project level even if the parent resources (organization or folder) has it enabled. See the Customizing policies for boolean constraints page for more details.

  2. Use exclusion filters

    Using an exclusion filter for serial port logs. accomplishes the same goal as the disabling the org policy, as there will be serial console logs in Logging. For more details, see the Exclusion filters page.