Known issues with Cloud Composer

This page lists known issues with Cloud Composer. Some fixes for these issues are in-progress, and will be available in future versions. Some issues affect older versions, and can be fixed by upgrading your environment.

Non-RFC 1918 address ranges are partially supported for Pods and Services

Cloud Composer depends on GKE to deliver support for non-RFC 1918 addresses for Pods and Services. Currently, only the following list of Non-RFC 1918 ranges is supported in Cloud Composer:

  • 100.64.0.0/10
  • 192.0.0.0/24
  • 192.0.2.0/24
  • 192.88.99.0/24
  • 198.18.0.0/15
  • 198.51.100.0/24
  • 203.0.113.0/24
  • 240.0.0.0/4

Airflow UI does not show tasks logs when DAG Serialization is on in Composer 1.10.2 and Composer 1.10.3

Enabling DAG serialization in environments using Composer versions 1.10.2 and 1.10.3 prevents logs from showing in the Airflow web server. Upgrade to version 1.10.4 (or later) to fix this issue.

GKE Workload Identity is not supported

You cannot turn on Workload Identity for Cloud Composer GKE clusters. As a result, you may see the WORKLOAD_IDENTITY_DISABLED finding in the Security Command Center.

GKE Pod Security Policies are not supported

You cannot turn on GKE Pod Security Policies for Cloud Composer GKE clusters. As a result, you may see the POD_SECURITY_POLICY_DISABLED finding in the Security Command Center.

Environment labels added during an update are not fully propagated to Cloud Composer dependencies

Updated labels are not applied to Cloud Storage buckets, Pub/Sub topics nor the Compute Engine VM. As a workaround, those labels can be applied by hand to the aforementioned resources.

GKE upgrades in the context of CVE-2020-14386 problem

We are working on addressing the vulnerability for all Cloud Composer environments. As part of the fix, all existing Cloud Composer's GKE clusters will get updated to a newer version.

Customers that decide to address the vulnerability immediately, can upgrade Composer GKE Cluster by following these instructions with the following considerations:

Step 1. If you're running a Cloud Composer version earlier than 1.7.2 then upgrade to a newer version of Cloud Composer. If you already have version 1.7.2 or later, please, go to the next point.

Step 2. Upgrade GKE cluster (master and nodes) to the latest 1.15 patch version containing the fix for this vulnerability.

Airflow tasks logs are unavailable in the Airflow web server after upgrading from Airflow 1.9.0 to Airflow 1.10.x

Airflow 1.10.x introduced backwards-incompatible changes to the naming convention for log files. Zone information is now added to the log names for Airflow tasks.

Airflow 1.9.0 stores and expects the log names to be in the following format: BUCKET/logs/DAG/2020-03-30T10:29:06/1.log Airflow 1.10.x stores and expects the log names to be in the following format: BUCKET/logs/DAG/2020-03-30T10:29:06+00:00/1.log

As a result, if you upgrade from Airflow 1.9.0 to Airflow 1.10.x and would like to read the log for a task executed with Airflow 1.9.0, the Airflow Web server will show the following error message: Unable to read remote log from BUCKET/logs/DAG/2020-03-30T10:29:06+00:00/1.log

Workaround: Rename the logs generated by Airflow 1.9.0 in the Cloud Storage bucket using the format: BUCKET/logs/DAG/2020-03-30T10:29:06+00:00/1.log

Cannot create Cloud Composer environments with the organization policy constraints/compute.disableSerialPortLogging enforced

Cloud Composer environment creation will fail if constraints/compute.disableSerialPortLogging is enforced on the target project.

Diagnosis

To determine if you're impacted by this issue, follow this procedure:

Go the GKE menu in Cloud Console. Visit the GKE menu

Then, select your newly created cluster. Check for the following error:

Not all instances running in IGM after 123.45s.
Expect <number of desired instances in IGM>. Current errors:

Constraint constraints/compute.disableSerialPortLogging violated for
project <target project number>.

Workarounds:

  1. Disable the organization policy on the project where the Cloud Composer environment will be created.

    An organization policy can always be disabled at the project level even if the parent resources (organization or folder) has it enabled. See the Customizing policies for boolean constraints page for more details.

  2. Use exclusion filters

    Using an exclusion filter for serial port logs. accomplishes the same goal as the disabling the org policy, as there will be serial console logs in Logging. For more details, see the Exclusion filters page.

Usage of Deployment Manager to manage GCP resources protected by VPC Service Controls

Composer uses Deployment Manager to create components of Cloud Composer environments.

In December 2020, you may have received information that you may need to perform additional VPC Service Controls configuration to be able to use Deployment Manager to manage resources protected by VPC Service Controls.

We would like to clarify that no action is required on your side if you are using Composer and you are not using Deployment Manager directly to manage GCP resources mentioned in the Deployment Manager's announcement.

Cannot delete an environment after its GKE cluster is deleted

If you delete your environment's cluster before the environment itself, then attempts to delete your environment result in the following error:

 Got error "" during CP_DEPLOYMENT_DELETING [Rerunning Task. ]

To delete an environment when its GKE cluster is already deleted:

  1. Open the Deployment Manager page in the Google Cloud Console.

    Open the Deployment Manager page

  2. Find all deployments marked with labels:

    • goog-composer-environment:<environment-name>
    • goog-composer-location:<environment-location>.

    You should see two deployments that are marked with the described labels:

    • A deployment named <environment-location>-<environment-name-prefix>-<hash>-sd
    • A deployment named addons-<uuid>
  3. Manually delete resources that are still listed in these two deployments and exist in the project (for example, Pub/Sub topics and subscriptions). To do so:

    1. Select the deployments.

    2. Click Delete.

    3. Select the Delete 2 deployments and all resources created by them, such as VMs, load balancers and disks option and click Delete all.

    The deletion operation fails, but the leftover resources are deleted.

  4. Delete the deployments using one of these options:

    • In Google Cloud Console, select both deployments again. Click Delete, then select the Delete 2 deployments, but keep resources created by them option.

    • Run a gcloud command to delete the deployments with the ABANDON policy:

      gcloud deployment-manager deployments delete addons-<uuid> \
          --delete-policy=ABANDON
      
      gcloud deployment-manager deployments delete <location>-<env-name-prefix>-<hash>-sd \
          --delete-policy=ABANDON
      
  5. Delete your Cloud Composer environment.