Release notes

These release notes apply to the Cloud Composer service. You can periodically check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.

To get the latest product updates delivered to you, add the URL of this page to your feed reader.

February 6, 2019

  • New versions of Cloud Composer images: composer-1.5.0-airflow-1.9.0, composer-1.5.0-airflow-1.10.0, and composer-1.5.0-airflow-1.10.1
  • Increased the disk size for the Apache Airflow webserver from 10 to 20 GB.
  • We now install Airflow dependencies of "mysql" instead of "devel".

    If your DAG uses dependencies that are available only in "devel", add the dependencies to your DAG as PyPI dependencies.

  • Backported Airflow upstream fix for the broken GCP connection in the DataflowJavaOperator and DataflowPythonOperator AIRFLOW-2009 in all Cloud Composer managed Airflow versions.
  • Backported Airflow upstream fixes for CVE-2018-20244: Stored XSS in Apache Airflow in all Cloud Composer managed Airflow versions (1.9.0, 1.10.0, 1.10.1).

    Due to this vulnerability, we recommend that you upgrade to the latest Cloud Composer version.

January 10, 2019

Added a Kubernetes pod liveness checker for the airflow-sqlproxy service to recover from the cloudsqlproxy process error.

New versions of Cloud Composer images: composer-1.4.2-airflow-1.9.0 and composer-1.4.2-airflow-1.10.0.

  • Fixed the issue where max_active_runs and concurrency are not updated in the Airflow web UI DAG Details page when configurations are updated.
  • Fixed the failed KubernetesPodOperator and GkePodOperator when a task a runs longer than 1 hour due to authentication issues in the Kuberentes client.

December 19, 2018

Cloud Composer is now available in the Northern Virginia (us-east4) region.

December 17, 2018

  • New versions of Cloud Composer images: composer-1.4.1-airflow-1.9.0 and composer-1.4.1-airflow-1.10.0.
  • The Airflow workers restart when a Celery concurrency configuration change is requested so that the change takes effect automatically.
  • The logs/ and data/ directories in Cloud Storage are mounted with file mode 755, enabling binaries to be placed in these directories and executed from Airflow.
  • The GCP_PROJECT environment variable is set to the Google Cloud Platform project ID in the Airflow webserver.

  • Fixed the broken Airflow webserver when DAG parsing times out on the webserver side.
  • Pinned the Airflow base Docker image tag for initial environment creation and future PyPI dependency installations.

November 26, 2018

GA launch of Python 3 support. Currently, the v1 API and the GCP Console support GA Python 3 environment creation.

New versions of Cloud Composer images: composer-1.4.0-airflow-1.9.0 and composer-1.4.0-airflow-1.10.0.

  • Fixed the Stackdriver logging severity level for airflow-sqlproxy and improved the labeling of logs.
  • Fixed boundless HTTP header growth in the set_user_agent googleapiclient methods in 1.10.0 images.

October 24, 2018

  • Multi-version Apache Airflow support is now in Beta. You can choose from multiple versions of Airflow when creating a new Cloud Composer environment. Multiple versions of Airflow can run in the same GCP project simultaneously.
  • Cloud Composer is now available in the Mumbai (asia-south1) region.
  • The Cloud Composer now displays in the top-right banner of the Airflow webserver UI.

New versions of Cloud Composer images: composer-1.3.0-airflow-1.9.0 and composer-1.3.0-airflow-1.10.0.

October 2, 2018

Support for Python 3 is now in Beta You can create Python 3 Cloud Composer environments and run Python 2 and Python 3 environments in the same GCP project simultaneously. Currently, the v1beta1 API and GCP Console support Python 3 environment creation.

For composer-1.2.0 environments, some commands in Python 2 Composer environments might run under the Python 3 runtime. Operators should explicitly specify python2 to ensure that commands run under Python 2. For more info, see PEP 394.

  • New version of the Cloud Composer image: composer-1.2.0-airflow-1.9.0.
  • To reduce potential scheduling delays due to DAG processing, the scheduler restart behavior is now time based (every 600 seconds), not run based.

September 17, 2018

  • New version of the Cloud Composer image: composer-1.1.1-airflow-1.9.0.
  • The following Airflow configurations are now updatable: core-dag_concurrency, core-parallelism, core-max_active_runs_per_dag, and scheduler-max_threads.

August 17, 2018

  • Support for GKE Shared VPC is now in Beta. For information about Cloud Composer configuration, see Creating Environments.
  • Added the following Airflow updates:
    • Backported network, subnetwork, and tags support in DataprocClusterCreateOperator to create Cloud Dataproc clusters in user-specified subnetworks.
    • Backported GKEPodOperator to enable launching Kubernetes pods in GKE clusters that are accessible from GCP connections.

  • New version of the Cloud Composer image: composer-1.1.0-airflow-1.9.0.
  • Cloud Composer environments running composer-1.0.0 or later now have GKE cluster auto-upgrade enabled by default.
  • Introduced additional liveness monitoring for the Cloud Composer Stackdriver fluentd agent.

Fixed the following Airflow issues:

  • Fixed the issue where the DataFlowJavaOperator and the DataFlowPythonOperator might run continually due to a job name mismatch.
  • Fixed a rare scheduler bug where the Airflow scheduler freezes by restarting the scheduler if its Stackdriver logs are old.

July 19, 2018

  • GA launch of Cloud Composer.
  • Cloud Composer is now available in the following regions:
    • asia-northeast1
    • us-east1
  • Revamped Stackdriver Logging as follows:
    • Stackdriver Logging now based on fluentd and now includes composer-agent, airflow-redis, and airflow-sqlproxy logs.
    • Exposed the Airflow worker and scheduler container logs in Stackdriver Logging, including exceptions and stderr.
    • Disabled Logging for Google Kubernetes Engine to avoid duplicate logging in Stackdriver Logging.
    • The log format for some raw Airflow logs stored in Cloud Storage now include the delimiter string, dag-id, task-id, and execution date.
  • Added new Cloud Identity and Access Management roles.
    • The composer.environmentAndStorageObjectAdmin and composer.environmentAndStorageObjectViewer roles provide users the permissions necessary to access environments and objects in Cloud Storage buckets.
    • The composer.worker role provides service accounts the permissions necessary to run a Cloud Composer VM.
  • Added the following Airflow updates:
    • Backported the KubernetesPodOperator to enable launching Docker containers in Cloud Composer.
    • Added PARQUET source format support in bigquery_hook.
    • Backported the JDBC/DBAPI autocommit issue fix.
    • Fixed the Airflow documentation link in the Airflow web interface.

Fixed the following Airflow issues:

  • Fixed the issue where Airflow tasks were stuck in queued or running states for extended periods of time before failing.
  • Fixed the Airflow web interface crashlooping issue that resulted from statsd exceptions.
  • Revised the Airflow web interface message when the TaskInstance RUN button is clicked. Note that the task instance restarts automatically if the DAG is running and its dependencies are met.

June 27, 2018

  • New version of the Cloud Composer image: composer-0.5.3-airflow-1.9.0.
  • Added requirement to include https://www.googleapis.com/auth/cloud-platform when specifying OAuth scopes during environment creation.
  • Improved health checking on Airflow to restart scheduler and worker pods if scheduled tasks are not queued or executed.
  • Improved messaging for create and update environment failures.

June 15, 2018

Audit logging is now available in Beta.

  • New version of the Cloud Composer image: composer-0.5.2-airflow-1.9.0.
  • Added OAuth scope configuration on the Google Cloud Platform Console Environment creation page
  • Increased the minimum disk size from 10 GB to 20 GB.
  • Improved error reporting.
  • Improved stability of the CeleryExecutor Redis message backend.

Fixed the BAD REQUEST error message for environment creation or deletion failures to provide information about the failures.

May 17, 2018

New version of the Cloud Composer image: composer-0.5.1-airflow-1.9.0.

  • Fixed permanent environment DELETE error that occurred when the Google Kubernetes Engine cluster is deleted or that occurred when in an error state prior to environment deletion.
  • Removed inapplicable "Failed to update environment from json" error message from worker/scheduler logs.

May 1, 2018

Beta launch of Cloud Composer (composer-0.5.0-airflow-1.9.0). This release includes Apache Airflow 1.9.0.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Composer