These release notes apply to the Cloud Composer service. You can periodically check this page for announcements about new or updated features, bug fixes, known issues, and deprecated functionality.
To get the latest product updates delivered to you, add the URL of this page to your
feed
reader, or add the feed URL directly: https://cloud.google.com/feeds/cloud-composer-release-notes.xml
November 25, 2019
- New versions of Cloud Composer images:
composer-1.8.2-airflow-1.9.0
,composer-1.8.2-airflow-1.10.1
,composer-1.8.2-airflow-1.10.2
, andcomposer-1.8.2-airflow-1.10.3
. The default iscomposer-1.8.2-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- Fixed an issue with triggering DAGs from the Web UI while DAG serialization is turned on (for Airflow 1.10.3).
- CloudSQL Proxy should now use less CPU power.
November 18, 2019
- New versions of Cloud Composer images:
composer-1.8.1-airflow-1.9.0
,composer-1.8.1-airflow-1.10.1
,composer-1.8.1-airflow-1.10.2
, andcomposer-1.8.1-airflow-1.10.3
. The default iscomposer-1.8.1-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- Fixed an issue that prevented upgrades from Airflow 1.10.2 to 1.10.3.
- Fixed an issue with triggering DAGs from the Web UI while DAG serialization is turned on.
October 31, 2019
Cloud Composer is now available in Frankfurt (europe-west3
).
October 30, 2019
- New versions of Cloud Composer images:
composer-1.8.0-airflow-1.9.0
,composer-1.8.0-airflow-1.10.1
,composer-1.8.0-airflow-1.10.2
, andcomposer-1.8.0-airflow-1.10.3
. The default iscomposer-1.8.0-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- Added support for Apache Airflow 1.10.3.
- Fixed a DAG serialization issue in Airflow 1.10.2.
- Fixed an issue with domain restricted sharing support.
October 18, 2019
- New versions of Cloud Composer images:
composer-1.7.9-airflow-1.9.0
,composer-1.7.9-airflow-1.10.1
, andcomposer-1.7.9-airflow-1.10.2
. The default iscomposer-1.7.9-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- Bucket Policy Only is now supported, and you will no longer need to disable it during Cloud Composer environment creation.
- Support for Domain restricted sharing is now in Beta. See Creating environments to learn how to enable this feature.
- Improved automatic zone selection during environment creation.
- Fixed an issue with Private IP setup during environment creation.
October 08, 2019
- New versions of Cloud Composer images:
composer-1.7.7-airflow-1.9.0
,composer-1.7.7-airflow-1.10.1
, andcomposer-1.7.7-airflow-1.10.2
. The default iscomposer-1.7.7-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- The ODBC Linux driver (
unixodbc-dev
) is now included by default in the Cloud Composer images. jsonschema
is now available as a Python dependency for DAG serialization.- Composer now ensures that you can only change the Airflow config
[core] store_serialized_dags
one way (from false to true). - Added default Airflow config and environment variables for DAG serialization.
- Environment update timeout has been extended to facilitate migrating large databases.
- New environments will use regional buckets instead of multi-regional. Previously-created environments will continue to use multi-regional buckets.
- Fixed an issue that sporadically caused Cloud Storage bucket synchronization to hang.
- Fixed an issue where service account permission errors would overwrite other errors.
- Fixed an issue that caused extraneous disk usage in the Airflow webserver.
September 20, 2019
Cloud Composer is now available in Zurich (europe-west6
).
September 12, 2019
- The ODBC Linux driver (
unixodbc-dev
) is now included by default in the Cloud Composer images. jsonschema
is now available as a Python dependency for DAG serialization.
- New versions of Cloud Composer images:
composer-1.7.6-airflow-1.9.0
,composer-1.7.6-airflow-1.10.1
, andcomposer-1.7.6-airflow-1.10.2
. The default iscomposer-1.7.6-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- New environments will use regional buckets instead of multi-regional. Previously-created environments will continue to use multi-regional buckets.
- Fixed an issue that sporadically caused Cloud Storage bucket synchronization to hang.
- Fixed an issue where service account permission errors would overwrite other errors.
- Fixed an issue that caused extraneous disk usage in the Airflow webserver.
August 30, 2019
Cloud Composer is now available in Sydney (australia-southeast1
) and Montreal (northamerica-northeast1
).
August 28, 2019
New versions of Cloud Composer images: composer-1.7.5-airflow-1.9.0
, composer-1.7.5-airflow-1.10.1
, and composer-1.7.5-airflow-1.10.2
. The default is composer-1.7.5-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- Fixed unwanted error output from scheduler pod in Private IP Composer environments.
- Improved reliability of Private IP environment creation in crowded IP address spaces.
- Fixed an issue with asynchronous DAG loading when handling Unicode strings.
- Fixed an issue that caused some environment deletion operations to fail.
- Fixed an issue that sometimes prevented the liveness DAG from running when there were failed scheduler pods.
- Improved error messages when corrupted Cloud Storage bucket permissions prevent operations from succeeding.
- Reduced the latency of failed Python package updates when invalid packages are selected.
- Backported AIRFLOW-4015 to support the
GET dag_runs
endpoint in Airflow 1.10.2. - Fixed an issue that caused image version upgrades to fail when the Airflow database was too large.
August 22, 2019
New versions of Cloud Composer images: composer-1.7.4-airflow-1.9.0
, composer-1.7.4-airflow-1.10.1
, and composer-1.7.4-airflow-1.10.2
. The default is composer-1.7.4-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- Fixed unwanted error output from scheduler pod in Private IP Composer environments.
- Improved reliability of Private IP environment creation in crowded IP address spaces.
- Fixed an issue with asynchronous DAG loading when handling Unicode strings.
- Fixed an issue that caused some environment deletion operations to fail.
- Fixed an issue that sometimes prevented the liveness DAG from running when there were failed scheduler pods.
- Improved error messages when corrupted Cloud Storage bucket permissions prevent operations from succeeding.
- Reduced the latency of failed Python package updates when invalid packages are selected.
- Backported AIRFLOW-4015 to support the
GET dag_runs
endpoint in Airflow 1.10.2. - Fixed an issue that caused image version upgrades to fail when the Airflow database was too large.
August 14, 2019
New versions of Cloud Composer images: composer-1.7.3-airflow-1.9.0
, composer-1.7.3-airflow-1.10.1
, and composer-1.7.3-airflow-1.10.2
. The default is composer-1.7.3-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
This release contains only internal reliability improvements.
June 21, 2019
Cloud Composer is now available in London (europe-west2
).
June 14, 2019
- New versions of Cloud Composer images:
composer-1.7.2-airflow-1.9.0
,composer-1.7.2-airflow-1.10.1
, andcomposer-1.7.2-airflow-1.10.2
. The default iscomposer-1.7.2-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release. - The Airflow configuration
webserver-workers
is now updatable. Available only in Airflow 1.10.2 and later.
- Fixed an asynchronous DAG loading issue that prevented the Airflow web server from loading some DAGs. Available only in
composer-1.7.2-airflow-1.10.2
versions and later. - Fixed an issue that caused image version upgrades to fail when migrating particularly large Airflow databases.
- Fixed an issue that sometimes prevented data plane monitoring metrics from being reported in Stackdriver.
- Fixed an issue that caused PyPI package installation to sometimes time out and fail.
May 28, 2019
To display DAGs faster and reduce potential downtime due to heavy DAG processing, Cloud Composer now supports asynchronous DAG loading in the Airflow web server. To enable, set the following Airflow configurations: [webserver]async_dagbag_loader=True
and [webserver]worker_refresh_interval=3600
. Available only in composer-1.7.1-Airflow-1.10.2
versions and later.
New versions of Cloud Composer images: composer-1.7.1-airflow-1.9.0
, composer-1.7.1-airflow-1.10.1
, and composer-1.7.1-airflow-1.10.2
. The default is composer-1.7.1-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
- Backported AIRFLOW-3143 to support auto-zone in DataprocClusterCreateOperator. Available only in
composer-1.7.1-airflow-1.10.2
versions and later. - Added back the list of blocked Airflow configurations for Airflow 1.10.2.
May 16, 2019
- Cloud Composer now supports Private Python package repositories. Available only in
composer-1.7.0-*
and later versions. - Added support for Apache Airflow 1.10.2.
- Added new health metrics in Stackdriver to monitor your environment:
composer.googleapis.com/environment/healthy
andcomposer.googleapis.com/environment/database_health
- VPC Native support is now in beta.
- Backported AIRFLOW-2747 to support explicit rescheduling of Airflow sensors. Available only in Airflow 1.10.2 versions and later.
- New versions of Cloud Composer images:
composer-1.7.0-airflow-1.9.0
,composer-1.7.0-airflow-1.10.1
, andcomposer-1.7.0-airflow-1.10.2
. The default iscomposer-1.7.0-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release. - To prevent co-locating workers on the same node, improved Airflow worker pods scheduling.
- When creating a Private IP Cloud Composer environment, the GKE master IP CIDR block no longer needs to be specified.
Starting from
composer-1.7.0-*
, Apache Airflow version 1.10.0 is no longer available for environment creation or in-place upgrade.
- Added back the
ping
command-line utility to the default Cloud Composer managed Docker images. - Backported AIRFLOW-2715 AIRFLOW-2715 to fix the
DataflowTemplateOperator
region support issue. Available only in Airflow 1.10.2 versions and later. - Fixed an issue that prevented in-place upgrades when the environment service account had only the
roles/composer.worker
role. - Backported fixes for CVE-2019-0216 and CVE-2019-0229.
- Fixed an issue that caused some environment deletion operations to fail.
- Fixed an issue that could cause Cloud Composer workloads to leak into the node pools added for
KubernetesPodOperator
workloads. - Fixed an issue in which the
airflow_db
connection did not work correctly for some environments. - Fixed an issue that prevented upgrading a Private IP Cloud Composer environment.
April 04, 2019
Stackdriver metrics for Cloud Composer is in beta. You can now use Stackdriver Monitoring to understand the performance and health of your Cloud Composer environments and examine Airflow metrics. Stackdriver is available for composer-1.6.0-airflow-1.9.0
and later versions.
New versions of Cloud Composer images: composer-1.6.1-airflow-1.9.0
, composer-1.6.1-airflow-1.10.0
, and composer-1.6.1-airflow-1.10.1
. The default is composer-1.6.1-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
Stackdriver Monitoring: In rare cases, a TRANSIENT_FAILURE
connection error occurs when Cloud Composer writes metrics to Stackdriver Monitoring. Typically, the automatic retry is successful, but occasionally, the failure state persists for an extended period of time.
April 03, 2019
- Private IP Cloud Composer environment is in beta. To isolate your workflows from the public internet, you can now assign only private IP (RFC 1918) addresses to the managed Google Kubernetes Engine and Cloud SQL VMs in your Cloud Composer environment.
- Image version upgrade is in beta. You can now perform an in-place upgrade on the Airflow version or Cloud Composer version that your environment runs.
New versions of Cloud Composer images: composer-1.6.0-airflow-1.9.0
, composer-1.6.0-airflow-1.10.0
, and composer-1.6.0-airflow-1.10.1
. The default is composer-1.6.0-airflow-1.9.0
. Upgrade your Cloud SDK to use features in this release.
Increased the operation timeout from 20 to 30 minutes to fix the sporadic, premature PyPI dependency update timeout issue.
- Currently, you cannot perform an image version upgrade.
- Currently, you cannot simultaneously create or delete two Cloud Composer environments on the same subnetwork if one of the two environments is a Private IP Cloud Composer environment.
- You must wait until the first Cloud Composer environment is created before creating the second environment to avoid environment creation failures.
- You must wait until the first Cloud Composer environment is deleted before deleting the second environment to avoid a GKE cluster resource leak. If the leak occurs, manually delete the GKE cluster for the Cloud Composer environment.
airflow_db
connection: The airflow_db
connection works only for the Airflow webserver. Workarounds are as follows:
- Update the
airflow_db
connection's Host component toairflow-sqlproxy-service.default
. This update breaks the ability to use the connection in the Airflow webserver's Ad Hoc Query page but is preferred for DAGs that rely on theairflow_db
connection. - Create a new connection that mirrors the
airflow_db
connection but usesairflow-sqlproxy-service.default
for the Host.
New versions of Cloud Composer images: composer-1.4.2-airflow-1.9.0
and composer-1.4.2-airflow-1.10.0
.
- Fixed the issue where
max_active_runs
andconcurrency
are not updated in the Airflow web UI DAG Details page when configurations are updated. - Fixed the failed
KubernetesPodOperator
andGkePodOperator
when a task a runs longer than 1 hour due to authentication issues in the Kuberentes client.
March 20, 2019
You can now configure Shared VPC in the GCP Console.
March 11, 2019
You can now access the Airflow web interface logs in Stackdriver Logging for Cloud Composer under airflow-webserver
.
- New versions of Cloud Composer images:
composer-1.5.2-airflow-1.9.0
,composer-1.5.2-airflow-1.10.0
, andcomposer-1.5.2-airflow-1.10.1
. The default iscomposer-1.5.2-airflow-1.9.0
. - For the Cloud SQL instance for Cloud Composer, the
wait_timeout
andinteractive_timeout
are reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock. - Added back the openssh-client package to the default Cloud Composer managed docker images.
- Backported the Apache Airflow fix for the SubDag failed by scheduler deadlock issue.
- Fixed an issue where a DAG backfill could not be disabled on a per-DAG basis.
March 07, 2019
You can now access the Airflow web interface logs in Stackdriver Logging for Cloud Composer under airflow-webserver
.
- New versions of Cloud Composer images:
composer-1.5.1-airflow-1.9.0
,composer-1.5.1-airflow-1.10.0
, andcomposer-1.5.1-airflow-1.10.1
. The default iscomposer-1.5.1-airflow-1.9.0
. - For the Cloud SQL instance for Cloud Composer, the
wait_timeout
andinteractive_timeout
are reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock. - Added back the openssh-client package to the default Cloud Composer managed docker images.
- Backported the Apache Airflow fix for the SubDag failed by scheduler deadlock issue.
- Fixed an issue where a DAG backfill could not be disabled on a per-DAG basis.
February 06, 2019
- Multi-version Apache Airflow support is now General Availability. You can select the Cloud Composer image version when you create a new environment without enabling Beta feature support.
- You can now view available Cloud Composer image versions in all Cloud Composer interfaces.
- Added support for Apache Airflow 1.10.1.
- New versions of Cloud Composer images:
composer-1.5.0-airflow-1.9.0
,composer-1.5.0-airflow-1.10.0
, andcomposer-1.5.0-airflow-1.10.1
- Increased the disk size for the Apache Airflow webserver from 10 to 20 GB.
We now install Airflow dependencies of "mysql" instead of "devel".
If your DAG uses dependencies that are available only in "devel", add the dependencies to your DAG as PyPI dependencies.
- Backported Airflow upstream fix for the broken GCP connection in the
DataflowJavaOperator
andDataflowPythonOperator
AIRFLOW-2009 in all Cloud Composer managed Airflow versions. Backported Airflow upstream fixes for CVE-2018-20244: Stored XSS in Apache Airflow in all Cloud Composer managed Airflow versions (1.9.0, 1.10.0, 1.10.1).
Due to this vulnerability, we recommend that you upgrade to the latest Cloud Composer version.
January 10, 2019
Added a Kubernetes pod liveness checker for the airflow-sqlproxy
service to recover from the cloudsqlproxy
process error.
December 19, 2018
Cloud Composer is now available in the Northern Virginia (us-east4
) region.
December 17, 2018
- New versions of Cloud Composer images:
composer-1.4.1-airflow-1.9.0
andcomposer-1.4.1-airflow-1.10.0
. - The Airflow workers restart when a Celery concurrency configuration change is requested so that the change takes effect automatically.
- The
logs/
anddata/
directories in Cloud Storage are mounted with file mode 755, enabling binaries to be placed in these directories and executed from Airflow. - The GCP_PROJECT environment variable is set to the Google Cloud Platform project ID in the Airflow webserver.
- Fixed the broken Airflow webserver when DAG parsing times out on the webserver side.
- Pinned the Airflow base Docker image tag for initial environment creation and future PyPI dependency installations.
November 26, 2018
GA launch of Python 3 support. Currently, the v1
API and the GCP Console support GA Python 3 environment creation.
New versions of Cloud Composer images: composer-1.4.0-airflow-1.9.0
and composer-1.4.0-airflow-1.10.0
.
- Fixed the Stackdriver logging severity level for
airflow-sqlproxy
and improved the labeling of logs. - Fixed boundless HTTP header growth in the
set_user_agent googleapiclient
methods in 1.10.0 images.
October 24, 2018
- Multi-version Apache Airflow support is now in Beta. You can choose from multiple versions of Airflow when creating a new Cloud Composer environment. Multiple versions of Airflow can run in the same GCP project simultaneously.
- Cloud Composer is now available in the Mumbai (
asia-south1
) region. - The Cloud Composer now displays in the top-right banner of the Airflow webserver UI.
New versions of Cloud Composer images: composer-1.3.0-airflow-1.9.0
and composer-1.3.0-airflow-1.10.0
.
October 02, 2018
Support for Python 3 is now in Beta You can create Python 3 Cloud Composer environments and run Python 2 and Python 3 environments in the same GCP project simultaneously. Currently, the v1beta1 API and GCP Console support Python 3 environment creation.
For composer-1.2.0 environments, some commands in Python 2 Composer environments might run under the Python 3 runtime. Operators should explicitly specify python2
to ensure that commands run under Python 2. For more info, see PEP 394.
- New version of the Cloud Composer image:
composer-1.2.0-airflow-1.9.0
. - To reduce potential scheduling delays due to DAG processing, the scheduler restart behavior is now time based (every 600 seconds), not run based.
September 17, 2018
- New version of the Cloud Composer image:
composer-1.1.1-airflow-1.9.0
. - The following Airflow configurations are now updatable: core-dag_concurrency, core-parallelism, core-max_active_runs_per_dag, and scheduler-max_threads.
August 17, 2018
- Support for GKE Shared VPC is now in Beta. For information about Cloud Composer configuration, see Creating Environments.
- Added the following Airflow updates:
- Backported network, subnetwork, and tags support in
DataprocClusterCreateOperator
to create Cloud Dataproc clusters in user-specified subnetworks. - Backported
GKEPodOperator
to enable launching Kubernetes pods in GKE clusters that are accessible from GCP connections.
- Backported network, subnetwork, and tags support in
- New version of the Cloud Composer image:
composer-1.1.0-airflow-1.9.0
. - Cloud Composer environments running composer-1.0.0 or later now have GKE cluster auto-upgrade enabled by default.
- Introduced additional liveness monitoring for the Cloud Composer Stackdriver fluentd agent.
Fixed the following Airflow issues:
- Fixed the issue where the
DataFlowJavaOperator
and theDataFlowPythonOperator
might run continually due to a job name mismatch. - Fixed a rare scheduler bug where the Airflow scheduler freezes by restarting the scheduler if its Stackdriver logs are old.
July 19, 2018
- GA launch of Cloud Composer.
- Cloud Composer is now available in the following regions:
asia-northeast1
us-east1
- Revamped Stackdriver Logging as follows:
- Stackdriver Logging now based on fluentd and now includes
composer-agent
,airflow-redis
, andairflow-sqlproxy
logs. - Exposed the Airflow worker and scheduler container logs in Stackdriver Logging, including exceptions and stderr.
- Disabled Logging for Google Kubernetes Engine to avoid duplicate logging in Stackdriver Logging.
- The log format for some raw Airflow logs stored in Cloud Storage now include the delimiter string, dag-id, task-id, and execution date.
- Stackdriver Logging now based on fluentd and now includes
- Added new Cloud Identity and Access Management roles.
- The
composer.environmentAndStorageObjectAdmin
andcomposer.environmentAndStorageObjectViewer
roles provide users the permissions necessary to access environments and objects in Cloud Storage buckets. - The
composer.worker
role provides service accounts the permissions necessary to run a Cloud Composer VM.
- The
- Added the following Airflow updates:
- Backported the
KubernetesPodOperator
to enable launching Docker containers in Cloud Composer. - Added
PARQUET
source format support inbigquery_hook
. - Backported the JDBC/DBAPI autocommit issue fix.
- Fixed the Airflow documentation link in the Airflow web interface.
- Backported the
- New version of the Cloud Composer image:
composer-1.0.0-airflow-1.9.0
. - Cloud Composer environments running composer-1.0.0 or later can now launch Cloud Dataflow jobs in supported Cloud Dataflow regions.
- Added the option to modify some celery configurations.
Fixed the following Airflow issues:
- Fixed the issue where Airflow tasks were stuck in queued or running states for extended periods of time before failing.
- Fixed the Airflow web interface crashlooping issue that resulted from statsd exceptions.
- Revised the Airflow web interface message when the TaskInstance RUN button is clicked. Note that the task instance restarts automatically if the DAG is running and its dependencies are met.
June 27, 2018
- New version of the Cloud Composer image:
composer-0.5.3-airflow-1.9.0
. - Added requirement to include
https://www.googleapis.com/auth/cloud-platform
when specifying OAuth scopes during environment creation. - Improved health checking on Airflow to restart scheduler and worker pods if scheduled tasks are not queued or executed.
- Improved messaging for create and update environment failures.
June 15, 2018
Audit logging is now available in Beta.
- New version of the Cloud Composer image:
composer-0.5.2-airflow-1.9.0
. - Added OAuth scope configuration on the Google Cloud Platform Console Environment creation page
- Increased the minimum disk size from 10 GB to 20 GB.
- Improved error reporting.
- Improved stability of the CeleryExecutor Redis message backend.
Fixed the BAD REQUEST
error message for environment creation or deletion failures to provide information about the failures.
May 17, 2018
New version of the Cloud Composer image: composer-0.5.1-airflow-1.9.0
.
- Fixed permanent environment
DELETE
error that occurred when the Google Kubernetes Engine cluster is deleted or that occurred when in an error state prior to environment deletion. - Removed inapplicable "Failed to update environment from json" error message from worker/scheduler logs.
May 01, 2018
Beta launch of Cloud Composer (composer-0.5.0-airflow-1.9.0
). This release includes Apache Airflow 1.9.0.