Release notes

These release notes apply to the Cloud Composer service. You can periodically check this page for announcements about new or updated features, bug fixes and deprecated functionality.

You can also check Composer's Known Issues page to learn more about known issues and possible workarounds.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, or browse and filter all release notes in the Google Cloud Console.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/composer-release-notes.xml

June 17, 2021

Cloud Composer 1.16.7 release started on June 17, 2021. Get ready for upcoming changes and features as we roll out the new release to all regions. This release is in progress at the moment. Listed changes and features might not be available in some regions yet.

New versions of Cloud Composer images:

  • composer-1.17.0-preview.3-airflow-2.0.1
  • composer-1.16.7-airflow-1.10.15
  • composer-1.16.7-airflow-1.10.14 (default)
  • composer-1.16.7-airflow-1.10.12

GCSfuse version was updated to 0.35.1 (latest release). Cloud Composer uses GCSfuse to sync files between the environment buckets and worker pods. The change improves the stability of the syncing process.

(Airflow 2) Preinstalled packages changed. Removed: google_cloud_build==2.0.0, mock==2.0.0, pbr==5.5.1. Downgraded overrides from 3.1.0 to 2.8.0.

For DAG runs with long-running tasks, task level logs are now periodically updated in the Airflow UI. Before this change, logs were only available in Airflow UI after the task was completed.

It is now possible to create environments with CMEK encryption in projects with enabled domain restricted sharing. Before the fix, an error related to insufficient Cloud Pub/Sub permissions was generated.

(Airflow 2) In the Airflow UI, you can now create connection types from the installed custom Airflow provider packages. Before, these connection types were not available.

Fixed a problem where the Airflow worker health was calculated incorrectly because of leftover queued tasks without DAGs being present in the Airflow database. This led to problems with task execution because Airflow workers were constantly restarted in healthy environments.

Fixed the cause of Liveness probe errored events that appeared in the scheduler and worker pod logs.

June 07, 2021

New versions of Cloud Composer images:

  • composer-1.17.0-preview.2-airflow-2.0.1
  • composer-1.16.6-airflow-1.10.15
  • composer-1.16.6-airflow-1.10.14 (default)
  • composer-1.16.6-airflow-1.10.12

You can now store values for the smtp_password Airflow configuration option in Secret Manager.

Increased the timeout for environment upgrade operations to support upgrades for databases up to 16 GB in size. If an upgrade operation times out and the Airflow database size is more than 10 GB, a warning message about the database size is generated.

Fixed memory issues that occurred while syncing files on machine types with more than 8 vCPUs.

DAG parsing and task processing in Airflow no longer fails because of incorrectly formatted Airflow logs. This happened due to a bug in Airflow log message formatting. Before this fix, errors related to sensor tasks with reschedule intervals shorter than scheduler processing time were not displayed.

(New environments only) Some log messages related to Airflow web server access were previously missing in Cloud Logging. This problem is fixed and these messages now appear in Cloud Logging.

(Available without upgrading) Updating environment labels now correctly overrides previous labels in billing reports.

Cloud Composer 1.10.4 has reached its end of full support period.

May 21, 2021

New versions of Cloud Composer images:

  • composer-1.17.0-preview.1-airflow-2.0.1
  • composer-1.16.5-airflow-1.10.15
  • composer-1.16.5-airflow-1.10.14 (default)
  • composer-1.16.5-airflow-1.10.12

Error messages about PyPI package conflicts now contain links to corresponding cluster build logs.

Cloud Composer 1.10.3 has reached its end of full support period.

May 13, 2021

Preview: Cloud Composer supports Airflow 2. For more information about transferring from environments with Airflow 1 to Airflow 2, see Migrate environments to Airflow 2.

Airflow 2.0.1 is available in Cloud Composer images.

You can now break down costs associated with particular Cloud Composer environments. User labels that you assign to your environments now appear in billing reports.

New versions of Cloud Composer images:

  • composer-1.17.0-preview.0-airflow-2.0.1
  • composer-1.16.4-airflow-1.10.15
  • composer-1.16.4-airflow-1.10.14 (default)
  • composer-1.16.4-airflow-1.10.12

For new Cloud Composer environments with Airflow 2, SMTP configuration properties for Airflow have new default values:

  • smtp_user is set to an empty value by default.
  • smtp_password is set to an empty value by default.
  • smtp_mail_from is set to a default value used by Airflow.

Improved the error message that is generated when the specified service account does not have enough permissions to run Airflow workloads.

Added troubleshooting information to error messages generated on Airflow web server deployment failures.

GKE clusters of new Cloud Composer environments use Container-Optimized OS with Containerd (cos_containerd) image type.

Kerberos client (krb5-user) package is pre-installed in Cloud Composer container images.

Some environment operations that failed because of networking problems are now retried instead of failing.

Database passwords are now redacted in error messages that appear in Composer Agent logs.

Error messages about dependency conflicts that happen when installing Python packages are now correctly reported.

When an environment upgrade fails because of package dependency conflicts, the error message contains detailed information about the conflict.

April 29, 2021

New versions of Cloud Composer images:

  • composer-1.16.3-airflow-1.10.15
  • composer-1.16.3-airflow-1.10.14 (default)
  • composer-1.16.3-airflow-1.10.12

When Airflow configuration is updated, an erroneous log message about a web server update failure no longer appears in logs.

Fixed problems with execution date in environment health monitoring when Airflow uses a custom time zone.

Cloud Composer versions 1.8.3 to 1.10.2 have reached their end of full support period.

April 22, 2021

New versions of Cloud Composer images:

  • composer-1.16.2-airflow-1.10.15
  • composer-1.16.2-airflow-1.10.14 (default)
  • composer-1.16.2-airflow-1.10.12

Airflow 1.10.10 is no longer included in Cloud Composer images.

When a GKE authorization error occurs during an environment operation, the GKE error message is reported and the operation fails immediately.

When an environment operation fails during the installation of PyPI packages, error messages generated by pip are now correctly reported.

When Airflow uses a non-UTC time zone, manually triggered DAGs are executed at correct times now. The monitoring panel displays the correct environment health status.

A deprecation message is now displayed for the xcom_push argument of KubernetesPodOperator.

April 15, 2021

New versions of Cloud Composer images:

  • composer-1.16.1-airflow-1.10.15
  • composer-1.16.1-airflow-1.10.14 (default)
  • composer-1.16.1-airflow-1.10.12
  • composer-1.16.1-airflow-1.10.10

If an environment's service account does not have required permissions for a requested operation, Cloud Composer generates an actionable error message. The operation fails faster in this case.

Fixed a bug that caused environment update and upgrade operations to fail with errors related to GKE cluster endpoints, instead of the actual root cause. This problem affected environments with installed custom PyPI packages.

April 08, 2021

Airflow 1.10.15 is available in Cloud Composer images.

New versions of Cloud Composer images:

  • composer-1.16.0-airflow-1.10.15
  • composer-1.16.0-airflow-1.10.14 (default)
  • composer-1.16.0-airflow-1.10.12
  • composer-1.16.0-airflow-1.10.10

In Airflow 1.10.14, PythonVirtualenvOperator now uses the Python version of the environment when a Python version is not specified.

Environments with already deleted GKE clusters can now be deleted as usual. Deleting such environments no longer requires a workaround.

March 31, 2021

New versions of Cloud Composer images:

  • composer-1.15.2-airflow-1.10.14 (default)
  • composer-1.15.2-airflow-1.10.12
  • composer-1.15.2-airflow-1.10.10

Irrelevant warnings about asynchronous DAG loading parameters no longer show up in the Airflow logs.

Corrected the validation of custom Cloud SQL and Airflow web server IP ranges that are specified during the environment creation. Changed the error code and the message that are returned when a specified CIDR range is not valid.

Fixed an Airflow web UI bug that caused the DAG Tree View page to crash in rare cases.

March 23, 2021

New versions of Cloud Composer images:

  • composer-1.15.1-airflow-1.10.14 (default)
  • composer-1.15.1-airflow-1.10.12
  • composer-1.15.1-airflow-1.10.10

During the environment creation, Cloud Composer checks that there are enough CPUs, according to the Cloud Compute CPU quota for a region. If there are not enough CPUs, the operation does not start.

Removed the "@-@" workload info prefixes from Airflow task logs in the Airflow UI and Cloud Logging. This change is available for Airflow 1.10.14.

For Cloud Composer versions 1.13.2 and later, regional base images are used for Airflow web server and worker-scheduler builds. This improves customized image build times.

Invalid resource names in API requests now cause a 4xx response. The invalid resource name is reported in the error message.

Improved the validation procedure for custom IP ranges that are specified during the environment creation. Changed the error code and the message that are returned when a specified CIDR range is not valid.

Fixed the documentation link in the Airflow UI.

Improved the file synchronization error handling for environments that run under the Domain Restricted Sharing organizational policy.

Improved error handling when creating node pools during upgrade operations. In some cases, the error was not reported when an upgrade operation failed on a timeout.

March 11, 2021

New versions of Cloud Composer images:

  • composer-1.15.0-airflow-1.10.14
  • composer-1.15.0-airflow-1.10.12 (default)
  • composer-1.15.0-airflow-1.10.10

DAG serialization is enabled by default in new environments created for Cloud Composer versions 1.15.0 and later. Upgrading an existing environment to 1.15.0 does not change the existing DAG serialization settings.

When creating new environments, enabling asynchronous DAG loading disables DAG serialization.

Before creating or updating an environment, Cloud Composer checks that required APIs are enabled in a project and that CIDR blocks specified for VPC Native and Private IP are valid. If these requirements are not met, Cloud Composer reports an error and the operation does not start. This change is available only for new Cloud Composer environments.

Environment deletion operations no longer fail when Artifact Registry API is disabled.

Fixed a bug that caused upgrades to fail during an in-cluster build in public IP environments.

Environment creation operations no longer fail in Private IP configurations that use Customer Managed Encryption Keys (CMEK).

The environment creation process now aborts early on any web server deployment failure.

Improved error reporting for web server deployment failures. App Engine errors that occur during the deployment are now marked as web server deployment errors.

March 05, 2021

New versions of Cloud Composer images:

  • composer-1.14.5-airflow-1.10.14
  • composer-1.14.5-airflow-1.10.12 (default)
  • composer-1.14.5-airflow-1.10.10

Improved the logging of Airflow exceptions. Full Python tracebacks for Airflow exceptions are reported and marked as errors in the logs.

Fixed a potential infinite loop in the airflow-monitoring pod. Environment health checks no longer get stuck after certain types of environment update operations.

Fixed the cause of failures when creating Qwiklabs environments.

When creating environments, unmet network requirements for pods and services cause the operation to fail immediately. Previously, the operation failed when a timeout was reached.

Cloud Composer acquires existing environment resources if they are available during an upgrade operation. Before, the operation could fail with the "ALREADY_EXISTS" error in some cases.

Added a precondition check for upgrade operations. This check verifies that GKE control plane can reach GKE nodes. Previously, if there was a networking problem with communication between the control plane and GKE nodes, the operation failed on a timeout.

Fixed a problem with airflow-monitoring not having logs after changing the machine type for GKE Cluster in a Cloud Composer environment.

PyPI packages can now be installed in Cloud Composer versions 1.11.0 and 1.11.1.

March 02, 2021

GA: Support for the Airflow Role-Based Access Control (RBAC) UI is now generally available.

GA: Support for Resource location restrictions and Data Residency is now generally available.

February 25, 2021

New versions of Cloud Composer images:

  • composer-1.14.4-airflow-1.10.14
  • composer-1.14.4-airflow-1.10.12 (default)
  • composer-1.14.4-airflow-1.10.10

When an environment update operation cannot start, an error message that lists possible causes for the error is generated.

Improved the syncing of DAGs and plugins to the Airflow web server. DAG parsing is now less likely to break because of race conditions.

Added FreeTDS system package to Cloud Composer images.

Updated apache-beam package version to 2.24.0 in Airflow 1.10.10 so that Dataflow jobs now correctly create partitioned BigQuery tables. Airflow versions 1.10.12 and 1.10.14 already have apache-beam version 2.27.0 installed.

Upgraded apache-airflow-backport-providers-google package to version 2021.2.5 in Airflow 1.10.12 and 1.10.14. This is potentially a breaking change because the package contains updates of Python Google Cloud libraries. For a list of new operators and for more information about breaking changes, see the 2021.2.5 release notes.

The google-cloud-pubsublite package is installed by default in Composer images for Airflow 1.10.12 and 1.10.14.

Improved the reliability of environment upgrade operations. Added new retrying procedures and enhanced existing ones.

Fixed the cause of several update-related errors.

When an environment deletion operation fails, a correct GKE error is displayed. Before, a different GKE error was displayed in some cases.

The Airflow scheduler liveness checker is now compatible with google-cloud-logging==2.2.0.

Fixed an error when some MsSQL operators were not working with Azure instances.

Fixed an error in GKE cluster builds. In private IP environments, an additional nodepool no longer remains after an update.

February 24, 2021

GA: Support for Customer Managed Encryption Keys (CMEK) is now generally available.

February 16, 2021

GA: Setting and updating machine types for CloudSQL/Web Server is now generally available.

GA: Support for Domain restricted sharing is now generally available.

Cloud Composer 1.14.3 release was rolled back. If you have an environment that was created with a composer-1.14.3-airflow-* image, you can later upgrade it to a newer version.

February 11, 2021

To upgrade environments that are deployed with VPC service controls, you must add Artifact Registry to the service perimeter and configure an additional firewall rule for *.pkg.dev.

For other types of environments, if your firewall configuration does not use the default rules, you might need to add a firewall rule for *.pkg.dev as well.

February 03, 2021

Timeouts for environment upgrade operations are increased.

On a failed environment upgrade operation, the created CloudSQL database is now correctly rolled back.

Create and update operations for environments no longer fail if your account doesn't have the serviceusage.services.get permission.

Fixed SQL operation conflicts that were occurring during environment upgrade operations.

Upgrade operations that might have resulted in a semi-upgraded environment state when the operation timeout was reached are now correctly rolled back and errors are reported.

You can now enable and disable RBAC in environments with installed custom PyPI packages.

New versions of Cloud Composer images:

  • composer-1.14.2-airflow-1.10.14
  • composer-1.14.2-airflow-1.10.12 (default)
  • composer-1.14.2-airflow-1.10.10

January 31, 2021

Preview: You can now configure Cloud Composer to use Artifact Registry instead of Container Registry.

January 28, 2021

The default Airflow version for Composer images changes from 1.10.10 to 1.10.12.

Airflow 1.10.9 is no longer included in Cloud Composer images.

Updated google-auth package version to 1.24.0 in Cloud Composer images so that it works with VPC Service Controls. Earlier versions of google-auth caused multiple DAG execution errors when used with VPC SC.

Updated pyarrow package version to 2.0.0 in Cloud Composer images to fix an error in BigQueryHook and Pandas integration. Earlier versions of pyarrow caused the to_pandas() got an unexpected keyword argument 'timestamp_as_object' error.

New versions of Cloud Composer images:

  • composer-1.14.1-airflow-1.10.14
  • composer-1.14.1-airflow-1.10.12 (default)
  • composer-1.14.1-airflow-1.10.10

January 22, 2021

  • If you run DAGs using Pandas and BigQuery in Composer version 1.14.0 and Airflow version 1.10.14, you must update the pyarrow PyPI package to version 1.0.0 or higher and apache-beam to version 2.27.0 or higher.
  • New versions of Cloud Composer images: composer-1.14.0-airflow-1.10.9, composer-1.14.0-airflow-1.10.10, composer-1.14.0-airflow-1.10.12, and composer-1.14.0-airflow-1.10.14. The default is composer-1.14.0-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
  • The max-pods-per-node parameter configures the maximum number of pods per node in the GKE cluster. You can set this parameter when you create a new environment. This feature is available in Google Cloud SDK and Beta API.
  • You can now specify maintenance windows for your environments. GKE cluster and SQL database are updated only during the specified period. This feature is available in Google Cloud SDK and Beta API.
  • The maximum number of pods per node in the GKE cluster changes from 100 to 32. This change affects all newly created environments. You can use the max-pods-per-node parameter when creating an environment to increase or decrease the number of pods.

January 13, 2021

  • Preview: You can now restart the Airflow web server using the command gcloud beta composer environments restart-web-server or the Beta API.

January 12, 2021

  • New versions of Cloud Composer images: composer-1.13.4-airflow-1.10.9, composer-1.13.4-airflow-1.10.10, and composer-1.13.4-airflow-1.10.12. The default is composer-1.13.4-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
  • If environment creation succeeds when the requester did not have the iam.serviceAccounts.actAs permission on the service account, Composer will now generate a warning in the audit log entry.
  • Error handling for files synchronization between buckets has been improved.
  • Preview: Added support for the Airflow Role-Based Access Control (RBAC) UI for Airflow version 1.10.10 or newer and Python 3. You can enable the Airflow RBAC UI by setting the [webserver]rbac=True Airflow configuration override.
  • Fixed an issue where connections were inserted into the Airflow database twice.
  • The Airflow UI will now always send requests to HTTPS addresses.

January 05, 2021

  • In Cloud Composer 1.15.0, DAG serialization will be enabled by default when creating new Cloud Composer environments. Starting from this version, if you do not want to use DAG serialization, you must explicitly disable it.

December 16, 2020

  • Preview: A new Logs tab has been added to the Environment details page.

December 14, 2020

  • New versions of Cloud Composer images: composer-1.13.3-airflow-1.10.9, composer-1.13.3-airflow-1.10.10, and composer-1.13.3-airflow-1.10.12. The default is composer-1.13.3-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
  • Composer will now fail faster when the network settings in Private IP environments prohibit the download of publicly stored Python packages.
  • Composer Agent error messages are now more descriptive.
  • Composer will now check whether the Artifact Registry API is enabled during updates (if it is required).

December 09, 2020

December 03, 2020

  • New versions of Cloud Composer images: composer-1.13.2-airflow-1.10.9, composer-1.13.2-airflow-1.10.10, and composer-1.13.2-airflow-1.10.12. The default is composer-1.13.2-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
  • Airflow worker pods are now forced to be scheduled on different nodes. Each node runs only a single Airflow worker pod.
  • You can now set web server network access control using the v1 Composer API.
  • New metrics have been added to monitor web server CPU and memory usage:
    • CPU usage time
    • CPU reserved cores
    • Memory bytes used
    • Memory quota
  • During environment creation and updates, Composer will now verify whether you have chosen a region compliant with any location restriction organization policies. Error reporting has also been improved in cases where location restrictions cause environment updates to fail.
  • Composer versions 1.8.1 and 1.8.2 have been deprecated.

November 23, 2020

  • Cloud Composer is now available in Los Angeles (us-west2).

November 20, 2020

  • Composer version 1.12.5 or newer: File synchronization between buckets in Domain restricted sharing compliant environments has been improved. If your environment was created before November 20, 2020 you must upgrade your environment to access this feature

November 09, 2020

  • New versions of Cloud Composer images: composer-1.13.0-airflow-1.10.6, composer-1.13.0-airflow-1.10.9, composer-1.13.0-airflow-1.10.10, and composer-1.13.0-airflow-1.10.12. The default is composer-1.13.0-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
  • Airflow 1.10.12: api-auth_backend is now set to airflow.api.auth.backend.deny_all by default. To use the experimental REST API, change the api-auth_backend configuration override to airflow.api.auth.backend.default.
  • Airflow 1.10.12 is now available for Cloud Composer.
  • The GKE release channel is set to STABLE for new and upgraded Composer environments.
  • The following Cloud Composer versions have been deprecated:
    • Composer-1.7.2
    • Composer-1.7.3
    • Composer-1.7.4
    • Composer-1.7.5
    • Composer-1.7.6
    • Composer-1.7.7
    • Composer-1.8.0

October 30, 2020

  • New versions of Cloud Composer images: composer-1.12.5-airflow-1.10.6, composer-1.12.5-airflow-1.10.9, and composer-1.12.5-airflow-1.10.10. The default is composer-1.12.5-airflow-1.10.10. Upgrade your Cloud SDK to use features in this release.
  • More detailed error messages are now provided for errors during Python package installation.
  • Fixed an issue where DAG definition files that exceeded 64kb were not working with [core] store_dag_code mode enabled. Now these files can be up to 16 MB.
  • Fixed an issue where updating Airflow configurations with value interpolation caused update requests to timeout; these requests are not supported and will now be rejected.
  • Improved GCSfuse stability to resolve intermittent issues where the mounted directory was unavailable.

October 23, 2020

  • New versions of Cloud Composer images: composer-1.12.4-airflow-1.10.6, composer-1.12.4-airflow-1.10.9, and composer-1.12.4-airflow-1.10.10. The default is composer-1.12.4-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.
  • The following Composer environment database metrics are now available in Cloud Monitoring: CPU usage, CPU cores, CPU utilization, memory usage, memory quota, memory utilization, disk usage, disk quota, disk utilization.
  • Worker liveness checks now also check whether the log syncing process is running.
  • Improved configuration synchronization with the Airflow web server, fixing an issue with loading logs.
  • The Airflow database will no longer disconnect after 30 seconds of query runtime.
  • Worker health checks will no longer overload the Airflow database.

October 15, 2020

  • New versions of Cloud Composer images: composer-1.12.3-airflow-1.10.6, composer-1.12.3-airflow-1.10.9, and composer-1.12.3-airflow-1.10.10. The default is composer-1.12.3-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.
  • Cloud Build logs from the tenant project are now published in the Composer logs. They are available under the log name build-log-webserver.
  • Airflow DAG processor manager logs are now published in the Composer logs. They are available under the log name dag-processor-manager.
  • If an update operation fails, links to the specific Cloud Build log will now be included in the error message.
  • Compatibility with Domain Restricted sharing has been improved. Upgrading your environment to the newest version of Composer can now enable or disable its compatibility with Domain Restricted Sharing based on your organization policy.
  • Setting or updating the machine type of the Airflow web server or Cloud SQL instance in Composer versions that don't support this feature (older than composer-1.7.2) will now return an error instead of failing silently.
  • Environments will now fall back to the in-cluster build when PyPI package installation fails due to Cloud Build unavailability under your VPC Service Controls configuration.
  • Airflow 1.10.10:
    • Fixed an issue with async DAG bag loading.
    • Task instance details will now render properly, even if there are no DAG runs related to the instance.

October 07, 2020

  • New versions of Cloud Composer images: composer-1.12.2-airflow-1.10.6, composer-1.12.2-airflow-1.10.9, and composer-1.12.2-airflow-1.10.10. The default is composer-1.12.2-airflow-1.10.9. Upgrade your Cloud SDK to use features in this release.
  • Added the log entry labels version_id and instance_id to differentiate the logs of different Airflow web server instances.
  • Airflow database upgrade logs are now published in the Composer logs under a separate log name.
  • Cloud Storage syncing logs are now published together in the Composer logs under a separate log name. They can be separated based further on pod_id.
  • Fixed upgrade rollback failures due to a mismatch in API versions error.
  • Improved handling of errors caused by a missing plugins/ directory in the Cloud Storage bucket.
  • Backported an Airflow change that fixes an issue with Airflow 1.10.9 and 1.10.10 that causes MySQL to deadlock on the rendered_task_instance_fields table when using DAG serialization.
  • Fixed a bug that undercharged customers for App Engine storage.
  • Backfilled GKE API versions to make older Composer environments GKE 1.16-proof.
  • Fixed an issue that caused bq load commands for files larger than 100MB to fail with a RedirectMissingLocation error.

September 28, 2020

  • New versions of Cloud Composer images: composer-1.12.1-airflow-1.10.6, composer-1.12.1-airflow-1.10.9, and composer-1.12.1-airflow-1.10.10. The default is composer-1.12.1-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
  • In-cluster image build logs will now appear in Monitoring under separate log names build-log-*, instead of in the Composer Agent logs.
  • You can now set or update machine types for the Airflow web server and Cloud SQL instance for any environment running a Composer version 1.7.2 or newer, regardless of Airflow version.
  • Airflow 1.10.6, 1.10.9, and 1.10.9: An Airflow change has been backported that lowers the network cost of DAG serialization.
  • Airflow 1.10.9: An Airflow change has been backported that improves GKE cluster resource usage, improving overall Airflow performance.

September 17, 2020

  • New versions of Cloud Composer images: composer-1.12.0-airflow-1.10.3, composer-1.12.0-airflow-1.10.6, composer-1.12.0-airflow-1.10.9, and composer-1.12.0-airflow-1.10.10. The default is composer-1.12.0-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
  • Airflow 1.10.10 is now available.
  • You can now securely store Airflow connections and secrets by configuring Composer environments with Secret Manager.
  • All new Environments now use Zonal DNS names.
  • Validation has been added to prevent setting DAG serialization and async DAG bag loading at the same time.
  • Fixed rendering templates in the Airflow UI when async DAG bag loading is on.
  • Fixed an issue with Airflow worker Pod placement on GKE Nodes after environment upgrade and update.
  • Added validation to the v1 API so that environment creation fails quickly if Domain Restricted Sharing is enforced in the project.
  • Environment variables set by the user at the environment creation are now correctly propagated to Pub/Sub topics.
  • A fix for the broken Airflow Sentry integration has been backported to older Composer Airflow versions.
  • Airflow 1.10.9:

    • Fixed an issue where task retries did not work while DAG serialization was turned on.
    • Fixed an issue that broke Airflow database connections when DAG serialization was turned on.
    • Fixed an issue where the default worker concurrency could not be set, causing problems with cluster resources usage.

August 21, 2020

  • New versions of Cloud Composer images: composer-1.11.3-airflow-1.10.3, composer-1.11.3-airflow-1.10.6, and composer-1.11.3-airflow-1.10.9. The default is composer-1.11.3-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
  • The max size for serialized DAGs has been increased to 128 MB.
  • PyPi dependency conflicts will now cause environment updates to fail by default. This can now be changed by setting the environmental variable IGNORE_PYPI_DEPENDENCY_CONFLICTS=True . This is the default behavior For older environments, but you can set IGNORE_PYPI_DEPENDENCY_CONFLICTS=False to force conflicts to fail.
  • The Airflow configuration [core]load_examples can no longer be modified.
  • Domain Restricted Sharing setup with buckets in the tenant project is now applied during environment creation using the Beta API only when it is required.
  • Airflow 1.10.3, 1.10.6 and 1.10.9:

  • Airflow 1.10.9:

    • Fixed an issue that caused BigQuery operators from backported providers to fail when DAG serialization was turned on.
    • The Airflow web server UI will now correctly reload when the plugin folder is modified.
    • Rendered template_fields can now be displayed without accessing DAG files.
  • Airflow 1.10.6: Fixed intermittent issues template rendering when DAG serialization is turned on.

August 10, 2020

  • New versions of Cloud Composer images: composer-1.11.2-airflow-1.10.3, composer-1.11.2-airflow-1.10.6, and composer-1.11.2-airflow-1.10.9. The default is composer-1.11.2-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
  • Airflow 1.10.6 and 1.10.9: You can now specify a location argument when creating a BigQueryCheckOperator to use it in a different region from the Composer environment.
  • Fixed GKE setting incompatibilities that broke environment creation for Composer versions between 1.7.2 and 1.8.3.
  • When DAG serialization is on, plugins and DAGs are no longer synced when the Airflow web server starts up. This fixes web server failures when plugins use custom PyPI packages.
  • Fixed intermittent failures when triggering a DAG from the Airflow Web UI with DAG serialization turned on.
  • Fixed update operations (installing Python dependencies and upgrading environments) for domain-scoped projects.
  • Fixed a broken link to the Airflow documentation in Airflow 1.10.9.

August 03, 2020

  • New versions of Cloud Composer images: composer-1.11.1-airflow-1.10.3, composer-1.11.1-airflow-1.10.6, and composer-1.11.1-airflow-1.10.9. The default is composer-1.11.1-airflow-1.10.6. Upgrade your Cloud SDK to use features in this release.
  • Composer now enforces iam.serviceAccounts.actAs permission checks on the service account specified during Composer environment creation. See Creating environments for details.
  • Private IP environments can now be creating using non-rfc 1918 CGN ranges (100.64.0.0/10)
  • New PyPi packages have been added for Composer version composer-1.11.0-airflow-1.10.6. These make it possible to install apache-airflow-backport-providers-google with no additional package upgrades.
  • The PyPi package google-cloud-datacatalog can now be installed on Composer environments running Airflow 1.10.6 and Python 3.
  • Cloud Composer 1.11.1+: Backport providers are installed by default for Airflow 1.10.6 and 1.10.9.
  • You can now use the label.worker_id filter in Cloud Monitoring logs to see logs sent out of a specific Airflow worker Pod.
  • With the Composer Beta API, you can now upgrade an environment to any of the three latest Composer versions (instead of just the latest).
  • You can now modify these previously blocked Airflow configurations: [scheduler] scheduler_heartbeat_sec, [scheduler] job_heartbeat_sec, [scheduler] run_duration
  • A more informative error message was added for environment creation failures caused by issues with Cloud SQL instance creation.
  • Improved error reporting has been added for update operations that change the web server image in cases where the error occurs before the new web server image is created.
  • The Airflow-worker liveness check has been changed so that a task just added to a queue will not fire an alert.
  • Reduced the amount of non-informative logs thrown by the environment in Composer 1.10.6.
  • Improved the syncing procedure for env_var.json in Airflow 1.10.9 (it should no longer throw "missing file:" errors).
  • Airflow-worker and airflow-scheduler will no longer throw "missing env_var.json" errors in Airflow 1.10.6.

July 30, 2020

Cloud Composer is now available in Osaka (asia-northeast2).

July 24, 2020

  • New versions of Cloud Composer images: composer-1.11.0-airflow-1.10.2, composer-1.11.0-airflow-1.10.3, composer-1.11.0-airflow-1.10.6, and composer-1.11.0-airflow-1.10.9. The default is composer-1.11.0-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
  • Airflow 1.10.9 is now supported.
  • Environment upgrades have been enabled for the latest two Composer versions (1.11.0 and 1.10.6).
  • Added a retry feature to the Airflow CeleryExecutor (disabled by default). You can configure the number of times Celery will attempt to execute a task by setting the [celery] max_command_attempts property. The delay between each retry can also be adjusted with [celery] command_retry_wait_duration (default: 5 seconds).
  • New PyPi packages have been added for Composer version composer-1.11.0-airflow-1.10.6. These make it possible to install apache-airflow-backport-providers-google with no additional package upgrades.
  • The PyPi package google-cloud-datacatalog can now be installed on Composer environments running Airflow 1.10.6 and Python 3.
  • Fixed synchronization of environment variables to the web server.
  • Improved error reporting when PyPI package installation fails.
  • Composer versions 1.6.1, 1.7.0, and 1.7.1 are now deprecated.

July 07, 2020

  • New versions of Cloud Composer images: composer-1.10.6-airflow-1.10.2, composer-1.10.6-airflow-1.10.3 and composer-1.10.6-airflow-1.10.6. The default is composer-1.10.6-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.

  • For Airflow 1.10.6 and later: The Airflow config property [celery] pool is now blocked.

  • The [core]sql_alchemy_pool_recycle Airflow setting has been modified to improve SQL connection reliability.

  • Fixed an issue with Airflow 1.10.6 environments where task logs were not visible in the UI when DAG serialization was enabled.
  • It is now possible to upgrade from Composer versions 1.1.1, 1.2.0, 1.3.0, 1.4.0, 1.4.1, 1.4.2, 1.5.0, and 1.5.2 to the newest version.
  • Fixed a misleading red environment health status bar when the [core]logging_level Airflow property is set to DEBUG.

June 30, 2020

Cloud Composer support for VPC Service Controls is now in Beta.

June 24, 2020

  • New versions of Cloud Composer images: composer-1.10.5-airflow-1.10.2, composer-1.10.5-airflow-1.10.3 and composer-1.10.5-airflow-1.10.6. The default is composer-1.10.5-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
  • Composer now uses the Kubernetes v1 API, and is compatible with GKE 1.16
  • An updated haproxy configuration for Composer increases the maximum number of connections to 2000, and changes load balancing to be based on the number of connections. These settings can be configured with environment variables.
  • Error messages for TP_APP_ENGINE_CREATING timeout and RPC delivery issues have been expanded.
  • Airflow Providers can now be installed inside Cloud Composer.
  • Error handling for rendering templates in the Airflow web server UI has been improved.
  • Fixed an issue with rendering task instance details (logs, task instance template, params) in the Airflow web server UI when DAG serialization is enabled.
  • Fixed an issue with DataFlowJavaOperator, so it can now be used with Apache Beam 2.20.
  • Improved error reporting for failing operations.
  • Memory consumption of the gcs-syncd container is now constrained to prevent system instability.

May 31, 2020

Cloud Composer is now available in Seoul (asia-northeast3).

May 26, 2020

  • New versions of Cloud Composer images: composer-1.10.4-airflow-1.10.2, composer-1.10.4-airflow-1.10.3 and composer-1.10.4-airflow-1.10.6. The default is composer-1.10.4-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
  • For Airflow 1.10.6 and later: The Airflow config property [celery] pool is now blocked.
  • Fixed an issue with Airflow 1.10.6 environments where task logs were not visible in the UI when DAG serialization was enabled.

May 15, 2020

  • New versions of Cloud Composer images: composer-1.10.3-airflow-1.10.2, composer-1.10.3-airflow-1.10.3 and composer-1.10.3-airflow-1.10.6. The default is composer-1.10.3-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
  • Resource quota limits have been updated, allowing environment administrators to set quotas with more granularity. The default quotas for read and write operations have also changed; see Cloud Composer resource quotas for details. The old limits are deprecated, but will not be removed from the Cloud Console Quotas page until a future release.
  • The machine type of the Airflow web server will now be preserved during Composer environment updates, including cases like new PyPi module installations, or adding new environment variables.
  • Synchronization of log files between the Airflow scheduler, web server and workers has been improved.
  • More useful error messages have been added for Composer environment upgrade failures.
  • Future change: Airflow 1.10.6 will become the default Airflow version for Composer environments in an upcoming release.
  • Composer version 1.6.1 has been deprecated.

May 08, 2020

Cloud Composer is now available in Hong Kong (asia-east2).

Cloud Composer is now available in Las Vegas (us-west4).

April 27, 2020

  • New versions of Cloud Composer images: composer-1.10.2-airflow-1.10.2, composer-1.10.2-airflow-1.10.3 and composer-1.10.2-airflow-1.10.6. The default is composer-1.10.2-airflow-1.10.3. Upgrade your Cloud SDK to use features in this release.
  • Fixed an issue with the CloudSQL Proxy HealthCheck that caused the Proxy Pod to restart repeatedly.
  • The fluentd spec for in-cluster build log exporting now correctly points to the production fluentd image from cloud-airflow-releaser. This fix is required for Composer to correctly perform in-cluster builds for VPC SC configuration.
  • Adjusted ImageBuilder to fix PyPI package installation issues when using VPC SC.
  • Fixed intermittent issues with airflow-monitoring during the initialization phase.
  • Fixed an issue that caused the Airflow scheduler and worker pods to take ~10 minutes to terminate.
  • Fixed an issue with upgrading the image version and improved error handling during Composer environment upgrades.
  • The oldest supported version of Composer is now composer-1.6.0-airflow-x.x.x

April 17, 2020

Composer version 1.10.1 has been rolled back. If you created an environment with composer-1.10.1-airflow-*, you can retrieve and delete the environment, but not update it. We recommend that you delete the environment and create a new environment with the latest image version. Refer to the March 20, 2020 release notes for default version.

April 10, 2020

Private IP Composer environments are now generally available (GA). See Configuring private IP to learn how to use this feature.

Support for Shared VPC networks is now generally available (GA).

April 05, 2020

Cloud Composer is now available in Salt Lake City (us-west3).

March 31, 2020

The new Composer monitoring dashboard is now in beta.

March 23, 2020

Cloud Composer is now available in Sao Paulo (southamerica-east1).

March 20, 2020

  • New versions of Cloud Composer images: composer-1.10.0-airflow-1.10.2, composer-1.10.0-airflow-1.10.3 and composer-1.10.0-airflow-1.10.6. The default is composer-1.10.0-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
  • Improved Composer logs: Composer Agent adds more detailed logs in Cloud Monitoring to describe the current stage of environment creation and provide better error messages if an operation fails.
  • MySQL passwords are now stored in Kubernetes Secrets instead of the config map.
  • You can now set the core.store_serialized_dags property to False after setting it to True. (Previously it was not possible to switch back.)
  • Composer now uses Pip version 19.0.2 for both Python 2 and Python 3 Airflow environments.
  • Added the Airflow property core.store_dag_code, which lets you see DAG code in the web UI while VPC-SC is enabled.
  • New metrics have been added to the monitoring API. See the full list here.
  • Fixed an issue where Airflow database connection errors did not propagate correctly.
  • Logs from the Composer Agent will now show the correct severity level in Cloud Monitoring.
  • Fixed an issue where network tags added to environments did not propagate to the node pools created during the in-cluster image building.
  • The Composer Agent will now wait for env_vars to be sent before starting the web server.
  • Backported a fix for SQL proxy, which improves the stability of SQL connections inside workers.
  • Added new scripts to handle synchronization of files from Customer Project buckets with tenant project buckets in DRS mode, so that DAG synchronization is not affected by long-running logs synchronization.
  • Fixed an issue that prevented the creation of environments in the same VPC but different regions.
  • Fixed an issue with updating the node count for Composer environments running versions 1.6.0 to 1.8.2.
  • Installing PyPI packages from private repositories in the public Internet (specified in the pip.conf file) now works in private IP Composer environments without having to configure Cloud NAT.

March 09, 2020

You can now control access to the Airflow web server, either allowing access from any IP address (default), or specifying which IP ranges have access. For details, see Creating environments.

February 28, 2020

  • New versions of Cloud Composer images: composer-1.9.2-airflow-1.10.1, composer-1.9.2-airflow-1.10.2, composer-1.9.2-airflow-1.10.3 and composer-1.9.2-airflow-1.10.6. The default is composer-1.9.2-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
  • (Beta) You can now create private IP Cloud Composer environments that are configured for Shared VPC.
  • Fixed an issue with updating Composer environments running versions 1.6.0 to 1.8.2. Updating the node count for an environment is still broken, and will be fixed in a future version.

    • Workaround: To resize the cluster for an environment using version 1.6.0 - 1.8.2, resize the default node pool in the cluster directly through GKE interface, or delete the Composer environment and recreate it with a different node count.
  • Fixed an issue with the Dataflow Python hook for Airflow 1.10.6 that restricted it to using Python 2. Dataflow will now use the same Python version as the Composer environment.

February 25, 2020

  • New versions of Cloud Composer images: composer-1.9.1-airflow-1.10.1, composer-1.9.1-airflow-1.10.2, composer-1.9.1-airflow-1.10.3 and composer-1.9.1-airflow-1.10.6. The default is composer-1.9.1-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
  • Fixed an issue where updating environment variables cleared the stored Fernet key.
  • Fixed an issue with running DAGs via the Airflow 1.10.2 REST API when dag_serialization is turned on.
  • Fixed an issue with environment creation for Composer versions 1.6.0 to 1.8.2 when using GKE version 1.14.
  • Improved the health check functionality for SQLProxy.

February 13, 2020

  • New versions of Cloud Composer images: composer-1.9.0-airflow-1.10.1, composer-1.9.0-airflow-1.10.2, composer-1.9.0-airflow-1.10.3 and composer-1.9.0-airflow-1.10.6. The default is composer-1.9.0-airflow-1.10.2. Upgrade your Cloud SDK to use features in this release.
  • Python 3 is now the default version for new Composer environments.
  • Airflow 1.9.0 is no longer supported for new Composer environments.
  • Fixed BigQuery operators for Python 3 in Airflow 1.10.6.
  • Fixed a bug where some deserialized tasks had no start_date in Airflow 1.10.6.
  • The Fernet Key is now stored in Kubernetes Secrets instead of the Config Map.
  • Workers now wait until they have synced with the dags folder before executing tasks.
  • New items have been added to the Airflow security properties blocklist in airflow.cfg: webserver-auth_backend and scheduler-auth_backend.
  • Fixed a rare bug where Airflow workers tried to execute tasks before the DAG code was synced to the Airflow worker volume.
  • Composer is planning to deprecate support for Airflow 1.10.1.
  • In the near future it will not be possible to create Composer environments for Python 2 from the user interface. You will still be able to create Python 2 environments using gcloud.

January 31, 2020

  • New versions of Cloud Composer images: composer-1.8.4-airflow-1.9.0, composer-1.8.4-airflow-1.10.1, composer-1.8.4-airflow-1.10.2, and composer-1.8.4-airflow-1.10.3. The default is composer-1.8.4-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • Deprecation policy update: It is no longer possible to create environments with image versions older than 1 year. Currently the oldest version allowed is composer-1.4.1.
  • The Python version for Dataflow Operators is no longer hard-coded. Composer users can now use Dataflow Operators with Python 2 and 3. (Airflow 1.10.3 only.)
  • Fixed an issue with App Engine health check errors when creating environments with Composer version 1.7.5 or lower.
  • The airflow_home configuration parameter is no longer added to airflow.cfg if using Airflow version 1.10.3 or greater.
  • Airflow 1.10.2 is now the default version used for new Composer environments when Airflow version is not specified.

December 16, 2019

  • New versions of Cloud Composer images: composer-1.8.3-airflow-1.9.0, composer-1.8.3-airflow-1.10.1, composer-1.8.3-airflow-1.10.2, and composer-1.8.3-airflow-1.10.3. The default is composer-1.8.3-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • Composer now uses OpenAPI to manage components running on GKE clusters. This ensures Composer will be compatible with future versions of GKE.
  • Fixed the formatting for validation errors returned by the projects.locations.environments.create API method.

November 25, 2019

  • New versions of Cloud Composer images: composer-1.8.2-airflow-1.9.0, composer-1.8.2-airflow-1.10.1, composer-1.8.2-airflow-1.10.2, and composer-1.8.2-airflow-1.10.3. The default is composer-1.8.2-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • Fixed an issue with triggering DAGs from the Web UI while DAG serialization is turned on (for Airflow 1.10.3).
  • CloudSQL Proxy should now use less CPU power.

November 18, 2019

  • New versions of Cloud Composer images: composer-1.8.1-airflow-1.9.0, composer-1.8.1-airflow-1.10.1, composer-1.8.1-airflow-1.10.2, and composer-1.8.1-airflow-1.10.3. The default is composer-1.8.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • Fixed an issue that prevented upgrades from Airflow 1.10.2 to 1.10.3.
  • Fixed an issue with triggering DAGs from the Web UI while DAG serialization is turned on.

October 31, 2019

Cloud Composer is now available in Frankfurt (europe-west3).

October 30, 2019

  • New versions of Cloud Composer images: composer-1.8.0-airflow-1.9.0, composer-1.8.0-airflow-1.10.1, composer-1.8.0-airflow-1.10.2, and composer-1.8.0-airflow-1.10.3. The default is composer-1.8.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • Added support for Apache Airflow 1.10.3.
  • Fixed a DAG serialization issue in Airflow 1.10.2.
  • Fixed an issue with domain restricted sharing support.

October 18, 2019

  • New versions of Cloud Composer images: composer-1.7.9-airflow-1.9.0, composer-1.7.9-airflow-1.10.1, and composer-1.7.9-airflow-1.10.2. The default is composer-1.7.9-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • Bucket Policy Only is now supported, and you will no longer need to disable it during Cloud Composer environment creation.
  • Support for Domain restricted sharing is now in Beta. See Creating environments to learn how to enable this feature.
  • Improved automatic zone selection during environment creation.
  • Fixed an issue with Private IP setup during environment creation.

October 08, 2019

  • New versions of Cloud Composer images: composer-1.7.7-airflow-1.9.0, composer-1.7.7-airflow-1.10.1, and composer-1.7.7-airflow-1.10.2. The default is composer-1.7.7-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • The ODBC Linux driver (unixodbc-dev) is now included by default in the Cloud Composer images.
  • jsonschema is now available as a Python dependency for DAG serialization.
  • Composer now ensures that you can only change the Airflow config [core] store_serialized_dags one way (from false to true).
  • Added default Airflow config and environment variables for DAG serialization.
  • Environment update timeout has been extended to facilitate migrating large databases.
  • New environments will use regional buckets instead of multi-regional. Previously-created environments will continue to use multi-regional buckets.
  • Fixed an issue that sporadically caused Cloud Storage bucket synchronization to hang.
  • Fixed an issue where service account permission errors would overwrite other errors.
  • Fixed an issue that caused extraneous disk usage in the Airflow webserver.

September 20, 2019

Cloud Composer is now available in Zurich (europe-west6).

September 12, 2019

  • The ODBC Linux driver (unixodbc-dev) is now included by default in the Cloud Composer images.
  • jsonschema is now available as a Python dependency for DAG serialization.
  • New versions of Cloud Composer images: composer-1.7.6-airflow-1.9.0, composer-1.7.6-airflow-1.10.1, and composer-1.7.6-airflow-1.10.2. The default is composer-1.7.6-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • New environments will use regional buckets instead of multi-regional. Previously-created environments will continue to use multi-regional buckets.
  • Fixed an issue that sporadically caused Cloud Storage bucket synchronization to hang.
  • Fixed an issue where service account permission errors would overwrite other errors.
  • Fixed an issue that caused extraneous disk usage in the Airflow webserver.

August 30, 2019

Cloud Composer is now available in Sydney (australia-southeast1) and Montreal (northamerica-northeast1).

August 28, 2019

New versions of Cloud Composer images: composer-1.7.5-airflow-1.9.0, composer-1.7.5-airflow-1.10.1, and composer-1.7.5-airflow-1.10.2. The default is composer-1.7.5-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

  • Fixed unwanted error output from scheduler pod in Private IP Composer environments.
  • Improved reliability of Private IP environment creation in crowded IP address spaces.
  • Fixed an issue with asynchronous DAG loading when handling Unicode strings.
  • Fixed an issue that caused some environment deletion operations to fail.
  • Fixed an issue that sometimes prevented the liveness DAG from running when there were failed scheduler pods.
  • Improved error messages when corrupted Cloud Storage bucket permissions prevent operations from succeeding.
  • Reduced the latency of failed Python package updates when invalid packages are selected.
  • Backported AIRFLOW-4015 to support the GET dag_runs endpoint in Airflow 1.10.2.
  • Fixed an issue that caused image version upgrades to fail when the Airflow database was too large.

August 22, 2019

New versions of Cloud Composer images: composer-1.7.4-airflow-1.9.0, composer-1.7.4-airflow-1.10.1, and composer-1.7.4-airflow-1.10.2. The default is composer-1.7.4-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

  • Fixed unwanted error output from scheduler pod in Private IP Composer environments.
  • Improved reliability of Private IP environment creation in crowded IP address spaces.
  • Fixed an issue with asynchronous DAG loading when handling Unicode strings.
  • Fixed an issue that caused some environment deletion operations to fail.
  • Fixed an issue that sometimes prevented the liveness DAG from running when there were failed scheduler pods.
  • Improved error messages when corrupted Cloud Storage bucket permissions prevent operations from succeeding.
  • Reduced the latency of failed Python package updates when invalid packages are selected.
  • Backported AIRFLOW-4015 to support the GET dag_runs endpoint in Airflow 1.10.2.
  • Fixed an issue that caused image version upgrades to fail when the Airflow database was too large.

August 14, 2019

New versions of Cloud Composer images: composer-1.7.3-airflow-1.9.0, composer-1.7.3-airflow-1.10.1, and composer-1.7.3-airflow-1.10.2. The default is composer-1.7.3-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

This release contains only internal reliability improvements.

June 21, 2019

Cloud Composer is now available in London (europe-west2).

June 14, 2019

  • New versions of Cloud Composer images: composer-1.7.2-airflow-1.9.0, composer-1.7.2-airflow-1.10.1, and composer-1.7.2-airflow-1.10.2. The default is composer-1.7.2-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • The Airflow configuration webserver-workers is now updatable. Available only in Airflow 1.10.2 and later.
  • Fixed an asynchronous DAG loading issue that prevented the Airflow web server from loading some DAGs. Available only in composer-1.7.2-airflow-1.10.2 versions and later.
  • Fixed an issue that caused image version upgrades to fail when migrating particularly large Airflow databases.
  • Fixed an issue that sometimes prevented data plane monitoring metrics from being reported in Stackdriver.
  • Fixed an issue that caused PyPI package installation to sometimes time out and fail.

May 28, 2019

To display DAGs faster and reduce potential downtime due to heavy DAG processing, Cloud Composer now supports asynchronous DAG loading in the Airflow web server. To enable, set the following Airflow configurations: [webserver]async_dagbag_loader=True and [webserver]worker_refresh_interval=3600. Available only in composer-1.7.1-Airflow-1.10.2 versions and later.

New versions of Cloud Composer images: composer-1.7.1-airflow-1.9.0, composer-1.7.1-airflow-1.10.1, and composer-1.7.1-airflow-1.10.2. The default is composer-1.7.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

  • Backported AIRFLOW-3143 to support auto-zone in DataprocClusterCreateOperator. Available only in composer-1.7.1-airflow-1.10.2 versions and later.
  • Added back the list of blocked Airflow configurations for Airflow 1.10.2.

May 16, 2019

  • Cloud Composer now supports Private Python package repositories. Available only in composer-1.7.0-* and later versions.
  • Added support for Apache Airflow 1.10.2.
  • Added new health metrics in Stackdriver to monitor your environment: composer.googleapis.com/environment/healthy and composer.googleapis.com/environment/database_health
  • VPC Native support is now in beta.
  • Backported AIRFLOW-2747 to support explicit rescheduling of Airflow sensors. Available only in Airflow 1.10.2 versions and later.
  • New versions of Cloud Composer images: composer-1.7.0-airflow-1.9.0, composer-1.7.0-airflow-1.10.1, and composer-1.7.0-airflow-1.10.2. The default is composer-1.7.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.
  • To prevent co-locating workers on the same node, improved Airflow worker pods scheduling.
  • When creating a Private IP Cloud Composer environment, the GKE master IP CIDR block no longer needs to be specified.
  • Starting from composer-1.7.0-*, Apache Airflow version 1.10.0 is no longer available for environment creation or in-place upgrade.

  • Added back the ping command-line utility to the default Cloud Composer managed Docker images.
  • Backported AIRFLOW-2715 AIRFLOW-2715 to fix the DataflowTemplateOperator region support issue. Available only in Airflow 1.10.2 versions and later.
  • Fixed an issue that prevented in-place upgrades when the environment service account had only the roles/composer.worker role.
  • Backported fixes for CVE-2019-0216 and CVE-2019-0229.
  • Fixed an issue that caused some environment deletion operations to fail.
  • Fixed an issue that could cause Cloud Composer workloads to leak into the node pools added for KubernetesPodOperator workloads.
  • Fixed an issue in which the airflow_db connection did not work correctly for some environments.
  • Fixed an issue that prevented upgrading a Private IP Cloud Composer environment.

April 04, 2019

Stackdriver metrics for Cloud Composer is in beta. You can now use Stackdriver Monitoring to understand the performance and health of your Cloud Composer environments and examine Airflow metrics. Stackdriver is available for composer-1.6.0-airflow-1.9.0 and later versions.

New versions of Cloud Composer images: composer-1.6.1-airflow-1.9.0, composer-1.6.1-airflow-1.10.0, and composer-1.6.1-airflow-1.10.1. The default is composer-1.6.1-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

Stackdriver Monitoring: In rare cases, a TRANSIENT_FAILURE connection error occurs when Cloud Composer writes metrics to Stackdriver Monitoring. Typically, the automatic retry is successful, but occasionally, the failure state persists for an extended period of time.

April 03, 2019

  • Private IP Cloud Composer environment is in beta. To isolate your workflows from the public internet, you can now assign only private IP (RFC 1918) addresses to the managed Google Kubernetes Engine and Cloud SQL VMs in your Cloud Composer environment.
  • Image version upgrade is in beta. You can now perform an in-place upgrade on the Airflow version or Cloud Composer version that your environment runs.

New versions of Cloud Composer images: composer-1.6.0-airflow-1.9.0, composer-1.6.0-airflow-1.10.0, and composer-1.6.0-airflow-1.10.1. The default is composer-1.6.0-airflow-1.9.0. Upgrade your Cloud SDK to use features in this release.

Increased the operation timeout from 20 to 30 minutes to fix the sporadic, premature PyPI dependency update timeout issue.

  • Currently, you cannot perform an image version upgrade.
  • Currently, you cannot simultaneously create or delete two Cloud Composer environments on the same subnetwork if one of the two environments is a Private IP Cloud Composer environment.
    • You must wait until the first Cloud Composer environment is created before creating the second environment to avoid environment creation failures.
    • You must wait until the first Cloud Composer environment is deleted before deleting the second environment to avoid a GKE cluster resource leak. If the leak occurs, manually delete the GKE cluster for the Cloud Composer environment.

airflow_db connection: The airflow_db connection works only for the Airflow webserver. Workarounds are as follows:

  • Update the airflow_db connection's Host component to airflow-sqlproxy-service.default. This update breaks the ability to use the connection in the Airflow webserver's Ad Hoc Query page but is preferred for DAGs that rely on the airflow_db connection.
  • Create a new connection that mirrors the airflow_db connection but uses airflow-sqlproxy-service.default for the Host.

New versions of Cloud Composer images: composer-1.4.2-airflow-1.9.0 and composer-1.4.2-airflow-1.10.0.

  • Fixed the issue where max_active_runs and concurrency are not updated in the Airflow web UI DAG Details page when configurations are updated.
  • Fixed the failed KubernetesPodOperator and GkePodOperator when a task a runs longer than 1 hour due to authentication issues in the Kuberentes client.

March 20, 2019

You can now configure Shared VPC in the GCP Console.

March 11, 2019

You can now access the Airflow web interface logs in Stackdriver Logging for Cloud Composer under airflow-webserver.

  • New versions of Cloud Composer images: composer-1.5.2-airflow-1.9.0, composer-1.5.2-airflow-1.10.0, and composer-1.5.2-airflow-1.10.1. The default is composer-1.5.2-airflow-1.9.0.
  • For the Cloud SQL instance for Cloud Composer, the wait_timeout and interactive_timeout are reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock.
  • Added back the openssh-client package to the default Cloud Composer managed docker images.

March 07, 2019

You can now access the Airflow web interface logs in Stackdriver Logging for Cloud Composer under airflow-webserver.

  • New versions of Cloud Composer images: composer-1.5.1-airflow-1.9.0, composer-1.5.1-airflow-1.10.0, and composer-1.5.1-airflow-1.10.1. The default is composer-1.5.1-airflow-1.9.0.
  • For the Cloud SQL instance for Cloud Composer, the wait_timeout and interactive_timeout are reduced to 1800 seconds to enable a faster deadlock recovery if an idle SQL connection holds a database lock.
  • Added back the openssh-client package to the default Cloud Composer managed docker images.

February 06, 2019

  • New versions of Cloud Composer images: composer-1.5.0-airflow-1.9.0, composer-1.5.0-airflow-1.10.0, and composer-1.5.0-airflow-1.10.1
  • Increased the disk size for the Apache Airflow webserver from 10 to 20 GB.
  • We now install Airflow dependencies of "mysql" instead of "devel".

    If your DAG uses dependencies that are available only in "devel", add the dependencies to your DAG as PyPI dependencies.

  • Backported Airflow upstream fix for the broken GCP connection in the DataflowJavaOperator and DataflowPythonOperator AIRFLOW-2009 in all Cloud Composer managed Airflow versions.
  • Backported Airflow upstream fixes for CVE-2018-20244: Stored XSS in Apache Airflow in all Cloud Composer managed Airflow versions (1.9.0, 1.10.0, 1.10.1).

    Due to this vulnerability, we recommend that you upgrade to the latest Cloud Composer version.

January 10, 2019

Added a Kubernetes pod liveness checker for the airflow-sqlproxy service to recover from the cloudsqlproxy process error.

December 19, 2018

Cloud Composer is now available in the Northern Virginia (us-east4) region.

December 17, 2018

  • New versions of Cloud Composer images: composer-1.4.1-airflow-1.9.0 and composer-1.4.1-airflow-1.10.0.
  • The Airflow workers restart when a Celery concurrency configuration change is requested so that the change takes effect automatically.
  • The logs/ and data/ directories in Cloud Storage are mounted with file mode 755, enabling binaries to be placed in these directories and executed from Airflow.
  • The GCP_PROJECT environment variable is set to the Google Cloud Platform project ID in the Airflow webserver.
  • Fixed the broken Airflow webserver when DAG parsing times out on the webserver side.
  • Pinned the Airflow base Docker image tag for initial environment creation and future PyPI dependency installations.

November 26, 2018

GA launch of Python 3 support. Currently, the v1 API and the GCP Console support GA Python 3 environment creation.

New versions of Cloud Composer images: composer-1.4.0-airflow-1.9.0 and composer-1.4.0-airflow-1.10.0.

  • Fixed the Stackdriver logging severity level for airflow-sqlproxy and improved the labeling of logs.
  • Fixed boundless HTTP header growth in the set_user_agent googleapiclient methods in 1.10.0 images.

October 24, 2018

  • Multi-version Apache Airflow support is now in Beta. You can choose from multiple versions of Airflow when creating a new Cloud Composer environment. Multiple versions of Airflow can run in the same GCP project simultaneously.
  • Cloud Composer is now available in the Mumbai (asia-south1) region.
  • The Cloud Composer now displays in the top-right banner of the Airflow webserver UI.

New versions of Cloud Composer images: composer-1.3.0-airflow-1.9.0 and composer-1.3.0-airflow-1.10.0.

October 02, 2018

Support for Python 3 is now in Beta You can create Python 3 Cloud Composer environments and run Python 2 and Python 3 environments in the same GCP project simultaneously. Currently, the v1beta1 API and GCP Console support Python 3 environment creation.

For composer-1.2.0 environments, some commands in Python 2 Composer environments might run under the Python 3 runtime. Operators should explicitly specify python2 to ensure that commands run under Python 2. For more info, see PEP 394.

  • New version of the Cloud Composer image: composer-1.2.0-airflow-1.9.0.
  • To reduce potential scheduling delays due to DAG processing, the scheduler restart behavior is now time based (every 600 seconds), not run based.

September 17, 2018

  • New version of the Cloud Composer image: composer-1.1.1-airflow-1.9.0.
  • The following Airflow configurations are now updatable: core-dag_concurrency, core-parallelism, core-max_active_runs_per_dag, and scheduler-max_threads.

August 17, 2018

  • Support for GKE Shared VPC is now in Beta. For information about Cloud Composer configuration, see Creating Environments.
  • Added the following Airflow updates:
    • Backported network, subnetwork, and tags support in DataprocClusterCreateOperator to create Cloud Dataproc clusters in user-specified subnetworks.
    • Backported GKEPodOperator to enable launching Kubernetes pods in GKE clusters that are accessible from GCP connections.
  • New version of the Cloud Composer image: composer-1.1.0-airflow-1.9.0.
  • Cloud Composer environments running composer-1.0.0 or later now have GKE cluster auto-upgrade enabled by default.
  • Introduced additional liveness monitoring for the Cloud Composer Stackdriver fluentd agent.

Fixed the following Airflow issues:

  • Fixed the issue where the DataFlowJavaOperator and the DataFlowPythonOperator might run continually due to a job name mismatch.
  • Fixed a rare scheduler bug where the Airflow scheduler freezes by restarting the scheduler if its Stackdriver logs are old.

July 19, 2018

  • GA launch of Cloud Composer.
  • Cloud Composer is now available in the following regions:
    • asia-northeast1
    • us-east1
  • Revamped Stackdriver Logging as follows:
    • Stackdriver Logging now based on fluentd and now includes composer-agent, airflow-redis, and airflow-sqlproxy logs.
    • Exposed the Airflow worker and scheduler container logs in Stackdriver Logging, including exceptions and stderr.
    • Disabled Logging for Google Kubernetes Engine to avoid duplicate logging in Stackdriver Logging.
    • The log format for some raw Airflow logs stored in Cloud Storage now include the delimiter string, dag-id, task-id, and execution date.
  • Added new Identity and Access Management roles.
    • The composer.environmentAndStorageObjectAdmin and composer.environmentAndStorageObjectViewer roles provide users the permissions necessary to access environments and objects in Cloud Storage buckets.
    • The composer.worker role provides service accounts the permissions necessary to run a Cloud Composer VM.
  • Added the following Airflow updates:
    • Backported the KubernetesPodOperator to enable launching Docker containers in Cloud Composer.
    • Added PARQUET source format support in bigquery_hook.
    • Backported the JDBC/DBAPI autocommit issue fix.
    • Fixed the Airflow documentation link in the Airflow web interface.

Fixed the following Airflow issues:

  • Fixed the issue where Airflow tasks were stuck in queued or running states for extended periods of time before failing.
  • Fixed the Airflow web interface crashlooping issue that resulted from statsd exceptions.
  • Revised the Airflow web interface message when the TaskInstance RUN button is clicked. Note that the task instance restarts automatically if the DAG is running and its dependencies are met.

June 27, 2018

  • New version of the Cloud Composer image: composer-0.5.3-airflow-1.9.0.
  • Added requirement to include https://www.googleapis.com/auth/cloud-platform when specifying OAuth scopes during environment creation.
  • Improved health checking on Airflow to restart scheduler and worker pods if scheduled tasks are not queued or executed.
  • Improved messaging for create and update environment failures.

June 15, 2018

Audit logging is now available in Beta.

  • New version of the Cloud Composer image: composer-0.5.2-airflow-1.9.0.
  • Added OAuth scope configuration on the Google Cloud Platform Console Environment creation page
  • Increased the minimum disk size from 10 GB to 20 GB.
  • Improved error reporting.
  • Improved stability of the CeleryExecutor Redis message backend.

Fixed the BAD REQUEST error message for environment creation or deletion failures to provide information about the failures.

May 17, 2018

New version of the Cloud Composer image: composer-0.5.1-airflow-1.9.0.

  • Fixed permanent environment DELETE error that occurred when the Google Kubernetes Engine cluster is deleted or that occurred when in an error state prior to environment deletion.
  • Removed inapplicable "Failed to update environment from json" error message from worker/scheduler logs.

May 01, 2018

Beta launch of Cloud Composer (composer-0.5.0-airflow-1.9.0). This release includes Apache Airflow 1.9.0.