Cloud Composer shared responsibility model

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

Running a business-critical application on Cloud Composer requires multiple parties to carry different responsibilities. While not an exhaustive list, this document lists the responsibilities for both Google and the Customer sides.

Google Responsibilities

• Hardening and patching the Cloud Composer environment's components and underlying infrastructure, including Google Kubernetes Engine cluster, Cloud SQL database (that hosts the Airflow database), Pub/Sub, Artifact Registry and other environment elements. In particular, this includes auto-upgrading the underlying infrastructure, including the GKE cluster and Cloud SQL instance of an environment.

  Note: Cloud Composer 1 is in the maintenance mode and new versions of Cloud Composer 1 with security fixes are no longer published. Migrate to Cloud Composer 2 to get the latest version updates with security improvements.

• Protecting access to Cloud Composer environments through incorporating access control provided by IAM, encrypting data at rest by default, providing additional customer-managed storage encryption, encrypting data in transit.

• Providing Google Cloud integrations for Identity and Access Management, Cloud Audit Logs and Cloud Key Management Service.

• Restricting and logging Google administrative access to customers' clusters for contractual support purposes with Access Transparency and Access Approval.

• Publishing information about backward incompatible changes between Cloud Composer and Airflow versions in Cloud Composer Release Notes.

• Keeping Cloud Composer documentation up to date:

  • Providing description of all functionalities provided by Cloud Composer.

  • Providing troubleshooting instructions that help to keep environments in a healthy state.

  • Publishing information about known issues with workarounds (if they exist).

• Resolving critical security incidents related to Cloud Composer environments and Airflow images provided by Cloud Composer (excluding customer-installed Python packages) by delivering new environment versions addressing the incidents.

• Depending on customer's Support Plan, troubleshooting of Cloud Composer environment health issues.

• Maintaining and expanding the functionality of the Cloud Composer Terraform provider.

• Cooperating with the Apache Airflow community to maintain and develop Google Airflow operators.

  Note: Google won't fix or troubleshoot issues in operator providers for third-party services or products.

• Troubleshooting and, if possible, fixing issues in Airflow core functionalities.

Customer responsibilities

• Upgrading to new Cloud Composer and Airflow versions to keep support for the product and to resolve security issues once Cloud Composer service publishes a Cloud Composer version that addresses the issues.

• Maintaining the DAGs code to keep it compatible with the used Airflow version.

• Keeping the environment's GKE cluster configuration intact, particularly including its auto-upgrade feature.

• Maintaining proper permissions in IAM for the environment's service account. Particularly, keeping permissions required by the Cloud Composer Agent and the environment's service account. Maintaining required permission for the CMEK key used for Cloud Composer environment encryption and rotating it according to your needs.

  Caution: We recommend to set up a user-managed service account for Cloud Composer environments that has only the required set of permissions that are necessary to run the environment and perform operations defined in your DAGs. The Composer Worker (composer.worker) role provides this required set of permissions in most cases. Add extra permissions to this service account only when it's necessary for the operation of your DAGs.
  
  Although we recommend against using this approach, if you don't specify an environment's service account, then your Cloud Composer environment uses the default Compute Engine service account. The default Compute Engine service account usually has the Editor basic role, which contains many more permissions than necessary to run Cloud Composer environments and thus creates a risk of DAGs using broader permissions than intended.

• Maintaining proper permissions in IAM for the environment's bucket and Artifact Registry repository where Cloud Composer's component images are stored.

  Caution: Users with read-write access to the following components:

  • Your environment's bucket
  • Artifact Registry repositories with container images used by: Airflow components, GKEPodOperator, or GKEStartPodOperator

  can deploy their own versions of DAGs or container images to an environment even without explicit Cloud Composer-related permissions. These DAGs or images can be later executed in your environment with the permissions of the Cloud Composer environment service account.

• Maintaining proper IAM permissions for a service account that performs PyPI packages installations. For more information, see Access control.

  Caution: Users with read-write access to the environment's bucket or those who can initiate PyPI packages installations can initate the process of building images on behalf of a service account which is used to perform such builds. This service account is called the environment's service account that is specified during the environment creation, It can be a user-provided service account, or the default service account.

• Maintaining proper end user permissions in IAM and Airflow UI Access Control configuration.

• Keeping Airflow database size below 16 GB through using the maintenance DAG.

• Resolving all DAG parsing issues before raising support cases to Cloud Customer Care.

• Naming DAGs in a proper way (for example, without using invisible characters like SPACE or TAB in DAG names) so that metrics can be reported correctly for DAGs.

• Upgrade the code of DAGs so that it doesn't use deprecated operators and migrate to their up to date alternatives. Deprecated operators might be removed from Airflow providers, which might impact your plans to upgrade to a later Cloud Composer or Airflow version. The deprecated operators are also not maintained and they must be used 'as is'.

• Configuring proper IAM permissions when using secret backends like Secret Manager so that the environment's service account has access to it.

• Adjusting Cloud Composer environment parameters (such as CPU and memory for Airflow components) and Airflow configurations to meet performance and load expectations of Cloud Composer environments using Cloud Composer optimization guide and environment scaling guide.

• Avoiding removing permissions required by Cloud Composer Agent and environment's service accounts (removing these permissions can lead either to failed management operations or to DAG and task failures).

• Keeping all services and APIs required by Cloud Composer always enabled. These dependencies must have quotas configured at levels required for Cloud Composer.

• Keeping Artifact Registry repositories that host container images used by Cloud Composer environments.

• Following recommendations and best practices for implementing DAGs.

• Diagnosing DAG and task failures using instructions for scheduler troubleshooting, DAG troubleshooting and triggerer troubleshooting.

• Avoiding installing or running additional components in the environment's GKE cluster that interfere with Cloud Composer components and prevent them from functioning correctly.

What's next

• Access control with IAM
• Clean up the Airflow database
• Security overview