Security best practices

Cloud Composer 3 | Cloud Composer 2 | Cloud Composer 1

Securing your Cloud Composer environment is crucial for protecting sensitive data and preventing unauthorized access. This page outlines key best practices, including recommendations for network security, Identity and Access Management, encryption, and environment configuration management.

To get more information about security features available in Cloud Composer, see Security overview.

Manage environment configuration and DAGs using version control

An example of an Airflow CI/CD pipeline
Figure 1. An example of an Airflow CI/CD pipeline (click to enlarge)

  • Create your environment using Terraform. In this way, you can store environment's configuration as code in a repository. In this way, changes to your environment configuration can be reviewed before they are applied, and you can reduce the number of users who have permissions to change the configuration by assigning roles with less permissions.

  • Deploy DAGs in your environment with a CI/CD pipeline, so that DAG code is retrieved from a repository. In this way, DAGs are reviewed and approved before the changes are merged to the version control system. During the review process, approvers make sure that DAGs meet the security criteria established within their teams. The review step is critical to prevent deployment of DAGs modifying the content of the environment's bucket.

  • In Identity and Access Management, disable direct access to DAGs and environment's configuration to regular users, as detailed in the Identity and Access Management section.

Network security

Identity and Access Management

  • Isolate permissions. Create environment service accounts and use different service accounts for different environments. Assign to these service accounts only permissions that are strictly necessary to operate these environments and perform operations defined in Airflow DAGs that they run.

  • Avoid using service accounts with broad permissions. While it is possible to create an environment that uses an account with broad permissions, such as those granted by the Editor basic role, this creates a risk of DAGs using broader permissions than intended.

  • Don't rely on default service accounts of Google services used by Cloud Composer. It is often impossible to reduce permissions available to these service accounts without also affecting other Google services in your project.

  • Make sure that you are familiar with security considerations for environment's service accounts and understand how this account interacts with permissions and roles that you grant to individual users in your project.

  • Adhere to the principle of least privilege. Grant only the minimum necessary permissions to users. For example, assign IAM roles, so that only administrators can access the environment's bucket , and direct access is disabled for regular users. For example, the Composer User role enables access only to DAG UI and Airflow UI.

  • Enforce Airflow UI Access Control, which allows to reduce visibility in Airflow UI and DAG UI based on user's Airflow role, and can be used to assign DAG-level permissions for individual DAGs.

  • Review regularly. Regularly audit IAM permissions and roles to identify and remove any excessive or unused privileges.

  • Beware of passing and storing sensitive data:

    • Exercise caution when passing storing sensitive data like personally identifiable information or passwords. Where required, use Secret Manager to securely store Airflow connections and Airflow secrets, API keys, passwords, and certificates. Don't store this information in your DAGs or environment variables.

    • Grant IAM permissions to the environment's bucket only to trusted users. Use per-object permissions, if possible. Security considerations for environment's service accounts list several ways in which users with access to the environment's bucket can perform actions on behalf of the environment's service account.

    • Make sure that you are familiar with what data is stored in the snapshots and provide permissions to create environment snapshots and access the bucket where they are stored only to trusted users.

    • All Cloud Composer's external interfaces use encryption by default. When connecting to external products and services, make sure that you use encrypted communication (SSL/TLS).

What's next