Managing Airflow Connections

This page describes how to use Airflow connections to access resources in Google Cloud Platform projects from a Cloud Composer environment.

Fernet key and secured connections

When you create a new environment, Cloud Composer generates a unique, permanent fernet key for the environment and secures connection extras by default. You can view the fernet_key in the Airflow Configuration. For information about how connections are secured, see Securing Connections.

Using the default connections

By default, Cloud Composer configures the following Airflow connections for Google Cloud Platform:

  • bigquery_default
  • google_cloud_default
  • google_cloud_datastore_default
  • google_cloud_storage_default

You can use these connections from your DAGs by using the default connection ID. The following example uses the BigQueryOperator with the default connection.

task_default = bigquery_operator.BigQueryOperator(
    task_id='task_default_connection',
    bql='SELECT 1', use_legacy_sql=False)

You can also specifying the connection ID explicitly when you create the operator.

task_explicit = bigquery_operator.BigQueryOperator(
    task_id='task_explicit_connection',
    bql='SELECT 1', use_legacy_sql=False,
    # Composer creates a 'google_cloud_default' connection by default.
    bigquery_conn_id='google_cloud_default')

Accessing resources in another project

The recommended way to allow your Cloud Composer environment to access resources in GCP projects is by using the default connections and by assigning the appropriate Cloud Identity and Access Management permissions to the service account associated with your environment.

The following sections provide examples for how to allow reads and writes to Cloud Storage buckets in your-storage-project for a Cloud Composer environment deployed in the project ID your-composer-project.

Determining the service account associated with your environment

Console

  1. In the GCP Console, open the Environments page.

    Open the Environments page

  2. In the Name column, click the name of the environment to open its Environment details page.
  3. Note the Service account. This value is an email address, such as service-account-name@your-composer-project.iam.gserviceaccount.com.

gcloud

Enter the following command and replace the VARIABLES with appropriate values:

gcloud composer environments describe ENVIRONMENT_NAME \
    --location LOCATION \
    --format="get(config.nodeConfig.serviceAccount)" 

The output shows an address, such as service-account-name@your-composer-project.iam.gserviceaccount.com.

Granting the appropriate IAM permissions to the service account

To allow reads and writes to Cloud Storage buckets in your-storage-project, grant the roles/storage.objectAdmin role to the service account associated with your Cloud Composer environment.

Console

  1. In the IAM & Admin page for your storage project.

    Open the IAM & Admin page

  2. Click Add members.

  3. In the Add members dialog, specify the full email address of the service account associated with your Cloud Composer environment.

  4. In the Select a role drop down, select the appropriate permissions. For this example, select the Storage > Object Admin role.

  5. Click Add.

gcloud

Use the gcloud projects add-iam-policy-binding command to add project-level IAM permissions. Replace the VARIABLES with appropriate values:

gcloud projects add-iam-policy-binding YOUR_STORAGE_PROJECT \
    --member=serviceAccount:SERVICE_ACCOUNT_EMAIL \
    --role=roles/storage.objectAdmin 

After the appropriate permissions are granted, you can access resources in the your-storage-project project with the same default Airflow connections that you use to access resources in the your-composer-project project.

Creating new Airflow connections

Before you begin

Grant the appropriate Cloud IAM permissions to the service account associated with your Cloud Composer environment and use the default connections in your DAG definitions. Follow the steps in this section if you are unable to do so.

Creating a connection to another project

The following steps provide examples for how to allow reads and writes to Cloud Storage buckets in your-storage-project for a Cloud Composer environment deployed in the project ID your-composer-project.

  1. Create a service account in your-storage-project and download a JSON key:

    1. In the GCP Console, open the Service Accounts page.

      Open the Service Accounts page

    2. Click Select a project.

    3. Select your project and click Open.
    4. Click Create Service Account.
    5. Enter a service account name, select a role you want to grant to the service account, such as Storage > Object Admin.
    6. Check Furnish a new private key and click Save.
    7. Open the JSON file in a plain text editor. The contents should look like the following:
      { "type": "service_account", "project_id": "your-storage-project", ... }
  2. Access the Airflow web interface for your Cloud Composer environment.

  3. In the Airflow web interface, open the Admin > Connections page.

    Airflow
   screenshot. Open the Admin Connections menu.

  4. To open the new connection form, click the Create tab.

    Airflow
   screenshot. Click the Create tab.

  5. Create a new connection:

  6. To choose a connection ID, fill out the Conn Id field, such as my_gcp_connection. Use this ID in your DAG definition files.

  7. In the Conn Type field, select the Google Cloud Platform option.

  8. Enter a value for the Project Id that corresponds to the project that your service account belongs to.

  9. Copy the contents of the service account JSON key file that you downloaded into the Keyfile JSON field.

  10. Enter a value in the Scopes field. It is recommended to use https://www.googleapis.com/auth/cloud-platform as the scope and to use Cloud IAM permissions on the service account to limit access to GCP resources.

  11. To create the connection, click Save.

Airflow
   screenshot. Click the Create tab.

To use the connection you created, set it as the corresponding connection ID argument when you construct a GCP Airflow operator.

task_custom = bigquery_operator.BigQueryOperator(
    task_id='task_custom_connection',
    bql='SELECT 1', use_legacy_sql=False,
    # Set a connection ID to use a connection that you have created.
    bigquery_conn_id='my_gcp_connection')
Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Composer