Configuring environments with Secret Manager

This page shows how to use Secret Manager to securely store Airflow connections and secrets.

Before you begin

To use Secret Manager, your Cloud Composer environment must use Airflow 1.10.10 or later and Python 3.6 or later. Python 2 is not supported.

Configuring your environment with Secret Manager

  1. Enable the Secret Manager API

    Secret Manager API

  2. Add connections and variables in Secret Manager (see Creating secrets and versions for details).

    Secrets must follow this format:

    • Variables: [variable_prefix][sep][variable_name]

      • The default value for [variable_prefix] is airflow-variables.
      • The default separator [sep] is -.
      • For example, if the variable name is example-var, then the secret name is airflow-variables-example-var.
    • Connections: [connection_prefix][sep][connection_name]

      • The default value for [connection_prefix] is airflow-connections.
      • The default separator [sep] is -.
      • For example, if the connection name is exampleConnection, then the secret name is airflow-connections-exampleConnection.
      • Connection values must use URI representation. For example, mysql://login:password@examplehost:9000.

        The URI must be URL-encoded (percent encoded). For example, a password that has a space symbol in it must be URL-encoded as follows: mysql://login:secret%20password@examplehost:9000.

        Airflow has a convenience method for generating connection URIs. An example of how to encode a complex URL with JSON extras is available in the Airflow documentation.

    If you want to use different values for [variable_prefix], [connection_prefix] or [sep], use the optional settings in the section: Enable and configure Secret Manager backend.

  3. Configure access control

    You must grant Airflow access to the secrets stored in Secret Manager. To do so, grant a role which includes the secretmanager.versions.access permission (such as Secret Manager Secret Accessor), to the service account under which your Cloud Composer environment accesses Secret Manager.

    By default, this account is the Compute Engine default service account, but you can specify a custom service account during the environment creation.

    You can grant this role at the Secret, Project, Folder, or Org level.

    Access Control

  4. Enable DAG serialization

    In general, you should only use Secret Backend from within the execute() methods of your operators, or with the Jinja templates of Airflow macros. For example, you can retrieve variables using var.value.example_var.

    The Airflow web server runs under a different service account with limited permissions, so it cannot access secrets in Secret Manager. If your DAG code accesses secrets during DAG processing (not just from tasks) and it is not possible to adjust it to access secrets from within the execute() methods, then enable DAG serialization. After you do so, the Airflow web server takes processed DAGs and does not need access to secrets.

  5. Enable and configure the Secret Manager backend

    1. Override the following Airflow configuration option:

      Section Key Value
      secrets backend
    2. Add optional settings by overriding this Airflow configuration option:

      Section Key Value
      secrets backend_kwargs See the following description.

      The backend_kwargs value is the JSON representation of the backend_kwargs object with the following fields:

      • connections_prefix: Specifies the prefix of the secret name to read in order to get Connections. Default is: airflow-connections.
      • variables_prefix: Specifies the prefix of the secret name to read in order to get Variables. Default is: airflow-variables.
      • gcp_key_path: Path to the Google Cloud Credential JSON file (if not provided, the default service account is used).
      • gcp_keyfile_dict: The Google Cloud Credential JSON dictionary. Mutually exclusive with gcp_key_path.
      • sep: separator used to concatenate connections_prefix and conn_id. Default: -.
      • project_id: The Google Cloud Project Id where secrets are stored.

      For example, the value of backend_kwargs can be: {"connections_prefix":"example-connections","variables_prefix":"example-variables", "sep":"-"}

Using Secret Manager with Cloud Composer

When fetching variables and connections, Cloud Composer checks Secret Manager first. If the requested variable or connection is not found, Cloud Composer then checks the environment variables and Airflow database.

Reading variables

Jinja templating

You can use Secret Manager to read variables with Jinja templating for templated operator fields (resolved at the execution time).

For the variable airflow-variables-secret_filename, use:

file_name = 'var.value.secret_filename'

Custom operators and callbacks

You can also use Secret Manager to read variables in custom operators or callback methods from operators. Reading variables from inside DAGs can negatively impact performance, so use Jinja templates if you want to use variables in your DAGs.

For the variable airflow-variables-secret_filename, use:

from airflow.models.variable import Variable
file_name = Variable.get('secret_filename')

Reading connections

Unless you are writing a custom operator, you should rarely need to access connections directly. Most hooks get the connection name as their instantiation parameter, and should retrieve connections from the secret backend automatically when tasks are executed.

Reading connections directly may be useful when writing your own hook.

For the connection airflow-connections-exampleConnection use:

from airflow.hooks.base_hook import BaseHook
exampleConnection = BaseHook.get_connection('exampleConnection')

BaseHook.get_connection returns a Connection object. It is possible to get the URI string representation of a connection like this:

exampleConnectionUri = BaseHook.get_connection('exampleConnection').get_uri()