Configuring environments with Secret Manager

This page shows how to use Secret Manager to securely store Airflow connections and secrets.

Prerequisites

To use Secret Manager, your Cloud Composer environment must use Airflow 1.10.10 or later and Python 3.6 or later (Python 2 is not supported).

Configuring your environment with Secret Manager

1. Enable the Secret Manager API

   Secret Manager API

2. Add connections and variables in Secret Manager (see Creating secrets and versions for details)

   Secrets must follow the following format:

   • variable: [variable_prefix][sep][variable_name]

     • The default value for [variable_prefix] is airflow-variables
     • The default separator [sep] is -
     • Example: For the variable name my-var, the secret name is: airflow-variables-my-var

   • connection: [connection_prefix][sep][connection_name]

     • The default value for [connection_prefix] is airflow-connections
     • The default separator [sep] is -
     • Example: For the connection myConnection, the secret name is airflow-connections-myConnection
     • Connection values must use URI representation, for example: mysql://myhost:9000. The URI must be URL-encoded (percent encoded). For example, the link to Some Page should be: [https://myhost:9000/Some%20Page](https://myhost:9000/Some%20Page). Airflow has a convenience method for generating connection URIs. An example of how to encode a complex URL with JSON extras is available in the Airflow documentation.

   If you want to use different values for [variable_prefix], [connection_prefix] or [sep], use the optional settings in the section: Enable and configure Secret Manager backend.

3. Configure access control

   Airflow must be granted access to the secrets stored in Secret Manager. Grant a role which includes the secretmanager.versions.access permission (such as Secret Manager Secret Accessor), to the service account under which your Cloud Composer environment will access Secret Manager. This role can be granted at the Secret, Project, Folder, or Org level.

   By default, this account is the Compute Engine default service account, but a custom service account can be specified during environment creation. You can also override the service account under which Airflow accesses Secret Manager by setting the gcp_key_path parameter of the secrets.backend_kwargs Airflow setting to point to a JSON file with the service account's credentials. This method is less advisable, as it requires storing service account credentials in a file, which increases the risk that it could be compromised.

   

4. Enable DAG serialization

   In general, you should only use Secret Backend from within the execute() methods of your operators, or with the JINJA templates of Airflow macros. For example, you can retrieve variables using var.value.my_var. The Airflow web server runs under a different service account with limited permissions, and will not be able to access secrets in Secret Manager.

   If your DAG code accesses secrets during processing (not just from tasks) and cannot be adjusted as recommended above, you must enable DAG serialization to use the Airflow web server.

5. Enable and configure the Secret Manager backend

   1. Select your environment from the Environments page to access the Environment details page.
   2. Click AIRFLOW CONFIGURATION OVERRIDES.

   3. Add the configuration:

      • Section = secrets
      • Key = backend
      • Value = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

      

      Caution: Do not use airflow.contrib.secrets.gcp_secrets_manager.CloudSecretsManagerBackend as this value will not allow you to view logs on the Airflow web server UI

   4. Add optional settings:

      • Section = secrets
      • Key = backend_kwargs

      • Value: The JSON representation of an object with following fields: (source)

        • connections_prefix: Specifies the prefix of the secret to read to get Connections. Default: airflow-connections
        • variables_prefix: Specifies the prefix of the secret to read to get Variables. Default: airflow-variables
        • gcp_key_path: Path to the Google Cloud Credential JSON file (if not provided, the default service account will be used).
        • gcp_keyfile_dict: The Google Cloud Credential JSON dictionary. Mutually exclusive with gcp_key_path
        • sep: separator used to concatenate connections_prefix and conn_id. Default: -
        • project_id: The Google Cloud Project Id to access secrets from.

      • For example: Value = {"connections_prefix":"my-connections","variables_prefix":"my-variables", "sep":"-"}

Using Secret Manager

When fetching variables and connections, Cloud Composer will check Secret Manager first. If the requested variable or connection is not found, Cloud Composer then checks the environment/metastore database.

Note: For security reasons, variables and connections from Secret Manager are not visible in the Airflow web server UI.

Reading variables

JINJA templating

You can use the Secret Manager to read variables with JINJA templating for templated operator fields (resolved at the execution time).

For the variable airflow-variables-secret_filename use:

file_name = 'var.value.secret_filename'

Custom operators and callbacks

You can also use it to read variables in custom operators or callback methods from operators. Reading variables from inside DAGs can negatively impact performance, so using JINJA templates is encouraged if you want to use variables in your DAGs.

For the variable airflow-variables-myVariable use

from airflow.models.variable import Variable
myValue = Variable.get('myVariable')

Reading connections

Unless you are writing a custom operator, you should rarely need to access connections directly. Most hooks get the connection name as their instantiation parameter, and should retrieve connections from the secret backend automatically when tasks are executed.

Reading connections directly may be useful when writing your own hook.

For the connection airflow-connections-myConnection use:

from airflow.hooks.base_hook import BaseHook
myConnection = BaseHook.get_connection('myConnection')

Note: BaseHook.get_connection returns a Connection object. It's possible to get the URI string representation with myConnectionUri = BaseHook.get_connection('myConnection').get_uri()