Cloud Composer 1 | Cloud Composer 2 | Cloud Composer 3
This page shows how to use Secret Manager to securely store Airflow connections and secrets.
Before you begin
- To use Secret Manager, your Cloud Composer environment must use Airflow 1.10.10 or later and Python 3.6 or later.
- Python 2 is not supported.
Configure Secret Manager for your environment
This section explains how to configure Secret Manager so that you can use secrets with your Cloud Composer environment.
Enable the Secret Manager API
Console
Enable the Secret Manager API.
gcloud
Enable the Secret Manager API:
gcloud services enable secretmanager.googleapis.com
Configure access control
You must configure access control so that Airflow can access secrets stored in Secret Manager.
To do so, the service account that accesses secrets must have a role with
the secretmanager.versions.access
permission. For example,
the Secret Manager Secret Accessor role includes this permission.
You can grant this role at the Secret, Project, Folder, or Org level.
Use one of the following options:
(Recommended) Grant this role to the service account of your environment.
Override the service account under which Airflow accesses Secret Manager.
- Grant this role to a service account.
- Set the
gcp_key_path
parameter of thebackend_kwargs
Airflow configuration option to point to a JSON file with the service account credentials.
Enable DAG serialization
In general, you should only use the Secret Manager backend from
within the execute()
methods of your operators, or with the
Jinja templates.
For example, you can retrieve variables using var.value.example_var
.
The Airflow web server runs under a different service account with limited
permissions, so it cannot access secrets in Secret Manager. If your
DAG code accesses secrets during DAG processing (not just from tasks)
and it is not possible to adjust it to access secrets from within
the execute()
methods, then
enable DAG serialization. After you do so, the Airflow
web server takes processed DAGs and does not need access to secrets.
Enable and configure the Secret Manager backend
Override the following Airflow configuration option:
Section Key Value secrets
backend
airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend
Add optional settings by overriding the following Airflow configuration option:
Section Key Value secrets
backend_kwargs
See the following description. The
backend_kwargs
value is the JSON representation of thebackend_kwargs
object with the following fields:connections_prefix
: prefix of the secret name to read in order to get Connections. The default isairflow-connections
.variables_prefix
: prefix of the secret name to read in order to get Variables. The default is:airflow-variables
.gcp_key_path
: path to the Google Cloud Credential JSON file (if not provided, the default service account is used).gcp_keyfile_dict
: Google Cloud Credential JSON dictionary. Mutually exclusive withgcp_key_path
.sep
: separator used to concatenateconnections_prefix
andconn_id
. The default is-
.project_id
: Google Cloud Project Id where secrets are stored.
For example, the value of
backend_kwargs
can be:{"project_id": "<project id>", "connections_prefix":"example-connections", "variables_prefix":"example-variables", "sep":"-"}
.
Add connections and variables in Secret Manager
Create secrets by following steps outlined in Creating secrets and versions.
Variables
- Must use the
[variables_prefix][sep][variable_name]
format. - The default value for
[variables_prefix]
isairflow-variables
. - The default separator
[sep]
is-
.
For example, if the variable name is example-var
, then the secret name
is airflow-variables-example-var
.
Connection names
- Must use the
[connection_prefix][sep][connection_name]
format. - The default value for
[connection_prefix]
isairflow-connections
. - The default separator
[sep]
is-
.
For example, if the connection name is exampleConnection
, then the secret
name is airflow-connections-exampleConnection
.
Connection values
Must use URI representation. For example,
postgresql://login:secret@examplehost:9000
.The URI must be URL-encoded (percent encoded). For example, a password that has a space symbol in it must be URL-encoded as follows:
postgresql://login:secret%20password@examplehost:9000
.
Airflow has a convenience method for generating connection URIs. An example of how to encode a complex URL with JSON extras is available in the Airflow documentation.
Use Secret Manager with Cloud Composer
When fetching variables and connections, Cloud Composer checks Secret Manager first. If the requested variable or connection is not found, Cloud Composer then checks the environment variables and the Airflow database.
Read variables using Jinja templating
You can use Secret Manager to read variables with Jinja templating for templated operator fields (resolved at the execution time).
For the airflow-variables-secret_filename
secret:
file_name = '{{var.value.secret_filename}}'
Read variables using custom operators and callbacks
You can also use Secret Manager to read variables in custom operators or callback methods from operators. Reading variables from inside DAGs can negatively impact performance, so use Jinja templates if you want to use variables in your DAGs.
For example, for the airflow-variables-secret_filename
secret:
from airflow.models.variable import Variable
file_name = Variable.get('secret_filename')
Read connections
Unless you are writing a custom operator, you should rarely need to access connections directly. Most hooks get the connection name as their instantiation parameter, and should retrieve connections from the secret backend automatically when tasks are executed.
Reading connections directly may be useful when writing your own hook.
For example, for the airflow-connections-exampleConnection
connection:
from airflow.hooks.base_hook import BaseHook
exampleConnection = BaseHook.get_connection('exampleConnection')
BaseHook.get_connection
returns a Connection
object. It is
possible to get the URI string representation of a connection like this:
exampleConnectionUri = BaseHook.get_connection('exampleConnection').get_uri()