Configure email notifications

Cloud Composer 1 | Cloud Composer 2

This page explains how to configure SMTP services for your Cloud Composer environment.

Before you begin

If your Cloud Composer environment is configured for Private IP, make sure that it has connectivity to the API endpoint for your external email service (such as https://api.sendgrid.com, or your preferred SMTP server).

Configure SendGrid email services

To receive notifications, configure your environment variables to send email through the SendGrid email service.

  1. If you haven't already, sign up with SendGrid in Google Cloud Console and create an API key. As a Google Cloud developer, you can start with 12,000 free emails per month.

    Go to SendGrid Email API

  2. Set the following environment variables for your environment:

    Name Value
    SENDGRID_MAIL_FROM The From: email address, such as noreply-composer@<your-domain>.
    SENDGRID_API_KEY Your SendGrid API key.
  3. To test SendGrid configuration:

    1. Create a test DAG that uses the EmailOperator.
    2. Upload the DAG to your environment and check that the EmailOperator task succeeds.
    3. Sign in to SendGrid in with your SendGrid credentials.
    4. In the SendGrid UI, go to Activity page.
    5. Search the list for the email. You should see that SendGrid processed and delivered the email.
    6. If the email is not processed and delivered:

      • Check your Sendgrid configurations.
      • Verify that the SENDGRID_MAIL_FROM and SENDGRID_API_KEY environment variables are correct.
      • Check the spam filter in your email client.

Configure third-party SMTP services

To send email through a third-party SMTP service, override the email_backend Airflow configuration option and configure other SMTP-related parameters.

To configure a third-party SMTP service, override the following Airflow configuration options:

Section Key Value
email email_backend airflow.utils.email.send_email_smtp
smtp smtp_host The hostname for the SMTP server.
smtp smtp_user The user name on the SMTP server.
smtp smtp_port The port for the SMTP server. Port 25 is not available. You can use other ports, such as standard SMTP ports 465 and 587.
smtp smtp_password Setting a password via smtp_password is not supported. To set an SMTP password, follow instructions provided in Configuring an SMTP password.
smtp smtp_mail_from The From: email address, such as noreply-composer@.
smtp smtp_starttls For enhanced security, set to True.
smtp smtp_ssl For enhanced security, set to True.

Configure an SMTP password for a third-party SMTP service

Keeping an SMTP password in plain text in Airflow configuration file is a bad security practice. That's why Cloud Composer does not support this method. Instead, you can use two other methods for configuring an SMTP password.

Using a command to retrieve an SMTP password

You can use a configuration override to specify a command that obtains the SMTP password. When communicating with your SMTP service, Airflow uses this command to get the value of the password.

To use this method, override the following Airflow configuration option:

Section Key Value
smtp smtp_password_cmd Specify a command that returns the SMTP password.

Using a secret stored in Secret Manager to retrieve an SMTP password

You can configure Secret Manager as your Airflow secrets backend.

Once you configure Secret Manager for your Composer environment, you can store an SMTP password in Secret Manager:

  1. Create a new secret:

    echo -n "SMTP_PASSWORD" | gcloud beta secrets create \
      airflow-variables-smtp-password \
      --data-file=- \
      --replication-policy=automatic
    

    Replace SMTP_PASSWORD with your SMTP password.

  2. Configure Airflow to obtain the SMTP password from Secret Manager. To do so, override the following Airflow configuration option:

    Section Key Value
    smtp smtp_password_secret smtp-password

What's next