Specify maintenance windows for your environments

This page explains how to define maintenance windows for your environments.

Cloud Composer environments are built on top of other Google Cloud services. Some of these services require maintenance operations to happen, for example, when patching the configuration of an existing resource. Maintenance operations are often required for GKE and Cloud SQL resources.

How to use maintenance windows

Maintenance operations might impact the execution of your DAGs and Airflow tasks, so we recommended you to do the following:

  1. Define maintenance windows for your Cloud Composer environments.

  2. Schedule DAG runs outside of specified maintenance windows by using start_date and schedule_interval parameters in your DAGs.

You can define maintenance windows for new and existing Cloud Composer environments. If you don't specify explicit maintenance windows, then time for maintenance operations is selected automatically without considering the schedule of your DAG runs.

Specify maintenance windows for new environments

Console

To define maintenance windows when you create an environment:

  1. In the Google Cloud Console, go to the Create environment page.

    Go to Create environment

  2. Expand the Networking, Airflow config overrides, and additional features section.

  3. In the Maintenance windows section, select the Set custom time for maintenance windows checkbox.

  4. Set Start time, Timezone, Days, and Length, so that combined time for the specified schedule is at least 12 hours in a 7-day rolling window. For example, a period of 4 hours every Monday, Wednesday, and Friday provides the required amount of time.

  5. Complete all other steps to create an environment.

gcloud

When you create an environment, the following arguments define maintenance windows parameters:

  • --maintenance-window-start sets the start time of a maintenance window.
  • --maintenance-window-end sets the end time of a maintenance window.
  • --maintenance-window-recurrence sets maintenance window recurrence.
gcloud beta composer environments create ENVIRONMENT_NAME \
    --location LOCATION \
    --maintenance-window-start 'DATETIME_START' \
    --maintenance-window-end 'DATETIME_END' \
    --maintenance-window-recurrence 'MAINTENANCE_RECURRENCE'

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • DATETIME_START with the start date and time in the date/time input format. Only the specified time of the day is used, the specified date is ignored.
  • DATETIME_END with the end date and time in the date/time input format. Only the specified time of the day is used, the specified date is ignored. The specified date and time must be after the start date.
  • MAINTENANCE_RECURRENCE with an RFC 5545 RRULE for maintenance windows recurrence. Cloud Composer supports two formats:

    • The FREQ=DAILY format specifies a daily recurrence.
    • The FREQ=WEEKLY;BYDAY=SU,MO,TU,WE,TH,FR,SA format specifies a recurrence on selected days of the week.

The following example specifies a 6-hour maintenance window between 01:00 and 07:00 (UTC) on Sundays, Wednesdays, and Saturdays. The 1 January, 2021 date is ignored.

gcloud beta composer environments create example-environment \
  --location us-central1 \
  --maintenance-window-start '2021-01-01T01:00:00Z' \
  --maintenance-window-end '2021-01-01T07:00:00Z' \
  --maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=SU,WE,SA'

Specify maintenance windows for existing environments

Console

To define or change maintenance windows for an existing environment, update the environment:

  1. In the Google Cloud Console, go to the Environments page.

Go to Environments

  1. Select an environment to view its details.

  2. Go to the Environment configuration tab.

  3. Next to the Maintenance windows entry, click Edit.

  4. In the Maintenance windows dialog, select the Set custom time for maintenance windows checkbox.

  5. Set Start time, Timezone, Days, and Length, so that combined time for the specified schedule is at least 12 hours in a 7-day rolling window. For example, a period of 4 hours every Monday, Wednesday, and Friday provides the required amount of time.

  6. Click Save and wait until your environment updates.

gcloud

When you update an environment, the following arguments define maintenance windows parameters:

  • --maintenance-window-start sets the start time of a maintenance window.
  • --maintenance-window-end sets the end time of a maintenance window.
  • --maintenance-window-recurrence sets maintenance window recurrence.
gcloud beta composer environments update ENVIRONMENT_NAME \
    --location LOCATION \
    --maintenance-window-start 'DATETIME_START' \
    --maintenance-window-end 'DATETIME_END' \
    --maintenance-window-recurrence 'MAINTENANCE_RECURRENCE'

Replace:

  • ENVIRONMENT_NAME with the name of the environment.
  • DATETIME_START with the start date and time in the date/time input format. Only the specified time of the day is used, the specified date is ignored.
  • DATETIME_END with the end date and time in the date/time input format. Only the specified time of the day is used, the specified date is ignored. The specified date and time must be after the start date.
  • MAINTENANCE_RECURRENCE with an RFC 5545 RRULE for maintenance windows recurrence. Cloud Composer supports two formats:

    • The FREQ=DAILY format specifies a daily recurrence.
    • The FREQ=WEEKLY;BYDAY=SU,MO,TU,WE,TH,FR,SA format specifies a recurrence on selected days of the week.

The following example specifies a 6-hour maintenance window between 01:00 and 07:00 (UTC) on Sundays, Wednesdays, and Saturdays. The 1 January, 2021 date is ignored.

gcloud beta composer environments update example-environment \
  --location us-central1 \
  --maintenance-window-start '2021-01-01T01:00:00Z' \
  --maintenance-window-end '2021-01-01T07:00:00Z' \
  --maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=SU,WE,SA'

What's next