Configuring maintenance windows and exclusions

This page shows how to use maintenance windows and maintenance exclusions to control when automatic cluster maintenance, such as auto-upgrades, can and cannot occur on your Google Kubernetes Engine (GKE) clusters.

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

• Using gcloud init, if you want to be walked through setting defaults.
• Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize gcloud to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

• Set your default project ID:

  gcloud config set project PROJECT_ID

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone COMPUTE_ZONE

• If you are working with Autopilot or regional clusters, set your default compute region:

  gcloud config set compute/region COMPUTE_REGION

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Configuring a maintenance window

To configure a maintenance window, you configure when it starts, how long it lasts, and how often it repeats. For example, you can configure a maintenance window that recurs weekly on Monday through Friday.

Note: You must allow at least 48 hours of maintenance availability in a 32-day rolling window. Only contiguous availability windows of at least four hours are considered.

You can configure a maintenance window using the Google Cloud Console or the gcloud command.

Note: When using the Google Cloud Console, times are always displayed using the local timezone.

Warning: Day of the week recurrences are always based on UTC. We recommend setting maintenance windows entirely in UTC as a result. While local time can be used when setting flags, the timezone will not be stored.

Creating a cluster with a simple maintenance window

You can create a simple maintenance window in gcloud by specifying the --maintenance-window flag. This flag allows you to specify a daily 4-hour maintenance window using a simplified format.

To create a new cluster with a simple maintenance window, run the following command:

gcloud container clusters create cluster-name \
  --maintenance-window start-time

where:

• cluster-name is the name of the new cluster.
• start-time is set to a 24-hour timestamp in UTC time, like 16:00.

The maintenance window runs each day at the specified start-time and runs for four hours.

Creating a cluster with a maintenance window

You can create a new cluster with a maintenance window by using the gcloud tool or the Google Cloud Console.

gcloud

To create a new cluster with a maintenance window, run the following command:

gcloud container clusters create cluster-name \
  --maintenance-window-start start-time \
  --maintenance-window-end end-time \
  --maintenance-window-recurrence rrule

where:

• cluster-name is the name of the new cluster.
• start-time is an RFC-5545 DTSTART.
• end-time is specified in the same format as start-time, but is only used to calculate the duration of the maintenance window. The value for end-time must be in the future, relative to start-time.
• rrule is an RFC-5545 RRULE. This is an extremely flexible format with multiple ways to specify recurrence rules.

For example, the following command creates a cluster named my-cluster with a maintenance window that starts at 2:00 AM UTC on August 1, 2019, finishes four hours later, and runs daily. You can learn more about formatting dates and times.

gcloud container clusters create my-cluster \
  --maintenance-window-start 2019-08-01T02:00:00Z \
  --maintenance-window-end 2019-08-01T06:00:00Z \
  --maintenance-window-recurrence FREQ=DAILY

Console

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click Create.

3. Configure your cluster as desired.

4. From the navigation pane, under Clusters, click Automation.

5. Select the Enable Maintenance Window checkbox.

6. Select the start time and length, then select the days of the week on which the maintenance window occurs. To edit the recurrence rule specification (RRule) directly, select Custom editor.

7. Click Create.

Configuring a maintenance window for an existing cluster

gcloud

To create or update a maintenance window for an existing cluster, run the following command:

gcloud container clusters update cluster-name \
  --maintenance-window-start start-time \
  --maintenance-window-end end-time \
  --maintenance-window-recurrence rrule

where:

• cluster-name is the name of the existing cluster.
• start-time is an RFC-5545 DTSTART.
• end-time is specified in the same format as start-time, but is only used to calculate the duration of the maintenance window. The value for end-time must be in the future, relative to start-time.
• rrule is an RFC-5545 RRULE. This is an extremely flexible format with multiple ways to specify recurrence rules.

Console

To create or modify a maintenance window for an existing cluster:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. In the cluster list, click the name of the cluster you want to modify.

3. Under Automation, click Edit maintenance policy next to the Maintenance window field.

4. Select the Enable Maintenance Window checkbox.

5. Select the start time and length, then select the days of the week on which the maintenance window occurs. To edit the RRule directly, select Custom editor.

6. Click Save Changes.

Manually finishing incomplete maintenance

If an upgrade or other automatic maintenance takes longer than the maintenance window to complete, GKE attempts to stop ongoing maintenance tasks and resumes them during the next occurrence of the maintenance window. If an automatic upgrade is canceled, and you have node auto-upgrades enabled, your nodes might be in a mixed-version state but your cluster should operate normally.

To manually upgrade your cluster, or cancel or roll back a partial upgrade, visit Manually upgrading a cluster.

Removing a maintenance window

gcloud

To remove a maintenance window from a cluster, run the following command:

gcloud container clusters update cluster-name --clear-maintenance-window

where cluster-name is the name of the existing cluster.

Console

To remove a maintenance window:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. In the cluster list, click the name of the cluster you want to modify.

3. Under Automation, click Edit maintenance policy next to the Maintenance window field.

4. Clear the Enable Maintenance Window checkbox.

5. Click Save Changes.

Example maintenance windows

The following examples illustrate some of the different ways you can configure a maintenance window. Only the relevant flags are shown, because the flags use the same syntax for creating a new cluster or updating an existing one.

Weekly on Tuesdays and Wednesdays, starting August 27, 2019, for the entire day

In this example, the difference between the start and end timestamps is a full day, so the maintenance window runs for 24 hours on both Tuesdays and Wednesdays.

--maintenance-window-start 2019-08-27T00:00:00Z \
--maintenance-window-end 2019-08-28T00:00:00Z \
--maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=TU,WE'

Daily on weekdays from 9:00-17:00 UTC-4

This example shows how to have a daily maintenance window, but skip weekends. This example specifies a non-UTC timezone.

--maintenance-window-start 2019-09-02T09:00:00-04:00 \
--maintenance-window-end 2019-09-02T17:00:00-04:00 \
--maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR'

Weekly at 4PM for 8 hours, UTC-7

If you do not specify a timezone for --maintenance-window-start, local time is used, as configured in your Google Cloud account. Google Cloud Console always uses local time.

--maintenance-window-start 2019-08-13T16:00:00-7:00 \
--maintenance-window-end 2019-08-14T00:00:00-7:00 \
--maintenance-window-recurrence FREQ=WEEKLY

Configuring a maintenance exclusion

To configure a maintenance exclusion, you configure its name (optional), start time, and end time. The maintenance exclusion can span multiple days.

The maintenance exclusion must allow 48 hours of maintenance availability in a 32-day rolling window. The duration of your maintenance windows directly affects the maximum maintenance exclusion and can be calculated using the following formula:

maximum maintenance exclusion (days) = 32-day rolling window - 48 hours maintenance availability / maintenance window hours per day

For example, if your normal maintenance window is 4 hours long, then your maximum maintenance exclusion is 20 days: (32 - 48/4 = 20).

You can configure a maximum of three maintenance exclusions on a cluster at any given time. You can manually remove maintenance exclusions, whether they have occurred or not.

You can configure a maintenance exclusion using the Google Cloud Console or the gcloud command.

Creating a cluster with a maintenance exclusion

Note: Currently, you cannot configure a maintenance exclusion when creating a new cluster using the gcloud command.

To create a cluster with a maintenance exclusion:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click Create.

3. Configure your cluster as desired.

4. From the navigation pane, under Cluster, click Automation.

5. Under Maintenance exclusions, click Add Maintenance Exclusion.

6. Select a Start time and an End time.

7. Click Create.

You can see an example maintenance exclusion for Black Friday.

Configuring a maintenance exclusion for an existing cluster

gcloud

To configure a maintenance exclusion for an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
  --add-maintenance-exclusion-name EXCLUSION_NAME
  --add-maintenance-exclusion-start START_DATE_TIME \
  --add-maintenance-exclusion-end END_DATE_TIME

Replace the following:

• CLUSTER_NAME: the name of your cluster.
• EXCLUSION_NAME: the name of the maintenance exclusion.
• START_DATE_TIME: the start date and time for the exclusion.
• END_DATE_TIME: the end date and time for the exclusion.

To view supported date and time formats, run gcloud topic datetimes.

Console

To configure a maintenance exclusion for an existing cluster:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. In the cluster list, click the name of the cluster you want to modify.

3. Under Automation, next to the Maintenance exclusions field, click Edit maintenance exclusions.

4. Under Maintenance exclusions, click Add Maintenance Exclusion.

5. Select a Start time and an End time.

6. Click Save Changes.

Removing a maintenance exclusion

A cluster can have a maximum of three active, non-elapsed maintenance exclusions at any time.

gcloud

To remove an existing maintenance exclusion, run the following command:

gcloud container clusters update cluster-name \
  --remove-maintenance-exclusion exclusion-name

where:

• cluster-name is the name of the existing cluster.
• exclusion-name is the name of the maintenance exclusion to remove.

Console

To remove a maintenance exclusion from an existing cluster:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. In the cluster list, click the name of the cluster you want to modify.

3. Under Automation, next to the Maintenance exclusions field, click Edit maintenance exclusions.

4. Under Maintenance exclusions, click Delete item next to the End time field for the exclusion you want to remove.

5. Click Save Changes.

To see all maintenance exclusions on a cluster, you can view the cluster's maintenance policy.

Example maintenance exclusion

The following example prevents maintenance over the four days encompassing Black Friday through Cyber Monday, the highest-volume sales period of the year for many retail businesses. This example shows how to prevent a maintenance window from occurring from Black Friday 2019 (November 29, 2019) to Cyber Monday 2019 (December 2, 2019), from midnight on the east coast (UTC-5) to 23:59:59 on the west coast (UTC-7).

gcloud container clusters update sample-cluster \
 --add-maintenance-exclusion-name black-friday \
 --add-maintenance-exclusion-start 2019-11-29T00:00:00-05:00 \
 --add-maintenance-exclusion-end 2019-12-02T23:59:59-07:00

Viewing a cluster's maintenance policy

To view a cluster's maintenance policy, including whether it has a maintenance window and all of its maintenance exclusions, use the following command:

gcloud container clusters describe cluster-name

What's next

• Learn more about cluster and node upgrades
• Enable node auto-upgrade
• Manually upgrade a cluster