Configuring maintenance windows and exclusions

This page shows how to use maintenance windows and maintenance exclusions to control when automatic cluster maintenance, such as auto-upgrades, can and cannot occur on your Google Kubernetes Engine clusters.

Before you begin

Before you start, make sure you have performed the following tasks:

• Ensure that you have enabled the Google Kubernetes Engine API.
Enable Google Kubernetes Engine API
• Ensure that you have installed the Cloud SDK.

Set up default gcloud settings using one of the following methods:

• Using gcloud init, if you want to be walked through setting defaults.
• Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

1. Run gcloud init and follow the directions:

   gcloud init

   If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

   gcloud init --console-only

2. Follow the instructions to authorize gcloud to use your Google Cloud account.
3. Create a new configuration or select an existing one.
4. Choose a Google Cloud project.
5. Choose a default Compute Engine zone.

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Using gcloud config

• Set your default project ID:

  gcloud config set project [PROJECT_ID]

• If you are working with zonal clusters, set your default compute zone:

  gcloud config set compute/zone [COMPUTE_ZONE]

• If you are working with regional clusters, set your default compute region:

  gcloud config set compute/region [COMPUTE_REGION]

• Update gcloud to the latest version:

  gcloud components update

Note: You can override these default settings in gcloud commands using the --project, --zone, and --region flags.

Configuring a maintenance window

To configure a maintenance window, you configure when it starts, how long it lasts, and how often it repeats.

Note: You must allow at least 24 hours of maintenance availability in a 14-day rolling window. Only contiguous availability windows of at least four hours are considered.

For example, you can configure a maintenance window that recurs weekly on Monday through Friday.

To configure a maintenance window, configure the following flags. You can configure a maintenance window using the Google Cloud Console or the gcloud command.

• --maintenance-window-start is an RFC-5545 DTSTART.
• --maintenance-window-end is specified in the same format as --maintenance-window-start, but is only used to calculate the duration of the maintenance window. --maintenance-window-end must be in the future, relative to --maintenance-window-start.
• --maintenance-window-recurrence is an RFC-5545 RRULE. This is an extremely flexible format with multiple ways to specify recurrence rules.

Note: The values for each of these flags must be enclosed in quote (") characters.

Creating a simple maintenance window

The --maintenance-window flag, which is generally available, allows you to specify a daily 4-hour maintenance window using a simplified format. You set it to a 24-hour timestamp in UTC time, like 16:00. Note: When using the Google Cloud Console, times are always displayed using the local timezone.

The maintenance window runs each day at that time and runs for four hours.

Creating a cluster with a maintenance window

gcloud

To create a new cluster with a maintenance window, use the gcloud container clusters create command and include the --maintenance-window-start,--maintenance-window-end and --maintenance-window-recurrence flags.

gcloud container clusters create [CLUSTER_NAME] \
  --maintenance-window-start "[TIMESTAMP]" \
  --maintenance-window-end "[TIMESTAMP]" \
  --maintenance-window-recurrence "[RRULE]"

Note: The values for each of these flags must be enclosed in quote (") characters.

For example, the following command creates a cluster named my-cluster with a maintenance window that starts at 2:00 AM UTC on August 1, 2019, finishes four hours later, and runs daily. You can learn more about formatting dates and times.

gcloud container clusters create my-cluster \
  --maintenance-window-start "2019-08-01T02:00:00Z" \
  --maintenance-window-end "2019-08-01T06:00:00Z" \
  --maintenance-window-recurrence "FREQ=DAILY"

Console

To create a new cluster with a maintenance window, perform the following steps:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click Create cluster.

3. Choose the Standard cluster template or choose an appropriate template for your workload. If you choose the Highly Available template, a regional cluster is created with a default maintenance window.

4. Configure your cluster as desired.

5. To customize the maintenance window, or if you used a cluster-creation template that does not include a maintenance window by default, click Availability, networking, security, and additional features.

6. In the Maintenance Window section, select the start time and length, then select the days of the week the maintenance window occurs on. To edit the RRULE directly, switch to the custom editor.

7. Click Create.

Configuring a maintenance window for an existing cluster

gcloud

To create or update a maintenance window for an existing cluster, use the gcloud container clusters update command and include the --maintenance-window-start,--maintenance-window-end and --maintenance-window-recurrence flags.

gcloud container clusters update [CLUSTER_NAME] \
  --maintenance-window-start "[TIMESTAMP]" \
  --maintenance-window-end "[TIMESTAMP]" \
  --maintenance-window-recurrence "[RRULE]"

Note: The values for each of these flags must be enclosed in quote (") characters.

You can use the --maintenance-window flag to create a simple maintenance window, but must specify the time in UTC, and cannot specify the start date. The first occurrence happens at the next occurrence of that time.

gcloud container clusters update my-cluster \
  --maintenance-window "[HH:MM]"

Console

To create or modify a maintenance window for an existing cluster:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Maintenance Window section, select the start time and length, then select the days of the week the maintenance window occurs on. To edit the RRULE directly, switch to the custom editor.

4. Click Save.

Manually finishing incomplete maintenance

If an upgrade or other automatic maintenance takes longer than the maintenance window to complete, GKE attempts to stop ongoing maintenance tasks and resumes them during the next occurrence of the maintenance window. If an automatic upgrade is canceled, and you have node auto-upgrades enabled, your nodes might be in a mixed-version state but your cluster should operate normally.

To manually upgrade your cluster, or cancel or roll back a partial upgrade, visit Manually upgrading a cluster.

Removing a maintenance window

gcloud

To remove a maintenance window, add the --clear-maintenance-window flag.

gcloud container clusters update [CLUSTER_NAME] --clear-maintenance-window

Console

To remove a maintenance window:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. From the Maintenance window drop-down menu, select Any time.

4. Click Save.

Example maintenance windows

The following examples illustrate some of the different ways you can configure a maintenance window. Only the relevant flags are shown, because the flags use the same syntax for creating a new cluster or updating an existing one.

Weekly on Tuesdays and Wednesdays, starting August 27, 2019, for the entire day

In this example, the difference between the start and end timestamps is a full day, so the maintenance window runs for 24 hours on both Tuesdays and Wednesdays.

--maintenance-window-start "2019-08-27T00:00:00Z" \
--maintenance-window-end "2019-08-28T00:00:00Z" \
--maintenance-window-recurrence "FREQ=WEEKLY;BYDAY=TU,WE"

Daily on weekdays from 9:00-17:00 UTC-4

This example shows how to have a daily maintenance window, but skip weekends. This example specifies a non-UTC timezone.

--maintenance-window-start "2019-09-02T09:00:00-04:00" \
--maintenance-window-end "2019-09-02T17:00:00-04:00" \
--maintenance-window-recurrence "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR"

Weekly at 4PM for 8 hours, UTC-7

If you do not specify a timezone for --maintenance-window-start, local time is used, as configured in your Google Cloud account. Google Cloud Console always uses local time.

--maintenance-window-start "2019-08-13T16:00:00-7:00" \
--maintenance-window-end "2019-08-14T00:00:00-7:00" \
--maintenance-window-recurrence "FREQ=WEEKLY"

Configuring a maintenance exclusion

To configure a maintenance exclusion, you configure its name (optional), start time, and end time. The maintenance exclusion can span multiple days.

Note: You must allow at least 24 hours of maintenance availability in a 14-day rolling window. Only contiguous availability windows of at least four hours are considered.

You can configure a maximum of three maintenance exclusions on a cluster at any given time. You can manually remove maintenance exclusions, whether they have occurred or not.

You can configure a maintenance exclusion using the Google Cloud Console or the gcloud command.

Creating a cluster with a maintenance exclusion

Note: You cannot currently configure a maintenance exclusion when creating a new cluster using the gcloud command. Instead, either create the cluster using Google Cloud Console. Otherwise, you can create the cluster, then configure the maintenance exclusion.

To create or modify a maintenance exclusion for an existing cluster:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Maintenance exclusion section, select Add maintenance exclusion. Select the start and end time.

4. Click Save.

You can see an example maintenance exclusion for Black Friday.

Removing a maintenance exclusion

A cluster can have a maximum of three active, non-elapsed maintenance exclusions at any time.

gcloud

To remove an existing maintenance exclusion, update the cluster and set the value of the --remove-maintenance-exclusion flag to the name of the maintenance exclusion to remove:

gcloud container clusters update [CLUSTER_NAME] \
  --remove-maintenance-exclusion [EXCLUSION-NAME]

Console

To remove a maintenance exclusion from an existing cluster:

1. Visit the Google Kubernetes Engine menu in Cloud Console.

   Visit the Google Kubernetes Engine menu

2. Click the cluster's Edit button, which looks like a pencil.

3. In the Maintenance exclusion section, click the X next to the exclusion to remove.

4. Click Save.

To see all maintenance exclusions on a cluster, you can view the cluster's maintenance policy.

Example maintenance exclusion

The following example prevents maintenance over the four days encompassing Black Friday through Cyber Monday, the highest-volume sales period of the year for many retail businesses. This example shows how to prevent a maintenance window from occurring from Black Friday 2019 (November 29, 2019) to Cyber Monday 2019 (December 2, 2019), from midnight on the east coast (UTC-5) to 23:59:59 on the west coast (UTC-7).

Note: The values for each of these flags must be enclosed in quote (") characters.

gcloud container clusters update [CLUSTER_NAME] \
 --add-maintenance-exclusion-name "black-friday" \
 --add-maintenance-exclusion-start "2019-11-29T00:00:00-05:00" \
 --add-maintenance-exclusion-end "2019-12-02T23:59:59-07:00"

Viewing a cluster's maintenance policy

To view a cluster's maintenance policy, including whether it has a maintenance window and all of its maintenance exclusions, use the gcloud container clusters describe command.

What's next

• Learn more about cluster and node upgrades
• Enable node auto-upgrade
• Manually upgrade a cluster