Configuring maintenance windows and exclusions

This page shows you how to use maintenance windows and maintenance exclusions to control when automatic cluster maintenance, such as auto-upgrades, can and cannot occur on your Google Kubernetes Engine (GKE) clusters.

Before you begin

Before you start, make sure you have performed the following tasks:

Set up default gcloud settings using one of the following methods:

  • Using gcloud init, if you want to be walked through setting defaults.
  • Using gcloud config, to individually set your project ID, zone, and region.

Using gcloud init

If you receive the error One of [--zone, --region] must be supplied: Please specify location, complete this section.

  1. Run gcloud init and follow the directions:

    gcloud init

    If you are using SSH on a remote server, use the --console-only flag to prevent the command from launching a browser:

    gcloud init --console-only
  2. Follow the instructions to authorize gcloud to use your Google Cloud account.
  3. Create a new configuration or select an existing one.
  4. Choose a Google Cloud project.
  5. Choose a default Compute Engine zone for zonal clusters or a region for regional or Autopilot clusters.

Using gcloud config

  • Set your default project ID:
    gcloud config set project PROJECT_ID
  • If you are working with zonal clusters, set your default compute zone:
    gcloud config set compute/zone COMPUTE_ZONE
  • If you are working with Autopilot or regional clusters, set your default compute region:
    gcloud config set compute/region COMPUTE_REGION
  • Update gcloud to the latest version:
    gcloud components update

Configuring a maintenance window

To configure a maintenance window, you configure when it starts, how long it lasts, and how often it repeats. For example, you can configure a maintenance window that recurs weekly on Monday through Friday.

You can configure a maintenance window by using the Google Cloud Console or the gcloud command-line tool.

Create a cluster with a simple maintenance window

You can create a simple maintenance window in gcloud tool by specifying the --maintenance-window flag. This flag allows you to specify a daily 4-hour maintenance window using a simplified format.

To create a new cluster with a simple maintenance window, run the following command:

gcloud container clusters create CLUSTER_NAME \
    --maintenance-window START_TIME

Replace the following:

  • CLUSTER_NAME: the name of your new cluster.
  • START_TIME: a 24-hour timestamp in UTC time, like 16:00.

The maintenance window runs each day at the specified START_TIME and runs for four hours.

Create a cluster with a maintenance window

You can create a new cluster with a maintenance window by using the gcloud tool or the Google Cloud Console.

gcloud

To create a new cluster with a maintenance window, run the following command:

gcloud container clusters create CLUSTER_NAME \
    --maintenance-window-start START_TIME \
    --maintenance-window-end END_TIME \
    --maintenance-window-recurrence RRULE

Replace the following:

  • CLUSTER_NAME: the name of the new cluster.
  • START_TIME: when to start the maintenance window, expressed as an RFC-5545 DTSTART value.
  • END_TIME: when to end the maintenance window, specified in the same format as START_TIME, but is only used to calculate the duration of the maintenance window. The value for END_TIME must be in the future, relative to START_TIME.
  • RRULE: an RFC-5545 RRULE. This is an extremely flexible format with multiple ways to specify recurrence rules.

For example, the following command creates a cluster named my-cluster with a maintenance window that starts at 2:00 AM UTC on August 1, 2019, finishes four hours later, and runs daily. You can learn more about formatting dates and times.

gcloud container clusters create my-cluster \
    --maintenance-window-start 2019-08-01T02:00:00Z \
    --maintenance-window-end 2019-08-01T06:00:00Z \
    --maintenance-window-recurrence FREQ=DAILY

Console

  1. Go to the Google Kubernetes Engine page in the Cloud Console.

    Go to Google Kubernetes Engine

  2. Click Create.

  3. Configure your cluster as desired.

  4. From the navigation pane, under Clusters, click Automation.

  5. Select the Enable Maintenance Window checkbox.

  6. Select the start time and length, then select the days of the week on which the maintenance window occurs. To edit the recurrence rule specification (RRule) directly, select Custom editor.

  7. Click Create.

Configure a maintenance window for an existing cluster

gcloud

To create or update a maintenance window for an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --maintenance-window-start START_TIME \
    --maintenance-window-end END_TIME \
    --maintenance-window-recurrence RRULE

Replace the following:

  • CLUSTER_NAME: the name of the existing cluster.
  • START_TIME: when to start the maintenance window, expressed as an RFC-5545 DTSTART value.
  • END_TIME: when to end the maintenance window, specified in the same format as START_TIME, but is only used to calculate the duration of the maintenance window. The value for END_TIME must be in the future, relative to START_TIME.
  • RRULE: an RFC-5545 RRULE. This is an extremely flexible format with multiple ways to specify recurrence rules.

Console

To create or modify a maintenance window for an existing cluster:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to modify.

  3. Under Automation, click Edit maintenance policy next to the Maintenance window field.

  4. Select the Enable Maintenance Window checkbox.

  5. Select the start time and length, then select the days of the week on which the maintenance window occurs. To edit the RRule directly, select Custom editor.

  6. Click Save Changes.

Manually finish incomplete maintenance

If an upgrade or other automatic maintenance takes longer than the maintenance window to complete, GKE attempts to stop ongoing maintenance tasks and resumes them during the next occurrence of the maintenance window. If an automatic upgrade is canceled, and you have node auto-upgrades enabled, your nodes might be in a mixed-version state but your cluster should operate normally.

To manually upgrade your cluster, or cancel or roll back a partial upgrade, visit Manually upgrading a cluster.

Remove a maintenance window

gcloud

To remove a maintenance window from a cluster, run the following command:

gcloud container clusters update CLUSTER_NAME --clear-maintenance-window

Replace CLUSTER_NAME with the name of the existing cluster.

Console

To remove a maintenance window:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to modify.

  3. Under Automation, click Edit maintenance policy next to the Maintenance window field.

  4. Clear the Enable Maintenance Window checkbox.

  5. Click Save Changes.

Example maintenance windows

The following examples illustrate some of the different ways you can configure a maintenance window. Only the relevant flags are shown, because the flags use the same syntax for creating a new cluster or updating an existing one.

Weekly on Tuesdays and Wednesdays, starting August 27, 2019, for the entire day

In this example, the difference between the start and end timestamps is a full day, so the maintenance window runs for 24 hours on both Tuesdays and Wednesdays.

--maintenance-window-start 2019-08-27T00:00:00Z \
--maintenance-window-end 2019-08-28T00:00:00Z \
--maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=TU,WE'
Daily on weekdays from 9:00-17:00 UTC-4

This example shows how to have a daily maintenance window, but skip weekends. This example specifies a non-UTC time zone.

--maintenance-window-start 2019-09-02T09:00:00-04:00 \
--maintenance-window-end 2019-09-02T17:00:00-04:00 \
--maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR'
Weekly at 4PM for 8 hours, UTC-7

If you do not specify a time zone for --maintenance-window-start, local time is used, as configured in your Google Cloud account. Google Cloud Console always uses local time.

--maintenance-window-start 2019-08-13T16:00:00-7:00 \
--maintenance-window-end 2019-08-14T00:00:00-7:00 \
--maintenance-window-recurrence FREQ=WEEKLY

Configuring a maintenance exclusion

To configure a maintenance exclusion, you configure its name (optional), start time, and end time. The maintenance exclusion can span multiple days.

You can configure a maximum of three maintenance exclusions on a cluster at any given time. You can manually remove maintenance exclusions, whether they have occurred or not.

Create a cluster with a maintenance exclusion

You can configure a maintenance exclusion when creating a new cluster by using the Google Cloud Console. This task cannot be performed in the gcloud tool.

To create a new cluster with a maintenance exclusion:

  1. Go to the Google Kubernetes Engine page in the Cloud Console.

    Go to Google Kubernetes Engine

  2. Click Create.

  3. Configure your cluster as desired.

  4. From the navigation pane, under Cluster, click Automation.

  5. Under Maintenance exclusions, click Add Maintenance Exclusion.

  6. Select a Start time and an End time.

  7. Click Create.

You can see an example maintenance exclusion for Black Friday.

Configure a maintenance exclusion for an existing cluster

You can configure a maintenance exclusion on an existing cluster by using the Google Cloud Console or the gcloud tool.

gcloud

To configure a maintenance exclusion for an existing cluster, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --add-maintenance-exclusion-name EXCLUSION_NAME
    --add-maintenance-exclusion-start START_DATE_TIME \
    --add-maintenance-exclusion-end END_DATE_TIME

Replace the following:

  • CLUSTER_NAME: the name of your cluster.
  • EXCLUSION_NAME: the name of the maintenance exclusion.
  • START_DATE_TIME: the start date and time for the exclusion.
  • END_DATE_TIME: the end date and time for the exclusion.

To view supported date and time formats, run gcloud topic datetimes.

Console

To configure a maintenance exclusion for an existing cluster:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to modify.

  3. Under Automation, next to the Maintenance exclusions field, click Edit maintenance exclusions.

  4. Under Maintenance exclusions, click Add Maintenance Exclusion.

  5. Select a Start time and an End time.

  6. Click Save Changes.

Remove a maintenance exclusion

A cluster can have a maximum of three active, non-elapsed maintenance exclusions at any time.

gcloud

To remove an existing maintenance exclusion, run the following command:

gcloud container clusters update CLUSTER_NAME \
    --remove-maintenance-exclusion EXCLUSION_NAME

Replace the following:

  • CLUSTER_NAME: the name of the existing cluster.
  • EXCLUSION_NAME: the name of the maintenance exclusion to remove.

Console

To remove a maintenance exclusion from an existing cluster:

  1. Go to the Google Kubernetes Engine page in Cloud Console.

    Go to Google Kubernetes Engine

  2. In the cluster list, click the name of the cluster you want to modify.

  3. Under Automation, next to the Maintenance exclusions field, click Edit maintenance exclusions.

  4. Under Maintenance exclusions, click Delete item next to the End time field for the exclusion you want to remove.

  5. Click Save Changes.

To see all maintenance exclusions on a cluster, you can view the cluster's maintenance policy.

Example maintenance exclusion

The following example prevents maintenance over the four days encompassing Black Friday through Cyber Monday, the highest-volume sales period of the year for many retail businesses. This example shows how to prevent a maintenance window from occurring from Black Friday 2019 (November 29, 2019) to Cyber Monday 2019 (December 2, 2019), from midnight on the east coast (UTC-5) to 23:59:59 on the west coast (UTC-7).

gcloud container clusters update sample-cluster \
    --add-maintenance-exclusion-name black-friday \
    --add-maintenance-exclusion-start 2019-11-29T00:00:00-05:00 \
    --add-maintenance-exclusion-end 2019-12-02T23:59:59-07:00

View a cluster's maintenance policy

To view a cluster's maintenance policy, including whether it has a maintenance window and all of its maintenance exclusions, use the following command:

gcloud container clusters describe CLUSTER_NAME

What's next