This page shows how to use maintenance windows and maintenance exclusions to control when automatic cluster maintenance, such as auto-upgrades, can and cannot occur on your Google Kubernetes Engine (GKE) clusters.
Before you begin
Before you start, make sure you have performed the following tasks:
- Ensure that you have enabled the Google Kubernetes Engine API. Enable Google Kubernetes Engine API
- Ensure that you have installed the Cloud SDK.
Set up default gcloud
settings using one of the following methods:
- Using
gcloud init
, if you want to be walked through setting defaults. - Using
gcloud config
, to individually set your project ID, zone, and region.
Using gcloud init
If you receive the error One of [--zone, --region] must be supplied: Please specify
location
, complete this section.
-
Run
gcloud init
and follow the directions:gcloud init
If you are using SSH on a remote server, use the
--console-only
flag to prevent the command from launching a browser:gcloud init --console-only
-
Follow the instructions to authorize
gcloud
to use your Google Cloud account. - Create a new configuration or select an existing one.
- Choose a Google Cloud project.
- Choose a default Compute Engine zone.
Using gcloud config
- Set your default project ID:
gcloud config set project PROJECT_ID
- If you are working with zonal clusters, set your default compute zone:
gcloud config set compute/zone COMPUTE_ZONE
- If you are working with regional clusters, set your default compute region:
gcloud config set compute/region COMPUTE_REGION
- Update
gcloud
to the latest version:gcloud components update
Configuring a maintenance window
To configure a maintenance window, you configure when it starts, how long it lasts, and how often it repeats. For example, you can configure a maintenance window that recurs weekly on Monday through Friday.
You can configure a maintenance window using the Google Cloud Console or the
gcloud
command.
Creating a cluster with a simple maintenance window
You can create a simple maintenance window in gcloud
by specifying the
--maintenance-window
flag. This flag allows you to specify a daily 4-hour
maintenance window using a simplified format.
To create a new cluster with a simple maintenance window, run the following command:
gcloud container clusters create cluster-name \
--maintenance-window start-time
where:
- cluster-name is the name of the new cluster.
- start-time is set to a 24-hour timestamp in UTC time, like
16:00
.
The maintenance window runs each day at the specified start-time and runs for four hours.
Creating a cluster with a maintenance window
You can create a new cluster with a maintenance window by using the
gcloud
tool or the Google Cloud Console.
gcloud
To create a new cluster with a maintenance window, run the following command:
gcloud container clusters create cluster-name \
--maintenance-window-start start-time \
--maintenance-window-end end-time \
--maintenance-window-recurrence rrule
where:
- cluster-name is the name of the new cluster.
- start-time is an RFC-5545 DTSTART.
- end-time is specified in the same format as start-time, but is only used to calculate the duration of the maintenance window. The value for end-time must be in the future, relative to start-time.
- rrule is an RFC-5545 RRULE. This is an extremely flexible format with multiple ways to specify recurrence rules.
For example, the following command creates a cluster named my-cluster
with
a maintenance window that starts at 2:00 AM UTC on August 1, 2019, finishes four
hours later, and runs daily. You can learn more about
formatting dates and times.
gcloud container clusters create my-cluster \
--maintenance-window-start 2019-08-01T02:00:00Z \
--maintenance-window-end 2019-08-01T06:00:00Z \
--maintenance-window-recurrence FREQ=DAILY
Console
Visit the Google Kubernetes Engine menu in Cloud Console.
Click add_box Create.
Configure your cluster as desired.
From the navigation pane, under Clusters, click Automation.
Select the Enable Maintenance Window checkbox.
Select the start time and length, then select the days of the week the maintenance window occurs on. To edit the
RRULE
directly, switch to the custom editor.Click Create.
Configuring a maintenance window for an existing cluster
gcloud
To create or update a maintenance window for an existing cluster, run the following command:
gcloud container clusters update cluster-name \
--maintenance-window-start start-time \
--maintenance-window-end end-time \
--maintenance-window-recurrence rrule
where:
- cluster-name is the name of the existing cluster.
- start-time is an RFC-5545 DTSTART.
- end-time is specified in the same format as start-time, but is only used to calculate the duration of the maintenance window. The value for end-time must be in the future, relative to start-time.
- rrule is an RFC-5545 RRULE. This is an extremely flexible format with multiple ways to specify recurrence rules.
Console
To create or modify a maintenance window for an existing cluster:
Visit the Google Kubernetes Engine menu in Cloud Console.
Click the cluster's Edit button, which looks like a pencil.
In the Maintenance Window section, select the start time and length, then select the days of the week the maintenance window occurs on. To edit the
RRULE
directly, switch to the custom editor.Click Save.
Manually finishing incomplete maintenance
If an upgrade or other automatic maintenance takes longer than the maintenance window to complete, GKE attempts to stop ongoing maintenance tasks and resumes them during the next occurrence of the maintenance window. If an automatic upgrade is canceled, and you have node auto-upgrades enabled, your nodes might be in a mixed-version state but your cluster should operate normally.
To manually upgrade your cluster, or cancel or roll back a partial upgrade, visit Manually upgrading a cluster.
Removing a maintenance window
gcloud
To remove a maintenance window from a cluster, run the following command:
gcloud container clusters update cluster-name --clear-maintenance-window
where cluster-name is the name of the existing cluster.
Console
To remove a maintenance window:
Visit the Google Kubernetes Engine menu in Cloud Console.
Click the cluster's Edit button, which looks like a pencil.
In the Maintenance window drop-down list, select Disabled.
Click Save.
Example maintenance windows
The following examples illustrate some of the different ways you can configure a maintenance window. Only the relevant flags are shown, because the flags use the same syntax for creating a new cluster or updating an existing one.
- Weekly on Tuesdays and Wednesdays, starting August 27, 2019, for the entire day
In this example, the difference between the start and end timestamps is a full day, so the maintenance window runs for 24 hours on both Tuesdays and Wednesdays.
--maintenance-window-start 2019-08-27T00:00:00Z \ --maintenance-window-end 2019-08-28T00:00:00Z \ --maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=TU,WE'
- Daily on weekdays from 9:00-17:00 UTC-4
This example shows how to have a daily maintenance window, but skip weekends. This example specifies a non-UTC timezone.
--maintenance-window-start 2019-09-02T09:00:00-04:00 \ --maintenance-window-end 2019-09-02T17:00:00-04:00 \ --maintenance-window-recurrence 'FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR'
- Weekly at 4PM for 8 hours, UTC-7
If you do not specify a timezone for
--maintenance-window-start
, local time is used, as configured in your Google Cloud account. Google Cloud Console always uses local time.--maintenance-window-start 2019-08-13T16:00:00-7:00 \ --maintenance-window-end 2019-08-14T00:00:00-7:00 \ --maintenance-window-recurrence FREQ=WEEKLY
Configuring a maintenance exclusion
To configure a maintenance exclusion, you configure its name (optional), start time, and end time. The maintenance exclusion can span multiple days.
The maintenance exclusion must allow 48 hours of maintenance availability in a 32-day rolling window. The duration of your maintenance windows directly affects the maximum maintenance exclusion and can be calculated using the following formula:
maximum maintenance exclusion (days) = 32-day rolling window - 48 hours maintenance availability / maintenance window hours per day
For example, if your normal maintenance window is 4 hours long, then your maximum maintenance exclusion is 20 days: (32 - 48/4 = 20).
You can configure a maximum of three maintenance exclusions on a cluster at any given time. You can manually remove maintenance exclusions, whether they have occurred or not.
You can configure a maintenance exclusion using the Google Cloud Console or the
gcloud
command.
Creating a cluster with a maintenance exclusion
To create or modify a maintenance exclusion for an existing cluster:
Visit the Google Kubernetes Engine menu in Cloud Console.
Click the cluster's Edit button, which looks like a pencil.
In the Maintenance exclusion section, select Add maintenance exclusion. Select the start and end time.
Click Save.
You can see an example maintenance exclusion for Black Friday.
Removing a maintenance exclusion
A cluster can have a maximum of three active, non-elapsed maintenance exclusions at any time.
gcloud
To remove an existing maintenance exclusion, run the following command:
gcloud container clusters update cluster-name \
--remove-maintenance-exclusion exclusion-name
where:
- cluster-name is the name of the existing cluster.
- exclusion-name is the name of the maintenance exclusion to remove.
Console
To remove a maintenance exclusion from an existing cluster:
Visit the Google Kubernetes Engine menu in Cloud Console.
Click the cluster's Edit button, which looks like a pencil.
In the Maintenance exclusion section, click the X next to the exclusion to remove.
Click Save.
To see all maintenance exclusions on a cluster, you can view the cluster's maintenance policy.
Example maintenance exclusion
The following example prevents maintenance over the four days encompassing Black Friday through Cyber Monday, the highest-volume sales period of the year for many retail businesses. This example shows how to prevent a maintenance window from occurring from Black Friday 2019 (November 29, 2019) to Cyber Monday 2019 (December 2, 2019), from midnight on the east coast (UTC-5) to 23:59:59 on the west coast (UTC-7).
gcloud container clusters update sample-cluster \
--add-maintenance-exclusion-name black-friday \
--add-maintenance-exclusion-start 2019-11-29T00:00:00-05:00 \
--add-maintenance-exclusion-end 2019-12-02T23:59:59-07:00
Viewing a cluster's maintenance policy
To view a cluster's maintenance policy, including whether it has a maintenance window and all of its maintenance exclusions, use the following command:
gcloud container clusters describe cluster-name
What's next
- Learn more about cluster and node upgrades
- Enable node auto-upgrade
- Manually upgrade a cluster