Rolling Out Updates to Managed Instance Groups

A managed instance group contains one or more virtual machine instances that are controlled using an instance template. To update instances in a managed instance group, you can make update requests to the group as a whole, using the Managed Instance Group Updater feature.

To learn more about instance groups, read the Instance Groups Overview.

The Managed Instance Group Updater allows you to easily deploy new versions of software to instances in your managed instance groups, while controlling the speed of deployment, the level of disruption to your service, and the scope of the update. The Updater offers two primary advantages:

• The rollout of an update happens automatically to your specifications, without the need for additional user input after the initial request.
• You can perform partial rollouts which allows for canary testing.

By allowing new software to be deployed inside an existing managed instance group, there is no need for you to reconfigure the instance group or reconnect load balancing, autoscaling, or autohealing each time a new version of software is rolled out. Without the Updater, new software versions must be deployed either by creating a new managed instance group with a new software version, requiring additional set up each time, or through a manual, user-initiated, instance-by-instance recreate. Both of these approaches require significant manual steps throughout the process.

Before you begin

• If you want to use the command-line examples in this guide:
  1. Install or update to the latest version of the gcloud command-line tool.
  2. Set a default region and zone.
• You have a managed instance group.
• You have an instance template you want to update your instance group to.

Starting a basic rolling update

A rolling update is an update that is gradually applied to all instances in an instance group until all instances have been updated. You can control various aspects of a rolling update, such as how many instances can be taken offline for the update, how long to wait between updating instances, whether the update affects all or just a portion of instances, and so on.

Here are things to keep in mind when making a rolling update:

• Updates are intent based. When you make the initial update request, the API returns a successful response to confirm that the request was valid but that does not indicate that the update was successful. You must check the status of the managed instance group to determine if your update was deployed successfully.

• The Instance Group Updater API is a declarative API. The API expects a request to specify the desired post update configuration of the managed instance group, rather than an explicit function call.

• The Updater feature supports up to two instance template versions in your managed instance group. This means that you can specify two different instance template versions for your managed instance group, which is useful for performing canary updates.

To start a basic rolling update where the update is applied to 100% of the instances in the group, follow the instructions below.

Console

1. Go to the Instance groups page in the GCP Console.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Under Template, pull down the drop down menu and select the new template to update to.
5. For the target size, enter 100% to update all instances.
6. Optionally, you can toggle configuration options like whether the update is proactive (the group actively replaces instances) or opportunistic (the group does not actively replace instances but applies the update when instances are replaced from other means). You can also supply max surge, max unavailable options, and minimum wait time.
7. Click Update to start the update.

gcloud

Using the gcloud tool, run the rolling-action start-update command:

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP] \
    --version template=[INSTANCE_TEMPLATE] [--zone [ZONE] | --region [REGION]]

where:

• [INSTANCE_GROUP] is the name of the instance group to update.
• [INSTANCE_TEMPLATE] is the new instance template to update the instance group to.
• Either a [ZONE] or [REGION] for this instance group. If the instance group is a zonal instance group, provide the zone. If it is regional, provide the region.

API

In the API, make a PATCH request to the following URL:

https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instanceGroupManagers/[MANAGED_INSTANCE_GROUP_NAME]

If the instance group is a regional managed instance group, replace zones/[ZONE] with regions/[REGION].

The request payload contains:

• The instance template to update the group to.
• An update policy for the request and other update options.

The following example is the minimal configuration necessary for initiating an update in the API.

If you don't specify otherwise, the maxSurge and maxUnavailable properties default to 1 multiplied by the number of affected zones, which means that the Updater only makes 1 instance unavailable in each affected zone, and only creates 1 additional instance per zone, during the update.

This example request updates 100% of instances to the new instance template.

{
  "instanceTemplate": "global/instanceTemplates/example-template",
  "updatePolicy": {
    "type": "proactive"
   }
 }

After you make a request, you can monitor the update to know when the update has finished.

Configuring options for your update

For more complex updates, you can configure additional options for a specific update request. These options are described below.

Note: If you are making a request directly to the API, some of the following options are required, while other tools like the gcloud tool provide default values if not specified.

Max surge

Set the maxSurge property to allow the Updater to temporarily create new instances above the targetSize during the update. For example, if you set maxSurge to 5, the managed instance group will create up to 5 new instances above your target size, with the new instance template. Setting a higher maxSurge value speeds up your update, at the cost of additional instances, which are billed according to the Compute Engine price sheet.

If you do not set the maxSurge value, the default value is used. For zonal managed instance groups, the default for maxSurge is 1. For regional managed instance groups, the default is [NUMBER_OF_ZONES], where [NUMBER_OF_ZONES] is the number of zones associated with the regional managed instance group.

This option is recognized only when configured with the REPLACE minimal action, but is not supported with the RESTART action setting. You can specify either a fixed number, or specify a percentage if the managed instance group has 10 or more instances. If you set a percentage, the Updater rounds up the number of instances if necessary.

maxSurge only works if you have enough quota or resources to support the additional resources.

Max unavailable

Set the maxUnavailable configuration so that only a certain number of instances are unavailable at any time during the update. For example, if you set maxUnavailable to 5, then only 5 instances will be taken offline for updating at a time. Use this parameter to control how disruptive the update is to your service and to control the rate at which the update is deployed.

This number also includes any instances that are unavailable for other reasons. For example, if the instance group is in the process of being resized up, instances in the middle of being created might be unavailable; these instances would count towards the maxUnavailable number. You can specify either a fixed number, or specify a percentage if the managed instance group has 10 or more instances. If you set a percentage, the Updater rounds down the number of instances, if necessary.

The default value for maxUnavailable in a zonal managed instance group is 1. In a regional managed instance group, the default is [NUMBER_OF_ZONES], the number of zones selected for the regional managed instance group, by default, 3.

Note: If you set both maxSurge and maxUnavailable properties and both properties resolve to 0, the Updater automatically sets maxUnavailable=1, to ensure that the update can always proceed.

Minimum wait time

Note: minReadySeconds is only available in the Beta API and may be deprecated in a future release.

Set the minReadySeconds to specify the amount of time to wait before considering a newly created or restarted instance as updated. Use this feature to control the rate at which the update is deployed. The timer starts when both of the following conditions are satisfied:

• The instance's status is RUNNING.
• If health checking is enabled, when the health check returns HEALTHY.

Note that for the health check to return healthy, the Updater will:

1. Wait for up to the period of time specified by autohealingPolicies.initialDelaySec for the health check to return as HEALTHY.
2. Then, waits for the period of time specified by minReadySeconds.

If the health check does not return HEALTHY within the initialDelaySec, then the Updater declares the VM instance as unhealthy and potentially stops the update. While the VM instance is waiting for verification during the initialDelaySec and the minReadySeconds time period, the instance's currentAction is VERIFYING. However, the underlying VM instance status remains RUNNING.

If there are no health checks for the instance group, then the timer starts when the instance's status is RUNNING. The maximum value for the minReadySeconds property is 3600 seconds (1 hour).

Minimal action

Set the minimal action property to control the minimum action that Updater must perform to update the instances in the group. For example, if you set REPLACE as the minimum action, then all instances that are affected will be deleted and replaced with a new instance, whether or not it is necessary.

Setting a minimal action guarantees that the Updater will perform that action, at a minimum. However, if the Updater determines that the minimal action you specify is not enough to perform the update, it might perform a more disruptive action. For example, if you set RESTART as the minimal action, the Updater attempts to restart instances to apply the update. However, the Updater will perform a more disruptive action if the update requires it. Changing an instance's OS, for example, cannot be done by restarting the instance so the Updater will replace the instances in the instance group with new VM instances.

Applicable actions are REPLACE or RESTART:

• RESTART: Restarts the instance (performs a stop and start request). Note that if your update request requires that the instance be replaced to pick up the changes (for example, changing the image would require that the instance is deleted and replaced), it will be forced to perform REPLACE.

• REPLACE: Deletes the existing instance and creates a new instance from the target template. The Updater creates a brand new instance with all new instance properties, such as new internal and external IP addresses.

The following diagram visualizes how these options affect your instances.

Additional update examples

Here are some command-line examples with common configuration options.

Perform a rolling update of all virtual machine instances, but create up to 5 new instances above the target size at a time

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
    --version template=[NEW_TEMPLATE] --max-surge 5 [--zone [ZONE] | --region [REGION]]

Perform a rolling update with at most 3 unavailable machines and a minimum wait time of 3 minutes before marking a new instance as available

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
    --version template=[NEW_TEMPLATE] --min-ready 3m \
    --max-unavailable 3 [--zone [ZONE] | --region [REGION]]

For example, if you have 1000 instances and you ran this command, the Updater will create up to 100 new instances before it starts to remove instances running the previous instance template.

Perform a rolling update of all virtual machine instances, but create up to 10% new instances above the target size at a time

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
    --version template=[NEW_TEMPLATE] --max-surge 10% [--zone [ZONE] | --region [REGION]]

Starting a canary update

The Instance Group Updater feature allows you to perform canary updates, so that you can test your updates on a random subset of instances before fully committing to the update.

A canary update is an update that is applied to a partial number of instances in the instance group. Canary updates let you test new features or upgrades on a subset of instances, instead of rolling out a potentially disruptive update to all your instances. If an update is not going well, you only need to roll back a small number of instances, minimizing the disruption for your users. From the perspective of the server, a canary update is the same as a standard rolling update, except that the number of instances that should be updated is less than the total size of the instance group. Like a standard rolling update, a canary update is disruptive to the instances affected; that is, the affected instances are deleted and replaced by new VM instances during the update.

To initiate a canary update:

• You specify up to two instance template versions, typically a new instance template to canary and the current instance template for the remainder of the instances. For example, you can specify that 20% of your instances are created based on new-instance-template while the rest of the instances continue to run on old-instance-template. You cannot specify more than two instance templates at a time.

• You must always specify a target size (targetSize) for the canary version. You cannot initiate a canary update if you omit the target size for the canary version. For example, if you specified that 10% of instances should be used for canarying, the remaining 90% will be untouched and use the current instance template.

Console

1. Go to the Instance groups page in the GCP Console.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Click Add template and choose the new instance template to canary.
5. Under Target size, enter the percentage or fixed number of instances you want to use to canary the new instance template.
6. Optionally, you can toggle configuration options like whether the update is proactive (the group actively deletes and replaces instances) or opportunistic (the group does not actively replace instances but applies the update when instances are created from other means). You can also supply max surge, max unavailable options, and minimum wait time.
7. Click Update to start the update.

gcloud

Using the gcloud command-line tool, provide both the current template and the new template to explicitly express how many instances should use each template:

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
    --version template=[CURRENT_TEMPLATE] \
    --canary-version template=[NEW_TEMPLATE],target-size=[SIZE] \
    [--zone [ZONE] | --region [REGION]]

where:

• [CURRENT_TEMPLATE] is the current template that the instance group is running.
• [NEW_TEMPLATE] is the new template you want to canary.
• [SIZE] is the number or percentage of instances you want to apply this update to. You must apply the target-size property to the --canary-version template. You can only set a percentage if the instance group contains 10 or more instances.
• Either a [ZONE] or [REGION] for this instance group. If the instance group is a zonal instance group, provide the zone. If it is regional, provide the region.

For example, the following command performs a canary update that rolls out my-template-b to 10% of instances in the instance group:

gcloud compute instance-groups managed rolling-action start-update my-ig1 \
        --version template=my-template-A --canary-version template=my-template-B,target-size=10%

API

In the API, make a PATCH request to the following URI:

https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instanceGroupManagers/[INSTANCE_GROUP_NAME]

The request payload should contain both the current instance template and the new instance template that you want to canary. For example:

{
 "versions": [
  {
   "instanceTemplate": "global/instanceTemplates/[NEW_TEMPLATE]",
   "targetSize": {
    "[percent|fixed]": [NUMBER|PERCENTAGE] # Use `fixed` for a specific number of instances
   }
  },
  {
   "instanceTemplate": "global/instanceTemplates/[CURRENT_TEMPLATE]"
  }
 ]
}

where:

• [NEW_TEMPLATE] is the name of the new template you want to canary.
• [NUMBER|PERCENTAGE] is the fixed number or percentage of instances to canary this update. You can only set a percentage if the instance group contains 10 or more instances. Otherwise, provide a fixed number instead.
• [CURRENT_TEMPLATE] is the name of the current template that the instance group is running

Rolling forward a canary update

After running a canary update, you can decide if you want to commit the update to 100% of the instance group or roll back.

Console

1. Go to the Instance groups page in the GCP Console.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Under Template, update the target size of the canary template to 100% to roll forward the template to all your instances. Alternatively, you can replace the primary template with the canary template and set the target size to 100%. Then, you can remove the second template field altogether.
5. Click Update to start the update.

gcloud

If you want to commit to your canary update, roll forward the update by making the same update request but setting only version and omitting --canary-version. Using the gcloud command-line tool:

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
    --version template=[NEW_TEMPLATE] [--zone [ZONE] | --region [REGION]]

API

In the API, make a PATCH request to the following URI:

https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instanceGroupManagers/[INSTANCE_GROUP_NAME]

In the request body, specify the new instance template as a version and omit the old instance template from your request body. Omit the target size specification to roll the update out to 100% of instances. For example, your request body would look like this:

{
"versions": [
   {
   "instanceTemplate": "global/instanceTemplates/[NEW_TEMPLATE]" # New instance template
   }
 ]
}

Replace [NEW_TEMPLATE] with the name of the new instance template you want to roll forward.

Starting an opportunistic or proactive update

By default, updates made using the gcloud command-line tool are proactive and updates initiated in the API are opportunistic.

For proactive updates, Compute Engine actively schedules actions to apply the requested updates to instances as necessary. In many cases, this often means deleting and recreating instances proactively.

Alternatively, you can choose to perform an opportunistic update, if a proactive update is potentially too disruptive. An opportunistic update is only applied when you manually initiate the update on selected instances or when new instances are created by the managed instance group. The managed instance group creates new instances when the managed instance group is resized either by another service, such as an autoscaler, or manually by a user. Compute Engine does not actively initiate requests to apply opportunistic updates.

In certain scenarios, an opportunistic update is useful because you don't want to cause instability to the system if it can be avoided. For example, if you have a non-critical update that can be applied as necessary without any urgency and you have a managed instance group that is actively being autoscaled, perform an opportunistic update so that Compute Engine does not actively tear down your existing instances to apply the update.

To choose whether an update is opportunistic or proactive, set the type property to OPPORTUNISTIC or PROACTIVE using either the gcloud command-line tool or the API.

Console

1. Go to the Instance groups page in the GCP Console.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Under Template, choose a template to update your instance group to and select a target size for the template.
5. Under Update mode, choose between either an opportunistic or proactive update.
6. Click Update to start the update.

gcloud

Using the gcloud command-line tool:

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
    --version template=[INSTANCE_TEMPLATE] \
    --type [opportunistic|proactive] [--zone [ZONE] | --region [REGION]]

API

In the request payload to start an update, include the type property in your updatePolicy:

{
"updatePolicy": {
  "type": "PROACTIVE" # Performs a proactive update
},
"versions": [{
  "instanceTemplate": "global/instanceTemplates/[NEW_TEMPLATE]",
  }]
}

where [NEW_TEMPLATE] is the name of the new template you want to canary. If you want an opportunistic update, replace PROACTIVE with OPPORTUNISTIC.

Updating selected instances

Beta

This is a Beta release of Updating selected instances. This feature is not covered by any SLA or deprecation policy and might be subject to backward-incompatible changes.

When you start an opportunistic update, you must wait for Compute Engine to roll out the update as opportunities arise. However, if you want more control over the rollout, you can manually initiate that update immediately on specific instances in your managed instance group.

Note: Set your update type to OPPORTUNISTIC to prevent a race between manually initiated updates and automatic proactive updates.

Manually initiating updates allows you to:

• Select the instances that you would like to update.
• Apply an update with the least of amount of disruption that is necessary for the update to complete. For example, if you are only updating metadata, it might not be necessary to restart the instance to complete the update. When manually initiating the update, the minimal necessary action is automatically performed.
• Enforce instance restart or recreation even if those actions are not necessary for applying the update. For example, you might want to restart a VM even if you are only updating its metadata because your guest software has to pick up the new metadata on VM boot.
• Prevent an update if it requires more disruption than you can afford.
• Perform an update of all selected instances immediately, without restricting the rollout by maxSurge and maxUnavailable constraints.

Minimal action and most disruptive allowed action

Depending on the nature of an update, it might disrupt the instance's state. For example, changing an instance's machine type requires the VM to restart while changing its boot image requires deleting and replacing the instance.

You can control the level of disruption by using the minimal-action and most-disruptive-allowed-action flags:

• minimal-action allows you to force a more disruptive action than is necessary.
• most-disruptive-allowed-action allows you to prevent an update if it requires more disruption than you can afford.

When updating selected instances, both of these flags accept the following actions:

Action	Description	Which instance properties can be updated?
NONE	Do not disrupt the instance at all.	None
REFRESH	Do not stop the instance.	Secondary disks, instance metadata, labels
RESTART	Stop the instance and start it again.	Machine type
REPLACE	Delete the instance and create it again.	All instance properties stored in the instance template.

By default, the minimal-action is NONE. If your update requires a more disruptive action than you set with this flag, Compute Engine performs the necessary action to execute the update.

By default, the most-disruptive-allowed-action is REPLACE. If your update requires a more disruptive action than the actoin you set with this flag, the update request will fail. For example, if you set "restart" as the most disruptive allowed action, then an attempt to update the boot disk image will fail because that update requires instance recreation, a more disruptive action than a restart.

Note: The most disruptive allowed action cannot be less disruptive than the minimal action.

You can update selected instances with the gcloud tool or the API.

gcloud

After starting an opportunistic update, use the update-instances sub-command to apply the update to specific instances.

gcloud beta compute instance-groups managed update-instances [INSTANCE_GROUP] \
    --instances [INSTANCE_NAMES] \
    --most-disruptive-allowed-action [DISRUPTION_LEVEL] \
    --minimal-action [DISRUPTION_LEVEL]

Where:

• [INSTANCE_GROUP] is the name of the instance group with pending updates.
• [INSTANCE_NAMES] is a list of instances to immediately update.
• [DISRUPTION_LEVEL] is the minimal or maximal disruption level: NONE, REFRESH, RESTART, REPLACE
  • The default minimal-action is NONE.
  • The default most-disruptive-allowed-action is REPLACE.

If you need to wait until all the specified instances are updated, check the group's status and wait until the group is stable.

API

After starting an opportunistic update, construct a POST request to the Beta regionInstanceGroupManagers.applyUpdatesToInstances method. For a zonal managed instance group, use the zonal instanceGroupManagers.applyUpdatesToInstances method.

POST https://www.googleapis.com/compute/beta/projects/[PROJECT]/regions/[REGION]/instanceGroupManagers/[INSTANCE_GROUP_NAME]/applyUpdatesToInstances
{
  "instances": "zones/[ZONE]/instances/[INSTANCE_NAME]","zones/[ZONE]/instances/[INSTANCE_NAME]"
  "minimalAction": [DISRUPTION_LEVEL],
  "mostDisruptiveAllowedAction": [DISRUPTION_LEVEL]
}

Where:

• [INSTANCE_GROUP_NAME] is the name of the instance group with pending updates.
• [ZONE] is the zone of an instance to immediately update.
• [INSTANCE_NAME] is the name of an instance to immediately update.
• [DISRUPTION_LEVEL] is the minimal or maximal disruption level: NONE, REFRESH, RESTART, REPLACE
  • The default minimalAction is NONE.
  • The default mostDisruptiveAllowedAction is REPLACE.

Similar to other managed instance group methods, applyUpdatesToInstances is intent-based, which means it returns an operation response. After the operation is DONE, listManagedInstances contains the list of instances with their currentAction fields changed to REFRESHING, RESTARTING or RECREATING. In case the operation fails, for example, due to concurrent changes to the group, the failure will be noted in the lastAttempt.errors field.

If you need to wait until all the specified instances are finished updating, check the group's status and wait until the group is stable.

Performing a rolling replace or restart

Alternatively, you can use the restart or replace commands to perform a rolling restart or a rolling replace of VM instances in the managed instance group. Similar to the start-update command, you can specify any of the configuration options for a restart or a replace. A rolling restart or replace does not change anything about the instance group, including the instance template. It simply refreshes the instances in the group using the method you choose.

There are many reasons why you might want a rolling replace or a rolling restart. For example, you might want to refresh your VM instances from time to time to:

• Clear up memory leaks.
• Restart your application so it can run from a fresh machine.
• Force a periodic replace as a best practice to test your VMs.
• Update your VM's operating system image or rerun startup scripts to update your software.

To perform a replace where all instances are deleted and replaced with new instances:

Console

1. Go to the Instance groups page in the GCP Console.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling restart/replace.
4. Choose whether you want to restart or replace your instances. Restarting performs the stop and start methods on the instances, while a replace actively deletes and creates instances.
5. Optionally, you can toggle configuration options like max surge, max unavailable options, and minimum wait time.
6. Click Restart or Replace button to start the update.

gcloud

gcloud compute instance-groups managed rolling-action replace [INSTANCE_GROUP]

This command replaces all instances in the managed instance group, one at a time. If a full replace is too disruptive, you can specify a rolling restart instead, which does not delete any instances and just restarts each instance.

gcloud compute instance-groups managed rolling-action restart [INSTANCE_GROUP]

You can further customize each of these commands with the same options available for updates (for example: maxSurge, maxUnavailable, min-ready).

API

In the API, make a PATCH request for the group and set a proactive updatePolicy where the minimalAction is either RESTART or REPLACE depending whether you want to perform a rolling replace, where each instance is deleted and replaced with a new instance, or a rolling restart, where each instance is stopped and restarted. In both cases of RESTART and REPLACE, you must always provide a versions.instanceTemplate and a versions.name property (for example, v2) to trigger the action.

A rolling replace might look like the following:

PATCH https://www.googleapis.com/compute/v1/projects/myproject/zones/[ZONE]/instanceGroupManagers/[INSTANCE_GROUP_NAME]

{
 "updatePolicy": {
  "minimalAction": "REPLACE",
  "type": "PROACTIVE"
 },
 "versions": [
  {
   "instanceTemplate": "global/instanceTemplates/example-template",
   "name": "v2"
  }
 ]
}

For a rolling restart:

PATCH https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instanceGroupManagers/[INSTANCE_GROUP_NAME]

{
 "updatePolicy": {
  "minimalAction": "RESTART",
  "type": "PROACTIVE"
 },
 "versions": [
  {
   "instanceTemplate": "global/instanceTemplates/example-template",
   "name": "v2"
  }
 ]
}

Additional replace/restart examples

Perform a rolling restart of all virtual machines, two at a time

This command restarts all virtual machines in the instance group, two at a time. Notice that no new instance template is specified.

gcloud compute instance-groups managed rolling-action restart [INSTANCE_GROUP_NAME] \
    --max-unavailable 2 [--zone [ZONE] | --region [REGION]]

Rolling restart all VMs as quickly as possible

gcloud compute instance-groups managed rolling-action restart [INSTANCE_GROUP_NAME] \
    --max-unavailable 100% [--zone [ZONE] | --region [REGION]]

Rolling replace all VMs as quickly as possible

gcloud compute instance-groups managed rolling-action replace [INSTANCE_GROUP_NAME]  \
    --max-unavailable 100% [--zone [ZONE] | --region [REGION]]

Updating a regional managed instance group

A regional managed instance group contains virtual machine instances spread across multiple zones within a region, as opposed to zonal managed instance groups, which only contain instances in one zone. Regional managed instance groups give you the ability to distribute your instances across more than one zone to improve your application's availability and to support extreme cases where one zone fails or an entire group of instances stops responding.

Updating a regional managed instance group using the Instance Group Updater feature is same as performing an update on a zonal managed instance group, with a few exceptions described below. When you initiate an update to a regional instance group, the Updater always updates instances proportionally and evenly across each zone; it is not possible to choose which instances in which zones are updated first or to update instances in only one zone.

Differences between updating a regional versus zonal managed instance group

Before you initiate an update to a regional managed instance group, you should be aware that there are a number of things that behave differently than for zonal managed instance groups:

• The default update parameters for regional managed instance groups are maxUnavailable=[NUMBER_OF_ZONES] and maxSurge=[NUMBER_OF_ZONES], where [NUMBER_OF_ZONES] is the number of zones selected for the regional managed instance group, by default, 3.

• If you are using fixed numbers when specifying an update, the fixed number must be either 0 or equal to or greater than the number of zones associated with the regional managed instance group. For example, if the instance group is distributed across three zones, then you cannot set maxSurge to 1 or to 2 because the Updater needs to create an additional instance in each of the three zones.

Using a fixed number or a percentage in update requests

If you specify a fixed number in your update requests, the number you specify is divided by the number of zones in the regional managed instance group and distributed evenly. For example, if you specify maxSurge=10, then the Updater divides 10 across the number of zones in the region and create new instances based on that number. If the number of instances does not divide evenly across zones, the Updater adds the additional instances to a random zone. So, for 10 instances across three zones, two of the zones will get 3 instances and one zone will get 4 instances. The same logic is applied to the maxUnavailable and the targetSize parameter for canary updates.

You can only specify a percentage if your managed instance group contains 10 or more VM instances. Percentages are handled slightly differently depending on the situation:

• If you specify a percentage of VM instances for a canary update, the Updater attempts to distribute the instances evenly across zones. The remainder is rounded either up or down in each zone but the total difference will not be more than 1 VM instance per group. For example, for a managed instance group with 10 instances and a target size percentage of 25%, the update will be rolled out to 2 to 3 VM instances.

• If you specify a percentage for update options like maxSurge and maxUnavailable, the percentages are rounded independently per zone. The same rules that apply to updating zonal managed instance groups apply here.

Monitoring updates

After you initiate a rolling update, it can take some time for the update to finish. You can monitor the progress of an update by inspecting the status of the managed instance group, or by checking the currentAction on each instance in the managed instance group.

Group status

At the group level, Compute Engine populates a read-only field called status that contains a versionTarget.isReached flag and an isStable flag. You can use the gcloud tool or the API to access these flags.

To verify that your update to the instance group has finished rolling out, check that status.versionTarget.isReached==true. To verify that all instances in the group are running and healthy (that is, that the currentAction of each managed instance is NONE), check that status.isStable==true. Remember that the stability of a managed instance group depends on group configurations beyond the Updater; for example, if your group is autoscaled, and if it is currently scaling up, then isStable==false due to the autoscaler operation.

You can also use the console to see the current and target number of instances being updated.

Console

You can monitor a rolling update for a group by going to the details page of the specific instance group.

1. Go to the Instance groups page in the GCP Console.

   Go to the Instance groups page

2. Select the instance group you want to monitor. The overview page for the instance group shows the template that each instance is using.
3. To view the details, click the Details tab.

The Details page shows the current and target number of instances being updated for each instance template and also displays the update parameters.

gcloud

gcloud compute instance-groups managed describe [INSTANCE_GROUP_NAME] \
    [--zone [ZONE] | --region [REGION]]

The gcloud tool returns detailed information about the instance group including the status.versionTarget.isReached and status.isStable fields.

API

In the API, make a POST request to the following URI:

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instanceGroupManagers/[INSTANCE_GROUP_NAME]/get

If the instance group is a regional managed instance group, replace zones/[ZONE] with regions/[REGION].

The API returns detailed information about the instance group including the status.versionTarget.isReached and status.isStable fields.

status.versionTarget.isReached

The rollout of an update is considered complete when all VM instances in the group have been or are being created in their target version. In the case of a full rollout, all instances are configured to use the new instance template. In the case of a partial rollout, the instances are configured according to the specified target split between instance templates.

You can verify whether the rollout of an update is complete by checking the value of status.versionTarget.isReached field of an instanceGroupManagers (or regionInstanceGroupManagers) resource.

status.versionTarget.isReached is set to true if all VM instances have been or are being created using the target version (versions[]).

status.versionTarget.isReached is set to false when at least one VM is not yet using the target version (versions[]). Or, in the case of a canary update, status.versionTarget.isReached is set to false when the number of VMs using a target version (versions[].instanceTemplates) does not match its target size (versions[].targetSize).

You can also use the gcloud beta compute instance-groups managed wait-until command with the --version-target-reached flag to wait until status.versionTarget.isReached is set to true for the group:

gcloud beta compute instance-groups managed wait-until INSTANCE_GROUP_NAME \
    --version-target-reached \
    [--zone [ZONE] | --region [REGION]]

Note: To verify that the update rollout is complete and also that all VM instances in the group are running and healthy, check the status.isStable flag in addition to status.versionTarget.isReached.

status.isStable

You can verify that a managed instance group is running and healthy by checking the value of the status.isStable field of the associated instanceGroupManagers or regionInstanceGroupManagers resource.

status.isStable set to false indicates that changes are active, pending, or that the managed instance group itself is being modified.

status.isStable set to true indicates the following:

• None of the instances in the managed instance group are undergoing any type of change and the currentAction for all instances is NONE.
• No changes are pending for instances in the managed instance group.
• The managed instance group itself is not being modified.

Managed instance groups can be modified in numerous ways. For example:

• You make a request to roll out a new instance template.
• You make a request to create, delete, resize or update instances in the group.
• An autoscaler requests to resize the group.
• An autohealer resource is replacing one or more unhealthy instances in the managed instance group.
• In a regional managed instance group, some of the instances are being redistributed.

As soon as all actions are finished, status.isStable is set to true again for that managed instance group.

You can also use the gcloud beta compute instance-groups managed wait-until command with the --stable flag, to wait until status.isStable is set to true for the group:

gcloud beta compute instance-groups managed wait-until INSTANCE_GROUP_NAME \
    --stable \
    [--zone [ZONE] | --region [REGION]]

Current actions on instances

To view current the currentAction being performed on each instance in your group, as well as the status of each instance, you can use the gcloud tool or the API.

gcloud

gcloud compute instance-groups managed list-instances [INSTANCE_GROUP_NAME] [--filter="zone:([ZONE])" | --filter="region:([REGION])"]

gcloud returns a response with a list of instances in the instance group and their respective statuses and current actions. For example:

NAME               ZONE           STATUS   ACTION    INSTANCE_TEMPLATE  VERSION_NAME  LAST_ERROR
vm-instances-9pk4  us-central1-f           CREATING  my-new-template
vm-instances-h2r1  us-central1-f           DELETING  my-old-template
vm-instances-j1h8  us-central1-f  RUNNING  NONE      my-old-template
vm-instances-ngod  us-central1-f  RUNNING  NONE      my-old-template

API

In the API, make a POST request to the following URI:

POST https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instanceGroupManagers/[INSTANCE_GROUP_NAME]/listManagedInstances

If the instance group is a regional managed instance group, replace zones/[ZONE] with regions/[REGION].

The API returns a list of instances for the group including each instance's instanceStatus and currentAction.

{
 "managedInstances": [
  {
   "instance": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances/vm-instances-prvp",
   "id": "5317605642920955957",
   "instanceStatus": "RUNNING",
   "instanceTemplate": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/instanceTemplates/[INSTANCE_TEMPLATE_NAME]",
   "currentAction": "REFRESHING"
  },
  {
   "instance": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances/vm-instances-pz5j",
   "currentAction": "DELETING"
  },
  {
   "instance": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances/vm-instances-w2t5",
   "id": "2800161036826218547",
   "instanceStatus": "RUNNING",
   "instanceTemplate": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/global/instanceTemplates/[INSTANCE_TEMPLATE_NAME]",
   "currentAction": "REFRESHING"
  }
 ]
}

For each instance in a managed instance group, the status of the instance is described by its instanceStatus field. To see a list of valid instanceStatus fields, see the documentation for Checking Instance Status.

If the instance is undergoing some type of change, the currentAction field will also be populated with one of the following actions to help you track the progress of the change. Otherwise, the currentAction field will be NONE.

Possible currentAction values are:

• ABANDONING: The instance is being removed from the managed instance group.
• CREATING: The instance is in the process of being created.
• CREATING_WITHOUT_RETRIES: The instance is being created without retries; if the instance fails to be created on the first try, the managed instance group will not try to replace the instance again.
• DELETING: The instance is in the process of being deleted.
• RECREATING: The instance was deleted and is being replaced.
• REFRESHING: The instance is being removed from its current target pools and being readded to the list of current target pools (this list might be the same or different from existing target pools).
• RESTARTING: The instance is in the process of being restarted using the stop and start methods.
• VERIFYING: The instance has been created and is in the process of being verified.
• NONE: No actions are being performed on the instance.

When an instance is successfully updated, its instanceStatus field will reflect the instance's current state.

Note: the pendingActions field is deprecated and will be eventually removed. Please contact [e-mail] to leave feedback if your workload relies on this field.

Rolling back an update

There is no explicit command for rolling back an update to a previous version but if you decide that you want to roll back an update (either a fully committed or canary update), you can do so by making a new update request and passing in the instance template that you want to roll back to.

For example, the following command rolls back an update as fast as possible:

gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
    --version template=[OLD_INSTANCE_TEMPLATE] --max-unavailable 100% [--zone [ZONE] | --region [REGION]]

Replace [OLD_INSTANCE_TEMPLATE] with the name of the old instance template you want to roll back to.

In the API, make a PATCH request to the following URI:

https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instanceGroupManagers/[INSTANCE_GROUP_NAME]

If the instance group is a regional managed instance group, replace zones/[ZONE] with regions/[REGION].

In the request body, specify the old instance template as a version:

{ "updatePolicy":
  {
    "maxUnavailable":
    {
      "percent": 100
    }
  },
 "versions": [
    {
      "instanceTemplate": "global/instanceTemplates/[OLD_TEMPLATE]" # Old instance template
    }
   ]
}

If it's a regional managed instance group with less than 10 instances, you must use a fixed value for maxUnavailable and set the value to the number of instances in the group:

{ "updatePolicy":
  {
    "maxUnavailable":
    {
      "fixed": [NUMBER_OF_INSTANCES_IN_GROUP]
    }
  },
 "versions": [
    {
      "instanceTemplate": "global/instanceTemplates/[OLD_TEMPLATE]" # Old instance template
    }
   ]
}

The Instance Group Updater service treats this as a regular update request so all of the update options described in this document can be specified with your request.

Caution: For canary updates, Google strongly recommends fully rolling back canary updates before you try to canary another version. Since there is no way to selectively choose which instances are updated based on which instance template, performing another canary update on top of an existing update might cause unintended consequences. As a best practice, you should fully roll back a canary update first and restart the canary process again from the beginning.

Controlling the speed of an update

By default, when you make an update request, the service performs the update as fast as possible. If you aren't sure you want to apply an update fully or are tentatively testing your changes, you can control the speed of the update using the following methods.

1. Start a canary update rather than a full update.
2. Set a large minReadySeconds value. Setting this value causes the service to wait this number of seconds before considering the instance successfully updated and proceeding to the next instance.
3. Enable health checking to cause the service to wait for your application to start and to report a healthy signal, before considering the instance successfully updated and proceeding to the next instance.
4. Set a low maxUnavailable and maxSurge value. This ensures that only a minimal number of instances are updated at a time.
5. Manually initiate updates on specific instances to update those instances immediately.

You can also use a combination of these parameters to control the rate of your update.

Stopping an update

There is no explicit method or command to stop an update. You can change an update from proactive to opportunistic and if the managed instance group is not being resized by other services like autoscaler, the change to opportunistic will effectively "stop" the update.

To change an update from proactive to opportunistic, run the following command:

gcloud compute instance-groups managed rolling-action stop-proactive-update [INSTANCE_GROUP_NAME] \
    [--zone [ZONE] | --region [REGION]]

If you decide you want to stop the update completely after converting it from proactive to opportunistic, you can stop it using these steps:

1. Make a request to determine how many instances have been updated.

   gcloud compute instance-groups managed list-instances [INSTANCE_GROUP_NAME] \
       [--zone [ZONE] | --region [REGION]]

   The gcloud tool returns a response with a list of instances in the instance group and their current statuses:

   NAME               ZONE           STATUS   ACTION    INSTANCE_TEMPLATE  VERSION_NAME  LAST_ERROR
   vm-instances-9pk4  us-central1-f  RUNNING  NONE      my-new-template
   vm-instances-j1h8  us-central1-f  RUNNING  NONE      my-new-template
   vm-instances-ngod  us-central1-f  RUNNING  NONE      my-old-template
   

   In this example, two instances have already been updated.

2. Next, make a request to perform a new "update" but pass in the number of instances that have already been updated as the target size:

   gcloud compute instance-groups managed rolling-action start-update [INSTANCE_GROUP_NAME] \
       --version template=my-old-template \
       --canary-version template=my-new-template,target-size=2 \
       [--zone [ZONE] | --region [REGION]]

   To the Updater service, this update appears "complete", so no other instances are updated, effectively stopping the update.

Relationship between versions and instanceTemplate properties for a managed instance group

Google recommends using the versions field to configure instance templates for managed instance groups.

The legacy instanceTemplate field overlaps in functionality with versions as both fields allow you to specify which instance template the managed instance group will use to create new instances. However, only the versions field allows you to specify an advanced, two template (canary) configuration.

For backwards compatibility, managed instance groups continue to support setting the top-level instanceTemplate field, even though we recommend that you switch to using only the versions field. Using both fields at the same time can lead to ambiguity and confusion.

If you specify both the instanceTemplate field and the versions field when calling the update() or patch() method, there are three possibilities of what could happen:

• You set both fields to the same value.

  This is a valid request. In this case, it creates no ambiguity and the new instance template is applied to the managed instance group.

  For example, in the following request, the top-level instanceTemplate and the versions field specify the same instance template which is different from the existing current template. The managed instance group will be updated to the new instance template:

  {
   "instanceTemplate": "global/instanceTemplates/new-template",
   "versions": [
    {
     "instanceTemplate": "global/instanceTemplates/new-template"
    }
   ],
   "updatePolicy": {
     "type": "PROACTIVE"
   }
  }
  

• You set both fields that do not match but only one value is different from the current instance template in the managed instance group.

  This is a valid request. The field that's different from the current setting is taken as the intended value. For example, you call the get() method, change one of the two fields and then call update() with just one changed field:

  {
   "instanceTemplate": "global/instanceTemplates/current-template",
   "versions": [
    {
     "instanceTemplate": "global/instanceTemplates/new-template"
    }
   ],
   "updatePolicy": {
     "type": "PROACTIVE"
   }
  }
  

• You set both fields but they don't match, have different values, and both are different from the current instance template in the managed instance group.

  This setting is invalid and will return an error as there is no clear intent.

  {
   "instanceTemplate": "global/instanceTemplates/new-template",
   "versions": [
    {
     "instanceTemplate": "global/instanceTemplates/a-different-new-template"
    }
   ],
   "updatePolicy": {
     "type": "PROACTIVE"
   }
  }
  

The versions field, instanceTemplate field, and the get() method

If you only specify one instance template, either through the top-level instanceTemplate field or through the versions field or through both, the get() method returns both fields in its response. This makes the new versions field backwards compatible. As long as you specify a single instance template in either of these fields, there is no change to what get() returns in the instanceTemplate field.

If the versions field has two instance templates specified, the get() method will return an empty top-level instanceTemplate field. There is no way to unambiguously express a canary, two instance template configuration in a the top-level instanceTemplate field so the field is not used during a canary update.

The versions field and the setInstanceTemplate() method

For backwards compatibility, the setInstanceTemplate() method behaves as it has previously, allowing you to change the template the managed instance group uses to create new instances. When you call this method, the versions field is overridden with the instance template specified by the setInstanceTemplate() method.

Furthermore, the method sets the updatePolicy to OPPORTUNISTIC. This prevents the managed instance group from actively deploying an instance template which was not explicitly specified in the versions field.

What's next

• Learn more about instance groups.
• Learn more about instance templates.