Rolling out updates to MIGs

A managed instance group (MIG) contains one or more virtual machine (VM) instances that are created by using an instance template. To update instances in a MIG, you can make update requests to the group as a whole, using the MIG Updater feature.

The managed instance group Updater lets you deploy new versions of software to instances in your managed instance groups, while controlling the speed of deployment, the level of disruption to your service, and the scope of the update. The Updater offers two primary advantages:

• The rollout of an update happens automatically to your specifications, without the need for additional user input after the initial request.
• You can perform partial rollouts, which allows for canary testing.

By allowing new software to be deployed inside an existing managed instance group, there is no need for you to reconfigure the instance group or reconnect load balancing, autoscaling, or autohealing each time a new version of software is rolled out. Without the Updater, new software versions must be deployed either by creating a new managed instance group with a new software version, requiring additional set up each time, or through a manual, user-initiated, instance-by-instance recreate. Both of these approaches require significant manual steps throughout the process.

Before you begin

• If you want to use the command-line examples in this guide:
  1. Install or update to the latest version of the gcloud command-line tool.
  2. Set a default region and zone.
• If you want to use the API examples in this guide, set up API access.
• You have a managed instance group.
• You have an instance template you want to update your instance group to.

Starting a basic rolling update

A rolling update is an update that is gradually applied to all instances in an instance group until all instances have been updated. You can control various aspects of a rolling update, such as how many instances can be taken offline for the update, how long to wait between updating instances, whether the update affects all or just a portion of instances, and so on.

Here are things to keep in mind when making a rolling update:

• Updates are intent based. When you make the initial update request, the API returns a successful response to confirm that the request was valid, but that doesn't indicate that the update was successful. You must check the status of the managed instance group to determine whether your update was deployed successfully.

• The Instance Group Updater API is a declarative API. The API expects a request to specify the desired post update configuration of the managed instance group, rather than an explicit function call.

• The Updater supports up to two instance template versions in your managed instance group. This means that you can specify two different instance template versions for your managed instance group, which is useful for performing canary updates.

To start a basic rolling update where the update is applied to 100% of the instances in the group, follow the instructions below.

Console

1. In the Cloud Console, go to the Instance groups page.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Under Template, click the drop down list and select the new template to update to.
5. For the target size, enter 100% to update all instances.
6. Optionally, you can toggle configuration options like whether the update is proactive (the group actively replaces instances) or opportunistic (the group does not actively replace instances but applies the update when instances are replaced from other means). You can also supply max surge, max unavailable options, and minimum wait time.
7. Click Update to start the update.

gcloud

Using the gcloud command-line tool, run the rolling-action start-update command:

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=instance-template-name
    [--zone zone | --region region]

Replace the following:

• instance-group-name: The name of the instance group to update.
• instance-template-name: The new instance template to update the instance group to.
• zone: The zone for this instance group, if the instance group is a zonal instance group.
• region: The region for this instance group, if the instance group is a regional instance group.

API

In the API, make a PATCH request to the instance group manager resource:

PATCH https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name

If the instance group is a regional managed instance group, replace zones/zone with regions/region.

The request payload contains:

• The instance template to update the group to.
• An update policy for the request and other update options.

The following example is the minimal configuration necessary for initiating an update in the API.

If you don't specify otherwise, the maxSurge and maxUnavailable properties default to 1 multiplied by the number of affected zones, which means that the Updater only makes 1 instance unavailable in each affected zone, and only creates 1 additional instance per zone, during the update.

This example request updates 100% of the instances to the new instance template.

{
  "instanceTemplate": "global/instanceTemplates/example-template",
  "updatePolicy": {
    "type": "proactive"
   }
 }

After you make a request, you can monitor the update to know when the update has finished.

Configuring options for your update

For more complex updates, you can configure additional options for a specific update request. These options are described below.

Note: If you are making a request directly to the API, some of the following options are required, while other tools like the gcloud tool provide default values if not specified.

Max surge

Set the maxSurge property to allow the Updater to temporarily create new instances above the targetSize during the update. For example, if you set maxSurge to 5, the managed instance group uses the new instance template to create up to 5 new instances above your target size. Setting a higher maxSurge value speeds up your update, at the cost of additional instances, which are billed according to the Compute Engine price sheet.

If you don't set the maxSurge value, the default value is used. For zonal managed instance groups, the default for maxSurge is 1. For regional managed instance groups, the default is the number of zones associated with the regional managed instance group.

This option is recognized only when configured with the REPLACE minimal action, but is not supported with the RESTART action setting. You can specify either a fixed number, or, if the managed instance group has 10 or more instances, a percentage. If you set a percentage, the Updater rounds up the number of instances if necessary.

maxSurge only works if you have enough quota or resources to support the additional resources.

Max unavailable

Set the maxUnavailable configuration so that only a certain number of instances are unavailable at any time during the update. For example, if you set maxUnavailable to 5, then only 5 instances are taken offline for updating at a time. Use this parameter to control how disruptive the update is to your service and to control the rate at which the update is deployed.

This number also includes any instances that are unavailable for other reasons. For example, if the instance group is in the process of being resized up, instances in the middle of being created might be unavailable; these instances would count toward the maxUnavailable number. You can specify either a fixed number, or, if the managed instance group has 10 or more instances, a percentage. If you set a percentage, the Updater rounds down the number of instances, if necessary.

The default value for maxUnavailable depends on the type of group you have.

• For zonal MIGs, the default is 1.
• For regional MIGs, the default is the number of zones that your MIG is distributed across, which is usually 3 but you might select a different number of zones.

Note: If you set both maxSurge and maxUnavailable properties and both properties resolve to 0, the Updater automatically sets maxUnavailable=1, to ensure that the update can always proceed.

Minimum wait time

Note: minReadySeconds is only available in the beta API and might be deprecated in a future release.

Set minReadySeconds to specify the amount of time to wait before considering a new or restarted instance as updated. Use this feature to control the rate at which the update is deployed. The timer starts when both of the following conditions are satisfied:

• The instance's status is RUNNING.
• If health checking is enabled, when the health check returns HEALTHY.

Note that for the health check to return healthy, the Updater:

1. Waits for up to the period of time specified by autohealingPolicies.initialDelaySec for the health check to return as HEALTHY.
2. Then, waits for the period of time specified by minReadySeconds.

If the health check doesn't return HEALTHY within the initialDelaySec, then the Updater declares the VM instance as unhealthy and potentially stops the update. While the VM instance is waiting for verification during the initialDelaySec and the minReadySeconds time period, the instance's currentAction is VERIFYING. However, the underlying VM instance status remains RUNNING.

If there are no health checks for the instance group, then the timer starts when the instance's status is RUNNING. The maximum value for the minReadySeconds property is 3600 seconds (1 hour).

Minimal action

Set the minimal action property to specify the minimum action that the Updater must perform to update the instances in the group. For example, if you set REPLACE as the minimum action, then all instances that are affected are deleted and replaced with new instances, whether or not it is necessary.

Setting a minimal action guarantees that the Updater performs that action, at a minimum. However, if the Updater determines that the minimal action you specify is not enough to perform the update, it might perform a more disruptive action. For example, if you set RESTART as the minimal action, the Updater attempts to restart instances to apply the update. However, the Updater performs a more disruptive action if the update requires it. Changing an instance's OS, for example, can't be done by restarting the instance so the Updater replaces the instances in the instance group with new VM instances.

Applicable actions are REPLACE or RESTART:

• RESTART. Restarts the instance (performs a stop and start request). If your update request requires that the instance be replaced to pick up the changes (for example, changing the image requires that the instance be deleted and replaced), the Updater performs a REPLACE.

• REPLACE. Deletes the existing instance and creates an instance from the target template. The Updater creates an instance with all new instance properties, such as new internal and external IP addresses.

The following diagram shows how these options affect your instances.

Replacement method

By default, when you update a MIG, the group deletes your VM instances and swaps them with new instances with new names. If you need to preserve the names of your VM instances, you can use the replacementMethod setting.

Preserving existing instance names might be useful if you have applications or systems that rely on using specific instance names. For example, some applications, like Memcached, rely on instance names because they don't have a discovery service; as a result, whenever an instance name changes, the application loses connection to that specific VM.

To preserve instance names, set the replacement method to recreate instead of substitute.

Valid updater replacementMethod values are:

• substitute (default).

  • Replaces VM instances faster during updates because new VMs are created before old ones are shut down. However, instance names aren't preserved because the names are still in use by the old instances.

• recreate

  • Preserves instance names through an update. Compute Engine releases the instance name as the old VM is shut down. Then Compute Engine creates a new instance using that same name.
  • To use this mode, you must set maxSurge to 0.

For more information, see Preserving instance names.

Additional update examples

Here are some command-line examples with common configuration options.

Perform a rolling update of all VM instances, but create up to 5 new instances above the target size at a time

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=new-template \
    --max-surge 5 \
    [--zone zone | --region region]

Perform a rolling update with at most 3 unavailable machines and a minimum wait time of 3 minutes before marking a new instance as available

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=new-template \
    --min-ready 3m \
    --max-unavailable 3 \
    [--zone zone | --region region]

For example, if you have 1,000 instances and you run this command, the Updater creates up to 100 instances before it starts to remove instances that are running the previous instance template.

Perform a rolling update of all VM instances, but create up to 10% new instances above the target size at a time

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=new-template \
    --max-surge 10% \
    [--zone zone | --region region]

Canary updates

The Updater lets you perform canary updates, so that you can test your updates on a random subset of instances before fully committing to the update.

A canary update is an update that is applied to a partial number of instances in the instance group. Canary updates let you test new features or upgrades on a random subset of instances, instead of rolling out a potentially disruptive update to all your instances. If an update is not going well, you only need to roll back a small number of instances, minimizing the disruption for your users. From the perspective of the server, a canary update is the same as a standard rolling update, except that the number of instances that should be updated is less than the total size of the instance group. Like a standard rolling update, a canary update is disruptive to the instances affected; that is, the affected instances are deleted and replaced by new VM instances during the update.

Starting a canary update

To initiate a canary update:

• You specify up to two instance template versions, typically a new instance template to canary and the current instance template for the remainder of the instances. For example, you can specify that 20% of your instances be created based on new-instance-template while the rest of the instances continue to run on the old-instance-template. You can't specify more than two instance templates at a time.

• You must always specify a target size (targetSize) for the canary version. You can't initiate a canary update if you omit the target size for the canary version. For example, if you specified that 10% of instances should be used for canarying, the remaining 90% are untouched and use the current instance template.

Console

1. In the Cloud Console, go to the Instance groups page.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Click Add template and choose the new instance template to canary.
5. Under Target size, enter the percentage or fixed number of instances you want to use to canary the new instance template.
6. Optionally, you can toggle configuration options like whether the update is proactive (the group actively deletes and replaces instances) or opportunistic (the group doesn't actively replace instances but applies the update when instances are created for other reasons). You can also supply max surge, max unavailable options, and minimum wait time.
7. Click Update to start the update.

gcloud

Using the gcloud command-line tool, provide both the current template and the new template to explicitly express how many instances should use each template:

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=current-template \
    --canary-version template=new-template,target-size=size \
    [--zone zone | --region region]

Replace the following:

• instance-group-name: The group name for the instance.
• current-template: The instance template that the instance group is running.
• new-template: The new template you want to canary.
• size: The number or percentage of instances that you want to apply this update to. You must apply the target-size property to the --canary-version template. You can only set a percentage if the instance group contains 10 or more instances.
• zone: A zone for this instance group. If the instance group is a zonal instance group, provide the zone.
• region: A region for this instance group. If the instance group is regional, provide the region.

For example, the following command performs a canary update that rolls out my-template-b to 10% of instances in the instance group:

gcloud compute instance-groups managed rolling-action start-update my-ig1 \
    --version template=my-template-A \
    --canary-version template=my-template-B,target-size=10%

API

In the API, make a PATCH request to the instance group manager resource:

PATCH https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name

In the request payload, include both the current instance template and the new instance template that you want to canary. For example:

{
 "versions": [
  {
   "instanceTemplate": "global/instanceTemplates/new-template",
   "targetSize": {
    "[percent|fixed]": number|percentage # Use `fixed` for a specific number of instances
   }
  },
  {
   "instanceTemplate": "global/instanceTemplates/current-template"
  }
 ]
}

Replace the following:

• new-template: The name of the new template you want to canary.
• number|percentage: The fixed number or percentage of instances to canary this update. You can only set a percentage if the instance group contains 10 or more instances. Otherwise, provide a fixed number instead.
• current-template: The name of the current instance template that the instance group is running.

Rolling forward a canary update

After running a canary update, you can decide whether you want to commit the update to 100% of the instance group or roll it back.

Console

1. In the Cloud Console, go to the Instance groups page.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Under Template, update the target size of the canary template to 100% to roll forward the template to all your instances. Alternatively, you can replace the primary template with the canary template and set the target size to 100%. Then, you can remove the second template field altogether.
5. Click Update to start the update.

gcloud

If you want to commit to your canary update, roll forward the update by making the same update request but setting only version and omitting --canary-version. Using the gcloud command-line tool:

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=new-template \
    [--zone zone | --region region]

API

In the API, make a PATCH request to the instance group manager resource:

PATCH https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name

In the request body, specify the new instance template as a version and omit the earlier instance template from your request body. Omit the target size specification to roll the update out to 100% of instances. For example, your request body would look like the following. Replace new-template with the name of the new instance template you want to roll forward.

{
"versions": [
   {
   "instanceTemplate": "global/instanceTemplates/new-template" # New instance template
   }
 ]
}

Starting an opportunistic or proactive update

By default, updates made using the gcloud command-line tool are proactive and updates initiated in the API are opportunistic.

For proactive updates, Compute Engine actively schedules actions to apply the requested updates to instances as necessary. In many cases, this often means deleting and recreating instances proactively.

Alternatively, you can choose to perform an opportunistic update if a proactive update is potentially too disruptive. An opportunistic update is only applied when you manually initiate the update on selected instances or when new instances are created by the managed instance group. The managed instance group creates new instances when the managed instance group is resized either by another service, such as an autoscaler, or manually by a user. Compute Engine does not actively initiate requests to apply opportunistic updates.

In certain scenarios, an opportunistic update is useful because you don't want to cause instability to the system if it can be avoided. For example, if you have a non-critical update that can be applied as necessary without any urgency and you have a managed instance group that is actively being autoscaled, perform an opportunistic update so that Compute Engine does not actively tear down your existing instances to apply the update. When resizing down, the autoscaler preferentially terminates instances with the old template as well as instances that are not yet RUNNING.

To choose whether an update is opportunistic or proactive, set the type property to OPPORTUNISTIC or PROACTIVE using either the gcloud command-line tool or the API.

Console

1. In the Cloud Console, go to the Instance groups page.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling update.
4. Under Template, choose a template to update your instance group to and select a target size for the template.
5. Under Update mode, choose between either an opportunistic or proactive update.
6. Click Update to start the update.

gcloud

Using the gcloud command-line tool:

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=instance-template \
    --type opportunistic|proactive \
    [--zone zone | --region region]

API

In the request payload to start an update, include the type property in your updatePolicy. Replace new-template with the name of the new template you want to canary. If you want an opportunistic update, replace PROACTIVE with OPPORTUNISTIC.

{
  "updatePolicy": {
    "type": "PROACTIVE" # Performs a proactive update
  },
  "versions": [{
    "instanceTemplate": "global/instanceTemplates/new-template",
    }]
}

Updating selected instances

When you start an opportunistic update, you must wait for Compute Engine to roll out the update as opportunities arise. However, if you want more control over the rollout, you can manually initiate that update immediately on specific instances in your managed instance group.

Note: Set your update type to OPPORTUNISTIC to prevent a race between manually initiated updates and automatic proactive updates.

Manually initiating updates lets you:

• Select the instances that you would like to update.
• Apply an update with the least of amount of disruption that is necessary for the update to complete. For example, if you are only updating metadata, it might not be necessary to restart the instance to complete the update. When manually initiating the update, the minimal necessary action is automatically performed.
• Enforce instance restart or recreation even if those actions aren't necessary for applying the update. For example, you might want to restart a VM even if you're only updating its metadata because your guest software has to pick up the new metadata on VM boot.
• Prevent an update if it requires more disruption than you can afford.
• Perform an update of all selected instances immediately, without restricting the rollout by maxSurge and maxUnavailable constraints.

Minimal action and most disruptive allowed action

Depending on the nature of an update, it might disrupt the instance's state. For example, changing an instance's boot disk requires deleting and replacing the instance.

You can control the level of disruption by using the minimal-action and most-disruptive-allowed-action flags:

• minimal-action lets you force a more disruptive action than is necessary.
• most-disruptive-allowed-action lets you prevent an update if it requires more disruption than you can afford.

When updating selected instances, both of these flags accept the following actions:

Action	Description	Which instance properties can be updated?
NONE	Do not disrupt the instance at all.	None.
REFRESH	Do not stop the instance.	Secondary disks, instance metadata, labels.
RESTART	Stop the instance and start it again.	Secondary disks, instance metadata, labels.
REPLACE	Delete the instance and create it again.	All instance properties stored in the instance template.

By default, the minimal-action is NONE. If your update requires a more disruptive action than you set with this flag, Compute Engine performs the necessary action to execute the update.

By default, the most-disruptive-allowed-action is REPLACE. If your update requires a more disruptive action than the action you set with this flag, the update request fails. For example, if you set "restart" as the most disruptive allowed action, then an attempt to update the boot disk image fails because that update requires instance recreation, a more disruptive action than a restart.

Note: The most disruptive allowed action can't be less disruptive than the minimal action.

You can update selected instances by using the gcloud tool or the API.

gcloud

After starting an opportunistic update, use the update-instances sub-command to apply the update to specific instances.

gcloud compute instance-groups managed update-instances instance-group-name \
    --instances instance-names \
    --most-disruptive-allowed-action disruption-level \
    --minimal-action disruption-level

Where:

• instance-group-name is the name of the instance group with pending updates.
• instance-names is a list of instances to immediately update.
• disruption-level is the minimal or maximal disruption level. This can be none, refresh, restart, or replace.
  • The default minimal-action is none.
  • The default most-disruptive-allowed-action is replace.

If you need to wait until all the specified instances are updated, check the group's status and wait until the group is stable.

API

After starting an opportunistic update, construct a POST request to the regionInstanceGroupManagers.applyUpdatesToInstances method. For a zonal managed instance group, use the zonal instanceGroupManagers.applyUpdatesToInstances method.

POST https://compute.googleapis.com/compute/v1/projects/project/regions/region/instanceGroupManagers/instance-group-name/applyUpdatesToInstances
{
  "instances": "zones/zone/instances/instance-name","zones/zone/instances/instance-name"
  "minimalAction": disruption-level,
  "mostDisruptiveAllowedAction": disruption-level
}

Where:

• instance-group-name is the name of the instance group with pending updates.
• zone is the zone of an instance to immediately update.
• instance-name is the name of an instance to immediately update.
• disruption-level is the minimal or maximal disruption level. This can be NONE, REFRESH, RESTART, or REPLACE.
  • The default minimalAction is NONE.
  • The default mostDisruptiveAllowedAction is REPLACE.

Similar to other managed instance group methods, applyUpdatesToInstances is intent-based, which means it returns an operation response. After the operation is DONE, listManagedInstances contains the list of instances with their currentAction fields changed to REFRESHING, RESTARTING, or RECREATING. If the operation fails, for example, due to concurrent changes to the group, the failure is noted in the lastAttempt.errors field.

If you need to wait until all the specified instances are finished updating, check the group's status and wait until the group is stable.

Performing a rolling replace or restart

Alternatively, you can use the restart or replace commands to perform a rolling restart or a rolling replace of VM instances in the managed instance group. Similar to the start-update command, you can specify any of the configuration options for a restart or a replace. A rolling restart or replace does not change anything about the instance group, including the instance template. It only refreshes the instances in the group using the method you choose.

There are many reasons why you might want a rolling replace or a rolling restart. For example, you might want to refresh your VM instances from time to time to:

• Clear up memory leaks.
• Restart your app so it can run from a fresh machine.
• Force a periodic replace as a best practice to test your VMs.
• Update your VM's operating system image or rerun startup scripts to update your software.

You can use the Google Cloud Console, the gcloud command-line tool, or the API to perform a replace where all instances are deleted and replaced with new instances:

Console

1. In the Cloud Console, go to the Instance groups page.

   Go to the Instance groups page

2. Select the instance group you want to update.
3. At the top of the page, click Rolling restart/replace.
4. Choose whether you want to restart or replace your instances. Restarting performs the stop and start methods on the instances, while a replace actively deletes and creates instances.
5. Optionally, you can toggle configuration options like max surge, max unavailable options, and minimum wait time.
6. Click the Restart or Replace button to start the update.

gcloud

gcloud compute instance-groups managed rolling-action replace instance-group-name

This command replaces all instances in the managed instance group, one at a time. If a full replace is too disruptive, you can specify a rolling restart instead, which doesn't delete any instances but only restarts each instance.

gcloud compute instance-groups managed rolling-action restart instance-group-name

You can further customize each of these commands with the same options available for updates (for example, maxSurge, maxUnavailable, min-ready).

API

In the API, make a PATCH request for the group and set a proactive updatePolicy where the minimalAction is either RESTART or REPLACE depending on whether you want to perform a rolling replace, where each instance is deleted and replaced with a new instance, or a rolling restart, where each instance is stopped and restarted. In both cases of RESTART or REPLACE, you must always provide a versions.instanceTemplate and a versions.name property (for example, v2) to trigger the action.

A rolling replace might look like the following:

PATCH https://compute.googleapis.com/compute/v1/projects/my-project/zones/zone/instanceGroupManagers/instance-group-name

{
 "updatePolicy": {
  "minimalAction": "REPLACE",
  "type": "PROACTIVE"
 },
 "versions": [
  {
   "instanceTemplate": "global/instanceTemplates/example-template",
   "name": "v2"
  }
 ]
}

For a rolling restart:

PATCH https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name

{
 "updatePolicy": {
  "minimalAction": "RESTART",
  "type": "PROACTIVE"
 },
 "versions": [
  {
   "instanceTemplate": "global/instanceTemplates/example-template",
   "name": "v2"
  }
 ]
}

Additional replace/restart examples

Perform a rolling restart of all VMs, two at a time

This command restarts all VMs in the instance group, two at a time. Notice that no new instance template is specified.

gcloud compute instance-groups managed rolling-action restart instance-group-name \
    --max-unavailable 2 \
    [--zone zone | --region region]

Perform a rolling restart of all VMs as quickly as possible

gcloud compute instance-groups managed rolling-action restart instance-group-name \
    --max-unavailable 100% \
    [--zone zone | --region region]

Perform a rolling replace of all VMs as quickly as possible

gcloud compute instance-groups managed rolling-action replace instance-group-name  \
    --max-unavailable 100% \
    [--zone zone | --region region]

Preserving instance names

If you need to preserve the names of your VM instances across an update, set the replacementMethod to recreate. You must also set maxUnavailable to be greater than 0 and maxSurge to be 0. Setting maxSurge to 0 causes your update to take longer to complete, but with replacementMethod:recreate, the updated instances keep their names.

If you do not specify a replacement method, the MIG's current updatePolicy.replacementMethod value is used. If it's not set then the default value (substitute) is used, which replaces VM instances with new instances that have randomly generated names.

Specify the replacement method using the gcloud command-line tool or the API.

gcloud

When issuing a rolling-action command, include --replacement-method=recreate.

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --replacement-method=recreate \
    --version template=new-template \
    --max-unavailable 5 \
    [--zone zone | --region region]

If you need to wait until all the specified instances are updated, check the group's status and wait until the group is stable.

API

Construct a request to the managed instance group's patch method and include the updatePolicy.replacementMethod field. For a regional MIG, use the regional patch method.

PATCH /compute/v1/projects/project/zones/zone/instanceGroupManagers/instance-group-name
{
    "updatePolicy": {
        "type": "PROACTIVE",
        "maxUnavailable": { "fixed": 5 },
        "replacementMethod": "RECREATE"
    },
    "versions": [ {
        "instanceTemplate": "global/instanceTemplates/new-template"
    } ]
}

Updating a regional managed instance group

A regional managed instance group contains VM instances that are spread across multiple zones within a region, as opposed to a zonal managed instance group, which only contains instances in one zone. Regional managed instance groups let you distribute your instances across more than one zone to improve your app's availability and to support extreme cases where one zone fails or an entire group of instances stops responding.

Updating a regional managed instance group using the Updater is same as performing an update on a zonal managed instance group, with a few exceptions described below. When you initiate an update to a regional instance group, the Updater always updates instances proportionally and evenly across each zone; it is not possible to choose which instances in which zones are updated first or to update instances in only one zone.

Differences between updating a regional versus zonal managed instance group

Before you initiate an update to a regional managed instance group, be aware that there are a number of things that behave differently from zonal managed instance groups:

• The default update parameters for regional managed instance groups are maxUnavailable=number-of-zones and maxSurge=number-of-zones, where number-of-zones is the number of zones selected for the regional managed instance group, which is, by default, 3.

• If you are using fixed numbers when specifying an update, the fixed number must be either 0 or equal to or greater than the number of zones associated with the regional managed instance group. For example, if the instance group is distributed across three zones, then you can't set maxSurge to 1 or to 2 because the Updater has to create an additional instance in each of the three zones.

Using a fixed number or a percentage in update requests

If you specify a fixed number in your update requests, the number you specify is divided by the number of zones in the regional managed instance group and distributed evenly. For example, if you specify maxSurge=10, then the Updater divides 10 across the number of zones in the region and creates instances based on that number. If the number of instances does not divide evenly across zones, the Updater adds the instances to a random zone. So, for 10 instances across three zones, two of the zones get 3 instances and one zone gets 4 instances. The same logic is applied to the maxUnavailable and the targetSize parameters for canary updates.

You can only specify a percentage if your managed instance group contains 10 or more VM instances. Percentages are handled slightly differently depending on the situation:

• If you specify a percentage of VM instances for a canary update, the Updater attempts to distribute the instances evenly across zones. The remainder is rounded either up or down in each zone but the total difference isn't more than 1 VM instance per group. For example, for a managed instance group with 10 instances and a target size percentage of 25%, the update is rolled out to 2 to 3 VM instances.

• If you specify a percentage for update options like maxSurge and maxUnavailable, the percentages are rounded independently per zone. The same rules that apply to updating zonal managed instance groups apply here.

Monitoring updates

After you initiate a rolling update, it can take some time for the update to finish. You can monitor the progress of an update by inspecting the status of the managed instance group or by checking the currentAction on each instance in the managed instance group.

Group status

At the group level, Compute Engine populates a read-only field called status that contains a versionTarget.isReached flag and an isStable flag. You can use the gcloud tool or the API to access these flags. You can also use the console to see the current and planned number of instances being updated.

Console

You can monitor a rolling update for a group by going to the details page of the specific instance group.

1. In the Cloud Console, go to the Instance groups page.

   Go to the Instance groups page

2. Select the instance group you want to monitor. The overview page for the instance group shows the template that each instance is using.
3. To view the details, click the Details tab.

The Details page shows the current and planned number of instances being updated for each instance template and also displays the update parameters.

gcloud

To verify that your update to the instance group has finished rolling out, use the describe command to check status.versionTarget.isReached==true. To verify that all instances in the group are running and healthy (that is, that the currentAction of each managed instance is NONE), check status.isStable==true.

Remember that the stability of a managed instance group depends on group configurations beyond the Updater; for example, if your group is autoscaled and if it is currently scaling out, then isStable==false because of the autoscaler operation.

gcloud compute instance-groups managed describe instance-group-name \
    [--zone zone | --region zone]

The gcloud tool returns detailed information about the instance group, including the status.versionTarget.isReached and status.isStable fields.

You can also use the gcloud compute instance-groups managed wait-until command with the --version-target-reached flag to wait until status.versionTarget.isReached is set to true for the group:

gcloud compute instance-groups managed wait-until instance-group-name \
    --version-target-reached \
    [--zone zone | --region region]

API

To verify that your update to the instance group has finished rolling out, use the instance group manager get method to check status.versionTarget.isReached==true. To verify that all instances in the group are running and healthy (that is, that the currentAction of each managed instance is NONE), check status.isStable==true.

Remember that the stability of a managed instance group depends on group configurations beyond the Updater; for example, if your group is autoscaled and if it is currently scaling out, then isStable==false because of the autoscaler operation.

GET https://compute.googleapis.com/compute/v1/projects/project-d/zones/zone/instanceGroupManagers/instance-group-name/get

If the instance group is a regional managed instance group, replace zones/zone with regions/region.

The API returns detailed information about the instance group, including the status.versionTarget.isReached and status.isStable fields.

Verifying whether an update rollout is complete

The rollout of an update is considered complete when all VM instances in the group have been or are being created in their target version. In the case of a full rollout, all instances are configured to use the new instance template. In the case of a partial rollout, the instances are configured according to the specified target split between instance templates.

You can verify whether the rollout of an update is complete by checking the value of the status.versionTarget.isReached field of an instanceGroupManagers (or regionInstanceGroupManagers) resource.

• status.versionTarget.isReached is set to true if all VM instances have been or are being created using the target version (versions[]).
• status.versionTarget.isReached is set to false when at least one VM is not yet using the target version (versions[]). Or, in the case of a canary update, status.versionTarget.isReached is set to false when the number of VMs using a target version (versions[].instanceTemplates) doesn't match its target size (versions[].targetSize).

Note: To verify that the update rollout is complete and also that all VM instances in the group are running and healthy, check both the status.isStable and status.versionTarget.isReached flags.

Checking whether managed instance group is stable

You can verify that a managed instance group is running and healthy by checking the value of the status.isStable field.

status.isStable set to false indicates that changes are active, pending, or that the managed instance group itself is being modified.

status.isStable set to true indicates the following:

• None of the instances in the managed instance group are undergoing any type of change and the currentAction for all instances is NONE.
• No changes are pending for instances in the managed instance group.
• The managed instance group itself is not being modified.

Managed instance groups can be modified in numerous ways. For example:

• You make a request to roll out a new instance template.
• You make a request to create, delete, resize or update instances in the group.
• An autoscaler requests to resize the group.
• An autohealer resource is replacing one or more unhealthy instances in the managed instance group.
• In a regional managed instance group, some of the instances are being redistributed.

As soon as all actions are finished, status.isStable is set to true again for that managed instance group.

Current actions on instances

You can see the currentAction being performed and the status of each instance in a managed instance group with the gcloud command-line tool or the API.

Permissions required for this task

To perform this task, you must have the following permissions:

• compute.instanceGroupManagers.get on the instance group
• compute.instanceGroupManagers.list on the instance group

gcloud

gcloud compute instance-groups managed list-instances instance-group-name \
[--filter="zone:(zone)" | --filter="region:(region)"]

gcloud returns a list of instances in the instance group and their respective statuses and current actions. For example:

NAME               ZONE           STATUS    ACTION    INSTANCE_TEMPLATE  VERSION_NAME  LAST_ERROR
vm-instances-9pk4  us-central1-f            CREATING  my-new-template
vm-instances-h2r1  us-central1-f  STOPPING  DELETING  my-old-template
vm-instances-j1h8  us-central1-f  RUNNING   NONE      my-old-template
vm-instances-ngod  us-central1-f  RUNNING   NONE      my-old-template

API

In the API, make a GET request to the regionInstanceGroupManagers.listManagedInstances method. For a zonal managed instance group, use the instanceGroupManagers.listManagedInstances method.

GET https://compute.googleapis.com/compute/v1/projects/project-id/regions/region/instanceGroupManagers/instance-group-name/listManagedInstances

The API returns a list of instances for the group including each instance's instanceStatus and currentAction.

{
 "managedInstances": [
  {
   "instance": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/vm-instances-prvp",
   "id": "5317605642920955957",
   "instanceStatus": "RUNNING",
   "instanceTemplate": "https://compute.googleapis.com/compute/v1/projects/project-id/global/instanceTemplates/instance-template-name",
   "currentAction": "REFRESHING"
  },
  {
   "instance": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/vm-instances-pz5j",
   "currentAction": "DELETING"
  },
  {
   "instance": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/vm-instances-w2t5",
   "id": "2800161036826218547",
   "instanceStatus": "RUNNING",
   "instanceTemplate": "https://compute.googleapis.com/compute/v1/projects/project-id/global/instanceTemplates/instance-template-name",
   "currentAction": "REFRESHING"
  }
 ]
}

For each instance in a managed instance group, the status of the instance is described by its instanceStatus field. To see a list of valid instanceStatus field values, see Checking an instance's status.

If the instance is undergoing some type of change, the currentAction field is populated with one of the following actions to help you track the progress of the change. Otherwise, the currentAction field is NONE.

Possible currentAction values are:

• ABANDONING. The instance is being removed from the managed instance group.
• CREATING. The instance is in the process of being created.
• CREATING_WITHOUT_RETRIES. The instance is being created without retries; if the instance isn't created on the first try, the managed instance group doesn't try to replace the instance again.
• DELETING. The instance is in the process of being deleted.
• RECREATING. The instance is being replaced.
• REFRESHING. The instance is being removed from its current target pools and being readded to the list of current target pools (this list might be the same or different from existing target pools).
• RESTARTING. The instance is in the process of being restarted using the stop and start methods.
• VERIFYING. The instance has been created and is in the process of being verified.
• NONE. No actions are being performed on the instance.

Note: the pendingActions field is deprecated and is to be removed. Contact [e-mail] to leave feedback if your workload relies on this field.

Rolling back an update

There is no explicit command for rolling back an update to a previous version, but if you decide to roll back an update (either a fully committed update or a canary update), you can do so by making a new update request and passing in the instance template that you want to roll back to.

For example, the following gcloud command rolls back an update as fast as possible. Replace old-instance-template with the name of the instance template you want to roll back to.

gcloud compute instance-groups managed rolling-action start-update instance-group-name \
    --version template=old-instance-template-name --max-unavailable 100% [--zone zone | --region region]

Replace old-instance-template-name with the name of the instance template you want to roll back to.

In the API, make a PATCH request to the instance group manager resource:

PATCH https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name

If the instance group is a regional managed instance group, replace zones/zone with regions/region.

In the request body, specify the earlier instance template as a version:

{ "updatePolicy":
  {
    "maxUnavailable":
    {
      "percent": 100
    }
  },
 "versions": [
    {
      "instanceTemplate": "global/instanceTemplates/old-template" # Old instance template
    }
   ]
}

If it's a regional managed instance group with less than 10 instances, you must use a fixed value for maxUnavailable and set the value to the number of instances in the group:

{ "updatePolicy":
  {
    "maxUnavailable":
    {
      "fixed": number-of-instances-in-group
    }
  },
 "versions": [
    {
      "instanceTemplate": "global/instanceTemplates/old-template" # Old instance template
    }
   ]
}

The Updater treats this as a regular update request, so all of the update options described in this document can be specified with your request.

Caution: For canary updates, Google strongly recommends fully rolling back canary updates before you try to canary another version. Because there is no way to selectively choose which instances are updated based on which instance template, performing another canary update on top of an existing update might cause unintended consequences. As a best practice, you should fully roll back a canary update first and restart the canary process again from the beginning.

Controlling the speed of an update

By default, when you make an update request, the service performs the update as fast as possible. If you aren't sure you want to apply an update fully or are tentatively testing your changes, you can control the speed of the update by using the following methods.

1. Start a canary update rather than a full update.
2. Set a large minReadySeconds value. Setting this value causes the service to wait this number of seconds before considering the instance successfully updated and proceeding to the next instance.
3. Enable health checking to cause the service to wait for your app to start and to report a healthy signal, before considering the instance successfully updated and proceeding to the next instance.
4. Set low maxUnavailable and maxSurge values. This ensures that only a minimal number of instances are updated at a time.
5. Manually initiate updates on specific instances to update those instances immediately.

You can also use a combination of these parameters to control the rate of your update.

Stopping an update

There is no explicit method or command to stop an update. You can change an update from proactive to opportunistic, and if the managed instance group is not being resized by other services like autoscaler, the change to opportunistic effectively stops the update.

To change an update from proactive to opportunistic, run the following command:

gcloud compute instance-groups managed rolling-action stop-proactive-update instance-group-name \
    [--zone zone | --region region]

To stop the update completely after converting it from proactive to opportunistic, you can stop it using these steps:

1. Make a request to determine how many instances have been updated.

   gcloud compute instance-groups managed list-instances instance-group-name \
       [--zone zone | --region region]

   The gcloud tool returns a response that includes a list of the instances in the instance group and their current statuses:

   >
   NAME               ZONE           STATUS   HEALTH_STATE  ACTION    INSTANCE_TEMPLATE  VERSION_NAME  LAST_ERROR
   vm-instances-9pk4  us-central1-f  RUNNING  HEALTHY       NONE      my-new-template
   vm-instances-j1h8  us-central1-f  RUNNING  HEALTHY       NONE      my-old-template
   vm-instances-ngod  us-central1-f  STAGING  UNKNOWN       CREATING  my-new-template
   

   In this example, two instances have already been updated.

2. Next, make a request to perform a new update, but pass in the number of instances that have already been updated as the target size:

   gcloud compute instance-groups managed rolling-action start-update instance-group-name \
       --version template=my-old-template \
       --canary-version template=my-new-template,target-size=2 \
       [--zone zone | --region region]

   To the Updater, this update appears complete, so no other instances are updated, effectively stopping the update.

Relationship between versions and instanceTemplate properties for a managed instance group

Google recommends using the versions field to configure instance templates for managed instance groups.

The legacy instanceTemplate field overlaps in functionality with versions because both fields let you specify which instance template the managed instance group uses to create instances. However, only the versions field lets you specify an advanced, two-template (canary) configuration.

For backward compatibility, managed instance groups continue to support setting the top-level instanceTemplate field, even though we recommend that you switch to using only the versions field. Using both the top-level instanceTemplate field and the versions field at the same time can lead to ambiguity and confusion.

If you specify both the instanceTemplate field and the versions field when calling the update() or patch() method, there are three possible outcomes:

• You set both fields to the same value.

  This is a valid request. In this case, it creates no ambiguity, and the new instance template is applied to the managed instance group.

  For example, in the following request, the top-level instanceTemplate and the versions field specify the same instance template which is different from the existing current template, so the managed instance group is updated to the new instance template:

  {
   "instanceTemplate": "global/instanceTemplates/new-template",
   "versions": [
    {
     "instanceTemplate": "global/instanceTemplates/new-template"
    }
   ],
   "updatePolicy": {
     "type": "PROACTIVE"
   }
  }
  

• You set both fields to values that don't match but only one value differs from the current instance template in the managed instance group.

  This is a valid request. The field that's different from the current setting is taken as the intended value. For example, you call the get() method, change one of the two fields, and then call update() with just one changed field:

  {
   "instanceTemplate": "global/instanceTemplates/current-template",
   "versions": [
    {
     "instanceTemplate": "global/instanceTemplates/new-template"
    }
   ],
   "updatePolicy": {
     "type": "PROACTIVE"
   }
  }
  

• You set both fields to values that don't match, and both values differ from the current instance template in the managed instance group.

  This setting is invalid and returns an error because there is no clear intent.

  {
   "instanceTemplate": "global/instanceTemplates/new-template",
   "versions": [
    {
     "instanceTemplate": "global/instanceTemplates/a-different-new-template"
    }
   ],
   "updatePolicy": {
     "type": "PROACTIVE"
   }
  }
  

The versions field, instanceTemplate field, and the get() method

If you only specify one instance template, either through the top-level instanceTemplate field or through the versions field or through both, the get() method returns both fields in its response. This makes the new versions field backward compatible. As long as you specify a single instance template in either of these fields, there is no change to what get() returns in the instanceTemplate field.

If the versions field has two instance templates specified, the get() method returns an empty top-level instanceTemplate field. There is no way to unambiguously express a canary, two-instance template configuration in the top-level instanceTemplate field, so the field is not used during a canary update.

The versions field and the setInstanceTemplate() method

For backward compatibility, the setInstanceTemplate() method behaves as it has previously, letting you change the template the managed instance group uses to create instances. When you call this method, the versions field is overridden with the instance template specified by the setInstanceTemplate() method.

The setInstanceTemplate() method also sets the updatePolicy to OPPORTUNISTIC. This prevents the managed instance group from actively deploying an instance template that isn't explicitly specified in the versions field.

What's next

• Learn more about instance groups.
• Learn more about instance templates.
• Learn how to use image families and a rolling replace to update the OS image on all VMs in a MIG.