Creating and managing regional MIGs

This document describes procedures that are specific to creating and managing regional managed instance groups (regional MIGs). For general information about creating MIGs, see Creating managed instance groups.

A regional MIG distributes your instances across multiple zones in a region, which increases the resilience of your MIG-based workloads. Using multiple zones protects you from extreme cases where all instances in a single zone fail.

For more information about regional MIGs, including why to choose them, their extra configuration options, and how they're different from zonal MIGs, see the Regional MIG overview.

Before you begin

If you want to use the API examples in this guide, set up API access.
If you want to use the command-line examples in this guide, install the gcloud command-line tool.
Create an instance template, which is a building block for a managed instance group.
Read about regions and zones, which contain resources such as CPUs, GPUs, and disks.

Limitations

With a regional MIG, you can create up to 2,000 VMs in a region, with a maximum of 1,000 VMs per zone. With a zonal MIG, you can create up to 1,000 VMs. If you need more, contact support.
When updating a MIG, you can specify up to 1,000 VMs in a single request.
If you want a stateful MIG, review the stateful MIG limitations.
If you want to use load balancing with a regional MIG, the following limitations apply:
- You cannot use the maxRate balancing mode.
- If you use an HTTP(S) load balancing scheme with a regional MIG, you must choose the maxRatePerInstance or maxUtilization balancing mode.
If you want to autoscale a regional MIG, the following limitations apply:
- To scale in and out, you must enable proactive instance redistribution. If you set the autoscaler's mode to only scale out, then you don't need to enable proactive instance distribution.
- If you want to autoscale a regional MIG based on Cloud Monitoring metrics, the following limitations apply:
  - You cannot use per-group metrics.
  - You cannot apply filters to per-instance metrics.

Creating a regional MIG

Use the Cloud Console, the gcloud tool or the Compute Engine API.

If there is not enough capacity in each zone to support VMs for the group, Compute Engine creates as many VMs as possible and continues attempting to create the remaining VMs when additional capacity becomes available.

Because you are creating a regional MIG, keep in mind that certain resources are zonal, such as persistent disks. If you are specifying zonal resources in your instance template, like additional persistent disks, the disk must be present in all zones so it can be attached to the VMs created by this regional MIG.

By default, if you do not explicitly specify individual zones in your request, Compute Engine automatically chooses three zones to create VMs in. If you need to create VMs in more than or fewer than three zones, or you want to pick which zones are used, you can provide a list of zones in your request. For more information, see Zone selection.

By default, proactive instance redistribution is enabled. If you need to manually manage the number of VMs in each zone, you can disable proactive instance redistribution and you cannot configure autoscaling.

Console

In the Cloud Console, go to the Instance groups page.
Go to the Instance groups page
Click Create Instance Group to create a new instance group.
Under Location, select Multiple zones.
Choose a desired region.
If you want to choose specific zones, click Configure zones to select the zones you want to use.
If you want to disable proactive instance redistribution
1. Ensure that Autoscaling mode is set to Don't autoscale.
2. Set Instance redistribution to Off.
Choose an instance template for the instance group or create a new one.
Specify the number of VMs for this group. Remember to provision enough VMs to support your application if a zone failure happens.
Continue with the rest of the MIG creation process.

gcloud

All MIGs require an instance template. If you don't have one, create an instance template. For example, the following command creates a basic instance template with default properties:

gcloud compute instance-templates create example-template

Next, use the instance-groups managed create command with the --region flag. For example, the following command creates a regional MIG in three zones within the us-east1 region:

gcloud compute instance-groups managed create example-rmig \
    --template example-template  \
    --size 30 \
    --region us-east1

If you want to select specific zones that the group should use, provide the --zones flag:

gcloud compute instance-groups managed create example-rmig \
    --template example-template \
    --size 30 \
    --zones us-east1-b,us-east1-c

If you want to disable proactive instance redistribution, set the --instance-redistribution-type flag to NONE. You cannot disable proactive instance redistribution if autoscaling is enabled.

gcloud compute instance-groups managed create example-rmig \
    --template example-template \
    --size 30 \
    --instance-redistribution-type NONE \
    --region us-east1

API

All MIGs require an instance template. If you don't have one, create an instance template.

Next, construct a POST request to the regionInstanceGroupManagers.insert method. In the request body, specify the group name, group size, and the URL to the instance template. Optionally, specify other fields, such as the base name for instances in the group.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers

{
  "baseInstanceName": "BASE_INSTANCE_NAME",
  "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
  "name": "INSTANCE_GROUP_NAME",
  "targetSize": "TARGET_SIZE"
}

Replace the following:

PROJECT_ID: the project ID for this request.
REGION: the region for the group.
BASE_INSTANCE_NAME: the instance name for each VM instance that is created as part of the group. For example, a base instance name of example-instance would create instances that have names like example-instance-[RANDOM_STRING] where [RANDOM_STRING] is generated by the server.
INSTANCE_TEMPLATE_NAME: the instance template to use.
INSTANCE_GROUP_NAME: the name of the MIG.
TARGET_SIZE: the target number of VMs for the group.

If you want to select specific zones or if you are creating VMs in a region with less than or more than three zones, include the distributionPolicy property in your request and supply a list of zones. Replace ZONE with the name of a zone to create VMs in.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers

{
  "baseInstanceName": "BASE_INSTANCE_NAME",
  "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
  "name": "INSTANCE_GROUP_NAME",
  "targetSize": "TARGET_SIZE",
  "distributionPolicy": {
     "zones": [
       {"zone": "zones/ZONE"},
       {"zone": "zones/ZONE"}
      ]
   }
}

For example, the following creates a regional MIG named example-rmig with 10 managed instances distributed across us-east1-b and us-east1-c zones:

POST https://compute.googleapis.com/compute/v1/projects/myproject/regions/us-east1/instanceGroupManagers

{
  "instanceTemplate": "global/instanceTemplates/example-instance",
  "name": "example-rmig",
  "targetSize": 10,
  "distributionPolicy": {
      "zones": [
        {"zone": "zones/us-east1-b"},
        {"zone": "zones/us-east1-c"}
      ]
   }
}

Working with managed instances in a regional MIG

If you need to act on specific managed instances in a MIG, for example, to create or delete instances with specific names, the process is the same for regional and zonal MIGs. See Working with managed instances.

Getting information about regional MIGs and managed instances

If you want to get information about your MIG, for example to view its configuration or inspect its status, or if you want to get information about the managed instances in a MIG, the process is the same for regional and zonal MIGs. See Getting information about MIGs and managed instances.

Updating a regional managed instance group

You can apply a new instance template to a regional MIG by using the Updater feature. For more information, see Updating a regional MIG.

If you want to add or remove instances in a MIG, see Working with managed instances.

If you want to use or update MIG features, see the docs for autohealing, load balancing, autoscaling, auto-updating, and stateful workloads.

Disabling and reenabling proactive instance redistribution

Proactive instance redistribution maintains an even number of VMs across the selected zones in the region. This configuration maximizes the availability of your application in the event of a zone-level failure.

Proactive instance redistribution is turned on by default for regional MIGs, but you can turn it off for non-autoscaled MIGs. When proactive instance redistribution is turned off, the group does not attempt to proactively redistribute VMs across zones. This is useful if you need to:

Manually delete or abandon managed instances from the group without affecting other running VMs.
Automatically delete a batch worker VM upon job completion without affecting other workers.
Protect VMs with stateful applications from unintended automatic deletion by proactive instance redistribution.

Use the Cloud Console, the gcloud tool or the Compute Engine API to create a regional MIG with proactive instance redistribution disabled, or to turn proactive instance redistribution off or on for an existing group.

Creating a group with proactive instance redistribution disabled

Console

In the Cloud Console, go to the Instance groups page.
Go to the Instance groups page
Click Create Instance Group to create a new instance group.
Under Location, select Multiple zones.
Choose a desired region.
If you want to choose specific zones, click Configure zones to select the zones you want to use.
To disable proactive instance redistribution:
1. Ensure that Autoscaling mode is set to Don't autoscale.
2. Set Instance redistributiond to Off.
Choose an instance template for the group or create a new one.
Specify the number of VMs for this group. Remember to provision enough VMs to support your application if a zone failure happens.
Continue with the rest of the MIG creation process.

gcloud

To create a new regional MIG without proactive instance redistribution, use the gcloud compute instance-groups managed create command with the --instance-redistribution-type flag set to NONE.

gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \
    --template INSTANCE_TEMPLATE_NAME \
    --size TARGET_SIZE \
    --zones ZONES \
    --instance-redistribution-type NONE

Replace the following:

INSTANCE_GROUP_NAME: the name for the MIG
INSTANCE_TEMPLATE_NAME: the name of the instance template to use for the group
TARGET_SIZE: the target size of the group
ZONES: the list of zones in a single region where you need to deploy VMs

For example:

gcloud compute instance-groups managed create example-rmig \
    --template example-template \
    --size 30 \
    --zones us-east1-b,us-east1-c \
    --instance-redistribution-type NONE

API

To create a non-autoscaled regional MIG without proactive instance redistribution, make a POST request to the regionInstanceGroupManagers.insert method. In the request body, include the updatePolicy property and set its instanceRedistributionType field to NONE.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/INSTANCE_GROUP_NAME

{
    "name": "INSTANCE_GROUP_NAME",
    "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
    "targetSize": "TARGET_SIZE",
    "distributionPolicy": {
        "zones": [
            {"zone": "zones/ZONE"},
            {"zone": "zones/ZONE"}
        ]
    },
    "updatePolicy": {
        "instanceRedistributionType": "NONE"
    }
}

Replace the following:

PROJECT_ID: the project ID for this request
REGION: the region for the instance group
INSTANCE_GROUP_NAME: the name for the MIG
INSTANCE_TEMPLATE_NAME: the name of the instance template to use for the group
TARGET_SIZE: the target size of the instance group
ZONE: the name of a zone in the single region where you need to deploy VMs

Turning off proactive instance redistribution

Before you can turn off proactive instance redistribution, you must set the autoscaler's mode to turn off autoscaling or to restrict it to only scale out.

Console

In the Cloud Console, go to the Instance groups page.
Go to the Instance groups page
Select the MIG to update and click Edit group.
If any autoscaling configuration exists, ensure that the Autoscaling mode is set to Don't autoscale.
Set Instance redistribution to Off to disable automatic redistribution.
Click Save.

gcloud

To turn off proactive instance redistribution for a non-autoscaled regional MIG, use the compute instance-groups managed update command with the --instance-redistribution-type flag set to NONE.

gcloud compute instance-groups managed update INSTANCE_GROUP_NAME \
    --instance-redistribution-type NONE \
    --region REGION

Replace the following:

INSTANCE_GROUP_NAME: the name of the MIG
REGION: the region of the instance group

API

To turn off proactive instance redistribution for a non-autoscaled regional MIG, make a PATCH request to the regionInstanceGroupManagers.patch method. In the request body, include the updatePolicy property and set its instanceRedistributionType field to NONE

 PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/[INSTANCE_GROUP_NAME

{
    "updatePolicy": {
         "instanceRedistributionType": "NONE"
    }
}

Replace the following:

PROJECT_ID: the project ID for this request
REGION: the region for the instance group
INSTANCE_GROUP_NAME: the name of a non-autoscaled MIGs

Turning on proactive instance distribution

To turn on proactive instance distribution, use a similar command as for turning off proactive instance redistribution, but set the instance redistribution type to PROACTIVE.

If you manually deleted or abandoned some managed instances resulting in an uneven distribution of VMs across the region, then, before you can reenable proactive instance redistribution, you must manually rebalance the group. The difference in the number of VMs between any two zones should not exceed 1 VM.

A regional MIG does not allow turning on proactive instance redistribution when VMs are distributed unevenly across zones (the difference in the number of VMs between two zones is 2 or more VMs). This is to prevent an unintended automatic deletion of VMs from zones with more VMs, which would be triggered to achieve even distribution.

Manually rebalancing a regional MIG

A MIG is not balanced if the difference in the number of VMs between two zones is 2 or more VMs. A MIG can fall out of balance if you disable proactive instance redistribution and then delete or abandon instances to cause an uneven distribution across zones.

You can manually achieve an even distribution of instances across zones by deleting VMs from zones with more instances or by resizing to add instances to zones with fewer VMs until the distribution is even.

When you resize a MIG that has proactive instance redistribution turned off, the group still opportunistically converges toward balance, treating each resize operation as an opportunity to balance the group:

When the group grows, the group always tries to add VMs to the zones with the smallest number of VMs.
When the group shrinks, the group always removes VMs from the zones with the largest number of VMs.

*Example of manually resizing a group to achieve even redistribution*

Simulating a zone outage for a regional MIG

To test that your regional MIG is overprovisioned enough and can survive a zone outage, you can use the following example to simulate a zonal failure.

This script stops and starts Apache as the default scenario. If this doesn't apply to your application, replace the commands that stop and start Apache with your own failure and recovery scenario.

Deploy and run this script continuously in every VM in the group. You can do this by adding the script to the instance template or by including the script in a custom image and using the image in the instance template.

zone_failure/failure.sh

View on GitHub

#!/usr/bin/env bash

# Copyright 2016 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -o nounset
set -o errexit
set -o pipefail

function GetMetadata() {
  curl -s "$1" -H "Metadata-Flavor: Google"
}

PROJECT_METADATA_URL="http://metadata.google.internal/computeMetadata/v1/project/attributes"
INSTANCE_METADATA_URL="http://metadata.google.internal/computeMetadata/v1/instance"
ZONE=$(GetMetadata "$INSTANCE_METADATA_URL/zone" | cut -d '/' -f 4)
INSTANCE_NAME=$(hostname)

# We keep track of the state to make sure failure and recovery is triggered only once.
STATE="healthy"
while true; do
  if [[ "$ZONE" = "$(GetMetadata $PROJECT_METADATA_URL/failed_zone)" ]] && \
     [[ "$INSTANCE_NAME" = *"$(GetMetadata $PROJECT_METADATA_URL/failed_instance_names)"* ]]; then
    if [[ "$STATE" = "healthy" ]]; then
      STATE="failure"
      # Do something to simulate failure here.
      echo "STARTING A FAILURE"
      /etc/init.d/apache2 stop
    fi
  else
    if [[ "$STATE" = "failure" ]] ; then
      STATE="healthy"
      # Do something to recover here.
      echo "RECOVERING FROM FAILURE"
      /etc/init.d/apache2 start
    fi
  fi
  sleep 5
done

Simulate a zone failure by setting these two project metadata fields:
- failed_zone: Sets the zone where you want to simulate the outage (limit the failure to just one zone).
- failed_instance_names: Choose the VMs to take offline by name (to limit the failure to only VM names containing this string).
You can set this metadata using the gcloud tool. For example, the following command sets the zone outage to the europe-west1-b zone and affects VMs that have names starting with base-instance-name:
```
gcloud compute project-info add-metadata --metadata failed_zone='europe-west1-b',failed_instance_names='base-instance-name-'
```
After you are done simulating the outage, recover from the failure by removing the metadata keys:
```
gcloud compute project-info remove-metadata --keys failed_zone,failed_instance_names
```

Here are some ideas for failure scenarios you can run using this script:

Stop your application completely to see how the MIG responds.
Make your VMs return as "unhealthy" on load balancing health checks.
Modify iptables to block some of the traffic to and from the VM.
Shutdown the VMs. By default, it will be recreated by the regional MIG shortly after but the new incarnation will immediately shutdown itself as soon as the script runs and as long as the metadata values are set. This will result in a crash loop.

What's next

Create an instance template to use in a managed instance group.
Choose a load balancer or add an instance group to a load balancer.
Enable autohealing for your managed instance group.
Enable autoscaling for your managed instance group.
Use the auto-updater to update your managed instances with a new template.