Creating and managing regional MIGs

This document describes procedures that are specific to creating and managing regional managed instance groups (MIGs). For general information about creating MIGs, see Creating managed instance groups.

A regional MIG distributes your instances across multiple zones in a region, which increases the resilience of your MIG-based workloads. Using multiple zones protects you from extreme cases where all instances in a single zone fail.

For more information about regional MIGs, including why to choose them, their extra configuration options, and how they're different from zonal MIGs, see the Regional MIG overview.

Before you begin

Limitations

  • With a regional MIG, you can create up to 2,000 VMs in a region, with a maximum of 1,000 VMs per zone. With a zonal MIG, you can create up to 1,000 VMs. If you need more, contact support.
  • When updating a MIG, you can specify up to 1,000 VMs in a single request.

  • If you want a stateful MIG, review the stateful MIG limitations.

  • You must select which zones are associated with a regional MIG when you create the regional MIG. After you choose specific zones during creation, you cannot change or update the zones later. But you can set the MIG's target distribution shape to specify how the group distributes its managed instances across the zones that you selected.

  • If you want to use load balancing with a regional MIG, the following limitations apply:

    • You cannot use the maxRate balancing mode.
    • If you use an HTTP(S) load balancing scheme with a regional MIG, you must choose the maxRatePerInstance or maxUtilization balancing mode.

  • If you want to autoscale a regional MIG, the following limitations apply:

Creating a regional MIG

Use the Cloud Console , the gcloud tool or the Compute Engine API.

If there is not enough capacity in each zone to support VMs for the group, Compute Engine creates as many VMs as possible and continues attempting to create the remaining VMs when additional capacity becomes available.

By default, if you do not explicitly specify individual zones in your request, Compute Engine automatically chooses three zones to create VMs in. If you need to create VMs in more than or fewer than three zones, or you want to pick which zones are used, you can provide a list of zones in your request. For more information, see Zone selection.

By default, instances are distributed evenly across selected zones. If you have zonal resources that you want to use with your regional MIG, for example, specific GPUs, existing zonal persistent disks, or zonal reservations, you can set the MIG's distribution shape to create instances in zones that contain those resources.

Because you are creating a regional MIG, keep in mind that certain resources are zonal, such as existing persistent disks. If you specify zonal resources in your instance template, like existing persistent disks, the disk must be present in all zones that you select, so it can be attached to the VMs created by this regional MIG.

By default, proactive instance redistribution is enabled. If you need to manually manage the number of VMs in each zone, you can disable proactive instance redistribution. If you disable proactive instance redistribution, and if you want to use autoscaling, you must restrict the autoscaler to only scale out.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to the Instance groups page

  2. Click Create Instance Group to create a new instance group.
  3. Select one of the New managed instance group options: stateless (default) or stateful.
  4. Under Location, select Multiple zones.
  5. Choose a region.
  6. If you want to choose specific zones, click Configure zones to select the zones you want to use.
  7. Under Target distribution shape, select Even. If you want to select a different shape, see Setting a policy for distributing instances across zones.
  8. If you want to disable proactive instance redistribution
    1. Ensure that Autoscaling mode is set to Don't autoscale.
    2. Set Instance redistribution to Off.
  9. Choose an instance template for the instance group or create a new one.
  10. Specify the number of VMs for this group. Remember to provision enough VMs to support your application if a zone failure happens.
  11. Continue with the rest of the MIG creation process.

gcloud

All MIGs require an instance template. If you don't have one, create an instance template. For example, the following command creates a basic instance template with default properties:

gcloud compute instance-templates create example-template

Next, use the instance-groups managed create command with the --region flag. For example, the following command creates a regional MIG in three zones within the us-east1 region:

gcloud compute instance-groups managed create example-rmig \
    --template example-template  \
    --size 30 \
    --region us-east1

If you want to select specific zones that the group should use, provide the --zones flag:

gcloud compute instance-groups managed create example-rmig \
    --template example-template \
    --size 30 \
    --zones us-east1-b,us-east1-c

If you want to disable proactive instance redistribution, set the --instance-redistribution-type flag to NONE. You cannot disable proactive instance redistribution if autoscaling is enabled.

gcloud compute instance-groups managed create example-rmig \
    --template example-template \
    --size 30 \
    --instance-redistribution-type NONE \
    --region us-east1

API

All MIGs require an instance template. If you don't have one, create an instance template.

Next, construct a POST request to the regionInstanceGroupManagers.insert method. In the request body, specify the group name, group size, and the URL to the instance template. Optionally, specify other fields, such as the base name for instances in the group.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers

{
  "baseInstanceName": "BASE_INSTANCE_NAME",
  "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
  "name": "INSTANCE_GROUP_NAME",
  "targetSize": "TARGET_SIZE"
}

Replace the following:

  • PROJECT_ID: the project ID for this request.
  • REGION: the region for the group.
  • BASE_INSTANCE_NAME: the instance name for each VM instance that is created as part of the group. For example, a base instance name of example-instance would create instances that have names like example-instance-[RANDOM_STRING] where [RANDOM_STRING] is generated by the server.
  • INSTANCE_TEMPLATE_NAME: the instance template to use.
  • INSTANCE_GROUP_NAME: the name of the MIG.
  • TARGET_SIZE: the target number of VMs for the group.

If you want to select specific zones or if you are creating VMs in a region with less than or more than three zones, include the distributionPolicy property in your request and supply a list of zones. Replace ZONE with the name of a zone to create VMs in.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers

{
  "baseInstanceName": "BASE_INSTANCE_NAME",
  "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
  "name": "INSTANCE_GROUP_NAME",
  "targetSize": "TARGET_SIZE",
  "distributionPolicy": {
     "zones": [
       {"zone": "zones/ZONE"},
       {"zone": "zones/ZONE"}
      ]
   }
}

For example, the following creates a regional MIG named example-rmig with 10 managed instances distributed across us-east1-b and us-east1-c zones:

POST https://compute.googleapis.com/compute/v1/projects/myproject/regions/us-east1/instanceGroupManagers

{
  "instanceTemplate": "global/instanceTemplates/example-instance",
  "name": "example-rmig",
  "targetSize": 10,
  "distributionPolicy": {
      "zones": [
        {"zone": "zones/us-east1-b"},
        {"zone": "zones/us-east1-c"}
      ]
   }
}

Working with managed instances in a regional MIG

If you need to act on specific managed instances in a MIG, for example, to create or delete instances with specific names, the process is the same for regional and zonal MIGs. See Working with managed instances.

Getting information about regional MIGs and managed instances

If you want to get information about your MIG, for example to view its configuration or inspect its status, or if you want to get information about the managed instances in a MIG, the process is the same for regional and zonal MIGs. See Getting information about MIGs and managed instances.

Updating a regional managed instance group

You can apply a new instance template to a regional MIG by using the Updater feature. For more information, see Updating a regional MIG.

If you want to add or remove instances in a MIG, see Working with managed instances.

If you want to use or update MIG features, see the docs for autohealing, load balancing, autoscaling, auto-updating, and stateful workloads.

You cannot select different zones for the MIG after the MIG is created. But you can update the MIG's target shape to change how the group distributes its managed instances across selected zones.

Setting a policy for distributing instances across zones

You can set a regional MIG's target distribution shape to one of the following options:

  • EVEN (default): the group schedules VM instance creation and deletion to achieve and maintain an even number of managed instances across the selected zones. The distribution is even when the number of managed instances does not differ by more than 1 between any two zones. Recommended for highly available serving workloads.

  • BALANCED: the group prioritizes acquisition of resources, scheduling VMs in zones where resources are available while distributing VMs as evenly as possible across selected zones to minimize the impact of zonal failure. Recommended for highly available serving or batch workloads that do not require autoscaling.

  • ANY: the group picks zones for creating VM instances to fulfill the requested number of VMs within present resource constraints and to maximize utilization of unused zonal reservations. Recommended for batch workloads that do not require high availability.

To help you choose, see the comparison table, use cases, and how distribution shapes work.

Set a target distribution shape when creating your group or update the target shape of an existing group.

Limitations

  • If you set the target distribution shape to BALANCED, you must disable proactive redistribution.
  • If you set the target distribution shape to BALANCED or to ANY, the following limitations apply:
    • Autoscaling is not supported. You can set the size of the MIG or add instances yourself.
    • Canary updates with two versions are not supported.
    • In case of limited availability of requested resources in the whole region, the group might schedule VM instance creation in a zone where the resources are already unavailable. You can try decreasing and increasing group size to get the requested resources in other zones.
    • Rolling updates that use the SUBSTITUTE replacement method will try to create the new updated instances in the same zone as the outdated machines, even if the zone doesn't have resources to accommodate the requirements of the new version. To mediate this behavior, you can delete the outdated VMs from the constrained zone, then increase the group size by the number of the deleted VMs. The group creates instances from the latest template in zones where capacity is available.

Creating a group with a target distribution shape

Use the Cloud Console , the gcloud tool or the Compute Engine API.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to the Instance groups page

  2. Click Create Instance Group to create a new instance group.
  3. Select one of the New managed instance group options: stateless (default) or stateful.
  4. Under Location, select Multiple zones.
  5. Select a region.
  6. If you want to choose specific zones, click Configure zones to select the zones you want to use.
  7. Choose a target distribution shape.
    1. If you want to select Any or Balanced, in the Autoscaling section, click Delete autoscaling configuration.
    2. If you want to select Balanced, in the Instance redistribution section, do not select Enable instance redistribution.
    3. In the Target distribution shape section, select a shape.
  8. Choose an instance template for the instance group or create a new one.
  9. Specify the number of VMs for this group. For highly available workloads, remember to provision enough VMs to support your application if a zone failure happens.
  10. Continue with the rest of the MIG creation process.

gcloud

Use the gcloud beta compute instance-groups managed create command and include the --target-distribution-shape flag.

gcloud beta compute instance-groups managed create INSTANCE_GROUP_NAME \
    --template TEMPLATE \
    --size SIZE \
    [--zones ZONES \]
    --target-distribution-shape SHAPE

Replace the following:

  • INSTANCE_GROUP_NAME: the name of the instance group.
  • TEMPLATE: the name of the instance template to use for the group.
  • SIZE: the target size of the instance group.
  • ZONES (optional): a list of zones in the region where you want to deploy VM instances.
  • SHAPE: the target distribution shape. This can be one of the following values:
    • EVEN (default): the group schedules VM instance creation and deletion to achieve and maintain an even number of managed instances across the selected zones. The distribution is even when the number of managed instances does not differ by more than 1 between any two zones. Recommended for highly available serving workloads.
    • BALANCED: the group prioritizes acquisition of resources, scheduling VMs in zones where resources are available while distributing VMs as evenly as possible across selected zones to minimize the impact of zonal failure. Recommended for highly available serving or batch workloads that do not require autoscaling.
    • ANY: the group picks zones for creating VM instances to fulfill the requested number of VMs within present capacity constraints and to maximize utilization of unused zonal reservations. Recommended for batch workloads that do not require high availability.

For example, to create a regional MIG with a balanced target distribution shape, set the --target-distribution-shape flag to balanced.

gcloud beta compute instance-groups managed create example-rmig \
    --template example-template \
    --size 30 \
    --zones us-east1-b,us-east1-c \
    --target-distribution-shape BALANCED \
    --instance-redistribution-type NONE

API

Call the regionInstanceGroupManagers.insert method. In the request body, include thedistributionPolicyproperty, and set itstargetShape` field.

POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/INSTANCE_GROUP_NAME

{
    "instanceTemplate": "global/instanceTemplates/TEMPLATE",
    "targetSize": SIZE,
    "distributionPolicy": {
        "zones": [
            {"zone": "zones/ZONE"},
            {"zone": "zones/ZONE"}
        ],
        "targetShape": SHAPE
    }
}

Replace the following:

  • PROJECT_ID: the project ID for this request.
  • REGION: the region for the instance group.
  • INSTANCE_GROUP_NAME: the name of the instance group.
  • TEMPLATE: the name of the instance template to use for the instance group.
  • SIZE: the target size of the instance group.
  • ZONE: the name of a zone in the region where you want to deploy VM instances.
  • SHAPE: the target distribution shape. This can be one of the following values:
    • EVEN (default): the group schedules VM instance creation and deletion to achieve and maintain an even number of managed instances across the selected zones. The distribution is even when the number of managed instances does not differ by more than 1 between any two zones. Recommended for highly available serving workloads.
    • BALANCED: the group prioritizes acquisition of resources, scheduling VMs in zones where resources are available while distributing VMs as evenly as possible across selected zones to minimize the impact of zonal failure. Recommended for highly available serving or batch workloads that do not require autoscaling.
    • ANY: the group picks zones for creating VM instances to fulfill the requested number of VMs within present capacity constraints and to maximize utilization of unused zonal reservations. Recommended for batch workloads that do not require high availability.

Changing the target distribution shape of an existing group

You can change the target distribution shape in an existing regional MIG but with the following limitations:

  • If you want to change the target distribution shape to BALANCED, you must first disable proactive redistribution.
  • If you want to change the target distribution shape to EVEN, and if the current distribution of instances is uneven, you must first disable proactive redistribution.
  • If you change the shape to EVEN, and you want to re-enable proactive redistribution, you must first manually rebalance the group.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to the Instance groups page

  2. Under the Name column of the list, click the name of the instance group where you want to change the target distribution shape.
  3. Click Edit group to modify this managed instance group.
  4. Under Target distribution shape, specify the shape you want.
  5. Click Save to apply the new template.

gcloud

Use the gcloud beta compute instance-groups managed update command and include the --target-distribution-shape flag.

gcloud beta compute instance-groups managed update INSTANCE_GROUP_NAME \
    --target-distribution-shape SHAPE

Replace the following:

  • INSTANCE_GROUP_NAME: the name of the instance group.
  • SHAPE: the target distribution shape. This can be one of the following values:
    • EVEN (default): the group schedules VM instance creation and deletion to achieve and maintain an even number of managed instances across the selected zones. The distribution is even when the number of managed instances does not differ by more than 1 between any two zones. Recommended for highly available serving workloads.
    • BALANCED: the group prioritizes acquisition of resources, scheduling VMs in zones where resources are available while distributing VMs as evenly as possible across selected zones to minimize the impact of zonal failure. Recommended for highly available serving or batch workloads that do not require autoscaling.
    • ANY: the group picks zones for creating VM instances to fulfill the requested number of VMs within present capacity constraints and to maximize utilization of unused zonal reservations. Recommended for batch workloads that do not require high availability.

API

Call the regionInstanceGroupManagers.patch method. In the request body, include the distributionPolicy property, and set its targetShape field.

PATCH https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/INSTANCE_GROUP_NAME

{
    "distributionPolicy": {
        "targetShape": SHAPE
    }
}

Replace the following:

  • PROJECT_ID: the project ID for this request.
  • REGION: the region for the instance group.
  • INSTANCE_GROUP_NAME: the name of the instance group.
  • SHAPE: the target distribution shape. This can be one of the following values:
    • EVEN (default): the group schedules VM instance creation and deletion to achieve and maintain an even number of managed instances across the selected zones. The distribution is even when the number of managed instances does not differ by more than 1 between any two zones. Recommended for highly available serving workloads.
    • BALANCED: the group prioritizes acquisition of resources, scheduling VMs in zones where resources are available while distributing VMs as evenly as possible across selected zones to minimize the impact of zonal failure. Recommended for highly available serving or batch workloads that do not require autoscaling.
    • ANY: the group picks zones for creating VM instances to fulfill the requested number of VMs within present capacity constraints and to maximize utilization of unused zonal reservations. Recommended for batch workloads that do not require high availability.

Viewing the configured instance distribution policy

Console

  1. In the Google Cloud Console, go to the Instance groups page.

    Go to the Instance groups page

    If you have existing instance groups, the page lists those groups.
  2. Click the name of the instance group that you want to examine. A page opens with the instance group properties and a list of instances that are included in the group.
  3. Click Details and look for the Target distribution shape section.

gcloud

Run the gcloud beta compute instance-groups managed describe command.

gcloud beta compute instance-groups managed describe INSTANCE_GROUP_NAME \
    --region REGION

The command returns the group's details, including the distributionPolicy.targetShape field:

...
distributionPolicy:
  targetShape: BALANCED
  zones:
  - zone: https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-f
  ...
name: my-group
region: https://www.googleapis.com/compute/v1/projects/my-project/regions/us-central1
...

API

Construct a GET request to the regionInstanceGroupManagers.get method.

GET https://www.googleapis.com/compute/beta/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/INSTANCE_GROUP_NAME

Replace the following:

  • PROJECT_ID: the project ID for this request
  • REGION: the region for the instance group
  • INSTANCE_GROUP_NAME: the name of the instance group

The target distribution shape is returned in the distributionPolicy.targetShape field. For example:

{
  "name": "my-instance-group",
  "distributionPolicy": {
    "targetShape": "BALANCED",
  },
  "targetSize": 50,
  ...
}

Disabling and reenabling proactive instance redistribution

Proactive instance redistribution maintains an even number of VMs across the selected zones in the region. This configuration maximizes the availability of your application in the event of a zone-level failure.

Proactive instance redistribution is turned on by default for regional MIGs, but you can turn it off for non-autoscaled MIGs. When proactive instance redistribution is turned off, the group does not attempt to proactively redistribute VMs across zones. This is useful if you need to:

  • Manually delete or abandon managed instances from the group without affecting other running VMs.
  • Automatically delete a batch worker VM upon job completion without affecting other workers.
  • Protect VMs with stateful applications from unintended automatic deletion by proactive instance redistribution.

Use the Cloud Console , the gcloud tool or the Compute Engine API to create a regional MIG with proactive instance redistribution disabled, or to turn proactive instance redistribution off or on for an existing group.

Creating a group with proactive instance redistribution disabled

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to the Instance groups page

  2. Click Create Instance Group to create a new instance group.
  3. Under Location, select Multiple zones.
  4. Choose a region.
  5. If you want to choose specific zones, click Configure zones to select the zones you want to use.
  6. To disable proactive instance redistribution:
    1. Ensure that Autoscaling mode is set to Don't autoscale.
    2. Set Instance redistributiond to Off.
  7. Choose an instance template for the group or create a new one.
  8. Specify the number of VMs for this group. Remember to provision enough VMs to support your application if a zone failure happens.
  9. Continue with the rest of the MIG creation process.

gcloud

To create a new regional MIG without proactive instance redistribution, use the gcloud compute instance-groups managed create command with the --instance-redistribution-type flag set to NONE.

gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \
    --template INSTANCE_TEMPLATE_NAME \
    --size TARGET_SIZE \
    --zones ZONES \
    --instance-redistribution-type NONE

Replace the following:

  • INSTANCE_GROUP_NAME: the name for the MIG
  • INSTANCE_TEMPLATE_NAME: the name of the instance template to use for the group
  • TARGET_SIZE: the target size of the group
  • ZONES: the list of zones in a single region where you want to deploy VMs

For example:

gcloud compute instance-groups managed create example-rmig \
    --template example-template \
    --size 30 \
    --zones us-east1-b,us-east1-c \
    --instance-redistribution-type NONE

API

To create a non-autoscaled regional MIG without proactive instance redistribution, make a POST request to the regionInstanceGroupManagers.insert method. In the request body, include the updatePolicy property, and set its instanceRedistributionType field to NONE.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/INSTANCE_GROUP_NAME

{
    "name": "INSTANCE_GROUP_NAME",
    "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE_NAME",
    "targetSize": "TARGET_SIZE",
    "distributionPolicy": {
        "zones": [
            {"zone": "zones/ZONE"},
            {"zone": "zones/ZONE"}
        ]
    },
    "updatePolicy": {
        "instanceRedistributionType": "NONE"
    }
}

Replace the following:

  • PROJECT_ID: the project ID for this request
  • REGION: the region for the instance group
  • INSTANCE_GROUP_NAME: the name for the MIG
  • INSTANCE_TEMPLATE_NAME: the name of the instance template to use for the group
  • TARGET_SIZE: the target size of the instance group
  • ZONE: the name of a zone in the single region where you want to deploy VMs

Turning off proactive instance redistribution

Before you can turn off proactive instance redistribution, you must set the autoscaler's mode to turn off autoscaling or to restrict it to only scale out.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to the Instance groups page

  2. Select the MIG to update and click Edit group.
  3. If any autoscaling configuration exists, ensure that the Autoscaling mode is set to Don't autoscale.
  4. Set Instance redistribution to Off to disable automatic redistribution.
  5. Click Save.

gcloud

To turn off proactive instance redistribution for a non-autoscaled regional MIG, use the compute instance-groups managed update command with the --instance-redistribution-type flag set to NONE.

gcloud compute instance-groups managed update INSTANCE_GROUP_NAME \
    --instance-redistribution-type NONE \
    --region REGION

Replace the following:

  • INSTANCE_GROUP_NAME: the name of the MIG
  • REGION: the region of the instance group

API

To turn off proactive instance redistribution for a non-autoscaled regional MIG, make a PATCH request to the regionInstanceGroupManagers.patch method. In the request body, include the updatePolicy property, and set its instanceRedistributionType field to NONE

 PATCH https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instanceGroupManagers/[INSTANCE_GROUP_NAME

{
    "updatePolicy": {
         "instanceRedistributionType": "NONE"
    }
}

Replace the following:

  • PROJECT_ID: the project ID for this request
  • REGION: the region for the instance group
  • INSTANCE_GROUP_NAME: the name of a non-autoscaled MIGs

Turning on proactive instance distribution

To turn on proactive instance distribution, use a similar command as for turning off proactive instance redistribution, but set the instance redistribution type to PROACTIVE.

If you manually deleted or abandoned some managed instances resulting in an uneven distribution of VMs across the region, then, before you can reenable proactive instance redistribution, you must manually rebalance the group. The difference in the number of VMs between any two zones should not exceed 1 VM.

A regional MIG does not allow turning on proactive instance redistribution when VMs are distributed unevenly across zones (the difference in the number of VMs between two zones is 2 or more VMs). This is to prevent an unintended automatic deletion of VMs from zones with more VMs, which would be triggered to achieve even distribution.

Manually rebalancing a regional MIG

A MIG is not balanced if the difference in the number of VMs between two zones is 2 or more VMs. A MIG can fall out of balance if you disable proactive instance redistribution and then delete or abandon instances to cause an uneven distribution across zones.

You can manually achieve an even distribution of instances across zones by deleting VMs from zones with more instances or by resizing to add instances to zones with fewer VMs until the distribution is even.

When you resize a MIG that has proactive instance redistribution turned off, the group still opportunistically converges toward balance, treating each resize operation as an opportunity to balance the group:

  • When the group grows, the group always tries to add VMs to the zones with the smallest number of VMs.
  • When the group shrinks, the group always removes VMs from the zones with the largest number of VMs.
Example of manually resizing a group to achieve even redistribution

Simulating a zone outage for a regional MIG

To test that your regional MIG is overprovisioned enough and can survive a zone outage, you can use the following example to simulate a zonal failure.

This script stops and starts Apache as the default scenario. If this doesn't apply to your application, replace the commands that stop and start Apache with your own failure and recovery scenario.

  1. Deploy and run this script continuously in every VM in the group. You can do this by adding the script to the instance template or by including the script in a custom image and using the image in the instance template.

    zone_failure/failure.sh 
    #!/usr/bin/env bash

# Copyright 2016 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

set -o nounset
set -o errexit
set -o pipefail

function GetMetadata() {
  curl -s "$1" -H "Metadata-Flavor: Google"
}

PROJECT_METADATA_URL="http://metadata.google.internal/computeMetadata/v1/project/attributes"
INSTANCE_METADATA_URL="http://metadata.google.internal/computeMetadata/v1/instance"
ZONE=$(GetMetadata "$INSTANCE_METADATA_URL/zone" | cut -d '/' -f 4)
INSTANCE_NAME=$(hostname)

# We keep track of the state to make sure failure and recovery is triggered only once.
STATE="healthy"
while true; do
  if [[ "$ZONE" = "$(GetMetadata $PROJECT_METADATA_URL/failed_zone)" ]] && \
     [[ "$INSTANCE_NAME" = *"$(GetMetadata $PROJECT_METADATA_URL/failed_instance_names)"* ]]; then
    if [[ "$STATE" = "healthy" ]]; then
      STATE="failure"
      # Do something to simulate failure here.
      echo "STARTING A FAILURE"
      /etc/init.d/apache2 stop
    fi
  else
    if [[ "$STATE" = "failure" ]] ; then
      STATE="healthy"
      # Do something to recover here.
      echo "RECOVERING FROM FAILURE"
      /etc/init.d/apache2 start
    fi
  fi
  sleep 5
done

  2. Simulate a zone failure by setting these two project metadata fields:

    • failed_zone: Sets the zone where you want to simulate the outage (limit the failure to just one zone).
    • failed_instance_names: Choose the VMs to take offline by name (to limit the failure to only VM names containing this string).

    You can set this metadata using the gcloud tool. For example, the following command sets the zone outage to the europe-west1-b zone and affects VMs that have names starting with base-instance-name:

    gcloud compute project-info add-metadata --metadata failed_zone='europe-west1-b',failed_instance_names='base-instance-name-'

  3. After you are done simulating the outage, recover from the failure by removing the metadata keys:

    gcloud compute project-info remove-metadata --keys failed_zone,failed_instance_names

Here are some ideas for failure scenarios you can run using this script:

  • Stop your application completely to see how the MIG responds.
  • Make your VMs return as "unhealthy" on load balancing health checks.
  • Modify iptables to block some of the traffic to and from the VM.
  • Shutdown the VMs. By default, it will be recreated by the regional MIG shortly after but the new incarnation will immediately shutdown itself as soon as the script runs and as long as the metadata values are set. This will result in a crash loop.

What's next