Creating managed instance groups

This document describes how to create a managed instance group (MIG). A MIG is a group of virtual machine (VM) instances that you control as a single entity. MIGs support features such as autohealing, load balancing, autoscaling, auto-updating, and stateful workloads.

You can create regional MIGs or zonal MIGs. Regional MIGs provide higher availability compared to zonal MIGs because the instances in a regional MIG are spread across multiple zones in a single region. This document provides information about creating either zonal or regional MIGs. However, regional MIGs have additional options and considerations. For more information about regional MIGs, see the Regional MIGs overview.

For conceptual information about instance groups and their features, see the Instance groups overview.

Before you begin

Limitations

  • With a regional MIG, you can create up to 2,000 VMs in a region, with a maximum of 1,000 VMs per zone. With a zonal MIG, you can create up to 1,000 VMs. If you need more, contact support.
  • When updating a MIG, you can specify up to 1,000 VMs in a single request.

  • If you want a stateful MIG, review the stateful MIG limitations.

  • If you want a regional MIG, review the regional MIG limitations.

  • Shared VPC on interfaces other than nic0 for managed instance groups is supported in gcloud tool and the API, but not in Cloud Console.

Stateless or stateful MIGs

You can use MIGs for stateless serving or batch workloads, such as a website frontend or image processing from a queue, or for stateful applications, such as databases or legacy applications.

Using MIGs for stateless applications

Stateless applications don't depend on the specific state of the underlying virtual machine (VM) instance to run. If you use a stateless MIG, your application should not depend on VM properties that don't persist, such as the contents of attached disks or in-memory data. If your application requires the preservation of VM properties, see Configuring stateful MIGs.

The default behavior for all persistent disks in a MIG is to delete or recreate them when the corresponding VM is deleted or recreated, so you should not rely on attached disks as persistent data. To retain your data in a stateless MIG, we recommend that you regularly maintain up-to-date custom images with updated software and configurations or use startup scripts and configure your applications to back up necessary data in another centralized location, such as Cloud Storage.

In your instance templates, you can specify a container image or a custom image with relevant startup scripts, so that when a VM is recreated, it has the necessary apps installed and has access to required data. For more information about creating instance templates, see Deterministic instance templates.

Using MIGs for stateful applications

Managed instance groups also support stateful applications. A stateful MIG preserves each VM's unique state (VM instance name, attached persistent disks, and/or metadata) on machine restart, recreation, autohealing, or update.

Use stateful MIGs for applications that have stateful data or configuration, such as databases, data processing applications, legacy applications, and long-running batch workloads with checkpointing.

Consider using stateful MIGs whenever you deploy a stateful application or cluster to Compute Engine and would like to improve its availability with autohealing and multi-zone deployment, or would like to simplify and accelerate updates with automated rolling updates.

For information about how to create a stateful MIG or add stateful configuration to an existing MIG, see Configuring stateful MIGs.

Creating a managed instance group

A MIG creates each of its managed instances based on the instance templates and optional stateful configuration that you specify. Each managed instance is a data entity within the MIG that contains the current status and intended state for an actual VM instance. MIGs maintain high availability of your applications by proactively keeping actual VMs available, that is, in the RUNNING state.

You can create a MIG through the Google Cloud Console, the gcloud tool, or the Compute Engine API. If you'd rather create a regional MIG, see Creating regional MIGs.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to Instance groups

  2. Click Create an instance group.

  3. If you want to create a stateful MIG, select the New managed instance group (stateful) option. To help you decide, see When to use stateful MIGs.

  4. Enter a name for the MIG, and select the zone where you want to locate the group.

  5. Under Group type, select Managed instance group.

  6. Under Instance template, select an instance template. If no templates are available, create an instance template.

  7. Specify the number of instances that you want to create in the group.

  8. Optionally, for stateless MIGs, enable Autoscaling so that the group automatically adds or removes instances based on its utilization

  9. Optionally, enable autohealing to perform application-based health checking on VMs within the group.

  10. Click Create to create the new group.

gcloud

Create an instance group with the instance-groups managed create command.

gcloud compute instance-groups managed create INSTANCE_GROUP_NAME \
    --size SIZE \
    --template INSTANCE_TEMPLATE \
    --zone ZONE

Replace the following:

  • INSTANCE_GROUP_NAME: the name for this instance group.
  • SIZE: the size of the instance group.
  • INSTANCE_TEMPLATE: the name of the instance template to use for this group.
  • ZONE: one of the zones available for Compute Engine. If you want to create a regional MIG, see Creating and managing regional MIGs.

You can optionally supply the --base-instance-name flag. Because these VMs are based on a common template, each VM is assigned a random string as part of its VM name. The base name is prepended to this random string. For example, if you set the base name to test, VMs will have names like test-yahs and test-qtyz. If you need specific names, see Creating instances with specific names in MIGs.

For example, the following command creates an instance group named example-group, with base VM name test. The group contains three instances:

gcloud compute instance-groups managed create example-group \
    --base-instance-name test \
    --size 3 \
    --template an-instance-template

API

Call the instanceGroupManagers.insert method or the regionInstanceGroupManagers.insert method. In the request body, include the group name, group size, and the URL to the instance template.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers

{
  "versions": [
    {
      "instanceTemplate": "global/instanceTemplates/INSTANCE_TEMPLATE"
    }
  ],
  "name": "INSTANCE_GROUP_NAME",
  "targetSize": SIZE
}

Replace the following:

  • PROJECT_ID: the project ID for the request.
  • ZONE: the zone for the request. For a regional MIG, replace zones/ZONE with regions/REGION and specify a region. For more information, see Creating and managing regional MIGs.
  • INSTANCE_GROUP_NAME: the name for this instance group.
  • SIZE: the size of the instance group.
  • INSTANCE_TEMPLATE: the instance template to use for this group.

You can optionally supply the base-name field. Because these VMs are based on a common template, each VM is assigned a random string as part of its VM name. The base name is prepended to this random string. For example, if you set the base name to test, VMs will have names like test-yahs and test-qtyz. If you need specific names, see Creating instances with specific names in MIGs.

Depending on how you configure and act on a MIG, various policies and actions can affect the instances in the group. To determine which managed instances are up and running, see Checking the status of managed instances.

Changing the instance template for a managed instance group

To learn how to apply a new instance template to a MIG, see Updating instances in a MIG.

Resizing a managed instance group

To add or remove VMs in a MIG, you can:

Automatically resizing a MIG

You can configure managed instance groups to automatically add or remove VMs based on their workloads. Your applications can gracefully handle increases in traffic, and you can reduce your costs when the need for compute resources is lower. To start scaling your managed instance groups, see Autoscaling groups of instances.

Manually setting the size of a MIG

If a managed instance group is not already set to automatically scale, you can resize the group manually to change the number of instances. If you increase the size, the managed instance group uses the current instance template to add new instances. If you decrease the size, the managed instance group deletes VMs from the group. The group deletes instances with a currentAction of DELETING, CREATING, and RECREATING before it deletes instances that are running with no scheduled actions.

If the group is part of a backend service that has enabled connection draining, it can take up to an additional 60 seconds after the connection draining duration has elapsed before the VM instance is removed or deleted.

You can resize a managed instance group by using the Google Cloud Console, the gcloud tool, or the Compute Engine API.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to Instance groups

  2. Under the Name column of the list, click the name of the instance group where you want to change the group size.

  3. Click Edit to modify this managed instance group.

  4. Under Number of instances, specify the number of instances that you want to include in this managed instance group. If Autoscaling is enabled, the group is automatically adding or removing instances as necessary. However, you can change the Minimum number of instances and the Maximum number of instances values to indirectly adjust the group size through Autoscaler.

  5. Click Save to apply the new template.

gcloud

Use the resize command.

gcloud compute instance-groups managed resize INSTANCE_GROUP_NAME \
    --size NEW_SIZE \
    --zone ZONE

API

Call the resize method for a zonal or regional MIG resource. Specify the new instance group size as a parameter.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME/resize?size=NEW_SIZE

If the instance group is a regional managed instance group, replace zones/ZONE with regions/REGION.

After you make a request to resize a managed instance group, the VMs start or stop as soon as the system can provision or delete them. This process can take a significant amount of time depending on the number of instances in the group. Verify the status of instances in your managed instance group.

Disabling creation retries mode during resize

By default, if the initial creation of a VM instance fails, the managed instance group continuously retries to create the VM, until the VM is successfully created. However, if you do not want automatic creation retries, you can disable the creation retry mode by setting the --nocreation-retries flag when you resize the instance group. In this mode, the managed instance group attempts to create all instances only once. If there is an error during instance creation, the managed instance group gives up on this instance, removes it from the group's list of managed instances, and decreases the target size of the managed instance group.

This mode applies only during the first creation attempt of a VM. If a VM is created successfully while this mode is enabled, the VM behaves the same way as all other VMs created with a regular resize request. In particular, if a running VM dies unexpectedly at a later time and needs to be recreated, this mode does not affect the recreation behavior in that scenario.

The disable creation retry mode is especially useful in scenarios where you have systems automatically creating groups of VMs where an exact number of VMs is not required. You might prefer to stabilize quickly on the size of the managed instance group and be flexible in the number of VMs in the group, rather than to wait indefinitely until all the requested instances are created, which might be delayed temporarily or permanently because of quota errors or other unrelated issues.

To resize a managed instance group in the disabled creation retries mode, use the gcloud tool or the Compute Engine API.

gcloud

Use the beta resize command with the --no-creation-retries flag.

gcloud beta compute instance-groups managed resize INSTANCE_GROUP_NAME --size NEW_SIZE \
    --no-creation-retries \
    --zone ZONE

API

Call the beta resizeAdvanced method for a zonal MIG resource. Specify the new group size and the noCreationRetries field in the request body.

POST https://compute.googleapis.com/compute/beta/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME/resizeAdvanced

{
 "targetSize": SIZE
 "noCreationRetries": true
}

You can see which instances are being created and in which mode by using the listManagedInstances method. Instances that are being created in the disabled creation retries mode have a currentAction of CREATING_WITHOUT_RETRIES.

Creating instances with specific names in MIGs

If you have a system that depends on specific names, you can create VMs with those names. The names that you assign to these managed instances persist if the VM is recreated. See Adding instances with specific names.

Deleting specific instances from a group

You can delete individual VM instances in a managed instance group. Deleting instances reduces the specified targetSize of the instance group and removes the VMs from any target pools of which they are a member.

Deleting instances from a managed instance group does not change any specified autoscaler settings. If you delete instances from a managed instance group, the autoscaler might detect an increase in the workload on the other instances in the group and increase the group size back to its previous level. To prevent this, turn off or delete the autoscaler before attempting to delete the instances.

If the group is part of a backend service that has enabled connection draining, it can take up to an additional 60 seconds after the connection draining duration has elapsed before the VM instance is removed or deleted.

Delete instances from a managed instance group by using the Google Cloud Console, the gcloud tool, or the Compute Engine API.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to Instance groups

  2. Under the Name column of the list, click the name of the instance group where you want to delete individual instances. A page opens with the instance group properties and a list of instances that are included in the group.

  3. In the Members section, from the list of instances select one or more instances that you want to delete.

  4. Click Delete instance. The selected instances are deleted.

gcloud

Use the delete-instances command.

gcloud compute instance-groups managed delete-instances INSTANCE_GROUP_NAME \
    --instances example-i3n2,example-z2x9 \
    --zone ZONE

API

Call the deleteInstances method for a zonal or regional MIG resource. In the request body, include the URLs to one or more instances that you want to delete.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME/deleteInstances

{
 "instances": [
  "zones/ZONE/instances/example-instance-i3n2",
  "zones/ZONE/instances/example-instance-l6n1"
 ]
}

If the instance group is a regional managed instance group, replace zones/ZONE with regions/REGION.

After you make a request to delete instances in a managed instance group, the instances stop as soon as the system can delete them. This process can take a significant amount of time depending on the number of instances that you delete from the group. Verify the status of instances in your managed instance group.

Abandoning instances from a group

You can separate a VM instance from a managed instance group to more easily debug issues with individual VMs without affecting the group as a whole. Abandoning an instance from a group also removes that VM from load balancers that were assigned to the managed instance group. Target pools that were manually assigned to specific individual instances are not removed.

Abandoning instances reduces the specified targetSize of the instance group, but does not change any specified autoscaler settings. Managed instance groups with an autoscaler continue to add or remove VMs automatically as necessary.

If the group is part of a backend service that has enabled connection draining, it can take up to an additional 60 seconds after the connection draining duration has elapsed before the VM instance is removed or deleted.

Abandon instances from a managed instance group by using the Google Cloud Console, the gcloud tool, or the Compute Engine API.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to Instance groups

  2. Under the Name column of the list, click the name of the instance group from which you want to remove instances. A page opens with the instance group properties and a list of instances that are included in the group.

  3. In the Members section, from the list of instances select one or more instances that you want to remove from the group.

  4. Click Remove from group. The selected instances leave the group, but continue to run outside of the group.

gcloud

To remove an instance from the instance group without deleting the instance, use the abandon-instances command.

gcloud compute instance-groups managed abandon-instances INSTANCE_GROUP_NAME \
    --instances example-i3n2,example-z2x9 \
    --zone ZONE

API

Call the abandonInstances method for a zonal or regional MIG resource. In the request body, include the URLs to one or more instances that you want to abandon.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME/abandonInstances

{
 "instances": [
  "zones/ZONE/instances/example-instance-i3n2",
  "zones/ZONE/instances/example-instance-l6n1"
 ]
}

If the instance group is a regional managed instance group, replace zones/ZONE with regions/REGION.

After you make a request to abandon instances from a managed instance group, the group removes the instances as soon as possible. Verify the status of instances in your managed instance group.

Recreating instances in the group

Recreating a managed instance deletes the specified VM and creates a new VM using the instance template that is assigned to the managed instance group.

Use this method to update selected VMs so that they use the latest instance template. If you need to recreate all of the VMs in a managed instance group, start a rolling update instead.

If the group is part of a backend service that has enabled connection draining, it can take up to an additional 60 seconds after the connection draining duration has elapsed before the VM instance is removed or deleted.

Recreate selected VMs in managed instance group by using the gcloud tool, or the Compute Engine API.

gcloud

Use the recreate-instances command.

gcloud compute instance-groups managed recreate-instances INSTANCE_GROUP_NAME \
    --instances example-i3n2,example-z2x9 \
    --zone ZONE

API

Call the recreateInstances method for a zonal or regional MIG resource. In the request body, include the URLs to one or more instances that you want to recreate.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME/recreateInstances

{
 "instances": [
  "zones/ZONE/instances/example-instance-i3n2",
  "zones/ZONE/instances/example-instance-l6n1"
 ]
}

If the instance group is a regional managed instance group, replace zones/ZONE with regions/REGION.

After you make a request to recreate instances in a managed instance group, the new VMs start as soon as the system can provision them. This process can take a significant amount of time depending on the number of instances that you recreate. Verify the status of instances in your managed instance group.

Deleting a managed instance group

When you delete a managed instance group, all VMs in the group are deleted. If you must keep any of the VMs in this managed instance group, abandon the instances first to remove the VMs from the group. Then, delete the managed instance group.

When you delete a managed instance group and its instances by using the Google Cloud Console or the gcloud tool, any attached autoscalers are automatically deleted. However, if you use the Compute Engine API, you must first issue separate requests to delete any attached autoscalers with autoscalers.delete or regionAutoscalers.delete.

Console

  1. In the Cloud Console, go to the Instance groups page.

    Go to Instance groups

  2. Select one or more groups on the list that you want to delete.

  3. Click Delete to delete the group and all of the VMs in the managed instance group.

gcloud

Use the delete command.

gcloud compute instance-groups managed delete INSTANCE_GROUP_NAME \
    --zone ZONE

API

Call the delete method for a zonal or regional MIG resource.

DELETE https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/INSTANCE_GROUP_NAME

If the instance group is a regional managed instance group, replace zones/ZONE with regions/REGION.

Creating groups of preemptible instances

You can use managed instance groups to quickly create multiple preemptible instances, which can reduce the costs of the VMs in your managed instance groups. For example, you can create a group of preemptible instances, use them to run a batch processing task, and then delete the group when the task is complete.

To create a group of preemptible instances, set the premptible option in an instance template, and then use the template to create the managed instance group.

Console

  1. In the Cloud Console, go to the Instance templates page.

    Go to the Instance templates page

  2. Click New instance template.

  3. Fill in the properties that you want for your instance template.

  4. Click Show advanced options to expand the Availability policies section.

  5. Set Preemptibility to On.

  6. Click Create to create the template.

  7. Use this template to create a managed instance group

gcloud

Create an instance template by using the instance-templates create command. Include the --preemptible flag.

gcloud compute instance-templates create INSTANCE_TEMPLATE \
    --preemptible

After you create the instance template, use it to create a managed instance group.

API

Call the instanceTemplates.insert method to create a new instance template. Include the scheduling.preemptible property and set it to true.

{
"name": "INSTANCE_TEMPLATE",
"properties": {
  "machineType": "zones/ZONE/machineTypes/MACHINE_TYPE",
  "networkInterfaces": [
    {
      "network": "global/networks/default",
      "accessConfigs":
      [
        {
          "name": "external-IP",
          "type": "ONE_TO_ONE_NAT"
        }
      ]
    }
  ],
  "scheduling":
  {
    "preemptible": true
  },
  "disks":
  [
    {
      "type": "PERSISTENT",
      "boot": true,
      "mode": "READ_WRITE",
      "initializeParams":
      {
        "sourceImage": "projects/debian-cloud/global/images/family/debian-9"
      }
    }
  ]
  }
}

After you create the instance template, use it to create a managed instance group.

Understanding the instanceTemplate and versions fields

When you create a managed instance group, you supply an instance template that the managed instance group uses to create individual VM instances. By default, Compute Engine describes the instance template used in two separate Compute Engine API properties: the top-level instanceTemplate property and the versions property. For example, in the following managed instance group, notice that both the instanceTemplate and versions fields are populated:

{

 "name": "example-group",
 "zone": "zones/us-central1-a",
 "instanceTemplate": "global/instanceTemplates/example-it",
 "versions": [
  {
   "name": "v3",
   "instanceTemplate": "global/instanceTemplates/example-it",
   "targetSize": {
    "calculated": 3
   }
  }
 ]...
}

Compute Engine automatically populates both the top-level instanceTemplate field and the versions field for backward compatibility. We recommend specifying the versions field and omitting the top-level instanceTemplates field whenever you can. However, if your application code currently sets the top-level instanceTemplate field, it is still a valid request.

To learn more, read about the Relationship between versions and instanceTemplate fields.

(Advanced) Canarying instance templates with a managed instance group

You can create a managed instance group where there are two groups of VMs that use different instance templates. For example, you might want to create a managed instance group that has 20 VM instances and 10 VMs should run on a specific operating image while the rest run on a different operating system image. This feature lets you compare the two different instance template versions before deciding on one.

For more information, read about Canary updates.

Managed instance groups and IAM

All the operations performed by Compute Engine as part of the managed instance group are performed by the Google APIs service account for your project. This per-project service account has an email address like the following, where PROJECT_ID is the numerical ID of the corresponding project:

PROJECT_ID@cloudservices.gserviceaccount.com

The Google APIs service account is different from the default Compute Engine service account.

It is up to you to ensure that the service account used by the managed instance group has enough privileges to create resources based on the instance template. In particular, this means that the service account needs to be granted the compute.instanceAdmin.v1 and, optionally, the serviceAccountUser role, to create and manage VMs in the instance group. The serviceAccountUser role is required only if the managed instance group will be creating VMs that can run as a service account. Keep in mind that this account is also used by other processes, including Deployment Manager.

When you create a managed instance group or update an instance template, Compute Engine validates that the Google APIs service account has the following:

  • serviceAccountUser role. Important if you plan to create instances that can run as a service account.
  • Permissions to all the resources referenced from instance templates, such as images, disks, VPC networks and subnets.

To learn more about service accounts, read the Service accounts overview.

Updating all instances in a managed instance group

See Updating instances in a MIG.

What's next