Getting info about MIGs and managed instances

Use the methods on this page to get info about your managed instance groups (MIGs) and to get information about the virtual machine (VM) instances within each MIG.

Before you begin

Getting info about MIGs

You can retrieve information about groups themselves, for example, to inspect the policies that are attached to the group and to check if the group is stable or undergoing changes. To get info about the VM instances within a MIG, see Getting info about managed instances.

Listing MIGs

To get a list of your managed instance groups, use the console, the gcloud tool, or the API.

Console

  1. In the Google Cloud Console, go to the Instance groups page.

    Go to the Instance groups page.

    If you have existing instance groups, the page lists those groups, including both managed instance groups and unmanaged instance groups.

    Managed instance groups have a value in the Template column.

gcloud

List all managed instance groups within a project:

gcloud compute instance-groups managed list

API

To list all managed instance groups within a zone, construct a GET request to the instanceGroupManagers.list method:

GET https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers

For regional (multiple-zone) managed instance groups, use the regionInstanceGroupManagers resource instead: regionInstanceGroupManagers.list.

GET https://compute.googleapis.com/compute/v1/projects/project-id/regions/region/instanceGroupManagers

Getting a MIG's properties

To get information about a specific MIG, including attached policies (for example, autohealing policies), use the console, the gcloud tool, or the API.

To get the group's id, you must use the gcloud tool or the API.

Console

  1. In the Google Cloud Console, go to the Instance groups page.

    Go to the Instance groups page.

    If you have existing instance groups, the page lists those groups. The page also lists unmanaged instance groups.

  2. Under the Name column of the list, click the name of the instance group that you want to examine. A page opens with the instance group properties and a list of instances that are included in the group.

gcloud

To get information about a specific group:

gcloud compute instance-groups managed describe instance-group \
    [--zone zone | --region region]

The command returns details about the group, including attached policies.

API

To get information about a specific group, construct a GET request to the instanceGroupManagers.get method and include the name of a specific managed instance group:

GET https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group

For regional (multiple-zone) managed instance groups, use the regionInstanceGroupManagers resource instead: regionInstanceGroupManagers.list and regionInstanceGroupManagers.get.

GET https://compute.googleapis.com/compute/v1/projects/project-id/regions/region/instanceGroupManagers/instance-group

The response includes details about the group, including attached policies except for autoscaling policies.

To get information about autoscalers, construct a GET request to the autoscalers.get method and include the name of a specific managed instance group:

GET https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/autoscalers/instance-group

For regional (multiple-zone) managed instance groups, use the regionAutoscalers resource instead: regionAutoscalers.get.

GET https://compute.googleapis.com/compute/v1/projects/project-id/regions/region/regionAutoscalers/instance-group

Checking whether a MIG is stable

Several commands and requests create, delete, and modify instances in a managed instance group. Those operations are returned as DONE after the group has scheduled actions to create, delete, or update those instances. However, this doesn't mean that instances in the group have been created, deleted, or updated until those underlying actions are complete. You must verify the status of the group, or alternatively verify the status of those instances.

A managed instance group is stable when all instances in the managed instance group are not undergoing any change, no future changes are scheduled for the instances, and the group itself is not being modified.

You might need to know whether the group is stable when you have dependencies that rely on all instances in the group being healthy and serving. For example, you might want to start scheduling jobs, sending traffic, or assigning key ranges after you have confirmation that all the VMs are running and healthy.

You can verify that a managed instance group is running and healthy by checking the value of the status.isStable field.

gcloud

Use the instance group describe command:

gcloud compute instance-groups managed describe instance-group-name \
    [--zone zone | --region region]

gcloud tool returns detailed information about the instance group including the status.isStable field.

To pause a script until the group is stable, use the wait-until command with the --stable flag. For example:

gcloud compute instance-groups managed wait-until instance-group-name \
    --stable \
    [--zone zone | --region region]
Waiting for group to become stable, current operations: deleting: 4
Waiting for group to become stable, current operations: deleting: 4
...
Group is stable

The command returns after status.isStable is set to true for the group.

API

For a zonal MIG, make a POST request to the instanceGroupManagers.get method:

POST https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name/get

For a regional managed instance group, replace zones/zone with regions/region:

POST https://compute.googleapis.com/compute/v1/projects/project-id/regions/region/instanceGroupManagers/instance-group-name/get

The API returns detailed information about the instance group including the status.isStable field.

status.isStable set to false indicates that changes are active, pending, or that the managed instance group itself is being modified.

status.isStable set to true indicates the following:

  • None of the instances in the managed instance group are undergoing any type of change and the currentAction for all instances is NONE.
  • No changes are pending for instances in the managed instance group.
  • The managed instance group itself is not being modified.

Managed instance groups can be modified in numerous ways. For example:

  • You make a request to roll out a new instance template.
  • You make a request to create, delete, resize or update instances in the group.
  • An autoscaler requests to resize the group.
  • An autohealer resource is replacing one or more unhealthy instances in the managed instance group.
  • In a regional managed instance group, some of the instances are being redistributed.

As soon as all actions are finished, status.isStable is set to true again for that managed instance group.

For individual VM instances, see Checking the status of managed instances.

Viewing historical autohealing operations

If you have configured autohealing, you can view past autohealing events.

Monitoring rolling updates

If you initiated a rolling update, and want to know when the update is complete, see Monitoring updates.

Viewing stateful configuration

If you configured a stateful MIG and want to review your configuration, see Viewing stateful configuration.

Getting info about the managed VM instances within a MIG

Listing a MIG's managed instances

To get a list of the managed instances in a group, including their states and templates, use the console, the gcloud tool, or the API.

Console

  1. In the Google Cloud Console, go to the Instance groups page.

    Go to the Instance groups page.

    If you have existing instance groups, the page lists those groups. The page also lists unmanaged instance groups.

  2. Under the Name column of the list, click the name of the instance group that you want to examine. A page opens with the instance group properties and a list of instances that are included in the group.

gcloud

Use the gcloud tool list-instances subcommand.

gcloud compute instance-groups managed list-instances instance-group \
    [--zone zone | --region region]

The command returns a list of instances and their details, including their current status, instance template, and last error.

API

For a zonal MIG, construct a GET request to the instanceGroupManagers.listManagedInstances method and include the name of a specific managed instance group:

GET https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group

For regional (multiple-zone) managed instance groups, use the regionInstanceGroupManagers resource instead: regionInstanceGroupManagers.list.

GET https://compute.googleapis.com/compute/v1/projects/project-id/regions/region/instanceGroupManagers/instance-group

The response includes a list of instances and their details, including their current status and instance template.

Checking the status of managed instances

Several commands and requests create, delete, and modify instances in a managed instance group. Those operations are returned as DONE after the group has scheduled actions to create, delete, or update those instances. However, this doesn't mean that instances in the group have been created, deleted, or updated until those underlying actions are complete. You must verify the status of those instances, or alternatively verify the status of the group itself.

You can see the currentAction being performed and the status of each instance in a managed instance group with the gcloud command-line tool or the API.

gcloud 

gcloud compute instance-groups managed list-instances instance-group-name \
[--filter="zone:(zone)" | --filter="region:(region)"]

gcloud returns a list of instances in the instance group and their respective statuses and current actions. For example:

NAME               ZONE           STATUS    ACTION    INSTANCE_TEMPLATE  VERSION_NAME  LAST_ERROR
vm-instances-9pk4  us-central1-f            CREATING  my-new-template
vm-instances-h2r1  us-central1-f  STOPPING  DELETING  my-old-template
vm-instances-j1h8  us-central1-f  RUNNING   NONE      my-old-template
vm-instances-ngod  us-central1-f  RUNNING   NONE      my-old-template

API

In the API, make a GET request to the regionInstanceGroupManagers.listManagedInstances method. For a zonal managed instance group, use the instanceGroupManagers.listManagedInstances method.

GET https://compute.googleapis.com/compute/v1/projects/project-id/regions/region/instanceGroupManagers/instance-group-name/listManagedInstances

The API returns a list of instances for the group including each instance's instanceStatus and currentAction.

{
 "managedInstances": [
  {
   "instance": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/vm-instances-prvp",
   "id": "5317605642920955957",
   "instanceStatus": "RUNNING",
   "instanceTemplate": "https://compute.googleapis.com/compute/v1/projects/project-id/global/instanceTemplates/instance-template-name",
   "currentAction": "REFRESHING"
  },
  {
   "instance": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/vm-instances-pz5j",
   "currentAction": "DELETING"
  },
  {
   "instance": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/vm-instances-w2t5",
   "id": "2800161036826218547",
   "instanceStatus": "RUNNING",
   "instanceTemplate": "https://compute.googleapis.com/compute/v1/projects/project-id/global/instanceTemplates/instance-template-name",
   "currentAction": "REFRESHING"
  }
 ]
}

For each instance in a managed instance group, the status of the instance is described by its instanceStatus field. To see a list of valid instanceStatus field values, see Checking an instance's status.

If the instance is undergoing some type of change, the currentAction field is populated with one of the following actions to help you track the progress of the change. Otherwise, the currentAction field is NONE.

Possible currentAction values are:

  • ABANDONING. The instance is being removed from the managed instance group.
  • CREATING. The instance is in the process of being created.
  • CREATING_WITHOUT_RETRIES. The instance is being created without retries; if the instance isn't created on the first try, the managed instance group doesn't try to replace the instance again.
  • DELETING. The instance is in the process of being deleted.
  • RECREATING. The instance is being replaced.
  • REFRESHING. The instance is being removed from its current target pools and being readded to the list of current target pools (this list might be the same or different from existing target pools).
  • RESTARTING. The instance is in the process of being restarted using the stop and start methods.
  • VERIFYING. The instance has been created and is in the process of being verified.
  • NONE. No actions are being performed on the instance.

To check if the group as a whole is stable or is undergoing changes, see Checking the status of the group.

Checking health states

If you have configured autohealing for your managed instance group, you can review the health state of each VM. See Checking health status.

Listing instance errors

If your managed instances encountered errors, you can view those errors to diagnose and mitigate the cause.

Use the Google Cloud Console, the gcloud tool, or the Compute Engine API to list errors.

Console

  1. In the Google Cloud Console, go to the Instance groups page.

    Go to the Instance groups page

  2. Under the Name column of the list, click the name of the instance group that you want to examine. A page opens with the instance group properties and a list of instances that are included in the group.

  3. Click Errors.

gcloud

Use the instance-groups managed list-errors command to list the most recent errors in the group

gcloud compute instance-groups managed list-errors instance-group-name 

     [--zone zone | --region region]

For example:

gcloud compute instance-groups managed list-errors example-group \
    --region us-west1
INSTANCE_URL                                                                                        ACTION    ERROR_CODE      ERROR_MESSAGE                                                                                     TIMESTAMP                      INSTANCE_TEMPLATE                                                                                                            VERSION_NAME

https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-a/instances/igm-1-v0hl  CREATING  QUOTA_EXCEEDED  Instance 'igm-1-v0hl' creation failed: Quota 'CPUS' exceeded.  Limit: 1.0 in zone us-central1-a.  2019-07-11T07:44:40.156-07:00  https://www.googleapis.com/compute/v1/projects/my-project/global/instanceTemplates/instance-template-1
https://www.googleapis.com/compute/v1/projects/my-project/zones/us-central1-a/instances/igm-1-qp3d  CREATING  QUOTA_EXCEEDED  Instance 'igm-1-qp3d' creation failed: Quota 'CPUS' exceeded.  Limit: 1.0 in zone us-central1-a.  2019-07-11T07:44:40.081-07:00  https://www.googleapis.com/compute/v1/projects/my-project/global/instanceTemplates/instance-template-1
...

API

Construct a GET request to the listErrors method for a zonal or regional managed instance group.

For example:

GET https://www.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name/listErrors&maxResults=100

The request gets a response similar to:


{
 "items": [
  {
   "error": {
    "code": "QUOTA_EXCEEDED",
    "message": "Instance 'example-mig-m1sz' creation failed: Quota 'CPUS' exceeded.  Limit: 50.0 in zone us-central1-c."
   },
   "timestamp": "2019-01-03T18:47:28.457-08:00",
   "instanceDetails" : {
    "instance": "zones/us-central1-c/instances/example-mig-m1sz",
    "action": "CREATING",
    "version": {
     "name": "example-version-name",
     "instanceTemplate": "global/instanceTemplates/example-template",
    }
   }
  }
 ],
 ...
 "nextPageToken": "Ch0yMDE5LTA3LTI5VDExOjA4OjA2LjYxMi0wNzowMBJ5aHR0cHM6Ly93d3cuZ29vZ2xlYXBpcy5jb20vY29tcHV0ZS9zdGFnaW5nX2FscGhhL3Byb2plY3RzL215cnpha2VyZXltcy1zdGFnaW5nL3pvbmVzL3VzLWNlbnRyYWwxLWlyMS9pbnN0YW5jZXMvaWdtLTEtcXAzZA=="
}

The listErrors API supports list pagination. To get the next page of results, use the nextPageToken field from one response in a subsequent request. For example: GET https://www.googleapis.com/compute/v1/projects/project-id/zones/zone/instanceGroupManagers/instance-group-name/listErrors&maxResults=100&nextPageToken=next-page-token, replacing next-page-token with the token that was obtained from the previous response.

The list of possible errors includes, but is not limited to, the following examples:

  • A new instance cannot be created due to:
    • Unavailability of preemptible capacity.
    • Lack of quota, for example, for CPUs in a given region.
    • The limit on the number of VM instances was reached for the network associated with the group.
    • The disk image specified in the instance template's properties.disks[].initializeParams.sourceImage field was deleted, or its permissions have changed.
    • The credit card on the billing account associated with the project refused a payment.
  • An existing instance cannot be deleted because:
    • The service account that group uses doesn't have compute.instances.delete permission.
    • Too many concurrent deletions were requested, exceeding rate limits.
  • An instance cannot be added to a target pool because the target pool was deleted.

After inspecting such errors, you can mitigate them. For example:

  • Move your workload to a different zone that has preemptible capacity.
  • Request a quota increase in a given region.
  • Add the compute.instances.delete permission to the service account used by the group.

If an action that is triggered by a managed instance group fails (for example, if an autoscaler encounters an error when attempting to increase the group's size), the group might immediately retry that action. If the last immediate retry attempt fails, the MIG records the error, and you can view the error by listing instance errors. After some backoff time, the group again retries the failed action. If the error persists, it might be recorded multiple times.

Compute Engine retains the most recent 500 errors for the last 8 days even if the underlying instances no longer exist or could not be created.

Checking if a VM instance is part of a MIG

To check if a specific VM is currently a member of a group, you can use the console or API.

Console

  1. In the Google Cloud Console, go to the VM instances page.

    Go to the VM instances page

  2. Click an instance to access its VM instance details.
  3. If the VM instance is a member of a managed instance group, the name of that managed instance group appears under the heading In use by. If the VM instance is not a member of a group, the heading In use by doesn't appear on the instance details page.

API

See Viewing referrers to VM instances.

Alternatively, to identify whether a VM is or was a member of a managed instance group, look for these two metadata keys in the instance's metadata:

  • instance-template indicates the template the VM was created from.
  • created-by indicates the managed instance group that created the VM.

Even if you abandon the instance, the VM will still have these metadata entries, unless you manually remove them.

For example, if there was an VM named random-instance-biy and you wanted to know whether the VM was created by a managed instance group, you can describe the instance and look for the metadata keys. For example:

gcloud compute instances describe random-instance-biy \
    --zone us-central1-f

gcloud tool returns a response similar to:

canIpForward: false
cpuPlatform: Intel Ivy Bridge
creationTimestamp: '2016-08-24T14:11:38.012-07:00'
...
metadata:
  items:
  - key: instance-template
    value: projects/123456789012/global/instanceTemplates/example-it
  - key: created-by
    value: projects/123456789012/zones/us-central1-f/instanceGroupManagers/igm-metadata

Viewing the preserved state of a managed instance

If you configured a stateful MIG and want to view the preserved state of each managed instance, see Viewing stateful configuration and preserved state.

What's next