Configuring stateful metadata in MIGs

Instance metadata is useful for setting properties for and communicating with your applications through the metadata server. For example, you can use metadata to configure the identity of the virtual machine (VM) instance, environment variables, information about cluster architecture, or data range that a VM is responsible for.

By configuring stateful metadata in a managed instance group (MIG), you ensure that instance-specific metadata is preserved on managed instance autohealing, update, and recreate events.

Configure stateful metadata individually for VM instances in a MIG by setting it in per-instance configs and applying the configuration. You can set a per-instance config on instance creation or against existing managed instances. After the per-instance config is applied, the MIG stores stateful metadata in the preserved state from config (preservedStateFromConfig) field of a managed instance.

Before you begin

Limitations

Stateful MIGs have the following limitations:

  • You cannot use autoscaling with stateful MIGs.
  • You cannot use proactive rolling updates if you configure stateful disks or stateful metadata.
    • You can control updates and limit disruption by updating specific instances instead.
    • If you use custom instance names and don't configure stateful disks or metadata, you can use proactive updates, but, to preserve instance names, you must set the replacement method to RECREATE.
  • When you permanently delete an instance (either manually or by resizing), the MIG does not preserve the instance's stateful metadata.
  • For stateful regional MIGs, you must disable proactive redistribution (set the redistribution type to NONE) to prevent deletion of stateful instances by automatic cross-zone redistribution.

Setting stateful metadata on instance creation

Set stateful metadata when manually creating instances in a MIG. This is useful for migrating a stateful application on standalone VMs to a stateful MIG and when creating stateful instances.

When you manually create an instance in a MIG and supply stateful metadata, the MIG performs the following tasks:

  1. Creates a managed instance from the instance template using the provided instance name.
  2. Creates a per-instance config with the provided stateful metadata and sets that metadata on the instance.
  3. Stores the stateful metadata in the preserved state from config (preservedStateFromConfig) of the associated managed instance.

gcloud

To create a managed instance with a custom name and set stateful metadata on that VM, use the gcloud compute instance-groups managed create-instance command with the --stateful-metadata flag.

gcloud compute instance-groups managed create-instance NAME \
  --instance INSTANCE_NAME \
  --stateful-metadata KEY=VALUE[,KEY=VALUE,...]

Replace the following:

  • NAME: the name of the MIG in which to create an instance
  • INSTANCE_NAME: the name of the instance to create
  • KEY and VALUE: stateful metadata key-value pairs to set individually for the instances in addition to metadata defined in the instance template
    • The key values that you set here take priority over any conflicting key values from the instance template

Example

You need to deploy a cluster of nodes, example-cluster, that can operate in one of two modes: active or standby. You set the mode individually for each VM in a cluster using metadata, for example: mode:active. You also configure how elaborate logging should be for each node, using a logging metadata key that can be set to basic or elaborate. The application on the node is configured using values from instance metadata.

To create an active node, node-12, with elaborate logging, you would run the following command:

gcloud compute instance-groups managed create-instance example-cluster \
  --instance node-12 \
  --stateful-metadata mode=active,logging=elaborate

The command creates a VM, node-12, in the example-cluster MIG and sets two metadata key-value pairs, mode:active and logging:elaborate, for the new instance.

API

To create one or multiple managed instances in a MIG with custom VM names and to set stateful metadata individually on these VMs, use the instanceGroupManagers.createInstances method. For a regional MIG, use the regionInstanceGroupManagers.createInstances method.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/NAME/createInstances
{
  "perInstanceConfigs": [
    {
      "name": "INSTANCE_NAME",
      "preservedState" : {
        "metadata": {
          "KEY" : "VALUE",
          ...
        }
      }
    },
    ...
  ]
}

Replace the following:

  • PROJECT_ID: the project ID for the request
  • ZONE: the zone where the MIG is located (applies to a zonal MIG)
    • For a regional MIG, replace zones/ZONE with regions/REGION and specify the region of the MIG
  • NAME: the name of the MIG in which to create an instance
  • INSTANCE_NAME: the name of the instance to create
  • KEY and VALUE: stateful metadata key-value pairs to set individually for the instances in addition to metadata defined in the instance template
    • The key values that you set here take priority over any conflicting key values from the instance template

Example

You need to deploy a cluster of nodes, example-cluster, that can operate in one of two modes: active or standby. You set the mode individually for each VM in a cluster using metadata, for example: mode:active. You also configure how elaborate logging should be for each node, using a logging metadata key that can be set to basic or elaborate. The application on the node is configured using values from instance metadata.

To create an active node, node-12, with elaborate logging, execute the following method:

POST https://compute.googleapis.com/compute/v1/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-cluster/createInstances

{
  "instance": [
    {
      "name": "node-12",
      "preservedState" : {
        "metadata": {
          "mode":"active",
          "logging":"elaborate"
        }
      }
    }
  ]
}

The method creates a VM, node-12, in the example-cluster MIG and sets two metadata key-value pairs, mode:active and logging:elaborate, for the new instance.

Setting, modifying, and removing stateful metadata individually for existing VM instances

Set, modify, or remove stateful metadata for an existing instance in a MIG by setting it in an associated per-instance config and then applying the configuration by updating the instance.

gcloud

To configure stateful metadata individually for a VM instance in a MIG, set or remove stateful metadata in the associated per-instance config. If you apply the configuration to the instance at the same time (--update-instance), you can choose whether to keep the instance running, restart it, or recreate it. If you do not apply the configuration (--no-update-instance), your changes do not take effect until you recreate or update the instance.

If a per-instance config doesn't exist for a given instance, use the gcloud compute instance-groups managed instance-configs create command with one of the following flags:

gcloud compute instance-groups managed instance-configs create NAME \
  --instance INSTANCE_NAME \
  --stateful-metadata KEY=VALUE[,KEY=VALUE,...] \
  [--no-update-instance | --update-instance] \
  [--instance-update-minimal-action MINIMAL_ACTION]

If a per-instance config already exists for a given instance, use the gcloud compute instance-groups managed instance-configs update command with:

  • The --stateful-metadata flag for setting or modifying metadata, or
  • The --remove-stateful-metadata flag for removing instance-specific stateful metadata.
gcloud compute instance-groups managed instance-configs update NAME \
  --instance INSTANCE_NAME \
  [--stateful-metadata KEY=VALUE[,KEY=VALUE,...]] \
  [--remove-stateful-metadata KEY[,KEY,...]] \
  [--no-update-instance | --update-instance] \
  [--instance-update-minimal-action MINIMAL_ACTION]

Replace the following:

  • NAME: The name of the managed instance group.
  • INSTANCE_NAME: The name of the instance for which to configure stateful metadata.
  • KEY=VALUE: Stateful metadata key-value pairs to set individually for the instance in addition to the metadata defined in the instance template. The key values that you set here take priority over any conflicting key values from the instance template.
  • KEY: Instance-specific stateful metadata keys to remove from the per-instance config.
    • If a value for the key is not defined by the instance template, the key-value pair is removed completely from the instance when the change is applied.
    • If a value for the key is defined by the instance template, the value from the instance template is set on the instance when the change is applied.
  • MINIMAL_ACTION: Perform at least the specified action when applying the per-instance config update to the instance. A MINIMAL_ACTION can only be set when the --update-instance flag is used. The value must be one of the following:

    • none: No action.
    • refresh: Apply updates that are possible to apply without stopping the instance.
    • restart: Stop the instance and then start it again.
    • replace: Recreate the instance.

    If omitted, the least disruptive action required by the update is used.

Example

You have a cluster of nodes, example-cluster, that can operate in one of two modes: active or standby. You set the mode individually for each VM in the cluster using metadata, for example: mode:active. You also configure how elaborate logging should be for each node, using a logging metadata key that can be set to basic or elaborate. The application on each node consumes the values from instance metadata.

The instance template defines mode:active and logging:basic metadata to be used as default for all instances. You have set logging:elaborate in a per-instance config for the node-12 VM in the cluster. Now, you want to switch node-12 to standby mode and switch logging to basic for this VM.

To switch the node-12 instance to standby and its logging to basic, run the following command:

gcloud compute instance-groups managed instance-configs update example-cluster \
  --instance node-12 \
  --stateful-metadata mode=standby \
  --remove-stateful-metadata logging

The command does the following:

  1. Sets mode:standby metadata in the per-instance config that is associated with the VM, node-12, in the example-cluster MIG.
  2. Removes logging:elaborate metadata from the per-instance config for the node-12 instance.
  3. Applies the per-instance config change to the node-12 VM:
    • Sets mode:standby metadata, according to the config.
    • Sets logging:basic metadata from the instance template because the value for the logging key is no longer defined by the per-instance config.
  4. The change is applied to the VM immediately by default because the flag --no-update-instance is omitted.
  5. The VM keeps running during the update because the --instance-update-minimal-action flag is omitted and the least disruptive action is chosen for the update by default, in this case: refresh.

API

To configure stateful metadata individually for existing VM instances in a MIG, set or remove the metadata in the associated per-instance configs. Then update the instance to apply the configuration.

If per-instance configs do not yet exist for the given instances, use the instanceGroupManagers.updatePerInstanceConfigs method with stateful metadata:

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/NAME/updatePerInstanceConfigs

{
  "perInstanceConfigs": [
    {
      "name": "INSTANCE_NAME",
      "preservedState" : {
        "metadata": {
          "KEY": "VALUE",
          ...
        }
      },
      "fingerprint: "FINGERPRINT"
    },
    ...
  ]
}

If per-instance configs already exist for the given instances, use the instanceGroupManagers.patchPerInstanceConfigs method

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instanceGroupManagers/NAME/patchPerInstanceConfigs

{
  "perInstanceConfigs": [
    {
      "name": "INSTANCE_NAME",
      "preservedState" : {
        "metadata": {
          "KEY": "VALUE",
          ...
        }
      },
      "fingerprint: "FINGERPRINT"
    },
    ...
  ]
}

Replace the following:

  • PROJECT_ID: The project ID for the request.
  • ZONE: The zone where the MIG is located (applies to a zonal MIG).
    • For a regional MIG, replace zones/ZONE with regions/REGION and specify the region of the MIG.
  • NAME: The name of the MIG.
  • INSTANCE_NAME: The name of the VM for which to configure stateful metadata.
  • KEY and VALUE: Stateful metadata key-value pairs to set individually for the instances in addition to any metadata defined in the instance template.
    • Stateful metadata values defined for the keys that already exist in the instance template override the values from the instance template.
    • Other metadata entries from the instance template remain unaffected and available.
    • Providing null as a value removes the key from the per-instance config.
  • FINGERPRINT: (Optional). The fingerprint for the given config if it already exists. Used for optimistic locking.

The updatePerInstanceConfigs and patchPerInstanceConfigs methods update the specified per-instance configs but do not apply the config updates to the associated VM instances. The changes are applied to a VM when you update or recreate the instance. To apply the changes to a VM, you can apply the update manually or use the Updater in proactive or opportunistic mode.

Example

You have a cluster of nodes, example-cluster, that can operate in one of two modes: active or standby. You set the mode individually for each VM in the cluster using metadata, for example: mode:active. You also configure how elaborate logging should be for each node, using a logging metadata key that can be set to basic or elaborate. The application on each node consumes the values from instance metadata.

The instance template defines mode:active and logging:basic metadata to be used as default for all instances. You have set logging:elaborate in a per-instance config for the node-12 VM in the cluster. Now, you want to switch node-12 to standby mode and switch logging to basic for this instance.

To switch the node-12 VM to standby and its logging to basic, patch the associated per-instance config by using the patchPerInstanceConfigs method:

POST https://compute.googleapis.com/compute/v1/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-cluster/patchPerInstanceConfigs

{
  "perInstanceConfigs": [
    {
      "name": "node-12",
      "preservedState" : {
        "metadata": {
          "mode": "standby",
          "logging": null
        }
      }
    }
  ]
}

The method does the following:

  1. Sets mode:standby metadata in the per-instance config associated with the VM, node-12, in the example-cluster MIG.
  2. Removes logging:elaborate metadata from the per-instance config because the supplied value is null.

The config update is not yet applied to the node-12 VM instance. The config update will be applied when you next recreate or update the instance or if you use proactive auto-updating.

To apply the per-instance config update to the node-12 VM instance, call the instanceGroupManagers.applyUpdatesToInstances method for the instance:

POST https://compute.googleapis.com/compute/v1/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-cluster/applyUpdatesToInstances

{
  "instances": ["/zones/us-east1-c/instances/node-12"],
  "minimalAction": "NONE"
}

The method applies the updated per-instance config to the node-12 VM:

  1. Sets mode:standby metadata, according to the per-instance config.
  2. Sets logging:basic metadata from the instance template because the value for the logging key is no longer defined by the per-instance config.
  3. The VM keeps running during the update because minimalAction is set to NONE, which allows the MIG to use the least disruptive action required for the update. An instance metadata update requires the REFRESH action, which does not disrupt a running instance.

Feedback

We want to learn about your use cases, challenges, and feedback about stateful MIGs. Please share your feedback with our team at mig-discuss@google.com.

What's next