How stateful MIGs work

A stateful managed instance group (stateful MIG) preserves the unique state of each virtual machine (VM) instance—including VM name, attached persistent disks, IP addresses, and/or metadata—on machine restart, recreation, auto-healing, or update.

This page describes how stateful MIGs work. See Configuring stateful MIGs to learn how to set up a stateful MIG.

Overview of how stateful MIGs work

A MIG is considered stateful if you have created a stateful configuration.

You create a stateful configuration by setting a non-empty stateful policy and/or one or more non-empty per-instance configurations:

The configuration is effective after you or the MIG applies it:

  • A MIG automatically applies your stateful policy configuration to new and existing instances.
  • When creating or updating per-instance configurations, you can choose whether to apply the new configuration manually or have it applied automatically.

After the stateful configuration (stateful policy and/or per-instance configurations) is applied, you can verify it by inspecting the preserved state of each managed instance.

Subsequent changes to your MIG's stateful configuration or size (for example, decreasing the MIG's size, or deleting or abandoning instances from the MIG) can affect the preserved states of the instances.

How stateful configuration is applied to managed instances

Your stateful configuration is effective after you or the MIG applies it. Applying stateful configuration to a MIG's instances depends on the configuration:

  • Stateful policy: The MIG automatically applies your stateful policy configuration to new and existing instances.
  • Per-instance configurations: When creating or updating per-instance configurations, you can choose whether to apply the new configuration manually or have it applied automatically.

Applying stateful configuration to managed instances.

How stateful policy updates are applied to instances

When you create or update a stateful policy, for example add or remove a stateful disk, the MIG applies your stateful policy configuration to all managed instances in the group automatically and asynchronously. A MIG also automatically applies your stateful policy configuration to new instances during their creation, for example, when a MIG's size is increased or when you create instances in the MIG manually.

After the configuration is applied, you can see the effect of the update in each managed instance's preserved state from policy.

Updates to a stateful policy do not disrupt running VMs.

When you update a stateful policy to add a stateful disk, the MIG updates each VM resource, changing the value of the disk's autoDelete flag (instances.disks[].autoDelete):

  • The MIG sets autoDelete to FALSE for disks that you configure as stateful. This prevents deletion of that disk on instance recreation by autohealing, updating, or manual recreation.
  • The MIG sets autoDelete to match your instance template configuration (instanceTemplates.disks[].autoDelete) for all disks that are to be stateless.

Changing the value of the autoDelete flag does not disrupt a running VM.

How per-instance configuration updates are applied to instances

When you create or update a per-instance configuration, you can choose whether to apply the new configuration manually or automatically. For more information, see Applying stateful configuration from per-instance configurations.

The following table shows the disruption levels that are required to apply different per-instance configuration updates to a VM:

Per-instance configuration update Disruption to VM required for applying
Configure a disk, defined by the instance template, to be stateful (added to the per-instance configuration) REFRESH
Configure a disk, defined by the instance template, to be stateless (removed from the per-instance configuration) REFRESH
Add a disk, not defined by the instance template, and attach it to the VM REFRESH
Remove a disk, not defined by the instance template, and detach it from the VM REFRESH
Add a metadata key-value pair REFRESH
Remove a metadata key-value pair REFRESH
Add an external boot disk, not created from the instance template, and attach it to the VM REPLACE
Remove an external boot disk, not created from the instance template, detach it from the VM, and create a boot disk from the instance template instead REPLACE
Set an internal IP address REPLACE
Remove an internal IP address REPLACE

When applying an updated per-instance configuration to the corresponding VM, the MIG performs the following actions depending on which stateful items are updated:

  • Adds (or removes) disks, IP addresses, or metadata to the preserved state from configuration in the corresponding managed instance.
  • Attaches (or detaches) disks that are not defined by the instance template to the VM.
  • Sets (or removes) metadata key-value pairs that are specific for the VM.
  • Assigns (or removes) specified IP addresses to the VM instance.

After a per-instance configuration is applied to a corresponding managed VM, you can see the effect of the update in the instance's preserved state from configuration.

Preserved state of a managed instance

When it is applied, the MIG translates your instance template and stateful configuration into a "preserved state" for each managed instance.

You can view the preserved state by inspecting a managed instance.

The MIG maintains these preserved states automatically, and the MIG automatically and asynchronously applies this state to each corresponding actual VM instance in the MIG.

Preserved state of managed VMs that are generated by applying stateful configuration.

The preserved state describes which individual items (persistent disks, IP addresses, metadata) are stateful for a given instance:

Preserved state generated from applied stateful configuration.

The preserved state generated based on a stateful policy is stored separately from the preserved state generated based on a per-instance configuration. The MIG combines both of them when recreating a VM, with the preserved state from a per-instance configuration taking priority.

Preserved state according to stateful policy

A stateful policy specifies items, present in all instances and defined in the MIG's instance template, to preserve individually for each VM instance in a MIG.

When applied, the MIG translates the stateful policy into instance-specific preserved states (managedInstances[].preservedStateFromPolicy). The MIG maintains these preserved states automatically.

The following example shows a MIG with two VM instances that use a stateful disk defined in a stateful policy that applies to every instance. There are no per-instance configurations in this example.

Preserved state generated from stateful policy only.

The preceding figure shows a MIG with two instances:

  • The instance template defines a boot disk with device name, boot-disk, and a disk with device name, data-disk, for all instances in the MIG.
  • The stateful policy declares data-disk as stateful. The boot disk remains stateless. Note that the disk with device name, data-disk, must be and is defined by the instance template.
  • After the configuration is applied, the MIG translates the stateful policy into instance-specific preserved states for each managed instance. The preserved states instruct the MIG to preserve the disk data-disk-1 for VM instance node-1 and the disk data-disk-2 for the instance node-2, because both of these disks have device name data-disk configured in the stateful policy.
  • This example has no per-instance configurations.

Preserved state according to per-instance configuration

A per-instance configuration specifies items that must be preserved for a particular VM. These items don't have to be defined in the MIG's instance template.

When applied, the MIG translates each per-instance configuration into a preserved state (preservedStateFromConfig) for the corresponding instance.

The following example shows a MIG with two VM instances for which stateful metadata and disks are defined in per-instance configurations (PICs) for every instance. There is no stateful policy in this example.

Preserved state generated from PICs only.

In the preceding figure:

  • The instance template defines a boot disk with device name boot-disk for all instances in the MIG. The boot disk is stateless for all VMs in the MIG.
  • Per-instance configurations define the states to preserve for two instances in the MIG: node-1 and node-2.
    • For the node-1 instance, the per-instance configuration defines a disk my-legacy-1 with device name legacy-disk and metadata node-id:xyz273.
    • For the node-2 instance, the per-instance configuration defines a disk my-logs-1 with device name logs-disk and metadata node-id:pqr851.
  • After the configuration is applied, the MIG automatically translates the per-instance configurations into preserved states for each managed instance. The preserved states instruct the MIG to attach and preserve the following:
    • Persistent disk my-legacy-1 and metadata node-id:xyz273 for VM node-1
    • Persistent disk my-logs-1 and metadata node-id:pqr851 for VM node-2
  • This example has no stateful policy.

Note that the disks and metadata in the preserved state from per-instance configurations are not defined by the instance template in this example; instead, they are defined by the per-instance configurations only. This is because the configuration that you specify in a per-instance configuration is specific for a particular VM, which means it does not have to be present in the instance template.

Per-instance configurations have priority over stateful policy and instance template

You can configure both a stateful policy and one or more per-instance configurations in one MIG. For example, in a stateful policy, you can define stateful disks that are present in all instances, and in per-instance configurations, you can define instance-specific metadata.

A managed instance's per-instance configuration takes priority over conflicting configuration in the instance template or in a stateful policy.

If you apply a per-instance configuration to add a disk or a network interface that is already defined in a stateful policy, the MIG stores the stateful configuration for that disk or network interface in the managed instance's preserved state from per-instance configuration (preservedStateFromConfig) and removes the conflicting entries from its preserved state from policy (preservedStateFromPolicy). The MIG must refresh the VM if the new preserved state is different from the previous one. The refresh could result in metadata change, external IP address change, or a disk swap to detach the disk from the last preserved state configuration and attach the disk specified in the new preserved state configuration.

In the following example, the per-instance configuration for VM instance node-1 redefines:

  • The preserved state for the disk with device name logs-disk, originally defined in the stateful policy
  • The value for metadata key logmonth, originally defined in the instance template.

Configuration from per-instance configurations takes priority over stateful policy and instance template.

In the preceding figure:

  • The instance template defines:
    • Three disks for all instances in the MIG, with device names boot-disk, data-disk, logs-disk.
    • Metadata common to all instances: logmonth:jan.
  • The stateful policy declares that disks with device names data-disk and logs-disk are stateful; the boot disk remains stateless.
  • A per-instance configuration for the instance node-1 redefines:
    • Stateful configuration for a disk with device name logs-disk: This instructs the MIG to attach the disk pd-logs-feb to node-1 under the logs-disk device name.
    • Metadata, defined in the instance template, with key value logmonth:jan: This instructs the MIG to set value logmonth:feb to node-1.
  • After you apply, the configuration, the MIG automatically translates the stateful policy and per-instance configuration into an instance-specific preserved state, stored in the managed instance.
    • The preserved state from policy instructs the MIG to preserve the disk data-disk-1 for the VM node-1. Note that the preserved state from policy does not include stateful configuration for the disk with device name logs-disk because this configuration is overridden by the configuration for logs-disk in the per-instance configuration.
    • The preserved state from configuration instructs the MIG to attach and preserve persistent disk logs-disk and to set and preserve metadata logmonth:feb for VM instance node-1. Note that the preserved state from configuration overrides the configuration for logs-disk from the stateful policy and overrides metadata logmonth:jan from the instance template.

How removing a resource from a stateful policy affects preserved state

If you remove a resource configuration from your stateful policy, the MIG automatically removes the corresponding preservedStateFromPolicy for all managed instances. The compute resources remain attached to the instances, but they are no longer stateful.

In the following example, removing a disk from the stateful policy leads to removal of that disk from preserved states from policy in all managed VMs. Those disks remain attached to their VMs, but they are no longer stateful and might be deleted and recreated on next VM recreation.

Removing a disk from a stateful policy.

If the same item, for example, a stateful persistent disk, is present both in the stateful policy and a per-instance configuration, and you remove its stateful configuration from the stateful policy only, the MIG doesn't remove it from the per-instance configuration. For the corresponding VM, the configured resource remains stateful.

In the following example, removing the disk from the stateful policy does not lead to removal of the disk from the per instance configuration. The disk remains stateful because it is still a part of preserved state from configuration.

Removing a disk from a stateful policy when a per-instance configuration also exists.

How removing items from per-instance configurations affects preserved state

If you remove the stateful configuration from a per-instance configuration, and apply the change, the MIG automatically removes the stateful configuration from the preserved state from configuration (preservedStateFromConfig) in the corresponding managed instance. The compute resources that are no longer part of any preserved state become stateless.

How removing stateful disks configuration from per-instance configurations affects preserved state

If you remove a stateful disk from a per-instance configuration and apply the change to the associated VM instance, the MIG does the following:

  • The disk configuration is removed from the instance's preserved state from configuration.
  • If a disk with the same device name is defined in the instance template but not configured in a stateful policy, then the disk stays attached to the given VM. However, the disk becomes stateless for the given VM and it might be recreated according to the instance template configuration on the next VM recreation, autohealing, or update event.
  • If a disk with the same device name is not defined in the instance template, then it is automatically detached from the VM immediately upon application of the updated per-instance configuration to the associated VM, regardless of its auto-delete configuration.
  • If a disk with the same device name is configured in a stateful policy, then its stateful policy configuration is translated into the preserved state from policy for the given managed instance, and the disk remains stateful.

In the following example, removing a blue and a green disk from node-1's per-instance configuration leads to removal of both disks from the node-1 managed instance's preserved state from configuration.

  • The blue disk remains attached to the node-1 VM instance, but it is now stateless and can be recreated on the next VM recreation according to the instance template configuration.
  • The green disk is detached from the node-1 VM instance because the instance template does not define a disk with the same device name.

Removing disks from a per-instance configuration.

How removing stateful metadata from per-instance configurations affects preserved state

Removing stateful metadata from a per-instance configuration and applying the change causes the MIG to immediately remove that stateful metadata from the corresponding managed instance's preserved state:

  • If you defined metadata with the same key in the instance template, the MIG immediately applies the value from the instance template to the instance.
  • If the metadata with the same key is not defined in the instance template, the MIG immediately removes the key value from the instance.

In the following example, removing mode:dev and id:xyz273 metadata from node-1's per-instance configuration leads to automatic removal of both key-value pairs from the node-1 managed instance's preserved state from configuration.

  • mode:dev is replaced by the instance template's mode:test in the VM.
  • id:xyz273 is removed from the VM immediately because the instance template does not have metadata with the same key id to replace it with.

Removing metadata from a per-instance configuration.

How removing stateful IP configuration from per-instance configurations affects preserved state

Removing the internal IP configuration from per-instance configuration makes the IP address for this VM stateless. No automated changes are performed on this VM, but the IP address can change after the VM is recreated, updated, or autohealed.

Fallback to stateful policy

If you remove the stateful configuration of a resource from a per-instance configuration, and you configured the same resource in the stateful policy, then the resource remains stateful according to the stateful policy.

The MIG automatically removes the item's stateful configuration from the preservedStateFromConfig and adds it to the preservedStateFromPolicy for the corresponding managed instance.

In the following example, removing a disk from node-1's per-instance configuration does not lead to removal of the disk from the stateful policy. The disk remains stateful according to the stateful policy:

  • The MIG automatically removes the disk from the preserveStateFromConfig for the node-1 managed instance because the disk is no longer part of its per-instance configuration.
  • The MIG automatically adds the disk to the preserveStateFromPolicy for the node-1 managed instance because the stateful policy configuration is still in place and is no longer in conflict with the node-1 per-instance configuration.

Removing a disk from a per-instance configuration but not from stateful policy.

Feedback

We want to learn about your use cases, challenges, and feedback about stateful MIGs. Please share your feedback with our team at mig-discuss@google.com.

What's next