How stateful MIGs work

A stateful managed instance group (stateful MIG) preserves the unique state of each virtual machine (VM) instance—including VM name, attached persistent disks, and/or metadata—on machine restart, recreation, auto-healing, or update.

This page describes how stateful MIGs work. See Configuring stateful MIGs to learn how to set up a stateful MIG.

Overview of how stateful MIGs work

A MIG is considered stateful if you have created a stateful configuration.

You create a stateful configuration by setting a non-empty stateful policy and/or one or more non-empty per-instance configs:

The configuration is effective after you or the MIG applies it:

  • A MIG automatically applies your stateful policy configuration to new and existing instances.
  • When creating or updating per-instance configs, you can choose whether to apply the new configuration manually or have it applied automatically.

After the stateful configuration (stateful policy and/or per-instance configs) is applied, you can verify it by inspecting the preserved state of each managed instance.

Subsequent changes to your MIG's stateful configuration or size (for example, decreasing the MIG's size, or deleting or abandoning instances from the MIG) can affect the preserved states of the instances.

How stateful configuration is applied to managed instances

Your stateful configuration is effective after you or the MIG applies it. Applying stateful configuration to a MIG's instances depends on the configuration:

  • Stateful policy: The MIG automatically applies your stateful policy configuration to new and existing instances.
  • Per-instance configs: When creating or updating per-instance configs, you can choose whether to apply the new configuration manually or have it applied automatically.

Applying stateful configuration to managed instances.

How stateful policy updates are applied to instances

When you create or update a stateful policy, for example add or remove a stateful disk, the MIG applies your stateful policy configuration to all managed instances in the group automatically and asynchronously. A MIG also automatically applies your stateful policy configuration to new instances during their creation, for example, when a MIG's size is increased or when you create instances in the MIG manually.

After the config is applied, you can see the effect of the update in each managed instance's preserved state from policy.

Updates to a stateful policy do not disrupt running VMs.

When you update a stateful policy to add a stateful disk, the MIG updates each VM resource, changing the value of the disk's autoDelete flag (instances.disks[].autoDelete):

  • The MIG sets autoDelete to FALSE for disks that you configure as stateful. This prevents deletion of that disk on instance recreation by autohealing, updating, or manual recreation.
  • The MIG sets autoDelete to match your instance template configuration (instanceTemplates.disks[].autoDelete) for all disks that are to be stateless.

Changing the value of the autoDelete flag does not disrupt a running VM.

How per-instance config updates are applied to instances

When you create or update a per-instance config, you can choose whether to apply the new configuration manually or have it applied automatically.

You can apply per-instance configs updates to the corresponding instances in the following ways:

  • You can manually apply the updates to specific instances to control the disruption level, the timing, and the sequence of the updates. Set the update type to opportunistic to prevent automatic updates and to only apply updates when a VM instance is manually recreated or updated.
  • You can manually create new instances in the MIG and specify their names and per-instance configs at creation time. The MIG applies per-instance configs immediately on VM creation.
  • When creating or updating a per-instance config using the gcloud command line, you can choose whether to apply the config to an associated managed instance at the same time or to postpone the application. In case of postponing, the update to the per-instance config won't be applied to the instance automatically. You must apply the update manually for it to take effect.

The following table shows the disruption levels that are required to apply different per-instance config updates to a VM:

Per-instance config update Disruption to VM required for applying
Configure a disk, defined by the instance template, to be stateful (added to the per-instance config) REFRESH
Configure a disk, defined by the instance template, to be stateless (removed from the per-instance config) REFRESH
Add a disk, not defined by the instance template, and attach it to the VM REFRESH
Remove a disk, not defined by the instance template, and detach it from the VM REFRESH
Add an external boot disk, not created from the instance template, and attach it to the VM REPLACE
Remove an external boot disk, not created from the instance template, detach it from the VM, and create a boot disk from the instance template instead REPLACE
Add a metadata key-value pair REFRESH
Remove a metadata key-value pair REFRESH

When applying an updated per-instance config to the corresponding VM, the MIG performs the following actions depending on which stateful items are updated:

  • Adds (or removes) disks or metadata to the preserved state from config in the corresponding managed instance.
  • Attaches (or detaches) disks that are not defined by the instance template to the VM.
  • Sets (or removes) metadata key-value pairs that are specific for the VM.

After a per-instance config is applied to a corresponding managed VM, you can see the effect of the update in the instance's preserved state from config.

Preserved state of a managed instance

When it is applied, the MIG translates your instance template and stateful configuration into a "preserved state" for each managed instance.

You can view the preserved state by inspecting a managed instance.

The MIG maintains these preserved states automatically, and the MIG automatically and asynchronously applies this state to each corresponding actual VM instance in the MIG.

Preserved state of managed VMs that are generated by applying stateful configuration.

The preserved state describes which individual items (persistent disks, metadata) are stateful for a given instance:

Preserved state generated from applied stateful config.

The preserved state generated based on a stateful policy is stored separately from the preserved state generated based on a per-instance config. The MIG combines both of them when recreating a VM, with the preserved state from a per-instance config taking priority.

Preserved state according to stateful policy

A stateful policy specifies items, present in all instances and defined in the MIG's instance template, to preserve individually for each VM instance in a MIG.

When applied, the MIG translates the stateful policy into instance-specific preserved states (managedInstances[].preservedStateFromPolicy). The MIG maintains these preserved states automatically.

The following example shows a MIG with two VM instances that use a stateful disk defined in a stateful policy that applies to every instance. There are no per-instance configs in this example.

Preserved state generated from stateful policy only.

The preceding figure shows a MIG with two instances:

  • The instance template defines a boot disk with device name, boot-disk, and a disk with device name, data-disk, for all instances in the MIG.
  • The stateful policy declares data-disk as stateful. The boot disk remains stateless. Note that the disk with device name, data-disk, must be and is defined by the instance template.
  • After the config is applied, the MIG translates the stateful policy into instance-specific preserved states for each managed instance. The preserved states instruct the MIG to preserve the disk data-disk-1 for VM instance node-1 and the disk data-disk-2 for the instance node-2, because both of these disks have device name data-disk configured in the stateful policy.
  • This example has no per-instance configs.

Preserved state according to per-instance config

A per-instance config specifies items that must be preserved for a particular VM. These items don't have to be defined in the MIG's instance template.

When applied, the MIG translates each per-instance config into a preserved state (preservedStateFromConfig) for the corresponding instance.

The following example shows a MIG with two VM instances for which stateful metadata and disks are defined in per-instance configs (PICs) for every instance. There is no stateful policy in this example.

Preserved state generated from PICs only.

In the preceding figure:

  • The instance template defines a boot disk with device name boot-disk for all instances in the MIG. The boot disk is stateless for all VMs in the MIG.
  • Per-instance configs define the states to preserve for two instances in the MIG: node-1 and node-2.
    • For the node-1 instance, the per-instance config defines a disk my-legacy-1 with device name legacy-disk and metadata node-id:xyz273.
    • For the node-2 instance, the per-instance config defines a disk my-logs-1 with device name logs-disk and metadata node-id:pqr851.
  • After the config is applied, the MIG automatically translates the per-instance configs into preserved states for each managed instance. The preserved states instruct the MIG to attach and preserve the following:
    • Persistent disk my-legacy-1 and metadata node-id:xyz273 for VM node-1
    • Persistent disk my-logs-1 and metadata node-id:pqr851 for VM node-2
  • This example has no stateful policy.

Note that the disks and metadata in the preserved state from per-instance configs are not defined by the instance template in this example; instead, they are defined by the per-instance configs only. This is because the configuration that you specify in a per-instance config is specific for a particular VM, which means it does not have to be present in the instance template.

Per-instance configs have priority over stateful policy and instance template

You can configure both a stateful policy and one or more per-instance configs in one MIG. For example, in a stateful policy, you can define stateful disks that are present in all instances, and in per-instance configs, you can define instance-specific metadata.

A managed instance's per-instance config takes priority over conflicting configuration in the instance template or in a stateful policy.

If you apply a per-instance config to add a disk that is already defined in a stateful policy, the MIG stores the stateful configuration for that disk in the managed instance's preserved state from config (preservedStateFromConfig) and removes the conflicting entries from its preserved state from policy (preservedStateFromPolicy). The MIG must refresh the VM if the new preserved state is different from the previous one. The refresh could result in metadata change, or a disk swap to detach the disk from the last preserved state configuration and attach the disk specified in the new preserved state configuration.

In the following example, the per-instance config for VM instance node-1 redefines:

  • The preserved state for the disk with device name logs-disk, originally defined in the stateful policy
  • The value for metadata key logmonth, originally defined in the instance template.

Configuration from per-instance configs takes priority over stateful policy and instance template.

In the preceding figure:

  • The instance template defines:
    • Three disks for all instances in the MIG, with device names boot-disk, data-disk, logs-disk.
    • Metadata common to all instances: logmonth:jan.
  • The stateful policy declares that disks with device names data-disk and logs-disk are stateful; the boot disk remains stateless.
  • A per-instance config for the instance node-1 redefines:
    • Stateful configuration for a disk with device name logs-disk: This instructs the MIG to attach the disk pd-logs-feb to node-1 under the logs-disk device name.
    • Metadata, defined in the instance template, with key value logmonth:jan: This instructs the MIG to set value logmonth:feb to node-1.
  • After you apply, the config, the MIG automatically translates the stateful policy and per-instance config into an instance-specific preserved state, stored in the managed instance.
    • The preserved state from policy instructs the MIG to preserve the disk data-disk-1 for the VM node-1. Note that the preserved state from policy does not include stateful configuration for the disk with device name logs-disk because this configuration is overridden by the configuration for logs-disk in the per-instance config.
    • The preserved state from config instructs the MIG to attach and preserve persistent disk logs-disk and to set and preserve metadata logmonth:feb for VM instance node-1. Note that the preserved state from config overrides the configuration for logs-disk from the stateful policy and overrides metadata logmonth:jan from the instance template.

How removing a resource from a stateful policy affects preserved state

If you remove a resource configuration from your stateful policy, the MIG automatically removes the corresponding preservedStateFromPolicy for all managed instances. The compute resources remain attached to the instances, but they are no longer stateful.

In the following example, removing a disk from the stateful policy leads to removal of that disk from preserved states from policy in all managed VMs. Those disks remain attached to their VMs, but they are no longer stateful and might be deleted and recreated on next VM recreation.

Removing a disk from a stateful policy.

If the same item, for example, a stateful persistent disk, is present both in the stateful policy and a per-instance config, and you remove its stateful configuration from the stateful policy only, the MIG doesn't remove it from the per-instance config. For the corresponding VM, the configured resource remains stateful.

In the following example, removing the disk from the stateful policy does not lead to removal of the disk from the per instance config. The disk remains stateful because it is still a part of preserved state from config.

Removing a disk from a stateful policy when a per-instance config also exists.

How removing items from per-instance configs affects preserved state

If you remove the stateful configuration from a per-instance config, and apply the change, the MIG automatically removes the stateful configuration from the preserved state from config (preservedStateFromConfig) in the corresponding managed instance. The compute resources that are no longer part of any preserved state become stateless.

How removing stateful disks configuration from per-instance configs affects preserved state

If you remove a stateful disk from a per-instance config and apply the change to the associated VM instance, the MIG does the following:

  • The disk configuration is removed from the instance's preserved state from config.
  • If a disk with the same device name is defined in the instance template but not configured in a stateful policy, then the disk stays attached to the given VM. However, the disk becomes stateless for the given VM and it might be recreated according to the instance template configuration on the next VM recreation, autohealing, or update event.
  • If a disk with the same device name is not defined in the instance template, then it is automatically detached from the VM immediately upon application of the updated per-instance config to the associated VM, regardless of its auto-delete configuration.
  • If a disk with the same device name is configured in a stateful policy, then its stateful policy configuration is translated into the preserved state from policy for the given managed instance, and the disk remains stateful.

In the following example, removing a blue and a green disk from node-1's per-instance config leads to removal of both disks from the node-1 managed instance's preserved state from config.

  • The blue disk remains attached to the node-1 VM instance, but it is now stateless and can be recreated on the next VM recreation according to the instance template configuration.
  • The green disk is detached from the node-1 VM instance because the instance template does not define a disk with the same device name.

Removing disks from a per-instance config.

How removing stateful metadata from per-instance configs affects preserved state

Removing stateful metadata from a per-instance config and applying the change causes the MIG to immediately remove that stateful metadata from the corresponding managed instance's preserved state:

  • If you defined metadata with the same key in the instance template, the MIG immediately applies the value from the instance template to the instance.
  • If the metadata with the same key is not defined in the instance template, the MIG immediately removes the key value from the instance.

In the following example, removing mode:dev and id:xyz273 metadata from node-1's per-instance config leads to automatic removal of both key-value pairs from the node-1 managed instance's preserved state from config.

  • mode:dev is replaced by the instance template's mode:test in the VM.
  • id:xyz273 is removed from the VM immediately because the instance template does not have metadata with the same key id to replace it with.

Removing metadata from a per-instance config.

Fallback to stateful policy

If you remove the stateful configuration of a resource from a per-instance config, and you configured the same resource in the stateful policy, then the resource remains stateful according to the stateful policy.

The MIG automatically removes the item's stateful configuration from the preservedStateFromConfig and adds it to the preservedStateFromPolicy for the corresponding managed instance.

In the following example, removing a disk from node-1's per-instance config does not lead to removal of the disk from the stateful policy. The disk remains stateful according to the stateful policy:

  • The MIG automatically removes the disk from the preserveStateFromConfig for the node-1 managed instance because the disk is no longer part of its per-instance config.
  • The MIG automatically adds the disk to the preserveStateFromPolicy for the node-1 managed instance because the stateful policy configuration is still in place and is no longer in conflict with the node-1 per-instance config.

Removing a disk from a per-instance config but not from stateful policy.

How operations affect preserved state

For information about how various configurations, MIG actions, or instance lifecycle events affect the preserved state of a managed instance, see the following sections:

Feedback

We want to learn about your use cases, challenges, and feedback about stateful MIGs. Please share your feedback with our team at mig-discuss@google.com.

What's next