Configuring stateful persistent disks in MIGs

You can configure any disk defined in the instance template to be stateful for all instances in a managed instance group (MIG) by adding that disk's device name to the MIG's stateful policy.

You can also configure stateful persistent disks individually for virtual machine (VM) instances in a MIG by setting per-instance configs; these disks don't need to be defined in the instance template.

Before you begin

Limitations

Stateful MIGs have the following limitations:

  • You cannot use autoscaling with stateful MIGs.
  • When you permanently delete an instance (either manually or by resizing down), the MIG does not preserve its stateful metadata.
  • For rolling updates, you must set the replacement method to RECREATE.
  • For stateful regional MIGs, you must disable proactive redistribution (set the redistribution type to NONE) to prevent deletion of stateful instances by automatic cross-zone redistribution.

When to use stateful persistent disks

Use stateful persistent disks to take advantage of VM

autohealing and auto-updating while still preserving the data on the disks. For more information, see use cases for stateful MIGs.

When you configure stateful disks, these disks are preserved through VM instance autohealing, updates, and recreation. But that also means that stateful disks cannot be recreated from the original image or updated to a new image.

As a best practice, we recommend keeping your boot disks stateless.

Keeping the boot disk stateless has the following benefits:

  • You can repair a boot disk that was corrupted by recreating it from its original image. Autohealing does such repairs automatically.
  • You can update a boot disk to the latest image with new versions and security patches. The MIG Updater can roll out the updates by recreating the boot disks.

For more information, see how autohealing and auto-updating handle preserved state.

Configuring stateful persistent disks for all VMs in a MIG

Configure any disk defined in an instance template to be stateful by adding that disk's device name to the MIG's stateful policy. The MIG treats disks with that device name as stateful for all existing and future VM instances.

Use the gcloud tool or the API to:

  • Configure stateful disks on MIG creation.
  • Set or update a stateful configuration for disks in an existing MIG.
  • Remove stateful disks from a MIG.

Configuring stateful disks on MIG creation

gcloud

To specify which disks from an instance template should be stateful on MIG creation, use the --stateful-disk flag with the gcloud beta compute instance-groups managed create command:

gcloud beta compute instance-groups managed create instance-group-name \
    --template instance-template \
    --size size \
    --stateful-disk device-name=device-name[,auto-delete=delete-rule]

Replace the following:

  • instance-group-name: The name of the managed instance group to create.
  • instance-template: The name of the instance template to use when creating instances.
  • size: The initial number of VMs you need in this group.
  • device-name: (Required.) The device name of a disk specified in the instance template.
  • delete-rule: (Optional.) A value that prescribes what should happen to a stateful disk when a VM is deleted. Available options are:

    • never: (Default.) Never delete the disk; instead, detach the disk when its VM is deleted.
    • on-permanent-instance-deletion: Delete the disk when its VM instance is permanently deleted from the instance group, for example, when the managed instance is deleted manually or when the group size is decreased.

    Regardless of the value of the delete rule, stateful disks are always preserved on VM autohealing, update, and recreation operations.

Example

You want to deploy a database with 12 shards, each with a stateless boot disk that contains the operating system and database binaries, and with a stateful data disk. Use the following steps:

  1. Create an instance template with a stateless boot disk based on the image img-example-db-v01, which has a pre-installed OS and database, and with a stateful data disk:

    gcloud beta compute instance-templates create example-database-template-v01 \
        --image img-example-db-v01 \
        --create-disk device-name=data-disk,mode=rw,image=empty10GBext4
    

    The --create-disk flag instructs the MIG to:

    1. Create a new 10 GB disk for each VM instance from an empty ext4 image, prepared beforehand.
    2. Attach the disk to its VM in read/write mode using device name data-disk.
  2. Create a MIG from the instance template and define the data disk as stateful:

    gcloud beta compute instance-groups managed create example-database-group \
      --template example-database-template-v01 \
      --base-instance-name shard \
      --size 12 \
      --stateful-disk device-name=data-disk,auto-delete=on-permanent-instance-deletion
    

    The device name data-disk is taken from the instance template. The data disk is configured to be deleted together with the VM instance when the VM is permanently deleted (either due to manual instance deletion or due to manual decrease of the group size). The data disk is preserved on autohealing, updates, and VM recreation.

  3. Verify that the data disk is configured in the stateful policy:

    gcloud beta compute instance-groups managed describe example-database-group
    
    
    baseInstanceName: shard
    ...
    name: example-database-group
    ...
    statefulPolicy:
      preservedState:
        disks:
          data-disk:
            autoDelete: ON_PERMANENT_INSTANCE_DELETION
    ...
    

    You can see that the stateful policy declares disks with device name data-disk as stateful, with a rule to delete such disks on permanent VM deletion.

API

To specify which disks from the instance template should be stateful on MIG creation, include them in the statefulPolicy field in the request body of the instanceGroupManagers.insert method:

POST https://compute.googleapis.com/compute/beta/projects/project/zones/zone/instanceGroupManagers

{
  "name": "name",
  "versions": [
    {
      "instanceTemplate": "global/instanceTemplates/template"
    }
  ],
  "targetSize": size,
  "statefulPolicy": {
    "preservedState": {
      "disks": {
        "device-name": {"autoDelete": "delete-rule" },
        "device-name": {"autoDelete": "delete-rule" }
      }
    }
  }
}

Replace the following:

  • project: The project ID for the request.
  • zone: The zone where the MIG is located (applies to a zonal MIG).
    • For a regional MIG, replace zones/zone with regions/region and specify the region of the MIG.
  • name: The name of the MIG to create.
  • template: The name of the instance template to use when creating instances.
  • size: The initial number of instances you need in this group.
  • device-name: (Required.) The device name of a disk specified in the instance template.
  • delete-rule: (Optional.) A value that prescribes what should happen to the stateful disk when the VM instance is deleted. The available options are:

    • never: (Default.) Never delete the disk; detach the disk when the VM is deleted.
    • on_permanent_instance_deletion: Delete the stateful disk when its VM is permanently deleted from the instance group, for example, when the managed instance is deleted manually or when the group size is decreased.

Example

You want to deploy a database with 12 shards, each with a stateless boot disk that contains the operating system and database binaries, and with a stateful data disk. Use the following steps.

  1. Create an instance template with a stateless boot disk based on the image img-example-db-v01, with pre-installed OS and database, and with a stateful data disk, using the instanceTemplates.insert method:

    POST https://compute.googleapis.com/compute/v1/projects/example-project/global/instanceTemplates
    
    {
      "name": "example-database-template-v01",
      "properties": {
        "machineType":"n1-standard-1",
        "disks": [
          {
            "boot": true,
            "deviceName": "boot-disk",
            "initializeParams": {
              "sourceImage": "projects/example-project/global/images/mg-example-db-v01"
            }
          },
          {
            "deviceName": "data-disk",
            "mode": "READ_WRITE",
            "initializeParams": {
              "sourceImage": "projects/example-project/global/images/empty10GBext4"
            }
          }
        ],
        "networkInterfaces": [
          {
            "network": "global/networks/default"
          }
        ]
      }
    }
    

    The data disk in the instance template has device name data-disk and is configured to be created from an empty ext4 image, prepared beforehand, and to be attached in read/write mode.

  2. Create a MIG from the instance template and define the data disk as stateful by using the instanceGroupManagers.insert method:

    POST https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers
    
    {
      "name": "example-database-group",
      "baseInstanceName": "shard",
      "versions": [
        {
          "instanceTemplate": "global/instanceTemplates/example-database-template-v01"
        }
      ],
      "targetSize": 12,
      "statefulPolicy": {
        "preservedState": {
          "disks": {
            "data-disk": {"autoDelete": "ON_PERMANENT_INSTANCE_DELETION" }
          }
        }
      }
    }
    

    The MIG creates 12 instances, each with a disk with the following properties:

    • A device name, data-disk, taken from the instance template.
    • A delete rule to delete the data disk when the VM is deleted (either due to manual instance deletion or due to manual decrease of the group size).
    • An entry in the preserved state from policy (preservedStateFromPolicy) of each managed instance so that the data disk is preserved on autohealing, updates, and instance recreation.
  3. Using the instanceGroupManagers.get method to verify that the data disk is configured in the stateful policy of the new instanceGroupManagers resource:

    GET https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-database-group
    
    
    {
      "name": "example-database-group",
      "baseInstanceName": "shard",
      ...
      "statefulPolicy": {
        "preservedState": {
          "disks": {
            "data-disk": {"autoDelete": "ON_PERMANENT_INSTANCE_DELETION" }
          }
        }
      }
      ...
    }
    

    You can see that stateful policy declares disks with device name data-disk as stateful with the rule to delete such disks on permanent instance deletion.

Setting and updating stateful configuration for disks in an existing MIG

If you run a stateful application on a stateless MIG (a MIG without any stateful configuration), you can configure existing disks that are defined in the instance template to be stateful for all instances in this MIG. This lets you preserve the disks on instance recreation, autohealing, and update operations, and optionally on deletion operations.

You can do the following operations:

  • Add disks that are defined in the instance template to the stateful policy of an existing MIG to declare them as stateful. This marks disks with the given device name as stateful for all existing and future instances in the MIG.
  • Update the stateful policy to change the stateful configuration for disks.

The MIG applies the updated configuration in the stateful policy automatically and asynchronously to all instances. Updates to disk configurations in a stateful policy do not disrupt running VMs. For more information, see applying stateful policy updates.

For a regional MIG, you must disable proactive cross zone instance redistribution before you can configure stateful disks. For more information, see how regional groups handle preserved state.

gcloud

To specify which disks from the instance template should be stateful or to update the stateful disk configuration for an existing MIG, use one or multiple --stateful-disk flags with the gcloud beta compute instance-groups managed update command:

gcloud beta compute instance-groups managed update name \
  --stateful-disk device-name=device-name[,auto-delete=delete-rule]

Replace the following:

  • name: (Required.) The name of the managed instance group to update.
  • device-name: (Required.) The device name of a disk that is specified in the instance template.
  • delete-rule: (Optional.) A value that prescribes what should happen to the stateful disk when a VM instance is deleted. The available options are:

    • never: (Default.) Never delete the disk, detach the disk when its instance is deleted.
    • on-permanent-instance-deletion: Delete the stateful disk when its instance is permanently deleted from the instance group, for example, when the managed instance is deleted manually or when the group size is decreased.

If a specified device name is already configured in the stateful policy, the command updates the configuration.

Example

You run a database with multiple shards on a MIG named example-database- group. Each VM in the MIG stores a shard on an additional disk with device name data-disk, which is defined by the instance template. The MIG has no stateful configuration, and you want to preserve the data disks on instance recreation, autohealing, and updates. You also want to protect the data disks from deletion when a VM is deleted.

  1. Update the MIG to define the data disk as stateful by using the following command:

    gcloud beta compute instance-groups managed update example-database-group \
      --stateful-disk device-name=data-disk,auto-delete=never
    

    As a result, the MIG applies the stateful policy configuration updates automatically and asynchronously to the data disks for all instances. The data disks are now preserved on autohealing, updates, and instance recreation, and the data disks are detached on instance deletion because the auto-delete rule is set to never.

  2. Verify that the data disk is configured in the stateful policy by running the gcloud beta compute instance-groups managed describe example-database-group command.

API

To specify which disks from the instance template should be stateful or to update the stateful disk configuration for an existing MIG, configure the disks in the MIG's stateful policy using the instanceGroupManagers.patch method:

PATCH https://compute.googleapis.com/compute/beta/projects/project/zones/zone/instanceGroupManagers/name

{
  "statefulPolicy": {
    "preservedState": {
      "disks": {
        "device-name": {"autoDelete": "delete-rule" },
        "device-name": {"autoDelete": "delete-rule" }
      }
    }
  }
}

Replace the following:

  • project: The project ID for the request.
  • zone: The zone where the MIG is located (applies to a zonal MIG).
    • For a regional MIG, replace zones/zone with regions/region and specify the region of the MIG.
  • name: The name of the MIG to update.
  • device-name: (Required.) The device name of a disk, specified in the instance template, for which you would like to update stateful configuration.
  • delete-rule: (Optional.) A value that prescribes what should happen to the stateful disk when a VM instance is deleted. The available options are:

    • never: (Default.) Never delete the disk, detach the disk when its instance is deleted.
    • on-permanent-instance-deletion: Delete the stateful disk when its instance is permanently deleted from the instance group, for example, when the instance is deleted manually or when the group size is decreased.

If the specified device name is already configured in the stateful policy, the patch method updates its configuration.

Example

You run a database with multiple shards on a MIG named example-database- group. Each VM in the MIG stores a shard on an additional disk with device name data-disk, which is defined by the instance template. The MIG has no stateful configuration, and you want to preserve the data disks on instance recreation, autohealing, and updates. You also want to protect the data disks from deletion when a VM is deleted.

  1. Patch the MIG to define the data disk as stateful:

    PATCH https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-database-group
    
    {
      "statefulPolicy": {
        "preservedState": {
          "disks": {
            "data-disk": {"autoDelete": "NEVER" }
          }
        }
      }
    }
    

    The MIG applies this stateful configuration automatically and asynchronously to the data disks for all instances. The data disks will be preserved on autohealing, updates, and instance recreation. The data disks will be detached on instance deletion because the autoDelete rule is set to NEVER.

  2. Verify that the data disk is configured in the stateful policy by viewing the instanceGroupManagers resource, returned by the instanceGroupManagers.get method.

Declaring previously stateful persistent disks as stateless

You might need to configure a stateful disk to be treated as stateless. For example:

  • If you rearchitect your app to move the state off the disk.
  • If you configured the disk to be stateful by mistake and would like to revert it.

To declare all disks with a given device name as stateless, remove the disk's configuration from the stateful policy using the gcloud tool or the API.

The MIG applies the change to the stateful policy automatically and asynchronously to all instances. Updates to disk configuration in a stateful policy do not disrupt running VM instances.

For more information, see:

gcloud

To specify which disks from a MIG's stateful policy to make stateless, use the --remove-stateful-disks flag with the gcloud beta compute instance-groups managed update command:

gcloud beta compute instance-groups managed update name \
  --remove-stateful-disks device-name[,device-name,...]

Replace the following:

  • name: The name of the MIG to update.
  • device-name: The device name of a disk to remove from the stateful policy and to treat as stateless. You can provide one or multiple device names in the list.

Example

You run a legacy application with multiple nodes on a MIG named example-legacy-group. Each VM in the MIG stores application data on a boot disk with device name boot-disk, which you configured as stateful in the MIG's stateful policy. You have moved application data to an additional disk and now want to make the boot disk stateless to make it easy to update to new images.

To remove the stateful configuration of the boot disk, update the managed instance group:

gcloud beta compute instance-groups managed update example-legacy-group \
  --remove-stateful-disks boot-disk

The MIG removes the stateful configuration for the device name boot-disk automatically and asynchronously for the boot disks of all instances in the group. The boot disks remain attached to the instances but are no longer stateful. When you recreate or update the instances, or when instances are autohealed, the MIG recreates the boot disks from the image specified in the instance template.

API

To specify which disks from a MIG's stateful policy to make stateless, remove each disk's configuration from the MIG's stateful policy using the instanceGroupManagers.patch method:

PATCH https://compute.googleapis.com/compute/beta/projects/project/zones/zone/instanceGroupManagers/name

{
  "statefulPolicy": {
    "preservedState": {
      "disks": {
        "device-name": null,
        ...
      }
    }
  }
}

Replace the following:

  • project: The project ID for the request.
  • zone: The zone where the MIG is located (applies to a zonal MIG).
    • For a regional MIG, replace zones/zone with regions/region and specify the region of the MIG.
  • name: The name of the MIG to update.
  • device-name: The device name of a disk that you want to remove from the stateful policy. Providing a null value leads to removal of the stateful configuration for that disk. You can provide one or multiple device names to remove.

Example

You run a legacy application with multiple nodes on a MIG named example-legacy-group. Each VM in the MIG stores application data on a boot disk with device name boot-disk, which you configured as stateful in the MIG's stateful policy. You have moved application data to an additional disk and now want to make the boot disk stateless to make it easy to update to new images.

To remove the stateful configuration of the boot disk, patch the managed instance group:

PATCH https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-legacy-group

{
  "statefulPolicy": {
    "preservedState": {
      "disks": {
        "boot-disk": null
      }
    }
  }
}

The MIG removes the stateful configuration for the device name boot-disk automatically and asynchronously for the boot disks of all instances in the group. The boot disks remain attached to the instances but are no longer stateful. When you recreate or update the instances, or when instances are autohealed, the MIG recreates the boot disks from the image specified in the instance template.

Removing stateful persistent disks from a MIG

You might need to completely remove a stateful disk from instances in a MIG, for example, if you re-architected your application and moved the state out of that disk.

MIGs do not allow removing stateful disks, so you must do the following steps:

  1. Remove the stateful configuration of the disk from the stateful policy. This makes disks with the given device name stateless.
  2. Detach the disks from the VMs if you still want to keep them.
  3. Roll out a new instance template that no longer defines the disk with the given device name.

For more information, see how removing a disk from stateful policy affects preserved state.

Configuring stateful persistent disks individually for a VM in a MIG

Configure stateful persistent disks for a specific VM in a MIG by adding the disk's device name to that VM's per-instance config.

Configuring stateful persistent disks individually for specific VMs in a MIG is useful if you need to:

  • Migrate existing workloads (bring existing disks) from standalone VMs to stateful MIGs to benefit from autohealing and easy updates.
  • Restore backups of disks, configured individually for VMs.
  • Attach stateful additional disks to a specific VM temporarily for testing, debugging, or copying data.

Adding existing stateful disks to new VMs in a MIG

You can add existing stateful disks to new instances that you manually create in a MIG. This is useful for migrating a stateful application from existing standalone VMs to a stateful MIG, for example:

  1. Create an instance template with common configuration for all VM instances.
  2. Detach the data disks from the standalone instances and delete these instances. You can also detach boot disks if they contain state that should be preserved.
  3. Create an empty MIG using the instance template created earlier.
  4. Create instances in the MIG with the appropriate names and associated disks from the previous step. The MIG responds to your request with the following actions:

    1. Creates a VM from the instance template using the provided instance name.
      • A regional MIG creates the VM in the same zone where the disk is located. If the disk is regional, the regional MIG creates the VM in any of the disk's replica zones.
    2. Creates a per-instance config with the provided stateful configuration for the disks.
    3. Attaches the disks to the new instance.

Add existing stateful disks when manually creating specific instances in a MIG using the gcloud tool or API.

gcloud

To create a VM with a custom name and attach one or more existing stateful disks to that VM, use the gcloud beta compute instance-groups managed create-instance command with one or multiple --stateful-disk flags.

gcloud beta compute instance-groups managed create-instance name \
  --instance instance-name \
  [--zone zone | --region region] \
  --stateful-disk device-name=device-name,source=disk[,mode=mode][,auto-delete=delete-rule]

Replace the following:

  • name: (Required.) The name of the MIG in which you need to create an instance.
  • instance-name: (Required.) The name of the new instance to create.
  • zone: The zone where the MIG is located (applies to a zonal MIG).
  • region: The region where the MIG is located (applies to a regional MIG).
  • device-name: (Required.) The device name to use when attaching the disk.
  • disk: (Required.) The URI of an existing persistent disk to attach under the specified device-name in the format projects/project-id/zones/zone/disks/disk-name for a zonal disk and projects/project-id/regions/region/disks/disk-name for a regional disk.
  • mode: (Optional.) Specifies the mode of the disk. Supported options are:
    • ro: Read-only.
    • rw: (Default.) Read/write.
  • delete-rule: (Optional.) A value that prescribes what should happen to a stateful disk when a VM instance is deleted. The available options are:

    • never: (Default.) Never delete the disk; instead, detach the disk when its instance is deleted.
    • on-permanent-instance-deletion: Delete the stateful disk when its instance is permanently deleted from the instance group, for example, when the instance is deleted manually or when the group size is decreased.

    Regardless of the value of the delete rule, stateful disks are always preserved on instance autohealing, update, and recreation operations.

Example

You want to have autohealing for a database server that is currently running on a standalone VM named db-instance and that currently stores data on a disk named db-data-disk-1.

Create a stateful MIG with autohealing, create a similar VM inside the MIG, and attach the existing data disk db-data-disk-1 to the new instance as a stateful disk:

  1. Stop the VM, db-instance, during a maintenance window.
  2. Create an instance template named db-template using the db-instance configuration.
  3. Detach db-data-disk-1 from db-instance and delete db-instance.
  4. Create an empty MIG, example-database-mig, from db-template, and configure autohealing.
  5. Create a managed instance with the original db-instance name and attach the db-data-disk-1 as a stateful disk:

    gcloud beta compute instance-groups managed create-instance example-database-mig \
      --instance db-instance \
      --zone us-east1-c \
      --stateful-disk device-name=data-disk,source=projects/example-project/zones/us-east1-c/disks/db-data-disk-1,auto-delete=never
    

    The command creates an instance, db-instance, in the MIG, creates a corresponding per-instance config with db-data-disk-1 stateful disk, and attaches the disk to the new VM, using data-disk as the device name.

API

To create one or multiple instances in a MIG, set custom VM names, and attach one or multiple existing stateful disks to these instances, use the instanceGroupManagers.createInstances method.

POST https://compute.googleapis.com/compute/beta/projects/project-id/zones/zone/instanceGroupManagers/name/createInstances

{
  "instances": [
    {
      "name": "instance-name",
      "preservedState" : {
        "disks": {
          "device-name" : {
            "source": "disk",
            "mode": "mode",
            "autoDelete": "delete-rule"
          },
          ...
        }
      }
    },
    ...
  ]
}

Replace the following:

  • project-id: Project ID for the request.
  • zone: The zone where the MIG is located (applies to a zonal MIG).
    • For a regional MIG, replace zones/zone with regions/region and specify the region of the MIG.
  • name: The name of the MIG in which to create an instance.
  • instance-name: (Required.) The name of the instance to create.
  • device-name: (Required.) The device name to use when attaching the disk.
  • disk: (Required.) URI of an existing persistent disk to attach under the specified device-name in the format projects/project-id/zones/zone/disks/disk-name for a zonal disk or projects/project-id/regions/region/disks/disk-name for a regional disk.
  • mode: (Optional.) Specifies the mode of the disk. Supported options are:
    • READ_ONLY: Read-only.
    • READ_WRITE: (Default.) Read/write.
  • delete-rule: (Optional.) A value that prescribes what should happen to a stateful disk when a VM is deleted. The available options are as follows:

    • never: (Default.) Never delete the disk; instead, detach the disk when its VM is deleted.
    • on-permanent-instance-deletion: Delete the stateful disk when its instance is permanently deleted from the instance group, for example, when the instance is deleted manually or when the group size is decreased.

    Regardless of the value of the delete rule, stateful disks are always preserved on instance autohealing, update, and recreation operations.

Example

You want to have autohealing for a database server that is currently running on a standalone VM named db-instance and that currently stores data on a disk named db-data-disk-1.

Create a stateful MIG with autohealing, create a similar instance inside the MIG, and attach the existing data disk db-data-disk-1 to the new VM as a stateful disk:

  1. Stop the VM, db-instance, during a maintenance window.
  2. Create an instance template named db-template using the db-instance configuration.
  3. Detach db-data-disk-1 from db-instance, and delete db-instance.
  4. Create an empty MIG, example-database-mig, from db-template, and configure autohealing.
  5. Create an instance with the original db-instance name, and attach the db-data-disk-1 as a stateful disk:

    POST https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-database-mig/createInstances
    
    {
      "instances": [
        {
          "name": "db-instance",
          "preservedState" : {
            "disks": {
              "data-disk" : {
                "source": "projects/example-project/zones/us-east1-c/disks/db-data-disk-1",
                "mode": "READ_WRITE",
                "autoDelete": "never"
              }
            }
          }
        }
      ]
    }
    

    The method creates an instance, db-instance, in the MIG, creates a corresponding per-instance config with the db-data-disk-1 stateful disk, and attaches the disk to the new instance, using data-disk as the device name.

Adding, declaring, and replacing stateful disks individually for VMs in a MIG

Configure stateful disks individually for a managed instance by adding or updating a stateful disk configuration in the associated per-instance config, for example:

  • Add a stateful disk from outside of a MIG to a VM in that MIG.
  • Declare a previously stateless disk as stateful for a VM in a MIG.
  • Replace a stateful disk with a different disk for a VM in a MIG.

Adding a stateful disk from outside of a MIG to a VM in that MIG. You can attach any disk from outside of a MIG to a managed instance by adding stateful configuration for the disk to the associated per-instance config. After you apply the config, the MIG automatically attaches the disk to the instance and treats it as stateful.

Declaring a previously stateless persistent disk as stateful. You can declare a previously stateless disk, currently attached to a VM, as stateful by adding stateful configuration for this disk, including its device name and URI, to the associated per-instance config. After you apply the config, the MIG starts preserving the disk as stateful.

Replacing a stateful disk with a different disk. Replacing one stateful disk with another stateful disk can be useful, for example, if you need access to a recovered backup. You can swap one stateful disk for another by updating the disk's URI while keeping the same device name in the per-instance config. After you apply the updated per-instance config, the MIG detaches the old disk and attaches the new one using the same device name. When applying the update, choose whether to keep the instance running, restart, or recreate it. Swapping a boot disk requires at least a VM restart.

gcloud

To configure stateful disks individually for a VM in a MIG, add or update stateful disk configuration in the associated per-instance config.

If a per-instance config does not yet exist for the instance, use the gcloud beta compute instance-groups managed instance-configs create command with one or multiple --stateful-disk flags:

gcloud beta compute instance-groups managed instance-configs create name \
--instance instance-name \
--stateful-disk device-name=device-name[,source=disk][,mode=mode][,auto-delete=delete-rule] \
[--no-update-instance | --update-instance] \
[--instance-update-minimal-action minimal-action]

If a per-instance config already exists for the instance, use the gcloud beta compute instance-groups managed instance-configs update command with one or multiple --stateful-disk flags.

The --update-instance flag (default) applies the changes immediately to the instance. If you use --no-update-instance, the changes remain unapplied and are applied when you next recreate or update the instance.

gcloud beta compute instance-groups managed instance-configs update name \
--instance instance-name \
--stateful-disk device-name=device-name[,source=disk][,mode=mode][,auto-delete=delete-rule] \
[--no-update-instance | --update-instance] \
[--instance-update-minimal-action minimal-action]

Replace the following:

  • name: (Required.) The name of the managed instance group.
  • instance-name: (Required.) The name of the VM instance for which to configure stateful disks.
  • device-name: (Required.) The device name used for attaching the disk.
  • disk: (Optional.) URI of an existing persistent disk to attach under the specified device-name, in the format projects/project-id/zones/zone/disks/disk-name for a zonal disk and projects/project-id/regions/region/disks/disk-name for a regional disk. If omitted, the stateful configuration is applied to a disk currently attached under the given device-name.
  • mode: (Optional.) Specifies the mode of the disk. You can only specify mode if you also specify source. Supported options are:
    • ro: Read-only.
    • rw: (Default.) Read/write.
  • delete-rule: (Optional.) A value that prescribes what should happen to a stateful disk when a VM is deleted. The available options are as follows:

    • never: (Default.) Never delete the disk; instead, detach the disk when its instance is deleted.
    • on-permanent-instance-deletion: Delete the stateful disk when its instance is permanently deleted from the instance group, for example, when the instance is deleted manually or when the group size is decreased.

    If omitted, the default value is used for new per-instance configs and current configuration is preserved for existing per-instance configs. Regardless of the value of the delete-rule, stateful disks are always preserved on instance autohealing, update, and recreation operations.

  • minimal-action: Perform at least the specified action when applying the per-instance config update to the instance. Must be used together with the --update-instance flag. The value must be one of:

    • none: No action.
    • refresh: Apply updates that are possible to apply without stopping the VM.
    • restart: Stop the VM and then start it again.
    • replace: Recreate the VM.

    If omitted, the least disruptive action required by the update is used.

Example

The data on a currently attached stateful disk, data-disk-1, got corrupted, and you want restore it from the latest backup. You created a disk, data-disk-2, from a snapshot to replace the corrupted disk in instance, db-instance-1, managed by a stateful MIG, example-database-mig. The original dis,k data-disk-1, is attached under the data-disk device name with an auto-delete rule to never delete the disk.

To replace data-disk-1 with data-disk-2, run the following command:

gcloud beta compute instance-groups managed instance-configs update example-database-mig \
  --instance db-instance-1 \
  --stateful-disk device-name=data-disk,source=projects/example-project/zones/us-east1-c/disks/data-disk-2 \
  --update-instance \
  --instance-update-minimal-action restart

The command does the following:

  1. Updates the per-instance config for db-instance-1:
    1. Updates the source for the disk with device name data-disk from data-disk-1 (last configuration) to data-disk-2 (new configuration).
    2. Keeps the auto-delete rule to never delete the disk because the auto-delete parameter is omitted in the --stateful-disk flag and, by default, the delete rule is never.
  2. Applies the per-instance config update to the db-instance-1 VM immediately because the --update-instance flag is included. The MIG detaches data-disk-1 and attaches data-disk-2 under the same device name, data-disk.
  3. Because the minimal action is set to restart, the MIG restarts the db-instance-1 instance to update the VM, which helps the database application to start using the new disk.

API

To configure stateful disks individually for VMs in a MIG, add or update the stateful disk configuration in the associated per-instance configs.

If per-instance configs do not yet exist for the given VMs, use the instanceGroupManagers.updatePerInstanceConfigs method with stateful configuration for one or multiple disks:

POST https://compute.googleapis.com/compute/beta/projects/project-id/zones/zone/instanceGroupManagers/name/updatePerInstanceConfigs

{
  "perInstanceConfigs": [
    {
      "name": "instance-name",
      "preservedState" : {
        "disks": {
          "device-name" : {
            "source": "disk",
            "mode": "mode",
            "autoDelete": "delete-rule"
          },
          ...
        }
      },
      "fingerprint: "fingerprint"
    },
    ...
  ]
}

If per-instance configs already exist for the given VMs, use the instanceGroupManagers.patchPerInstanceConfigs method with stateful configuration for one or multiple disks:

POST https://compute.googleapis.com/compute/beta/projects/project-id/zones/zone/instanceGroupManagers/name/patchPerInstanceConfigs

{
  "perInstanceConfigs": [
    {
      "name": "instance-name",
      "preservedState" : {
        "disks": {
          "device-name" : {
            "source": "disk",
            "mode": "mode",
            "autoDelete": "delete-rule"
          },
          ...
        }
      },
      "fingerprint: "fingerprint"
    },
    ...
  ]
}

Replace the following:

  • project-id: The project ID for the request.
  • zone: The zone where the MIG is located (applies to a zonal MIG).
    • For a regional MIG, replace zones/zone with regions/region and specify the region of the MIG.
  • name: The name of the MIG.
  • instance-name: (Required.) The name of the VM, for which to configure stateful disks.
  • device-name: (Required.) The device name used for attaching the disk.
  • disk: (Optional.) The URI of an existing persistent disk to attach under the specified device-name in the format projects/project-id/zones/zone/disks/disk-name for a zonal disk and projects/project-id/regions/region/disks/disk-name for a regional disk. If omitted, the stateful configuration is applied to a disk currently attached under the given device name.
  • mode: (Optional.) Specifies the mode of the disk. mode can only be specified if source is given. Supported options are:
    • READ_ONLY: Read-only.
    • READ_WRITE: (Default.) Read/write.
  • delete-rule: (Optional.) A value that prescribes what should happen to a stateful disk when a VM is deleted. The available options are as follows:

    • never: (Default.) Never delete the disk; instead, detach the disk when its instance is deleted.
    • on-permanent-instance-deletion: Delete the stateful disk when its instance is permanently deleted from the instance group, for example, when the instance is deleted manually or when the group size is decreased.

    If omitted, the default value is used for new per-instance configs and current configuration is preserved for existing per-instance configs. Regardless of the value of the delete rule, stateful disks are always preserved on instance autohealing, update, and recreation operations.

  • fingerprint: (Optional). The fingerprint for the given config if it already exists. Used for optimistic locking. To see the latest fingerprint, make a get() request to retrieve the resource.

The updatePerInstanceConfigs and patchPerInstanceConfigs methods update the specified per-instance configs but do not apply the config updates to the associated managed VMs. The changes are applied to an instance when the MIG is instructed to recreate or update the instance. Apply the update manually to apply the changes to an instance.

Example

The data on a currently attached stateful disk, data-disk-1, got corrupted, and you want restore it from the latest backup. You created a disk, data-disk-2, from a snapshot to replace the corrupted disk in instance, db-instance-1, managed by a stateful MIG, example-database-mig. The original dis,k data-disk-1, is attached under the data-disk device name with an auto-delete rule to never delete the disk.

To update the per-instance config for db-instance-1 with the new disk, call the patchPerInstanceConfigs method:

POST https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-database-mig/patchPerInstanceConfigs

{
  "perInstanceConfigs": [
    {
      "name": "db-instance-1",
      "preservedState" : {
        "disks": {
          "data-disk" : {
            "source": "projects/example-project/zones/us-east1-c/disks/data-disk-2"
          }
        }
      }
    }
  ]
}

The method patches the per-instance config for db-instance-1:

  1. Updates the source for a disk with device name data-disk from data-disk-1 (last configuration) to data-disk-2 (new configuration).
  2. Keeps mode and autoDelete parameters unchanged because the parameters are omitted in the request.

The config update is not yet applied to the db-instance-1 VM. The MIG applies the config update when you recreate or update the instance.

To apply the per-instance config update to the db-instance-1 VM, call the instanceGroupManagers.applyUpdatesToInstances method for the instance:

POST https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-database-mig/applyUpdatesToInstances
{
  "instances": ["/zones/us-east1-c/instances/db-instance-1"],
  "minimalAction": "RESTART"
}

The method updates the managed instance's preserved state, detaching data-disk-1 and attaching data-disk-2 under the same device name data-disk. Because the minimalAction is set to RESTART, the method restarts the db-instance-1 VM, which lets the database application start using the new disk.

Detaching a stateful disk or declaring it stateless for an individual VM

You might need to detach a stateful disk or configure it to be treated as stateless for an individual VM. For example:

  • If you rearchitect your app to move the state off the disk.
  • If you configure the disk to be stateful by mistake and would like to revert it.

Detach a stateful disk or make it stateless for an individual VM by removing the disk's stateful configuration from the associated per-instance config or deleting the entire per-instance config. When you apply the change:

  • If the disk is not defined in the instance template, the MIG detaches the disk .
    • The MIG does not delete the disk when you delete its configuration from the per-instance config, regardless of the auto-delete rule in the configuration.
  • If the disk is defined by the instance template, the MIG treats the disk as stateless, which means that the MIG recreates the disk from its source in the instance template on subsequent instance recreation, update, or autohealing events.

Removing a disk configuration from a per-instance config does not restart a running VM instance, unless you explicitly choose to do so.

To learn more, read:

gcloud

To detach stateful disks or declare them stateless individually for a VM in a MIG, remove the stateful disk configuration from the associated per-instance config or delete the whole per-instance config if it doesn't contain any other state.

To remove a stateful disk configuration from the associated per-instance config, use the gcloud beta compute instance-groups managed instance-configs update command with the --remove-stateful-disks flag. The --update-instance flag (default) applies the changes immediately to the instance. If you use --no-update-instance, the changes remain unapplied and are applied when you next recreate or update the instance.

gcloud beta compute instance-groups managed instance-configs update name \
  --instance instance-name \
  --remove-stateful-disks device-name[,device-name,...] \
  [--no-update-instance | --update-instance] \
  [--instance-update-minimal-action minimal-action]

Replace the following:

  • name: (Required.) The name of the MIG.
  • instance-name: (Required.) Name of the VM from which to remove stateful configuration.
  • device-name: (Required.) The device name used for attaching the disk.
  • minimal-action: Perform at least the specified action when updating the VM with its per-instance config. Can only be used together with --update-instance. The value must be one the following:

    • none: No action.
    • refresh: Apply updates that are possible to apply without stopping the VM.
    • restart: Stop the VM and then start it again.
    • replace: Recreate the VM.

    If omitted, the least disruptive action required by the update is used.

Example

You run a legacy application on a MIG named example-legacy-group. Each VM in the MIG stores application data on a boot disk with device name, boot-disk. Using per-instance configs, you configured each boot disk to be stateful. Now you have moved application data to an additional disk, and you want to make the boot disk stateless for each VM to facilitate updating to new images.

For each instance, for example, for node-1, run the command:

gcloud beta compute instance-groups managed instance-configs update example-legacy-group \
  --instance node-1 \
  --remove-stateful-disks boot-disk \
  --update-instance

The command does the following:

  1. Removes configuration for the disk with device name boot-disk from the per-instance config for node-1.
  2. Applies the per-instance config update to the node-1 VM immediately because the --update-instance flag is included. The MIG removes the boot disk from the managed instance's preservedStateFromConfig and treats the boot disk as stateless, which means that the MIG recreates the disk from its boot image in the instance template on subsequent instance recreation, update, or autohealing events.

API

To detach stateful disks or declare them stateless individually for a VM in a MIG, remove the stateful disk configuration from the associated per-instance config or delete the whole per-instance config if it doesn't contain any other state.

To remove a stateful disk configuration from the associated per-instance config, use the instanceGroupManagers.patchPerInstanceConfigs method:

POST https://compute.googleapis.com/compute/beta/projects/project-id/zones/zone/instanceGroupManagers/name/patchPerInstanceConfigs

{
  "perInstanceConfigs": [
    {
      "name": "instance-name",
      "preservedState" : {
        "disks": {
          "device-name" : null
          },
          ...
        }
      },
      "fingerprint: "fingerprint"
    },
    ...
  ]
}

Replace the following:

  • project-id: The project ID for the request.
  • zone: The zone where the MIG is located (applies to a zonal MIG).
    • For a regional MIG, replace zones/zone with regions/region and specify the region of the MIG.
  • name: The name of the MIG.
  • instance-name: (Required.) The name of the VM from which to remove stateful configuration.
  • device-name: (Required.) The device name used for attaching the disk.
  • fingerprint: The fingerprint for the given config if it already exists. Used for optimistic locking. To see the latest fingerprint, make a get() request to retrieve the resource.

The patchPerInstanceConfigs method patches the specified per-instance configs but does not apply the changes to the associated VMs. The changes are applied to a VM when you recreate or update the instance. Apply the changes manually to a VM, or use proactive auto-updating.

Example

You run a legacy application on a MIG named example-legacy-group. Each VM in the MIG stores application data on a boot disk with device name boot-disk. You configured the boot disk as stateful in the MIG's per- instance configs when migrating the standalone VMs into the MIG. You have moved application data to an additional disk and now want to make the boot disk stateless for each VM to make it easy to update to new images.

  1. Call the patchPerInstanceConfigs method for the instances, for example, for node-1 with a null value for the boot disk:

    POST https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-legacy-group/patchPerInstanceConfigs
    
    {
      "perInstanceConfigs": [
        {
          "name": "node-1",
          "preservedState" : {
            "disks": {
              "boot-disk" : null
            }
          }
        }
      ]
    }
    

    The method removes configuration for the disk with device name boot- disk from the per-instance config for node-1. The config update is not yet applied to the node-1 VM instance. The config update will be applied on the next instance recreation or update.

  2. To apply the per-instance config update to the node-1 VM instance, call the instanceGroupManagers.applyUpdatesToInstances method for the instance:

    POST https://compute.googleapis.com/compute/beta/projects/example-project/zones/us-east1-c/instanceGroupManagers/example-legacy-group/applyUpdatesToInstances
    {
      "instances": ["/zones/us-east1-c/instances/node-1"]
    }
    

    The MIG removes the boot disk from the preservedStateFromConfig for the node-1 instance and treats the disk as stateless. That is, the MIG recreates the disk from its boot image in the instance template on subsequent instance recreation, update, or autohealing events.

Feedback

We want to learn about your use cases, challenges, and feedback about stateful MIGs. Please share your feedback with our team at mig-discuss@google.com.

What's next