How to migrate a group of individual instances to a stateful MIG using Python script

A GCP Compute Engine managed instance group (MIG) supports any VM configuration that you need, and it helps manage the VMs for you. For example, when a VM in a MIG unexpectedly stops running, the MIG recreates that VM according to the configuration that you set. You can also set up an application-based health check to verify that your application responds as expected on each VM.

In addition to that, MIGs also allows you to deploy complex stateful applications, such as databases (Cassandra, ElasticSearch) or data processing applications (Kafka, Flink), where preservation of individual VM state (for example, a database shard, or app configuration) is important. 

If you are just starting to work with GCP and want to deploy a stateful application, then it’s highly recommended to use a stateful MIG configuration. A stateful MIG allows you to preserve the unique state of each instance (including instance name, attached persistent disks, IP addresses - available in Preview, and metadata) on VM restart, recreation, auto-healing, or update. You can learn more about Stateful MIGs by reading the docs.

The goal of this blog post is to help those who already have an existing stateful application running on standalone (unmanaged) Compute Engine instances and who want to automatically migrate them to a stateful MIG using a Python script. The script code and detailed instructions for its use and installation can be found here. In case you want to do this whole process manually, you can go through the Migrate an existing workload to a stateful managed instance group tutorial.

Let's dive into the logic of the script and see how it can be executed. 

Script requirements

Before running the script, make sure your existing instances meet the following criteria:

• All source instances must have the same instance configuration (for example, machine type). Remember that all instances in a MIG are created according to the same instance template. The script creates a template for you based on one of your existing instances.

• Your boot disks should be stateless. In case you must preserve state on your boot disks, then you can still create a stateful MIG, but you cannot update operating systems or software by rolling out boot image updates.

• The script will stop your source VM instances. The script leaves all source VMs stopped with their disks intact, for easy reverting if the MIG doesn't work as expected. This results in additional costs, for the following reasons:

1. The script doesn't detach or delete the original disks.

2. The script creates images from existing disks.

• You do not depend on your VM names. Source VM names remain unchanged. The new VMs in the MIG will have new names. Pay attention to this if you are somehow tied to the names of your VMs.

Script steps

1. Stop all instances

2. If needed, create a boot disk image

3. Create an instance template based on the properties of a chosen instance, while ignoring attached data disks.

4. Create an empty MIG.

5. For each instance in the original group, perform the following steps:

1. Clone all instance disks except the boot disk.

2. Create an instance in the MIG based on the instance template, and include the cloned disks from the source instance.

6. Print commands for cleaning up the source instances after you have verified that the stateful MIG serves your needs.

If you are ready to use the script, follow the link for installation and execution.

Usage example

Let’s now imagine some instance configuration and show how the script proceeds. Suppose we have three standalone instances with names “alpha”, “beta”, and “gamma” in the us-central1-a zone and we want to migrate them to a MIG with the name “greek”.

Instance “alpha” has two additional disks: “alpha-disk-1” and “alpha-disk-2”, instance “beta” has one additional disk “beta-disk-1”, and “gamma” has zero additional disks. At the same time we would like to create a disk image for the boot image.

Run the following command:

$ python3 migrate_script.py -s alpha beta gamma -z us-central1-a -m greek --image_for_boot_disk

Let’s take a look on the output step by step:

1. All the instances are stopped at the very beginning.

2. Because we set the “image_for_boot_disk” flag, an image for the boot disk is created.

3. By default, alpha is chosen to specify machine configuration in the instance template (it’s possible to set another instance explicitly).

4. An empty MIG “gamma” is created.

5. For each individual instance, we create copies of all its disks (not to damage source instances data) and attach them to VM in the MIG.

If you are convinced that everything is fine with the newly created MIG, you can delete your source instances. Use the following command:

$ gcloud compute instances delete alpha beta gamma

If instead, for any reason, you want to return to your initial configuration, use the following clean-up commands in the terminal or in the Cloud Shell:

Congratulations! Your MIG is set up! What’s next?

- Configure autohealing

- Use a stateful policy instead of per-instance configurations

- Add more VMs