About resize requests in a MIG

This document describes how resize requests in a managed instance group (MIG) work and their limitations. Use resize requests to create virtual machine (VM) instances with GPUs all at once in a MIG.

Creating VMs all at once in a MIG through a resize request is useful in the following scenarios:

  • When you want GPU VMs for a specific time only, a resize request increases the chances of obtaining GPUs, which are highly-demanded resources.

  • When you want an exact number of VMs to run a job, a resize request helps you to create VMs all at once. By using resize requests, you can also avoid unnecessary charges for the partial capacity that Compute Engine creates while you wait for all the resources to be available.

How resize requests work

When creating a resize request, you must specify the following:

  • resizeBy: the number of VMs that you want to create all at once as part of the request.

  • requestedRunDuration: the duration for which the VMs created as part of the request must run. The run duration must be between 10 minutes and 7 days. At the end of the run duration, the MIG deletes the created VMs.

When you create a resize request in a MIG, Compute Engine sets the state of the request to CREATING, and then transitions it to ACCEPTED when the request is created. Dynamic Workload Scheduler (DWS), the underlying scheduler mechanism, schedules resize requests created across Compute Engine based on requested durations and resource availability.

After DWS schedules the creation of the requested number of VMs, the MIG increases its target size by the number of requested VMs and creates managed instances that are in a CREATING status. These managed instances represent the VMs that the MIG will create when the resize request succeeds. You can't delete the managed instances that are in CREATING status unless you cancel the resize request.

If you lack quota for the requested resources or the resources are temporarily unavailable, the DWS persists the request until you have sufficient quota and the resources become available.

An accepted resize request remains as such until Compute Engine sets its state to one of the following:

  • SUCCEEDED: the MIG created the requested number of VMs all at once. The VMs run until the MIG deletes them after the specified run duration ends, or until you delete the VMs.

  • FAILED: the resize request failed due to a technical error and Compute Engine decreased the target size of the MIG by the number of requested VMs.

  • CANCELLED: a user canceled the resize request and Compute Engine decreased the target size of the MIG by the number of requested VMs. If you want to stop an accepted resize request from creating VMs, you must cancel the resize request, and then you can optionally delete it. If you don't delete a canceled resize request, Compute Engine automatically deletes it 14 days after it's canceled.

To check the state of an accepted resize request or troubleshoot it, view the details of the resize request.

If you delete a MIG containing resize requests, this operation also deletes any resize requests and VMs in the MIG. However, if you delete a MIG when the MIG is creating VMs to fulfill a resize request, Compute Engine waits until the MIG has finished creating the requested number of VMs and the state of the resize request transitions to SUCCEEDED before deleting the MIG.

Limitations

The following sections outline the limitations for creating resize requests in a MIG.

Limitations for resize requests

For resize requests, the following limitations apply:

  • You can use resize request to obtain GPU VMs only.

  • You can create resize request only in zonal MIGs.

  • You can only cancel accepted (ACCEPTED) resize requests.

  • You can only delete a resize request after it succeeds (SUCCEEDED), fails (FAILED), or a user cancels it (CANCELLED).

Limitations for the instance template

For the instance template used in the MIG in which you want to create resize requests, the following limitations apply:

Limitations for the MIG

For the MIG in which you want to create resize requests, the following limitations apply:

Pricing

There are no costs associated with creating, canceling, or deleting resize requests in a MIG. You incur charges when the MIG creates the requested number VMs all at once. When the MIG deletes the VMs after their run duration ends, or you manually delete the VMs, you stop incurring charges.

What's next