Getting Compute Engine resources for batch processing just got easier
Product Manager, Google Compute Engine
If you need to run an embarrassingly parallel batch processing workload, it can be tricky to decide how many instances to create in each zone while accounting for available resources, quota limits and your reservations. We are excited to announce a new method of obtaining Compute Engine instances for batch processing that accounts for availability of resources in zones of a region. Now available in preview for regional managed instance groups, you can do this simply by specifying the
ANY value in the API.
The capacity-aware deployment method is particularly useful if you need to easily create many instances with a special configuration such as virtual machines (VM) with a specific CPU platform or GPU model, preemptible VMs, or instances with a large number of cores or memory size.
Now, when deploying instances to run embarrassingly parallel batch processing, such as financial modeling or rendering, you no longer have to figure out which zones support the required hardware and how many instances to create in each zone in a region to accommodate the requested capacity.
Assuming that any distribution of instances across zones works for your batch processing job, and that the workload doesn’t require resilience against zone-level failure, you can now delegate the job of obtaining the requested capacity to a regional managed instance group. A regional MIG with the new distribution shape
ANY automatically deploys instances to zones where resources are available to fulfill your request, accounting for your quota limits. This works both when you create a group or when you increase it in size.
If you use reservations to ensure that resources are available for your computation, you should specify reservation affinity in a group’s instance template. A regional MIG with distribution shape
ANY utilizes the specified reservations efficiently by prioritizing consumption of unused reserved capacity before provisioning additional resources.
A regional MIG with distribution shape ANY automatically deploys instances to zones where capacity is available, takes quotas into account, and prioritizes consumption of a specified reservation.
Depending on the availability of the requested resources, a regional MIG with
ANY distribution might deploy all instances to a single zone or spread the instances across multiple zones. The distribution shape
ANY is not suitable for highly available serving workloads such as frontend web services because a zone-level failure could result in all or most of the instances becoming unavailable if they happen to be deployed to the zone that failed.
To configure the new distribution shape
ANY when creating a regional MIG, look under the “Target distribution shape” setting on Create instance group screen in the Google Cloud Console:
Configuring a regional managed instance group's distribution shape
You can also set the distribution shape to
ANY for an existing regional MIG—for example, by running a
Summing it up
Obtaining capacity to run an embarrassingly parallel batch processing workload is easier with a regional MIG’s new distribution shape
ANY. When deciding how many instances to create in each zone, the regional MIG accounts for availability of resources in each zone, accounts for your quota limits, and prioritizes consumption of specified reservations.
Visit the regional MIG documentation to learn more about creating instances using the new distribution shape
Continue your learning at the Cloud Technical Series digital event, March 23-26, and go deeper into VM migration, application modernization, GKE, data analytics, AI/ML and more. Register here.