Create groups of GPU VMs in bulk

You can create a group of VMs that have attached GPUs by using the bulk creation process. With the bulk creation process, you get upfront validation where the request fails fast if it is not feasible. Also if you use the region flag, the bulk creation API automatically chooses the zone that has the capacity to fulfill the request. For more information about the bulk creation, see About bulk creation of VMs.

Before you begin

  • If you want to use the command-line examples in this guide, do the following:
    1. Install or update to the latest version of the Google Cloud CLI.
    2. Set a default region and zone.
  • If you want to use the API examples in this guide, set up API access.
  • To review additional prerequisite steps such as selecting an OS image and checking GPU quota, review the overview document.

Overview

When creating VMs with attached GPUs using the bulk creation method, you can choose to create VMs in a region (such as us-central1) or in a specific zone such as (us-central1-a).

If you choose to specify a region, Compute Engine places the VMs in any zone within the region that supports GPUs.

Create groups of accelerator-optimized VMs

The accelerator-optimized machine family is available in A2 standard and ultra machine types, and the G2 standard machine type.

Each accelerator-optimized machine type has a fixed number of a specific NVIDIA GPU model attached.

  • For A2 accelerator-optimized machine types, NVIDIA A100 GPUs are attached. These are available in both A100 40GB and A100 80GB options.
  • For G2 accelerator-optimized machine types, NVIDIA L4 GPUs are attached.

You create a group of accelerator-optimized VMs by using either the Google Cloud CLI, or the Compute Engine API.

gcloud

To create a group of VMs, use the gcloud compute instances bulk create command. For more information about the parameters and how to use this command, see Create VMs in bulk.

The following optional flags are shown in the example command:

  • The --preemptible flag which configures your VM as a preemptible VM instance. This reduces the cost of your VM and the attached GPUs. For more information, see GPUs on preemptible instances.

  • The --accelerator flag to specify a virtual workstation. NVIDIA RTX Virtual Workstations (vWS) are supported for only G2 VMs.

Example

This example creates two VMs that have attached GPUs by using the following specifications:

gcloud compute instances bulk create \
    --name-pattern="my-test-vm-#" \
    --region=REGION \
    --count=2 \
    --machine-type=MACHINE_TYPE \
    --boot-disk-size=200 \
    --image=IMAGE \
    --image-project=IMAGE_PROJECT \
    --on-host-maintenance=TERMINATE --restart-on-failure \
    [--preemptible] \
    [--accelerator=type=nvidia-l4-vws,count=VWS_ACCELERATOR_COUNT]

Replace the following:

  • REGION: the region for the VMs. This region must support your selected GPU model.

  • MACHINE_TYPE: the machine type that you selected. Choose from one of the following:

    • An A2 machine type.
    • A G2 machine type. G2 machine types also support custom memory. Memory must be a multiple of 1024 MB and within the supported memory range. For example, to create a VM with 4 vCPUs and 19 GB of memory specify --machine-type=g2-custom-4-19456.

  • IMAGE: an operating system image that supports GPUs.

    If you want to use the latest image in an image family, replace the --image flag with the --image-family flag and set its value to an image family that supports GPUs. For example: --image-family=rocky-linux-8-optimized-gcp.

    You can also specify a custom image or Deep Learning VM Images.

  • IMAGE_PROJECT: the Compute Engine image project that the OS image belongs to. If using a custom image or Deep Learning VM Images, specify the project that those images belong to.

  • VWS_ACCELERATOR_COUNT: the number of virtual GPUs that you need.

If successful, the output is similar to the following:

NAME          ZONE
my-test-vm-1  us-central1-b
my-test-vm-2  us-central1-b
Bulk create request finished with status message: [VM instances created: 2, failed: 0.]

API

Use the instances.bulkInsert method with the required parameters to create multiple VMs in a zone. For more information about the parameters and how to use this command, see Create VMs in bulk.

Example

This example creates two VMs that have attached GPUs by using the following specifications:

  • VM names: my-test-vm-1, my-test-vm-2

  • Each VM has two GPUs attached, specified by using the appropriate accelerator-optimized machine type

    POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/regions/REGION/instances/bulkInsert
{
"namePattern":"my-test-vm-#",
"count":"2",
"instanceProperties": {
  "machineType":MACHINE_TYPE,
  "disks":[
    {
      "type":"PERSISTENT",
      "initializeParams":{
        "diskSizeGb":"200",
        "sourceImage":SOURCE_IMAGE_URI
      },
      "boot":true
    }
  ],
  "name": "default",
  "networkInterfaces":
  [
    {
      "network": "projects/PROJECT_ID/global/networks/default"
    }
  ],
  "scheduling":{
    "onHostMaintenance":"TERMINATE",
    "automaticRestart":true
  }
}
}

Replace the following:

  • PROJECT_ID: your project ID
  • REGION: the region for the VMs. This region must support your selected GPU model.

  • MACHINE_TYPE: the machine type that you selected. Choose from one of the following:

    • An A2 machine type.
    • A G2 machine type. G2 machine types also support custom memory. Memory must be a multiple of 1024 MB and within the supported memory range. For example, to create a VM with 4 vCPUs and 19 GB of memory specify --machine-type=g2-custom-4-19456.

  • SOURCE_IMAGE_URI: the URI for the specific image or image family that you want to use.

    For example:

    • Specific image: "sourceImage": "projects/rocky-linux-cloud/global/images/rocky-linux-8-optimized-gcp-v20220719"
    • Image family: "sourceImage": "projects/rocky-linux-cloud/global/images/family/rocky-linux-8-optimized-gcp".

    When you specify an image family, Compute Engine creates a VM from the most recent, non-deprecated OS image in that family. For more information about when to use image families, see Image family best practices.

Additional settings:

  • You can reduce the cost of your VM and the attached GPUs by using preemptible VMs. For more information, see GPUs on preemptible VM instances. To set the VM to be preemptible, add the "preemptible": true option to your request.

    "scheduling":
  {
    "onHostMaintenance": "terminate",
    "automaticRestart": true,
    "preemptible": true
  }

  • For G2 VMs, NVIDIA RTX Virtual Workstations (vWS) are supported. To specify a virtual workstation, add the guestAccelerators option to your request. Replace VWS_ACCELERATOR_COUNT with the number of virtual GPUs that you need.

    "guestAccelerators":
 [
   {
     "acceleratorCount": VWS_ACCELERATOR_COUNT,
     "acceleratorType": "projects/PROJECT_ID/zones/ZONEacceleratorTypes/nvidia-l4-vws"
   }
 ]

Limitations

A2 standard

  • You don't receive sustained use discounts for VMs that use A2 standard machine types.
  • You can only use A2 standard machine types in certain regions and zones.
  • The A2 standard machine types is only available on the Cascade Lake platform.
  • You can't use regional persistent disks on VMs that use A2 standard machine types.
  • You can't use the a2-megagpu-16g A2 standard machine type on Windows operating systems. When using Windows operating systems, choose a different A2 standard machine type.
  • You can't do a quick format of the attached Local SSDs on Windows VMs that use A2 standard machine types. To format these Local SSDs, you must do a full format by using the diskpart utility and specifying format fs=ntfs label=tmpfs.

A2 ultra

  • You don't receive sustained use discounts for VMs that use A2 ultra machine types.
  • You can only use A2 ultra machine types in certain regions and zones.
  • The A2 ultra machine types is only available on the Cascade Lake platform.
  • You can't use regional persistent disks on VMs that use A2 ultra machine types.
  • You can't do a quick format of the attached Local SSDs on Windows VMs that use A2 ultra machine types. To format these Local SSDs, you must do a full format by using the diskpart utility and specifying format fs=ntfs label=tmpfs.
  • You can't change the machine type for a VM that uses the A2 ultra machine family. If you need to use a different A2 ultra machine type, or any other machine family, you must create a new VM.
  • You can't change any other machine type to an A2 ultra machine type. If you need to create a VM that uses an A2 ultra machine type, you must create a new VM.

G2 standard

  • You don't receive sustained use discounts for VMs that use G2 standard machine types.
  • You can only use G2 standard machine types in certain regions and zones.
  • The G2 standard machine types is only available on the Cascade Lake platform.
  • You can't use regional persistent disks on VMs that use G2 standard machine types.
  • Standard persistent disks (pd-standard) are not supported on VMs that use G2 standard machine types. For supported disk types, see Supported disk types for G2.
  • You can't create Multi-Instance GPUs on G2 standard machine types.
  • You can't use Deep Learning VM Images as boot disks for your VMs that use G2 standard machine types.
  • The current default driver for Container-Optimized OS doesn't support L4 GPUs running on G2 machine types. Container-Optimized OS also only support a select set of drivers. If you want to use Container-Optimized OS on G2 machine types, review the following notes:
    • Use a Container-Optimized OS version that supports the minimum recommended NVIDIA driver version 525.60.13 or later. For more information, review the Container-Optimized OS release notes.
    • When you install the driver, specify the latest available version that works for the L4 GPUs. For example, sudo cos-extensions install gpu -- -version=525.60.13.
  • You must use the Google Cloud CLI or the Compute Engine API to create G2 VMs for the following scenarios:
    • You want to specify custom memory values.
    • You want to customize the number of visible CPU cores.

Create groups of N1-general purpose VMs

You create a group of VMs with attached GPUs by using either the Google Cloud CLI, or the Compute Engine API.

This section describes how to create multiple VMs using the following GPU types:

NVIDIA GPUs:

  • NVIDIA T4: nvidia-tesla-t4
  • NVIDIA P4: nvidia-tesla-p4
  • NVIDIA P100: nvidia-tesla-p100
  • NVIDIA V100: nvidia-tesla-v100
  • NVIDIA K80: nvidia-tesla-k80. See NVIDIA K80 EOL.

NVIDIA RTX Virtual Workstation (vWS) (formerly known as NVIDIA GRID):

  • NVIDIA T4 Virtual Workstation: nvidia-tesla-t4-vws
  • NVIDIA P4 Virtual Workstation: nvidia-tesla-p4-vws

  • NVIDIA P100 Virtual Workstation: nvidia-tesla-p100-vws

    For these virtual workstations, an NVIDIA RTX Virtual Workstation (vWS) license is automatically added to your VM.

gcloud

To create a group of VMs, use the gcloud compute instances bulk create command. For more information about the parameters and how to use this command, see Create VMs in bulk.

Example

The following example creates two VMs with attached GPUs using the following specifications:

  • VM names: my-test-vm-1, my-test-vm-2
  • VMs created in any zone in us-central1 that supports GPUs
  • Each VM has two T4 GPUs attached, specified by using the accelerator type and accelerator count flags
  • Each VM has GPU drivers installed
  • Each VM uses the Deep Learning VM image pytorch-latest-gpu-v20211028-debian-10
gcloud compute instances bulk create \
    --name-pattern="my-test-vm-#" \
    --count=2 \
    --region=us-central1 \
    --machine-type=n1-standard-2 \
    --accelerator type=nvidia-tesla-t4,count=2 \
    --boot-disk-size=200 \
    --metadata="install-nvidia-driver=True" \
    --scopes="https://www.googleapis.com/auth/cloud-platform" \
     --image=pytorch-latest-gpu-v20211028-debian-10 \
    --image-project=deeplearning-platform-release \
    --on-host-maintenance=TERMINATE --restart-on-failure

If successful, the output is similar to the following:

NAME          ZONE
my-test-vm-1  us-central1-b
my-test-vm-2  us-central1-b
Bulk create request finished with status message: [VM instances created: 2, failed: 0.]

API

Use the instances.bulkInsert method with the required parameters to create multiple VMs in a zone. For more information about the parameters and how to use this command, see Create VMs in bulk.

Example

The following example creates two VMs with attached GPUs using the following specifications:

  • VM names: my-test-vm-1, my-test-vm-2
  • VMs created in any zone in us-central1 that supports GPUs
  • Each VM has two T4 GPUs attached, specified by using the accelerator type and accelerator count flags
  • Each VM has GPU drivers installed
  • Each VM uses the Deep Learning VM image pytorch-latest-gpu-v20211028-debian-10

Replace PROJECT_ID with your project ID.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/region/us-central1/instances/bulkInsert

{
    "namePattern":"my-test-vm-#",
    "count":"2",
    "instanceProperties": {
      "machineType":"n1-standard-2",
      "disks":[
        {
          "type":"PERSISTENT",
          "initializeParams":{
            "diskSizeGb":"200",
            "sourceImage":"projects/deeplearning-platform-release/global/images/pytorch-latest-gpu-v20211028-debian-10"
          },
          "boot":true
        }
      ],
      "name": "default",
      "networkInterfaces":
      [
        {
          "network": "projects/PROJECT_ID/global/networks/default"
        }
      ],
      "guestAccelerators":
      [
        {
          "acceleratorCount": 2,
          "acceleratorType": "nvidia-tesla-t4"
        }
      ],
      "scheduling":{
        "onHostMaintenance":"TERMINATE",
        "automaticRestart":true
      },
      "metadata":{
        "items":[
          {
            "key":"install-nvidia-driver",
            "value":"True"
          }
        ]
      }
  }
 }

What's next?