Adding or removing GPUs

Compute Engine provides graphics processing units (GPUs) that you can add to your virtual machine instances. You can use these GPUs to accelerate specific workloads on your instances such as machine learning and data processing.

For more information about what you can do with GPUs and what types of GPU hardware are available, read GPUs on Compute Engine.

Before you begin

• If you want to use the command-line examples in this guide:
  1. Install or update to the latest version of the gcloud command-line tool.
  2. Set a default region and zone.
• If you want to use the API examples in this guide, set up API access.
• Read about GPU pricing on Compute Engine to understand the cost to use GPUs on your instances.
• Read about restrictions for instances with GPUs to learn how these instances function differently than most instances.
• Learn how instances function when you schedule them to terminate for host maintenance events. If you add GPUs to your instances, you must also set the instance to terminate during host maintenance.
• Check the quotas page to ensure that you have enough GPUs available for your project. If you need additional GPU quota, request a quota increase.

Creating an instance with a GPU

Before you create an instance with a GPU, select which boot disk image you want to use for the instance, and ensure that the appropriate GPU driver is installed.

If you are using GPUs for machine learning, you can use a Deep Learning VM image for your instance. The Deep Learning VM images have GPU drivers pre-installed, and include packages such as TensorFlow and PyTorch. You can also use the Deep Learning VM images for general GPU workloads. For information about the images available, and the packages installed on the images, see the Deep Learning VM documentation.

You can also use any public image or custom image, but some images might require a unique driver or install process that is not covered in this guide. You must identify what drivers are appropriate for your images.

For steps to install drivers, see installing GPU drivers.

Note: Check the quotas page to ensure that you have enough GPUs available in your project. If you require additional GPU quota, request a quota increase. If your project has an established billing history, it will receive quota automatically after you submit the request.

When you create an instance with one or more GPUs, you must set the instance to terminate on host maintenance. Instances with GPUs cannot live migrate because they are assigned to specific hardware devices. See GPU restrictions for details.

Create an instance with one or more GPUs using the Google Cloud Console, the gcloud command-line tool, or the API.

Console

1. Go to the VM instances page.

   Go to the VM instances page

2. Click Create instance.
3. Select a zone where GPUs are available. See the list of available zones with GPUs.
4. In the Machine configuration section, select the machine type that you want to use for this instance. Alternatively, you can specify custom machine type settings if desired.
5. In the Machine configuration section, click CPU platform and GPU to see advanced machine type options and available GPUs.
6. Click GPUs to see the list of available GPUs.
7. Specify the GPU type and the number of GPUs that you need.
8. If necessary, adjust the machine type to accommodate your desired GPU settings. If you leave these settings as they are, the instance uses the predefined machine type that you specified before opening the machine type customization screen.
9. To configure your boot disk, in the Boot disk section, click Change.
10. In the Public images tab, choose an operating system and version.
11. Click Save to confirm your boot disk options.
12. Configure any other instance settings that you require. For example, you can change the Preemptibility settings to configure your instance as a preemptible instance. This reduces the cost of your instance and the attached GPUs. Read GPUs on preemptible instances to learn more.
13. At the bottom of the page, click Create to create the instance.

gcloud

Use the regions describe command to ensure that you have sufficient GPU quota in the region where you want to create instances with GPUs. Replace region with the region that you want to check for GPU quota.

gcloud compute regions describe region

Note: To protect Compute Engine systems and users, new projects have a global GPU quota, which limits the total number of GPUs you can create in any supported zone. When you request a GPU quota, you must request a quota for the GPU models that you want to create in each region, and an additional global quota for the total number of GPUs of all types in all zones.

To create and start an instance use the gcloud compute instances create command with the following flags:

gcloud compute instances create instance-name \
    --machine-type machine-type --zone zone \
    --accelerator type=accelerator-type,count=accelerator-count \
    --image-family image-family --image-project image-project \
    --maintenance-policy TERMINATE --restart-on-failure \
    [--preemptible]

Replace the following:

• instance-name: The name for the new instance.
• machine-type: The machine type that you selected for the instance. See GPUs on Compute Engine to see what machine types are available based on your desired GPU count.
• zone: The zone for the instance.
• image-family: An image family that support GPUs.
• image-project: The image project that the image family belongs to.
• accelerator-count: The number of GPUs that you want to add to your instance. See GPUs on Compute Engine for a list of GPU limits based on the machine type of your instance.

• accelerator-type: The GPU model that you want to use. Use one of the following values:

  • NVIDIA® Tesla® T4: nvidia-tesla-t4
  • NVIDIA® Tesla® T4 Virtual Workstation with NVIDIA® GRID®: nvidia-tesla-t4-vws
  • NVIDIA® Tesla® P4: nvidia-tesla-p4
  • NVIDIA® Tesla® P4 Virtual Workstation with NVIDIA® GRID®: nvidia-tesla-p4-vws
  • NVIDIA® Tesla® P100: nvidia-tesla-p100
  • NVIDIA® Tesla® P100 Virtual Workstation with NVIDIA® GRID®: nvidia-tesla-p100-vws
  • NVIDIA® Tesla® V100: nvidia-tesla-v100
  • NVIDIA® Tesla® K80: nvidia-tesla-k80

  See GPUs on Compute Engine for a list of available GPU models.

• --preemptible is an optional flag that configures your instance as a preemptible instance. This reduces the cost of your instance and the attached GPUs. Read GPUs on preemptible instances to learn more.

For example, you can use the following gcloud command to start an Ubuntu 16.04 instance with 1 NVIDIA Tesla K80 GPU and 2 vCPUs in the us-east1-d zone.

gcloud compute instances create gpu-instance-1 \
    --machine-type n1-standard-2 --zone us-east1-d \
    --accelerator type=nvidia-tesla-k80,count=1 \
    --image-family ubuntu-1604-lts --image-project ubuntu-os-cloud \
    --maintenance-policy TERMINATE --restart-on-failure

This example command starts the instance, but CUDA and the driver must be installed on the instance ahead of time.

API

Identify the GPU type that you want to add to your instance. Submit a GET request to list the GPU types that are available to your project in a specific zone.

GET https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/acceleratorTypes

Replace the following:

• project-id: The project ID.
• zone: The zone from which you want to list the available GPU types.

In the API, create a POST request to create a new instance. Include the acceleratorType parameter to specify which GPU type you want to use, and include the acceleratorCount parameter to specify how many GPUs you want to add. Also set the onHostMaintenance parameter to TERMINATE.

POST https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances?key={YOUR_API_KEY}
{
  "machineType": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/machineTypes/machine-type",
  "disks":
  [
    {
      "type": "PERSISTENT",
      "initializeParams":
      {
        "diskSizeGb": "disk-size",
        "sourceImage": "https://compute.googleapis.com/compute/v1/projects/image-project/global/images/family/image-family"
      },
      "boot": true
    }
  ],
  "name": "instance-name",
  "networkInterfaces":
  [
    {
      "network": "https://compute.googleapis.com/compute/v1/projects/project-id/global/networks/network"
    }
  ],
  "guestAccelerators":
  [
    {
      "acceleratorCount": accelerator-count,
      "acceleratorType": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/acceleratorTypes/accelerator-type"
    }
  ],
  "scheduling":
  {
    "onHostMaintenance": "terminate",
    "automaticRestart": true,
    ["preemptible": true]
  },
}

Replace the following:

• instance-name: The name of the instance.
• project-id: Your project ID.
• zone: The zone for th instance.
• machine-type: The machine type that you selected for the instance. See GPUs on Compute Engine to see what machine types are available based on your desired GPU count.
• image-family: A boot disk image for your instance. Specify an image family from the list of available public images.
• image-project: The image project that the image family belongs to.
• disk-size: The size of your boot disk in GB.
• network: The VPC network that you want to use for the instance. You can specify default to use your default network.
• accelerator-count: The number of GPUs that you want to add to your instance. See GPUs on Compute Engine for a list of GPU limits based on the machine type of your instance.
• accelerator-type: The GPU model that you want to use. See GPUs on Compute Engine for a list of available GPU models.
• "preemptible": true is an optional parameter that configures your instance as a preemptible instance. This reduces the cost of your instance and the attached GPUs. To learn more, see GPUs on preemptible instances.

Install the GPU driver on your instance so that your system can use the device.

Adding or removing GPUs on existing instances

You can add or detach GPUs on your existing instances, but you must first stop the instance and change its host maintenance setting so that it terminates rather than live-migrating. Instances with GPUs cannot live migrate because they are assigned to specific hardware devices. See GPU restrictions for details.

Also be aware that you must install GPU drivers on this instance after you add a GPU. The boot disk image that you used to create this instance determines what drivers you need. You must identify what drivers are appropriate for the operating system on your instance's persistent boot disk images. Read installing GPU drivers for details.

You can add or remove GPUs from an instance using the Google Cloud Console or the API.

Console

You can add or remove GPUs from your instance by stopping the instance and editing your instance's configuration.

1. Verify that all of your critical applications are stopped on the instance. You must stop the instance before you can add a GPU.

2. Go to the VM instances page to see your list of instances.

   Go to the VM instances page

3. On the list of instances, click the name of the instance where you want to add GPUs. The instance details page opens.

4. At the top of the instance details page, click Stop to stop the instance.

5. After the instance stops running, click Edit to change the instance properties.

6. If the instance has a shared-core machine type, you must change the machine type to have one or more vCPUs. You cannot add accelerators to instances with shared-core machine types.

7. In the Machine configuration section, click CPU platform and GPU to see advanced machine type options and available GPUs.

8. Click GPUs to see the list of available GPUs.

9. Select the number of GPUs and the GPU model that you want to add to your instance. Alternatively, you can set the number of GPUs to None to remove existing GPUs from the instance.

10. If you added GPUs to an instance, set the host maintenance setting to Terminate. If you removed GPUs from the instance, you can optionally set the host maintenance setting back to Migrate VM instance.

11. At the bottom of the instance details page, click Save to apply your changes.

12. After the instance settings are saved, click Start at the top of the instance details page to start the instance again.

API

You can add or remove GPUs from your instance by stopping the instance and changing your instance's configuration through the API.

1. Verify that all of your critical applications are stopped on the instance and then create a POST command to stop the instance so it can move to a host system where GPUs are available.

   POST https://compute.googleapis.com/compute/v1/projects/compute/zones/zone/instances/instance-name/stop
   

   Replace the following:

   • instance-name: The name of the instance to stop. This is the instance that you want to attach GPUs to.
   • zone: The zone for where the instance is located.

2. Identify the GPU type that you want to add to your instance. Submit a GET request to list the GPU types that are available to your project in a specific zone.

   GET https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/acceleratorTypes
   

   Replace the following:

   • project-id: Your project ID.
   • zone: The zone where you want to list the available GPU types.

3. If the instance has a shared-core machine type, you must change the machine type to have one or more vCPUs. You cannot add accelerators to instances with shared-core machine types.

4. After the instance stops, create a POST request to add or remove one or more GPUs to your instance.

   POST https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/instances/instance-name/setMachineResources
   
       {
        "guestAccelerators": [
         {
           "acceleratorCount": accelerator-count,
           "acceleratorType": "https://compute.googleapis.com/compute/v1/projects/project-id/zones/zone/acceleratorTypes/accelerator-type"
         }
        ]
       }
   

   Replace the following:

   • instance-name: The name of the instance.
   • project-id: Your project ID.
   • zone: The zone for thes instance.
   • accelerator-count: The number of GPUs that you want to attach to your instance. For a list of GPU limits based on the machine type of your instance, see GPUs on Compute Engine.
   • accelerator-type: The GPU model that you want to use. For a list of available GPU models, see GPUs on Compute Engine.

5. Create a POST command to set the scheduling options for the instance. If you are adding GPUs to an instance, you must specify "onHostMaintenance": "TERMINATE". Optionally, if you are removing GPUs from an instance you can specify "onHostMaintenance": "MIGRATE".

   POST https://compute.googleapis.com/compute/v1/projects/compute/zones/zone/instances/instance-name/setScheduling
   
   {
   "onHostMaintenance": "maintenance-type",
   "automaticRestart": true
   }
   

   Replace the following:

   • instance-name: The name of the instance where you want to add GPUs.
   • zone: The zone where the instance is located.
   • maintenance-type" The action you want your instance to take when host maintenance is necessary. Specify TERMINATE if you are adding GPUs to your instance. Alternatively, you can specify "onHostMaintenance": "MIGRATE" if you have removed all of the GPUs from your instance and want the instance to resume migration on host maintenance events.

6. Start the instance.

   POST https://compute.googleapis.com/compute/v1/projects/compute/zones/zone/instances/instance-name/start
   

   Replace the following:

   • instance-name: The name of the instance that you want to add GPUs to.
   • zone: The zone where the instance is located.

Next install the GPU driver on your instance so that your system can use the device.

What's next?

• Learn more about GPUs on Compute Engine.
• Add Local SSDs to your instances. Local SSD devices pair well with GPUs when your apps require high-performance storage.
• Create groups of GPU instances using instance templates.
• To monitor GPU performance, see Monitoring GPU performance.
• To optimize GPU performance, see Optimizing GPU performance.
• To handle GPU host maintenance, see Handling GPU host maintenance events.
• Try the Running TensorFlow Inference Workloads at Scale with TensorRT5 and NVIDIA T4 GPU tutorial.