Add or remove GPUs

Linux Windows

Compute Engine provides graphics processing units (GPUs) that you can add to your virtual machine (VM) instances. You can use these GPUs to accelerate specific workloads on your VMs such as machine learning and data processing.

You can only use two machine families when running GPUs on Compute Engine:

The accelerator-optimized machine family: A3, A2, and G2.
The N1 general-purpose machine family. You can use most N1 machine types except the N1 shared-core machine type. If you are not using an N1 general-purpose machine, you can switch to an N1 general-purpose machine and then add the GPUs.

Before you begin

To review additional prerequisite steps such as selecting an OS image and checking GPU quota, review the overview document.
If you haven't already, set up authentication. Authentication is the process by which your identity is verified for access to Google Cloud services and APIs. To run code or samples from a local development environment, you can authenticate to Compute Engine as follows.

Select the tab for how you plan to use the samples on this page:
Console

When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.
REST

To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

Accelerator-optimized VMs

Each accelerator-optimized machine type has a specific model of NVIDIA GPUs attached.

For A3 accelerator-optimized machine types, NVIDIA H100 80GB GPUs are attached.
For A2 accelerator-optimized machine types, NVIDIA A100 GPUs are attached. These are available in both A100 40GB and A100 80GB options.
For G2 accelerator-optimized machine types, NVIDIA L4 GPUs are attached.

You can modify each accelerator-optimized VM as follows:

For A2 standard VMs, you can modify the GPU count by switching from one A2 standard machine type to another A2 standard machine type.
For A3 standard and A2 ultra VMs, you can't modify the machine type. If you are using an A3 standard or A2 ultra machine type for your VM and you need to change the machine type, create a new VM.
For G2 standard VMs, you can do the following:
- You can modify the GPU count by switching from one G2 standard machine type to another G2 standard machine type.
- You can switch from a G2 standard machine type to a machine type from a different machine family such as general-purpose or compute-optimized. See Change the machine type.
You can't remove GPUs from an accelerator-optimized machine type. If you no longer require GPUs, complete the following:
- For A2 standard or ultra VMs, create a new VM.
- For G2 VMs, change to a machine type from a different machine family.

Modify the GPU count

You can modify the GPU count of an A2 standard or G2 accelerator-optimized VM by using either the Google Cloud console, or REST.

Console

You can modify the number of GPUs for your VM by stopping the VM and editing the VM configuration.

Verify that all of your critical applications are stopped on the VM.
In the Google Cloud console, go to the VM instances page to see your list of VMs.

Go to VM instances
Click the name of the VM that you want to modify the number of GPUs for. The Details page opens.
Complete the following steps from the Details page.
1. If the VM is running, on the toolbar, click Stop. Then, wait for the VM to stop.
2. On the toolbar, click Edit.
3. In the Machine configuration section, select the GPUs machine family, and then do the following:
  1. In the Number of GPUs list, increase or decrease the GPU count.
    
    Note: Each accelerator-optimized machine type has a specific number of GPUs attached. If you adjust the number of GPUs, the machine type changes.
4. To apply your changes, click Save.
5. To restart the VM, click Start/Resume.

REST

You can modify the number of GPUs on your VM by stopping the VM and changing the machine type. Each accelerator-optimized machine type has a specific number of GPUs attached. If you change the machine type, this adjusts the number of GPUs that are attached to the VM.

Verify that all of your critical applications are stopped on the VM, and then create a POST command to stop the VM so it can move to a host system where GPUs are available.
```
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/stop
```

After the VM stops, create a POST request to modify the machine type.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/setMachineType

{
   machineType: "zones/ZONE/machineTypes/MACHINE_TYPE"
}

Start the VM.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/start

Replace the following:

PROJECT_ID: your project ID.
VM_NAME: the name of the VM that you want to add GPUs to.
ZONE: the zone where the VM is located. This zone must support GPUs.
MACHINE_TYPE: the machine type that you want to use. It must be one of the following:
- If your VM uses an A2 standard machine, select another A2 machine type.
- If your VM uses a G2 machine type, select another G2 machine type. G2 machine types also support custom memory. Memory must be a multiple of 1024 MB and within the supported memory range. For example, to create a VM with 4 vCPUs and 19 GB of memory specify --machine-type=g2-custom-4-19456.

Limitations

A2 standard

You don't receive sustained use discounts and flexible committed use discounts for VMs that use A2 standard machine types.
You can only use A2 standard machine types in certain regions and zones.
You can't use regional persistent disks on VMs that use A2 standard machine types.
The A2 standard machine type is only available on the Cascade Lake platform.
If your VM uses an A2 standard machine type, you can only switch from one A2 standard machine type to another A2 standard machine type. You can't change to any other machine type. For more information, see Modify accelerator-optmized VMs.
You can't use the a2-megagpu-16g A2 standard machine type on Windows operating systems. When using Windows operating systems, choose a different A2 standard machine type.
You can't do a quick format of the attached Local SSDs on Windows VMs that use A2 standard machine types. To format these Local SSDs, you must do a full format by using the diskpart utility and specifying format fs=ntfs label=tmpfs.
A2 standard machine types don't support sole-tenancy.

A2 ultra

You don't receive sustained use discounts and flexible committed use discounts for VMs that use A2 ultra machine types.
You can only use A2 ultra machine types in certain regions and zones.
You can't use regional persistent disks on VMs that use A2 ultra machine types.
The A2 ultra machine type is only available on the Cascade Lake platform.
If your VM uses an A2 ultra machine type, you can't change the machine type. If you need to use a different A2 ultra machine type, or any other machine type, you must create a new VM.
You can't change any other machine type to an A2 ultra machine type. If you need to create a VM that uses an A2 ultra machine type, you must create a new VM.
You can't do a quick format of the attached Local SSDs on Windows VMs that use A2 ultra machine types. To format these Local SSDs, you must do a full format by using the diskpart utility and specifying format fs=ntfs label=tmpfs.

G2 standard

You don't receive sustained use discounts and flexible committed use discounts for VMs that use G2 standard machine types.
You can only use G2 standard machine types in certain regions and zones.
You can't use regional persistent disks on VMs that use G2 standard machine types.
The G2 standard machine type is only available on the Cascade Lake platform.
Standard persistent disks (pd-standard) are not supported on VMs that use G2 standard machine types. For supported disk types, see Supported disk types for G2.
You can't create Multi-Instance GPUs on G2 standard machine types.
If you need to change the machine type of a G2 VM, review Modify accelerator-optmized VMs.
You can't use Deep Learning VM Images as boot disks for your VMs that use G2 standard machine types.
The current default driver for Container-Optimized OS doesn't support L4 GPUs running on G2 machine types. Container-Optimized OS also only support a select set of drivers. If you want to use Container-Optimized OS on G2 machine types, review the following notes:
- Use a Container-Optimized OS version that supports the minimum recommended NVIDIA driver version 525.60.13 or later. For more information, review the Container-Optimized OS release notes.
- When you install the driver, specify the latest available version that works for the L4 GPUs. For example, sudo cos-extensions install gpu -- -version=525.60.13.
You must use the Google Cloud CLI or REST to create G2 VMs for the following scenarios:
- You want to specify custom memory values.
- You want to customize the number of visible CPU cores.

N1-general purpose VMs

This section covers how to add, modify, or remove GPUs from a N1-general purpose machine.

In summary, the process to add, modify, or remove GPUs from an existing VM is as follows:

Check that your VM has a boot disk size of at least 40 GB.
Stop the VM.
Add, modify, or remove the GPUs.

If your VM didn't have GPUs attached before, you need to complete the following steps:
1. Prepare your VM for the modification.
2. Modify the host maintenance setting for the VM. VMs with GPUs cannot live migrate because they are assigned to specific hardware devices. For more information, see GPU restrictions.
3. Change the machine type. GPUs are only supported on select machine types.
4. Install a GPU driver on your VM, so that your system can use the device.

Prepare your VM

When a GPU is added to a VM, the order of the network interface can change.

Most public images on Compute Engine don't have persistent network interface names and adjust to the new order.

However, if you are using either SLES or a custom image, you must update the system setting to prevent the network interface from persisting. To prevent the network interface from persisting, run the following command on your VM:

 rm /etc/udev/rules.d/70-persistent-net.rules

Add GPUs or modify GPU type on existing VMs

This section covers how to add GPUs, or modify the GPU type on an existing N1 general-purpose VMs. This procedure supports the following GPU types:

NVIDIA GPUs:

NVIDIA T4: nvidia-tesla-t4
NVIDIA P4: nvidia-tesla-p4
NVIDIA P100: nvidia-tesla-p100
NVIDIA V100: nvidia-tesla-v100
NVIDIA K80: nvidia-tesla-k80. See NVIDIA K80 EOL.

NVIDIA RTX Virtual Workstation (vWS) (formerly known as NVIDIA GRID):

NVIDIA T4 Virtual Workstation: nvidia-tesla-t4-vws
NVIDIA P4 Virtual Workstation: nvidia-tesla-p4-vws
NVIDIA P100 Virtual Workstation: nvidia-tesla-p100-vws

For these virtual workstations, an NVIDIA RTX Virtual Workstation (vWS) license is automatically added to your VM.

Console

To add GPUs or modify the GPU type, complete the following steps.

Verify that all of your critical applications are stopped on the VM.
In the Google Cloud console, go to the VM instances page to see your list of VMs.

Go to VM instances
Click the name of the VM that you want to update. The Details page opens.
Complete the following steps from the Details page.
1. If the VM is running, on the toolbar, click Stop. Then, wait for the VM to stop.
2. On the toolbar, click Edit.
3. In the Machine configuration section, select the GPUs machine family, and then do the following:
  1. In the GPU type list, select or switch to any of the GPU types supported on N1 VMs.
  2. In the Number of GPUs list, select the number of GPUs.
  3. If your GPU model supports NVIDIA RTX Virtual Workstations (vWS) for graphics workloads, and you plan on running graphics-intensive workloads on this VM, select Enable Virtual Workstation (NVIDIA GRID).
4. If your VM didn't have GPUs attached before, complete the following:
  1. If the VM has a shared-core machine type, you must change the machine type. In the Machine type list, select one of the preset N1 machine types. Alternatively, you can also specify custom machine type settings.
  2. In the Management section, complete the following:
    1. In the On host maintenance list, select Terminate VM instance. VMs with attached GPUs can't live migrate. See Handle GPU host events.
    2. In the Automatic restart list, select On.
5. To apply your changes, click Save.
6. To restart the VM, click Start/Resume.

REST

You can add or modify GPUs on your VM by stopping the VM and changing your VM's configuration through the API.

Verify that all of your critical applications are stopped on the VM and then create a POST command to stop the VM so it can move to a host system where GPUs are available.
```
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/stop
```
If your VM didn't have GPUs attached before, complete the following steps:
1. Identify the GPU type that you want to add to your VM. You can submit a GET request to list the GPU types that are available to your project in a specific zone.
```
GET https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/acceleratorTypes
```
2. If the VM has a shared-core machine type, you must change the machine type to have one or more vCPUs. You cannot add accelerators to VMs with shared-core machine types.
3. Create a POST command to set the scheduling options for the VM.
```
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/setScheduling

{
"onHostMaintenance": "TERMINATE",
"automaticRestart": true
}
```

Create a POST request to add or modify the GPUs that are attached to your VM.

POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/setMachineResources

    {
     "guestAccelerators": [
      {
        "acceleratorCount": ACCELERATOR_COUNT,
        "acceleratorType": "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/acceleratorTypes/ACCELERATOR_TYPE"
      }
     ]
    }

Start the VM.
```
POST https://compute.googleapis.com/compute/v1/projects/PROJECT_ID/zones/ZONE/instances/VM_NAME/start
```
Replace the following:
- PROJECT_ID: your project ID.
- VM_NAME: the name of the VM that you want to add GPUs to.
- ZONE: the zone where the VM is located.
- ACCELERATOR_COUNT: the number of GPUs that you want attached to your VM. For a list of GPU limits based on the machine type of your VM, see GPUs on Compute Engine.
- ACCELERATOR_TYPE: the GPU model that you want to attach or switch to. If you plan on running graphics-intensive workloads on this VM, use one of the virtual workstation models.
  
  Choose one of the following values:
  - NVIDIA GPUs:
    - NVIDIA T4: nvidia-tesla-t4
    - NVIDIA P4: nvidia-tesla-p4
    - NVIDIA P100: nvidia-tesla-p100
    - NVIDIA V100: nvidia-tesla-v100
    - NVIDIA K80: nvidia-tesla-k80. See NVIDIA K80 EOL.
  - NVIDIA RTX Virtual Workstation (vWS) (formerly known as NVIDIA GRID):
    - NVIDIA T4 Virtual Workstation: nvidia-tesla-t4-vws
    - NVIDIA P4 Virtual Workstation: nvidia-tesla-p4-vws
    - NVIDIA P100 Virtual Workstation: nvidia-tesla-p100-vws
    For these virtual workstations, an NVIDIA RTX Virtual Workstation (vWS) license is automatically added to your VM.

Install drivers

To install the drivers, choose one of the following options:

If you plan to run graphics-intensive workloads, such as those for gaming and visualization, install drivers for the NVIDIA RTX Virtual Workstation.
For most workloads, install the GPU drivers.

Remove GPUs

This section covers how to remove the following GPU types from an existing N1 general-purpose VM.

NVIDIA GPUs:

NVIDIA T4: nvidia-tesla-t4
NVIDIA P4: nvidia-tesla-p4
NVIDIA P100: nvidia-tesla-p100
NVIDIA V100: nvidia-tesla-v100
NVIDIA K80: nvidia-tesla-k80. See NVIDIA K80 EOL.

NVIDIA RTX Virtual Workstation (vWS) (formerly known as NVIDIA GRID):

NVIDIA T4 Virtual Workstation: nvidia-tesla-t4-vws
NVIDIA P4 Virtual Workstation: nvidia-tesla-p4-vws
NVIDIA P100 Virtual Workstation: nvidia-tesla-p100-vws

For these virtual workstations, an NVIDIA RTX Virtual Workstation (vWS) license is automatically added to your VM.

You can use the Google Cloud console to remove GPUs from an existing VM. To remove GPUs, complete the following steps:

Verify that all of your critical applications are stopped on the VM.
In the Google Cloud console, go to the VM instances page to see your list of VMs.

Go to VM instances
Click the name of the VM that you want to remove GPUs from. The Details page opens.
Complete the following steps from the Details page.
1. If the VM is running, on the toolbar, click Stop. Then, wait for the VM to stop.
2. On the toolbar, click Edit.
3. In the Machine configuration section, select the General purpose machine family, and then do the following:
  1. To view attached GPUs, expand Advanced configurations.
  2. In the GPUs section, remove GPUs using one of the following options:
    - To remove some GPUs, in the Number of GPUs list, select a new number.
    - To remove all GPUs, click Delete GPU.
4. Optional: Modify the VM host maintenance policy setting. VMs with GPUs must have the host maintenance policy set to Terminate VM instance. But if you removed all GPUs, you have the option to live migrate this VM during host maintenance. For more information, see Set VM host maintenance policy.
5. To apply your changes, click Save.
6. To restart the VM, click Start/Resume.

What's next?

Learn more about GPU platforms.
Add Local SSDs to your instances. Local SSD devices pair well with GPUs when your apps require high-performance storage.
Create groups of GPU instances using instance templates.
To monitor GPU performance, see Monitoring GPU performance.
To improve network performance, see Use higher network bandwidth.
To handle GPU host maintenance, see Handling GPU host events.
Try the Running TensorFlow Inference Workloads at Scale with TensorRT5 and NVIDIA T4 GPU tutorial.