Troubleshoot GPU VMs

This page shows you how to resolve issues for VMs running on Compute Engine that have attached GPUs.

If you are trying to create a VM with attached GPUs and are getting errors, review Troubleshooting resource availability errors and Troubleshooting creating and updating VMs.

Xid messages

After you create a VM that has attached GPUs, you must install NVIDIA device drivers on your GPU VMs so that your applications can access the GPUs. However, sometimes these drivers return error messages.

An Xid message is an error report from the NVIDIA driver that is printed to the operating system's kernel log or event log for your Linux VM. These messages are placed in the /var/log/messages file.

For more information about Xid messages including potential causes, see NVIDIA documentation.

The following section provides guidance on handling some Xid messages grouped by the most common types: GPU memory errors, GPU System Processor (GSP) errors, and illegal memory access errors.

GPU memory errors

GPU memory is the memory that is available on a GPU that can be used for temporary storage of data. GPU memory is protected with Error Correction Code, ECC, which detects and corrects single bit errors (SBE) and detects and reports Double Bit Errors (DBE).

Prior to the release of the NVIDIA A100 GPUs, dynamic page retirement was supported. For NVIDIA A100 and later GPU releases (such as NVIDIA H100), row remap error recovery is introduced. ECC is enabled by default. Google highly recommends keeping ECC enabled.

The following are common GPU memory errors and their suggested resolutions.

Xid error message Resolution
Xid 48: Double Bit ECC
  1. Stop your workloads.
  2. Delete and recreate the VM. If the error persists, file a case with Cloud Customer Care.
Xid 63: ECC page retirement or row remapping recording event
  1. Stop your workloads.
  2. Reset the GPUs.
Xid 64: ECC page retirement or row remapper recording failure

And the message contains the following information:

Xid 64: All reserved rows for bank are remapped
  1. Stop your workloads.
  2. Delete and recreate the VM. If the error persists, file a case with Cloud Customer Care.

If you get at least two of the following Xid messages together:

  • Xid 48
  • Xid 63
  • Xid 64

And the message contains the following information:

Xid XX: row remap pending
  1. Stop your workloads.
  2. Reset the GPUs. Resetting the GPU allows the row remap and page retirement process to complete and heal the GPU.
Xid 92: High single-bit ECC error rate This Xid message is returned after the GPU driver corrects a correctable error, and it shouldn't affect your workloads. This Xid message is informational only. No action is needed.
Xid 94: Contained ECC error
  1. Stop your workloads.
  2. Reset the GPUs.
Xid 95: Uncontained ECC error
  1. Stop your workloads.
  2. Reset the GPUs.

GSP errors

A GPU System Processor (GSP) is a microcontroller that runs on GPUs and handles some of the low level hardware management functions.

Xid error message Resolution
Xid 119: GSP RPC timeout
  1. Stop your workloads.
  2. Delete and recreate the VM. If the error persists, collect the NVIDIA bug report and file a case with Cloud Customer Care.
Xid 120: GSP error

Illegal memory access errors

The following Xids are returned when applications have illegal memory access issues:

  • Xid 13: Graphics Engine Exception
  • Xid 31: GPU memory page fault

Illegal memory access errors are typically caused by your workloads trying to access memory that is already freed or is out of bounds. This can be caused by issues such as the dereferencing of an invalid pointer, or an out bounds array.

To resolve this issue, you need to debug your application. To debug your application, you can use cuda-memcheck and CUDA-GDB.

In some very rare cases, hardware degradation might cause illegal memory access errors to be returned. To identify if the issue is with your hardware, use NVIDIA Data Center GPU Manager (DCGM). You can run dcgmi diag -r 3 or dcgmi diag -r 4 to run different levels of test coverage and duration. If you identify that the issue is with the hardware, file a case with Cloud Customer Care.

Other common Xid error messages

Xid error message Resolution
Xid 74: NVLINK error
  1. Stop your workloads.
  2. Reset the GPUs.
Xid 79: GPU has fallen off the bus

This means the driver is not able to communicate with the GPU.

 Reboot the VM.

Reset GPUs

Some issues might require you to reset your GPUs. To reset GPUs, complete the following steps:

  • For N1, G2, and A2 VMs, reboot the VM by running sudo reboot.
  • For A3 VMs, run nvidia-smi reset GPU.

If errors persist after resetting the GPU, you need to delete and recreate the VM.

If the error persists after a delete and recreate, file a case with Cloud Customer Care to move the VM into the repair stage.