Troubleshoot GPUs in GKE

Autopilot Standard

This page shows you how to resolve issues related to GPUs in Google Kubernetes Engine (GKE).

If you need additional assistance, reach out to Cloud Customer Care.

GPU driver installation

This section provides troubleshooting information for automatic NVIDIA device driver installation in GKE.

Driver installation fails in Ubuntu nodes

If you use Ubuntu nodes that have attached L4 GPUs or H100 GPUs, the default GPU driver that GKE installs might not be at or later than the required version for those GPUs. As a result, the GPU device plugin Pod remains stuck in the Pending state and your GPU workloads on those nodes might experience issues.

To resolve this issue, we recommend upgrading to the following GKE versions which install GPU driver version 535 as the default driver:

1.26.15-gke.1483000 and later
1.27.15-gke.1039000 and later
1.28.11-gke.1044000 and later
1.29.6-gke.1073000 and later
1.30.2-gke.1124000 and later

Alternatively, you can manually install driver version 535 or later by running the following command:

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/ubuntu/daemonset-preloaded-R535.yaml

GPU device plugins fail with CrashLoopBackOff errors

The following issue occurs if you used the manual driver installation method in your node pool prior to January 25, 2023 and later upgraded your node pool to a GKE version that supports automatic driver installation. Both installation workloads exist at the same time and try to install conflicting driver versions on your nodes.

The GPU device plugin init container fails with the Init:CrashLoopBackOff status. The logs for the container are similar to the following:

failed to verify installation: failed to verify GPU driver installation: exit status 18

To resolve this issue, try the following methods:

Remove the manual driver installation DaemonSet from your cluster. This deletes the conflicting installation workload and lets GKE automatically install a driver to your nodes.

Note: Ensure that all of your node pools use automatic installation before you delete the DaemonSet.
```
kubectl delete -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
```
Re-apply the manual driver installation DaemonSet manifest to your cluster. On January 25, 2023, we updated the manifest to ignore nodes that use automatic driver installation.
```
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml
```
Disable automatic driver installation for your node pool. The existing driver installation DaemonSet should work as expected after the update operation completes.
```
gcloud container node-pools update POOL_NAME \
    --accelerator=type=GPU_TYPE,count=GPU_COUNT,gpu-driver-version=disabled \
    --cluster=CLUSTER_NAME \
    --location=LOCATION
```
Replace the following:
- POOL_NAME: the name of the node pool.
- GPU_TYPE: the GPU type that the node pool already uses.
- GPU_COUNT: the number of GPUs that are already attached to the node pool.
- CLUSTER_NAME: the name of the GKE cluster that contains the node pool.
- LOCATION: the Compute Engine location of the cluster.

Error: "Container image cos-nvidia-installer:fixed is not present with pull policy of Never." or "Container image ubuntu-nvidia-installer:fixed is not present with pull policy of Never."

This issue occurs when the nvidia-driver-installer Pods are in the PodInitializing state and the GPU plugin device or the GPU driver installer Pods report the following error. The specific error message depends on the operating system running on your node:

COS

Container image "cos-nvidia-installer:fixed" is not present with pull policy of Never.

Ubuntu

Container image "gke-nvidia-installer:fixed" is not present with pull policy of Never.

This issue can occur when the garbage collector removes the preloaded NVIDIA driver image to free space on a node. When the driver Pod is recreated or its container is restarted, GKE won't be able to locate the preloaded image.

To mitigate the garbage collection issue when you are running COS, upgrade your GKE nodes to one of these versions that contain the fix:

1.25.15-gke.1040000 and later
1.26.10-gke.1030000 and later
1.27.6-gke.1513000 and later
1.28.3-gke.1061000 and later

If your nodes are running Ubuntu, no fix is available yet for this garbage collection issue. To mitigate this issue on Ubuntu, you can run a privileged container that interacts with the host to ensure the correct setup of NVIDIA GPU drivers. To do so, run sudo /usr/local/bin/nvidia-container-first-boot from your node or apply the following manifest:

apiVersion: v1
kind: Pod
metadata:
  name: gke-nvidia-installer-fixup
spec:
  nodeSelector:
    cloud.google.com/gke-os-distribution: ubuntu
  hostPID: true
  containers:
  - name: installer
    image: ubuntu
    securityContext:
      privileged: true
    command:
      - nsenter
      - -at
      - '1'
      - --
      - sh
      - -c
      - "/usr/local/bin/nvidia-container-first-boot"
  restartPolicy: Never

Another potential cause of the issue is when the NVIDIA driver images are lost after node reboot or host maintenance. This may occur on confidential nodes, or nodes with GPUs, that use ephemeral local SSD storage. In this situation, GKE preloads the nvidia-installer-driver container images on nodes and moves them from the boot disk to the local SSD on first boot.

To confirm there was a host maintenance event, use the following log filter:

resource.type="gce_instance"
protoPayload.serviceName="compute.googleapis.com"
log_id("cloudaudit.googleapis.com/system_event")

To mitigate the host maintenance issue, upgrad your GKE version to one of these versions:

1.27.13-gke.1166000 and later
1.29.3-gke.1227000 and later
1.28.8-gke.1171000 and later

Error: failed to configure GPU driver installation dirs: failed to create lib64 overlay: failed to create dir /usr/local/nvidia/lib64: mkdir /usr/local/nvidia/lib64: not a directory.

You encounter this error from the GPU driver installer container inside the GPU device plugin when NCCL fastsocket is enabled:

failed to configure GPU driver installation dirs: failed to create lib64 overlay: failed to create dir /usr/local/nvidia/lib64: mkdir /usr/local/nvidia/lib64: not a directory.

This issue only happens on clusters and nodes running GKE 1.28 and 1.29.

The issue is caused by a NCCL fastsocket race condition with the GPU driver installer.

To mitigate this issue, upgrade your GKE version to one of these versions:

1.28.8-gke.1206000 and later
1.29.3-gke.1344000 and later

Error: Failed to get device for nvidia0: device nvidia0 not found.

The following error indicates that XID 62 and RmInitAdapter failed for GPU with minor 0:

Failed to get device for nvidia0: device nvidia0 not found.

NVIDIA driver version 525.105.17 has a bug that can cause communication errors (XID) and prevent the GPU from initializing properly, leading to a failure to initialize the GPU.

To fix this issue, upgrade the NVIDIA driver to driver version 525.110.11 or later.

Reset GPUs on A3 VMs

Some issues might require you to reset the GPU on an A3 VM.

To reset the GPU, follow these steps:

Remove Pods that request GPU resources from the node where you need to reset the GPU.

Disable the GPU device plugin on the node:

kubectl get nodes \
    --selector=kubernetes.io/hostname=NODE_NAME \
    --no-headers | awk '{print $1}' \
    | xargs -I{} kubectl label node {} gke-no-default-nvidia-gpu-device-plugin=true

Replace NODE_NAME with the name of the node.

Connect to the VM backing the node.

In the SSH session, reset the GPU:

/home/kubernetes/bin/nvidia/bin/nvidia-smi --gpu-reset

Re-enable the GPU device plugin:

kubectl get nodes --selector=kubernetes.io/hostname=NODE_NAME \
    --no-headers \| awk '{print $1}' \
    | xargs -I{} kubectl label node {} gke-no-default-nvidia-gpu-device-plugin=false \
    --overwrite

What's next

If you need additional assistance, reach out to Cloud Customer Care.