Troubleshooting Cloud TPU errors and performance issues

These troubleshooting documents describe error conditions and performance issues you might see while training with Cloud TPUs using TensorFlow, JAX, and PyTorch.

Monitoring with Stacktrace describes how to create log-based metrics that can be used to create alerts and visualizing dashboards to help debug errors and performance issues.

If you cannot tell whether the problem you are seeing is specific to a particular framework, start with Troubleshooting TensorFlow - TPU.

Monitoring with Stacktrace

• Monitoring with Stackdriver

  Create log-based metrics, alerts, and visualizing dashboards to help debug errors and performance issues.

Troubleshooting TensorFlow - TPU

• Troubleshooting TensorFlow - TPU models on Cloud TPU

  Identify and resolve problems you might encounter while training TensorFlow models on Cloud TPU.

Troubleshooting PyTorch - TPU

• Troubleshoot PyTorch - TPU models on Cloud TPU

  Identify and resolve problems you might encounter while training PyTorch models on Cloud TPU.

Troubleshooting JAX - TPU

• Troubleshoot JAX - TPU models on Cloud TPU

  Identify and resolve problems you might encounter while training JAX models on Cloud TPU.