Troubleshooting Cloud TPU errors and performance issues

These troubleshooting documents describe error conditions and performance issues you might see while training with Cloud TPUs using TensorFlow, JAX, and PyTorch.

If you cannot tell whether the problem you are seeing is specific to a particular framework, start with Troubleshooting TensorFlow - TPU.

Monitoring with Google Cloud Monitoring

• Monitoring with Google Cloud Monitoring

  Create log-based metrics, alerts, and visualizing dashboards to help debug errors and performance issues.

Troubleshooting TensorFlow - TPU

• Troubleshooting TensorFlow - TPU models on Cloud TPU

  Identify and resolve problems you might encounter while training TensorFlow models on Cloud TPU.

Troubleshooting PyTorch - TPU

• Troubleshoot PyTorch - TPU models on Cloud TPU

  Identify and resolve problems you might encounter while training PyTorch models on Cloud TPU.

Troubleshooting JAX - TPU

• Troubleshoot JAX - TPU models on Cloud TPU

  Identify and resolve problems you might encounter while training JAX models on Cloud TPU.

Cloud TPU Error Glossary

• Cloud TPU error glossary

  A glossary of common errors and solutions for Cloud TPU.