Preemptible TPUs

Important: TPU Spot VMs are the latest version of preemptible TPUs. New and existing preemptible TPUs continue to be supported, and TPU Spot VMs use the same pricing model as preemptible TPUs. However, TPU Spot VMs provide new features that preemptible TPUs don't support. For example, TPU Spot VMs don't have the 24 hour run time limit of preemptible TPUs. For more information, see Manage TPU Spot VMs.

Preemptible TPUs cost much less than non-preemptible TPUs. The Cloud TPU service might preempt (shut down) these TPUs at any time, if it requires additional TPU resources.

Creating a preemptible TPU VM

gcloud

$ gcloud compute tpus tpu-vm create demo-tpu \
  --zone=europe-west4-a \
  --accelerator-type=v3-8 \
  --version=tpu-vm-tf-2.18.0-pjrt \
  --preemptible
  

where:

• demo-tpu is a name for the TPU.
• accelerator-type is the type of TPU. For more information about supported accelerator types for each TPU version, see TPU versions.
• version is the Cloud TPU VM image to install.
• preemptible allows Cloud TPU to preempt the TPU.

Pricing and quota for preemptible TPUs

Pricing for preemptible TPUs is significantly lower than for normal TPUs. For details, see the pricing page. You are not charged for TPUs if they are preempted in the first minute after you create them.

Quota for preemptible TPUs is generally higher, and is separate from the quota for normal TPUs. See the quota page.

Detecting if a TPU has been preempted

You use the following gcloud command to check whether the Cloud TPU service has preempted your TPU:

List your available TPUs:

gcloud compute tpus tpu-vm list
  --zone=us-central1-b
  

The output of the command displays the details of the TPUs created in your project. If a TPU has been preempted, the status changes from READY to PREEMPTED.

For example:

NAME       ZONE           ACCELERATOR_TYPE  NETWORK_ENDPOINT   NETWORK  RANGE          STATUS
demo-tpu   us-central1-b  v2-8              10.240.1.2:8470    default  10.240.1.0/29  PREEMPTED

Note: If your Cloud TPU is preempted, you must delete it and create a new one as described in Managing TPUs.

Designing your machine learning application to run on preemptible TPUs

Make sure your application is resilient to restarts of the VM and TPU, by saving model checkpoints regularly and by configuring your application to restore the most recent checkpoint on restart.