Manage TPU Spot VMs

Spot VMs make unused capacity available at highly-discounted rates. Spot VMs can be preempted (shut down) at any time, but unlike preemptible TPUs, there is no limit on runtime duration. You can't restart TPU Spot VMs, and you must recreate them after preemption. For more information about Spot VMs in general, see the Compute Engine documentation about Spot VMs.

Note: TPU Spot VMs only support the TPU VM architecture.

Create TPU Spot VMs

You can create TPU Spot VMs using queued resources, which adds your create request to a queue and lets you receive capacity once it becomes available. Creating TPUs as queued resources is a best practice. For more information, see Queued resources user guide.

Create TPU Spot VMs as queued resources by adding the --spot flag to the queued resources create command:

gcloud

gcloud alpha compute tpus queued-resources create QUEUED_RESOURCE_ID \
  --node-id=NODE_ID \
  --zone=europe-west4-a \
  --accelerator-type=v3-8 \
  --runtime-version=tpu-vm-tf-2.16.1-pod-pjrt \
  --spot

curl

curl -X POST https://tpu.googleapis.com/v2alpha1/projects/PROJECT_ID/locations/europe-west4-a/queuedResources?queued_resource_id=QUEUED_RESOURCE_ID \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{
    tpu: {
      node_spec: {
        parent: "projects/PROJECT_ID/locations/europe-west4-a",
        node_id: "NODE_ID",
        node: {
          accelerator_type: "v3-8",
          runtime_version: "tpu-vm-tf-2.16.1-pod-pjrt"
        }
      }
    },
    spot: {}
  }'

You can also create TPU Spot VMs without using queued resources by adding the --spot flag to the create command:

gcloud

gcloud compute tpus tpu-vm create TPU_NAME \
  --zone=europe-west4-a \
  --accelerator-type=v3-8 \
  --version=tpu-vm-tf-2.16.1-pod-pjrt \
  --spot

curl

curl -X POST https://tpu.googleapis.com/v2/projects/PROJECT_ID/locations/europe-west4-a/nodes?node_id=TPU_NAME \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -d '{
    accelerator_type: "v3-8",
    runtime_version: "tpu-vm-tf-2.16.1-pod-pjrt",
    network_config: {enable_external_ips: true},
    scheduling_config: {spot: true}
  }'

Check if a TPU VM is a Spot VM

To confirm if your TPU VM is a Spot VM, use the describe command:

gcloud compute tpus tpu-vm describe TPU_NAME --zone=europe-west4-a

If the TPU VM is a Spot VM, then the output will include the spot field set to true, similar to the following:

...
schedulingConfig:
  spot: true
...

Pricing and quota

Pricing for TPU Spot VMs is significantly lower than for on-demand and reserved TPUs. For more information about pricing, see Cloud TPU pricing.

You need preemptible quota to use TPU Spot VMs. For more information, see Quotas.