This page explains how to create and use a preemptible virtual machine (VM) instance. A preemptible instance is an instance you can create and run at a much lower price than normal instances. However, Compute Engine might terminate (preempt) these instances if it requires access to those resources for other tasks. Preemptible instances will always terminate after 24 hours. To learn more about preemptible instances, read the preemptible instances documentation.
Preemptible instances are recommended only for fault-tolerant applications that can withstand instance preemptions. Make sure your application can handle preemptions before you decide to create a preemptible instance. To understand the risks and value of preemptible instances, read the preemptible instances documentation.
Before you begin
- If you want to use the command-line examples in this guide:
- Install or update to the latest version of the gcloud command-line tool.
- Set a default region and zone.
- If you want to use the API examples in this guide, set up API access.
- Read the Preemptible VM instances documentation.
Creating a preemptible instance
Create a preemptible instance through the Google Cloud Console, the
gcloud
tool, or the
API.
Console
Creating a preemptible instance is the same as creating a normal instance,
but you enable the preemptible
property.
- In the Cloud Console, go to the VM Instances page.
- Click Create instance.
- On the Create a new instance page, fill in the properties for your instance.
- Click Management, security, disks, networking, sole tenancy.
- Under Availability policy, set the Preemptibility option to On. This setting disables automatic restart for the instance, and sets the host maintenance action to Terminate.
- Click Create to create the instance.
gcloud
With gcloud compute
, use the same instances create
command that
you would use to create a normal instance, but add the --preemptible
flag.
gcloud compute instances create [INSTANCE_NAME] --preemptible
where [INSTANCE_NAME]
is the name of the instance.
API
In the API, construct a normal request to
create an instance,
but include the
preemptible
property under scheduling
and set it to true
. For example:
POST https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances
{
'machineType': 'zones/[ZONE]/machineTypes/[MACHINE_TYPE]',
'name': '[INSTANCE_NAME]',
'scheduling':
{
'preemptible': true
},
...
}
Preemptible CPU quotas
Preemptible instances require available CPU quotas like regular instances. To avoid preemptible instances consuming the CPU quotas for your regular instances, you can request a special "Preemptible CPU" quota. After Compute Engine grants you preemptible CPU quota in that region, all preemptible instances will count against that quota. All regular instances will continue to count against the regular CPU quota.
In regions where you don’t have preemptible CPU quota, you can use regular CPU
quota to launch preemptible instances. You will also need sufficient IP and
disk quota, as usual. Preemptible CPU quota is not visible in the gcloud
tool or
Cloud Console quota pages unless Compute Engine has granted
the quota.
For more information about quotas, visit the Resource Quotas page.
Handling preemption with a shutdown script
When your instance is preempted, you can use a shutdown script to perform cleanup actions before the instance stops. For example, you can gracefully terminate a running process and copy a checkpoint file to Cloud Storage.
The following is a shutdown script that you can add to a running preemptible
instance or add to a new preemptible instance when you create it. This script
runs when the instance starts to shut down, before the operating
system's normal kill
command terminates all remaining processes. After
gracefully terminating the desired program, the script performs a parallel
upload of a checkpoint file to a Google Cloud Storage bucket.
#!/bin/bash
MY_PROGRAM="[PROGRAM_NAME]" # For example, "apache2" or "nginx"
MY_USER="[LOCAL_USERNAME]"
CHECKPOINT="/home/$MY_USER/checkpoint.out"
GSUTIL_OPTS="-m -o GSUtil:parallel_composite_upload_threshold=32M"
BUCKET_NAME="[BUCKET_NAME]" # For example, "my-checkpoint-files" (without gs://)
echo "Shutting down! Seeing if ${MY_PROGRAM} is running."
# Find the newest copy of $MY_PROGRAM
PID="$(pgrep -n "$MY_PROGRAM")"
if [[ "$?" -ne 0 ]]; then
echo "${MY_PROGRAM} not running, shutting down immediately."
exit 0
fi
echo "Sending SIGINT to $PID"
kill -2 "$PID"
# Portable waitpid equivalent
while kill -0 "$PID"; do
sleep 1
done
echo "$PID is done, copying ${CHECKPOINT} to gs://${BUCKET_NAME} as ${MY_USER}"
su "${MY_USER}" -c "gsutil $GSUTIL_OPTS cp $CHECKPOINT gs://${BUCKET_NAME}/"
echo "Done uploading, shutting down."
To add this script to an instance, configure the script to work with an application on your instance and add it to your instance metadata.
- Copy or download the shutdown script to your local workstation.
- Open the file for edit and change the following variables:
[PROGRAM_NAME]
is the name of the process or program you want to shut down. For example,apache2
ornginx
.[LOCAL_USER]
is the username you are logged into the virtual machine as.[BUCKET_NAME]
is the name of the Cloud Storage bucket where you want to save the program's checkpoint file. Note the bucket name does not start withgs://
in this case.
- Save your changes.
- Add the shutdown script to a new instance or an existing instance.
This script assumes:
The instance was created with at least read-write access to Cloud Storage. See the authentication documentation for instructions about how to create an instance with the appropriate scopes.
You have an existing Cloud Storage bucket and permission to write to it.
Checking if an instance is preemptible
You can check if an instance is configured to be preemptible using the
Cloud Console, the
gcloud
tool, or the
API.
Console
Check if an instance is preemptible by viewing the instance properties.
- Go to the VM instances page.
- Select your project and click Continue.
- Click the name of the instance that you want to check. This opens the instance details page.
- The preemptible status is specified in the Availability policies section of the instance details.
gcloud
In gcloud compute
, use instances describe
to get information
about an instance, including whether the instance is preemptible.
gcloud compute instances describe [INSTANCE_NAME]
where [INSTANCE_NAME]
is the name of the instance.
The response information includes the preemptible status in the scheduling section.
... scheduling: automaticRestart: false onHostMaintenance: TERMINATE preemptible: true ...
API
To check if an instance is preemptible, use the API to send a GET
request to the URI of the instance.
GET https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances/[INSTANCE_NAME]
The response information includes the preemptible status under scheduling
.
{ "kind": "compute#instance", "id": "4468501694759003918", "creationTimestamp": "2015-04-15T15:40:59.004-07:00", "zone": "https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-f", "status": "RUNNING", "name": "example-instance", "scheduling": { "preemptible": true }, ... }
Alternatively, you can determine if an instance is preemptible from inside the
instance itself. Simply check the metadata server for the
scheduling/preemptible
value in your instance's
default instance metadata.
For example, use curl
from within your instance to obtain the value for
scheduling/preemptible
:
curl "http://metadata.google.internal/computeMetadata/v1/instance/scheduling/preemptible" -H "Metadata-Flavor: Google"
TRUE
If this value is TRUE
, the instance is preemptible.
Detecting if an instance was preempted
Determine if an instance was preempted with the
Google Cloud Console, the
gcloud
tool, or the
API.
Console
You can check if an instance was preempted by checking the system activity
logs.
- Go to the Logs page.
- Select your project and click Continue.
- Add
compute.instances.preempted
to the filter by label or text search field. - Optionally, you can also enter an instance name if you want to see preemption operations for a specific instance.
- Press enter to apply the specified filters. The Cloud Console updates the list of logs to show only the operations where an instance was preempted.
- Select an operation in the list to see details about the instance that was preempted.
gcloud
Use the gcloud compute operations list
command with a filter parameter to
get a list of preemption events in your project.
gcloud compute operations list \
--filter="operationType=compute.instances.preempted"
You can use the filter param to further scope the results. For example, to see preemption events only for instances within a managed instance group:
gcloud compute operations list \
--filter="operationType=compute.instances.preempted AND targetLink:instances/[BASE_INSTANCE_NAME]"
gcloud
returns a response similar to:
NAME TYPE TARGET HTTP_STATUS STATUS TIMESTAMP systemevent-xxxxxxxx compute.instances.preempted us-central1-f/instances/example-instance-xxx 200 DONE 2015-04-02T12:12:10.881-07:00
An operation type of compute.instances.preempted
indicates that the
instance was preempted. You can use the operations describe
command to get
more information about a specific preemption operation.
gcloud compute operations describe \
systemevent-xxxxxxxx
gcloud
returns a response similar to:
... operationType: compute.instances.preempted progress: 100 selfLink: https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-f/operations/systemevent-xxxxxxxx startTime: '2015-04-02T12:12:10.881-07:00' status: DONE statusMessage: Instance was preempted. ...
API
To get a list of recent system operations, send a GET
request to the URI
of zone operations.
GET https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/operations
The response contains a list of recent operations.
{ "kind": "compute#operation", "id": "15041793718812375371", "name": "systemevent-xxxxxxxx", "zone": "https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-f", "operationType": "compute.instances.preempted", "targetLink": "https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/us-central1-f/instances/example-instance", "targetId": "12820389800990687210", "status": "DONE", "statusMessage": "Instance was preempted.", ... }
To scope the response to show only preemption operations, you can add a
filter to your API request:
operationType="compute.instances.preempted"
. To see preemption operations
for a specific instance, add a targetLink
param to the filter:
operationType="compute.instances.preempted" AND
targetLink="https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances/[INSTANCE_NAME]"
.
Alternatively, you can determine if an instance was preempted from inside the
instance itself. This is useful if you want to handle a shutdown due to a
Compute Engine preeemption differently from a normal
shutdown in a shutdown script. To do this, simply check
the metadata server for the preempted
value in your instance's
default instance metadata.
For example, use curl
from within your instance to obtain the value for
preempted
:
curl "http://metadata.google.internal/computeMetadata/v1/instance/preempted" -H "Metadata-Flavor: Google"
TRUE
If this value is TRUE
, the instance was preempted by Compute Engine,
otherwise it will be FALSE
.
If you want to use this outside of a shutdown script, you can append ?wait_for_change=true to the URL. This performs a hanging HTTP GET request that will only return when the metadata has changed and the instance has been preempted.
curl "http://metadata.google.internal/computeMetadata/v1/instance/preempted?wait_for_change=true" -H "Metadata-Flavor: Google"
TRUE
Best practices
Here are some best practices to help you get the most out of preemptible VM instances.
Pick smaller machine shapes
Resources for preemptible VM instances come out of excess and backup Google Cloud capacity. It's often easier to get lots of preemptible capacity with smaller machine types than larger ones. The preemption rates for smaller machine types with less than 32 cores are also historically lower than for larger machine types.
You might also get more spare capacity by using a custom machine type that is in between the predefined types. For example, there's likely more capacity for a custom machine type with 48 vCPUs than there are n1-standard-64s.
Run large preemptible VM clusters during off peak times
The load on Google Cloud data centers varies with location and time of day, but generally lowest on nights and weekends. As such, nights and weekends are the best times to run large preemptible VM clusters.
Design your applications to be fault and preemption tolerant
It's important to be prepared for the fact that there will be changes in preemption patterns at different points in time. For example, it is possible that if a zone suffers a partial outage, large numbers of preemptible instances could be preempted in order to make room for regular instances that need to be moved as part of the recovery. In that small window of time, the preemption rate would look very different than on any other day. If your application assumes that preemptions will always be done in small groups, you might not be prepared for such an event. You can test your application's behavior under a preemption event by stopping the VM instance.
Retry creating instances that have been preempted
If your VM instance been preempted, try creating new preemptible instances once or twice before falling back to regular instances. Depending on your requirements, it might be a good idea to combine regular and preemptible instances in your clusters to ensure that work proceeds at an adequate pace.
Use shutdown scripts
Manage shutdown and preemption notices with a shutdown script that can save a job's progress so that it can pick up where it left off, rather than start over from scratch.
What's next?
- Read the Preemptible VM instances documentation.
- Read about Shutdown scripts.
- Refer to the preemptible instance pricing.
- Connect to your instance.