Cluster Scheduled Deletion

To help avoid incurring Google Cloud Platform charges for an inactive cluster, use Cloud Dataproc's Cluster Scheduled Deletion feature when you create a cluster. This feature provides options to delete a cluster:

  • after a specified cluster idle period
  • at a specified future time
  • after a specified period that starts from the time of submission of the cluster creation request

Using Cluster Scheduled Deletion

gcloud command

You can create a cluster with the Cluster Scheduled Deletion feature by passing the following scheduled deletion flags to the /sdk/gcloud/reference/beta/dataproc/clusters/create command.

FlagDescriptionFinest GranularityMin ValueMax Value
--max-idle1The duration from the moment when the cluster enters the idle state to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" or "1d" (30 minutes or 1 day from when the cluster becomes idle).1 second10 minutes14 days
--expiration-time2The time to start deleting the cluster in ISO 8601 datetime format. An easy way to generate the datetime in correct format is through the Timestamp Generator. For example, "2017-08-22T13:31:48-08:00" specifies an expiration time of 13:21:48 in the UTC -8:00 time zone.1 second10 minutes from the current time 14 days from the current time
--max-age2The duration from the moment of submitting the cluster create request to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" (30 minutes from now); "1d" (1 day from now).1 second10 minutes14 days
gcloud beta dataproc clusters create cluster-name \
    ‑‑max-idle=duration \ 
    ‑‑expiration-time=time \ 
    ... other args ...

You can update a cluster that was created with the scheduled deletion feature to change or remove scheduled deletion settings by passing the following scheduled deletion flags to the gcloud beta dataproc clusters update command (other cluster update flags cannot be combined with scheduled deletion flags).

FlagDescriptionFinest GranularityMin ValueMax Value
--max-idle1The duration from the moment when the cluster enters the idle state to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" or "1d" (30 minutes or 1 day from when the cluster becomes idle).1 second10 minutes14 days
--no-max-idleCancels cluster auto-deletion by cluster idle duration previously set by the max-idle flagnot applicablenot applicablenot applicable
--expiration-time2The time to start deleting the cluster in ISO 8601 datetime format. An easy way to generate the datetime in correct format is through the Timestamp Generator. For example, "2017-08-22T13:31:48-08:00" specifies an expiration time of 13:21:48 in the UTC -8:00 time zone.1 second10 minutes from the current time, and the new time must not be earlier than the previously set time.14 days from the current time
--max-age2The duration from the moment of submitting the cluster update request to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" (30 minutes from now); "1d" (1 day from now).1 second10 minutes, and the updated scheduled deletion time (update time + new max-age duration) must not be earlier than the previously set cluster deletion time.14 days
--no-max-ageCancels cluster auto-deletion by maximum cluster age previously set by the max-age or expiration-time flagnot applicablenot applicablenot applicable
gcloud beta dataproc clusters update cluster-name \ 
    ‑‑max-idle=duration \ 
    --no-max-age

REST API

Creating a Cluster with Scheduled Deletion

See GceClusterConfig.ClusterLifecycleConfig.

You can set the following fields in ClusterLifecycleConfig in your cluster.create or cluster.patch request.

Flag Description Finest Granularity Min Value Max Value
idleDeleteTtl1 The duration from the moment when the cluster enters the idle state to the moment when the cluster starts to delete. Provide a duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". 1 second 10 minutes from the time of creating or updating the cluster. When updating a cluster, the new value must be greater than the previously set value. Submit a cluster.patch request with an empty duration to cancel a previously set idleDeleteTtl value. 14 days
autoDeleteTime2 The time to start deleting the cluster. Provide a timestamp in RFC 3339 UTC "Zulu" format, accurate to nanoseconds. Example: "2014-10-02T15:01:23.045123456Z". 1 second 10 minutes from the current time. When updating a cluster, the new time must be later than the previously set time. 14 days from the current time
autoDeleteTtl2 The duration from the moment of submitting the cluster create or update request to the moment when the cluster starts to delete. Provide a duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". 1 second 10 minutes. When updating a cluster, the new scheduled deletion time (update time + new max-age duration) must be later than the previously set cluster deletion time. Submit a cluster.patch request with an empty duration to cancel a previously set autoDeleteTtl value. 14 days

Console

Currently, Cluster Scheduled Deletion is not supported in the Google Cloud Platform Console.

Viewing Scheduled Deletion cluster settings

gcloud command

You can use the gcloud beta clusters list command to confirm that a cluster has scheduled deletion enabled.

 gcloud beta dataproc clusters list
...
NAME         WORKER_COUNT ... SCHEDULED_DELETE
cluster-id   number       ... enabled
...

You can use the gcloud beta clusters describe command to check a cluster's LifecycleConfig scheduled deletion settings.

gcloud beta dataproc clusters describe cluster-name
...
lifecycleConfig:
  autoDeleteTime: '2018-11-28T19:33:48.146Z'
  idleDeleteTtl: 1800s
  idleStartTime: '2018-11-28T18:33:48.146Z'
...

The autoDeleteTime and idleDeleteTtl are the scheduled deletion configuration values previously set by the user on the cluster. Cloud Dataproc generates the idleStartTime value, which is the latest cluster idle start time. Cloud Dataproc deletes the cluster if the cluster remains idle at idleStartTime + idleDeleteTtl.

REST API

You can make a clusters.list request to confirm that a cluster has scheduled deletion enabled.

Console

Currently, Cluster Scheduled Deletion is not supported in the Google Cloud Platform Console.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataproc Documentation