Cluster Scheduled Deletion

To help avoid incurring Google Cloud Platform charges for an inactive cluster, use Cloud Dataproc's Cluster Scheduled Deletion feature when you create a cluster. This feature provides options to delete a cluster:

  • after a specified cluster idle period
  • at a specified future time
  • after a specified period that starts from the time of submission of the cluster creation request

Using Cluster Scheduled Deletion

gcloud command

You can create a cluster with the Cluster Scheduled Deletion feature using the gcloud beta dataproc clusters create command.

You can pass the following flags to the create command:

FlagDescriptionFinest GranularityMin ValueMax Value
--max-idle1The duration from the moment when the cluster enters the idle state to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" or "1d" (30 minutes or 1 day from when the cluster becomes idle).1 second10 minutes14 days
--expiration-time2The time to start deleting the cluster in ISO 8601 datetime format. An easy way to generate the datetime in correct format is through the Timestamp Generator. For example, "2017-08-22T13:31:48-08:00" specifies an expiration time of 13:21:48 in the UTC -8:00 time zone.1 second10 minutes from the current time 14 days from the current time
--max-age2The duration from the moment of submitting the cluster create request to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" (30 minutes from now); "1d" (1 day from now).1 second10 minutes14 days
gcloud beta dataproc clusters create \
  ‑‑max-idle=duration
  ‑‑expiration-time=time \
  other args ...


You can update a cluster that was created with the scheduled deletion feature to change or remove scheduled deletion settings using the gcloud beta dataproc clusters update command.

You can pass the following scheduled deletion flags to the update command (other cluster update flags cannot be combined with scheduled deletion flags):

FlagDescriptionFinest GranularityMin ValueMax Value
--max-idle1The duration from the moment when the cluster enters the idle state to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" or "1d" (30 minutes or 1 day from when the cluster becomes idle).1 second10 minutes14 days
--no-max-idleCancels cluster auto-deletion by cluster idle duration previously set by the max-idle flagnot applicablenot applicablenot applicable
--expiration-time2The time to start deleting the cluster in ISO 8601 datetime format. An easy way to generate the datetime in correct format is through the Timestamp Generator. For example, "2017-08-22T13:31:48-08:00" specifies an expiration time of 13:21:48 in the UTC -8:00 time zone.1 second10 minutes from the current time, and the new time must not be earlier than the previously set time.14 days from the current time
--max-age2The duration from the moment of submitting the cluster update request to the moment when the cluster starts to delete. Provide the duration in IntegerUnit format, where the unit can be "s, m, h, d" (seconds, minutes, hours, days, respectively). Examples: "30m" (30 minutes from now); "1d" (1 day from now).1 second10 minutes, and the updated scheduled deletion time (update time + new max-age duration) must not be earlier than the previously set cluster deletion time.14 days
--no-max-ageCancels cluster auto-deletion by maximum cluster age previously set by the max-age or expiration-time flagnot applicablenot applicablenot applicable
gcloud beta dataproc clusters update \
  ‑‑max-idle=duration
  --no-max-age

REST API

See GceClusterConfig.ClusterLifecycleConfig.

You can set the following fields in ClusterLifecycleConfig in your cluster.create or cluster.patch request.

Flag Description Finest Granularity Min Value Max Value
idleDeleteTtl1 The duration from the moment when the cluster enters the idle state to the moment when the cluster starts to delete. Provide a duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". 1 second 10 minutes from the time of creating or updating the cluster. When updating a cluster, the new value must be greater than the previously set value. Submit a cluster.patch request with an empty duration to cancel a previously set idleDeleteTtl value. 14 days
autoDeleteTime2 The time to start deleting the cluster. Provide a timestamp in RFC 3339 UTC "Zulu" format, accurate to nanoseconds. Example: "2014-10-02T15:01:23.045123456Z". 1 second 10 minutes from the current time. When updating a cluster, the new time must be later than the previously set time. 14 days from the current time
autoDeleteTtl2 The duration from the moment of submitting the cluster create or update request to the moment when the cluster starts to delete. Provide a duration in seconds with up to nine fractional digits, terminated by 's'. Example: "3.5s". 1 second 10 minutes. When updating a cluster, the new scheduled deletion time (update time + new max-age duration) must be later than the previously set cluster deletion time. Submit a cluster.patch request with an empty duration to cancel a previously set autoDeleteTtl value. 14 days

Console

Currently, Cluster Scheduled Deletion is not supported in the Google Cloud Platform Console.

Was this page helpful? Let us know how we did:

Send feedback about...

Cloud Dataproc Documentation