Dataproc Serverless staging buckets

Dataproc Serverless creates a Cloud Storage staging bucket in your project or reuses an existing staging bucket from previous batch creation requests. Note that this is the default bucket created by Dataproc on Compute Engine clusters (see Dataproc staging and temp buckets).

• Dataproc Serverless stores workload dependencies, config files, and job driver console output in the staging bucket.

• Dataproc Serverless sets regional staging buckets in Cloud Storage locations according to the Compute Engine zone where your workload is deployed, and then creates and manages these project-level, per-location buckets. Dataproc Serverless-created staging buckets are shared among workloads in the same region, and are created with a Cloud Storage soft delete retention duration set to 0 seconds.

To locate the Dataproc default staging bucket using the Google Cloud console Cloud Storage Browser, filter results using the "dataproc-staging-" prefix.