Dataproc Serverless lets you run Spark batch workloads without requiring you to provision and manage your own cluster. Specify workload parameters, and then submit the workload to the Dataproc Serverless service. The service will run the workload on a managed compute infrastructure, autoscaling resources as needed. Dataproc Serverless charges apply only to the time when the workload is executing.
Dataproc Serverless for Spark compared to Dataproc on Compute Engine
Dataproc on Compute Engine is ideal for users who want to provision and manage infrastructure, then execute workloads on Spark and other open source processing frameworks. The following table list key differences between the Dataproc on Compute Engine and Dataproc Serverless for Spark.
|Capability||Dataproc Serverless for Spark||Dataproc on Compute Engine|
|Processing frameworks||Spark 3.2||Spark 3.1 and earlier versions. Other open source frameworks, such as, Hive|
|Resource management||Spark based||YARN based|
|Interactive sessions||Planned (Google managed)||Yes (customer managed)|
|VM access (for example, SSH)||No||Yes|
|Java versions||Java 11||Previous versions supported|
- An OS Login policy is not applicable to or supported by Dataproc Serverless.
If your organization enforces an
OS Loginpolicy, its Dataproc Serverless workloads will fail.
Dataproc Serverless for Spark workload capabilities
You can run the following Spark workload types on the Dataproc Serverless for Spark service:
- Spark SQL
- Spark R
- Spark Java/Scala
- You can specify the Spark properties when you submit a Spark batch workload.