Supported Dataproc Serverless for Spark runtime versions
Version | Last Updated | Released On | Supported Until | Available Until | Notes |
---|---|---|---|---|---|
Spark runtime 2.2 LTS | 2024/08/22 | 2024/03/27 | 2026/09/30 | 2028/09/30 | General Availability release. This is the default version. |
Spark runtime 1.2 LTS | 2024/08/22 | 2024/03/27 | 2026/09/30 | 2028/09/30 | General Availability release. |
Spark runtime 1.1 LTS | 2024/08/22 | 2023/01/27 | 2025/07/31 | 2027/07/31 | General Availability release. |
How to choose a Dataproc Serverless for Spark runtime version
The current default Dataproc Serverless runtime version is listed in Supported Dataproc Serverless for Spark runtime versions. You can use the Google Cloud console, gcloud CLI, and the Dataproc API to choose a different runtime version when you submit a batch workload.
Console
- Open the Dataproc Create batch page in the Google Cloud console.
- Under Container, in the Runtime version section, select one of the listed Dataproc Serverless runtime versions.
gcloud
Use the dataproc batches submit --version
flag to specify a
Dataproc Serverless runtime version.
Example:
gcloud dataproc batches submit SPARK_WORKLOAD_TYPE \ --region=REGION \ --version=VERSION \ other args ...
Replace the following:
SPARK_WORKLOAD_TYPE: A Spark workload type, for example,
spark
.REGION: An available Compute Engine region, for example,
us-central1
.VERSION: Optional. A Dataproc Serverless runtime version, for example,
1.2
.
API
Use the RuntimeConfig.version
field as part of a batches.create
API call to specify a Dataproc Serverless runtime version.
Unsupported Dataproc Serverless for Spark runtime versions
The following Dataproc Serverless versions are unsupported.
Version | Includes | Released On | Last Updated | Available Until | Notes |
---|---|---|---|---|---|
Spark runtime 2.1 | Apache Spark 3.4.0 Cloud Storage connector 2.2.20 BigQuery connector 0.28.1 Java 17 Conda 23.3 Python 3.11 R 4.2 Scala 2.13 |
2022/01/19 | 2024/5/30 | 2026/04/30 | Unsupported as of 2024/05/30. 2.1.50 was the final released version. |
Spark runtime 2.0 | Apache Spark 3.3.4 Cloud Storage connector 2.2.20 Java 17 Conda 22.9 Python 3.10 R 4.1 Scala 2.13 |
2022/10/21 | 2024/08/22 | 2026/08/31 | Unsupported as of 2024/08/22. 2.0.85 was the final released version. |
Spark runtime 1.0 | Apache Spark 3.2.3 Cloud Storage connector 2.2.11 Conda 4.11 Java 11 Python 3.9 R 4.1 Scala 2.12.17 |
2022/01/19 | 2023/2/03 | 2025/01/31 | Unsupported as of 2023/02/03. 1.0.29 was the final released version. |