Dataproc Serverless for Spark runtime versions

• Dataproc Serverless LTS (Long-Time-Support) runtime versions are supported for 30 months. Dataproc Serverless non-LTS runtime versions are supported for 12 months.
• Dataproc Serverless runtime versions continue to be available for two years after their end-of-support date.

Supported Dataproc Serverless for Spark runtime versions

Version	Last Updated	Released On	Supported Until	Available Until	Notes
Spark runtime 2.2 LTS	2024/10/31	2024/03/27	2026/09/30	2028/09/30	General Availability release. This is the default version.
Spark runtime 1.2 LTS	2024/10/31	2024/03/27	2026/09/30	2028/09/30	General Availability release.
Spark runtime 1.1 LTS	2024/10/31	2023/01/27	2025/07/31	2027/07/31	General Availability release.

How to choose a Dataproc Serverless for Spark runtime version

The current default Dataproc Serverless runtime version is listed in Supported Dataproc Serverless for Spark runtime versions. You can use the Google Cloud console, gcloud CLI, and the Dataproc API to choose a different runtime version when you submit a batch workload.

Note: Dataproc Serverless for Spark runtime versions 1.2 and 2.1+ don't support subminor version pinning.

Console

1. Open the Dataproc Create batch page in the Google Cloud console.
2. Under Container, in the Runtime version section, select one of the listed Dataproc Serverless runtime versions.

gcloud

Use the dataproc batches submit --version flag to specify a Dataproc Serverless runtime version.

Example:

gcloud dataproc batches submit SPARK_WORKLOAD_TYPE \
    --region=REGION \
    --version=VERSION \
    other args ...

Replace the following:

• SPARK_WORKLOAD_TYPE: A Spark workload type, for example, spark.

• REGION: An available Compute Engine region, for example, us-central1.

• VERSION: Optional. A Dataproc Serverless runtime version, for example, 1.2.

API

Use the RuntimeConfig.version field as part of a batches.create API call to specify a Dataproc Serverless runtime version.

Unsupported Dataproc Serverless for Spark runtime versions

The following Dataproc Serverless versions are unsupported.

Version	Includes	Released On	Last Updated	Available Until	Notes
Spark runtime 2.1	Apache Spark 3.4.0 Cloud Storage connector 2.2.20 BigQuery connector 0.28.1 Java 17 Conda 23.3 Python 3.11 R 4.2 Scala 2.13	2022/01/19	2024/5/30	2026/04/30	Unsupported as of 2024/05/30. 2.1.50 was the final released version.
Spark runtime 2.0	Apache Spark 3.3.4 Cloud Storage connector 2.2.20 Java 17 Conda 22.9 Python 3.10 R 4.1 Scala 2.13	2022/10/21	2024/08/22	2026/08/31	Unsupported as of 2024/08/22. 2.0.84 was the final released version.
Spark runtime 1.0	Apache Spark 3.2.3 Cloud Storage connector 2.2.11 Conda 4.11 Java 11 Python 3.9 R 4.1 Scala 2.12.17	2022/01/19	2023/2/03	2025/01/31	Unsupported as of 2023/02/03. 1.0.29 was the final released version.