Google Dataproc uses Ubuntu, Debian, and Rocky Linux image versions to bundle operating system, big data components, and Google Cloud Platform connectors into one package that is deployed on a cluster. For more information, see Dataproc Versioning.
Default Dataproc image version
Dataproc updates the default image version to the latest generally available Debian-based Dataproc image version one month after its General availability (GA) release date.
Supported Dataproc image versions
Debian images
The following Debian-based image versions are supported in Dataproc clusters. Note that new clusters will be created to include any sub-minor patches that have been made to a version since its release.
Version | Last Updated | Released On | Supported Until | Available Until | Notes |
---|---|---|---|---|---|
2.2-debian12 | 2024/08/19 | 2023/12/08 | 2025/12/31 | 2027/12/31 | General availability release. |
2.1-debian11 | 2024/08/19 | 2022/12/12 | 2024/12/31 | 2026/12/31 | General availability release. |
2.0-debian10 | 2024/08/19 | 2021/01/22 | 2024/07/31 | 2026/07/31 | General availability release. |
Ubuntu images
The following Ubuntu LTS-based image versions are supported in Dataproc clusters. Note that new clusters will be created to include any sub-minor patches that have been made to a version since its release.
Version | Last Updated | Released On | Supported Until | Available Until | Notes |
---|---|---|---|---|---|
2.2-ubuntu22 | 2024/08/19 | 2023/12/08 | 2025/12/31 | 2027/12/31 | General availability release. |
2.1-ubuntu20 | 2024/08/19 | 2022/12/12 | 2024/12/31 | 2026/12/31 | General availability release. |
2.0-ubuntu18 | 2024/08/19 | 2021/01/22 | 2024/07/31 | 2026/07/31 | General availability release. |
Rocky Linux images
The following Rocky Linux-based image versions are supported in Dataproc clusters. Note that new clusters will be created to include any sub-minor patches that have been made to a version since its release.
Version | Last Updated | Released On | Supported Until | Available Until | Notes |
---|---|---|---|---|---|
2.2-rocky9 | 2024/08/19 | 2023/12/08 | 2025/12/31 | 2027/12/31 | General availability release. |
2.1-rocky8 | 2024/08/19 | 2022/12/12 | 2024/12/31 | 2026/12/31 | General availability release. |
2.0-rocky8 | 2024/08/19 | 2022/02/18 | 2024/07/31 | 2026/07/31 | General availability release. |
Unsupported Dataproc image versions
The following Dataproc versions are unsupported. Dataproc does not provide updates and support for clusters created with these versions. Although you can continue running a cluster that was created with an unsupported version, replacing the cluster with a new cluster that is created with a supported version is recommended.
Version | Includes | Released On | Last Updated | Notes |
---|---|---|---|---|
1.5-debian10/-ubuntu18/-rocky8 |
Apache Spark 2.4.8 Apache Hadoop 2.10.2 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 2.1.9-hadoop2 Python 3.7 Scala 2.12.10 Zookeeper 3.4.14 |
2020/03/25:debian10/ubuntu18 2022/02/18:rocky8 |
2023/04/28 | Unsupported as of 2023/04/28. 1.5.89-debian10/-ubuntu18/-rocky8 was the final released version. |
2.0-centos8 |
Apache Spark 3.1.2 Apache Hadoop 3.2.2 Apache Pig 0.18.0-SNAPSHOT Apache Hive 3.1.2 Cloud Storage connector 2.2.4-hadoop3 Python 3.8 Scala 2.12.14 Zookeeper 3.4.14 |
2021/03/16 | 2022/02/01 | Unsupported as of 2022/02/01. 2.0.30-centos8 was the final released version. |
1.5-centos8 |
Apache Spark 2.4.8 Apache Hadoop 2.10.1 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 2.1.5-hadoop2 Python 3.7 Scala 2.12.10 Zookeeper 3.4.14 |
2020/12/14 | 2022/02/01 | Unsupported as of 2022/02/01. 1.5.56-centos8 was the final released version. |
1.4-debian10/-ubuntu18 |
Apache Spark 2.4.8 Apache Hadoop 2.9.2 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 1.9.18-hadoop2 Python 3.6 Scala 2.11.12 Zookeeper 3.4.14 |
2019/03/22 | 2022/02/01 | Unsupported as of 2022/02/01. 1.4.80-debian10/-ubuntu18 was the final released version. |
1.3-debian10/-ubuntu18 |
Apache Spark 2.3.4 Apache Hadoop 2.9.2 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 1.9.18-hadoop2 Python 2.7 Scala 2.11.8 Zookeeper 3.4.13 |
2018/06/29 | 2021/12/22 | Unsupported as of 2021/08/01. 1.3.95-debian10/-ubuntu18 was the final released version, which has log4j2 vulnerabilities addressed. Note: previously released versions are vulnerable and must be upgraded. |
1.4-debian9 |
Apache Spark 2.4.5 Apache Hadoop 2.9.2 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 1.9.17-hadoop2 Python 3.6 Scala 2.11.12 Zookeeper 3.4.13 |
2019/03/22 | 2020/07/10 | Unsupported as of 2020/07/10. 1.4.33-debian9 was the final released version. |
1.3-debian9 |
Apache Spark 2.3.4 Apache Hadoop 2.9.2 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 1.9.17-hadoop2 Python 2.7 Scala 2.11.8 Zookeeper 3.4.13 |
2018/06/29 | 2020/07/10 | Unsupported as of 2020/07/10. 1.3.62-debian9 was the final released version. |
1.2-debian9 |
Apache Spark 2.2.3 Apache Hadoop 2.8.5 Apache Pig 0.16.0 Apache Hive 2.1.1 Cloud Storage connector 1.6.10-hadoop2 BigQuery connector 0.10.11-hadoop2 Python 2.7 Scala 2.11.8 Zookeeper 3.4.13 |
2017/07/21 | 2020/07/10 | Unsupported as of 2020/07/10. 1.2.102-debian9 was the final released version. |
1.1-debian9 |
Apache Spark 2.0.2 Apache Hadoop 2.7.7 Apache Pig 0.16.0 Apache Hive 2.1.1 Cloud Storage connector 1.6.10-hadoop2 BigQuery connector 0.10.11-hadoop2 |
2016/08/08 | 2019/09/26 | Unsupported as of 2019/10/01. 1.1.121-debian9 is the final released version. |
1.0-debian9 |
Apache Spark 1.6.2 Apache Hadoop 2.7.4 Apache Pig 0.15.0 Apache Hive 1.2.1 Cloud Storage connector 1.6.10-hadoop2 BigQuery connector 0.10.11-hadoop2 |
2016/02/22 | 2019/05/09 | GA image first release. Unsupported as of 2019/04/01. 1.0.119-debian9 was the final released version. |
0.2 |
Apache Spark 1.5.2 Apache Hadoop 2.7.1 Apache Pig 0.15.0 Apache Hive 1.2.1 Cloud Storage connector 1.5.1-hadoop2 BigQuery connector 0.7.7-hadoop2 |
2015/11/18 | 2016/08/02 | Beta image second release. |
0.1 |
Apache Spark 1.5.0 Apache Hadoop 2.7.1 Apache Pig 0.14.10 Apache Hive 1.0 Cloud Storage connector 1.5.1-hadoop2 BigQuery connector 0.7.7-hadoop2 |
2015/09/23 | 2016/08/02 | Dataproc beta release. Spark 1.5 has been compiled against Hive 1.2. |