Google Dataproc uses image versions to bundle operating system, big data components, and Google Cloud Platform connectors into one package that is deployed on a cluster. For more information, see Dataproc Versioning.
Supported Dataproc versions
Debian images
The following Debian 10-based image versions are supported in Dataproc clusters. Note that new clusters will be created to include any sub-minor patches that have been made to a version since its release.
| Version | Last Updated | Released On | Notes |
|---|---|---|---|
| preview-debian10 | 2020/11/16 | 2020/06/11 | Preview release. |
| 1.5-debian10 | 2020/11/16 | 2020/03/25 | General availability release. |
| 1.4-debian10 | 2020/11/16 | 2019/03/22 | General availability release. |
| 1.3-debian10 | 2020/11/16 | 2018/06/29 | General availability release. |
Ubuntu images
The following Ubuntu 18.04 LTS-based image versions are supported in Dataproc clusters. Note that new clusters will be created to include any sub-minor patches that have been made to a version since its release.
| Version | Last Updated | Released On | Notes |
|---|---|---|---|
| preview-ubuntu18 | 2020/11/16 | 2020/06/11 | Preview release. |
| 1.5-ubuntu18 | 2020/11/16 | 2020/03/25 | General availability release. |
| 1.4-ubuntu18 | 2020/11/16 | 2019/03/22 | General availability release. |
| 1.3-ubuntu18 | 2020/11/16 | 2019/03/22 | General availability release. |
Unsupported Dataproc versions
The following Dataproc versions are unsupported. Dataproc does not provide updates and support for clusters created with these versions. Although you can continue running a cluster that was created with an unsupported version, replacing the cluster with a new cluster that is created with a supported version is recommended.
| Version | Includes | Released On | Last Updated | Notes |
|---|---|---|---|---|
1.4-debian9 |
Apache Spark 2.4.5 Apache Hadoop 2.9.2 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 1.9.17-hadoop2 Python 3.6 Scala 2.11.12 Zookeeper 3.4.13 |
2019/03/22 | 2020/07/10 | Unsupported as of 2020/07/10. 1.4.33-debian9 was the final released version. |
1.3-debian9 |
Apache Spark 2.3.4 Apache Hadoop 2.9.2 Apache Pig 0.17.0 Apache Hive 2.3.7 Cloud Storage connector 1.9.17-hadoop2 Python 2.7 Scala 2.11.8 Zookeeper 3.4.13 |
2018/06/29 | 2020/07/10 | Unsupported as of 2020/07/10. 1.3.62-debian9 was the final released version. |
1.2-debian9 |
Apache Spark 2.2.3 Apache Hadoop 2.8.5 Apache Pig 0.16.0 Apache Hive 2.1.1 Cloud Storage connector 1.6.10-hadoop2 BigQuery connector 0.10.11-hadoop2 Python 2.7 Scala 2.11.8 Zookeeper 3.4.13 |
2017/07/21 | 2020/07/10 | Unsupported as of 2020/07/10. 1.2.102-debian9 was the final released version. |
1.1-debian9 |
Apache Spark 2.0.2 Apache Hadoop 2.7.7 Apache Pig 0.16.0 Apache Hive 2.1.1 Cloud Storage connector 1.6.10-hadoop2 BigQuery connector 0.10.11-hadoop2 |
2016/08/08 | 2019/09/26 | Unsupported as of 2019/10/01. 1.1.121-debian9 is the final released version. |
1.0-debian9 |
Apache Spark 1.6.2 Apache Hadoop 2.7.4 Apache Pig 0.15.0 Apache Hive 1.2.1 Cloud Storage connector 1.6.10-hadoop2 BigQuery connector 0.10.11-hadoop2 |
2016/02/22 | 2019/05/09 | GA image first release. Unsupported as of 2019/04/01. 1.0.119-debian9 was the final released version. |
0.2 |
Apache Spark 1.5.2 Apache Hadoop 2.7.1 Apache Pig 0.15.0 Apache Hive 1.2.1 Cloud Storage connector 1.5.1-hadoop2 BigQuery connector 0.7.7-hadoop2 |
2015/11/18 | 2016/08/02 | Beta image second release. |
0.1 |
Apache Spark 1.5.0 Apache Hadoop 2.7.1 Apache Pig 0.14.10 Apache Hive 1.0 Cloud Storage connector 1.5.1-hadoop2 BigQuery connector 0.7.7-hadoop2 |
2015/09/23 | 2016/08/02 | Dataproc beta release. Spark 1.5 has been compiled against Hive 1.2. |