Google Cloud Dataproc uses image versions to bundle operating system, big data components, and Google Cloud Platform connectors into one package that is deployed on a cluster. For more information, see Cloud Dataproc Versioning.
Supported Cloud Dataproc versions
The following versions are supported in Cloud Dataproc clusters. Note that new clusters will be created to include any sub-minor patches that have been made to a version since its release.
| Version | Includes | Released On | Last Updated (sub-minor version) |
Notes |
|---|---|---|---|---|
1.2 |
Apache Spark 2.2.1 Apache Hadoop 2.8.3 Apache Pig 0.16.0 Apache Hive 2.1.1* Cloud Storage connector 1.6.6-hadoop2 BigQuery connector 0.10.7-hadoop2 |
2017/07/18 | 2018/06/08 (1.2.38) |
This is the default version for new clusters beginning 2017/08/08. |
1.1 |
Apache Spark 2.0.2 Apache Hadoop 2.7.3 Apache Pig 0.16.0 Apache Hive 2.1.1* Cloud Storage connector 1.6.6-hadoop2 BigQuery connector 0.10.7-hadoop2 |
2016/08/08 | 2018/06/08 (1.1.74) |
This was the default version until 2017/08/08. |
1.0 |
Apache Spark 1.6.2 Apache Hadoop 2.7.3 Apache Pig 0.15.0 Apache Hive 1.2.1* Cloud Storage connector 1.6.6-hadoop2 BigQuery connector 0.10.7-hadoop2 |
2016/02/22 | 2018/06/08 (1.0.83) |
General availability release. |
Other versions
These versions are not officially supported, and are provided for testing or preview purposes. The subcomponent (Hadoop, Spark, Hive, Pig, etc.) versions listed below are subject to change.
| Version | Includes | Last Updated | Notes |
|---|---|---|---|
preview |
Apache Spark 2.3.0 Apache Hadoop 2.9.0 Apache Pig 0.17.0 Apache Hive 2.3.2 Apache Tez 0.9.0 Cloud Storage connector 1.8.1-hadoop2 |
2018/05/28 | Preview of next version but components in next stable version may differ. |
Unsupported Cloud Dataproc versions
The following versions are no longer supported in Cloud Dataproc clusters. You can create clusters with these image versions, however, these versions are no longer updated and are no longer supported. If you are using one of these versions you should consider upgrading to a more recent and supported version.
| Version | Includes | Released On | Last Updated | Notes |
|---|---|---|---|---|
0.2 |
Apache Spark 1.5.2 Apache Hadoop 2.7.1 Apache Pig 0.15.0 Apache Hive 1.2.1 Cloud Storage connector 1.5.1-hadoop2 BigQuery connector 0.7.7-hadoop2 |
2015/11/18 | 2016/08/02 | Beta image second release. |
0.1 |
Apache Spark 1.5.0 Apache Hadoop 2.7.1 Apache Pig 0.14.10 Apache Hive 1.0 Cloud Storage connector 1.5.1-hadoop2 BigQuery connector 0.7.7-hadoop2 |
2015/09/23 | 2016/08/02 | Cloud Dataproc beta release. Spark 1.5 has been compiled against Hive 1.2. |
