Dataproc is a managed Apache Spark and Apache Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning. Dataproc automation helps you create clusters quickly, manage them easily, and save money by turning clusters off when you don't need them. With less time and money spent on administration, you can focus on your jobs and your data. Learn more

Guides

Reference

Resources

Training and tutorials

Try Dataproc tutorials, courses, and self-paced training from Google Cloud Skills Boost.

Tutorial

Run a Spark job on Google Kubernetes Engine

Submit Spark jobs to a running Google Kubernetes Engine cluster from the Dataproc Jobs API.

Learn more  
Training

Introduction to Cloud Dataproc: Hadoop and Spark on Google Cloud

This course features a combination of lectures, demos, and hands-on labs to create a Dataproc cluster, submit a Spark job, and then shut down the cluster.

Learn more  
Training

Machine Learning with Spark on Dataproc

This course features a combination of lectures, demos, and hands-on labs to implement logistic regression using a machine learning library for Apache Spark running on a Dataproc cluster to develop a model for data from a multivariable dataset.

Learn more  

Use cases

Explore use cases, reference architectures, whitepapers, best practices, and industry solutions.

Solutions

Workflow scheduling solutions

Schedule workflows on Google Cloud.

Learn more  
Use Case

Migrate HDFS Data from On-Premises to Google Cloud

How to move data from on-premises Hadoop Distributed File System (HDFS) to Google Cloud.

Learn more  
Best Practice

Manage Java and Scala dependencies for Apache Spark

Recommended approaches to including dependencies when you submit a Spark job to a Dataproc cluster.

Learn more  
Best Practice

Using Apache Hive on Dataproc

Use Apache Hive on Dataproc in an efficient and flexible way by storing Hive data in Cloud Storage and hosting the Hive metastore in a MySQL database on Cloud SQL.

Learn more  

Code samples

Dive into coding with examples that demonstrate how to use and connect Google Cloud services.

github

Python API samples

Call Dataproc APIs from Python.

Open GitHub  

github

Java API samples

Call Dataproc APIs from Java.

Open GitHub  

github

Node.js API samples

Call Dataproc APIs from Node.js.

Open GitHub  

github

Go API samples

Call Dataproc APIs from Go.

Open GitHub  

Videos