View product documentation

Dataproc is a managed Apache Spark and Apache Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning. Dataproc automation helps you create clusters quickly, manage them easily, and save money by turning clusters off when you don't need them. With less time and money spent on administration, you can focus on your jobs and your data. Learn more

Get started for free

Start your next project with $300 in free credit

Build and test a proof of concept with the free trial credits and free monthly usage of 20+ products.

Documentation resources

Find quickstarts and guides, review key references, and get help with common issues.
Explore self-paced training from Google Cloud Skills Boost, use cases, reference architectures, and code samples with examples of how to use and connect Google Cloud services.
Training
Training and tutorials

Run a Spark job on Google Kubernetes Engine

Submit Spark jobs to a running Google Kubernetes Engine cluster from the Dataproc Jobs API.

Learn more  
Training
Training and tutorials

Introduction to Cloud Dataproc: Hadoop and Spark on Google Cloud

This course features a combination of lectures, demos, and hands-on labs to create a Dataproc cluster, submit a Spark job, and then shut down the cluster.

Learn more  
Training
Training and tutorials

Machine Learning with Spark on Dataproc

This course features a combination of lectures, demos, and hands-on labs to implement logistic regression using a machine learning library for Apache Spark running on a Dataproc cluster to develop a model for data from a multivariable dataset.

Learn more  

Related videos

🧠 Problem: Too much data for embedding? 🤯🧩 Solution: Chunk your data! ✂️

A core principal of RAG is to search through your data to use as an input for LLMs. You can embed the data into a vector so it is searchable, but if you have a large video or entire book that will be too much data in one embedding. Let's talk about

GenAI is a game changer for podcasts 💡🎧

Gemini is revolutionizing how I learn from tech podcasts! It helps me pinpoint the exact discussions that align with my interests, so I can deep-dive into the topics that matter most. What tools are you using to stay informed in the fast-paced world

Build an AI marketing platform and support tools with HighLevel and Firestore

Learn how innovators like HighLevel are migrating to Firestore in order to realize significant total cost of ownership savings, while incorporating Firestore with AI to build differentiated solutions for their customers. This session will also cover