Dataproc is a fully managed and highly scalable service for running Apache Spark, Apache Flink, Presto, and 30+ open source tools and frameworks. Use Dataproc for data lake modernization, ETL, and secure data science, at planet scale, fully integrated with Google Cloud, at a fraction of the cost.
Modernize your open source data processing
Whether you need VMs or Kubernetes, extra memory for Presto, or even GPUs, Dataproc can help accelerate your data and analytics processing through on-demand purpose-built or serverless environments.
Intelligent and seamless OSS for data science
Enable data scientists and data analysts to seamlessly perform data science jobs through native integrations with Vertex AI.
Submit Spark jobs which auto-provision and auto-scale. More details with the quickstart link below.
Dataproc initialization actions
Add other OSS projects to your Dataproc clusters with pre-built initialization actions.
Open source connectors
Libraries and tools for Apache Hadoop interoperability.
Dataproc Workflow Templates
The Dataproc WorkflowTemplates API provides a flexible and easy-to-use mechanism for managing and executing workflows.
Enterprises are migrating their existing on-premises Apache Hadoop and Spark clusters over to Dataproc to manage costs and unlock the power of elastic scale. With Dataproc, enterprises get a fully managed, purpose-built cluster that can autoscale to support any data or analytics processing job.
Migrate HDFS data to Google Cloud
Create your ideal data science environment by spinning up a purpose-built Dataproc cluster. Integrate open source software like Apache Spark, NVIDIA RAPIDS, and Jupyter notebooks with Google Cloud AI services and GPUs to help accelerate your machine learning and AI development.
Use Dataproc and Apache Spark ML for machine learning
IT governed open source data science with Dataproc Hub
|Serverless Spark||Deploy Spark applications and pipelines that autoscale without any manual infrastructure provisioning or tuning.|
|Resizable clusters||Create and scale clusters quickly with various virtual machine types, disk sizes, number of nodes, and networking options.|
|Autoscaling clusters||Dataproc autoscaling provides a mechanism for automating cluster resource management and enables automatic addition and subtraction of cluster workers (nodes).|
|Cloud integrated||Built-in integration with Cloud Storage, BigQuery, Cloud Bigtable, Cloud Logging, Cloud Monitoring, and AI Hub, giving you a more complete and robust data platform.|
|Versioning||Image versioning allows you to switch between different versions of Apache Spark, Apache Hadoop, and other tools.|
|Highly available||Run clusters in high availability mode with multiple main nodes and set jobs to restart on failure to help ensure your clusters and jobs are highly available.|
|Cluster scheduled deletion||To help avoid incurring charges for an inactive cluster, you can use Dataproc's scheduled deletion, which provides options to delete a cluster after a specified cluster idle period, at a specified future time, or after a specified time period.|
|Automatic or manual configuration||Dataproc automatically configures hardware and software but also gives you manual control.|
|Developer tools||Multiple ways to manage a cluster, including an easy-to-use web UI, the Cloud SDK, RESTful APIs, and SSH access.|
|Initialization actions||Run initialization actions to install or customize the settings and libraries you need when your cluster is created.|
|Optional components||Use optional components to install and configure additional components on the cluster. Optional components are integrated with Dataproc components and offer fully configured environments for Zeppelin, Druid, Presto, and other open source software components related to the Apache Hadoop and Apache Spark ecosystem.|
|Custom images||Dataproc clusters can be provisioned with a custom image that includes your pre-installed Linux operating system packages.|
|Flexible virtual machines||Clusters can use custom machine types and preemptible virtual machines to make them the perfect size for your needs.|
|Component Gateway and notebook access||Dataproc Component Gateway enables secure, one-click access to Dataproc default and optional component web interfaces running on the cluster.|
|Workflow templates||Dataproc workflow templates provide a flexible and easy-to-use mechanism for managing and executing workflows. A workflow template is a reusable workflow configuration that defines a graph of jobs with information on where to run those jobs.|