Dataproc

Deploy Spark applications and pipelines that autoscale without any manual infrastructure provisioning or tuning. 

Create and scale clusters quickly with various virtual machine types, disk sizes, number of nodes, and networking options.

Dataproc autoscaling provides a mechanism for automating cluster resource management and enables automatic addition and subtraction of cluster workers (nodes).

Built-in integration with Cloud Storage, BigQuery, Dataplex, Vertex AI, Composer, Bigtable, Cloud Logging, and Cloud Monitoring, giving you a more complete and robust data platform.

Dataproc automatically configures hardware and software but also gives you manual control.

Multiple ways to manage a cluster, including an easy-to-use web UI, the Cloud SDK, RESTful APIs, and SSH access.

Run initialization actions to install or customize the settings and libraries you need when your cluster is created.

Use optional components to install and configure additional components on the cluster. Optional components are integrated with Dataproc components and offer fully configured environments for Zeppelin, Presto, and other open source software components related to the Apache Hadoop and Apache Spark ecosystem.

Dataproc serverless Spark can be provisioned with custom docker containers. Dataproc clusters can be provisioned with a custom image that includes your pre-installed Linux operating system packages.

Clusters can use custom machine types and preemptible virtual machines to make them the perfect size for your needs.

Dataproc workflow templates provide a flexible and easy-to-use mechanism for managing and executing workflows. A workflow template is a reusable workflow configuration that defines a graph of jobs with information on where to run those jobs. 

Standardize security, cost, and infrastructure policies across a fleet of clusters. You can create policies for resource management, security, or network at a project level. You can also make it easy for users to use the correct images, components, metastore, and other peripheral services, enabling you to manage your fleet of clusters and serverless Spark policies in the future. 

Dataproc recommended alerts allow customers to adjust the thresholds for the pre-configured alerts to get alerts on idle, runaway clusters, jobs, overutilized clusters and more. Customers can further customize these alerts and even create advanced cluster and job management capabilities. These capabilities allow customers to manage their fleet at scale.

Dataproc on GDC enables you to run Spark on the GDC Edge Appliance in your data center. Now you can use the same Spark applications on Google Cloud as well as on sensitive data in your data center.

Dataproc Metastore is a fully managed, highly available Hive metastore (HMS) with fine-grained access control. Multi-regional Dataproc Metastore provides active-active DR and resilience against regional outages.