Concepts

Dataproc overview

• What is Dataproc?

  An introduction to Dataproc and its features.

Accessing clusters

• Cluster web interfaces

  Available web interfaces for Dataproc's open source components, and how to connect to them.

• Component gateway^Alpha

  Use the component gateway to connect to cluster components.

• Network configuration

  Configure your cluster's network.

Components

• Overview

  Overview of cluster components.

• Anaconda optional component

  Install the Anaconda component on your cluster.

• Druid optional component^Alpha

  Install the Druid component on your cluster.

• Hive WebHCat optional component

  Install the Hive WebHCat component on your cluster.

• Jupyter optional component

  Install the Jupyter component on your cluster.

• Presto optional component^Beta

  Install the Presto component on your cluster.

• Zeppelin optional component

  Install the Zeppelin component on your cluster.

• Zookeeper optional component^Beta

  Install the Zookeeper component on your cluster.

Compute options

• Custom machine types

  Dataproc lets you specify custom machine types for special workloads.

• GPU Clusters

  Use Graphics Processing Units (GPUs) with your Dataproc clusters.

• Local Solid State Drives

  Attach local SSDs to Dataproc clusters.

• Minimum CPU platform^Beta

  Specify a minimum CPU platform for your Dataproc cluster.

• Persistent Solid State Drive (PD-SSD) boot disks

  Create clusters with persistent SSD boot disks.

• Preemptible VMs

  Preemptible instances are affordable nodes that are reclaimed if they're needed for other tasks, providing you with more flexibility when building your cluster.

Configuring and running jobs

• Life of a job

  Understand Dataproc job throttling.

• Restartable jobs

  Create jobs that restart on failure. Great for long-running and streaming jobs.

• Dataproc jobs on GKE^Alpha

  Running Dataproc job on Google Kubernetes Engine

Configuring clusters

• Autoscaling clusters

  Learn how to use autoscaling to automatically resize clusters to meet the demands of user workloads.

• Auto Zone placement

  Let Dataproc select a zone for your cluster.

• Cluster metadata

  Learn about Dataproc's cluster metadata and how to set your own custom metadata.

• Cluster properties

  Configuration properties for Dataproc's open source components and how to access them.

• Customer managed encryption keys (CMEK)

  Manage encrypted keys for Dataproc cluster and job data.

• Enhanced Flexibility Mode^Alpha

  Keep jobs running by saving intermediate data to a distributed file system.

• High availability mode

  Increase the resilience of HDFS and YARN to service unavailability

• Initialization actions

  Specify actions to run on all or some cluster nodes on setup.

• Network configuration

  Configure your cluster's network.

• Scaling Clusters

  Increase or decrease the number of worker nodes in a cluster, even while jobs are running.

• Scheduled Deletion^Beta

  Delete your cluster after a specified period or at a specified time.

• Security Configuration^Beta

  Enable cluster security features.

• Service accounts

  Understand Dataproc service accounts.

• Single node clusters

  Create lightweight sandbox clusters with only one node.

• Staging bucket

  Learn about the Dataproc staging bucket.

Connectors

• BigQuery connector

  Use the BigQuery for Apache Hadoop on your Dataproc clusters.

• BigQuery connector code samples

  View the BigQuery code samples.

• Cloud Bigtable with Dataproc

  Use Cloud Bigtable Apache HBase-compatible API with your Dataproc clusters.

• Cloud Storage connector

  Use the Cloud Storage connector for Hadoop on your Dataproc clusters.

• Installing the Cloud Storage connector

  Using the Cloud Storage connector on other (non-Dataproc) clusters.

Data storage

• Hadoop data storage

  Understand Hadoop Data Storage on Dataproc.

Identity and Access Management (IAM)

• Dataproc permissions and IAM roles

  Set up IAM roles to allow users and groups to access your project's Dataproc resources.

• Dataproc principals and roles

  Understand Dataproc principals and the roles required to create, manage, and run tasks on a cluster.

• Dataproc Granular IAM

  Set up granular cluster-specific permissions.

Dataproc Regional endpoints

• Regional endpoints

  Use regional endpoints with Dataproc resources.

Versioning

• Overview

  Software versions used on Dataproc clusters and how to select them.

• Overview

  Software versions used on Dataproc clusters and how to select them.

• 1.4.x release versions

  Dataproc image version 1.4.

• 1.3.x release versions

  Dataproc image version 1.3.

• 1.2.x release versions

  Dataproc image version 1.2.

• 1.1.x release versions

  Dataproc image version 1.1.

• Image version list

  List of versions currently supported in Dataproc clusters.

Workflow Templates

• Overview

  Learn about workflow templates.

• Monitoring and debugging workflows

  How to monitor and debug workflows.

• Parameterization

  Learn how to parameterize your workflow templates.

• Using YAML files

  Learn how to use YAML files in your workflow.

• Using cluster selectors

  Learn how to use cluster selectors in your workflow.

• Using workflows

  Learn how to set up and run workflows.