Cloud Dataproc clusters
Cloud SDK contains tools and libraries that enable you to easily create and manage resources on Google Cloud Platform. Cloud SDK makes it easy for you to create and manage Cloud Dataproc clusters on Cloud Platform.
Other Spark and Hadoop clusters
These tools are also useful for Spark and Hadoop clusters that are not running on Cloud Dataproc, such as a self-managed cluster running on Compute Engine.
Command-line utilities (bdutil)
The bdutil package contains a set of scripts to help you deploy and manage self-managed Spark and Hadoop clusters directly on Google Cloud Platform. To install bdutil clone the bdutil GitHub repository.
With Google connectors, you can use Google Cloud Platform services, such as Cloud Storage, in your Spark and Hadoop clusters.
You can manually install these connectors on new or existing self-managed clusters. Your clusters can use these connectors even if they do not run on Google Cloud Platform. For example, you can use the Cloud Storage connector with a cluster running on-premises or in another cloud.
Cloud Storage Connector—The Cloud Storage connector lets you run Hadoop or Spark jobs directly on data in Cloud Storage, and offers benefits over choosing Hadoop Distributed File System (HDFS) as your default file system.
BigQuery Connector—You can use a BigQuery connector to enable programmatic read/write access to BigQuery. This is ideal for processing data that you've already stored in BigQuery.