Cloud Storage connector

The Cloud Storage connector is an open source Java library that lets you run Apache Hadoop or Apache Spark jobs directly on data in Cloud Storage, and offers a number of benefits over choosing the Hadoop Distributed File System (HDFS).

Connector Support. The Cloud Storage connector is supported by Google Cloud for use with Google Cloud products and use cases, and when used with Dataproc is supported at the same level as Dataproc (see Get support).

Benefits of the Cloud Storage connector

• Direct data access – Store your data in Cloud Storage and access it directly. You do not need to transfer it into HDFS first.
• HDFS compatibility – You can easily access your data in Cloud Storage using the gs:// prefix instead of hdfs://.
• Interoperability – Storing data in Cloud Storage enables seamless interoperability between Spark, Hadoop, and Google services.
• Data accessibility – When you shut down a Hadoop cluster, unlike HDFS, you continue to have access to your data in Cloud Storage.
• High data availability – Data stored in Cloud Storage is highly available and globally replicated without a loss of performance.
• No storage management overhead – Unlike HDFS, Cloud Storage requires no routine maintenance, such as checking the file system, or upgrading or rolling back to a previous version of the file system.
• Quick startup – In HDFS, a MapReduce job can't start until the NameNode is out of safe mode, a process that can take from a few seconds to many minutes depending on the size and state of your data. With Cloud Storage, you can start your job as soon as the task nodes start, which leads to significant cost savings over time.

Getting the connector

Dataproc clusters

The Cloud Storage connector is installed by default on all Dataproc cluster nodes under /usr/local/share/google/dataproc/lib/ directory.

Note: When running the connector inside of Compute Engine VMs, including Dataproc clusters, google.cloud.auth.service.account.enable is set to false by default, which means you don't need to manually configure a service account for the connector; it will automatically get the service account credential from the VM metadata server. However, you just need to make sure the VM service account has permission to access your Cloud Storage bucket.

If your application depends a connector version that is different from the default connector version deployed on your Dataproc cluster, you must either:

1. create a new cluster with the --metadata GCS_CONNECTOR_VERSION=x.y.z flag, which will update the connector used by your application to the specified connector version, or
2. include and relocate the connector classes and connector dependencies for the version you are using into your application's jar so that the connector version you are using will not conflict with the connector version deployed on your Dataproc cluster (see this example of dependencies relocation in Maven).

Non-Dataproc clusters

You can install and use the Cloud Storage connector on an Apache Hadoop/Spark cluster, for example, to move on-premises HDFS data to Cloud Storage.

To download the Cloud Storage connector for Hadoop:

• latest version located in Cloud Storage bucket (not recommended for use in production):
  • Cloud Storage connector for Hadoop 1.x
  • Cloud Storage connector for Hadoop 2.x
  • Cloud Storage connector for Hadoop 3.x
• specific version from Cloud Storage bucket by substituting Hadoop and Cloud Storage connector versions in the gcs-connector-HADOOP_VERSION-CONNECTOR_VERSION.jar name pattern:
  • gs://hadoop-lib/gcs/gcs-connector-hadoop2-2.1.1.jar
• specific version from Apache Maven repository (you should download a shaded jar that has -shaded suffix in the name):
  • Cloud Storage connector for Hadoop 1.x
  • Cloud Storage connector for Hadoop 2.x
  • Cloud Storage connector for Hadoop 3.x

See Installing the connector on GitHub to to install, configure, and test the Cloud Storage connector.

Using the connector

There are multiple ways to access data stored in Cloud Storage:

• In a Spark (or PySpark) or Hadoop application using the gs:// prefix.
• The hadoop shell: hadoop fs -ls gs://bucket/dir/file.
• The Cloud Console Cloud Storage browser.
• Using the gsutil cp or gsutil rsync commands.

Resources

• Use the Cloud Storage connector with Apache Spark
• Apache Hadoop FileSystem API

Java version

The Cloud Storage connector requires Java 8.

Apache Maven Dependency Information

<dependency>
    <groupId>com.google.cloud.bigdataoss</groupId>
    <artifactId>gcs-connector</artifactId>
    <version>insert "hadoopX-X.X.X" connector version number here</version>
    <scope>provided</scope>
</dependency>

For more detailed information, see the Cloud Storage connector release notes and Javadoc reference.

What's next

• Learn more about Cloud Storage