Installing the Cloud Storage connector

You can install and use the Cloud Storage Connector on an Apache Hadoop/Spark cluster—for example, to move on-premises HDFS data to Cloud Storage. Note that you can install the connector on a Spark standalone cluster, but you'll need set up the config file properly as noted in installation step 3.

The Cloud Storage connector is automatically installed on Cloud Dataproc clusters.

Steps to install the connector

1. Download the Cloud Storage Connector

   Hadoop 1.x

   1. Download the Cloud Storage connector for Hadoop 1.x
   2. Copy the jarfile into your hadoop/lib dir (see the next command example for Spark standalone mode).

      cp ~/Downloads/gcs-connector-latest-hadoop1.jar /your/hadoop/dir/lib/

      When running a Spark standalone cluster:

      cp ~/Downloads/gcs-connector-latest-hadoop1.jar $SPARK_HOME/jars directory

   Hadoop 2.x

   1. Download the Cloud Storage connector for Hadoop 2.x.
   2. Copy the jarfile into your $HADOOP_COMMON_LIB_JARS_DIR (see the next command example for Spark standalone mode).

      cp ~/Downloads/gcs-connector-latest-hadoop2.jar $HADOOP_COMMON_LIB_JARS_DIR.

      When running a Spark standalone cluster:

      cp ~/Downloads/gcs-connector-latest-hadoop2.jar $SPARK_HOME/jars directory

2. Set up service-account "keyfile" authentication.

   1. Make sure you have enabled the Compute Engine API in your project.
   2. Visit Google Cloud Platform Console→APIs & Services→Credentials, and select Create Credentials→service account key.

      

   3. Select Service account→Compute Engine default service account and Key type→JSON, then click Create to download the key.

      

   4. Keep track of the downloaded .json file. You may want to to rename it before placing it in a directory that is more easily accessible from Hadoop. For example:

      cp ~/Downloads/project-id-xxxxxxx.json /path/to/hadoop/conf/gcskey.json
      

      To help protect the key file, you can provide read-only access to owner and group, but restrict access to others (chmod 440 /path/to/hadoop/conf/gcskey.json).

3. Add the following entries to the conf/core-site.xml file:

   Required entries:
   <property>
     <name>google.cloud.auth.service.account.enable</name>
     <value>true</value>
   </property>
   <property>
     <name>google.cloud.auth.service.account.json.keyfile</name>
     <value>full path to JSON keyfile downloaded for service account</value>
   </property>
   

   Multiple Node Setup: In a cluster with multiple nodes, you must replicate the conf/core-site.xml setup on every node.
   Spark standalone mode: Spark may not install a core-site.xml in its conf dir, so you may need to create one or, alternatively, set spark.hadoop.* properties in the spark-defaults.conf file (see Custom Hadoop/Hive Configuration). For example, instead of setting the required properties in core-site.xml, you can set the properties in spark-defaults.conf:

   spark.hadoop.google.cloud.auth.service.account.enable true
   google.cloud.auth.service.account.json.keyfile path to keyfile
   

   .

Testing the installation

With the above settings on your master and worker nodes, you should be ready to test the Cloud Storage connector by running:

hadoop fs -ls gs://the-bucket-you-want-to-list