Connecting to Cloud Storage Buckets

This page introduces Cloud Storage as an option for storing your machine learning data and training output, and describes how to give your Cloud TPU access to the data objects on Cloud Storage.

Before you begin

You need a Cloud TPU Service Account in order to access a Cloud Storage bucket.

  1. Create a Cloud TPU Service Account for your project.

    gcloud beta services identity create --service tpu.googleapis.com --project $PROJECT_ID
    

    The command returns a Cloud TPU Service Account with following format:

    service-PROJECT_NUMBER@cloud-tpu.iam.gserviceaccount.com
    
  2. Follow the instructions found in either the Cloud TPU quickstart guide or the Creating and deleting TPUs document to configure your Google Cloud project and create your Cloud TPU VM and TPU resources.

Writing data to Cloud Storage

Console

  1. Go to the Cloud Storage page on the Google Cloud console.

    Go to the Cloud Storage page

  2. Create a new bucket, specifying the following options:

    • A unique name of your choosing.
    • Default storage class: Standard
    • Location: us-central1

gsutil

  1. Use the gsutil mb command to create a Cloud Storage bucket:

    gsutil mb -l region gs://bucket-name/

    where:

    • region is the region where you created the Cloud TPU. For example: us-central1. Cloud TPU is available in the following regions:

      US

      Cloud TPU v2 and Preemptible v2 us-central1
      Cloud TPU v3 and Preemptible v3 us-central1
      Cloud TPU v4 and Preemptible v4 us-central2
      Cloud TPU v2 Pod us-central1
      Cloud TPU v4 Pod us-central2

      Europe

      Cloud TPU v2 and Preemptible v2 europe-west4
      Cloud TPU v3 and Preemptible v3 europe-west4
      Cloud TPU v2 Pod europe-west4
      Cloud TPU v3 Pod europe-west4

      Asia Pacific

      Cloud TPU v2 and Preemptible v2 asia-east1
    • bucket-name is the name of the bucket you want to create.

  2. Use the gsutil cp command to write files to the Cloud Storage bucket:

    gsutil cp -r local-data-dir gs://bucket-name

    where local-data-dir is a local path to your data. For example:$HOME/your-data

Giving your Cloud TPU access to Cloud Storage

You need to give your Cloud TPU read/write access to your Cloud Storage objects. To do that, you must grant the required access to the Cloud TPU Service Account used by the Cloud TPU. Follow these steps to find the Cloud TPU Service Account and grant the necessary access:

Authorize the Cloud TPU Service Account

If you store training data on Cloud Storage, the Cloud TPU Service Account needs read and write permission on the bucket.

Console

  1. Go to the Cloud Storage browser page to view the buckets you own.

    Go to the Cloud Storage browser

  2. Select the bucket whose ACL you want to modify.

  3. Select the Permissions tab.

  4. Select Add to add a new permission and type the complete Service Account name in the New Principals edit box.

  5. If you are reading from this bucket, you must authorize the TPU Service Account to read from the resource. Do this by granting the Service Account the Storage Legacy > Storage Legacy Bucket Reader role.

  6. If you are writing to this bucket, you must authorize the TPU Service Account to write to the resource. Do this by granting the Service Account the Storage Legacy > Storage Legacy Bucket Writer role.

gsutil

  1. If you are reading from this bucket, grant read permission for the Cloud TPU Service Account:

     gsutil acl ch -u tpu-service-account:READER gs://bucket-name

  2. If you are writing to this bucket, grant write permission for the Cloud TPU Service Account:

     gsutil acl ch -u tpu-service-account:WRITER gs://bucket-name

Using IAM permissions for Cloud TPU (Alternative)

If you want to grant broader permissions instead of whitelisting access to each bucket explicitly, you can grant the Identity Access Management (IAM) Storage Admin role to the Cloud TPU Service Account.

  1. Go to the IAM page for your project.

    Go to IAM

  2. Click the +Add button to add principals to the project.

  3. Enter the names of the Cloud TPU Service Account in the Principals text box.

  4. Click the Roles dropdown list.

  5. Enable the following roles:

    • Project > Viewer

    • Storage > Storage Admin

Cloud Storage FUSE

Cloud Storage FUSE allows you to mount and access Cloud Storage buckets as local file systems. This allows applications to read and write objects in your bucket using standard file system semantics.

See the Cloud Storage FUSE documentation for details about how Cloud Storage FUSE works and a description of how Cloud Storage FUSE operations map to Cloud Storage operations. You can find additional information about how to use Cloud Storage FUSE, such as how to install the gcsfuse CLI and mounting buckets on GitHub.

What's next