Connecting to Cloud Storage Buckets

This page introduces Cloud Storage as an option for storing your machine learning data, and describes how to give your Cloud TPU access to the data objects on Cloud Storage.

Before you begin

  1. Create a Cloud TPU Service Account for your project.

    gcloud beta services identity create --service tpu.googleapis.com --project $PROJECT_ID
    

    The command returns a Cloud TPU Service Account with following format:

    service-PROJECT_NUMBER@cloud-tpu.iam.gserviceaccount.com
    
  2. Follow the instructions found in either the Cloud TPU quickstart guide or the Creating and deleting TPUs document to configure your Google Cloud project, create a Compute Engine VM, and create a Cloud TPU resource.

Writing data to Cloud Storage

Console

  1. Go to the Cloud Storage page on the Cloud Console.

    Go to the Cloud Storage page

  2. Create a new bucket, specifying the following options:

    • A unique name of your choosing.
    • Default storage class: Standard
    • Location: us-central1

gsutil

  1. Use the gsutil mb command to create a Cloud Storage bucket:

    gsutil mb -l region gs://bucket-name/

    where:

    • region is the region where you created the Cloud TPU. For example: us-central1. Cloud TPU is available in the following regions:

      US

      Cloud TPU v2 and Preemptible v2 us-central1
      Cloud TPU v3 and Preemptible v3 us-central1
      Cloud TPU v2 Pod (beta) us-central1

      Europe

      Cloud TPU v2 and Preemptible v2 europe-west4
      Cloud TPU v3 and Preemptible v3 europe-west4
      Cloud TPU v2 Pod (beta) europe-west4
      Cloud TPU v3 Pod (beta) europe-west4

      Asia Pacific

      Cloud TPU v2 and Preemptible v2 asia-east1
    • bucket-name is the name of the bucket you want to create.

  2. Use the gsutil cp command to write files to the Cloud Storage bucket:

    gsutil cp -r local-data-dir gs://bucket-name

    where local-data-dir is a local path to your data. For example:$HOME/your-data

Giving your Cloud TPU access to Cloud Storage

You need to give your Cloud TPU read/write access to your Cloud Storage objects. To do that, you must grant the required access to the Cloud TPU Service Account used by the Cloud TPU. Follow these steps to find the Cloud TPU Service Account and grant the necessary access:

Authorize the Cloud TPU Service account

If you store training data on Cloud Storage, the Cloud TPU service account needs read and write permission on the bucket.

Console

  1. Go to the Cloud Storage browser page to view the buckets you own.

    Go to the Cloud Storage browser

  2. Navigate to the bucket whose ACL you want to modify.

  3. On that bucket, select Edit Bucket Permissions.

  4. If you are reading from this bucket, you must authorize the TPU service account to read from the resource. Do this by granting the service account the Storage Legacy > Storage Legacy Bucket Reader role.

  5. If you are writing to this bucket, you must authorize the TPU service account to write to the resource. Do this by granting the service account the Storage Legacy > Storage Legacy Bucket Writer role.

gsutil

  1. If you are reading from this bucket, grant read permission for the Cloud TPU Service Account:

     gsutil acl ch -u tpu-service-account:READER gs://bucket-name

  2. If you are writing to this bucket, grant write permission for the Cloud TPU Service Account:

     gsutil acl ch -u tpu-service-account:WRITER gs://bucket-name

Using IAM permissions for Cloud TPU (Alternative)

If you want to grant broader permissions instead of whitelisting access to each bucket explicitly, you can grant the Identity Access Management (IAM) Storage Admin role to the Cloud TPU Service Account.

  1. Go to the IAM page for your project.

    Go to IAM

  2. Click the +Add button to add members to the project.

  3. Enter the names of the Cloud TPU Service Account in the Members text box.

  4. Click the Roles dropdown list.

  5. Enable the following roles:

    • Project > Viewer

    • Storage > Storage Admin

What's next