Connecting to Cloud Storage Buckets

This page introduces Cloud Storage as an option for storing your machine learning data and training output, and describes how to give your Cloud TPU access to the data objects on Cloud Storage.

Before you begin

You need a Cloud TPU Service Account in order to access a Cloud Storage bucket.

Note: Once you set up a Service Account for your project, it stays set while the project is active; you do not have to set up the Service Account every time you run a training under the same project.

1. Create a Cloud TPU Service Account for your project.

   gcloud beta services identity create --service tpu.googleapis.com --project $PROJECT_ID
   

   The command returns a Cloud TPU Service Account with following format:

   [e-mail]
   

   Important: This command creates a service agent, a special type of Google-managed service account which is not listed in your Service accounts page on the Google Cloud console.

2. Follow the instructions found in either the Cloud TPU quickstart guide or the Creating and deleting TPUs document to configure your Google Cloud project and create your Cloud TPU VM and TPU resources.

Writing data to Cloud Storage

Note: Set up the Cloud Storage bucket in the same location (region) that you set up the Cloud TPU. See Create buckets for all options available for managing storage buckets.

Console

1. Go to the Cloud Storage page on the Google Cloud console.

   Go to the Cloud Storage page

2. Create a new bucket, specifying the following options:

   • A unique name of your choosing.
   • Default storage class: Standard
   • Location: us-central1

gsutil

1. Use the gsutil mb command to create a Cloud Storage bucket:

   gsutil mb -l region gs://bucket-name/

   where:

   • region is the region where you created the Cloud TPU. For example: us-central1. Cloud TPU is available in the following regions:

     US

     Cloud TPU v2 and Preemptible v2	us-central1
     Cloud TPU v3 and Preemptible v3	us-central1
     Cloud TPU v4 and Preemptible v4	us-central2
     Cloud TPU v2 Pod	us-central1
     Cloud TPU v4 Pod	us-central2

     Europe

     Cloud TPU v2 and Preemptible v2	europe-west4
     Cloud TPU v3 and Preemptible v3	europe-west4
     Cloud TPU v2 Pod	europe-west4
     Cloud TPU v3 Pod	europe-west4

     Asia Pacific

     Cloud TPU v2 and Preemptible v2

     asia-east1

   • bucket-name is the name of the bucket you want to create.

2. Use the gsutil cp command to write files to the Cloud Storage bucket:

   gsutil cp -r local-data-dir gs://bucket-name

   where local-data-dir is a local path to your data. For example:$HOME/your-data

Giving your Cloud TPU access to Cloud Storage

You need to give your Cloud TPU read/write access to your Cloud Storage objects. To do that, you must grant the required access to the Cloud TPU Service Account used by the Cloud TPU. Follow these steps to find the Cloud TPU Service Account and grant the necessary access:

Authorize the Cloud TPU Service Account

Using fine-grained ACLs for Cloud TPU (Recommended)

If you store training data on Cloud Storage, the Cloud TPU Service Account needs read and write permission on the bucket.

Console

1. Go to the Cloud Storage browser page to view the buckets you own.

   Go to the Cloud Storage browser

2. Select the bucket whose ACL you want to modify.

3. Select the Permissions tab.

4. Select Add to add a new permission and type the complete Service Account name in the New Principals edit box.

5. If you are reading from this bucket, you must authorize the TPU Service Account to read from the resource. Do this by granting the Service Account the Storage Legacy > Storage Legacy Bucket Reader role.

6. If you are writing to this bucket, you must authorize the TPU Service Account to write to the resource. Do this by granting the Service Account the Storage Legacy > Storage Legacy Bucket Writer role.

gsutil

1. If you are reading from this bucket, grant read permission for the Cloud TPU Service Account:

    gsutil acl ch -u tpu-service-account:READER gs://bucket-name

2. If you are writing to this bucket, grant write permission for the Cloud TPU Service Account:

    gsutil acl ch -u tpu-service-account:WRITER gs://bucket-name

Using IAM permissions for Cloud TPU (Alternative)

If you want to grant broader permissions instead of whitelisting access to each bucket explicitly, you can grant the Identity Access Management (IAM) Storage Admin role to the Cloud TPU Service Account.

1. Go to the IAM page for your project.

   Go to IAM

2. Click the +Add button to add principals to the project.

3. Enter the names of the Cloud TPU Service Account in the Principals text box.

4. Click the Roles dropdown list.

5. Enable the following roles:

   • Project > Viewer

   • Storage > Storage Admin

What's next

• For more information about creating Cloud Storage buckets and writing data to those buckets, see the Cloud Storage documentation.
• For more information about Service Accounts, see the authentication overview.