Connecting to Cloud Storage Buckets

This page introduces Cloud Storage as an option for storing your machine learning data, and describes how to give your Cloud TPU access to the data objects on Cloud Storage.

Before you begin

Follow the Cloud TPU quickstart guide or the custom setup guide to configure your Google Cloud Platform project, create a Compute Engine VM, and create a Cloud TPU resource.

Writing data to Cloud Storage

Console

  1. Go to the Cloud Storage page on the GCP Console.

    Go to the Cloud Storage page

  2. Create a new bucket, specifying the following options:

    • A unique name of your choosing.
    • Default storage class: Regional
    • Location: us-central1

gsutil

  1. Use the gsutil mb command to create a Cloud Storage bucket:

    gsutil mb -l ${TPU_ZONE} gs://YOUR-BUCKET-NAME/

    where:

    • TPU_ZONE is the zone where you created the Cloud TPU. Cloud TPU is available in the following zones:

      US

      Cloud TPU v2 and Preemptible v2 us-central1-b
      us-central1-c
      us-central1-f ( TFRC program only)
      Cloud TPU v3 (beta) and Preemptible v3 (beta) us-central1-b
      us-central1-f
      ( TFRC program only)

      Europe

      Cloud TPU v2 and Preemptible v2 europe-west4-a
      Cloud TPU v3 (beta) and Preemptible v3 (beta) europe-west4-a

      Asia Pacific

      Cloud TPU v2 and Preemptible v2 asia-east1-c

    • YOUR-BUCKET-NAME is the name of the bucket you want to create.

  2. Use the gsutil cp command to write files to the Cloud Storage bucket:

    gsutil cp -r ${LOCAL_DATA_DIR} ${CLOUD_DATA_DIR}

    where:

    • LOCAL_DATA_DIR is a local path to your data. For example: $HOME/your-data
    • CLOUD_DATA_DIR is a path within Cloud Storage. For example: gs://YOUR-BUCKET-NAME

Giving your Cloud TPU access to Cloud Storage

You need to give your Cloud TPU read/write access to your Cloud Storage objects. To do that, you must grant the required access to the service account used by the Cloud TPU. Follow these steps to find the Cloud TPU service account and grant the necessary access:

Locate the Cloud TPU service account

Determine the name of the Cloud TPU service account. The name follows this format:

    service-[PROJECT_NUMBER]@cloud-tpu.iam.gserviceaccount.com

See how to find your project number.

Authorize the service account

If you store training data on Cloud Storage, the Cloud TPU service account needs read and write permission on the bucket.

Console

  1. Go to the Cloud Storage browser page to view the buckets you own.

    Go to the Cloud Storage browser

  2. Navigate to the bucket whose ACL you want to modify.

  3. On that bucket, select Edit Bucket Permissions.

  4. If you are reading from this bucket, you must authorize the TPU service account to read from the resource. Do this by granting the service account the Storage Legacy > Storage Legacy Bucket Reader role.

  5. If you are writing to this bucket, you must authorize the TPU service account to write to the resource. Do this by granting the service account the Storage Legacy > Storage Legacy Bucket Writer role.

gsutil

  1. If you are reading from this bucket, grant read permission for the TPU service account:

    gsutil acl ch -u [SERVICE_ACCOUNT]:READER gs://[BUCKET_NAME]

  2. If you are writing to this bucket, grant write permission for the TPU service account:

    gsutil acl ch -u [SERVICE_ACCOUNT]:WRITER gs://[BUCKET_NAME]

Using IAM permissions for Cloud TPU (Alternative)

If you want to grant broader permissions instead of whitelisting access to each bucket explicitly, you can grant the Identity Access Management (IAM) Storage Admin role to the Cloud TPU service account.

  1. Go to the Cloud IAM page for your project.

    Go to Cloud IAM

  2. Click the +Add button to add members to the project.

  3. Enter the names of the Cloud TPU service accounts in the Members text box.

  4. Click the Roles dropdown list.

  5. Enable the following roles:

    • Project > Viewer

    • Storage > Storage Admin

What's next

Was this page helpful? Let us know how we did:

Send feedback about...