Configure access to a source: Cloud Storage

Storage Transfer Service uses a Google-managed service account, known as a service agent, to move data from a Cloud Storage source bucket. This service agent is created the first time that you call googleServiceAccounts.get.

The source bucket does not need to belong to the same project as the service agent. The steps are the same regardless of which project the bucket is in.

User permissions

In order to grant the required permissions to the service agent, you must have the relevant permissions on the source bucket:

• storage.buckets.getIamPolicy
• storage.buckets.setIamPolicy

The Storage Legacy Bucket Owner role (roles/storage.legacyBucketOwner) or the Storage Admin role (roles/storage.admin) provide the required permissions.

Auto-granting permissions in the Google Cloud console

If you're using the Google Cloud console to create your transfer, and have the permissions listed in User permissions, the service agent will automatically be granted the required permissions on your source bucket.

You can skip the steps on this page and go directly to creating a transfer.

Required permissions

The service agent must have the following permissions for the source bucket:

Permission	Description
storage.buckets.get	Allows the service agent to get the location of the bucket.
storage.objects.list	Allows the service agent to list objects in the bucket.
storage.objects.get	Allows the service agent to read objects in the bucket.
storage.objects.create	Allows the service agent to create objects in the bucket. Required if the transfer is to a file system. Storage Transfer Service creates and populates two folders in the source bucket: cloud-ingest/ contains file system information, and storage-transfer/ contains transfer logs.
storage.objects.delete	Allows the service agent to delete objects in the bucket. Required if you set deleteObjectsFromSourceAfterTransfer to true.

The following predefined roles together grant the required permissions:

• One of:
  • Storage Object Viewer (roles/storage.objectViewer) if the transfer is to another Cloud Storage bucket.
  • Storage Object Creator (roles/storage.objectCreator) if the transfer is to a file system.
• Plus one of:
  • Storage Legacy Bucket Writer (roles/storage.legacyBucketWriter) if object delete permission is required.
  • Storage Legacy Bucket Reader (roles/storage.legacyBucketReader) if object delete permission is not required.

Any Cloud Storage role marked as a legacy role can only be granted at the bucket level.

For a complete list of Cloud Storage roles and the permissions they contain, see IAM roles.

For object ACLs when ACL_PRESERVE is set, the Storage Legacy Object Owner role (roles/storage.legacyObjectOwner) is additionally required for the source object. For ACL_PRESERVE, manually add the service account to have the Storage Legacy Object Owner role to give the source object permission storage.objects.getIamPolicy.

Grant the required permissions

To grant the required permissions to the service agent, follow the steps below.

Find the service agent's email

1. Go to the googleServiceAccounts.get reference page.

   An interactive panel opens, titled Try this method.

2. In the panel, under Request parameters, enter your project ID. The project you specify here must be the project you're using to manage Storage Transfer Service, which might be different from the source bucket's project.

3. Click Execute.

   Your service agent's email is returned as the value of accountEmail. Copy this value.

   The service agent's email uses the format [e-mail].

Add the service agent to a bucket-level policy

Console

1. In the Google Cloud console, go to the Cloud Storage Buckets page.
   

   Go to Buckets

2. Click the Bucket overflow menu () associated with the bucket to which you want to grant a principal a role.

3. Choose Edit access.

4. Click the + Add principal button.

5. In the New principals field, enter your service agent's account email.

6. Select Storage Object Viewer or Storage Object Creator from the Select a role drop-down menu.

7. Click Add another role.

8. Select either Storage Legacy Bucket Writer or Storage Legacy Bucket Reader.

9. Click Save.

gcloud

Use the gcloud storage buckets add-iam-policy-binding command:

gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
--member=serviceAccount:YOUR_AGENT_EMAIL --role=roles/storage.objectViewer

gcloud storage buckets add-iam-policy-binding gs://BUCKET_NAME \
--member=serviceAccount:YOUR_AGENT_EMAIL --role=roles/storage.legacyBucketReader

Where:

• BUCKET_NAME is the name of the bucket you are granting the principal access to. For example, my-bucket.
• YOUR_AGENT_EMAIL is the agent account email you copied in Find the service agent's email.

Code samples

C++

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage C++ API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

namespace gcs = ::google::cloud::storage;
using ::google::cloud::StatusOr;
[](gcs::Client client, std::string const& bucket_name,
   std::string const& role, std::string const& member) {
  auto policy = client.GetNativeBucketIamPolicy(
      bucket_name, gcs::RequestedPolicyVersion(3));

  if (!policy) throw std::move(policy).status();

  policy->set_version(3);
  for (auto& binding : policy->bindings()) {
    if (binding.role() != role || binding.has_condition()) {
      continue;
    }
    auto& members = binding.members();
    if (std::find(members.begin(), members.end(), member) == members.end()) {
      members.emplace_back(member);
    }
  }

  auto updated = client.SetNativeBucketIamPolicy(bucket_name, *policy);
  if (!updated) throw std::move(updated).status();

  std::cout << "Updated IAM policy bucket " << bucket_name
            << ". The new policy is " << *updated << "\n";
}

C#

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage C# API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

using Google.Apis.Storage.v1.Data;
using Google.Cloud.Storage.V1;
using System;
using System.Collections.Generic;

public class AddBucketIamMemberSample
{
    public Policy AddBucketIamMember(
        string bucketName = "your-unique-bucket-name",
        string role = "roles/storage.objectViewer",
        string member = "serviceAccount:[e-mail]")
    {
        var storage = StorageClient.Create();
        var policy = storage.GetBucketIamPolicy(bucketName, new GetBucketIamPolicyOptions
        {
            RequestedPolicyVersion = 3
        });
        // Set the policy schema version. For more information, please refer to https://cloud.google.com/iam/docs/policies#versions.
        policy.Version = 3;

        Policy.BindingsData bindingToAdd = new Policy.BindingsData
        {
            Role = role,
            Members = new List<string> { member }
        };

        policy.Bindings.Add(bindingToAdd);
        var bucketIamPolicy = storage.SetBucketIamPolicy(bucketName, policy);
        Console.WriteLine($"Added {member} with role {role} " + $"to {bucketName}");
        return bucketIamPolicy;
    }
}

Go

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage Go API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

import (
	"context"
	"fmt"
	"io"
	"time"

	"cloud.google.com/go/iam"
	"cloud.google.com/go/storage"
)

// addBucketIAMMember adds the bucket IAM member to permission role.
func addBucketIAMMember(w io.Writer, bucketName string) error {
	// bucketName := "bucket-name"
	ctx := context.Background()
	client, err := storage.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("storage.NewClient: %w", err)
	}
	defer client.Close()

	ctx, cancel := context.WithTimeout(ctx, time.Second*10)
	defer cancel()

	bucket := client.Bucket(bucketName)
	policy, err := bucket.IAM().Policy(ctx)
	if err != nil {
		return fmt.Errorf("Bucket(%q).IAM().Policy: %w", bucketName, err)
	}
	// Other valid prefixes are "serviceAccount:", "user:"
	// See the documentation for more values.
	// https://cloud.google.com/storage/docs/access-control/iam
	identity := "group:[e-mail]"
	var role iam.RoleName = "roles/storage.objectViewer"

	policy.Add(identity, role)
	if err := bucket.IAM().SetPolicy(ctx, policy); err != nil {
		return fmt.Errorf("Bucket(%q).IAM().SetPolicy: %w", bucketName, err)
	}
	// NOTE: It may be necessary to retry this operation if IAM policies are
	// being modified concurrently. SetPolicy will return an error if the policy
	// was modified since it was retrieved.
	fmt.Fprintf(w, "Added %v with role %v to %v\n", identity, role, bucketName)
	return nil
}

Java

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

import com.google.cloud.Binding;
import com.google.cloud.Policy;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class AddBucketIamMember {
  /** Example of adding a member to the Bucket-level IAM */
  public static void addBucketIamMember(String projectId, String bucketName) {
    // The ID of your GCP project
    // String projectId = "your-project-id";

    // The ID of your GCS bucket
    // String bucketName = "your-unique-bucket-name";

    // For more information please read:
    // https://cloud.google.com/storage/docs/access-control/iam
    Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();

    Policy originalPolicy =
        storage.getIamPolicy(bucketName, Storage.BucketSourceOption.requestedPolicyVersion(3));

    String role = "roles/storage.objectViewer";
    String member = "group:[e-mail]";

    // getBindingsList() returns an ImmutableList and copying over to an ArrayList so it's mutable.
    List<Binding> bindings = new ArrayList(originalPolicy.getBindingsList());

    // Create a new binding using role and member
    Binding.Builder newMemberBindingBuilder = Binding.newBuilder();
    newMemberBindingBuilder.setRole(role).setMembers(Arrays.asList(member));
    bindings.add(newMemberBindingBuilder.build());

    // Update policy to add member
    Policy.Builder updatedPolicyBuilder = originalPolicy.toBuilder();
    updatedPolicyBuilder.setBindings(bindings).setVersion(3);
    Policy updatedPolicy = storage.setIamPolicy(bucketName, updatedPolicyBuilder.build());

    System.out.printf("Added %s with role %s to %s\n", member, role, bucketName);
  }
}

Node.js

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage Node.js API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The role to grant
// const roleName = 'roles/storage.objectViewer';

// The members to grant the new role to
// const members = [
//   'user:[e-mail]',
//   'group:[e-mail]',
// ];

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

async function addBucketIamMember() {
  // Get a reference to a Google Cloud Storage bucket
  const bucket = storage.bucket(bucketName);

  // For more information please read:
  // https://cloud.google.com/storage/docs/access-control/iam
  const [policy] = await bucket.iam.getPolicy({requestedPolicyVersion: 3});

  // Adds the new roles to the bucket's IAM policy
  policy.bindings.push({
    role: roleName,
    members: members,
  });

  // Updates the bucket's IAM policy
  await bucket.iam.setPolicy(policy);

  console.log(
    `Added the following member(s) with role ${roleName} to ${bucketName}:`
  );

  members.forEach(member => {
    console.log(`  ${member}`);
  });
}

addBucketIamMember().catch(console.error);

PHP

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage PHP API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

use Google\Cloud\Storage\StorageClient;

/**
 * Adds a new member / role IAM pair to a given Cloud Storage bucket.
 *
 * @param string $bucketName The name of your Cloud Storage bucket.
 *        (e.g. 'my-bucket')
 * @param string $role The role to which the given member should be added.
 *        (e.g. 'roles/storage.objectViewer')
 * @param string[] $members The member(s) to be added to the role.
 *        (e.g. ['group:[e-mail]'])
 */
function add_bucket_iam_member(string $bucketName, string $role, array $members): void
{
    $storage = new StorageClient();
    $bucket = $storage->bucket($bucketName);

    $policy = $bucket->iam()->policy(['requestedPolicyVersion' => 3]);
    $policy['version'] = 3;

    $policy['bindings'][] = [
        'role' => $role,
        'members' => $members
    ];

    $bucket->iam()->setPolicy($policy);

    printf('Added the following member(s) to role %s for bucket %s' . PHP_EOL, $role, $bucketName);
    foreach ($members as $member) {
        printf('    %s' . PHP_EOL, $member);
    }
}

Python

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage Python API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

from google.cloud import storage


def add_bucket_iam_member(bucket_name, role, member):
    """Add a new member to an IAM Policy"""
    # bucket_name = "your-bucket-name"
    # role = "IAM role, e.g., roles/storage.objectViewer"
    # member = "IAM identity, e.g., user: [e-mail]"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)

    policy = bucket.get_iam_policy(requested_policy_version=3)

    policy.bindings.append({"role": role, "members": {member}})

    bucket.set_iam_policy(policy)

    print(f"Added {member} with role {role} to {bucket_name}.")

Ruby

To learn how to install and use the client library for Cloud Storage, see Cloud Storage client libraries. For more information, see the Cloud Storage Ruby API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for client libraries.

def add_bucket_iam_member bucket_name:
  # The ID of your GCS bucket
  # bucket_name = "your-unique-bucket-name"

  require "google/cloud/storage"

  storage = Google::Cloud::Storage.new
  bucket = storage.bucket bucket_name

  role   = "roles/storage.objectViewer"
  member = "group:[e-mail]"

  bucket.policy requested_policy_version: 3 do |policy|
    policy.bindings.insert role: role, members: [member]
  end

  puts "Added #{member} with role #{role} to #{bucket_name}"
end

JSON

Note: This method replaces any existing IAM policy set on a bucket. If you want to modify the existing policy, retrieve it with the getIamPolicy method, edit it as needed, and use the edited version in your setIamPolicy request.

1. Have gcloud CLI installed and initialized, which lets you generate an access token for the Authorization header.

2. Create a JSON file that contains the following information:

   {
   "bindings":[
     {
       "role": "roles/storage.objectViewer",
       "members":[
         "YOUR_AGENT_EMAIL"
       ]
     },
     {
       "role": "roles/storage.legacyBucketReader",
       "members":[
         "YOUR_AGENT_EMAIL"
       ]
     }
   ]
   }

   Where:

   • YOUR_AGENT_EMAIL is the agent account email you copied in Find the service agent's email.

3. Use cURL to call the JSON API with a PUT setIamPolicy request:

   curl -X PUT --data-binary @JSON_FILE_NAME \
   -H "Authorization: Bearer OAUTH2_TOKEN" \
   -H "Content-Type: application/json" \
   "https://storage.googleapis.com/storage/v1/b/BUCKET_NAME/iam"

   Where:

   • JSON_FILE_NAME is the path for the file that you created in Step 2.
   • OAUTH2_TOKEN is the access token you generated in Step 1.
   • BUCKET_NAME is the name of the bucket to which you want to give the principal access. For example, my-bucket.