Configure access to a source: Amazon S3

You can set up access to an Amazon S3 bucket using either of two methods:

Supported regions

Storage Transfer Service is currently available to transfer data from the following Amazon S3 regions:

ap-east-1 ca-central-1 me-south-1
ap-northeast-1 eu-central-1 sa-east-1
ap-northeast-2 eu-north-1 us-east-1
ap-south-1 eu-west-1 us-east-2
ap-southeast-1 eu-west-2 us-west-1
ap-southeast-2 eu-west-3 us-west-2

Required permissions

In order to use Storage Transfer Service to move data from an Amazon S3 bucket, you must have an AWS Identity and Access Management user account that has certain permissions for the bucket:

Permission Description Use
s3:ListBucket Allows Storage Transfer Service to list objects in the bucket. Always required.
s3:GetObject Allows Storage Transfer Service to read objects in the bucket. Required if you are transferring the current version of all objects. If your manifest specifies an object version, use s3:GetObjectVersion instead.
s3:GetObjectVersion Allows Storage Transfer Service to read specific versions of objects in the bucket. Required if your manifest specifies an object version. Otherwise, use s3:GetObject.
s3:GetBucketLocation Allows Storage Transfer Service to get the location of the bucket. Always required.
s3:DeleteObject Allows Storage Transfer Service to delete objects in the bucket. Required if you set deleteObjectsFromSourceAfterTransfer to true.

Access credentials

  1. Create an AWS Identity and Access Management (AWS IAM) user with a name that you can easily recognize, such as transfer-user. Ensure the name follows the AWS IAM user name guidelines (see Limitations on IAM Entities and Objects).

  2. Give the AWS IAM user the ability to do the following:

    • List the Amazon S3 bucket.
    • Get the location of the bucket.
    • Read the objects in the bucket.
    • If you plan to delete objects from the source after the objects are transferred, grant the user Delete objects permissions.
  3. Create at least one access/secret key pair for the transfer job that you plan to set up. You can also create a separate access/secret key pair for each transfer job.

    Avoid using an access/secret key pair that can access all resources of the AWS account.

Federated identity

Storage Transfer Service uses a Google-managed service account to move your data from an Amazon S3 source bucket. The service account's format is typically project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com.

  1. To find your service account's format and create the service account if it doesn't already exist, use the googleServiceAccounts.get API call.

  2. Get the subjectId for the Google-managed service account that will run your transfer.

  3. Create the following Amazon Resource Name (ARN) IAM role with AssumeRoleWithWebIdentity permissions:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Federated": "accounts.google.com"
          },
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "accounts.google.com:sub": "SUBJECT_ID"
            }
          }
        }
      ]
    }
    

    For more information about ARNs, see IAM ARNs.

  4. Add permissions that allow Storage Transfer Service to access Amazon S3 resources. To do so, attach the following policy to the ARN IAM role, which can be done through the AWS IAM Console:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
              "s3:Get*",
              "s3:List*",
              "s3:Delete*"
           ],
          "Resource": [
              "arn:aws:s3:::DOC-EXAMPLE-BUCKET",
              "ars::aws:s3:::DOC-EXAMPLE-BUCKET/*"
          ]
        }
      ]
    }
    

    For more information about Amazon S3 policies, see Bucket Policy Examples.

  5. Restore any objects that are archived to Amazon Glacier. Objects in Amazon S3 that are archived to Amazon Glacier are not accessible until they are restored. For more information, see the Migrating to Cloud Storage from Amazon Glacier white paper.

Creating an access key

These steps give an overview of the process of creating Amazon S3 access key credentials that can be used in data transfers from an Amazon S3 bucket to a Cloud Storage bucket. For detailed information, see Creating an IAM User in Your AWS Account and Bucket Policy Examples.

For information on our data retention policy for user credentials, see User credentials.

  1. Create a new user in the AWS Identity and Access Management console.

  2. Note or download the access credentials.

    The downloaded credentials contain the user name, access key ID, and secret access key. When you configure the transfer job in Cloud Storage, you only need the access key ID and secret access key.

  3. Attach a managed policy to the IAM user that contains the permissions needed to complete a transfer.

    Attach the AmazonS3FullAccess policy if your transfer job is configured to delete source objects; otherwise, attach the AmazonS3ReadyOnlyAccess policy. For example, the AmazonS3FullAccess managed policy attached to a user through the IAM console is:

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "s3:*",
          "Resource": "*"
        }
      ]
    }
    
  4. Optional: Create a policy that is more restrictive than the managed policies.

    For example, you can create a policy that limits access to just the Amazon S3 bucket. For more information, see Bucket Policy Examples.