Hide
Cloud Storage

Cloud Storage Transfer Service

Cloud Storage Transfer Service enables you to import large amounts of online data into Google Cloud Storage quickly and cost-effectively. To use Cloud Storage Transfer Service, you set up a transfer from a data source to data sink. Data sources can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or another Google Cloud Storage bucket. Data sinks are always a Google Cloud Storage bucket.

Example uses of Cloud Storage Transfer Service include:

  • Backing up data to a Google Cloud Storage bucket from other storage providers.

  • Moving data from a Standard Storage bucket to a Nearline Storage bucket to lower your storage costs.

Cloud Storage Transfer Service has options that make data transfers and synchronization between data sources and data sinks easier. For example, you can:

  • Schedule one-time transfers or recurring transfers.

  • Schedule periodic synchronization from data source to data sink with advanced filters based on file creation dates, file-name filters, and the times of day you prefer to import data.

  • Delete source objects after transferring them.

There are a number of ways you can work with Cloud Storage Transfer Service:

  • Use the Google Developers Console UI to create and manage transfers. This is often the easiest and quickest way to get started with Cloud Storage Transfer Service. For more information, see Using the Developers Console.

  • Use a Google APIs Client Library in a language of your choice. See the Developer's Guide.

  • If you are comfortable with REST interfaces, you can work directly with the Storage Transfer Service API. See Creating A Client for information about getting authentication tokens to use in your requests.

You can use gsutil, a flexible command-line tool for working with Google Cloud Storage. With gsutil you can also work with Amazon S3 buckets, and transfer data from Amazon S3 to Google Cloud Storage using a daisy chain approach. For more information, see the help for the -D flag of the gsutil cp command.

For less than 1 TB of data to transfer, you should use gsutil. For greater than 10 TB of data to transfer, use should use Storage Transfer Service. For values values between 1 and 10 TB, either tool is appropriate. Use this guidance as a starting point, but keep in mind that the specifics of your transfer scenario will determine which tool is more appropriate.

The rest of this page discusses concepts that are independent of how you choose to work with Cloud Storage Transfer Service.

Configuring Access

Cloud Storage Transfer Service uses a Google Service Account to access Google Cloud Storage buckets in your Google Developers Console project. For example, a service account looks like

storage-transfer-<accountID>@partnercontent.gserviceaccount.com

where <accountID> is specific to your project.

The service account must be granted write access to any bucket that is designated as a data sink in a transfer. For information about how to add the service to access your bucket, see Setting Bucket Permissions.

Depending on your transfer data source, take one of the following actions.

Google Cloud Storage bucket

If your source data is another Google Cloud Storage bucket and the transfer does not delete from the source, then grant the service account the Can View role. If the transfer deletes from the source, then grant the service account the Can Edit role. For more information, see Project Members and Permissions.

Amazon S3 bucket

If your source data is an Amazon S3 bucket, then set up an AWS Identity and Access Management (IAM) user so that you give the user the ability to list the Amazon S3 bucket, get the location of the bucket, and read the objects in the bucket. You must configure the AWS IAM user using Amazon S3 tools such as the AWS Management Console. For more information, see Creating an IAM User in Your AWS Account and Bucket Policy Examples.

These AWS IAM user credentials (access/secret key) must be entered into Storage Transfer Service when you configure a transfer from Amazon S3. We recommend that you create an access/secret key pair for each transfer or one for a group of transfers at most. Avoid using an access/secret key pair that can access all resources of the AWS account.

HTTP URLs

If your source data is a list of HTTP URLs, then the objects that the URLs point to must allow public access.

Pricing

Storage Transfer Service is free to use; however, there are charges related to using the service:

  • When transferring data from an external source into Google Cloud Storage, you may incur egress and operation charges based on the pricing policy of the source provider. For example, when moving data from Amazon Simple Storage Service (Amazon S3) to Google Cloud Storage, the pricing in the Amazon S3 Pricing page applies for requests and data transferred out.

  • When transferring data from one Google Cloud Storage bucket to another, you may incur transfer charges for transferring between buckets in different locations. For more information see Network Pricing. In addition, early deletion from Google Cloud Storage Nearline will incur a cost. For more information see Google Cloud Storage Nearline pricing.

  • When you use Storage Transfer Service, operation charges apply for managing objects in buckets both in Google Cloud Storage and storage providers external to Google. For example, a transfer operation from an external provider into Google Cloud Storage might need to list bucket contents in both the source and destination locations. For more information, see Operation Pricing and the appropriate pricing page for the source provider.