Cloud Storage Transfer Service enables you to import large amounts of online data into Google Cloud Storage quickly and cost-effectively. To use Cloud Storage Transfer Service, you set up a transfer from a data source to data sink. Data sources can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or another Google Cloud Storage bucket. Data sinks are always a Google Cloud Storage bucket.
Example uses of Cloud Storage Transfer Service include:
Backing up data to a Google Cloud Storage bucket from other storage providers.
Moving data from a Standard Storage bucket to a Nearline Storage bucket to lower your storage costs.
Cloud Storage Transfer Service has options that make data transfers and synchronization between data sources and data sinks easier. For example, you can:
Schedule one-time transfers or recurring transfers.
Schedule periodic synchronization from data source to data sink with advanced filters based on file creation dates, file-name filters, and the times of day you prefer to import data.
Delete source objects after transferring them.
There are a number of ways you can work with Cloud Storage Transfer Service:
Use the Google Developers Console UI to create and manage transfers. This is often the easiest and quickest way to get started with Cloud Storage Transfer Service. For more information, see Using the Developers Console.
Use a Google APIs Client Library in a language of your choice. See the Developer's Guide.
If you are comfortable with REST interfaces, you can work directly with the Storage Transfer Service API. See Creating A Client for information about enabling the API and getting authentication tokens to use in your requests.
You can use
gsutil, a flexible command-line tool for working with
Google Cloud Storage. With
gsutil you can also work with Amazon S3 buckets, and
transfer data from Amazon S3 to Google Cloud Storage using a daisy chain approach. For
more information, see the help for the
-D flag of the
For less than 1 TB of data to transfer, you should use
gsutil. For greater
than 10 TB of data to transfer, use should use Storage Transfer Service. For values
values between 1 and 10 TB, either tool is appropriate. Use this guidance as a
starting point, but keep in mind that the specifics of your transfer scenario
will determine which tool is more appropriate.
The rest of this page discusses concepts that are independent of how you choose to work with Cloud Storage Transfer Service.
Cloud Storage Transfer Service uses a Google Service Account to access Google Cloud Storage buckets in your Google Developers Console project. For example, a service account looks like
<accountID> is specific to your project.
The service account must be granted write access to any bucket that is designated as a data sink in a transfer. For information about how to add the service to access your bucket, see Setting Bucket Permissions.
Depending on your transfer data source, take one of the following actions.
- Google Cloud Storage bucket
If your source data is another Google Cloud Storage bucket and the transfer does not delete from the source, then grant the service account the
Can Viewrole. If the transfer deletes from the source, then grant the service account the
Can Editrole. For more information, see Project Members and Permissions.
- Amazon S3 bucket
If your source data is an Amazon S3 bucket, then set up an AWS Identity and Access Management (IAM) user so that you give the user the ability to list the Amazon S3 bucket, get the location of the bucket, and read the objects in the bucket. You must configure the AWS IAM user using Amazon S3 tools such as the AWS Management Console. For more information, see Creating an IAM User in Your AWS Account and Bucket Policy Examples.
These AWS IAM user credentials (access/secret key) must be entered into Storage Transfer Service when you configure a transfer from Amazon S3. We recommend that you create an access/secret key pair for each transfer or one for a group of transfers at most. Avoid using an access/secret key pair that can access all resources of the AWS account.
- HTTP URLs
If your source data is a list of HTTP URLs, then the objects that the URLs point to must allow public access.
Charges related to using Storage Transfer Service include:
- When transferring data from an external source into Google Cloud Storage, you may incur egress
and operation charges based on the pricing policy of the source provider. For example, when moving
data from Amazon Simple Storage Service (Amazon S3) to Google Cloud Storage, the pricing in the Amazon S3 Pricing page applies for requests and
data transferred out.
- When transferring data from one Google Cloud Storage bucket to another,
you may incur transfer charges for transferring between buckets
in different locations. For more information see
Network Pricing. In addition,
early deletion from Google Cloud Storage Nearline will incur a cost. For more information see
Google Cloud Storage Nearline pricing.
- When you use Storage Transfer Service, operation charges apply for managing objects in buckets both in Google Cloud Storage and storage providers external to Google. For example, a transfer operation from an external provider into Google Cloud Storage might need to list bucket contents in both the source and destination locations. For more information, see Operation Pricing and the appropriate pricing page for the source provider.