This page describes Cloud Storage Transfer Service, which you can use to quickly import online data into Google Cloud Storage. You can also transfer data within Google Cloud Storage, from one bucket to another.
To learn how to set up a transfer in the Cloud Platform Console, follow the instructions in Creating and Managing Transfers with the Console.
What is Cloud Storage Transfer Service?
Cloud Storage Transfer Service transfers data from a data source to a data sink. Your data source can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or a Google Cloud Storage bucket. Your data sink (the destination) is always a Google Cloud Storage bucket.
You can use Cloud Storage Transfer Service to:
- Back up data to a Google Cloud Storage bucket from other storage providers.
- Move data from a Standard Storage bucket to a Nearline Storage bucket to lower your storage costs.
Cloud Storage Transfer Service has options that make data transfers and synchronization between data sources and data sinks easier. For example, you can:
- Schedule one-time transfers or recurring transfers.
- Schedule periodic synchronization from data source to data sink with advanced filters based on file creation dates, file-name filters, and the times of day you prefer to import data.
- Delete source objects after transferring them.
There are a number of ways you can work with Cloud Storage Transfer Service:
- Use the Google Cloud Platform Console UI to create and manage transfers. This is often the easiest and quickest way to get started with Cloud Storage Transfer Service. For more information, see Creating and Managing Transfers with the Console.
- Use a Google API Client Library in a language of your choice. See Creating a Storage Transfer Service Client.
- If you are comfortable with REST APIs, you can work directly with the Storage Transfer Service API. See Creating a Storage Transfer Service Client for information about enabling the API and getting authentication tokens to use in your requests.
Should you use gsutil or Cloud Storage Transfer Service?
The gsutil command-line tool also enables you to transfer data between
Cloud Storage and other locations. You can even use gsutil to work with
Amazon S3 buckets and transfer data from Amazon S3 to Google Cloud Storage using a
daisy-chain approach. (For more information, see the help for the
-D flag of
gsutil cp command.)
Follow these rules of thumb when deciding whether to use gsutil or Cloud Storage Transfer Service:
- When transferring less than 1 TB of data, use gsutil.
- When transferring more than 10 TB of data, use Storage Transfer Service.
- For between 1 and 10 TB of data, use either gsutil or Storage Transfer Service.
Use this guidance as a starting point. The specific details of your transfer scenario will also help you determine which tool is more appropriate.
Charges related to using Storage Transfer Service include:
- When transferring data from an external source into Google Cloud Storage, you may incur egress
and operation charges based on the pricing policy of the source provider. For example, when moving
data from Amazon Simple Storage Service (Amazon S3) to Google Cloud Storage, the pricing in the Amazon S3 Pricing page applies for requests and
data transferred out.
- When transferring data from any source into Google Cloud Storage, you will incur charges for insertion of
each object into Google Cloud Storage (a
PUToperation). For more information, see Operation Pricing.
- When transferring data from one Cloud Storage bucket to another,
you may incur transfer charges for transferring between buckets
in different locations. For more information, see
Network Pricing. In addition,
early deletion from Google Cloud Storage Nearline will incur a cost. For more information see
Google Cloud Storage Nearline pricing.
- When you use Storage Transfer Service, operation charges apply for managing objects in buckets both in Google Cloud Storage and storage providers external to Google. For example, a transfer operation from an external provider into Google Cloud Storage might need to list bucket contents in both the source and destination locations. For more information, see Operation Pricing and the appropriate pricing page for the source provider.