This page describes Cloud Storage Transfer Service, which you can use to quickly import online data into Google Cloud Storage. You can also transfer data within Google Cloud Storage, from one bucket to another.
To learn how to set up a transfer in the Cloud Platform Console, follow the instructions in Creating and Managing Transfers with the Console.
What is Cloud Storage Transfer Service?
Cloud Storage Transfer Service transfers data from a data source to a data sink. Your data source can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or a Google Cloud Storage bucket. Your data sink (the destination) is always a Google Cloud Storage bucket.
You can use Cloud Storage Transfer Service to:
- Back up data to a Google Cloud Storage bucket from other storage providers.
- Move data from a Multi-Regional Storage bucket to a Nearline Storage bucket to lower your storage costs.
Cloud Storage Transfer Service has options that make data transfers and synchronization between data sources and data sinks easier. For example, you can:
- Schedule one-time transfers or recurring transfers.
- Delete existing objects in the destination bucket if they don't have a corresponding object in the source.
- Delete source objects after transferring them.
- Schedule periodic synchronization from data source to data sink with advanced filters based on file creation dates, file-name filters, and the times of day you prefer to import data.
By default, Cloud Storage Transfer Service copies a file from the data source if the file doesn't exist in the the data sink or if it differs between the version in the source and the sink. The default is also to retain files in the source after the transfer.
There are a number of ways you can work with Cloud Storage Transfer Service:
- Use the Google Cloud Platform Console UI to create and manage transfers. This is often the easiest and quickest way to get started with Cloud Storage Transfer Service. For more information, see Creating and Managing Transfers with the Console.
- Use a Google API Client Library in a language of your choice. See Creating a Storage Transfer Service Client.
- If you are comfortable with REST APIs, you can work directly with the Storage Transfer Service API. See Creating a Storage Transfer Service Client for information about enabling the API and getting authentication tokens to use in your requests.
Should you use gsutil or Cloud Storage Transfer Service?
The gsutil command-line tool also enables you to transfer data between Cloud Storage and other locations. While you can use gsutil to work with Amazon S3 buckets and transfer data from Amazon S3 to Google Cloud Storage, Storage Transfer Service is recommended for this use case.
Follow these rules of thumb when deciding whether to use gsutil or Cloud Storage Transfer Service:
- When transferring data from an on-premise location, use gsutil.
- When transferring data from another cloud storage provider, use Storage Transfer Service.
- Otherwise, evaluate both tools with respect to your specific scenario.
Use this guidance as a starting point. The specific details of your transfer scenario will also help you determine which tool is more appropriate.
Charges related to using Storage Transfer Service include:
- When transferring data from an external source into Google Cloud Storage, you may incur egress
and operation charges based on the pricing policy of the source provider. For example, when moving
data from Amazon Simple Storage Service (Amazon S3) to Google Cloud Storage, the pricing in the Amazon S3 Pricing page applies for requests and
data transferred out.
- When transferring data from any source into Google Cloud Storage, you will incur charges for insertion of
each object into Google Cloud Storage (a
PUToperation). For more information, see Operation Pricing.
- When transferring data from one Cloud Storage bucket to another, you may incur transfer
charges for transferring between buckets in different locations. For more information, see
Network Pricing. In addition,
early deletion from Google Cloud Storage Nearline or Google Cloud Storage Coldline incurs a cost. For more information see
Nearline Storage and Coldline Storage
- When you use Storage Transfer Service, operation charges apply for managing objects in buckets both in Google Cloud Storage and storage providers external to Google. For example, a transfer operation from an external provider into Google Cloud Storage might need to list bucket contents in both the source and destination locations. For more information, see Operation Pricing and the appropriate pricing page for the source provider.