This page describes Storage Transfer Service, which you can use to quickly import online data into Cloud Storage. You can also transfer data within Cloud Storage, from one bucket to another.
To learn how to set up a transfer in the GCP Console, follow the instructions in Creating and Managing Transfers with the Console.
What is Storage Transfer Service?
Storage Transfer Service transfers data from an online data source to a data sink. Your data source can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or a Cloud Storage bucket. Your data sink (the destination) is always a Cloud Storage bucket.
You can use Storage Transfer Service to:
- Back up data to a Cloud Storage bucket from other storage providers.
- Move data from a Multi-Regional Storage bucket to a Nearline Storage bucket to lower your storage costs.
Storage Transfer Service performs a data transfer with a transfer operation. Transfer operations are scheduled and configured through a transfer job. Storage Transfer Service has options that make data transfers and synchronization between data sources and data sinks easier. For example, you can:
- Schedule one-time transfer operations or recurring transfer operations.
- Delete existing objects in the destination bucket if they don't have a corresponding object in the source.
- Delete source objects after transferring them.
- Schedule periodic synchronization from data source to data sink with advanced filters based on file creation dates, file-name filters, and the times of day you prefer to import data.
By default, Storage Transfer Service copies a file from the data source if the file doesn't exist in the data sink or if it differs between the version in the source and the sink. The default is also to retain files in the source after the transfer operation.
Who can use Storage Transfer Service?
In order to have full access to Storage Transfer Service, you must be the
OWNER of the project that creates the transfer job. If you are a
of the project, you can view and list transfer jobs and transfer operations
associated with the data sink.
The project that creates the transfer job does not have to be associated with the buckets that act as the source or sink, but additional permissions are required to set up and use sources and sinks. See Configuring Access to Data Sources and Sinks for information on how to do this.
There are a number of ways you can work with Storage Transfer Service:
- Use the Google Cloud Platform Console UI to create and manage transfer jobs. This is often the easiest and quickest way to get started with Storage Transfer Service. For more information, see Creating and Managing Transfers with the Console.
- Use a Google API Client Library in a language of your choice. See Creating a Storage Transfer Service Client.
- If you are comfortable with REST APIs, you can work directly with the Storage Transfer Service API. See Creating a Storage Transfer Service Client for information about enabling the API and getting authentication tokens to use in your requests.
Should you use gsutil or Storage Transfer Service?
The gsutil command-line tool also enables you to transfer data between Cloud Storage and other locations. While you can use gsutil to work with Amazon S3 buckets and transfer data from Amazon S3 to Cloud Storage, Storage Transfer Service is recommended for this use case.
Follow these rules of thumb when deciding whether to use gsutil or Storage Transfer Service:
- When transferring data from an on-premises location, use gsutil.
- When transferring data from another cloud storage provider, use Storage Transfer Service.
- Otherwise, evaluate both tools with respect to your specific scenario.
Use this guidance as a starting point. The specific details of your transfer scenario will also help you determine which tool is more appropriate.
There are no extra costs for using the Storage Transfer Service; however, normal Cloud Storage and external provider costs apply when using the Storage Transfer Service. For example:
When transferring data from an external source into Cloud Storage, you may incur egress and operation charges based on the pricing policy of the source provider. For example, when moving data from Amazon Simple Storage Service (Amazon S3) to Cloud Storage, the pricing in the Amazon S3 Pricing page applies for requests and data transferred out of Amazon S3.
When transferring data from a Cloud Storage bucket to another Cloud Storage bucket in a different location, you incur transfer charges. For more information, see Network Pricing.
When transferring data from any source into a Cloud Storage bucket, you incur a
PUToperation charge for each object added to the bucket. For more information, see Operations Pricing.
When transferring Nearline Storage or Coldline Storage data from one Cloud Storage bucket to another, you incur data retrieval costs, and you incur early deletion fees if they apply. For more information see Nearline Storage and Coldline Storage pricing.
When you use the Storage Transfer Service, operation charges apply for any object management that occurs, both in Cloud Storage and in external storage providers external. For example, a transfer operation from an external provider into Cloud Storage may need to list the contents of both the source and destination buckets. For more information, see Operation Pricing for Cloud Storage or the appropriate pricing page for the source provider.