This page describes Storage Transfer Service, which you can use to quickly import online data into Cloud Storage. You can also transfer data within Cloud Storage, from one bucket to another.
To learn how to set up a transfer in the GCP Console, follow the instructions in Creating and Managing Transfers with the Console.
What is Storage Transfer Service?
Storage Transfer Service transfers data from an online data source to a data sink. Your data source can be an Amazon Simple Storage Service (Amazon S3) bucket, an HTTP/HTTPS location, or a Cloud Storage bucket. Your data sink (the destination) is always a Cloud Storage bucket.
You can use Storage Transfer Service to:
- Back up data to a Cloud Storage bucket from other storage providers.
- Move data from a Multi-Regional Storage bucket to a Nearline Storage bucket to lower your storage costs.
Storage Transfer Service performs a data transfer with a transfer operation. Transfer operations are scheduled and configured through a transfer job. Storage Transfer Service has options that make data transfers and synchronization between data sources and data sinks easier. For example, you can:
- Schedule one-time transfer operations or recurring transfer operations.
- Delete existing objects in the destination bucket if they don't have a corresponding object in the source.
- Delete source objects after transferring them.
- Schedule periodic synchronization from data source to data sink with advanced filters based on file creation dates, file-name filters, and the times of day you prefer to import data.
By default, Storage Transfer Service copies a file from the data source if the file doesn't exist in the data sink or if it differs between the version in the source and the sink. The default is also to retain files in the source after the transfer operation.
Who can use Storage Transfer Service?
In order to have full access to Storage Transfer Service, you must be the EDITOR
or OWNER of the project that creates the transfer job. If you are a VIEWER
of the project, you can view and list transfer jobs and transfer operations
associated with the data sink.
The project that creates the transfer job does not have to be associated with the buckets that act as the source or sink, but additional permissions are required to set up and use sources and sinks. See Configuring Access to Data Sources and Sinks for information on how to do this.
Available interfaces
There are a number of ways you can work with Storage Transfer Service:
- Use the Google Cloud Platform Console UI to create and manage transfer jobs. This is often the easiest and quickest way to get started with Storage Transfer Service. For more information, see Creating and Managing Transfers with the Console.
- Use a Google API Client Library in a language of your choice. See Creating a Storage Transfer Service Client.
- If you are comfortable with REST APIs, you can work directly with the Storage Transfer Service API. See Creating a Storage Transfer Service Client for information about enabling the API and getting authentication tokens to use in your requests.
Should you use gsutil or Storage Transfer Service?
The gsutil command-line tool also enables you to transfer data between Cloud Storage and other locations. While you can use gsutil to work with Amazon S3 buckets and transfer data from Amazon S3 to Cloud Storage, Storage Transfer Service is recommended for this use case.
Follow these rules of thumb when deciding whether to use gsutil or Storage Transfer Service:
- When transferring data from an on-premises location, use gsutil.
- When transferring data from another cloud storage provider, use Storage Transfer Service.
- Otherwise, evaluate both tools with respect to your specific scenario.
Use this guidance as a starting point. The specific details of your transfer scenario will also help you determine which tool is more appropriate.
