Differences between Storage Transfer Service options

This document summarizes the differences between file system transfers and transfers from cloud object storage sources (Amazon S3, Azure Storage, and Cloud Storage) or URL lists to Cloud Storage.

Feature Cloud object storage to Cloud Storage File system transfers Notes
Agents and agent pools No Required

Transfers involving file systems require agents with root access to the file system. Agents are grouped together and managed with agent pools.

See Manage agent pools to get started.

Sync Options to overwrite objects that exist in Cloud Storage No overwrite option

Storage Transfer Service uses checksum metadata, when possible, to detect changes between objects in the source storage system and Cloud Storage. For file systems, Storage Transfer Service uses the source object's last modified time and size and compares those to the last modified time and size when the object was last copied to Cloud Storage.

Cloud object storage transfers

When Storage Transfer Service detects a new or changed object on the source storage system, it copies the complete object to Cloud Storage. You can change this behavior, so that Storage Transfer Service overwrites objects with the same name that exist in Cloud Storage regardless of changed state. For more information, see TransferOptions.

File system transfers

When Storage Transfer Service detects a new or changed object on the file system, it copies the complete object to Cloud Storage. This behavior isn't changeable. To copy files that exist in Cloud Storage, either delete the corresponding objects in the destination Cloud Storage bucket, or choose a new prefix for the destination objects.

Bandwidth control Not supported Supported—can set limits across all transfers in the project in MB/s increments
Google Cloud's operations suite monitoring Not supported Supported for agents only For Cloud-to-Google Cloud, you can poll the API to get the status, speed, and so forth of your transfer jobs. For more information, see TransferJobs API description.
Transfer logs Not supported Supported—you can view a record of all files copied and any errors. For errors in Cloud-to-Google Cloud transfers you can view the error samples in Google Cloud console.
Name-based source data filtering Supported—include and exclude prefixes for source objects Not supported To specify a subset of files for a file system transfer, you create a separate staging directory with only the files you intend to transfer, and then start a transfer job with that directory as the source.
Modified-time source-data filtering Supported—include source files based on last modified time Not supported To specify a subset of files for a file system transfer, you create a separate staging directory with only the files you intend to transfer, and then start a transfer job with that directory as the source.
IAM
  • Storage Transfer Admin, Storage Transfer User, and Storage Transfer Viewer roles
  • Custom IAM permissions
Storage Transfer Admin and Storage Transfer User roles The Cloud console for Storage Transfer Service will not work properly if the custom role is missing required permissions. For example, some parts of the Cloud console assume the role has read access to display an item before editing it, so a role with only write permissions will experience Cloud console screens that don't work.