This document summarizes the differences between file system transfers and transfers from cloud object storage sources (Amazon S3, Azure Storage, and Cloud Storage) or URL lists to Cloud Storage.
| Feature | Cloud object storage to Cloud Storage | File system transfers | Notes |
|---|---|---|---|
| Agents and agent pools | No | Required | Transfers involving file systems require agents with root access to the file system. Agents are grouped together and managed with agent pools. See Manage agent pools to get started. |
| Sync | Options to overwrite objects that exist in Cloud Storage | No overwrite option | Storage Transfer Service uses checksum metadata, when possible, to detect changes between objects in the source storage system and Cloud Storage. For file systems, Storage Transfer Service uses the source object's last modified time and size and compares those to the last modified time and size when the object was last copied to Cloud Storage. Cloud object storage transfers When Storage Transfer Service detects a new or changed object on the source storage system, it copies the complete object to Cloud Storage. You can change this behavior, so that Storage Transfer Service overwrites objects with the same name that exist in Cloud Storage regardless of changed state. For more information, see TransferOptions. File system transfers When Storage Transfer Service detects a new or changed object on the file system, it copies the complete object to Cloud Storage. This behavior isn't changeable. To copy files that exist in Cloud Storage, either delete the corresponding objects in the destination Cloud Storage bucket, or choose a new prefix for the destination objects. |
| Bandwidth control | Not supported | Supported—can set limits across all transfers in the project in MB/s increments | |
| Google Cloud's operations suite monitoring | Not supported | Supported for agents only | For Cloud-to-Google Cloud, you can poll the API to get the status, speed, and so forth of your transfer jobs. For more information, see TransferJobs API description. |
| Transfer logs | Supported. See Configure transfer logs. | Supported. View a record of all files copied and any errors. See Transfer log format. | |
| Name-based source data filtering | Supported—include and exclude prefixes for source objects | Not supported | To specify a subset of files for a file system transfer, you create a separate staging directory with only the files you intend to transfer, and then start a transfer job with that directory as the source. |
| Modified-time source-data filtering | Supported—include source files based on last modified time | Not supported | To specify a subset of files for a file system transfer, you create a separate staging directory with only the files you intend to transfer, and then start a transfer job with that directory as the source. |
| IAM |
|
Storage Transfer Admin and Storage Transfer User roles | The Cloud console for Storage Transfer Service will not work properly if the custom role is missing required permissions. For example, some parts of the Cloud console assume the role has read access to display an item before editing it, so a role with only write permissions will experience Cloud console screens that don't work. |