Storage Transfer Service release notes

This page documents production updates to Storage Transfer Service. We recommend that Storage Transfer Service developers periodically check this list for any new announcements.

You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or you can programmatically access release notes in BigQuery.

To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly: https://cloud.google.com/feeds/storagetransfer-release-notes.xml

July 18, 2022

Detailed logging for objects copied between AWS S3, Azure Blob Storage, ADLS Gen 2, and Cloud Storage with Storage Transfer Service is now generally available (GA).

With detailed logs of individual objects available in Cloud Logging, you can verify what was transferred and perform additional data integrity checks. This launch simplifies monitoring, reporting, and troubleshooting. Read Cloud Logging for Storage Transfer Service for details.

June 29, 2022

Expanded overwrite options are new generally available (GA). The overwriteWhen field can be used to specify whether data that already exists in the destination should be overwritten always, never, or only when ETags and checksum values indicate that the file has changed.

Metadata preservation options are now generally available (GA). This includes the option of preserving POSIX attributes and symlinks when transferring to, from, and between POSIX filesystems; as well as object ACLs, CMEK, temporary holds, and object creation time when transferring between Cloud Storage buckets.

See Metadata preservation for details.

June 06, 2022

Storage Transfer Service now offers a merged, unified console experience for cloud and file system transfers. All transfer jobs irrespective of source can be tracked though a single interface. This launch simplifies job creation, monitoring, and troubleshooting.

April 19, 2022

Storage Transfer Service now provides more options for when to overwrite files that already exist in the destination. The new overwriteWhen field provides three options, that apply to all transfers, including those to or from file systems.

  • NEVER provides defense in depth for archival cases, where data is not intended to be overwritten. Users no longer need to rely on a retention policy to protect their data.
  • DIFFERENT uses ETags and checksum values to only overwrite a file if the contents have changed.
  • ALWAYS overwrites any existing files with the same name. Avoids LIST operations on the destination when transferring into Cloud Storage.

April 12, 2022

Storage Transfer Service now offers a predefined role to simplify permission assignment to transfer agents. The roles/storagetransfer.transferAgent role grants a minimum set of permissions required for the service to communicate with agents and eliminates the need to assign each permission individually.

The role should be granted to the user account or service account being used to authenticate the agents. See the On-premises agent account documentation for more details.

March 30, 2022

Creating and managing data transfers with the gcloud command-line tool is now generally available (GA).

You can use gcloud commands to perform agent installation, manage agent pool lifecycles, and orchestrate transfer jobs. This launch simplifies writing scripts to automate transfer workflow.

The GA launch adds support for transfers between file systems, metadata preservation, and manifests. It also introduces the gcloud transfer authorize command to inspect and grant required permissions for transfers.

March 23, 2022

Storage Transfer Service now enforces the Resource Location Restriction, which is part of the Org Policy Service. This allows an organization to define the allowed regions in which location-based Google Cloud resources, such as transfer jobs, can be created.

To learn how Storage Transfer Service chooses the region in which to run a transfer job, refer to Location of Storage Transfer Service jobs.

March 14, 2022

Between April 2, 2022 and December 31, 2022, transfers using Storage Transfer Service will not result in Cloud Storage charges for certain transfer scenarios.

For more details, refer to the Storage Transfer Service pricing page.

March 08, 2022

Storage Transfer Service now offers more control over preserving metadata when transferring between Cloud Storage buckets. Choose to retain or discard metadata including object ACLs, customer-managed encryption keys (CMEK), temporary holds, and object creation time. In addition, storage class can be set to any supported value, allowing you to change storage class at scale.

For details, refer to Metadata preservation.

March 03, 2022

Storage Transfer Service now supports Cloud Client Libraries, which are the recommended option for accessing Cloud APIs programmatically. This launch significantly reduces the amount of code you need to write; see Getting started and Migrating to the Cloud Client Library for more details.

February 08, 2022

Support for agent pools is now generally available (GA) .

You can use agent pools to create isolated groups of agents as a source or sink entity in a transfer job. This enables you to transfer data from multiple data centers and filesystems concurrently, without creating multiple projects for a large transfer spanning multiple filesystems and data centers.

January 27, 2022

Storage Transfer Service now offers Preview support for moving data between two filesystems and keeping them in sync on a periodic schedule. Transfers can be orchestrated by creating a transfer job through the API or gcloud command-line tool. See Transfer data between POSIX file systems for details.

This launch offers a managed way to migrate from a self-managed filesystem to Filestore. If you have on-premises systems generating massive amounts of data that needs to be processed in Google Cloud, you can now use Storage Transfer Service to accelerate data transfer from an on-prem filesystem to a cloud filesystem.

Storage Transfer Service now offers Preview support for preserving POSIX attributes and symlinks when transferring to, from, and between POSIX filesystems. Attributes include the user ID of the owner, the group ID of the owning group, the mode or permissions, the modification time, and the size of the file. See Metadata preservation for details.

December 13, 2021

Integration with AWS Security Token Service is now generally available (GA) for Storage Transfer Service.

Security conscious customers can use Storage Transfer Service to perform transfers from AWS S3 without passing long-term AWS S3 credentials, which have to be rotated or explicitly revoked when they are no longer needed. Refer to the Amazon S3 > Federated Identity tab when setting up access to your data source.

Creating and managing data transfers with the gcloud command-line tool is now available in Preview. You can use gcloud commands to perform agent installation, manage agent pool lifecycles, and orchestrate transfer jobs. This launch simplifies writing scripts to automate transfer workflow.

December 06, 2021

Storage Transfer Service now offers Preview support for detailed logging for objects copied between AWS S3, Azure Blob, ADLS Gen 2, and Cloud Storage. With detailed logs of individual objects available in Cloud Logging, you can verify what was transferred and perform additional data integrity checks. This launch simplifies monitoring, reporting, and troubleshooting. Read Configure transfer logs for details.

December 03, 2021

Support for transferring data from Azure ADLS Gen 2 to Cloud Storage with Storage Transfer Service is now generally available (GA).

December 02, 2021

The Storage Transfer Service API for managing on-premises transfers is now generally available (GA). Customers can use RESTful APIs to automate their on-prem to Cloud transfer workflow.

For more information, see Managing Transfer for on-premises jobs.

Storage Transfer Service now offers preview support for Manifest. You can use Manifest to transfer a specific list of objects, object versions, and files from cloud and on-premises sources. Programmatic users can use the output of an upstream operation generating a list of files and objects as an input for Storage Transfer Service to act upon.

November 04, 2021

Storage Transfer Service now offers Preview support for agent pools. You can use agent pools to create isolated groups of agents as a source or sink entity in a transfer job. This enables you to transfer data from multiple data centers and filesystems concurrently, without creating multiple projects for a large transfer spanning multiple filesystems and data centers.

November 01, 2021

Storage Transfer Service now offers Preview support for exporting data from Cloud Storage to a POSIX file system. You can use this bidirectional data movement capability to move data in and out of Cloud Storage, on-premises clusters, and edge locations including Google Distributed Cloud. For more information, see Download data from Cloud Storage.

July 26, 2021

Storage Transfer Service now offers Public Preview support for managing on-premises transfer via API. Customers can use RESTful APIs to automate their on-prem to Cloud transfer workflow.

For more information, see Managing Transfer for on-premises jobs.

July 01, 2021

Transfer service for on-premises data support for delete from source is now Generally Available. For more information, see Data consistency details.

Storage Transfer Service offers Preview for Integration with AWS Security Token Service. Security conscious customers can now use Storage Transfer Service to perform transfers from AWS S3 without passing any security credentials. This release will alleviate the security burden associated with passing long-term AWS S3 credentials, which have to be rotated or explicitly revoked when they are no longer needed. Refer Amazon Web Services (AWS) S3 Federated Identity credentials when setting up access to your data source.

June 18, 2021

Storage Transfer Service offers Preview support for transferring data from Azure ADLS Gen 2 to Cloud Storage.

June 01, 2021

As of June 1 2021 all Transfer service for on-premises projects added a per-project service account. Actions are required before September 1, 2021 to ensure minimal disruptions to your transfers. Please check your email for detailed instructions.

April 09, 2021

Obtaining the status of the latest transfer operation is in now generally available.

March 31, 2021

Storage Transfer Service support for hourly schedules and modifying a job's schedule is now generally available. For more information, see Schedule.

Storage Transfer Service support for specifying source and destination paths when creating a transfer is now generally available. For more information, see Specifying source and destination paths.

Storage Transfer Service now supports event-driven transfer workflows, a new capability that lets customers start Transfer jobs at once based on an event trigger. Customers can create Transfer jobs without specifying a schedule and start a TransferOperation using transferJobs.run from an existing TransferJob configuration.

January 20, 2021

Storage Transfer Service offers Preview support for specifying source and destination paths when creating a transfer. For more information, see Specifying source and destination paths.

January 19, 2021

Transfer service for on-premises data has updated the cost to $0.0125 per GB transferred to the destination successfully. For more information, see Transfer for on-premises pricing details.

November 09, 2020

Storage Transfer Service offers Preview support for hourly schedules and modifying a job's schedule. For more information, see Schedule.

October 13, 2020

September 01, 2020

Storage Transfer Service integration for VPC Service Controls is now generally available.

July 31, 2020

Transfers from Microsoft Azure Blob Storage are now generally available.

March 31, 2020

Beta stage integration with VPC Service Controls

Transfer service on-premises: Beta stage integration with VPC Service Controls

March 23, 2020

Transfer Service for on-premises data is now generally available. For more information, see Transfer Service for on-premises data Overview.

Storage Transfer Service now offers Beta support for transfers from Microsoft Azure Blob Storage.

March 20, 2020

Storage Transfer Service supports Pub/Sub notifications for transfer jobs for external cloud provider to Cloud Storage transfers.

March 16, 2020

Storage Transfer Service now allows you to specify files to transfer based on their last modification times.

March 09, 2020

Storage Transfer Service supports custom job IDs.

December 04, 2019

Storage Transfer Service for on-premises is now in beta.

September 05, 2019

API documentation update to clarify transferJobs API descriptions, specifically: scheduleStartDate, scheduleEndDate, and startTimeOfDay.

August 05, 2019

Updates to API documentation.

March 20, 2019

Adding new IAM roles for Storage Transfer Service.

August 06, 2018

Updated documentation for configuring access to the data sink.

December 07, 2017

Documentation updates for IAM permissions.

September 18, 2017

Documentation update for access control.

September 07, 2017

Updates to API documentation.

August 31, 2016

Documentation update that provides clearer instructions for configuring a Cloud Storage data source.

August 25, 2016

Documentation update to add information for configuring access to a data sink and supported AWS regions.

June 15, 2016

API documentation update.

April 25, 2016

Documentation updates and content reorganization.

April 04, 2016

Updates to API documentation for scheduling jobs.

February 23, 2016

Documentation update to support scheduling recurring transfers with Cloud Console.

December 12, 2015

Updated Getting Started document for changes to the Create Transfer workflow.

September 23, 2015

Several updates to API documentation.

August 06, 2015

Documentation includes instructions for enabling API for non-UI users.

August 05, 2015

API documentation now includes "Try It" sections.

July 28, 2015

Documentation update for using prefixes in Storage Transfer Service user interface.

July 22, 2015

Storage Transfer Service, a cloud-to-cloud transfer tool, launched.