Storage Transfer Service can listen to event notifications in AWS or Google Cloud to automatically transfer data that has been added or updated in the source location. Event-driven transfers are supported from AWS S3 or Cloud Storage to Cloud Storage.
Event-driven transfers listen to Amazon S3 Event Notifications sent to Amazon SQS for AWS S3 sources. Cloud Storage sources send notifications to a Pub/Sub subscription.
Benefits of event-driven transfers
Because event-driven transfers listen for changes to the source bucket, updates are copied to the destination in near-real time. Storage Transfer Service doesn't need to execute a list operation against the source, saving time and money.
Use cases include:
Event-driven analytics: Replicate data from AWS to Cloud Storage to perform analytics and processing.
Cloud Storage replication: Enable automatic, asynchronous object replication between Cloud Storage buckets.
Event-driven transfers with Storage Transfer Service differ from typical Cloud Storage replication by creating a copy of your data in a different bucket.
This provides benefits such as:
- Keeping development and production data in separate namespaces.
- Sharing data without providing access to the original bucket.
- Backing up to a different continent, or to an area not covered by dual-region and multi-region storage.
DR/HA setup: Replicate objects from source to backup destination in order of minutes:
- Cross-cloud backup: Create a copy of AWS S3 backup on Cloud Storage.
- Cross-region or cross-project backup: Create a copy of Cloud Storage bucket in a different region or project.
Live migration: Event-driven transfer can power low-downtime migration, on the order of minutes of downtime, as a follow up step to one-time batch migration.