Use cross-bucket replication

This page describes how to use cross-bucket replication, which uses Storage Transfer Service to copy new and updated objects asynchronously from a source bucket to a destination bucket. When you use cross-bucket replication, you create and manage replication jobs, which are a type of job in Storage Transfer Service.

Before you begin

To use cross-bucket replication, you must complete prerequisite steps. For instructions, refer to Before you begin in the Storage Transfer Service documentation.

Create a replication job

Console

When you create a cross-bucket replication job using the Google Cloud console, the Storage Transfer Service service agent is automatically granted the storage.legacyBucketOwner role on the source bucket and the storage.legacyBucketWriter role on the destination bucket.

When using the Google Cloud console, you can create a replication job for existing buckets or for new buckets during the bucket creation process.

To create a replication job for a new bucket, follow the instructions for creating a new bucket.

To create a replication job for an existing bucket, complete the following steps:

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket whose objects you want to replicate.

  3. On the Bucket details page, click the Configuration tab.

  4. Locate the Cross-bucket replication option and click Edit.

  5. In the Edit cross-bucket replication pane that opens, click Add a destination.

  6. In the Choose a destination section, select a destination bucket, then click Next.

  7. In the Choose replication settings section, do the following:

    • Optional: To filter objects to replicate by object name prefix, select the Replicate objects based on prefix checkbox in the Choose which objects to replicate section.

      • To include objects by prefix, enter a prefix in the Include objects with prefix section, then click Add a prefix.

      • To exclude objects by prefix, enter a prefix in the Exclude objects with prefix section, then click Add a prefix.

    • Optional: To set a storage class for replicated objects, select a storage class from the menu in the Set storage class for replicated objects section.

      If you skip this step, replicated objects use the destination bucket's storage class by default.

    1. Click Save.

Command line

For Google Cloud CLI instructions on how to create a replication job, refer to Create a replication job using the Google Cloud CLI.

REST APIs

JSON API

For JSON API instructions on how to create a replication job, refer to Create a replication job using the JSON API.

To check the status of the replication job, view Cloud Logging for Storage Transfer Service logs.

List replication jobs

Console

You cannot list replication jobs using the Google Cloud console. See View replication job for instructions on how to view a single replication job at a time.

Command line

For Google Cloud CLI instructions on how to list replication jobs, refer to List replication jobs using the Google Cloud CLI.

REST APIs

JSON API

For JSON API instructions on how to list replication jobs, refer to List replication jobs using the JSON API.

View a replication job

Console

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket whose cross-bucket replication job you want to view.

  3. On the Bucket details page, click the Configuration tab.

  4. Locate the Cross-bucket replication option and click Edit.

    The Edit cross-bucket replication pane appears, which displays the replication job for each destination bucket.

On the Buckets page, you can view the Replication column, which displays whether a bucket has a Turbo replication job or a cross-bucket replication job running. For instructions on displaying the Replication column, see Show columns.

Command line

For Google Cloud CLI instructions on how to view a replication job, refer to View a replication job using the Google Cloud CLI.

REST APIs

JSON API

For JSON API instructions on how to view a replication job, refer to View a replication job using the JSON API.

Update a replication job

You can update the following fields of a replication job:

  • The description of the replication job

  • The configuration for running a replication job

  • The configuration of notifications published to Pub/Sub

  • The logging behavior for replication job operations

  • The status of the replication job (whether it's enabled, disabled, or deleted)

Console

When using the Google Cloud console, you can only update a replication job by pausing or unpausing the job.

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket that you want to pause or unpause replicating.

  3. On the Bucket details page, click the Configuration tab.

  4. Locate the Cross-bucket replication option and click Edit.

  5. In the Edit cross-bucket replication pane that appears, click Pause or Unpause next to the replication job you want to update.

Command line

For Google Cloud CLI instructions on how to update a replication job, refer to Update a replication job using the Google Cloud CLI.

REST APIs

JSON API

For JSON API instructions on how to update a replication job, refer to Update a replication job using the JSON API.

Delete a replication job

Console

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click the name of the source bucket you want to stop replicating.

  3. On the Bucket details page, click the Configuration tab.

  4. Locate the Cross-bucket replication option and click Edit.

  5. In the Edit cross-bucket replication pane that appears, click Delete next to the replication job you want to delete.

  6. In the dialogue that appears, click Confirm.

Command line

For Google Cloud CLI instructions on how to delete a replication job, refer to Delete a replication job using the Google Cloud CLI.

REST APIs

JSON API

For JSON API instructions on how to delete a replication job, refer to Delete a replication job using the JSON API.