Filtering objects from transfers

This page shows you how to include and exclude paths from transfers using includePrefix and excludePrefix. To learn how to create a manifest of specific objects to transfer, see Transfer specific files or objects using a manifest.

You can use includePrefix and excludePrefix when creating transfers programmatically to limit what objects Storage Transfer Service operates on. includePrefix and excludePrefix are optional fields that you can specify in your TransferSpec when creating a transfer. An includePrefix specifies particular paths to include for a transfer, and an excludePrefix specifies particular paths to exclude.

If a path includes both objects that you do and don't want to transfer, you can use includePrefix and excludePrefix together to specify which objects that you want to transfer.

Example objects and paths

The examples in this document use the following sample objects and paths:

object_1
object_2
path_1/object_3/
path_2/object_4
 path_1/subpath_1/object_5
path_1/subpath_2/object_6
path_2/subpath_3/object_7
path_2/subpath_4/object_8

Including paths

You can use includePrefix when creating a transfer to instruct Storage Transfer Service to consider objects in the listed paths for transfer, and to ignore objects not under those paths.

For example, to include objects under path_1/, pass includePrefix the following:

includePrefix: "path_1/"

This includes objects directly under path_1/, path_1/subpath_1, and path_1/subpath_2/. The following objects are included in the transfer:

path_1/object_3
 path_1/subpath_1/object_5
path_1/subpath_2/object_6

You can also specify particular paths to include. For example, you can pass includePrefix the following:

includePrefix: "path_1/subpath_2/", "path_1/subpath_3/"

In this case, the transfer includes the following objects:

path_1/subpath_2/object_6
 path_2/subpath_3/object_7

When you use includePrefix, paths that you don't specifically include aren't transferred to the Cloud Storage destination bucket.

Excluding paths

Using excludePrefix when creating a transfer instructs Storage Transfer Service to ignore the listed paths for transfer.

To exclude objects under path_1/, pass excludePrefix the following:

excludePrefix: "path_1/"

This excludes objects under path_1/, path_1/subpath_1/, and path_1/subpath_2/. In this case, the following objects are included in the transfer:

object_1
object_2
 path_2/object_4
path_2/subpath_3/object_7
path_2/subpath_4/object_8

You can also specify specific paths to exclude. For example, you can pass excludePrefix the following:

excludePrefix: "path_1/subpath_2/", "path_2/subpath_3/"

In this case, the transfer includes the following objects:

object_1
object_2
path_1/object_3
path_2/object_4
 path_1/subpath_1/object_5
path_2/subpath_4/object_8

Including and excluding paths simultaneously

You can apply an excludePrefix and an includePrefix together, in which case the excludePrefix further limits what the includePrefix includes in the transfer.

When specifying both includePrefix and excludePrefix, each excludePrefix must start with any of the values you specified for includePrefix.

For example, to include objects under path_1/ and exclude objects under subpath_1/ pass the following:

includePrefix: "path_1/"
excludePrefix: "path_1/subpath_1/"

In this case, the transfer includes the following objects:

path_1/object_3
 path_1/subpath_2/object_6

To include all objects under path_1/ and path_2/, except items in either path_1/subpath_1/ or path_2/subpath_3/, pass the following:

includePrefix: "path_1/", "path_2/"
excludePrefix: "path_1/subpath_1/", "path_2/subpath_3/"

In this case, the transfer includes the following objects:

path_1/object_3
path_2/object_4
 path_1/subpath_2/object_6
path_2/subpath_4/object_8

Examples of incorrect path inclusion or exclusion

The following sections include examples to avoid when using include or exclude paths, and how to correct them so that they work correctly.

Including a path that is used in another include-prefix

Each includePrefix should specify a distinct portion of the object namespace. The following example is incorrect because the second value is already included in the namespace of the first value:

includePrefix: "path_1/", "path_1/subpath_1"

In this example, the includePrefix values are invalid because the second includePrefix, "path_1/subpath_1", is already included within "path_1/". To fix this, choose to include either one or the other:

  • includePrefix: "path_1"
  • includePrefix: "path_1/subpath_1"

Using an exclude-prefix that doesn't start with an include-prefix

Each excludePrefix should start with any of the specified includePrefix values. The following example is incorrect because the excludePrefix values don't start with the specified includePrefix values:

includePrefix: "path_1/", "path_2/"
excludePrefix: "subpath_1/", "subpath_4/"

In this example, the excludePrefix values are invalid because they don't start with either of the includePrefix values, path_1 or path_2. To fix this, ensure that the excludePrefix includes a full path listed in includePrefix:

includePrefix: "path_1/", "path_2/"
excludePrefix: "path_1/subpath_1/", "path_2/subpath_4/"

What's next