This page shows you how to include and exclude paths from transfers using
includePrefix
and
excludePrefix
.
To learn how to create a manifest of specific objects to transfer, see
Transfer specific files or objects using a manifest.
You can use includePrefix
and excludePrefix
when
creating transfers programmatically to limit
what objects Storage Transfer Service operates on. includePrefix
and excludePrefix
are optional fields that you can specify in your
TransferSpec
when creating a transfer.
An includePrefix
specifies particular paths to include for a transfer, and an
excludePrefix
specifies particular paths to exclude.
If a path includes both objects that you do and don't want to transfer, you can
use includePrefix
and excludePrefix
together to specify which objects that
you want to transfer.
Example objects and paths
The examples in this document use the following sample objects and paths:
object_1 object_2 path_1/object_3/ path_2/object_4 |
path_1/subpath_1/object_5 path_1/subpath_2/object_6 path_2/subpath_3/object_7 path_2/subpath_4/object_8 |
Including paths
You can use includePrefix
when creating a transfer to instruct
Storage Transfer Service to consider objects in the listed paths for transfer,
and to ignore objects not under those paths.
For example, to include objects under path_1/
, pass includePrefix
the
following:
includePrefix: "path_1/"
This includes objects directly under path_1/
, path_1/subpath_1
,
and path_1/subpath_2/
. The following objects are included in the transfer:
path_1/object_3 |
path_1/subpath_1/object_5 path_1/subpath_2/object_6 |
You can also specify particular paths to include. For example, you can pass
includePrefix
the following:
includePrefix: "path_1/subpath_2/", "path_1/subpath_3/"
In this case, the transfer includes the following objects:
path_1/subpath_2/object_6 |
path_2/subpath_3/object_7 |
When you use includePrefix
, paths that you don't specifically include aren't
transferred to the Cloud Storage destination bucket.
Excluding paths
Using excludePrefix
when creating a transfer instructs Storage Transfer Service to
ignore the listed paths for transfer.
To exclude objects under path_1/
, pass excludePrefix
the following:
excludePrefix: "path_1/"
This excludes objects under path_1/
, path_1/subpath_1/
, and
path_1/subpath_2/
. In this case, the following objects are included in the
transfer:
object_1 object_2 |
path_2/object_4 path_2/subpath_3/object_7 path_2/subpath_4/object_8 |
You can also specify specific paths to exclude. For example, you can pass
excludePrefix
the following:
excludePrefix: "path_1/subpath_2/", "path_2/subpath_3/"
In this case, the transfer includes the following objects:
object_1 object_2 path_1/object_3 path_2/object_4 |
path_1/subpath_1/object_5 path_2/subpath_4/object_8 |
Including and excluding paths simultaneously
You can apply an excludePrefix
and an includePrefix
together, in which case
the excludePrefix
further limits what the includePrefix
includes in the
transfer.
When specifying both includePrefix
and excludePrefix
, each excludePrefix
must start with any of the values you specified for includePrefix
.
For example, to include objects under path_1/
and exclude objects under
subpath_1/
pass the following:
includePrefix: "path_1/"
excludePrefix: "path_1/subpath_1/"
In this case, the transfer includes the following objects:
path_1/object_3 |
path_1/subpath_2/object_6 |
To include all objects under path_1/
and path_2/
, except items in either
path_1/subpath_1/
or path_2/subpath_3/
, pass the following:
includePrefix: "path_1/", "path_2/"
excludePrefix: "path_1/subpath_1/", "path_2/subpath_3/"
In this case, the transfer includes the following objects:
path_1/object_3 path_2/object_4 |
path_1/subpath_2/object_6 path_2/subpath_4/object_8 |
Examples of incorrect path inclusion or exclusion
The following sections include examples to avoid when using include or exclude paths, and how to correct them so that they work correctly.
Including a path that is used in another include-prefix
Each includePrefix
should specify a distinct portion of the object namespace.
The following example is incorrect because the second value is already included
in the namespace of the first value:
includePrefix: "path_1/", "path_1/subpath_1"
In this example, the includePrefix
values are invalid because the second
includePrefix
, "path_1/subpath_1"
, is already included within "path_1/"
.
To fix this, choose to include either one or the other:
includePrefix: "path_1"
includePrefix: "path_1/subpath_1"
Using an exclude-prefix that doesn't start with an include-prefix
Each excludePrefix
should start with any of the specified includePrefix
values. The following example is incorrect because the excludePrefix
values
don't start with the specified includePrefix
values:
includePrefix: "path_1/", "path_2/"
excludePrefix: "subpath_1/", "subpath_4/"
In this example, the excludePrefix
values are invalid because they don't start
with either of the includePrefix
values, path_1
or path_2
. To fix this,
ensure that the excludePrefix
includes a full path listed in includePrefix
:
includePrefix: "path_1/", "path_2/"
excludePrefix: "path_1/subpath_1/", "path_2/subpath_4/"
What's next
- Learn how to transfer a specific list of files or objects.