Configuration for running a transfer.
JSON representation |
---|
{ "objectConditions": { object ( |
Fields | |
---|---|
objectConditions |
Only objects that satisfy these object conditions are included in the set of data source and data sink objects. Object conditions based on objects' "last modification time" do not exclude objects in a data sink. |
transferOptions |
If the option |
transferManifest |
A manifest file provides a list of objects to be transferred from the data source. This field points to the location of the manifest file. Otherwise, the entire source bucket is used. ObjectConditions still apply. |
sourceAgentPoolName |
Specifies the agent pool name associated with the posix data source. When unspecified, the default name is used. |
sinkAgentPoolName |
Specifies the agent pool name associated with the posix data sink. When unspecified, the default name is used. |
Union field data_sink . The write sink for the data. data_sink can be only one of the following: |
|
gcsDataSink |
A Cloud Storage data sink. |
posixDataSink |
A POSIX Filesystem data sink. |
Union field data_source . The read source of the data. data_source can be only one of the following: |
|
gcsDataSource |
A Cloud Storage data source. |
awsS3DataSource |
An AWS S3 data source. |
httpDataSource |
An HTTP URL data source. |
posixDataSource |
A POSIX Filesystem data source. |
azureBlobStorageDataSource |
An Azure Blob Storage data source. |
awsS3CompatibleDataSource |
An AWS S3 compatible data source. |
Union field
|
|
gcsIntermediateDataLocation |
For transfers between file systems, specifies a Cloud Storage bucket to be used as an intermediate location through which to transfer data. See Transfer data between file systems for more information. |
GcsData
In a GcsData resource, an object's name is the Cloud Storage object's name and its "last modification time" refers to the object's updated
property of Cloud Storage objects, which changes when the content or the metadata of the object is updated.
JSON representation |
---|
{ "bucketName": string, "path": string } |
Fields | |
---|---|
bucketName |
Required. Cloud Storage bucket name. Must meet Bucket Name Requirements. |
path |
Root path to transfer objects. Must be an empty string or full path name that ends with a '/'. This field is treated as an object prefix. As such, it should generally not begin with a '/'. The root path value must meet Object Name Requirements. |
PosixFilesystem
A POSIX filesystem resource.
JSON representation |
---|
{ "rootDirectory": string } |
Fields | |
---|---|
rootDirectory |
Root directory path to the filesystem. |
AwsS3Data
An AwsS3Data resource can be a data source, but not a data sink. In an AwsS3Data resource, an object's name is the S3 object's key name.
JSON representation |
---|
{
"bucketName": string,
"awsAccessKey": {
object ( |
Fields | |
---|---|
bucketName |
Required. S3 Bucket name (see Creating a bucket). |
awsAccessKey |
Input only. AWS access key used to sign the API requests to the AWS S3 bucket. Permissions on the bucket must be granted to the access ID of the AWS access key. For information on our data retention policy for user credentials, see User credentials. |
path |
Root path to transfer objects. Must be an empty string or full path name that ends with a '/'. This field is treated as an object prefix. As such, it should generally not begin with a '/'. |
roleArn |
The Amazon Resource Name (ARN) of the role to support temporary credentials via When a role ARN is provided, Transfer Service fetches temporary credentials for the session using a |
credentialsSecret |
The Resource name of a secret in Secret Manager. AWS credentials must be stored in Secret Manager in JSON format: { "accessKeyId": "ACCESS_KEY_ID", "secretAccessKey": "SECRET_ACCESS_KEY" }
See Configure access to a source: Amazon S3 for more information. If This feature is in preview. Format: |
AwsAccessKey
AWS access key (see AWS Security Credentials).
For information on our data retention policy for user credentials, see User credentials.
JSON representation |
---|
{ "accessKeyId": string, "secretAccessKey": string } |
Fields | |
---|---|
accessKeyId |
Required. AWS access key ID. |
secretAccessKey |
Required. AWS secret access key. This field is not returned in RPC responses. |
HttpData
An HttpData resource specifies a list of objects on the web to be transferred over HTTP. The information of the objects to be transferred is contained in a file referenced by a URL. The first line in the file must be "TsvHttpData-1.0"
, which specifies the format of the file. Subsequent lines specify the information of the list of objects, one object per list entry. Each entry has the following tab-delimited fields:
HTTP URL — The location of the object.
Length — The size of the object in bytes.
MD5 — The base64-encoded MD5 hash of the object.
For an example of a valid TSV file, see Transferring data from URLs.
When transferring data based on a URL list, keep the following in mind:
When an object located at
http(s)://hostname:port/<URL-path>
is transferred to a data sink, the name of the object at the data sink is<hostname>/<URL-path>
.If the specified size of an object does not match the actual size of the object fetched, the object is not transferred.
If the specified MD5 does not match the MD5 computed from the transferred bytes, the object transfer fails.
Ensure that each URL you specify is publicly accessible. For example, in Cloud Storage you can share an object publicly and get a link to it.
Storage Transfer Service obeys
robots.txt
rules and requires the source HTTP server to supportRange
requests and to return aContent-Length
header in each response.ObjectConditions
have no effect when filtering objects to transfer.
JSON representation |
---|
{ "listUrl": string } |
Fields | |
---|---|
listUrl |
Required. The URL that points to the file that stores the object list entries. This file must allow public access. Currently, only URLs with HTTP and HTTPS schemes are supported. |
AzureBlobStorageData
An AzureBlobStorageData resource can be a data source, but not a data sink. An AzureBlobStorageData resource represents one Azure container. The storage account determines the Azure endpoint. In an AzureBlobStorageData resource, a blobs's name is the Azure Blob Storage blob's key name.
JSON representation |
---|
{
"storageAccount": string,
"azureCredentials": {
object ( |
Fields | |
---|---|
storageAccount |
Required. The name of the Azure Storage account. |
azureCredentials |
Required. Input only. Credentials used to authenticate API requests to Azure. For information on our data retention policy for user credentials, see User credentials. |
container |
Required. The container to transfer from the Azure Storage account. |
path |
Root path to transfer objects. Must be an empty string or full path name that ends with a '/'. This field is treated as an object prefix. As such, it should generally not begin with a '/'. |
credentialsSecret |
The Resource name of a secret in Secret Manager. The Azure SAS token must be stored in Secret Manager in JSON format: { "sasToken" : "SAS_TOKEN" }
See Configure access to a source: Microsoft Azure Blob Storage for more information. If This feature is in preview. Format: |
AzureCredentials
Azure credentials
For information on our data retention policy for user credentials, see User credentials.
JSON representation |
---|
{ "sasToken": string } |
Fields | |
---|---|
sasToken |
Required. Azure shared access signature (SAS). For more information about SAS, see Grant limited access to Azure Storage resources using shared access signatures (SAS). |
AwsS3CompatibleData
An AwsS3CompatibleData resource.
JSON representation |
---|
{ "bucketName": string, "path": string, "endpoint": string, "region": string, // Union field |
Fields | |
---|---|
bucketName |
Required. Specifies the name of the bucket. |
path |
Specifies the root path to transfer objects. Must be an empty string or full path name that ends with a '/'. This field is treated as an object prefix. As such, it should generally not begin with a '/'. |
endpoint |
Required. Specifies the endpoint of the storage service. |
region |
Specifies the region to sign requests with. This can be left blank if requests should be signed with an empty region. |
Union field data_provider . Specifies the metadata of the S3 compatible data provider. Each provider may contain some attributes that do not apply to all S3-compatible data providers. When not specified, S3CompatibleMetadata is used by default. data_provider can be only one of the following: |
|
s3Metadata |
A S3 compatible metadata. |
S3CompatibleMetadata
S3CompatibleMetadata contains the metadata fields that apply to the basic types of S3-compatible data providers.
JSON representation |
---|
{ "authMethod": enum ( |
Fields | |
---|---|
authMethod |
Specifies the authentication and authorization method used by the storage service. When not specified, Transfer Service will attempt to determine right auth method to use. |
requestModel |
Specifies the API request model used to call the storage service. When not specified, the default value of RequestModel REQUEST_MODEL_VIRTUAL_HOSTED_STYLE is used. |
protocol |
Specifies the network protocol of the agent. When not specified, the default value of NetworkProtocol NETWORK_PROTOCOL_HTTPS is used. |
listApi |
The Listing API to use for discovering objects. When not specified, Transfer Service will attempt to determine the right API to use. |
AuthMethod
The authentication and authorization method used by the storage service.
Enums | |
---|---|
AUTH_METHOD_UNSPECIFIED |
AuthMethod is not specified. |
AUTH_METHOD_AWS_SIGNATURE_V4 |
Auth requests with AWS SigV4. |
AUTH_METHOD_AWS_SIGNATURE_V2 |
Auth requests with AWS SigV2. |
RequestModel
The request model of the API.
Enums | |
---|---|
REQUEST_MODEL_UNSPECIFIED |
RequestModel is not specified. |
REQUEST_MODEL_VIRTUAL_HOSTED_STYLE |
Perform requests using Virtual Hosted Style. Example: https://bucket-name.s3.region.amazonaws.com/key-name |
REQUEST_MODEL_PATH_STYLE |
Perform requests using Path Style. Example: https://s3.region.amazonaws.com/bucket-name/key-name |
NetworkProtocol
The agent network protocol to access the storage service.
Enums | |
---|---|
NETWORK_PROTOCOL_UNSPECIFIED |
NetworkProtocol is not specified. |
NETWORK_PROTOCOL_HTTPS |
Perform requests using HTTPS. |
NETWORK_PROTOCOL_HTTP |
Not recommended: This sends data in clear-text. This is only appropriate within a closed network or for publicly available data. Perform requests using HTTP. |
ListApi
The Listing API to use for discovering objects.
Enums | |
---|---|
LIST_API_UNSPECIFIED |
ListApi is not specified. |
LIST_OBJECTS_V2 |
Perform listing using ListObjectsV2 API. |
LIST_OBJECTS |
Legacy ListObjects API. |
ObjectConditions
Conditions that determine which objects are transferred. Applies only to Cloud Data Sources such as S3, Azure, and Cloud Storage.
The "last modification time" refers to the time of the last change to the object's content or metadata — specifically, this is the updated
property of Cloud Storage objects, the LastModified
field of S3 objects, and the Last-Modified
header of Azure blobs.
Transfers with a PosixFilesystem
source or destination don't support ObjectConditions
.
JSON representation |
---|
{ "minTimeElapsedSinceLastModification": string, "maxTimeElapsedSinceLastModification": string, "includePrefixes": [ string ], "excludePrefixes": [ string ], "lastModifiedSince": string, "lastModifiedBefore": string } |
Fields | |
---|---|
minTimeElapsedSinceLastModification |
Ensures that objects are not transferred until a specific minimum time has elapsed after the "last modification time". When a A duration in seconds with up to nine fractional digits, ending with ' |
maxTimeElapsedSinceLastModification |
Ensures that objects are not transferred if a specific maximum time has elapsed since the "last modification time". When a A duration in seconds with up to nine fractional digits, ending with ' |
includePrefixes[] |
If you specify The following are requirements of
The max size of For more information, see Filtering objects from transfers. |
excludePrefixes[] |
If you specify The following are requirements of
The max size of For more information, see Filtering objects from transfers. |
lastModifiedSince |
If specified, only objects with a "last modification time" on or after this timestamp and objects that don't have a "last modification time" are transferred. The
A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
lastModifiedBefore |
If specified, only objects with a "last modification time" before this timestamp and objects that don't have a "last modification time" are transferred. A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: |
TransferOptions
TransferOptions define the actions to be performed on objects in a transfer.
JSON representation |
---|
{ "overwriteObjectsAlreadyExistingInSink": boolean, "deleteObjectsUniqueInSink": boolean, "deleteObjectsFromSourceAfterTransfer": boolean, "overwriteWhen": enum ( |
Fields | |
---|---|
overwriteObjectsAlreadyExistingInSink |
When to overwrite objects that already exist in the sink. The default is that only objects that are different from the source are ovewritten. If true, all objects in the sink whose name matches an object in the source are overwritten with the source object. |
deleteObjectsUniqueInSink |
Whether objects that exist only in the sink should be deleted. Note: This option and |
deleteObjectsFromSourceAfterTransfer |
Whether objects should be deleted from the source after they are transferred to the sink. Note: This option and |
overwriteWhen |
When to overwrite objects that already exist in the sink. If not set, overwrite behavior is determined by |
metadataOptions |
Represents the selected metadata options for a transfer job. |
OverwriteWhen
Specifies when to overwrite an object in the sink when an object with matching name is found in the source.
Enums | |
---|---|
OVERWRITE_WHEN_UNSPECIFIED |
Overwrite behavior is unspecified. |
DIFFERENT |
Overwrites destination objects with the source objects, only if the objects have the same name but different HTTP ETags or checksum values. |
NEVER |
Never overwrites a destination object if a source object has the same name. In this case, the source object is not transferred. |
ALWAYS |
Always overwrite the destination object with the source object, even if the HTTP Etags or checksum values are the same. |
MetadataOptions
Specifies the metadata options for running a transfer.
JSON representation |
---|
{ "symlink": enum ( |
Fields | |
---|---|
symlink |
Specifies how symlinks should be handled by the transfer. By default, symlinks are not preserved. Only applicable to transfers involving POSIX file systems, and ignored for other transfers. |
mode |
Specifies how each file's mode attribute should be handled by the transfer. By default, mode is not preserved. Only applicable to transfers involving POSIX file systems, and ignored for other transfers. |
gid |
Specifies how each file's POSIX group ID (GID) attribute should be handled by the transfer. By default, GID is not preserved. Only applicable to transfers involving POSIX file systems, and ignored for other transfers. |
uid |
Specifies how each file's POSIX user ID (UID) attribute should be handled by the transfer. By default, UID is not preserved. Only applicable to transfers involving POSIX file systems, and ignored for other transfers. |
acl |
Specifies how each object's ACLs should be preserved for transfers between Google Cloud Storage buckets. If unspecified, the default behavior is the same as ACL_DESTINATION_BUCKET_DEFAULT. |
storageClass |
Specifies the storage class to set on objects being transferred to Google Cloud Storage buckets. If unspecified, the default behavior is the same as |
temporaryHold |
Specifies how each object's temporary hold status should be preserved for transfers between Google Cloud Storage buckets. If unspecified, the default behavior is the same as |
kmsKey |
Specifies how each object's Cloud KMS customer-managed encryption key (CMEK) is preserved for transfers between Google Cloud Storage buckets. If unspecified, the default behavior is the same as |
timeCreated |
Specifies how each object's |
Symlink
Whether symlinks should be skipped or preserved during a transfer job.
Enums | |
---|---|
SYMLINK_UNSPECIFIED |
Symlink behavior is unspecified. |
SYMLINK_SKIP |
Do not preserve symlinks during a transfer job. |
SYMLINK_PRESERVE |
Preserve symlinks during a transfer job. |
Mode
Options for handling file mode attribute.
Enums | |
---|---|
MODE_UNSPECIFIED |
Mode behavior is unspecified. |
MODE_SKIP |
Do not preserve mode during a transfer job. |
MODE_PRESERVE |
Preserve mode during a transfer job. |
GID
Options for handling file GID attribute.
Enums | |
---|---|
GID_UNSPECIFIED |
GID behavior is unspecified. |
GID_SKIP |
Do not preserve GID during a transfer job. |
GID_NUMBER |
Preserve GID during a transfer job. |
UID
Options for handling file UID attribute.
Enums | |
---|---|
UID_UNSPECIFIED |
UID behavior is unspecified. |
UID_SKIP |
Do not preserve UID during a transfer job. |
UID_NUMBER |
Preserve UID during a transfer job. |
Acl
Options for handling Cloud Storage object ACLs.
Enums | |
---|---|
ACL_UNSPECIFIED |
ACL behavior is unspecified. |
ACL_DESTINATION_BUCKET_DEFAULT |
Use the destination bucket's default object ACLS, if applicable. |
ACL_PRESERVE |
Preserve the object's original ACLs. This requires the service account to have storage.objects.getIamPolicy permission for the source object. Uniform bucket-level access must not be enabled on either the source or destination buckets. |
StorageClass
Options for handling Google Cloud Storage object storage class.
Enums | |
---|---|
STORAGE_CLASS_UNSPECIFIED |
Storage class behavior is unspecified. |
STORAGE_CLASS_DESTINATION_BUCKET_DEFAULT |
Use the destination bucket's default storage class. |
STORAGE_CLASS_PRESERVE |
Preserve the object's original storage class. This is only supported for transfers from Google Cloud Storage buckets. REGIONAL and MULTI_REGIONAL storage classes will be mapped to STANDARD to ensure they can be written to the destination bucket. |
STORAGE_CLASS_STANDARD |
Set the storage class to STANDARD. |
STORAGE_CLASS_NEARLINE |
Set the storage class to NEARLINE. |
STORAGE_CLASS_COLDLINE |
Set the storage class to COLDLINE. |
STORAGE_CLASS_ARCHIVE |
Set the storage class to ARCHIVE. |
TemporaryHold
Options for handling temporary holds for Google Cloud Storage objects.
Enums | |
---|---|
TEMPORARY_HOLD_UNSPECIFIED |
Temporary hold behavior is unspecified. |
TEMPORARY_HOLD_SKIP |
Do not set a temporary hold on the destination object. |
TEMPORARY_HOLD_PRESERVE |
Preserve the object's original temporary hold status. |
KmsKey
Options for handling the KmsKey setting for Google Cloud Storage objects.
Enums | |
---|---|
KMS_KEY_UNSPECIFIED |
KmsKey behavior is unspecified. |
KMS_KEY_DESTINATION_BUCKET_DEFAULT |
Use the destination bucket's default encryption settings. |
KMS_KEY_PRESERVE |
Preserve the object's original Cloud KMS customer-managed encryption key (CMEK) if present. Objects that do not use a Cloud KMS encryption key will be encrypted using the destination bucket's encryption settings. |
TimeCreated
Options for handling timeCreated
metadata for Google Cloud Storage objects.
Enums | |
---|---|
TIME_CREATED_UNSPECIFIED |
TimeCreated behavior is unspecified. |
TIME_CREATED_SKIP |
Do not preserve the timeCreated metadata from the source object. |
TIME_CREATED_PRESERVE_AS_CUSTOM_TIME |
Preserves the source object's timeCreated metadata in the customTime field in the destination object. Note that any value stored in the source object's customTime field will not be propagated to the destination object. |
TransferManifest
Specifies where the manifest is located.
JSON representation |
---|
{ "location": string } |
Fields | |
---|---|
location |
Specifies the path to the manifest in Cloud Storage. The Google-managed service account for the transfer must have |