HttpData(mapping=None, *, ignore_unknown_fields=False, **kwargs)
An HttpData resource specifies a list of objects on the web to be
transferred over HTTP. The information of the objects to be
transferred is contained in a file referenced by a URL. The first
line in the file must be "TsvHttpData-1.0"
, which specifies the
format of the file. Subsequent lines specify the information of the
list of objects, one object per list entry. Each entry has the
following tab-delimited fields:
HTTP URL — The location of the object.
Length — The size of the object in bytes.
MD5 — The base64-encoded MD5 hash of the object.
For an example of a valid TSV file, see Transferring data from
URLs <https://cloud.google.com/storage-transfer/docs/create-url-list>
__.
When transferring data based on a URL list, keep the following in mind:
When an object located at
http(s)://hostname:port/<URL-path>
is transferred to a data sink, the name of the object at the data sink is<hostname>/<URL-path>
.If the specified size of an object does not match the actual size of the object fetched, the object is not transferred.
If the specified MD5 does not match the MD5 computed from the transferred bytes, the object transfer fails.
Ensure that each URL you specify is publicly accessible. For example, in Cloud Storage you can share an object publicly and get a link to it.
Storage Transfer Service obeys
robots.txt
rules and requires the source HTTP server to supportRange
requests and to return aContent-Length
header in each response.ObjectConditions][google.storagetransfer.v1.ObjectConditions]
have no effect when filtering objects to transfer.
Attribute | |
---|---|
Name | Description |
list_url |
str
Required. The URL that points to the file that stores the object list entries. This file must allow public access. Currently, only URLs with HTTP and HTTPS schemes are supported. |