This page explains how to create a URL list and test your process for
generating MD5 hashes. You can use Storage Transfer Service to transfer data from a list
of public data locations to a Cloud Storage bucket. After you make the list of
locations, you must upload the list to a URL that begins with
When you configure your transfer, you simply refer to the
Creating the URL list
Use the following process to create a URL list:
Create a tab-separated values (TSV) file.
The URL list must be a tab-separated values (TSV) file that lists the URLs you want to download.
Set the first line to the format specifier,
Add additional lines for each object to transfer. Include the following tab-separated fields, in order, on each line:
The HTTP or HTTPS URL of a source object. Ensure that each URL you specify is publicly accessible. For example, in Cloud Storage, you can share an object publicly and get a link to it. Also, ensure that the server's
robots.txtfile allows access to the URL.
Check that the server that hosts each object supports
Rangerequests and returns a
Content-Lengthheader in each response.
The size of the object in bytes. Make sure that the specified size matches the actual size of the object when it is fetched. If the size does not match, the object will not be transferred.
- The Base64-encoded MD5 hash of the object. Make sure that the specified MD5 matches the MD5 computed from the transferred bytes. If the MD5 does not match, the object transfer will fail. For more information, see Generating MD5 hashes.
Ensure that when your web server returns the URL list, it sets a strong
Etagheader in the HTTP response.
The following example shows a TSV file that identifies two objects to transfer:
TsvHttpData-1.0 https://example.com/buckets/obj1 1357 wHENa08V36iPYAsOa2JAdw== https://example.com/buckets/obj2 2468 R9acAaveoPd2y8nniLUYbw==
Generating MD5 hashes
As noted above, your URL list must provide an MD5 hash for each object that is being transferred.
Use the following public object to verify that you are generating MD5 hashes correctly:
This object has a Base64-encoded MD5 hash of
Copy the object to a local file called
md5-test, and verify the hash using
openssl md5 -binary md5-test | openssl enc -base64