This page discusses XML API multipart uploads in Cloud Storage. This upload method uploads files in parts and then assembles them into a single object using a final request. XML API multipart uploads are compatible with Amazon S3 multipart uploads.
An XML API multipart upload allows you to upload data in multiple parts and then assemble them into a final object. This behavior has several advantages, particularly for large files:
You can upload parts simultaneously, reducing the time it takes to upload the data in its entirety.
If one of the upload operations fails, you only have to re-upload a portion of the overall object, instead of restarting from the beginning.
Since the total file size is not specified in advance, you can use XML API multipart uploads for streaming uploads or for compressing data on-the-fly while uploading.
An XML API multipart upload has three required steps:
Initiate the upload using a
POSTrequest, which includes specifying any metadata that the completed object should have. The response returns an
UploadIdthat you use in all subsequent requests associated with the upload.
Upload the data using one or more
PUTrequests. Note that there are limits to the number of parts and their sizes.
Complete the upload using a
POSTrequest. This request overwrites any existing object in the bucket with the same name.
There is no limit to how long a multipart upload and its uploaded parts can remain unfinished or idle in a bucket, but note that successfully uploaded parts count toward your monthly storage usage. You can avoid a buildup of abandoned multipart uploads by using Object Lifecycle Management to automatically remove multipart uploads when they reach a specified age.
The following limitations apply to using XML API multipart uploads:
- An object uploaded using this method cannot be copied or rewritten, unless you compose the object prior to the copy or rewrite.
- Preconditions are not supported in the requests.
- MD5 hashes do not exist for objects uploaded using this method.
- This upload method is not supported in the Google Cloud console, Google Cloud CLI, or Cloud Storage client libraries.
Keep in mind the following when working with XML API multipart uploads:
While you can initiate an upload and upload parts, the request to complete the upload fails if it would overwrite an object that has a hold on it or an unfulfilled retention period.
You can list ongoing uploads in a bucket, but only a completed upload appears in the normal list of objects in the bucket.
Uploaded parts are subject to early deletion charges if they use an applicable storage class and one of the following occurs before the part reaches its minimum storage duration:
- The upload completes but the part is not used in the completion request.
- The part is overwritten by another uploaded part.
- The overall multipart upload is aborted, either directly or through Object Lifecycle Management.
The storage duration for each part in a multipart upload begins at the time the upload of the part completes.
- Explore additional uploading methods for Cloud Storage.
- Learn about truncated exponential backoff and when to retry requests.