Performing resumable uploads

This page describes how to make a resumable upload request in the Cloud Storage JSON and XML API. This protocol allows you to resume an upload operation after a communication failure interrupts the flow of data. For details about this feature and when to use it, see Resumable Uploads.

Initiating a resumable upload session

To initiate a resumable upload session:

JSON API

1. Create a POST request to the URI https://storage.googleapis.com/upload/storage/v1/b/[BUCKET_NAME]/o.

2. Add the query parameter uploadType=resumable. For example, for a bucket named myBucket:

   POST https://storage.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable

3. If you do not have any metadata for the file, add a name query parameter to identify which resource the upload is associated with. For example, to specify that an object's name is myObject:

   POST https://storage.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable&name=myObject

4. If you have metadata for the file, add the metadata to the request body in JSON format. Otherwise, leave the request body empty.

5. Add the following HTTP headers:

   • X-Upload-Content-Type. Optional. Set to the MIME type of the file data, which is transferred in subsequent requests. If the MIME type of the data is not specified in metadata or through this header, the object is served as application/octet-stream.
   • X-Upload-Content-Length. Optional. Set to the number of bytes of file data, which will be transferred in subsequent requests.
   • Content-Type. Required if you have metadata for the file. Set to application/json; charset=UTF-8.
   • Content-Length. Required unless you are using chunked transfer encoding. Set to the number of bytes in the body of this initial request.
   • Origin. if you have enabled Cross-Origin Resource Sharing. You must also use this header in subsequent upload requests.

6. Send the request.

Example

The following example shows how to initiate a resumable session using a bucket named myBucket. You can get an authorization access token from the OAuth 2.0 Playground.

POST https://storage.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable HTTP/1.1
  Authorization: Bearer [YOUR_AUTH_TOKEN]
  Content-Length: 38
  Content-Type: application/json; charset=UTF-8
  X-Upload-Content-Type: image/jpeg
  X-Upload-Content-Length: 2000000
  {
    "name": "myObject"
  }

The next section describes how to handle the response.

XML API

To begin a resumable upload, send a POST Object request to Cloud Storage. The POST Object request does not contain the file you are uploading. Rather, it contains a few headers that inform the Cloud Storage system that you want to perform a resumable upload. Specifically, the POST Object request must have the following:

• An empty entity body.
• A Content-Length request header, which must be set to 0.
• An [x-goog-resumable][x-goog-resumable] header, which must be set to start.
• If you have enabled [Cross-Origin Resource Sharing][cors], an Origin header that you use in subsequent upload requests for the file you are uploading.

Example The following example shows how to initiate a resumable upload for a file named music.mp3 that's being uploaded into a bucket named example.

    POST /music.mp3 HTTP/1.1
    Host: example.storage.googleapis.com
    Date: Fri, 01 Oct 2010 21:56:18 GMT
    Content-Length: 0
    Content-Type: audio/mpeg
    x-goog-resumable: start
    Authorization: Bearer ya29.AHES6ZRVmB7fkLtd1XTmq6mo0S1wqZZi3-Lh_s-6Uw7p8vtgSwg
    

Saving the resumable session URI

Copy and save the resumable session URI so you can use it for subsequent requests.

Note: A resumable session URI expires after one week.

JSON API

If the session initiation request succeeds, the response includes a 200 OK HTTP status code. In addition, it includes a Location header that specifies the resumable session URI. Use the resumable session URI to upload the file data and query the upload status.

Example

The following example shows a response that includes a resumable session URI to a bucket named myBucket:

HTTP/1.1 200 OK
  Location: https://storage.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable&upload_id=xa298sd_sdlkj2
  Content-Length: 0

XML API

After you initiate the resumable upload with a POST Object request, Cloud Storage responds with a 201 Created status message. The status message includes a Location header whose value is the resumable session URI. You must save the session URI, because you will use it in all further requests during your upload operation.

Example The following example shows the response to the Post Object request that was shown in the "Initiating a resumable upload" section.

  HTTP/1.1 201 Created
  Location: https://example.storage.googleapis.com/music.mp3?upload_id=tvA0ExBntDa...gAAEnB2Uowrot
  Date: Fri, 01 Oct 2010 21:56:18 GMT
  Content-Length: 0
  Content-Type: audio/mpeg
  

Uploading the file

JSON API

There are two ways to upload a file with a resumable session:

• In a single request. This approach is usually best, since it requires fewer requests and thus has better performance.

• In multiple chunks. Use this approach if:

  • You need to reduce the amount of data transferred in any single request. You might need to do this when there is a fixed time limit for individual requests, as is true for certain classes of App Engine requests.
  • You need to provide a customized indicator showing the upload progress.

Single request

To upload the file in a single request:

1. Create a PUT request to the resumable session URI.
2. Add the file's data to the request body.
3. Add a Content-Length HTTP header, set to the number of bytes in the file.
4. Send the request.

If the upload request is interrupted, or if you receive a 5xx response, follow the procedure in Resuming an interrupted upload.

If the request succeeds, you receive a 200 OK or 201 Created response, along with any metadata associated with the resource.

Example The following example shows a resumable request to upload an entire 2,000,000-byte JPEG file into myBucket, using the resumable session URI obtained in the previous step:

PUT https://storage.googleapis.com/upload/storage/v1/b/myBucket/o?   uploadType=resumable&upload_id=xa298sd_sdlkj2 HTTP/1.1
Content-Length: 2000000
Content-Type: image/jpeg
Content-Range: bytes 0-1999999

Multiple chunks

To upload the file in multiple chunks:

1. Create a PUT request to the resumable session URI.
2. Add the chunk's data to the request body. Create chunks in multiples of 256 KB (256 x 1024 bytes) in size, except for the final chunk that completes the upload. Keep the chunk size as large as possible so that the upload is efficient.
3. Add the following HTTP headers:
   • Content-Length. Set to the number of bytes in the current chunk.
   • Content-Range: Set to show which bytes in the file you are uploading. For example, Content-Range: bytes 0-524287/2000000 shows that you are uploading the first 524,288 bytes (256 x 1024 x 2) in a 2,000,000 byte file.

4. Send the request, and process the response. If the upload request is interrupted, or if you receive a 5xx response, follow the procedure in Resuming an interrupted upload.

5. Repeat steps 1 through 4 for each remaining chunk in the file. Use the Range header in the response to determine where to start the next chunk. Do not assume that the server received all bytes sent in the previous request.

   When the entire file upload is complete, you receive a 200 OK or 201 Created response, along with any metadata associated with the resource.

Example

The following example shows a request that sends the first 524,288 bytes (512 KB) of the file into myBucket, using the resumable session URI obtained in the previous step:

PUT https://storage.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable&upload_id=xa298sd_sdlkj2 HTTP/1.1
      Content-Length: 524288
      Content-Type: image/jpeg
      Content-Range: bytes 0-524287/2000000 

If the request succeeds, the server responds with 308 Resume Incomplete' along with aRange` header that identifies the total number of bytes that have been stored so far:

HTTP/1.1 308 Resume Incomplete
    Content-Length: 0
    Content-Range: bytes=0-524287

Use the upper value returned in the Range header to determine where to start the next chunk. Continue to PUT each chunk of the file until the entire file has been uploaded.

When the entire upload is complete, you receive a 200 OK or 201 Created response, along with any metadata associated with the resource.

XML API

Implement a PUT Object request that sends the file blocks to Cloud Storage. Use the session URI you obtained as the PUT request's request URI. The request also includes a Content-Length header, which you must use to specify the size of the file you are uploading.

As with the POST Object request, you must use the standard Cloud Storage host name in the request (storage.googleapis.com). You do not need to use an explicit authentication token since the session URI is, in effect, an authentication token.

Example

The following example shows how to upload the music.mp3 file that was initiated in Step 1:

  PUT https://example.storage.googleapis.com/music.mp3?upload_id=tvA0ExBntDa...gAAEnB2Uowrot HTTP/1.1
  Date: Fri, 01 Oct 2010 21:56:18 GMT
  Content-Length: 7351375
  

If the PUT Object request is not interrupted and the file is successfully uploaded, Cloud Storage responds with a 200 OK status code. If the upload is interrupted, you can resume the upload by performing Steps 4, 5, and 6.

You should query for the number of bytes it has received by sending another PUT Object request. The PUT Object request must have the following:

• An empty entity body.
• A Content-Length request header, which must be set to 0.
• A Content-Range request header, which specifies the byte range you are seeking status for.
• A request URI equal to the session URI for the resumable upload.

The value of the Content-Range request header must be in the following format:

Content-Range: bytes */<content-length>

Where <content-length> is the value of the Content-Length header that you specified in the original PUT Object request (Step 3).

In addition, you must use the standard Cloud Storage host name in the request (storage.googleapis.com).

The following example shows how to query the Cloud Storage system after a resumable upload is interrupted:

  PUT https://example.storage.googleapis.com/music.mp3?upload_id=tvA0ExBntDa...gAAEnB2Uowrot HTTP/1.1
  Date: Fri, 01 Oct 2010 22:25:53 GMT
  Content-Range: bytes */7351375
  Content-Length: 0
  

Resuming an interrupted upload

If an upload request is terminated before receiving a response, or if you receive a 503 or 500 Service Unavailable response, then you need to resume the interrupted upload. To resume an interrupted upload:

JSON API

1. To request the upload status, create an empty PUT request to the resumable session URI.

2. Add a Content-Range header indicating that the current position in the file is unknown.

   For example, set the Content-Range to */2000000 if your total file length is 2,000,000 bytes.

   If you don't know the full size of the file, set the Content-Range to */*.

3. Send the request.

4. Process the response.

   • A 200 OK or 201 Created response indicates that the upload was completed, and no further action is necessary.
   • A 308 Resume Incomplete response indicates that you need to continue uploading the file.

5. If you received a 308 Resume Incomplete response, process the response's Range header, which specifies which bytes Cloud Storage has received so far. You will use this number in the next step. The response does not have a Range header if Cloud Storage has not yet received any bytes.

   For example, a Range header of bytes=0-42 indicates that the first 43 bytes of the file have been received.

6. Now that you know where to resume the upload, continue uploading the file, either by sending the remaining data or by sending the next chunk. Include a Content-Range header indicating which portion of the file you are sending.

   For example, Content-Range: bytes 43-1999999/2000000 indicates that you are sending bytes 43 through 1,999,999.

XML API

You can resume the upload operation by sending a PUT Object request. The PUT Object request must have the following:

• An entity body containing the range of bytes that still need to be uploaded. You can determine this range by subtracting the Range (which you obtained in Step 5) from the Content-Length (which you specified in Step 3).
• A Content-Length request header, which specifies the number of bytes you are uploading in the current request.
• A Content-Range request header, which specifies the byte range you are uploading in the request.

• A request URI equal to the session URI for the resumable upload.

  You must use the standard Cloud Storage host name in the request (storage.googleapis.com).

  Example

  The following example shows a PUT Object request that resumes the upload of the music.mp3 file into the example bucket.

  PUT https://example.storage.googleapis.com/music.mp3?upload_id=tvA0ExBntDa...gAAEnB2Uowrot HTTP/1.1
  Date: Fri, 01 Oct 2010 22:25:53 GMT
  Content-Range: bytes 2359296-7351374/7351375
  Content-Length: 4992079
  

  You can perform steps 4, 5, or 6 as many times as necessary, but when retrying requests, use truncated exponential backoff. For an example, see the boto implementation of this logic.

  When the file is successfully uploaded, Cloud Storage responds with a 200 OK status code.

Read more details about Optional Optimization

Cancelling an upload

If you want to cancel the upload and prevent any further action on it, issue a DELETE request on the unique upload URI. After the DELETE request has succeeded, future attempts to query or resume the upload will result in a 4xx response.

JSON API

Example

The following example shows how to cancel a resumable upload:

DELETE https://storage.googleapis.com/upload/storage/v1/b/myBucket/o?uploadType=resumable&upload_id=xa298sd_sdlkj2 HTTP/1.1
Content-Length: 0

XML API

If you want to cancel the upload and prevent any further action on it, issue a DELETE request on the unique upload URI. After the DELETE request has succeeded, future attempts to query or resume the upload will result in a 4xx response.

Example

The following example shows how to cancel a resumable upload:

 DELETE https://example.storage.googleapis.com/myFile.zip?upload_id=tvA0ExBntDa...gAAEnB2Uowrot HTTP/1.1
    Content-Length: 0
    

What's next?

• Learn more about resumable uploads
• Read an overview for the JSON API and XML API
• Learn how to batch multiple requests to the Cloud Storage JSON API.