Performing resumable uploads

Go to concepts

This page describes how to make a resumable upload request in the Cloud Storage JSON and XML APIs. This protocol allows you to resume an upload operation after a communication failure interrupts the flow of data.

Initiating a resumable upload session

To initiate a resumable upload session:

JSON API

  1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

  2. Optionally, create a .json file that contains the metadata you want to set on the object that you're uploading.

  3. Use cURL to call the JSON API with a POST Object request:

    curl -i -X POST --data-binary @METADATA_LOCATION \
    -H "Authorization: Bearer OAUTH2_TOKEN" \
    -H "Content-Type: application/json" \
    -H "Content-Length: INITIAL_REQUEST_LENGTH" \
    "https://storage.googleapis.com/upload/storage/v1/b/BUCKET_NAME/o?uploadType=resumable&name=OBJECT_NAME"

    Where:

    • METADATA_LOCATION is the local path to the JSON file containing the optional metadata you specified in the previous step. If you are not including a metadata file, exclude this, along with --data-binary @ and the Content-Type header.
    • OAUTH2_TOKEN is the access token you generated in Step 1.
    • INITIAL_REQUEST_LENGTH is the number of bytes in the body of this initial request, for example 79. Not required if you are using chunked transfer encoding.
    • BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.
    • OBJECT_NAME is the name you want to give your object. For example, pets/dog.png. This is not required if you included a name in the object metadata file in Step 2.

    If you have enabled Cross-Origin Resource Sharing, you should also include an Origin header in both this and subsequent upload requests.

    Optional headers that you can add to the request include X-Upload-Content-Type and X-Upload-Content-Length.

    If successful, the response includes a 200 status code.

  4. Save the resumable session URI given in the Location header of the response to your POST Object request.

    This URI is used in subsequent requests to upload the object data.

XML API

  1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials.

  2. Use cURL to call the XML API with a POST Object request that has an empty body:

    curl -i -X POST -H "Authorization: Bearer OAUTH2_TOKEN" \
    -H "Content-Length: 0" \
    -H "Content-Type: OBJECT_CONTENT_TYPE" \
    -H "x-goog-resumable: start" \
    "https://storage.googleapis.com/BUCKET_NAME/OBJECT_NAME"

    Where:

    • OAUTH2_TOKEN is the access token you generated in Step 1.
    • OBJECT_CONTENT_TYPE is the content type of the object. For example, image/png. If you do not specify a content type, the Cloud Storage system sets the Content-Type metadata for the object to be application/octet-stream.
    • BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.
    • OBJECT_NAME is the name you want to give your object. For example, pets/dog.png.

    If you have enabled Cross-Origin Resource Sharing, you should also include an Origin header in both this and subsequent upload requests.

    If successful, the response includes a 201 status message.

  3. Save the resumable session URI given in the Location header of the response to your POST Object request.

    This URI is used in subsequent requests to upload the object data.

Uploading the file

Once you have initiated a resumable upload, there are two ways to upload the object's data:

  • In a single chunk: This approach is usually best, since it requires fewer requests and thus has better performance.
  • In multiple chunks: Use this approach if you need to reduce the amount of data transferred in any single request, such as when there is a fixed time limit for individual requests, or if you don't know the total size of the upload at the time the upload begins.

Single chunk upload

To upload the file in a single chunk:

JSON API

  1. Use cURL to call the JSON API with a PUT Object request:

    curl -i -X PUT --data-binary @OBJECT_LOCATION \
    -H "Content-Length: OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.
    • OBJECT_SIZE is the number of bytes in your object. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

XML API

  1. Use cURL to call the XML API with a PUT Object request:

    curl -i -X PUT --data-binary @OBJECT_LOCATION \
    -H "Content-Length: OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.
    • OBJECT_SIZE is the number of bytes in your object. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

If the upload completes in its entirety, you receive a 200 OK or 201 Created response, along with any metadata associated with the resource.

If the upload request is interrupted, or if you receive a 5xx response, follow the procedure in Resuming an interrupted upload.

Multiple chunk upload

To upload the file in multiple chunks:

JSON API

  1. Create a chunk of data from the file you want to upload.

    The chunk size should be a multiple of 256 KiB (256 x 1024 bytes), unless it's the last chunk that completes the upload. Larger chunk sizes typically make uploads more efficient. We recommend using 8 MiB for the chunk size.

  2. Use cURL to call the JSON API with a PUT Object request:

    curl -i -X PUT --data-binary @CHUNK_LOCATION \
    -H "Content-Length: CHUNK_SIZE" \
    -H "Content-Range: bytes CHUNK_FIRST_BYTE-CHUNK_LAST_BYTE/TOTAL_OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • CHUNK_LOCATION is the local path to the chunk that you're currently uploading.
    • CHUNK_SIZE is the number of bytes you're uploading in the current request. For example, 524288.
    • CHUNK_FIRST_BYTE is the starting byte in the overall object that the chunk you're uploading contains.
    • CHUNK_LAST_BYTE is the ending byte in the overall object that the chunk you're uploading contains.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

    An example Content-Range is Content-Range: bytes 0-524287/2000000. See Content-Range for more information about this header.

    If the request succeeds, the server responds with 308 Resume Incomplete. The response contains a Range header. Use the upper value in this header to determine where to start the next chunk. You should not assume that the server received all bytes sent in the request.

  3. Repeat the above steps for each remaining chunk of the data you want to upload.

XML API

  1. Create a chunk of data from the file you want to upload.

    The chunk size should be a multiple of 256 KiB (256 x 1024 bytes), unless it's the last chunk that completes the upload. Larger chunk sizes typically make uploads more efficient. We recommend using 8 MiB for the chunk size.

  2. Use cURL to call the XML API with a PUT Object request:

    curl -i -X PUT --data-binary @CHUNK_LOCATION \
    -H "Content-Length: CHUNK_SIZE" \
    -H "Content-Range: bytes CHUNK_FIRST_BYTE-CHUNK_LAST_BYTE/TOTAL_OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • CHUNK_LOCATION is the local path to the chunk that you're currently uploading.
    • CHUNK_SIZE is the number of bytes you're uploading in the current request. For example, 524288.
    • CHUNK_FIRST_BYTE is the starting byte in the overall object that the chunk you're uploading contains.
    • CHUNK_LAST_BYTE is the ending byte in the overall object that the chunk you're uploading contains.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

    An example Content-Range is Content-Range: bytes 0-524287/2000000. See Content-Range for more information about this header.

    If the request succeeds, the server responds with 308 Resume Incomplete. The response contains a Range header. Use the upper value in this header to determine where to start the next chunk. You should not assume that the server received all bytes sent in the request.

  3. Repeat the above steps for each remaining chunk of the data you want to upload.

Once the upload completes in its entirety, you receive a 200 OK or 201 Created response, along with any metadata associated with the resource.

If any of the chunk uploads are interrupted, or if you receive a 5xx response, follow the procedure in Resuming an interrupted upload.

Checking the status of a resumable upload

If your resumable upload is interrupted, or you're not sure the upload completed, you can check the status of the upload:

JSON API

  1. Use cURL to call the XML API with a PUT Object request:

    curl -i -X PUT \
    -H "Content-Length: 0" \
    -H "Content-Range: bytes */OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • OBEJCT_SIZE is the total number of bytes in your object. If you don't know the full size of your object, use * for this value.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

XML API

  1. Use cURL to call the XML API with a PUT Object request:

    curl -i -X PUT \
    -H "Content-Length: 0" \
    -H "Content-Range: bytes */OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • OBEJCT_SIZE is the total number of bytes in your object. If you don't know the full size of your object, use * for this value.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

A 200 OK or 201 Created response indicates that the upload was completed, and no further action is necessary.

A 308 Resume Incomplete response indicates that you need to continue uploading the file.

  • If Cloud Storage has not yet received any bytes, the 308 response does not have a Range header. In this case, you should start your upload from the beginning.
  • Otherwise, the 308 response has a Range header, which specifies which bytes Cloud Storage has received so far. Use this value when resuming an interrupted upload.

Resuming an interrupted upload

If an upload request is terminated before receiving a response, or if you receive a 503 or 500 response, then you need to resume the interrupted upload from where it left off. To resume an interrupted upload:

JSON API

  1. Check the status of your resumable upload.

  2. Save the upper value of the Range header contained in the response to your status check. If the response does not have a Range header, Cloud Storage has not yet received any bytes, and you should upload from the beginning.

  3. Ensure that that object data you're about to upload begins at the byte following the upper value in the Range header.

  4. Use cURL to call the JSON API with a PUT Object request that picks up at the byte following the value in the Range header:

    curl -i -X PUT --data-binary @PARTIAL_OBJECT_LOCATION \
    -H "Content-Length: UPLOAD_SIZE_REMAINING" \
    -H "Content-Range: bytes NEXT_BYTE-LAST_BYTE/TOTAL_OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • PARTIAL_OBJECT_LOCATION is the local path to the remaining portion of data that you want to upload.
    • UPLOAD_SIZE_REMAINING is the number of bytes you're uploading in the current request. For example, uploading the rest of an object with a total size of 20000000 that was interrupted after bytes 0-42 uploaded would have an UPLOAD_SIZE_REMAINING of 1999957.
    • NEXT_BYTE is the next integer after the value you saved in step 2. For example, if 42 is the upper value in step 2, the value for NEXT_BYTE is 43.
    • LAST_BYTE is the ending byte contained in this PUT request. For example, to finish uploading an object whose total size is 20000000, the value for LAST_BYTE is 1999999.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

XML API

  1. Check the status of your resumable upload.

  2. Save the upper value of the Range header contained in the response to your status check. If the response does not have a Range header, Cloud Storage has not yet received any bytes, and you should upload from the beginning.

  3. Ensure that that object data you're about to upload begins at the byte following the upper value in the Range header.

  4. Use cURL to call the XML API with a PUT Object request that picks up at the byte following the value in the Range header:

    curl -i -X PUT --data-binary @PARTIAL_OBJECT_LOCATION \
    -H "Content-Length: UPLOAD_SIZE_REMAINING" \
    -H "Content-Range: bytes NEXT_BYTE-LAST_BYTE/TOTAL_OBJECT_SIZE" \
    "SESSION_URI"

    Where:

    • PARTIAL_OBJECT_LOCATION is the local path to the remaining portion of data that you want to upload.
    • UPLOAD_SIZE_REMAINING is the number of bytes you're uploading in the current request. For example, uploading the rest of an object with a total size of 20000000 that was interrupted after bytes 0-42 uploaded would have an UPLOAD_SIZE_REMAINING of 1999957.
    • NEXT_BYTE is the next integer after the value you saved in step 2. For example, if 42 is the upper value in step 2, the value for NEXT_BYTE is 43.
    • LAST_BYTE is the ending byte contained in this PUT request. For example, to finish uploading an object whose total size is 20000000, the value for LAST_BYTE is 1999999.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

You can resume uploads as many times as necessary while the session URI is active; the session URI expires after a week. When the file is successfully uploaded, Cloud Storage responds with a 200 OK or 201 created status code.

Cancelling an upload

To cancel a resumable upload and prevent any further action on it:

JSON API

  1. Use cURL to call the JSON API with a DELETE request:

    curl -i -X DELETE -H "Content-Length: 0" \
  "SESSION_URI"

    Where:

If successful, the response contains a 499 status code. Future attempts to query or resume the upload result in a 4xx response.

XML API

  1. Use cURL to call the XML API with a DELETE request:

    curl -i -X DELETE -H "Content-Length: 0" \
  "SESSION_URI"

    Where:

If successful, the response contains a 204 status code, and future attempts to query or resume the upload also result in a 204 response.

What's next?