Perform resumable uploads

Overview

This page describes how to make a resumable upload request in the Cloud Storage JSON and XML APIs. This protocol lets you resume an upload operation after a communication failure interrupts the flow of data.

For information on using resumable uploads in the Google Cloud CLI and client libraries, see How tools and APIs use resumable uploads.

Required roles

To get the permissions that you need to perform a resumable upload, ask your administrator to grant you the Storage Object User (roles/storage.objectUser) IAM role for the bucket. This predefined role contains the permissions required to upload an object to a bucket. To see the exact permissions that are required, expand the Required permissions section:

Required permissions

  • storage.objects.create
  • storage.objects.delete
    • This permission is only required for uploads that overwrite an existing object.

You can also get these permissions with other predefined roles or custom roles.

For information about granting roles for buckets, see Use IAM with buckets.

Initiate a resumable upload session

To initiate a resumable upload session:

JSON API

  1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials. For instructions, see API authentication.
  2. Optionally, create a JSON file that contains the metadata you want to set on the object that you're uploading. For example, the following JSON file sets the contentType metadata of the object you want to upload to image/png:

    {
        "contentType": "image/png"
    }
  3. Use cURL to call the JSON API with a POST Object request:

    curl -i -X POST --data-binary @METADATA_LOCATION \
        -H "Authorization: Bearer OAUTH2_TOKEN" \
        -H "Content-Type: application/json" \
        -H "Content-Length: INITIAL_REQUEST_LENGTH" \
        "https://storage.googleapis.com/upload/storage/v1/b/BUCKET_NAME/o?uploadType=resumable&name=OBJECT_NAME"

    Where:

    • METADATA_LOCATION is the local path to the JSON file containing the optional metadata you specified in the previous step. If you are not including a metadata file, exclude this, along with --data-binary @ and the Content-Type header.
    • OAUTH2_TOKEN is the access token you generated in Step 1.
    • INITIAL_REQUEST_LENGTH is the number of bytes in the body of this initial request, for example 79. Not required if you are using chunked transfer encoding.
    • BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.
    • OBJECT_NAME is the URL-encoded name you want to give your object. For example, pets/dog.png, URL-encoded as pets%2Fdog.png. This is not required if you included a name in the object metadata file in Step 2.

    If you have enabled Cross-Origin Resource Sharing, you should also include an Origin header in both this and subsequent upload requests.

    Optional headers that you can add to the request include X-Upload-Content-Type and X-Upload-Content-Length.

    If successful, the response includes a 200 status code.

  4. Save the resumable session URI given in the Location header of the response to your POST Object request.

    This URI is used in subsequent requests to upload the object data.

XML API

  1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials. For instructions, see API authentication.
  2. Use cURL to call the XML API with a POST Object request that has an empty body:

    curl -i -X POST -H "Authorization: Bearer OAUTH2_TOKEN" \
        -H "Content-Length: 0" \
        -H "Content-Type: OBJECT_CONTENT_TYPE" \
        -H "x-goog-resumable: start" \
        "https://storage.googleapis.com/BUCKET_NAME/OBJECT_NAME"

    Where:

    • OAUTH2_TOKEN is the access token you generated in Step 1.
    • OBJECT_CONTENT_TYPE is the content type of the object. For example, image/png. If you do not specify a content type, the Cloud Storage system sets the Content-Type metadata for the object to be application/octet-stream.
    • BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.
    • OBJECT_NAME is the URL-encoded name you want to give your object. For example, pets/dog.png, URL-encoded as pets%2Fdog.png.

    If you have enabled Cross-Origin Resource Sharing, you should also include an Origin header in both this and subsequent upload requests.

    If successful, the response includes a 201 status message.

  3. Save the resumable session URI given in the Location header of the response to your POST Object request.

    This URI is used in subsequent requests to upload the object data.

Upload the data

Once you have initiated a resumable upload, there are two ways to upload the object's data:

  • In a single chunk: This approach is usually best, since it requires fewer requests and thus has better performance.
  • In multiple chunks: Use this approach if you need to reduce the amount of data transferred in any single request, such as when there is a fixed time limit for individual requests, or if you don't know the total size of the upload at the time the upload begins.

Single chunk upload

To upload the data in a single chunk:

JSON API

  1. Use cURL to call the JSON API with a PUT Object request:

    curl -i -X PUT --data-binary @OBJECT_LOCATION \
        -H "Content-Length: OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.
    • OBJECT_SIZE is the number of bytes in your object. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

    Optionally, you can include headers prefixed with X-Goog-Meta- to add custom metadata for the object as part of this request.

XML API

  1. Use cURL to call the XML API with a PUT Object request:

    curl -i -X PUT --data-binary @OBJECT_LOCATION \
        -H "Content-Length: OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.
    • OBJECT_SIZE is the number of bytes in your object. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

If the upload completes in its entirety, you receive a 200 OK or 201 Created response, along with any metadata associated with the resource.

If the upload request is interrupted, or if you receive a 5xx response, follow the procedure in Resuming an interrupted upload.

Multiple chunk upload

To upload the data in multiple chunks:

JSON API

  1. Create a chunk of data from the overall data you want to upload.

    The chunk size should be a multiple of 256 KiB (256 x 1024 bytes), unless it's the last chunk that completes the upload. Larger chunk sizes typically make uploads faster, but note that there's a tradeoff between speed and memory usage. It's recommended that you use at least 8 MiB for the chunk size.

  2. Use cURL to call the JSON API with a PUT Object request:

    curl -i -X PUT --data-binary @CHUNK_LOCATION \
        -H "Content-Length: CHUNK_SIZE" \
        -H "Content-Range: bytes CHUNK_FIRST_BYTE-CHUNK_LAST_BYTE/TOTAL_OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • CHUNK_LOCATION is the local path to the chunk that you're currently uploading.
    • CHUNK_SIZE is the number of bytes you're uploading in the current request. For example, 8388608.
    • CHUNK_FIRST_BYTE is the starting byte in the overall object that the chunk you're uploading contains.
    • CHUNK_LAST_BYTE is the ending byte in the overall object that the chunk you're uploading contains.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

    An example Content-Range is Content-Range: bytes 0-8388607/20000000. See Content-Range for more information about this header.

    If the request succeeds, the server responds with 308 Resume Incomplete. The response contains a Range header.

  3. Repeat the above steps for each remaining chunk of data that you want to upload, using the upper value contained in the Range header of each response to determine where to start each successive chunk; you should not assume that the server received all bytes sent in any given request.

    Optionally, in the final request of the resumable upload you can include headers prefixed with X-Goog-Meta- to add custom metadata for the object.

XML API

  1. Create a chunk of data from the overall data you want to upload.

    The chunk size should be a multiple of 256 KiB (256 x 1024 bytes), unless it's the last chunk that completes the upload. Larger chunk sizes typically make uploads faster, but note that there's a tradeoff between speed and memory usage. It's recommended that you use at least 8 MiB for the chunk size.

  2. Use cURL to call the XML API with a PUT Object request:

    curl -i -X PUT --data-binary @CHUNK_LOCATION \
        -H "Content-Length: CHUNK_SIZE" \
        -H "Content-Range: bytes CHUNK_FIRST_BYTE-CHUNK_LAST_BYTE/TOTAL_OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • CHUNK_LOCATION is the local path to the chunk that you're currently uploading.
    • CHUNK_SIZE is the number of bytes you're uploading in the current request. For example, 8388608.
    • CHUNK_FIRST_BYTE is the starting byte in the overall object that the chunk you're uploading contains.
    • CHUNK_LAST_BYTE is the ending byte in the overall object that the chunk you're uploading contains.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

    An example Content-Range is Content-Range: bytes 0-8388607/20000000. See Content-Range for more information about this header.

    If the request succeeds, the server responds with 308 Resume Incomplete. The response contains a Range header.

  3. Repeat the above steps for each remaining chunk of data that you want to upload, using the upper value contained in the Range header of each response to determine where to start each successive chunk; you should not assume that the server received all bytes sent in any given request.

Once the upload completes in its entirety, you receive a 200 OK or 201 Created response, along with any metadata associated with the resource.

If any of the chunk uploads are interrupted, or if you receive a 5xx response, you should either resend the interrupted chunk in its entirety or resume the interrupted upload from where it left off.

Check the status of a resumable upload

If your resumable upload is interrupted, or you're not sure the upload completed, you can check the status of the upload:

JSON API

  1. Use cURL to call the JSON API with a PUT Object request:

    curl -i -X PUT \
        -H "Content-Length: 0" \
        -H "Content-Range: bytes */OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • OBJECT_SIZE is the total number of bytes in your object. If you don't know the full size of your object, use * for this value.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

XML API

  1. Use cURL to call the XML API with a PUT Object request:

    curl -i -X PUT \
        -H "Content-Length: 0" \
        -H "Content-Range: bytes */OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • OBJECT_SIZE is the total number of bytes in your object. If you don't know the full size of your object, use * for this value.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

A 200 OK or 201 Created response indicates that the upload was completed, and no further action is necessary.

A 308 Resume Incomplete response indicates that you need to continue uploading the data.

  • If Cloud Storage has not yet persisted any bytes, the 308 response does not have a Range header. In this case, you should start your upload from the beginning.
  • Otherwise, the 308 response has a Range header, which specifies which bytes Cloud Storage has persisted so far. Use this value when resuming an interrupted upload.

Resume an interrupted upload

If an upload request is terminated before receiving a response, or if you receive a 503 or 500 response, then you need to resume the interrupted upload from where it left off. To resume an interrupted upload:

JSON API

  1. Check the status of your resumable upload.

  2. Save the upper value of the Range header contained in the response to your status check. If the response does not have a Range header, Cloud Storage has not yet persisted any bytes, and you should upload from the beginning.

  3. Ensure that that object data you're about to upload begins at the byte following the upper value in the Range header.

  4. If the interrupted request contained headers prefixed with X-Goog-Meta-, include those headers in the request to resume your upload.

  5. Use cURL to call the JSON API with a PUT Object request that picks up at the byte following the value in the Range header:

    curl -i -X PUT --data-binary @PARTIAL_OBJECT_LOCATION \
        -H "Content-Length: UPLOAD_SIZE_REMAINING" \
        -H "Content-Range: bytes NEXT_BYTE-LAST_BYTE/TOTAL_OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • PARTIAL_OBJECT_LOCATION is the local path to the remaining portion of data that you want to upload.
    • UPLOAD_SIZE_REMAINING is the number of bytes you're uploading in the current request. For example, uploading the rest of an object with a total size of 20000000 that was interrupted after bytes 0-42 uploaded would have an UPLOAD_SIZE_REMAINING of 19999957.
    • NEXT_BYTE is the next integer after the value you saved in step 2. For example, if 42 is the upper value in step 2, the value for NEXT_BYTE is 43.
    • LAST_BYTE is the ending byte contained in this PUT request. For example, to finish uploading an object whose total size is 20000000, the value for LAST_BYTE is 19999999.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

XML API

  1. Check the status of your resumable upload.

  2. Save the upper value of the Range header contained in the response to your status check. If the response does not have a Range header, Cloud Storage has not yet persisted any bytes, and you should upload from the beginning.

  3. Ensure that that object data you're about to upload begins at the byte following the upper value in the Range header.

  4. Use cURL to call the XML API with a PUT Object request that picks up at the byte following the value in the Range header:

    curl -i -X PUT --data-binary @PARTIAL_OBJECT_LOCATION \
        -H "Content-Length: UPLOAD_SIZE_REMAINING" \
        -H "Content-Range: bytes NEXT_BYTE-LAST_BYTE/TOTAL_OBJECT_SIZE" \
        "SESSION_URI"

    Where:

    • PARTIAL_OBJECT_LOCATION is the local path to the remaining portion of data that you want to upload.
    • UPLOAD_SIZE_REMAINING is the number of bytes you're uploading in the current request. For example, uploading the rest of an object with a total size of 20000000 that was interrupted after bytes 0-42 uploaded would have an UPLOAD_SIZE_REMAINING of 19999957.
    • NEXT_BYTE is the next integer after the value you saved in step 2. For example, if 42 is the upper value in step 2, the value for NEXT_BYTE is 43.
    • LAST_BYTE is the ending byte contained in this PUT request. For example, to finish uploading an object whose total size is 20000000, the value for LAST_BYTE is 19999999.
    • TOTAL_OBJECT_SIZE is the total size of the object you are uploading. For example, 20000000.
    • SESSION_URI is the value returned in the Location header when you initiated the resumable upload.

You can resume uploads as many times as necessary while the session URI is active; the session URI expires after a week. When the data is successfully uploaded, Cloud Storage responds with a 200 OK or 201 created status code.

Cancel an upload

To cancel an incomplete resumable upload and prevent any further action on it:

JSON API

  1. Use cURL to call the JSON API with a DELETE request:

    curl -i -X DELETE -H "Content-Length: 0" \
      "SESSION_URI"

    Where:

If successful, the response contains a 499 status code. Future attempts to query or resume the upload result in a 4xx response.

XML API

  1. Use cURL to call the XML API with a DELETE request:

    curl -i -X DELETE -H "Content-Length: 0" \
      "SESSION_URI"

    Where:

If successful, the response contains a 204 status code, and future attempts to query or resume the upload also result in a 204 response.

What's next