Cloud Storage supports streaming data to a bucket without requiring that the data first be saved to a file. This is useful when you want to upload data but don't know the final size at the start of the upload, such as when generating the upload data from a process, or when compressing an object on-the-fly.
Using checksum validation when streaming
Because a checksum can only be supplied in the initial request of an upload, it's often not feasible to use Cloud Storage's checksum validation when streaming. It's recommended that you always use checksum validation, and you can manually do so after a streaming upload completes; however, validating after the transfer completes means that any corrupted data is accessible during the time it takes to confirm the corruption and remove it.
If you require checksum validation prior to the upload completing and the data becoming accessible, then you should not use a streaming upload. You should use a different upload option that performs checksum validation prior to finalizing the object.
Prerequisites
Prerequisites can vary based on the tool used:
Console
In order to complete this guide using the Google Cloud console, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.
For a list of permissions required for specific actions, see IAM permissions for the Google Cloud console.
For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
Command line
In order to complete this guide using a command-line utility, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.
For a list of permissions required for specific actions, see IAM permissions for gsutil commands.
For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
Client libraries
In order to complete this guide using the Cloud Storage client libraries, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.
Unless otherwise noted, client library requests are made through the JSON API and require permissions as listed in IAM permissions for JSON methods. To see which JSON API methods are invoked when you make requests using a client library, log the raw requests.
For a list of relevant IAM roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
REST APIs
JSON API
In order to complete this guide using the JSON API, you must have the proper IAM permissions. If the bucket you want to access for streaming exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.
For a list of permissions required for specific actions, see IAM permissions for JSON methods.
For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.
Stream an upload
The following examples show how to perform a streaming upload from a process to a Cloud Storage object:
Console
The Google Cloud console does not support streaming uploads. Use the gcloud CLI instead.
Command line
gcloud
Pipe the data to the
gcloud storage cp
command and use a dash for the source URL:PROCESS_NAME | gcloud storage cp - gs://BUCKET_NAME/OBJECT_NAME
Where:
PROCESS_NAME
is the name of the process from which you are collecting data. For example,collect_measurements
.BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that is created from the data. For example,data_measurements
.
gsutil
Pipe the data to the
gsutil cp
command and use a dash for the source URL:PROCESS_NAME | gsutil cp - gs://BUCKET_NAME/OBJECT_NAME
Where:
PROCESS_NAME
is the name of the process from which you are collecting data. For example,collect_measurements
.BUCKET_NAME
is the name of the bucket containing the object. For example,my_app_bucket
.OBJECT_NAME
is the name of the object that is created from the data. For example,data_measurements
.
Client libraries
For more information, see the
Cloud Storage C++ API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage C# API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Go API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Java API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Node.js API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage PHP API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Python API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
For more information, see the
Cloud Storage Ruby API
reference documentation.
To authenticate to Cloud Storage, set up Application Default Credentials.
For more information, see
Set up authentication for a local development environment.
C++
C#
Go
Java
Node.js
PHP
Python
Ruby
REST APIs
JSON API
To perform a streaming upload, follow the instructions for performing a resumable upload with the following considerations:
When uploading the file data itself, use a multiple chunk upload.
Since you don't know the total file size until you get to the final chunk, use a
*
for the total file size in theContent-Range
header of intermediate chunks.For example, if the first chunk you upload has a size of 512 KiB, the
Content-Range
header for the chunk isbytes 0-524287/*
. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has aContent-Range
header with the valuebytes 524288-588287/588288
.
XML API
To perform a streaming upload, use one of the following methods:
A resumable upload, with the following adjustments:
When uploading the file data itself, use a multiple chunk upload.
Since you don't know the total file size until you get to the final chunk, use a
*
for the total file size in theContent-Range
header of intermediate chunks.For example, if the first chunk you upload has a size of 512 KiB, the
Content-Range
header for the chunk isbytes 0-524287/*
. If your upload has 64000 bytes remaining after the first chunk, you then send a final chunk that contains the remaining bytes and has aContent-Range
header with the valuebytes 524288-588287/588288
.
What's next
- Stream a download.
- Learn more about decompressive transcoding.
- Learn more about uploads and downloads.