Upload objects from a filesystem

This page shows you how to upload objects to your Cloud Storage bucket from your local filesystem. An uploaded object consists of the data you want to store along with any associated metadata. For a conceptual overview, including how to choose the optimal upload method based on your file size, see Uploads and downloads.

For instructions on uploading from memory, see Upload objects from memory.

Required permissions

Console

In order to complete this guide using the Google Cloud console, you must have the proper IAM permissions. If the bucket you want to upload to exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.

For a list of permissions required for specific actions, see IAM permissions for the Google Cloud console.

For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.

Command line

In order to complete this guide using a command-line utility, you must have the proper IAM permissions. If the bucket you want to upload to exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.

For a list of permissions required for specific actions, see IAM permissions for gsutil commands.

For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.

Client libraries

In order to complete this guide using the Cloud Storage client libraries, you must have the proper IAM permissions. If the bucket you want to upload to exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.

Unless otherwise noted, client library requests are made through the JSON API and require permissions as listed in IAM permissions for JSON methods. To see which JSON API methods are invoked when you make requests using a client library, log the raw requests.

For a list of relevant IAM roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.

REST APIs

JSON API

In order to complete this guide using the JSON API, you must have the proper IAM permissions. If the bucket you want to upload to exists in a project that you did not create, you might need the project owner to give you a role that contains the necessary permissions.

For a list of permissions required for specific actions, see IAM permissions for JSON methods.

For a list of relevant roles, see Cloud Storage roles. Alternatively, you can create a custom role that has specific, limited permissions.

Upload an object to a bucket

Complete the following steps to upload an object to a bucket:

Console

  1. In the Google Cloud console, go to the Cloud Storage Buckets page.

    Go to Buckets

  2. In the list of buckets, click on the name of the bucket that you want to upload an object to.

  3. In the Objects tab for the bucket, either:

    • Drag and drop the desired files from your desktop or file manager to the main pane in the Google Cloud console.

    • Click the Upload Files button, select the files you want to upload in the dialog that appears, and click Open.

To learn how to get detailed error information about failed Cloud Storage operations in the Google Cloud console, see Troubleshooting.

Command line

gcloud

Use the gcloud storage cp command:

gcloud storage cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME/

Where:

  • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.

  • DESTINATION_BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.

If successful, the response looks like the following example:

Completed files 1/1 | 164.3kiB/164.3kiB

You can set fixed-key and custom object metadata as part of your object upload by using command flags.

gsutil

Use the gsutil cp command:

gsutil cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME/

Where:

  • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.

  • DESTINATION_BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.

If successful, the response looks like the following example:

Operation completed over 1 objects/58.8 KiB.

You can set fixed-key and custom object metadata as part of your object upload in the headers of the request by using the -h global option.

Client libraries

C++

For more information, see the Cloud Storage C++ API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

google/cloud/storage/examples/storage_object_file_transfer_samples.cc 
namespace gcs = ::google::cloud::storage;
using ::google::cloud::StatusOr;
[](gcs::Client client, std::string const& file_name,
   std::string const& bucket_name, std::string const& object_name) {
  // Note that the client library automatically computes a hash on the
  // client-side to verify data integrity during transmission.
  StatusOr<gcs::ObjectMetadata> metadata = client.UploadFile(
      file_name, bucket_name, object_name, gcs::IfGenerationMatch(0));
  if (!metadata) throw std::move(metadata).status();

  std::cout << "Uploaded " << file_name << " to object " << metadata->name()
            << " in bucket " << metadata->bucket()
            << "\nFull metadata: " << *metadata << "\n";
}

C#

For more information, see the Cloud Storage C# API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

storage/api/Storage.Samples/UploadFile.cs 

using Google.Cloud.Storage.V1;
using System;
using System.IO;

public class UploadFileSample
{
    public void UploadFile(
        string bucketName = "your-unique-bucket-name",
        string localPath = "my-local-path/my-file-name",
        string objectName = "my-file-name")
    {
        var storage = StorageClient.Create();
        using var fileStream = File.OpenRead(localPath);
        storage.UploadObject(bucketName, objectName, null, fileStream);
        Console.WriteLine($"Uploaded {objectName}.");
    }
}

Go

For more information, see the Cloud Storage Go API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

storage/objects/upload_file.go 
import (
	"context"
	"fmt"
	"io"
	"os"
	"time"

	"cloud.google.com/go/storage"
)

// uploadFile uploads an object.
func uploadFile(w io.Writer, bucket, object string) error {
	// bucket := "bucket-name"
	// object := "object-name"
	ctx := context.Background()
	client, err := storage.NewClient(ctx)
	if err != nil {
		return fmt.Errorf("storage.NewClient: %w", err)
	}
	defer client.Close()

	// Open local file.
	f, err := os.Open("notes.txt")
	if err != nil {
		return fmt.Errorf("os.Open: %w", err)
	}
	defer f.Close()

	ctx, cancel := context.WithTimeout(ctx, time.Second*50)
	defer cancel()

	o := client.Bucket(bucket).Object(object)

	// Optional: set a generation-match precondition to avoid potential race
	// conditions and data corruptions. The request to upload is aborted if the
	// object's generation number does not match your precondition.
	// For an object that does not yet exist, set the DoesNotExist precondition.
	o = o.If(storage.Conditions{DoesNotExist: true})
	// If the live object already exists in your bucket, set instead a
	// generation-match precondition using the live object's generation number.
	// attrs, err := o.Attrs(ctx)
	// if err != nil {
	// 	return fmt.Errorf("object.Attrs: %w", err)
	// }
	// o = o.If(storage.Conditions{GenerationMatch: attrs.Generation})

	// Upload an object with storage.Writer.
	wc := o.NewWriter(ctx)
	if _, err = io.Copy(wc, f); err != nil {
		return fmt.Errorf("io.Copy: %w", err)
	}
	if err := wc.Close(); err != nil {
		return fmt.Errorf("Writer.Close: %w", err)
	}
	fmt.Fprintf(w, "Blob %v uploaded.\n", object)
	return nil
}

Java

For more information, see the Cloud Storage Java API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

samples/snippets/src/main/java/com/example/storage/object/UploadObject.java 

import com.google.cloud.storage.BlobId;
import com.google.cloud.storage.BlobInfo;
import com.google.cloud.storage.Storage;
import com.google.cloud.storage.StorageOptions;
import java.io.IOException;
import java.nio.file.Paths;

public class UploadObject {
  public static void uploadObject(
      String projectId, String bucketName, String objectName, String filePath) throws IOException {
    // The ID of your GCP project
    // String projectId = "your-project-id";

    // The ID of your GCS bucket
    // String bucketName = "your-unique-bucket-name";

    // The ID of your GCS object
    // String objectName = "your-object-name";

    // The path to your file to upload
    // String filePath = "path/to/your/file"

    Storage storage = StorageOptions.newBuilder().setProjectId(projectId).build().getService();
    BlobId blobId = BlobId.of(bucketName, objectName);
    BlobInfo blobInfo = BlobInfo.newBuilder(blobId).build();

    // Optional: set a generation-match precondition to avoid potential race
    // conditions and data corruptions. The request returns a 412 error if the
    // preconditions are not met.
    Storage.BlobWriteOption precondition;
    if (storage.get(bucketName, objectName) == null) {
      // For a target object that does not yet exist, set the DoesNotExist precondition.
      // This will cause the request to fail if the object is created before the request runs.
      precondition = Storage.BlobWriteOption.doesNotExist();
    } else {
      // If the destination already exists in your bucket, instead set a generation-match
      // precondition. This will cause the request to fail if the existing object's generation
      // changes before the request runs.
      precondition =
          Storage.BlobWriteOption.generationMatch(
              storage.get(bucketName, objectName).getGeneration());
    }
    storage.createFrom(blobInfo, Paths.get(filePath), precondition);

    System.out.println(
        "File " + filePath + " uploaded to bucket " + bucketName + " as " + objectName);
  }
}

Node.js

For more information, see the Cloud Storage Node.js API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

The following sample uploads an individual object:

samples/uploadFile.js 
/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The path to your file to upload
// const filePath = 'path/to/your/file';

// The new ID for your GCS file
// const destFileName = 'your-new-file-name';

// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

async function uploadFile() {
  const options = {
    destination: destFileName,
    // Optional:
    // Set a generation-match precondition to avoid potential race conditions
    // and data corruptions. The request to upload is aborted if the object's
    // generation number does not match your precondition. For a destination
    // object that does not yet exist, set the ifGenerationMatch precondition to 0
    // If the destination object already exists in your bucket, set instead a
    // generation-match precondition using its generation number.
    preconditionOpts: {ifGenerationMatch: generationMatchPrecondition},
  };

  await storage.bucket(bucketName).upload(filePath, options);
  console.log(`${filePath} uploaded to ${bucketName}`);
}

uploadFile().catch(console.error);

The following sample uploads multiple objects concurrently:

samples/uploadManyFilesWithTransferManager.js 
/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The ID of the first GCS file to download
// const firstFileName = 'your-first-file-name';

// The ID of the second GCS file to download
// const secondFileName = 'your-second-file-name;

// Imports the Google Cloud client library
const {Storage, TransferManager} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

// Creates a transfer manager client
const transferManager = new TransferManager(storage.bucket(bucketName));

async function uploadManyFilesWithTransferManager() {
  // Uploads the files
  await transferManager.uploadManyFiles([firstFilePath, secondFilePath]);

  for (const filePath of [firstFilePath, secondFilePath]) {
    console.log(`${filePath} uploaded to ${bucketName}.`);
  }
}

uploadManyFilesWithTransferManager().catch(console.error);

The following sample uploads all objects with a common prefix concurrently:

samples/uploadDirectoryWithTransferManager.js 
/**
 * TODO(developer): Uncomment the following lines before running the sample.
 */
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';

// The local directory to upload
// const directoryName = 'your-directory';

// Imports the Google Cloud client library
const {Storage, TransferManager} = require('@google-cloud/storage');

// Creates a client
const storage = new Storage();

// Creates a transfer manager client
const transferManager = new TransferManager(storage.bucket(bucketName));

async function uploadDirectoryWithTransferManager() {
  // Uploads the directory
  await transferManager.uploadManyFiles(directoryName);

  console.log(`${directoryName} uploaded to ${bucketName}.`);
}

uploadDirectoryWithTransferManager().catch(console.error);

PHP

For more information, see the Cloud Storage PHP API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

storage/src/upload_object.php 
use Google\Cloud\Storage\StorageClient;

/**
 * Upload a file.
 *
 * @param string $bucketName The name of your Cloud Storage bucket.
 *        (e.g. 'my-bucket')
 * @param string $objectName The name of your Cloud Storage object.
 *        (e.g. 'my-object')
 * @param string $source The path to the file to upload.
 *        (e.g. '/path/to/your/file')
 */
function upload_object(string $bucketName, string $objectName, string $source): void
{
    $storage = new StorageClient();
    if (!$file = fopen($source, 'r')) {
        throw new \InvalidArgumentException('Unable to open file for reading');
    }
    $bucket = $storage->bucket($bucketName);
    $object = $bucket->upload($file, [
        'name' => $objectName
    ]);
    printf('Uploaded %s to gs://%s/%s' . PHP_EOL, basename($source), $bucketName, $objectName);
}

Python

For more information, see the Cloud Storage Python API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

The following sample uploads an individual object:

samples/snippets/storage_upload_file.py 
from google.cloud import storage


def upload_blob(bucket_name, source_file_name, destination_blob_name):
    """Uploads a file to the bucket."""
    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"
    # The path to your file to upload
    # source_file_name = "local/path/to/file"
    # The ID of your GCS object
    # destination_blob_name = "storage-object-name"

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(destination_blob_name)

    # Optional: set a generation-match precondition to avoid potential race conditions
    # and data corruptions. The request to upload is aborted if the object's
    # generation number does not match your precondition. For a destination
    # object that does not yet exist, set the if_generation_match precondition to 0.
    # If the destination object already exists in your bucket, set instead a
    # generation-match precondition using its generation number.
    generation_match_precondition = 0

    blob.upload_from_filename(source_file_name, if_generation_match=generation_match_precondition)

    print(
        f"File {source_file_name} uploaded to {destination_blob_name}."
    )

The following sample uploads multiple objects concurrently:

samples/snippets/storage_transfer_manager_upload_many.py 
def upload_many_blobs_with_transfer_manager(
    bucket_name, filenames, source_directory="", processes=8
):
    """Upload every file in a list to a bucket, concurrently in a process pool.

    Each blob name is derived from the filename, not including the
    `source_directory` parameter. For complete control of the blob name for each
    file (and other aspects of individual blob metadata), use
    transfer_manager.upload_many() instead.
    """

    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"

    # A list (or other iterable) of filenames to upload.
    # filenames = ["file_1.txt", "file_2.txt"]

    # The directory on your computer that is the root of all of the files in the
    # list of filenames. This string is prepended (with os.path.join()) to each
    # filename to get the full path to the file. Relative paths and absolute
    # paths are both accepted. This string is not included in the name of the
    # uploaded blob; it is only used to find the source files. An empty string
    # means "the current working directory". Note that this parameter allows
    # directory traversal (e.g. "/", "../") and is not intended for unsanitized
    # end user input.
    # source_directory=""

    # The maximum number of processes to use for the operation. The performance
    # impact of this value depends on the use case, but smaller files usually
    # benefit from a higher number of processes. Each additional process occupies
    # some CPU and memory resources until finished.
    # processes=8

    from google.cloud.storage import Client, transfer_manager

    storage_client = Client()
    bucket = storage_client.bucket(bucket_name)

    results = transfer_manager.upload_many_from_filenames(
        bucket, filenames, source_directory=source_directory, max_workers=processes
    )

    for name, result in zip(filenames, results):
        # The results list is either `None` or an exception for each filename in
        # the input list, in order.

        if isinstance(result, Exception):
            print("Failed to upload {} due to exception: {}".format(name, result))
        else:
            print("Uploaded {} to {}.".format(name, bucket.name))

The following sample uploads all objects with a common prefix concurrently:

samples/snippets/storage_transfer_manager_upload_directory.py 
def upload_directory_with_transfer_manager(bucket_name, source_directory, processes=8):
    """Upload every file in a directory, including all files in subdirectories.

    Each blob name is derived from the filename, not including the `directory`
    parameter itself. For complete control of the blob name for each file (and
    other aspects of individual blob metadata), use
    transfer_manager.upload_many() instead.
    """

    # The ID of your GCS bucket
    # bucket_name = "your-bucket-name"

    # The directory on your computer to upload. Files in the directory and its
    # subdirectories will be uploaded. An empty string means "the current
    # working directory".
    # source_directory=""

    # The maximum number of processes to use for the operation. The performance
    # impact of this value depends on the use case, but smaller files usually
    # benefit from a higher number of processes. Each additional process occupies
    # some CPU and memory resources until finished.
    # processes=8

    from pathlib import Path

    from google.cloud.storage import Client, transfer_manager

    storage_client = Client()
    bucket = storage_client.bucket(bucket_name)

    # Generate a list of paths (in string form) relative to the `directory`.
    # This can be done in a single list comprehension, but is expanded into
    # multiple lines here for clarity.

    # First, recursively get all files in `directory` as Path objects.
    directory_as_path_obj = Path(source_directory)
    paths = directory_as_path_obj.rglob("*")

    # Filter so the list only includes files, not directories themselves.
    file_paths = [path for path in paths if path.is_file()]

    # These paths are relative to the current working directory. Next, make them
    # relative to `directory`
    relative_paths = [path.relative_to(source_directory) for path in file_paths]

    # Finally, convert them all to strings.
    string_paths = [str(path) for path in relative_paths]

    print("Found {} files.".format(len(string_paths)))

    # Start the upload.
    results = transfer_manager.upload_many_from_filenames(
        bucket, string_paths, source_directory=source_directory, max_workers=processes
    )

    for name, result in zip(string_paths, results):
        # The results list is either `None` or an exception for each filename in
        # the input list, in order.

        if isinstance(result, Exception):
            print("Failed to upload {} due to exception: {}".format(name, result))
        else:
            print("Uploaded {} to {}.".format(name, bucket.name))

Ruby

For more information, see the Cloud Storage Ruby API reference documentation.

To authenticate to Cloud Storage, set up Application Default Credentials. For more information, see Set up authentication for a local development environment.

google-cloud-storage/samples/storage_upload_file.rb 
def upload_file bucket_name:, local_file_path:, file_name: nil
  # The ID of your GCS bucket
  # bucket_name = "your-unique-bucket-name"

  # The path to your file to upload
  # local_file_path = "/local/path/to/file.txt"

  # The ID of your GCS object
  # file_name = "your-file-name"

  require "google/cloud/storage"

  storage = Google::Cloud::Storage.new
  bucket  = storage.bucket bucket_name, skip_lookup: true

  file = bucket.create_file local_file_path, file_name

  puts "Uploaded #{local_file_path} as #{file.name} in bucket #{bucket_name}"
end

Terraform

You can use a Terraform resource to upload an object. Either content or source must be specified.

# Upload files
# Discussion about using tf to upload a large number of objects
# https://stackoverflow.com/questions/68455132/terraform-copy-multiple-files-to-bucket-at-the-same-time-bucket-creation

# The text object in Cloud Storage
resource "google_storage_bucket_object" "default" {
  name = "new-object"
  # Uncomment and add valid path to an object.
  #  source       = "/path/to/an/object"
  content      = "Data as string to be uploaded"
  content_type = "text/plain"
  bucket       = google_storage_bucket.static.id
}

REST APIs

JSON API

The JSON API distinguishes between media uploads, in which only object data is included in the request, and JSON API multipart uploads, in which both object data and object metadata are included in the request.

Media upload (a single-request upload without object metadata)

  1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials. For instructions, see API authentication.

  2. Use cURL to call the JSON API with a POST Object request:

    curl -X POST --data-binary @OBJECT_LOCATION \
    -H "Authorization: Bearer OAUTH2_TOKEN" \
    -H "Content-Type: OBJECT_CONTENT_TYPE" \
    "https://storage.googleapis.com/upload/storage/v1/b/BUCKET_NAME/o?uploadType=media&name=OBJECT_NAME"

    Where:

    • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.
    • OAUTH2_TOKEN is the access token you generated in Step 1.
    • OBJECT_CONTENT_TYPE is the content type of the object. For example, image/png.
    • BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.
    • OBJECT_NAME is the URL-encoded name you want to give your object. For example, pets/dog.png, URL-encoded as pets%2Fdog.png.

JSON API multipart upload (a single-request upload that includes object metadata)

  1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials. For instructions, see API authentication.

  2. Create a multipart/related file that contains the following information:

    --BOUNDARY_STRING
Content-Type: application/json; charset=UTF-8

OBJECT_METADATA

--BOUNDARY_STRING
Content-Type: OBJECT_CONTENT_TYPE

OBJECT_DATA
--BOUNDARY_STRING--

    Where:

    • BOUNDARY_STRING is a string you define that identifies the different parts of the multipart file. For example, separator_string.
    • OBJECT_METADATA is metadata you want to include for the file, in JSON format. At a minimum, this section should include a name attribute for the object, for example {"name": "myObject"}.
    • OBJECT_CONTENT_TYPE is the content type of the object. For example, text/plain.
    • OBJECT_DATA is the data for the object.

    For example:

    --separator_string
Content-Type: application/json; charset=UTF-8

{"name":"my-document.txt"}

--separator_string
Content-Type: text/plain

This is a text file.
--separator_string--

  3. Use cURL to call the JSON API with a POST Object request:

    curl -X POST --data-binary @MULTIPART_FILE_LOCATION \
    -H "Authorization: Bearer OAUTH2_TOKEN" \
    -H "Content-Type: multipart/related; boundary=BOUNDARY_STRING" \
    -H "Content-Length: MULTIPART_FILE_SIZE" \
    "https://storage.googleapis.com/upload/storage/v1/b/BUCKET_NAME/o?uploadType=multipart"

    Where:

    • MULTIPART_FILE_LOCATION is the local path to the multipart file you created in step 2. For example, Desktop/my-upload.multipart.
    • OAUTH2_TOKEN is the access token you generated in Step 1.
    • BOUNDARY_STRING is the boundary string you defined in Step 2. For example, my-boundary.
    • MULTIPART_FILE_SIZE is the total size, in bytes, of the multipart file you created in Step 2. For example, 2000000.
    • BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.

If the request succeeds, the server returns the HTTP 200 OK status code along with the file's metadata.

XML API

  1. Get an authorization access token from the OAuth 2.0 Playground. Configure the playground to use your own OAuth credentials. For instructions, see API authentication.

  2. Use cURL to call the XML API with a PUT Object request:

    curl -X PUT --data-binary @OBJECT_LOCATION \
    -H "Authorization: Bearer OAUTH2_TOKEN" \
    -H "Content-Type: OBJECT_CONTENT_TYPE" \
    "https://storage.googleapis.com/BUCKET_NAME/OBJECT_NAME"

    Where:

    • OBJECT_LOCATION is the local path to your object. For example, Desktop/dog.png.
    • OAUTH2_TOKEN is the access token you generated in Step 1.
    • OBJECT_CONTENT_TYPE is the content type of the object. For example, image/png.
    • BUCKET_NAME is the name of the bucket to which you are uploading your object. For example, my-bucket.
    • OBJECT_NAME is the URL-encoded name you want to give your object. For example, pets/dog.png, URL-encoded as pets%2Fdog.png.

You can set additional object metadata as part of your object upload in the headers of the request in the same way the above example sets Content-Type. When working with the XML API, metadata can only be set at the time the object is written, such as when uploading, copying, or replacing the object. For more information, see Editing object metadata.

What's next

Try it for yourself

If you're new to Google Cloud, create an account to evaluate how Cloud Storage performs in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Try Cloud Storage free