Folders

This page discusses folders in Cloud Storage and how they vary across the Cloud Storage tools.

Note: Folders in the Google Cloud resource hierarchy are different from the folders concept covered in this page, which only applies to buckets and objects in Cloud Storage.

Overview

Cloud Storage operates with a flat namespace, which means that folders don't actually exist within Cloud Storage. If you create an object named folder1/file.txt in the bucket your-bucket, the path to the object is your-bucket/folder1/file.txt, but there is no folder named folder1; instead, the string folder1 is part of the object's name.

However, the Google Cloud console and gsutil provide the illusion of a hierarchical file tree:

• The Google Cloud console creates a visual representation of folders that resembles a local file browser.

• gsutil simulates the typical command-line directory experience using a variety of rules.

Tools

Click the tabs below to see how each tool handles folders.

Console

In the Google Cloud console, you can create an empty folder in a bucket, or upload an existing folder.

When you upload an existing folder, the name of the folder becomes part of the path for all the objects contained in the folder. Any subfolders and the objects they contain are also included in the upload.

To create a folder:

1. In the Google Cloud console, go to the Cloud Storage Buckets page.
   

   Go to Buckets

2. Navigate to the bucket.

3. Click on Create folder to create an empty new folder, or Upload folder to upload an existing folder.

gsutil

To achieve the illusion of a hierarchical file tree, gsutil applies the following rules to determine whether the destination URL in a command should be treated as an object name or a folder:

1. If the destination URL ends with a / character, gsutil treats the destination URL as a folder. For example, consider the following command where your-file is the name of a file:

   gsutil cp your-file gs://your-bucket/abc/

   As a result, gsutil creates an object named gs://your-bucket/abc/your-file.

2. If you copy multiple source files to a destination URL, gsutil treats the destination URL as a folder. For example, consider the following command where your-dir is a folder containing files such as file1 and file2:

   gsutil cp -r your-dir gs://your-bucket/abc

   As a result, gsutil creates the objects gs://your-bucket/abc/your-dir/file1 and gs://your-bucket/abc/your-dir/file2.

3. If none of the above rules applies, gsutil checks the objects in the bucket to determine if the destination URL is an object name or a folder. For example, consider the following command where your-file is the name of a file:

   gsutil cp your-file gs://your-bucket/abc

   gsutil makes an object listing request for your-bucket, using the / delimiter and prefix=abc, to determine whether there are objects in the bucket whose path starts with gs://your-bucket/abc/. If so, gsutil treats gs://your-bucket/abc as a folder name, and the above command creates the object gs://your-bucket/abc/your-file. Otherwise, gsutil creates the object gs://your-bucket/abc.

gsutil's rule-based approach differs from the way many tools work, which create 0-byte objects to mark the existence of folders. gsutil understands several conventions used by such tools, such as the convention of adding _$folder$ to the end of the name of the 0-byte object, but gsutil does not require such marker objects to implement naming behavior consistent with UNIX commands.

Retries and naming

If you enable automatic retries through gsutil, you may encounter a problem where a first attempt only copies a subset of files, and subsequent attempts encounter an already existing destination folder and name your objects incorrectly.

For example, consider the following command, where there are subfolders under your-dir/ such as dir1 and dir2, and both subfolders contain the file abc:

gsutil cp -r ./your-dir gs://your-bucket/new

If gs://your-bucket/new doesn't exist yet, gsutil creates the following objects on the first successful attempt:

gs://your-bucket/new/dir1/abc
gs://your-bucket/new/dir2/abc

On the next successful attempt, gsutil creates the following objects:

gs://your-bucket/new/your-dir/dir1/abc
gs://your-bucket/new/your-dir/dir2/abc

To make gsutil work consistently on every attempt, try the following:

1. Add a slash to the end of the destination URL so gsutil always treats it as a folder.

2. Use gsutil rsync. Since rsync doesn't use the Unix cp-defined folder naming rules, it works consistently whether the destination subfolder exists or not.

3. If gsutil rsync doesn't work for you, create a placeholder object to establish that the destination is a folder. For example:

   gsutil cp some-file gs://your-bucket/new/placeholder

   With the placeholder object present, running the gsutil cp -r command above consistently treats gs://your-bucket/new as a folder. Once you have at least one of your objects under gs://your-bucket/new, you can delete the placeholder object and continue your uploads.

For more information on how names are constructed, see gsutil help cp.

Additional notes

• You cannot create a zero-byte object to mimic an empty folder using gsutil.

• If you use scripts to build file paths by combining subpaths, note that because / is just a character that happens to be in the name of the object, gsutil interprets gs://my-bucket/folder/ to be a different object from gs://my-bucket//folder.

Pricing considerations

A downside of the gsutil naming approach is it requires an extra object listing before performing a cp or mv command, such as a recursive copy of a folder to the cloud. However, those listings are relatively inexpensive, because they use delimiter and prefix parameters to limit result data. gsutil also makes only one object listing request per cp or mv command, and thus amortizes the cost across all transferred objects.

REST APIs

JSON API

Folders don't exist in the JSON API, but you can narrow down the objects you list using the prefix and delimiter query parameters.

For example, to list all the objects in the bucket my-bucket with the prefix folder/subfolder/, make an object listing request using this URL:

"https://storage.googleapis.com/storage/v1/b/my-bucket/o?prefix=folder/subfolder/"

XML API

Folders don't exist in the XML API, but you can narrow down the objects you list using the prefix and delimiter query parameters.

For example, to list all the objects in the bucket my-bucket with the prefix folder/subfolder/, make an object listing request using this URL:

"https://storage.googleapis.com/my-bucket?prefix=folder/subfolder/"

Note: If you create an empty folder using the Google Cloud console, Cloud Storage creates a zero-byte object as a placeholder. For example, if you create a folder called folder in a bucket called my-bucket, a zero- byte object called gs://my-bucket/folder/ is created. This placeholder is discoverable by other tools when listing the objects in the bucket, for example when using the gsutil ls command.

What's next

• Upload objects to a Cloud Storage bucket.