Top-Level Command-Line Options

Description

gsutil supports separate options for the top-level gsutil command and the individual sub-commands (like cp, rm, etc.) The top-level options control behavior of gsutil that apply across commands. For example, in the command:

gsutil -m cp -p file gs://bucket/obj

the -m option applies to gsutil, while the -p option applies to the cp sub-command.

Options

-D

Shows HTTP requests/headers and additional debug info needed when posting support requests, including exception stack traces.

-DD Same as -D, plus HTTP upstream payload.
-h

Allows you to specify certain HTTP headers, for example:

gsutil -h "Cache-Control:public,max-age=3600" \
       -h "Content-Type:text/html" cp ...

Note that you need to quote the headers/values that contain spaces (such as "Content-Disposition: attachment; filename=filename.ext"), to avoid having the shell split them into separate arguments.

The following headers are stored as object metadata and used in future requests on the object:

Cache-Control
Content-Disposition
Content-Encoding
Content-Language
Content-Type

The following headers are used to check data integrity:

Content-MD5

gsutil also supports custom metadata headers with a matching Cloud Storage Provider prefix, such as:

x-goog-meta-

Note that for gs:// URLs, the Cache Control header is specific to the API being used. The XML API accepts any cache control headers and returns them during object downloads. The JSON API respects only the public, private, no-cache, max-age, and no-transform cache control headers.

See gsutil help setmeta for the ability to set metadata fields on objects after they have been uploaded.

-i

Allows you to use the configured credentials to impersonate a service account, for example:

gsutil -i "service-account@google.com" ls gs://pub

Note that this setting will be ignored by the XML API and S3. See gsutil help creds for more information on impersonating service accounts.

-m

Causes supported operations (acl ch, acl set, cp, mv, rm, rsync, and setmeta) to run in parallel. This can significantly improve performance if you are performing operations on a large number of files over a reasonably fast network connection.

gsutil performs the specified operation using a combination of multi-threading and multi-processing. The number of threads and processors are determined by parallel_thread_count and parallel_process_count, respectively. These values are set in the .boto configuration file or specified in individual requests with the -o top-level flag. Because gsutil has no built-in support for throttling requests, you should experiment with these values. The optimal values can vary based on a number of factors, including network speed, number of CPUs, and available memory.

Using the -m option can consume a significant amount of network bandwidth and cause problems or make your performance worse if you use a slower network. For example, if you start a large rsync operation over a network link that's also used by a number of other important jobs, there could be degraded performance in those jobs. Similarly, the -m option can make your performance worse, especially for cases that perform all operations locally, because it can "thrash" your local disk.

To prevent such issues, reduce the values for parallel_thread_count and parallel_process_count, or stop using the -m option entirely. One tool that you can use to limit how much I/O capacity gsutil consumes and prevent it from monopolizing your local disk is ionice (built in to many Linux systems). For example, the following command reduces the I/O priority of gsutil so it doesn't monopolize your local disk:

ionice -c 2 -n 7 gsutil -m rsync -r ./dir gs://some bucket

If a download or upload operation using parallel transfer fails before the entire transfer is complete (e.g. failing after 300 of 1000 files have been transferred), you must restart the entire transfer.

Also, although most commands normally fail upon encountering an error when the -m flag is disabled, all commands continue to try all operations when -m is enabled with multiple threads or processes, and the number of failed operations (if any) are reported as an exception at the end of the command's execution.

-o Set/override values in the boto configuration value, in the format <section>:<name>=<value>, e.g. gsutil -o "Boto:proxy=host" ... This will not pass the option to gsutil integration tests, which run in a separate process.
-q Causes gsutil to perform operations quietly, i.e., without reporting progress indicators of files being copied or removed, etc. Errors are still reported. This option can be useful for running gsutil from a cron job that logs its output to a file, for which the only information desired in the log is failures.
-u

Allows you to specify a user project to be billed for the request. For example:

gsutil -u "bill-this-project" cp ...