Whether your business is early in its journey or well on its way to digital transformation, Google Cloud's solutions and technologies help chart a path to success.
This page describes the installation and setup of gsutil, a tool that enables
you to access Cloud Storage from the command-line.
Simply installing gsutil gives you immediate read and/or write access to
public data. Authenticating to the Cloud Storage service gives you read
and/or write access to protected data that has been shared with you. Enabling
billing gives you the ability to create and manage your own buckets.
System requirements
The gsutil tool runs on Linux/Unix, Mac OS, and Windows (XP or later).
If you plan to use composite objects, you need to install
compiled crcmod. On Windows, this is only available for 32-bit Python.
Installing gsutil
The officially supported installation and update method for gsutil is as part
of the Google Cloud SDK.
Installing gsutil as part of the Google Cloud SDK
Follow the instructions for your operating system to install gsutil as a part of
the Google Cloud SDK:
Linux
Enter the following at a command prompt:
curl https://sdk.cloud.google.com | bash
Restart your shell:
exec -l $SHELL
Run gcloud init to
initialize the gcloud environment:
gcloud init
Debian/Ubuntu
Cloud SDK is available in package format for installation on Debian and Ubuntu systems. This
package contains the gcloud, gcloud alpha, gcloud beta,
gsutil, and bq commands only. It does not include kubectl
or the App Engine extensions required to deploy an application using gcloud commands.
If you want these components, you must install them separately as described
later in this section.
You can install the Cloud SDK from a Debian/Ubuntu package for all official
Ubuntu releases that have not
reached end of life, and Debian stable
releases from Wheezy forward.
Note: If you are using an instance on Google Compute Engine, Cloud SDK is
installed by default. You can still manually install Cloud SDK using the instructions below.
Add the Cloud SDK distribution URI as a package source:
echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
Launch the installer and follow the prompts. The installer is signed by Google LLC.
Cloud SDK requires Python. Supported versions are 3.5 to 3.7, and 2.7.9 or higher.
The installer will install all necessary dependencies, including the needed Python version.
While Cloud SDK currently uses Python 2 by default, you can use an existing Python installation
if necessary by unchecking the option to 'Install Bundled Python'.
.
After installation has completed, accept the following options:
Start Cloud SDK Shell
Run gcloud init
The installer starts a terminal window and runs the
gcloud init command.
The default installation does not include the App Engine extensions required to deploy an
application using gcloud commands. These components can be installed using the
Cloud SDK component manager.
Perform updates with the components update command: gcloud components update.
There are several ways to install gsutil as a stand-alone product. You may
prefer one of these methods if you do not want any of the other components
that come with the Cloud SDK, or if you are managing packages with PyPI.
If you are installing gsutil as a standalone, an additional system requirement
is that you must have Python installed on your computer. gsutil officially
supports minor versions 2.7.x through 3.7.x. Python is installed by default on
most distributions of Linux and macOS, but not on Windows; you must install
Python before you can run gsutil on Windows.
Expand the desired method for instructions.
Installing from tar or zip archive
Linux and macOS
You can download gsutil as
gsutil.tar.gz
or as gsutil.zip.
It doesn't matter which of these you use; however, these installation instructions
assume you are using gsutil.tar.gz.
Whichever format you download, gsutil is bundled in a single archive.
Save the archive in a convenient location.
Extract the archive files.
Open a shell window, change directories to where you downloaded gsutil,
and run the following command:
tar xfz gsutil.tar.gz -C $HOME
This installs gsutil in your home directory and creates a directory
named gsutil. The gsutil directory should
contain the gsutil tool, a gslib directory, and a third_party directory that
contains several directories, including the
boto library.
If you want to install gsutil in a different location,
replace $HOME with another location on your computer.
Add gsutil to your PATH environment variable.
Add the following line to your ~/.bashrc file on Linux or to your
~/.bash_profile on macOS:
export PATH=${PATH}:$HOME/gsutil
If you did not install gsutil in your home directory, be sure to
change $HOME/gsutil so that it matches the location of
gsutil.
Restart your shell or terminal program.
In some cases you might have to restart your shell or terminal
program in order for the PATH environment variable to take effect.
You're ready to start using gsutil. To see a listing of gsutil commands,
type gsutil at the command prompt.
Install a version of Python that works with gsutil
(see System Requirements above). We
recommend that you install a 32-bit version of Python and
that you install Python in its default location (for example,
C:\Python27).
gsutil is bundled in a single archive named gsutil.zip. Save the
archive in a convenient location, such as C:\.
Extract the archive files.
We recommend that you extract the archive files into your root
directory. This should create a directory named gsutil
in your root directory (for example, C:\gsutil), which
contains the gsutil tool and all the libraries and other files it
needs.
Start a command prompt window and run gsutil to get a list of gsutil
commands.
To run gsutil commands on Windows, you must run the Python
interpreter. For example, to see a listing of gsutil commands, you
must type python gsutil at the command prompt. Also, if
you run gsutil outside of the C:\gsutil directory, you
must provide the full path to the gsutil executable. For example, to
run gsutil in the C:\Users\Admin\MyDocuments directory,
you must type python C:\gsutil\gsutil at the command
prompt.
You're ready to start using gsutil.
Facilitating the use of gsutil
You can make it easier to run gsutil by creating a gsutil.cmd command file that
contains the following (assuming you installed Python and gsutil as described above):
Then, if you add the directory location of gsutil.cmd to the system path,
you can run gsutil command from anywhere, where
command is a valid gsutil command.
Updating stand-alone gsutil
When used as a stand-alone product, gsutil has a built-in update command
that checks for the latest gsutil release and offers to let you automatically
update to it if it differs from the version you have currently installed.
Use the command gsutil update (or python gsutil
update for Windows).
Installing from the Python package index (PyPI)
Experimental!
Installing gsutil via PyPI is an experimental and rapidly changing new feature
for Cloud Storage. Unfortunately, this means that we may make
backwards-incompatible changes. We will inform the community when this feature
is no longer experimental.
CentOS, RHEL, and Fedora
Install required system packages.
Several packages are required to successfully install gsutil from
PyPI. You can install them with the following command:
We recommend using the pip installer. You can install it with the
following command:
sudo yum install python-pip
If you are using CentOS or RHEL and yum can't find the package
python-pip, it's likely because you do not have the Extra
Packages for Enterprise Linux (EPEL) repository source enabled. To
enable it,
follow
these instructions.
Install gsutil from PyPI.
To install gsutil from PyPI, use the following command:
sudo pip install gsutil
You're ready to start using gsutil. To see a listing of gsutil commands,
type gsutil at the command prompt.
Debian and Ubuntu
Install required system packages.
Several packages are required to successfully install gsutil from
PyPI. You can install them with the following command:
We recommend using the pip installer. You can install it with the
following command:
sudo apt-get install python-pip
Install gsutil from PyPI.
To install gsutil from PyPI, use the following command:
sudo pip install gsutil
You're ready to start using gsutil. To see a listing of gsutil commands,
type gsutil at the command prompt.
macOS
Install pip.
We recommend using the pip installer. You can install it with the
following command:
sudo easy_install -U pip
Install gsutil from PyPI.
To install gsutil from PyPI, use the following command:
sudo pip install gsutil
You're ready to start using gsutil. To see a listing of gsutil commands,
type gsutil at the command prompt.
Windows
Install pip.
If you use Python version 2.7.9 or above, run the command:
python -m ensurepip
Otherwise, download the file get-pip.py,
which allows you to install pip, and then install pip with the
following command:
python get-pip.py
Install gsutil from PyPI.
To install gsutil from PyPI, run the following command:
pip install gsutil
You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.
Updating PyPI-installed gsutil
The gsutil update command is not available when installing via PyPI. To
update gsutil, use the pip installer:
sudo pip install -U gsutil
Setting Up Credentials to Access Protected Data
Installing gsutil allows you to download and access publicly-accessible objects,
but in order
to access protected data or write to a protected bucket, you need to set up
credentials (authenticate). For example, if someone else has created a
Cloud Storage account and uploaded data that is only accessible to you or other specific
individuals, you must set up your credentials to the Cloud Storage service to be able
to access this data.
When using gsutil as part of the Google Cloud SDK, OAuth2 is used to
authenticate and authorize access. Note that you may have already performed
the following steps if you performed
Installing gsutil as part of the Google Cloud SDK above. If you
ran gcloud init previously, you are asked if you want to re-initialize the
configuration or create a new one.
Open a command prompt instance.
Run gcloud init at the command prompt.
You receive the following output:
Welcome! This command will take you through the configuration of gcloud.
Your current configuration has been set to: [default]
To continue, you must login. Would you like to login (Y/n)?
Type "Y" and press enter to login.
The command prints a URL and tries to open a browser window to request access to your project.
If a browser window can be opened, you receive the following output:
Your browser has been opened to visit:
https://accounts.google.com/o/oauth2/auth?redirect_uri=http%3A%2F%2F...
Go to the next step.
If you are working on a local machine and your browser doesn't
automatically load the URL, retry the gcloud init command with the --console-only flag:
gcloud init --console-only
If you use this command, or if the Cloud SDK detects that a browser can
not be opened (such as if you are working on a remote machine), you
receive the following output:
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?redirect_uri=urn%3Aietf%3Awg%3A...
Enter verification code:
Open a browser and navigate to the URL provided.
If prompted, sign in with the Google account associated with your
Cloud Storage data in the browser window.
Grant access.
If the browser window opened automatically, review the application
permissions and click Accept when you are ready. The verification
code is then automatically sent to the command line tool.
If you are working on a remote machine or used the --console-only flag,
copy the verification code from the URL and paste it to the terminal
command line, after Enter verification code:.
Choose the default project for this configuration.
After setting up your credentials, the gcloud command-line tool prompts
you for a default project for this configuration and provides a list of
available projects. Select a project ID from the list.
When you set this property, gsutil commands that require a project, such
as gsutil mb, use the default project ID unless you override them with
the -p flag or set the CLOUDSDK_CORE_PROJECT environment variable.
That's it. You're ready to access protected data. To see a listing of gsutil
commands, type gsutil at the command prompt.
Authenticate stand-alone gsutil
Expand these instructions for establishing your
credentials if you did not install gsutil as part of the Google Cloud SDK.
Open a new command prompt.
(Windows only): Switch to the directory where you installed gsutil (the
default is C:\gsutil).
Run gsutil config at the command prompt (for Windows:
python gsutil config).
The first time you run this command, gsutil prints a URL for the
authorization page where you can authorize gsutil to access Cloud Storage.
The gsutil dialog looks similar to the following:
This script creates a boto config file at <your_homedir>/.boto
containing your credentials, based on your responses to the following questions.
Please navigate your browser to the following URL: <http://urlto/authorization/dialog>
In your browser you should see a page that requests you to authorize gsutil to access
Cloud Storage on your behalf. After you approve, an authorization code is displayed.
Enter the authorization code:
Note: If you want gsutil to launch a browser
window for you, use gsutil config -b (for Windows: python
gsutil config -b). If you are using SSH to connect to a remote machine,
the -b option may not work.
Copy and paste the URL into a browser window.
If prompted, sign in with the Google account associated with your
Cloud Storage data in the browser window.
Click the Allow Access button.
This authorizes gsutil to access Cloud Storage.
Copy the authorization code that appears on the next page into the gsutil
prompt and press Enter.
After you provide an authorization code, gsutil prints the following:
Please navigate your browser to the following URL:
https://console.cloud.google.com/
What is your project-id?
Enter a project ID into the gsutil prompt.
If you were given a project ID, you can enter it here. Otherwise, copy
and paste the URL into a browser window, and log in to the
Google Cloud Console to find a project ID. (Find an ID by clicking
Manage all projects found in the drop-down in the upper right).
This project is set as your default for gsutil. You can only set
one project ID for gsutil at a time. You can change the project ID for
gsutil by editing the boto configuration file.
You're ready to access protected data. To see a listing of gsutil commands,
type gsutil at the command prompt.
Authenticate with HMAC
While OAuth 2.0 is the recommended way to authenticate gsutil, you can also
use HMAC keys for your credentials. To authenticate with HMAC,
use the -a flag in the gsutil config command:
gsutil config -a
Using this command takes you through a similar process as authenticating the
stand-alone gsutil (see previous section), except you are
prompted for the access ID and secret associated with your HMAC key.
If you are using the Google Cloud SDK, you also should disable credential
passing using the command:
gcloud config set pass_credentials_to_gsutil false
