Install gsutil

This page describes the installation and setup of gsutil, a tool that enables you to access Cloud Storage from the command-line.

Simply installing gsutil gives you immediate read and/or write access to public data. Authenticating to the Cloud Storage service gives you read and/or write access to protected data that has been shared with you. Enabling billing gives you the ability to create and manage your own buckets.

System requirements

• The gsutil tool runs on Linux/Unix, Mac OS, and Windows (XP or later).

• If you plan to use composite objects, you need to install compiled crcmod. On Windows, this is only available for 32-bit Python.

Warning: On some Linux distributions, another tool named gsutil, by GrandStream BudgeTone, is pre-installed. If you run this command instead of the Cloud Storage gsutil, it will likely print an error message similar to "Choose one of -b, -d, -e, or -r to do something". If this happens, you can either move the Cloud Storage version of gsutil to the front of your PATH environment variable, or you can specify the full path when running Cloud Storage gsutil (for example, /home/users/joan/gsutil/gsutil ls).

Installing gsutil

Note: If you have previously installed the Google Cloud SDK or if you are using a Google Compute Engine instance, then you already have gsutil installed.

The officially supported installation and update method for gsutil is as part of the Google Cloud SDK.

Installing gsutil as part of the Google Cloud SDK

Follow the instructions for your operating system to install gsutil as a part of the Google Cloud SDK:

Linux

1. Enter the following at a command prompt:

   curl https://sdk.cloud.google.com | bash

2. Restart your shell:

   exec -l $SHELL

3. Run gcloud init to initialize the gcloud environment:

   gcloud init

Debian/Ubuntu

Cloud SDK is available in package format for installation on Debian and Ubuntu systems. This package contains the gcloud, gcloud alpha, gcloud beta, gsutil, and bq commands only. It does not include kubectl or the App Engine extensions required to deploy an application using gcloud commands. If you want these components, you must install them separately as described later in this section.

You can install the Cloud SDK from a Debian/Ubuntu package for all official Ubuntu releases that have not reached end of life, and Debian stable releases from Wheezy forward.

Note: If you are using an instance on Google Compute Engine, Cloud SDK is installed by default. You can still manually install Cloud SDK using the instructions below.

1. Create a variable for the correct distribution:

   CLOUD_SDK_REPO="cloud-sdk-$(grep VERSION_CODENAME /etc/os-release | cut -d '=' -f 2)"

2. Add the Cloud SDK distribution URI as a package source:

   echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list

   Note: If you have apt-transport-https installed, you can use "https" instead of "http" in this step.
3. Import the Google Cloud public key:

   curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

   Troubleshooting Tip: If you are unable to get latest updates due to an expired key, obtain the latest apt-get.gpg key file.
4. Update and install the Cloud SDK:

   sudo apt-get update && sudo apt-get install google-cloud-sdk

   Note: For additional apt-get options, such as disabling prompts or dry runs, refer to the apt-get man pages.
5. Optionally, install any of these additional components:
   • google-cloud-sdk-app-engine-python
   • google-cloud-sdk-app-engine-python-extras
   • google-cloud-sdk-app-engine-java
   • google-cloud-sdk-app-engine-go
   • google-cloud-sdk-datalab
   • google-cloud-sdk-datastore-emulator
   • google-cloud-sdk-pubsub-emulator
   • google-cloud-sdk-cbt
   • google-cloud-sdk-cloud-build-local
   • google-cloud-sdk-bigtable-emulator
   • kubectl

   For example, the google-cloud-sdk-app-engine-java component can be installed as follows:

   sudo apt-get install google-cloud-sdk-app-engine-java

6. Run gcloud init to get started:

   gcloud init

macOS

1. Enter the following at a command prompt:

   curl https://sdk.cloud.google.com | bash

2. Restart your shell:

   exec -l $SHELL

3. Run gcloud init to initialize the gcloud environment:

   gcloud init

Windows

1. Download the Cloud SDK installer. The installer is signed by Google Inc.

2. Launch the installer and follow the prompts.

   Cloud SDK requires Python 2 with a release version of Python 2.7.9 or later. The installer will install all necessary dependencies, including the needed Python version, by default. If you already have Python 2.x.y installed and want to use the existing installation, you can uncheck the option to install Bundled Python.

   Note: As of Cloud SDK version 206.0.0, the gcloud CLI has experimental support for running using a Python 3.4+ interpreter (run gcloud topic startup for exclusions and more information on configuring your Python interpreter). All other Cloud SDK tools still require a Python 2.7 interpreter.
3. After installation has completed, accept the following options:
   • Start Cloud SDK Shell
   • Run gcloud init

The installer starts a terminal window and runs the gcloud init command.

4. The default installation does not include the App Engine extensions required to deploy an application using gcloud commands. These components can be installed using the Cloud SDK component manager.

Note: On Google Compute Engine instances running Windows Server, Internet Explorer is set for Enhanced Security Configuration. This setting prevents you from running the installer above. Disable Enhanced Security Configuration first, restart Internet Explorer and then run the installer. For instructions on disabling Enhanced Security on your Windows Server Instance, open Windows Help and Support and search for "IE ESC".

Perform updates with the components update command: gcloud components update.

To learn how to use gsutil, see Quickstart: Using the gsutil Tool exercise, or run gsutil help.

Alternative installation methods

There are several ways to install gsutil as a stand-alone product. You may prefer one of these methods if you do not want any of the other components that come with the Cloud SDK, or if you are managing packages with PyPI.

If you are installing gsutil as a standalone, an additional system requirement is that you must have Python 2.7.x installed on your computer. gsutil does not currently run on Python 3.x. Python is installed by default on most distributions of Linux and macOS. Python is not installed by default on Windows; you must install Python before you can run gsutil on Windows.

Expand the desired method for instructions.

Installing from tar or zip archive

Linux and macOS

1. You can download gsutil as gsutil.tar.gz or as gsutil.zip. It doesn't matter which of these you use; however, these installation instructions assume you are using gsutil.tar.gz.

   Whichever format you download, gsutil is bundled in a single archive. Save the archive in a convenient location.

2. Extract the archive files.

   Open a shell window, change directories to where you downloaded gsutil, and run the following command:

   tar xfz gsutil.tar.gz -C $HOME

   This installs gsutil in your home directory and creates a directory named gsutil. The gsutil directory should contain the gsutil tool, a gslib directory, and a third_party directory that contains several directories, including the boto library.

   If you want to install gsutil in a different location, replace $HOME with another location on your computer.

3. Add gsutil to your PATH environment variable.

   Add the following line to your ~/.bashrc file on Linux or to your ~/.bash_profile on macOS:

   export PATH=${PATH}:$HOME/gsutil

   If you did not install gsutil in your home directory, be sure to change $HOME/gsutil so that it matches the location of gsutil.

4. Restart your shell or terminal program.

   In some cases you might have to restart your shell or terminal program in order for the PATH environment variable to take effect.

You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Windows

1. Download and install Python.

   Install a version of Python that works with gsutil (see System Requirements above). We recommend that you install a 32-bit version of Python and that you install Python in its default location (for example, C:\Python27).

2. Download gsutil.

   gsutil is bundled in a single archive named gsutil.zip. Save the archive in a convenient location, such as C:\.

3. Extract the archive files.

   We recommend that you extract the archive files into your root directory. This should create a directory named gsutil in your root directory (for example, C:\gsutil), which contains the gsutil tool and all the libraries and other files it needs.

4. Start a command prompt window and run gsutil to get a list of gsutil commands.

   To run gsutil commands on Windows, you must run the Python interpreter. For example, to see a listing of gsutil commands, you must type python gsutil at the command prompt. Also, if you run gsutil outside of the C:\gsutil directory, you must provide the full path to the gsutil executable. For example, to run gsutil in the C:\Users\Admin\MyDocuments directory, you must type python C:\gsutil\gsutil at the command prompt.

You're ready to start using gsutil.

Facilitating the use of gsutil

You can make it easier to run gsutil by creating a gsutil.cmd command file that contains the following (assuming you installed Python and gsutil as described above):

cmd.exe /C C:\Python27\python.exe "C:\gsutil\gsutil" %*"

Then, if you add the directory location of gsutil.cmd to the system path, you can run gsutil command from anywhere, where command is a valid gsutil command.

Updating stand-alone gsutil

When used as a stand-alone product, gsutil has a built-in update command that checks for the latest gsutil release and offers to let you automatically update to it if it differs from the version you have currently installed. Use the command gsutil update (or python gsutil update for Windows).

Installing from the Python package index (PyPI)

Experimental!

Installing gsutil via PyPI is an experimental and rapidly changing new feature for Google Cloud Storage. Unfortunately, this means that we may make backwards-incompatible changes. We will inform the community when this feature is no longer experimental.
 

CentOS, RHEL, and Fedora

1. Install required system packages.

   Several packages are required to successfully install gsutil from PyPI. You can install them with the following command:

   sudo yum install gcc openssl-devel python-devel python-setuptools libffi-devel

2. Install pip.

   We recommend using the pip installer. You can install it with the following command:

   sudo yum install python-pip

   If you are using CentOS or RHEL and yum can't find the package python-pip, it's likely because you do not have the Extra Packages for Enterprise Linux (EPEL) repository source enabled. To enable it, follow these instructions.

3. Install gsutil from PyPI.

   To install gsutil from PyPI, use the following command:

   sudo pip install gsutil

You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Debian and Ubuntu

1. Install required system packages.

   Several packages are required to successfully install gsutil from PyPI. You can install them with the following command:

   sudo apt-get install gcc python-dev python-setuptools libffi-dev

2. Install pip.

   We recommend using the pip installer. You can install it with the following command:

   sudo apt-get install python-pip

3. Install gsutil from PyPI.

   To install gsutil from PyPI, use the following command:

   sudo pip install gsutil

You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

macOS

1. Install pip.

   We recommend using the pip installer. You can install it with the following command:

   sudo easy_install -U pip

2. Install gsutil from PyPI.

   To install gsutil from PyPI, use the following command:

   sudo pip install gsutil

You're ready to start using gsutil. To see a listing of gsutil commands, type gsutil at the command prompt.

Windows

Installing gsutil from PyPI is not supported on Windows.

Updating PyPI-installed gsutil

The gsutil update command is not available when installing via PyPI. To update gsutil, use the pip installer:

sudo pip install -U gsutil

Setting Up Credentials to Access Protected Data

Installing gsutil allows you to download and access publicly-accessible objects, but in order to access protected data or write to a protected bucket, you need to set up credentials (authenticate). For example, if someone else has created a Cloud Storage account and uploaded data that is only accessible to you or other specific individuals, you must set up your credentials to the Cloud Storage service to be able to access this data.

If you installed gsutil as part of the Google Cloud SDK, follow these instructions. Otherwise, see the Authenticate stand-alone gsutil section.

Note: To check if you are using gsutil from the Cloud SDK or as a stand-alone, use gsutil version -l, and in the output look for "using cloud sdk".

Authenticate Google Cloud SDK

When using gsutil as part of the Google Cloud SDK, OAuth2 is used to authenticate and authorize access. Note that you may have already performed the following steps if you performed Installing gsutil as part of the Google Cloud SDK above. If you ran gcloud init previously, you are asked if you want to re-initialize the configuration or create a new one.

1. Open a command prompt instance.

2. Run gcloud init at the command prompt.

   You receive the following output:

   Welcome! This command will take you through the configuration of gcloud.
    
   Your current configuration has been set to: [default]
    
   To continue, you must login. Would you like to login (Y/n)?
   

3. Type "Y" and press enter to login.

   • The command prints a URL and tries to open a browser window to request access to your project. If a browser window can be opened, you receive the following output:

     Your browser has been opened to visit:
      
     https://accounts.google.com/o/oauth2/auth?redirect_uri=http%3A%2F%2F...
     

     Go to the next step.

   • If you are working on a local machine and your browser doesn't automatically load the URL, retry the gcloud init command with the --console-only flag:

     gcloud init --console-only
     

     If you use this command, or if the Cloud SDK detects that a browser can not be opened (such as if you are working on a remote machine), you receive the following output:

     Go to the following link in your browser:
      
     https://accounts.google.com/o/oauth2/auth?redirect_uri=urn%3Aietf%3Awg%3A...
      
     Enter verification code:
     

     Open a browser and navigate to the URL provided.

4. If prompted, sign in with the Google account associated with your Cloud Storage data in the browser window.

5. Grant access.

   • If the browser window opened automatically, review the application permissions and click Accept when you are ready. The verification code is then automatically sent to the command line tool.

   • If you are working on a remote machine or used the --console-only flag, copy the verification code from the URL and paste it to the terminal command line, after Enter verification code:.

6. Choose the default project for this configuration.

   After setting up your credentials, the gcloud command-line tool prompts you for a default project for this configuration and provides a list of available projects. Select a project ID from the list.

   When you set this property, gsutil commands that require a project, such as gsutil mb, use the default project ID unless you override them with the -p flag or set the CLOUDSDK_CORE_PROJECT environment variable.

Note: If you have the Compute Engine API enabled, gcloud init may next ask you to set a project default zone and region. Your choice will not affect the performance of gsutil.

That's it. You're ready to access protected data. To see a listing of gsutil commands, type gsutil at the command prompt.

Authenticate stand-alone gsutil

Expand these instructions for establishing your credentials if you did not install gsutil as part of the Google Cloud SDK.

1. Open a new command prompt.
2. (Windows only): Switch to the directory where you installed gsutil (the default is C:\gsutil).
3. Run gsutil config at the command prompt (for Windows: python gsutil config).

   The first time you run this command, gsutil prints a URL for the authorization page where you can authorize gsutil to access Cloud Storage. The gsutil dialog looks similar to the following:

   This script creates a boto config file at <your_homedir>/.boto
   containing your credentials, based on your responses to the following questions.
   
   Please navigate your browser to the following URL: <http://urlto/authorization/dialog>
   In your browser you should see a page that requests you to authorize gsutil to access
   Cloud Storage on your behalf. After you approve, an authorization code is displayed.
   
   Enter the authorization code:
   

   Note: If you want gsutil to launch a browser window for you, use gsutil config -b (for Windows: python gsutil config -b). If you are using SSH to connect to a remote machine, the -b option may not work.

   
   
4. Copy and paste the URL into a browser window.

   If prompted, sign in with the Google account associated with your Cloud Storage data in the browser window.

5. Click the Allow Access button.

   This authorizes gsutil to access Cloud Storage.

6. Copy the authorization code that appears on the next page into the gsutil prompt and press Enter.

   After you provide an authorization code, gsutil prints the following:

   Please navigate your browser to the following URL:
   https://console.cloud.google.com/
   What is your project-id?
   

7. Enter a project ID into the gsutil prompt.

   If you were given a project ID, you can enter it here. Otherwise, copy and paste the URL into a browser window, and log in to the Google Cloud Platform Console to find a project ID. (Find an ID by clicking Manage all projects found in the drop-down in the upper right).

   This project is set as your default for gsutil. You can only set one project ID for gsutil at a time. You can change the project ID for gsutil by editing the boto configuration file.

You're ready to access protected data. To see a listing of gsutil commands, type gsutil at the command prompt.

Authenticate with HMAC

While OAuth 2.0 is the recommended way to authenticate gsutil, you can also use HMAC authentication and developer keys. To do so, use the -a flag in the gsutil config command:

gsutil config -a

Using this command takes you through a similar process as authenticating the stand-alone gsutil (see previous section), except you are prompted for your google access key ID and your google secret access key.

If you are using the Google Cloud SDK, you also should disable credential passing using the command:

gcloud config set pass_credentials_to_gsutil false