Credential Types Supporting Various Use Cases

Overview

gsutil currently supports several types of credentials/authentication. Each of these type of credentials is discussed in more detail below, along with information about configuring and using credentials via the Cloud SDK.

Configuring/Using Credentials Via Cloud Sdk Distribution Of Gsutil

When gsutil is installed/used via the Cloud SDK ("gcloud"), credentials are stored by Cloud SDK in a non-user-editable file located under ~/.config/gcloud (any manipulation of credentials should be done via the gcloud auth command). If you need to set up multiple credentials (e.g., one for an individual user account and a second for a service account), the gcloud auth command manages the credentials for you, and you switch between credentials using the gcloud auth command as well (for more details, see https://cloud.google.com/sdk/gcloud/reference/auth).

Once credentials have been configured via gcloud auth, those credentials are used regardless of whether the user has any boto configuration files (which are located at ~/.boto unless a different path is specified in the BOTO_CONFIG environment variable). However, gsutil still looks for credentials in the boto config file if a type of non-Cloud Storage credential is needed that's not stored in the gcloud credential store (e.g., an HMAC credential for an S3 account).

Supported Credential Types

gsutil supports several types of credentials (the specific subset depends on which distribution of gsutil you are using; see above discussion).

OAuth2 user account:

This type of credential can be used for authenticating requests on behalf of a specific user (which is probably the most common use of gsutil). This is the default type of credential that is created when you run gcloud init. This credential type is not supported for stand-alone versions of gsutil. For more details about OAuth2 authentication, see: https://developers.google.com/accounts/docs/OAuth2#scenarios

HMAC:

This type of credential can be used by programs that are implemented using HMAC authentication, which is an authentication mechanism supported by certain other cloud storage service providers. This type of credential can also be used for interactive use when moving data to/from service providers that support HMAC credentials. This is the type of credential that is created when you run gsutil config -a.

Note that it's possible to set up HMAC credentials for both Cloud Storage and another service provider; or to set up OAuth2 user account credentials for Cloud Storage and HMAC credentials for another service provider. To do so, after you run the gcloud init command, you can edit the generated ~/.boto config file and look for comments for where other credentials can be added.

For more details about HMAC authentication, see HMAC keys for Cloud Storage.

OAuth2 service account:

This is a credential you can use when authenticating on behalf of a service or application (as opposed to a user). For example, if you intend to run gsutil out of a nightly cron job to upload/download data, using a service account means the cron job does not depend on credentials of an individual employee at your company. This is the type of credential that is configured when you run gcloud auth login with the --cred-file flag (or gsutil config -e when using stand-alone versions of gsutil).

You should generally avoid this credential type because it requires storing highly-privileged credentials in your local environment, creating potential security risks. Instead, service account impersonation or workload identity federation are the recommended alternatives when authenticating on behalf of a service or application.

It is important to note that a service account is considered an Editor by default for the purposes of API access, rather than an Owner. In particular, the fact that Editors have OWNER access in the default object and bucket ACLs, but the predefined ACL options remove OWNER access from Editors, can lead to unexpected results. The solution to this problem is to use "gsutil acl ch" instead of "gsutil acl set <predefined-ACL>" to change permissions on a bucket.

To set up a service account for use with gsutil, see Authorize a service account using a service account key.

For more details about OAuth2 service accounts, see Using OAuth 2.0 for server to server applications.

Compute Engine internal service account:

This is the type of service account used for accounts hosted by App Engine or Compute Engine. Such credentials are created automatically for you on Compute Engine when you run the gcloud compute instances create command and the credentials can be controlled with the --scopes flag.

For more details about using service account credentials for authenticating workloads on Compute Engine, see https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances.

For more details about App Engine service accounts, see https://developers.google.com/appengine/docs/python/appidentity/overview

Service account impersonation:

Impersonating a service account is useful in scenarios where you need to grant short-term access to specific resources. For example, if you have a bucket of sensitive data that is typically read-only and want to temporarily grant write access through a trusted service account.

You can specify which service account to use for impersonation by running gsutil -i, gsutil config -e and editing the boto configuration file, or gcloud config set auth/impersonate_service_account [service_account_email_address].

In order to impersonate, your original credentials need to be granted roles/iam.serviceAccountTokenCreator on the target service account. For more information see https://cloud.google.com/docs/authentication/use-service-account-impersonation.

External account credentials (workload identity federation):

Using workload identity federation, you can access Google Cloud resources from Amazon Web Services (AWS), Microsoft Azure or any identity provider that supports OpenID Connect (OIDC) or SAML 2.0.

For more information see https://cloud.google.com/iam/docs/workload-identity-federation.