Authenticate workloads using service accounts

---

This page describes how to use service accounts to enable apps running on your virtual machine (VM) instances to authenticate to Google Cloud APIs and authorize access to resources.

For more information about how Compute Engine uses service accounts, see the service accounts overview.

Before you begin

• If you want to use the command-line examples in this guide, do the following:
  1. Install or update to the latest version of the Google Cloud CLI.
  2. Set a default region and zone.
• If you want to use the API examples in this guide, set up API access.
• Read the Service accounts overview.

Creating a new service account

You can create and set up a new service account using IAM. After creating an account, grant the account one or more IAM roles, and then authorize a virtual machine instance to run as that service account.

Console

To create a new service account:

1. Create a new service account as described in Creating a service account.

2. Get the service account's email. You need the email to set up an instance to run as this service account. Verify the service account's email in the console:

   1. Go to the Service Accounts page.

      Go to Service Accounts

   2. If prompted, select a project.

   3. Look for your new service account and make note of the service account email.

   Usually, the service account's email is derived from the service account ID, in the format:

   [SERVICE-ACCOUNT-NAME]@[PROJECT_ID].iam.gserviceaccount.com

3. Grant IAM roles to the service account. If you do not grant any roles, the service account will not have access to any services. For a full list of IAM roles, see Understanding Roles on the IAM documentation.

4. Next, set up an instance to run as a service account. Follow the instructions to set up an instance to run as a service account.

Terraform

To create a service account, you can use the google_service_account resource.

compute/service_account_for_instances/main.tf

View on GitHub Feedback

resource "google_service_account" "default" {
  account_id   = "service-account-id"
  display_name = "Service Account"
}

Remember to replace the placeholder values for the account_id and the display_name attributes.

To learn how to apply or remove a Terraform configuration, see Basic Terraform commands.

Setting up a new instance to run as a service account

After creating a new service account, you can create new virtual machine instances to run as the service account. If the service account is in a different project than the instances, you must configure the service account for a resource in a different project.

If you want to assign or change a service account for an existing instance, see Changing the service account and access scopes for an instance instead.

You can enable multiple virtual machine instances to use the same service account, but a virtual machine instance can only have one service account identity. If you assign the same service account to multiple virtual machine instances, any subsequent changes you make to the service account will affect instances using the service account. This includes any changes you make to the IAM roles granted to the service account. For example, if you remove a role, all instances using the service account will lose permissions granted by that role.

Generally, you can just set the cloud-platform access scope to allow access to most of the Cloud APIs, then grant the service account only relevant IAM roles. The combination of access scopes granted to the virtual machine instance and the IAM roles granted to the service account determines the amount of access the service account has for that instance. The service account can execute API methods only if they are allowed by both the access scope and its IAM roles.

Alternatively, you can choose to set specific scopes that permit access to the particular API methods that the service will call. For example, to call the instances.insert method requires authorization with either the https://www.googleapis.com/auth/compute scope or the https://www.googleapis.com/auth/cloud-platform scope as well as an IAM role that grants access to that method. You could set the compute scope in place of the cloud-platform scope, which would give the service access to call methods in Compute Engine but no access to call API methods outside of Compute Engine.

You can set up a new instance to run as a service account through the Google Cloud console, the Google Cloud CLI, or directly through the API.

Console

1. Go to the Create an instance page.

   Go to Create an instance

2. Specify the VM details.

3. In the Identity and API access section, choose the service account you want to use from the drop-down list.

4. Continue with the VM creation process.

gcloud

To create a new instance and authorize it to run as a custom service account using the Google Cloud CLI, provide the service account email and desired access scopes for the instance.

gcloud compute instances create [INSTANCE_NAME] \
    --service-account [SERVICE_ACCOUNT_EMAIL] \
    --scopes [SCOPES,...]

where:

• [SERVICE_ACCOUNT_EMAIL] is the service account email you want to use. For example: [e-mail]. If you don't know what the email is, learn how to obtain a service account email.
• [INSTANCE_NAME] is the name of the instance.
• [SCOPES] is a comma-separated list of full scope URIs, or scope aliases provided in the description for the --scopes flag.

For example:

gcloud compute instances create example-vm \
    --service-account [e-mail] \
    --scopes https://www.googleapis.com/auth/cloud-platform

The gcloud CLI also offers scope aliases in place of the longer scope URIs. For example, the scope for full access to Cloud Storage is https://www.googleapis.com/auth/devstorage.full_control. The alias for this scope is storage-full.

You can see a list of scopes and scope aliases on the instances create page in the description for the --scopes flag. The help for the instances create command also lists these scopes and aliases:

gcloud compute instances create --help

Specify the alias the same way you would specify the normal scope URI. For example:

gcloud compute instances create [INSTANCE_NAME] \
    --service-account [SERVICE_ACCOUNT_EMAIL] \
    --scopes cloud-platform

Note: These aliases are recognized only by the gcloud CLI. The API and other libraries do not recognize these aliases, so you must specify the full scope URI.

Terraform

To set up a new instance to run as a service account, you can use the google_compute_instance resource.

compute/service_account_for_instances/main.tf

View on GitHub Feedback

resource "google_compute_instance" "default" {
  name         = "my-test-vm"
  machine_type = "n1-standard-1"
  zone         = "us-central1-a"

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }

  // Local SSD disk
  scratch_disk {
    interface = "SCSI"
  }

  network_interface {
    network = "default"

    access_config {
      // Ephemeral public IP
    }
  }

  service_account {
    # Google recommends custom service accounts with `cloud-platform` scope with
    # specific permissions granted via IAM Roles.
    # This approach lets you avoid embedding secret keys or user credentials
    # in your instance, image, or app code
    email  = google_service_account.default.email
    scopes = ["cloud-platform"]
  }
}

API

In the API, construct a standard request to create an instance, but include the serviceAccounts property. Obtain your service account email, and include it the email property, along with the desired access scopes for the instance.

POST https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances

{
  "machineType": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/machineTypes/[MACHINE_TYPE]",
  "name": "[INSTANCE_NAME]",
  "serviceAccounts": [
   {
    "email": "[SERVICE_ACCOUNT_EMAIL]",
    "scopes": ["https://www.googleapis.com/auth/cloud-platform"]
   }
  ],
  ...
}

After you have set up an instance to run as the service account, an application running on the instance can use one of the following methods for authentication:

• For most applications, choose one of the following:
  • Use Application Default Credentials and a client library
  • Use gcloud and gsutil commands
• For applications that require an OAuth2 access token, request and use access tokens directly from the metadata server

Authenticating applications using service account credentials

After setting up an instance to run as a service account, you can use service account credentials to authenticate applications running on the instance.

Authenticating applications with a client library

Client libraries can use Application Default Credentials to authenticate with Google APIs and send requests to those APIs. Application Default Credentials lets applications automatically obtain credentials from multiple sources so you can test your application locally and then deploy it to a Compute Engine instance without changing the application code.

For information about setting up Application Default Credentials, see Provide credentials to Application Default Credentials.

This example uses the Python client library to authenticate and make a request to the Cloud Storage API to list the buckets in a project. The example uses the following procedure:

1. Obtain the necessary authentication credentials for the Cloud Storage API and initialize the Cloud Storage service with the build() method and the credentials.
2. List buckets in Cloud Storage.

You can run this sample on an instance that has access to manage buckets in Cloud Storage.

Note: You might need to run this script with sudo.

compute/auth/application_default.py

View on GitHub Feedback

import argparse

import googleapiclient.discovery


def create_service():
    # Construct the service object for interacting with the Cloud Storage API -
    # the 'storage' service, at version 'v1'.
    # Authentication is provided by application default credentials.
    # When running locally, these are available after running
    # `gcloud auth application-default login`. When running on Compute
    # Engine, these are available from the environment.
    return googleapiclient.discovery.build('storage', 'v1')


def list_buckets(service, project_id):
    buckets = service.buckets().list(project=project_id).execute()
    return buckets


def main(project_id):
    service = create_service()
    buckets = list_buckets(service, project_id)
    print(buckets)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('project_id', help='Your Google Cloud Project ID.')

    args = parser.parse_args()

    main(args.project_id)

Authenticating applications directly with access tokens

For most applications, you can authenticate by using Application Default Credentials, which finds credentials and manages tokens for you. However, if your application requires you to provide an OAuth2 access token, Compute Engine lets you get an access token from its metadata server for use in your application.

There are several options for obtaining and using these access tokens to authenticate your applications. For example, you can use curl to create a simple request, or use a programming language like Python for more flexibility.

cURL

To use curl to request an access token and send a request to an API:

1. On the instance where your application runs, query the metadata server for an access token by running the following command:

   $ curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
   -H "Metadata-Flavor: Google"

   The request returns a response similar to:

   {
         "access_token":"ya29.AHES6ZRN3-HlhAPya30GnW_bHSb_QtAS08i85nHq39HE3C2LTrCARA",
         "expires_in":3599,
         "token_type":"Bearer"
    }

   For API requests you need to include the access_token value, not the entire response. If you have the jq command-line JSON processor installed you can use the following command to extract the access token value from the response:

   $ ACCESS_TOKEN=`curl \
   "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
   -H "Metadata-Flavor: Google" | jq -r '.access_token'`
   

2. Copy the value of the access_token property from the response and use it to send requests to the API. For example, the following request prints a list of instances in your project from a certain zone:

   $ curl https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances \
   -H "Authorization":"Bearer [ACCESS_TOKEN]"
   

   where:

   • [PROJECT_ID] is the project ID for this request.
   • [ZONE] is the zone to list instances from.
   • [ACCESS_TOKEN] is the access token value you got from step 1.
   Note: You can use the access token only for scopes that you specified when you created the instance. For example, if the instance has been granted only the https://www.googleapis.com/auth/storage-full scope for Cloud Storage, then it can't use the access token to make a request to BigQuery.

   For information about the parameters that you can set in your request, see the parameters documentation.

Python

This example demonstrates how to request a token to access the Cloud Storage API in a Python application. The example uses the following procedure:

1. Request an access token from the metadata server.
2. Extract the access token from the server response.
3. Use the access token to make a request to Cloud Storage.
4. If the request is successful, the script prints the response.

compute/auth/access_token.py

View on GitHub Feedback

import argparse

import requests


METADATA_URL = 'http://metadata.google.internal/computeMetadata/v1/'
METADATA_HEADERS = {'Metadata-Flavor': 'Google'}
SERVICE_ACCOUNT = 'default'


def get_access_token():
    url = '{}instance/service-accounts/{}/token'.format(
        METADATA_URL, SERVICE_ACCOUNT)

    # Request an access token from the metadata server.
    r = requests.get(url, headers=METADATA_HEADERS)
    r.raise_for_status()

    # Extract the access token from the response.
    access_token = r.json()['access_token']

    return access_token


def list_buckets(project_id, access_token):
    url = 'https://www.googleapis.com/storage/v1/b'
    params = {
        'project': project_id
    }
    headers = {
        'Authorization': 'Bearer {}'.format(access_token)
    }

    r = requests.get(url, params=params, headers=headers)
    r.raise_for_status()

    return r.json()


def main(project_id):
    access_token = get_access_token()
    buckets = list_buckets(project_id, access_token)
    print(buckets)


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('project_id', help='Your Google Cloud project ID.')

    args = parser.parse_args()

    main(args.project_id)

Access tokens expire after a short period of time. The metadata server caches access tokens until they have 5 minutes of remaining time before they expire. You can request new tokens as frequently as you like, but your applications must have a valid access token for their API calls to succeed.

Authenticating tools on an instance using a service account

Some applications might use commands from the gcloud and gsutil tools, which are included by default in most Compute Engine images. These tools automatically recognize an instance's service account and relevant permissions granted to the service account. Specifically, if you grant the correct roles to the service account, you can use the gcloud and gsutil tools from your instances without having to use gcloud auth login.

This service account recognition happens automatically and applies only to the gcloud and gsutil tools that are included with the instance. If you create new tools or add custom tools, you must authorize your application using a client library or by using access tokens directly in your application.

To take advantage of automatic service account recognition, grant the appropriate IAM roles to the service account and set up an instance to run as a service account. For example, if you grant a service account the roles/storage.objectAdmin role, the gsutil tool can automatically manage and access Cloud Storage objects.

Likewise, if you enable roles/compute.instanceAdmin.v1 for the service account, the gcloud compute tool can automatically manage instances.

Changing the service account and access scopes for an instance

If you want to run the VM as a different identity, or you determine that the instance needs a different set of scopes to call the required APIs, you can change the service account and the access scopes of an existing instance. For example, you can change access scopes to grant access to a new API, you can remove the service account and access scopes to prevent a VM from accessing any Google Cloud services, or you can change a VM so that it runs as a service account that you created instead of the Compute Engine default service account. However, Google recommends that you use fine-grained IAM policies instead of relying on access scopes to control resource access for the service account.

To change an instance's service account and access scopes, the instance must be temporarily stopped. To stop your instance, read the documentation for Stopping an instance. After changing the service account or access scopes, remember to restart the instance. Use one of the following methods to the change service account or access scopes of the stopped instance.

Console

Note: If your instance uses a service account other than the Compute Engine default service account, use the gcloud CLI or the API to change access scopes.

1. Go to the VM instances page.

   Go to VM instances

2. Click the VM instance name for which you want to change the service account.

3. If the instance is not stopped, click Stop. Wait for the instance to be stopped.

4. Next, click Edit.

5. Scroll down to the Service Account section.

6. From the drop-down list, select the service account to assign to the instance.

   • If you choose the default service account, you can modify its access scopes in the Google Cloud console.
     • To change scopes, in the Access scopes section, select Set access for each API and set the appropriate scopes for your needs.
     • If you're not sure of the proper access scopes to set, choose Allow full access to all Cloud APIs and then make sure to restrict access by setting IAM roles on the service account.
   • If you choose a different service account, the VM's access scope defaults to the cloud-platform scope. You can modify the scope by using the gcloud CLI or Compute Engine API.
   • For more information about setting access scopes, see Best practices.

7. Click Save to save your changes.

gcloud

Use the instances set-service-account command and provide the instance name, the service account email, and the desired scopes. For more information about setting access scopes, see Best practices.

gcloud compute instances set-service-account [INSTANCE_NAME] \
   [--service-account [SERVICE_ACCOUNT_EMAIL] | --no-service-account] \
   [--no-scopes | --scopes [SCOPES,...]]

where:

• [SERVICE_ACCOUNT_EMAIL] is the service account email you want to use. For example: [e-mail].
• [INSTANCE_NAME] is the name of the instance.
• [SCOPES] is a comma-separated list of full scope URIs, or scope aliases provided in the description for the --scopes flag. If you want to remove all scopes for the instance, use the --no-scopes flag instead.

For example, the following command assigns the service account [e-mail] to an instance called example-instance and sets access scopes on that instance to allow read/write access to Compute Engine and read-only access to Cloud Storage:

gcloud compute instances set-service-account example-instance \
   --service-account [e-mail] \
   --scopes compute-rw,storage-ro

API

In the API, make a POST request to the setServiceAccount method:

https://compute.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances/[INSTANCE_NAME]/setServiceAccount

where:

• [PROJECT_ID] is the project ID for this request.
• [ZONE] is the zone where this instance belongs to.
• [INSTANCE_NAME] is the name of the instance.

In the request body, provide the email address of the service account and the desired scope URIs for the instance. For more information about setting access scopes, see Best practices.

{
  "email": "[SERVICE_ACCOUNT_EMAIL]",
  "scopes": [
    "[SCOPE_URI]",
    "[SCOPE_URI]",
    ...
  ]
}

For example, the following request uses the service account email [e-mail] and sets a Cloud Storage and BigQuery scope:

{
  "email": "[e-mail]",
  "scopes": [
    "https://www.googleapis.com/auth/bigquery",
    "https://www.googleapis.com/auth/devstorage.read_only"
  ]
}

Obtaining a service account email

To identify a service account, you need the service account email. Obtain a service account email through one of the following options:

Console

1. Go to the Service Accounts page.

   Go to Service Accounts

2. If prompted, select a project. The service accounts page lists all service accounts for the project and their emails.

gcloud

Use the gcloud compute instances describe command from your local machine:

gcloud compute instances describe [INSTANCE_NAME] --format json

{
      ...
      "serviceAccounts":[
         {
            "email":"[e-mail]",
            "scopes":[
               "https://www.googleapis.com/auth/devstorage.full_control"
            ]
         }
      ]
      ...
   }

If the instance isn't using a service account, you receive a response without the serviceAccounts property.

Metadata Server

Query the metadata server from within the instance itself. Make a request to http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/:

user@myinst:~$ curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/" \
-H "Metadata-Flavor: Google"

If you enabled one or more service accounts when you created the instance, this curl command returns output similar to the following:

[e-mail]/
default/

If the instance isn't using a service account, you receive an empty response.

API

Make a request to the Service Accounts API.

Using the Compute Engine Default Service Account

If you are familiar with the Compute Engine default service account and want to use the credentials provided by the default service account instead of creating new service accounts, you can grant IAM roles to the default service account.

By default, all Compute Engine instances can run as the default service account. When you create an instance using the Google Cloud CLI or the Google Cloud console, and omit any service account specifications, the default service account is assigned to the instance.

Before you assign IAM roles to the default service account, note that:

• Granting an IAM role to the default service account affects all instances that are running as the default service account. For example, if you grant the default service account the roles/storage.objectAdmin role, all instances running as the default service account with the required access scopes will have permissions granted by the roles/storage.objectAdmin role. Likewise, if you limit access by omitting certain roles, it will affect all instances running as the default service account.

• You must revoke project editor permission for the service account. The default service account is added as a project editor to projects by default. To use IAM roles, you must revoke the project editor permission.

If you are unsure about granting IAM roles to the default service account, create a new service account instead.

Caution: If you have existing instances that are currently using the default service account and are relying on editor access, this process requires that you revoke editor access to the account. Instances relying on editor permissions won't be able to use those permissions to access other Google APIs. Most Google Cloud Platform products provide IAM roles, so you should be able to grant roles to replicate the access provided by project editor but in some cases, certain APIs do not yet support IAM roles. In these cases, you will need to rely on the project editor role for the time being.

Follow these instructions to grant an IAM role to the default service account:

1. In the Google Cloud console, go to the IAM page.

   Go to IAM

2. If prompted, select a project.

3. Look for the service account named Compute Engine Default Service Account.

4. In the Role(s) column, expand the drop down menu for the Compute Engine Default Service Account.

5. Remove Editor access and save your changes.

6. Next, grant IAM roles to the service account.

Any virtual machine instances that are currently running as the default service account will now have access to other Google Cloud APIs according to the IAM roles you granted to the account.

If you want to set up a new instance to run as the default service account, follow these instructions:

Console

1. Go to the Create an instance page.

   Go to Create an instance

2. Specify the VM details.

3. In the Identity and API access section, choose Compute Engine default service account from the Service account drop-down list.

4. Continue with the VM creation process.

gcloud

To create a new instance and authorize it to have full access to all Google Cloud services using the default service account:

gcloud compute instances create [INSTANCE_NAME] \
     --scopes cloud-platform

API

In the API, construct a standard request to create an instance, but include the serviceAccounts property. Obtain your default service account ID, and include it as the service account's email. Then, set one or more scopes in the scopes property.

POST https://compute.googleapis.com/compute/v1/projects/zones/[ZONE]/instances

{
  "machineType": "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/machineTypes/[MACHINE_TYPE]",
  "name": "[INSTANCE_NAME]",
  "serviceAccounts": [
   {
    "email": "[DEFAULT_SERVICE_ACCOUNT_EMAIL]",
    "scopes": ["https://www.googleapis.com/auth/cloud-platform"]
   }
  ],
  ...
}

Best practices

• Google recommends that each VM instance that needs to call a Google API should run as a service account with the minimum permissions necessary for that VM to do its job.

• Follow these instructions to configure service accounts for your VMs:

  1. Create a new service account rather than using the Compute Engine default service account.
  2. Grant IAM roles to that service account for only the resources that it needs.
  3. Configure the VM to run as the new service account you created.
  4. Grant your VM the https://www.googleapis.com/auth/cloud-platform scope to allow access to most of the Google Cloud APIs, so that the IAM permissions of the VM are completely determined by the IAM roles that you granted to the VM's service account. The service account can only execute API methods that are allowed by both the access scope and the service account's specific IAM roles.

• Limit the privileges of service accounts and regularly check your service account permissions to make sure they are up-to-date.

• Delete service accounts with caution. Make sure your critical applications are no longer using a service account before deleting it. If you're not sure whether a service account is being used, we recommend disabling the service account instead of deleting it. Disabled service accounts can be re-enabled if they are still needed.

• Mitigate the security risks for your service account. For more information, see Best practices for working with service accounts.

What's next

• Learn more about Service Accounts.
• Learn more about Compute Engine IAM Roles.
• Learn more about best practices for working with service accounts.

Try it for yourself

If you're new to Google Cloud, create an account to evaluate how Compute Engine performs in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

Try Compute Engine free