Best practices for managing service account keys

Unlike normal users, service accounts do not have passwords. Instead, service accounts use RSA key pairs for authentication: If you know the private key of a service account's key pair, you can use the private key to create a JWT bearer token and use the bearer token to request an access token. The resulting access token reflects the service account's identity and you can use it to interact with Google Cloud APIs on the service account's behalf.

Because the private key lets you authenticate as the service account, having access to the private key is similar to knowing a user's password.

The key pairs used by service accounts fall into two categories, Google-managed and user-managed. Google-managed key pairs and user-managed key pairs differ in where their private key is stored, how they can be used for authentication, and how they must be secured:

  • Google-managed key pairs are automatically generated and fully managed by Identity and Access Management. IAM maintains Google-managed key pairs for all service accounts, and rotates them periodically. You can download the public key of a Google-managed key pair, but you can't access its private key.

    To use the private key of a Google-managed key pair, you must request IAM to perform operations for you. For example, you can invoke the signBlob or signJwt operations to let IAM use a service account's Google-managed private key to sign a piece of data for you. Similarly, by attaching a service account to a compute resource, you can authorize IAM to use the Google-managed key whenever the compute resource requests service account credentials.

    Requesting IAM to use a service account's Google-managed private key for you requires special permissions. For example, to invoke signBlob, you must have the iam.serviceAccounts.signBlob permission on the respective service account. Similarly, to attach a service account to a compute resource, you must have the iam.serviceAccounts.actAs permission. To secure the service account, and protect against privilege-escalation threats, you must ensure that only authorized users are granted roles that include these permissions.

  • User-managed key pairs are additional key pairs that you can associate with a service account, either by uploading a public key or by letting IAM generate a key pair for you. In both cases, IAM only stores the public key while you own the private key.

    The private key, along with some metadata is called a service account key and typically stored in JSON format.

    Because you own the private key, you are responsible for keeping it confidential, storing it securely, and rotating it periodically. Anybody who can access the private key can use it to create a JWT bearer token and authenticate as that service account.

Service account keys can become a security risk if not managed carefully. The main threats are related to service account keys are:

  • Credential leakage: Service account keys might inadvertently end up in places where they are not supposed to be stored. A bad actor can use a leaked service account key to authenticate and gain a foothold in your environment.
  • Privilege escalation: If a bad actor gets access to a poorly secured service account key, they might be able to use the key to escalate their privileges.
  • Information disclosure: Service account keys might inadvertently disclose confidential metadata.
  • Non-repudiation: By authenticating using a service account key and letting the service account carry out operations on their behalf, a bad actor might conceal their identity and actions.

The best way to mitigate these threats is to avoid user-managed service account keys and to use other methods to authenticate service accounts whenever possible. You can also use IAM conditions and VPC Service Controls to restrict what resources can potentially be accessed by a compromised service account.

For situations where using alternatives authentication methods aren't viable, this guide presents best practices for managing, using, and securing service account keys.

Protecting against credential leakage

Like a username and password, service account keys are a form of credential. If a user can access a valid service account key, they can use it to authenticate and access the resources the respective service account has been granted access to.

To bad actors, service account keys can be even more valuable than a leaked password: Attempting to sign in by using a leaked password is unlikely to succeed if the user account has been configured to use 2-step verification and login challenges. In contrast, authenticating by using a leaked service account key is likely to succeed as service accounts are not subject to any additional sign-in verifications.

Bad actors might look for service account keys in a variety of places, including:

  • Source code repositories of open-source projects
  • Public Cloud Storage buckets
  • Public data dumps of breached services

In addition to public locations, bad actors might look for service account keys in private locations they've compromised. Examples include:

  • Email inboxes
  • File shares
  • Backup storage
  • Temporary file system directories

An effective way to lower the risk of leaking service account keys is to reduce the number of keys in circulation and to disincentivize the creation of new keys. The following sections describe how you can limit the number of service account keys in circulation, and what other measures can help you limit the risk of leaking service accounts.

Provide alternatives to creating service account keys

Trying to reduce the number of service account keys in circulation can be challenging if users in your organization find it easier to create new service account keys than to use more secure alternatives.

To prevent that service account keys are created out of convenience, make sure that users are aware of alternatives to using service account keys, and are able to use these alternative methods if they need to:

After you've made sure that users can use alternative methods to authenticate service accounts, use organization policy constraints to limit service account key usage to selected projects.

Use organization policy constraints to limit which projects can create service account keys

There are several ways to avoid the need to create service account keys, including attaching a service account to a resource, using Workload Identity, or using Workload Identity Federation. Given these alternatives, it's best to consider the use of service account keys as an exception rather than the norm.

To prevent unnecessary usage of service account keys, use organization policy constraints:

Modifying organization policy constraints requires the orgpolicy.policy.set permission. Because neither the Owner (roles/owner) nor the Editor (roles/editor) role includes this permission, constraints can also be effective in non-production environments where some principals might have Owner or Editor access to projects.

Don't leave service account keys in temporary locations

When you create a service account key by using the Google Cloud Console, most browsers immediately download the new key and save it in a download folder on your computer. You should immediately move the key to the location where you want to store it. Make sure you're not accidentally leaving a copy in the download folder or the recycle bin of your computer.

You can reduce the risk of accidentally leaving copies of service account keys in temporary locations by using the gcloud command-line tool: The gcloud iam service-accounts keys create command lets you write the service account key file straight to the location where you need it. Also, on most operating systems, the gcloud tool automatically adjusts file permissions so that the file is only accessible by you.

Don't pass service account keys between users

When you deploy an application that requires a service account key, you might not have the permission to create a service account key yourself. Instead, you might have to request a different person to create a service account key for you.

In scenarios where multiple users are involved in the creation and deployment of a service account key, there is an increased risk that copies of the key remain in mailboxes, chat histories, or other locations. Whenever a handover between users is necessary, it can be more secure to upload a service account key:

  1. As the user deploying the application, create a self-signed certificate that uses an RSA 2048-bit key pair on the target machine. To create the certificate, you can use openssl, certutil, New-SelfSignedCertificate, or other operating system tools.
  2. Pass the certificate file to the user who has the permission to upload the certificate while keeping the private key on the target machine. When passing the certificate, make sure that it can't be replaced or tampered with, but you don't need to keep it confidential.
  3. As the user who has the necessary permissions to manage service account keys, upload the certificate to associate it with a service account.

By following this process, you avoid passing the private key and instead only exchange public information between users.

Don't submit service account keys to source code repositories

Service account keys are credentials, and must be protected from unauthorized access. If you submit a service account key to a source code repository, there is an increased risk that the key becomes accessible to unauthorized users and bad actors:

  • Bad actors might scan the source code of public source repositories for leaked keys.
  • In the future, you might decide to turn a private source repository into a public repository, without checking it for keys first.
  • Other team members might store copies of the source code on their workstation.

When you work on code that uses a service account key, always store the service account key separate from the source code to reduce the risk of accidentally submitting the key to the source repository. In many cases, you can further reduce this risk by not using service account keys at all during development and using your personal credentials instead of service account keys instead.

Additionally, set up your source control system so that it detects accidental submissions of service account keys:

Don't embed service account keys in program binaries

Service account keys are strings that match a certain pattern, and they can be identified even if embedded in other files or binaries. If a bad actor has access to the binary, they can extract any service account keys that are embedded in the binary.

Program binaries for server-side applications might be hosted in artifact repositories or they might be copied to developer workstations for debugging purposes. Keeping service account keys separate from the program binaries helps ensure that a user who can access the binary does not implicitly get access to service account credentials.

  • For client-side applications such as tools, desktop programs, or mobile apps, don't use service accounts. Instead, let users authenticate with their own credentials by using the OAuth-consent flow.
  • For server-side applications, don't embed service account keys into the binary. Instead, keep the keys separate from the application binary.

Use insights and metrics to identify unused service account keys

To minimize the number of valid service account keys in circulation, it's best to disable keys as soon as they aren't needed anymore, then delete the keys when you're certain that they are no longer needed. If you're unsure whether a key is still in use or not, you can use service account insights and authentication metrics:

Because service accounts belong to a Google Cloud project, insights and metrics must be tracked individually for each project.

Rotate service account keys to reduce potential damage caused by leaked keys

Although you can reduce the probability of accidentally leaking a service account key, it can be difficult to eliminate the risk completely.

If a key is leaked, bad actors might not find it immediately – instead, it might take several days or weeks before somebody finds the key. You can take advantage of this delay by periodically invalidating your service account keys and replacing them with new keys: The more often you rotate service account keys, the less likely it is that a leaked service account key is still valid when a bad actor finds it.

To rotate keys, you can follow a push- or pull-based model:

  • In a push-based model, you use a centralized tool or system that periodically creates new service account keys and pushes them to the individual applications to replace existing keys. Afterwards, the tool deletes obsolete service account keys.

    In this model, only the central tool requires permissions to create or upload new service account keys, but it requires these permissions for all service accounts that it manages.

  • In a pull-based model, each application handles key rotation itself. Periodically, the application uses the IAM API to create or upload a new service account key while authenticating with its existing key. After it has obtained a new service account key, the application deletes its previous key.

    In this model, each application's service account must have the permission to create or upload new service account keys, but only for itself.

Use uploaded keys to let keys expire automatically

Service account keys that you create and download from IAM don't have an expiry date and stay valid until you delete them.

You can limit the validity of service account keys by uploading a service account key instead and by specifying a Valid To date in the X.509 certificate file. Once the expiration date passes, the key can't be used for authentication anymore, but will stay associated with the service account until you delete it.

Limiting the validity of service account keys can limit your security risk and can serve as a fail-safe if key rotation fails. At the same time, limiting the validity of keys increases the risk of an application failure caused by a key that hasn't been renewed in time.

Protecting against privilege escalation

Using service account keys can expose you to privilege escalation attacks if the keys are less well secured than the resources they grant access to.

As an example, suppose a bad actor has already gained a foothold in your environment and now tries to access certain Google Cloud resources. They might still lack the permissions to access these resources, but their privileges might suffice to access a service account key that is stored on a VM, file share, or another less-well secured location. By authenticating using the service account key, the bad actor can assume the identity of the service account. The service account might let the bad actor access resources they previously did not have access to, thereby escalating the bad actor's privileges.

Because a service account key indirectly grants access to resources on Google Cloud, you must consider the key itself to be as valuable, and as much worth protecting, as the resources themselves.

The following sections describe best practices for protecting service account keys and reducing the risk of unauthorized access and resulting privilege escalation.

Avoid storing keys on a file system

Service account keys created by using the Google Cloud Console or the gcloud tool are JSON files, and you can copy these files to the file system of the machine where they are needed. But storing service account keys as files on a file system can expose you to several risks, including:

  • Some file systems such as NTFS use inherited permissions by default. Unless disabled, a permission added to a parent folder might inadvertently cause a key file to become more widely accessible and visible to unauthorized users.
  • In a virtualized environment, bad actors might be able to undermine file system security by accessing the underlying virtual disk.
  • File system access and permission changes are often not audit-logged. If file permissions are inadvertently changed and the key becomes visible to unauthorized users, it might be difficult to analyze when and by whom these changes were made.
  • Files can be easily copied and thus exfiltrated if a bad actor gains access.

Whenever possible, avoid storing service account keys on a file system. If you can't avoid storing keys on disk, make sure to restrict access to the key file, configure file access auditing, and encrypt the underlying disk.

Use an HSM or TPM to store keys

When you create a service account key by using the Google Cloud Console or the gcloud tool, the private key is generated by Google Cloud and then revealed to you. Many security risks associated with service account keys stem from the fact that the private key is, temporarily or permanently, available in clear text and can therefore be difficult to protect.

Instead of letting Google Cloud generate a key pair, you can use a hardware security module (HSM) or Trusted Platform Module (TPM) to create and manage keys:

  1. Use a HSM or TPM to generate a RSA key pair.
  2. Use the key pair to create a self-signed certificate.
  3. Upload the certificate as a service account key.
  4. Let the application use the HSM or TPM's signing API to sign the JWT for authenticating the service account.

An HSM or TPM lets you use a private key without ever revealing the key in clear text. Using an HSM or TPM to manage service account keys therefore helps you enforce access control while also mitigating the risk of keys being copied to other systems.

Some platforms provide abstractions that let you take advantage of a TPM without having to directly interact with it. For example, Windows lets you manage TPM-protected keys by using the CryptoNG API in combination with the Microsoft Platform Crypto Provider.

Service account keys managed by a TPM are unique to a physical or virtual machine. You can still let multiple machines share a service account by associating each machine's key with a common service account.

Use a software-based key store

In situations where using a hardware-based key store isn't viable, use a software-based key store to manage service account keys. Similar to hardware-based options, a software-based key store lets users or applications use service account keys without revealing the private key. Software-based key store solutions can help you control key access in a fine-grained manner and can also ensure that each key access is logged.

The security of a software-based key store typically depends on how its master key is protected. Before you use a software-based key store, make sure to review:

  • how the master key is secured at rest,
  • how the unsealing process works, and who is able to initiate it,
  • how keys are protected from being extracted from memory,
  • how the key store is protected from being undermined if a bad actor gains shell access or hypervisor access to the underlying system.

Don't store keys in Secret Manager or other cloud-based secret stores

Google Cloud as well as other cloud providers provide managed key stores that let you manage various types of secrets. Examples include Secret Manager, Azure KeyVault, or AWS Secret Manager. If you store a service account key in one of these key stores, you must use the cloud provider's IAM system to control access to the secret. Instead of using IAM to control access to a secret, which then grants access to resources on Google Cloud, use workload identity federation to manage access without creating a service account key.

Don't use the Editor role in projects that allow service account key creation or upload

A key difference between the Editor (roles/editor) and Owner (roles/owner) basic roles is that the Editor role doesn't let you change IAM policies or roles. With the Editor role, you therefore can't easily extend your own access or grant other users access to project resources.

The limitations of the Editor role can be undermined if a project contains service accounts. Because the Editor roles grant permission to create or upload service account keys, a bad actor can create new keys for existing service accounts and use these keys to either escalate their own access, or to hand the keys to other users to obtain access to project resources.

Instead of using the Editor role, or any other basic role, it's best to use the more narrowly defined predefined roles, or to create custom roles that only grant necessary permissions.

If you need to use the Editor role, disable service account key upload and key creation by using organization policy constraints to help ensure that the Editor role can't be abused for privilege escalation.

Avoid using service account keys for domain-wide delegation

Domain-wide delegation lets you impersonate a user so that you can access a user's data without any manual authorization on their part. Although examples illustrating the use of domain-wide delegation commonly suggest the use of service account keys, using service account keys is not necessary to perform domain-wide delegation.

When using domain-wide delegation, avoid service account keys and use the signJwt API instead:

  1. Authenticate a service account by using an attached service account, Workload Identity, or Workload identity federation first.
  2. Construct a JWT and use the sub claim to specify the email address of the user for which you're requesting delegated access.
  3. Use the signJwt API to sign the JWT.
  4. Pass the signed JWT to the OAuth2 Token resource to obtain an access token.

By following this approach, you avoid having to manage a service account key, resulting in a setup that can be secured more easily.

Protecting against information disclosure threats

Avoid disclosing confidential information in uploaded X.509 certificates

For each service account key, IAM lets you download a X.509 certificate from the endpoint https://www.googleapis.com/service_accounts/v1/metadata/x509/ACCOUNT_EMAIL. This endpoint is public and doesn't require authentication.

For Google-managed keys and user-managed keys that you created by using the Google Cloud Console or the gcloud tool, the X.509 certificates are created automatically and only contain basic metadata such as the email address and expiry date.

For uploaded service account keys, the X.509 certificate provided by the public endpoint is the same certificate as the one you uploaded. If the certificate you uploaded contained any optional attributes (such as address or location information embedded in the common name), then this information also becomes publicly accessible. A bad actor might use this information to learn more about your environment.

To avoid disclosing confidential information, don't add any optional attributes to uploaded X.509 certificates and use a generic Subject.

Protecting against non-repudiation threats

When you notice suspicious activity affecting your Google Cloud resources and want to analyze its origins, you need data that lets you reconstruct the chain of events that led to the suspicious activity. The primary source of data to perform such analysis are typically audit logs.

Analyzing audit logs can become more difficult when service accounts are involved: if an activity was initiated by a service account, the log entry contains the service account's email address, but you also need to find out which user or application was using the service account at the time.

The following sections contain best practices for using service account keys in a way that helps you track their usage.

Use a dedicated service account for each application

All audit log records contain a principalEmail field that identifies the principal that initiated the activity. If you share a service account key across multiple applications, then it can be difficult to identify which application performed an activity because audit log records contain the same principalEmail value.

Instead of sharing a key among multiple applications, create a dedicated service account for each application. That way, the principalEmail field lets you identify the application associated with a service account which can help you reconstruct the chain of events that led to a suspicious activity.

Use a dedicated key for each machine that runs an application

If you run multiple copies of the same application across multiple machines, then the principalEmail field might let you identify the application, but not the machine where a particular activity originated from.

To help you narrow down the potential sources of suspicious activity, create individual keys for each copy of the application. That way, you can use the serviceAccountKeyName field that many services add to audit log records to distinguish which machine an activity originated from.

What's next