Handle compromised Google Cloud credentials

Google Cloud credentials control access to your resources hosted on Google Cloud. To help keep your data secure and protected from attackers, you must handle your credentials with utmost care.

We recommend that you protect all of your Google Cloud credentials from unintended access. These credentials include but are not limited to the following:

• Service credentials:

  • Service account private keys (JSON and p12 files)
  • API keys
  • OAuth2 client ID secrets

• User credentials that are created and managed on developer workstations or other computers:

  • Google Cloud CLI credentials

  • Application Default Credentials

  • Browser cookies

Google Cloud CLI credentials are stored in the user's home directory. You can list them in Google Cloud CLI using the gcloud auth list command. Application Default Credentials are stored on the developer's workstation. Browser cookies are browser-specific, but are typically stored on the developer's workstation.

If you suspect that any of your credentials have been compromised, you must take immediate action to limit the impact of the compromise on your Google Cloud account.

Monitor for credential compromise

To monitor for potential compromise, consider the following:

• Monitor for suspicious account activity such as privilege escalation and multiple account creations. Monitor for these activities using Cloud Audit Logs, Policy Intelligence and Security Command Center. Use the following Security Command Center services and capabilities:

  • Event Threat Detection to identify threats that are based on administrator activities, Groups changes, and Identity and Access Management (IAM) permission changes. For each threat category, recommended investigation steps are provided to aid in your response.
  • Sensitive Actions Service to track actions in your organization, folders, and projects that could be damaging to your business if they are taken by a malicious actor.
  • Cloud Infrastructure Entitlement Management (CIEM) (Preview) to manage access for identities and generate findings for misconfigurations.

• Monitor user logins in Google Workspace and Cloud Identity. To better track issues, consider exporting the logs to Cloud Logging.

• Monitor for secrets in your code repositories, using tools such as Anomaly Detection or secret scanning.

• Monitor for anomalies in service account key usage using Cloud Monitoring or CIEM.

Ensure that your security operations center (SOC) is notified promptly and has the playbooks, tools, and access that are required to respond quickly to a suspected credential compromise. Use the Security Command Center Enterprise tier to enable SIEM and SOAR capabilities such as playbooks, response workflows, and automated actions. You can also integrate Security Command Center with your existing SIEM or import logs into Google Security Operations for further analysis.

Protect your Google Cloud resources from a compromised credential

Complete the steps in the following sections as soon as you can to help protect your resources if you suspect a credential is compromised.

Revoke and reissue credentials

If you suspect a credential is compromised, revoke and re-issue it. Proceed carefully to ensure you don't suffer a service outage as a result of revoking credentials.

In general, to reissue credentials, you generate a new credential, push it to all services and users that need it, and then revoke the old credential.

The following sections provide specific instructions for each type of credential.

Replace a service account key

1. In the Google Cloud console, go to the Service accounts page.

   Go to Service accounts

2. Locate the affected service account.

3. Create a new key for the service account.

4. Push the new key to all the locations in which the old key was in use.

5. Delete the old key.

Important: Deleting a service account key does not revoke short-lived credentials that were issued based on the key. To revoke a compromised short-lived credential, you must disable or delete the service account that the credential represents. If you do so, any workload that uses the service account will immediately lose access to your resources.

For more information, see Create service accounts.

Regenerate API keys

1. In the Google Cloud console, go to the Credentials page.

   Go to Credentials

2. Create a new API key using the Create credentials button. Configure the new key to be the same as the compromised API key. The restrictions on the API key must match; otherwise you might suffer an outage.

3. Push the API key to all locations in which the old key was in use.

4. Delete the old key.

For more information, see Authenticate by using API keys.

Reset an OAuth2 client ID secret

Changing a client ID secret will cause a temporary outage while the secret is rotated.

1. In the Google Cloud console, go to the Credentials page.

   Go to Credentials

2. Select the compromised OAuth2 client ID and edit it.

3. Click Reset Secret.

4. Push the new secret to your application.

For more information, see Setting up OAuth 2.0 and Using OAuth 2.0 to access Google APIs.

Remove Google Cloud CLI credentials as an administrator

As a Google Workspace administrator, remove access to Google Cloud CLI from the user's list of connected apps. For more information, see View and remove access to third-party applications.

When the user accesses Google Cloud CLI again, it will automatically ask them to re-authorize the application.

Remove Google Cloud CLI credentials as a user

1. Open the list of apps with access to your Google Account.

2. Remove Google Cloud CLI from the list of connected apps.

When you access Google Cloud CLI again, it will automatically ask you to re-authorize the application.

Revoke Application Default Credentials as an administrator

If you suspect that an Application Default Credential is compromised, you can revoke it. This procedure can cause a temporary outage until the credentials file is recreated.

As a Google Workspace administrator, remove access to the Google Auth Library from the user's list of connected apps. For more information, see View and remove access to third-party applications.

Revoke Application Default Credentials as a user

If you suspect that an Application Default Credential that you created is compromised, you can revoke it. This procedure can cause a temporary outage until the credentials file is recreated. This procedure can only be completed by the owner of the compromised credential.

1. Install and initialize the Google Cloud CLI, if you haven't already.

2. Authorize gcloud CLI with your user identity, not with a service account:

    gcloud auth login
   

   For more information, see Authorize the gcloud CLI.

3. Revoke the credentials:

     gcloud auth application-default revoke
   

4. Optionally, delete the application_default_credentials.json file. The location depends on your operating system:

   • Linux, macOS: $HOME/.config/gcloud/
   • Windows: %APPDATA%\gcloud\

5. Recreate the credentials file:

    gcloud auth application-default login
   

Invalidate browser cookies as an administrator

If you suspect browser cookies are compromised, Google Workspace administrators can sign a user out of their account.

In addition, immediately force a password change.

These actions invalidate all existing cookies, and the user is asked to sign in again.

Invalidate browser cookies as a user

If you suspect browser cookies are compromised, sign out of your Google Account and change your password immediately.

These actions invalidate all your existing cookies. The next time you access Google Cloud, you must sign in again.

Look for unauthorized access and resources

After you revoke compromised credentials and restored your service, review all access to your Google Cloud resources. You can use Logging or Security Command Center.

In Logging, complete the following:

1. Examine your audit logs in the Google Cloud console.

   Go to Logs Explorer

2. Search all potentially affected resources, and make sure that all account activity (especially related to the compromised credentials) are as expected.

In Security Command Center, complete the following:

1. In the Google Cloud console, go to the Security Command Center Findings page.

   Go to Findings

2. If necessary, select your Google Cloud project or organization.

3. In the Quick filters section, click an appropriate filter to display the finding that you need in the Findings query results table. For example, if you select Event Threat Detection or Container Threat Detection in the Source display name subsection, only findings from the selected service appear in the results.

   The table is populated with findings for the source you selected.

4. To view details of a specific finding, click the finding name under Category. The finding details pane expands to display a summary of the finding's details.

5. To display all findings that were caused by the same user's actions:

   1. On the finding details pane, copy the email address next to Principal email.
   2. Close the pane.

   3. In Query editor, enter the following query:

      access.principal_email="USER_EMAIL"
      

      Replace USER_EMAIL with the email address that you previously copied.

      Security Command Center displays all findings that are associated with actions taken by the user that you specified.

Delete all unauthorized resources

Make sure that there are no unexpected resources, such as VMs, App Engine apps, service accounts, Cloud Storage buckets, and so forth, that the compromised credential could access.

After you are satisfied that you have identified all unauthorized resources, you can choose to delete these resources immediately. This is especially important for Compute Engine resources, because attackers can use compromised accounts to exfiltrate data or otherwise compromise your production systems.

Alternatively, you can try to isolate unauthorized resources to allow your own forensics teams to perform additional analysis.

Contact Cloud Customer Care

For help with finding the Google Cloud logs and tools that you require for your investigation and mitigation steps, contact Customer Care and open a support case.

Best practices to avoid compromised credentials

This section describes best practices that you can implement to help you avoid compromised credentials.

Separate credentials from code

Manage and store your credentials separately from your source code. It is extremely common to accidentally push both credentials and source code to a source management site like GitHub, which makes your credentials vulnerable to attack.

If you are using GitHub or other public repository, you can implement tools such as Anomaly Detection or secret scanning, which warns you about exposed secrets in your GitHub repositories. To stop keys from being committed to your GitHub repositories, consider using tools such as git-secrets.

Use secret management solutions such as Secret Manager and Hashicorp Vault to store your secrets, rotate them regularly, and apply least privilege.

Implement service account best practices

To help protect service accounts, review the best practices for working with service accounts.

Limit session lengths

To force periodic re-authentication, limit the time that sessions remain active for Google and Google Cloud accounts. For more information, see the following:

• Set session length for Google Workspace

• Set session length for Google Cloud services

• Set session length for Cloud Identity

Use VPC Service Controls to limit access

To limit the impact of compromised credentials, create service perimeters using VPC Service Controls. When you configure VPC Service Controls, resources inside the perimeter can only communicate with other resources inside the perimeter.