Investigating and responding to threats

This topic offers informal guidance to help you investigate and respond to threats, and use additional resources to add context to Security Command Center findings. Following these steps helps you understand what happened during a potential attack and develop possible responses for affected resources.

The techniques on this page are not guaranteed to be effective against any previous, current, or future threats you face. See Remediating threats to understand why Security Command Center does not provide official remediation guidance for threats.

Before you begin

You need adequate Identity and Access Management (IAM) roles to view or edit findings and logs, and modify Google Cloud resources. If you encounter access errors in Security Command Center, ask your administrator for assistance and see Access control to learn about roles. To resolve resource errors, read documentation for affected products.

Understanding threat findings

Event Threat Detection produces security findings by matching events in your Cloud Logging and Google Workspace log streams to known indicators of compromise (IoC). IoCs, developed by internal Google security sources, identify potential vulnerabilities and attacks. Event Threat Detection also detects threats by identifying known adversarial tactics, techniques, and procedures in your logging stream, and by detecting deviations from historically observed behavior of your organization.

Container Threat Detection generates findings by collecting and analyzing low-level observed behavior in the guest kernel of containers.

Findings are written to Security Command Center and Cloud Logging.

Reviewing findings

To review threat findings, do the following:

Legacy Findings page

1. In the Google Cloud console, go to the Security Command Center Findings page.

   Go to Findings

2. If necessary, select your Google Cloud project or organization.

   

3. On the Findings page, next to View by, click Source type.

4. Select Event Threat Detection or Container Threat Detection.

   The table is populated with findings for the source you selected.

5. To view details for a specific finding, click the finding name under category in the table.

Findings page (Preview)

If you opted to upgrade to the Findings Workflow Improvements, follow these steps:

1. In the Google Cloud console, go to the Security Command Center Findings page.

   Go to Findings

2. If necessary, select your Google Cloud project or organization.

   

3. In the Quick filters section, in the Source display name subsection, select Event Threat Detection or Container Threat Detection.

   The table is populated with findings for the source you selected.

4. To view details of a specific finding, click the finding name under Category. The finding details pane expands to display the finding's attributes.

5. To view the finding's JSON definition, click the JSON tab.

Findings provide the names and numeric identifiers of resources involved in an incident, along with environment variables and asset properties. You can use that information to quickly isolate affected resources and determine the potential scope of an event.

To aid in your investigation, threat findings also contain links to the following external resources:

• MITRE ATT&CK framework entries. The framework explains techniques for attacks against cloud resources and provides remediation guidance.
• VirusTotal, an Alphabet-owned service that provides context on potentially malicious files, URLs, domains, and IP addresses.

The following sections outline potential responses to threat findings.

Event Threat Detection responses

To learn more about Event Threat Detection, see how Event Threat Detection works.

Evasion: Access from Anonymizing Proxy

Anomalous access from an anonymous proxy is detected by examining Cloud Audit Logs for Google Cloud service modifications that originated from anonymous proxy IP addresses, like Tor IP addresses.

To respond to these findings, do the following:

Step 1: Review finding details

1. Open an Evasion: Access from Anonymizing Proxy finding, as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • principalEmail: the account that made the changes (a potentially compromised account)
   • Ip: The proxy IP address where the changes are conducted from

Step 2: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Proxy: Multi-hop Proxy.
2. Contact the owner of the account in the principalEmail field. Confirm whether the action was conducted by the legitimate owner.
3. To develop a response plan, combine your investigation results with MITRE research.

Defense Evasion: Modify VPC Service Control

Audit logs are examined to detect changes to VPC Service Controls perimeters that would lead to a reduction in the protection offered by that perimeter. The following are some examples:

• A project is removed from a perimeter
• An access level policy is added to an existing perimeter
• One or more services are added to the list of accessible services

To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Defense Evasion: Modify VPC Service Control finding, as directed in Reviewing findings.
2. In the Finding Details pane, under Additional Information, note the following fields:
   • access.principalEmail: the account that performed the modification
   • resource.name: the name of the VPC Service Controls perimeter that was modified

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • properties
     • name: the name of the VPC Service Controls perimeter that was modified
     • policyLink: the link to the access policy that controls the perimeter
     • delta: the changes, either REMOVE or ADD, to a perimeter that reduced its protection
       • restricted_resources: the projects that follow the restrictions of this perimeter. Protection is reduced if you remove a project
       • restricted_services: the services that are forbidden from running by the restrictions of this perimeter. Protection is reduced if you remove a restricted service
       • allowed_services: the services that are allowed to run under this perimeter's restrictions. Protection is reduced if you add an allowed service
       • access_levels: the access levels that are configured to allow access to resources under the perimeter. Protection is reduced if you add more access levels
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings

Step 2: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. Find admin activity logs related to VPC Service Controls changes by using the following filters:
   • protoPayload.methodName:"AccessContextManager.UpdateServicePerimeter"
   • protoPayload.methodName:"AccessContextManager.ReplaceServicePerimeters"

Step 3: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Defense Evasion: Modify Authentication Process.
2. Review related findings by clicking the link in relatedFindingURI.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 4: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the VPC Service Controls policy and perimeter.
• Consider reverting the changes for the perimeter until the investigation is completed.
• Consider revoking Access Context Manager roles on the principal that modified the perimeter until the investigation is completed.
• Investigate how the reduced protections have been used. For example, if "BigQuery Data Transfer Service API" is enabled, or added as allowed service, check who started using that service and what they are transferring.

Discovery: Can get sensitive Kubernetes object check

A potentially malicious actor attempted to determine what sensitive objects in GKE they can query for, by using the kubectl auth can-i get command. Specifically, the actor ran any of the following commands:

• kubectl auth can-i get '*'
• kubectl auth can-i get secrets
• kubectl auth can-i get clusterroles
• kubectl auth can-i get clusterrolebindings

Step 1: Review finding details

1. Open the Discovery: Can get sensitive Kubernetes object check finding as directed in Reviewing findings.

2. In the finding details, on the Summary tab, note the values of following fields:

   • Under What was detected:
     • Kubernetes access reviews: the requested access review information, based on the SelfSubjectAccessReview k8s resource.
     • Principal email: the account that made the call.
   • Under Affected resource:
     • Resource display name: the Kubernetes cluster where the action occurred.
   • Under Related links:
     • Cloud Logging URI: link to Logging entries.

Step 2: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in the Cloud Logging URI field.

2. On the page that loads, check for other actions taken by the principal by using the following filters:

   • resource.labels.cluster_name="CLUSTER_NAME"

   • protoPayload.authenticationInfo.principalEmail="PRINCIPAL_EMAIL"

     Replace the following:

   • CLUSTER_NAME: the value that you noted in the Resource display name field in the finding details.

   • PRINCIPAL_EMAIL: the value that you noted in the Principal email field in the finding details.

Step 3: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Discovery.
2. Confirm the sensitivity of the object queried and determine whether there are other signs of malicious activity by the principal in the logs.

3. If the principal email is not a service account, contact the owner of the account to confirm whether the legitimate owner conducted the action.

   If the principal email is a service account (IAM or Kubernetes), identify the source of the access review to determine its legitimacy.

4. To develop a response plan, combine your investigation results with MITRE research.

Exfiltration: BigQuery Data Exfiltration

Data exfiltration from BigQuery is detected by examining audit logs for three scenarios:

• A resource is saved outside of your organization.
• VPC Service Controls blocks a copy operation.
• An attempt is made to access BigQuery resources that are protected by VPC Service Controls.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Exfiltration: BigQuery Data Exfiltration finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.

   • exfiltration

     • sources: details about the tables from which data was exfiltrated
     • targets: details about the tables where exfiltrated data was stored

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   • properties

     • projectId: the finding involves an asset in this project
     • jobLink: the link to the BigQuery job that exfiltrated data

     • Query: the SQL query run on the BigQuery dataset

     • userEmail: the account used to exfiltrate the data

   • contextUris: internal and external resources to add context to findings

     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings

Step 2: Investigate in Chronicle

You can use Chronicle to investigate this finding. Chronicle is a Google Cloud service that lets you investigate threats and pivot through related entities in a unified timeline. Chronicle enriches findings data, letting you identify indicators of interest and simplify investigations.

1. Go to the Security Command Center Findings page in the Google Cloud console.

   Go to Findings

2. Next to View by, click Source Type.

3. In the Source type list, select Event Threat Detection.

   The table populates with findings for the source type you selected.

4. In the table, under category, click on an Exfiltration: BigQuery Data Exfiltration finding.

5. In the Finding Details pane, click Investigate in Chronicle.

6. Follow the instructions in Chronicle's guided user interface.

Use the following guides to conduct investigations in Chronicle:

• Investigating a user
• Investigating an asset

Step 3: Review permissions and settings

1. In the Google Cloud console, go to the IAM page .

   Go to IAM

2. If necessary, select the project listed in projectID.

3. On the page that appears, in the Filter box, enter the email address listed inuserEmail and check what permissions are assigned to the account.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. Find admin activity logs related to BigQuery jobs by using the following filters:
   • protoPayload.methodName="Jobservice.insert"
   • protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"

Step 5: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service: Exfiltration to Cloud Storage.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type on the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 6: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with exfiltrated data.
• Consider revoking permissions for userEmail until the investigation is completed.
• To stop further exfiltration, add restrictive IAM policies to the impacted BigQuery datasets (exfiltration.sources and exfiltration.targets).
• To scan impacted datasets for sensitive information, use Cloud Data Loss Prevention. You can also send Cloud DLP data to Security Command Center. Depending on the quantity of information, Cloud DLP costs can be significant. Follow best practices for keeping Cloud DLP costs under control.
• To limit access to the BigQuery API, use VPC Service Controls.
• To identify and fix overly permissive roles, use IAM Recommender.

Exfiltration: BigQuery Data Extraction

Data exfiltration from BigQuery is detected by examining audit logs for two scenarios:

• A resource is saved to a Cloud Storage bucket outside of your organization.
• A resource is saved to a publicly accessible Cloud Storage bucket owned by your organization.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Exfiltration: BigQuery Data Extraction finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • access.principalEmail: the account used to exfiltrate the data
   • resource.name: the name of the BigQuery resource whose data was exfiltrated
   • resource.project_name: the project containing the BigQuery data that was exfiltrated

   • exfiltration

     • sources: details about the BigQuery table from which data was exfiltrated
     • targets: details about the Cloud Storage buckets where data was exported to

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   • properties
     • jobLink: the link to the BigQuery job that exfiltrated data
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings

Step 2: Review permissions and settings

1. In the Google Cloud console, go to the IAM page .

   Go to IAM

2. If necessary, select the project listed in resource.project_name (from Step 1).

3. On the page that appears, in the Filter box, enter the email address listed in access.principalEmail (from Step 1) and check what permissions are assigned to the account.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. Find admin activity logs related to BigQuery jobs by using the following filters:
   • protoPayload.methodName="Jobservice.insert"
   • protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service: Exfiltration to Cloud Storage.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type on the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with exfiltrated data.
• Consider revoking permissions for the principal in the access.principalEmail field until the investigation is completed.
• To stop further exfiltration, add restrictive IAM policies to the impacted BigQuery datasets (exfiltration.sources).
• To scan impacted datasets for sensitive information, use Cloud Data Loss Prevention. You can also send Cloud DLP data to Security Command Center. Depending on the quantity of information, Cloud DLP costs can be significant. Follow best practices for keeping Cloud DLP costs under control.
• To limit access to the BigQuery API, use VPC Service Controls.
• If you are the owner of the bucket, consider revoking public access permissions.
• To identify and fix overly permissive roles, use IAM Recommender.

Exfiltration: BigQuery Data to Google Drive

Data exfiltration from BigQuery is detected by examining audit logs for the following scenario:

• A resource is saved to a Google Drive folder.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Exfiltration: BigQuery Data to Google Drive finding, as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • access.principalEmail: the account used to exfiltrate the data
   • resource.name: the name of the BigQuery resource whose data was exfiltrated
   • resource.project_name: the project containing the BigQuery data that was exfiltrated
   • exfiltration
     • sources: details about the BigQuery table from which data was exfiltrated
     • targets: details about the Google Drive files where data was exported to

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields:

   • properties
     • jobLink: the link to the BigQuery job that exfiltrated data
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings

Step 2: Review permissions and settings

1. In the Google Cloud console, go to the IAM page .

   Go to IAM

2. If necessary, select the project listed in resource.project_name (from Step 1).

3. On the page that appears, in the Filter box, enter the email address listed in access.principalEmail (from Step 1) and check what permissions are assigned to the account.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. Find admin activity logs related to BigQuery jobs by using the following filters:
   • protoPayload.methodName="Jobservice.insert"
   • protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service: Exfiltration to Cloud Storage.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type on the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with exfiltrated data.
• Consider revoking permissions for the principal in the access.principalEmail field until the investigation is completed.
• To stop further exfiltration, add restrictive IAM policies to the impacted BigQuery datasets (exfiltration.sources).
• To scan impacted datasets for sensitive information, use Cloud Data Loss Prevention. You can also send Cloud DLP data to Security Command Center. Depending on the quantity of information, Cloud DLP costs can be significant. Follow best practices for keeping Cloud DLP costs under control.
• To limit access to the BigQuery API, use VPC Service Controls.
• To identify and fix overly permissive roles, use IAM Recommender.

Exfiltration: Cloud SQL Data Exfiltration

Data exfiltration from Cloud SQL is detected by examining audit logs for two scenarios:

• Live instance data exported to a Cloud Storage bucket outside the organization.
• Live instance data exported to a Cloud Storage bucket that is owned by the organization and is publicly accessible.

All Cloud SQL instance types are supported.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Exfiltration: Cloud SQL Data Exfiltration finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • access.principalEmail: the account used to exfiltrate the data
   • resource.name: the resource name of the Cloud SQL whose data was exfiltrated
   • resource.project_name: the project containing the Cloud SQL instance the backup was created from
   • exfiltration
     • sources: details about the Cloud SQL instance whose data was exfiltrated
     • targets: details about the Cloud Storage bucket the data was exported to

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • properties
     • bucketAccess: whether the Cloud Storage bucket is publicly accessible or external to the organization
     • exportScope: how much of the data was exported (the whole instance, one or more databases, one or more tables, or a subset specified by a query)
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to other Cloud SQL exfiltration findings for the same Cloud SQL instance

Step 2: Review permissions and settings

1. In the Google Cloud console, go to the IAM page .

   Go to IAM

2. If necessary, select the project of the instance listed in resource.project_name (from Step 1).

3. On the page that appears, in the Filter box, enter the email address listed in access.principalEmail (from Step 1) and check what permissions are assigned to the account.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI (from Step 1). The Logs Explorer page includes all logs related to the relevant Cloud SQL instance.

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service: Exfiltration to Cloud Storage.
2. Review related findings by clicking the link in relatedFindingURI (from Step 1). Related findings have the same finding type on the same Cloud SQL instance.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with exfiltrated data.
• Consider revoking permissions for access.principalEmail until the investigation is completed.
• To stop further exfiltration, add restrictive IAM policies to the impacted Cloud SQL instances.
  • MySQL
  • PostgreSQL
  • SQL Server
• To limit access to and export from the Cloud SQL Admin API, use VPC Service Controls.
• To identify and fix overly permissive roles, use IAM Recommender.

Exfiltration: Cloud SQL Restore Backup to External Organization

Data exfiltration from a Cloud SQL backup is detected by examining audit logs to determine whether data from the backup has been restored to a Cloud SQL instance outside the organization. All Cloud SQL instance and backup types are supported.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Exfiltration: Cloud SQL Restore Backup to External Organization finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • access.principalEmail: the account used to exfiltrate the data
   • resource.name: the resource name of the backup that was restored
   • resource.parent_name: the resource name of the Cloud SQL instance the backup was created from
   • resource.project_name: the project containing the Cloud SQL instance the backup was created from
   • exfiltration
     • sources: details about the Cloud SQL instance the backup was created from
     • targets: details about the Cloud SQL instance the backup data was restored to

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields:

   • properties/restoreToExternalInstance
     • backupId: the ID of the backup run that was restored
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to other Cloud SQL exfiltration findings for the same Cloud SQL instance

Step 2: Review permissions and settings

1. In the Google Cloud console, go to the IAM page .

   Go to IAM

2. If necessary, select the project of the instance listed in resource.project_name (from Step 1).

3. On the page that appears, in the Filter box, enter the email address listed in access.principalEmail (from Step 1) and check what permissions are assigned to the account.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI (from Step 1). The Logs Explorer page includes all logs related to the relevant Cloud SQL instance.

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service: Exfiltration to Cloud Storage.
2. Review related findings by clicking the link in relatedFindingURI (from Step 1). Related findings have the same finding type on the same Cloud SQL instance.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with exfiltrated data.
• Consider revoking permissions for access.principalEmail until the investigation is completed.
• To stop further exfiltration, add restrictive IAM policies to the impacted Cloud SQL instances.
  • MySQL
  • PostgreSQL
  • SQL Server
• To limit access to the Cloud SQL Admin API, use VPC Service Controls.
• To identify and fix overly permissive roles, use IAM Recommender.

Exfiltration: Cloud SQL Over-Privileged Grant

Detects when all privileges over a PostgreSQL database (or all functions or procedures in a database) are granted to one or more database users.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Exfiltration: Cloud SQL Over-Privileged Grant finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • database.display_name: the name of the database in the Cloud SQL for PostgreSQL instance that was affected
   • database.query the PostgreSQL query executed that granted the privileges
   • database.user_name the PostgreSQL user who granted excess privileges
   • database.grantees the grantees of the overbroad privileges
   • resource.name: the resource name of the Cloud SQL for PostgreSQL instance that was affected
   • resource.parent_name: the resource name of the Cloud SQL for PostgreSQL instance
   • resource.project_name: the project containing the Cloud SQL for PostgreSQL instance

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields:

   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to other exfiltration findings for the same Cloud SQL for PostgreSQL instance

Step 2: Review database privileges

1. Connect to the PostgreSQL database.
2. List and show access privileges for the following:
   • Databases. Use the \l or \list metacommand and check what privileges are assigned for the database listed in database.display_name (from Step 1).
   • Functions or procedures. Use the \df metacommand and check what privileges are assigned for functions or procedures in the database listed in database.display_name (from Step 1).

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI (from Step 1). The Logs Explorer page includes all logs related to the relevant Cloud SQL instance.
2. On the page that loads, check logs for PostgreSQL pgaudit logs (which contain queries executed) for that database by using the following filters:
   • protoPayload.request.database="DATABASE"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service.
2. To determine if additional remediation steps are necessary, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the instance with overprivileged grants.
• Consider revoking all permissions for database.grantees until the investigation is completed.
• To limit access to the database (from database.display_name of Step 1, revoke unnecessary permissions from the grantees (from database.grantees of Step 1.

Initial Access: Database Superuser Writes to User Tables

Detects when the Cloud SQL database superuser (postgres for PostgreSQL and root for MySQL) writes to user tables. The superuser (a role with very broad access) generally should not be used to write to user tables. A user with more limited access should be used for normal daily activity. When a superuser writes to a user table, that could indicate that an attacker has escalated privileges or has compromised the default database user and is modifying data. It could also simply indicate normal but unsafe practices.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Initial Access: Database Superuser Writes to User Tables finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • database.display_name: the name of the database in the Cloud SQL instance that was affected
   • database.query the SQL query executed while writing to user tables
   • database.user_name the superuser
   • resource.name: the resource name of the Cloud SQL instance that was affected
   • resource.parent_name: the resource name of the Cloud SQL instance
   • resource.project_name: the project containing the Cloud SQL instance

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields:

   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to other Cloud SQL exfiltration findings for the same Cloud SQL instance

Step 2: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI (from Step 1). The Logs Explorer page includes all logs related to the relevant Cloud SQL instance.
2. Check the logs for PostgreSQL pgaudit logs or Cloud SQL for MySQL audit logs, which contain the queries executed by the superuser, by using the following filters:
   • protoPayload.request.user="SUPERUSER"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service.
2. To determine if additional remediation steps are necessary, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Review the users allowed to connect to the database.

  • For PostgreSQL, see Create and manage users
  • For MySQL, see Manage users with built-in authentication

• Consider changing the password for the superuser.

  • For PostgreSQL, see Set the password for the default user
  • For MySQL, see Set the password for the default user

• Consider creating a new, limited access user for the different types of queries used on the instance.

  • Grant the new user only the necessary permissions needed to execute their queries.

    • For PostgreSQL, see Grant (command)
    • For MySQL, see Access Control and Account Management

  • Update the credentials for the clients that connect to the Cloud SQL instance

Brute Force: SSH

Detection of successful brute force of SSH on a host. To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Brute Force: SSH finding, as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • sourceLogId: the project ID and timestamp to identify the log entry
     • projectId: the project that contains the finding
   • Attempts: the number of login attempts
     • sourceIP: the IP address that launched the attack
     • username: the account that logged in
     • vmName: the name of the virtual machine
     • authResult: the SSH authentication result
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings of the same type

Step 2: Investigate in Chronicle

You can use Chronicle to investigate this finding. Chronicle is a Google Cloud service that lets you investigate threats and pivot through related entities in an easy to use timeline. Chronicle enriches findings data, letting you identify indicators of interest and simplify investigations.

1. Go to the Security Command Center Findings page in the Google Cloud console.

   Go to Findings

2. Next to View by, click Source Type.

3. In the Source type list, select Event Threat Detection.

   The table populates with findings for the source type you selected.

4. In the table, under category, click on a Brute Force: SSH finding.

5. In the Finding Details pane, click Investigate in Chronicle.

6. Follow the instructions in Chronicle's guided user interface.

Use the following guides to conduct investigations in Chronicle:

• Investigating a user
• Investigating an IP address
• Investigating an asset

Step 3: Review permissions and settings

1. In the Google Cloud console, go to the Dashboard.

   Go to the Dashboard

2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in projectId.

3. Navigate to the Resources card and click Compute Engine.

4. Click the VM instance that matches the name and zone in vmName. Review instance details, including network and access settings.

5. In the navigation pane, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules on port 22.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. On the page that loads, find VPC Flow Logs related to srcIP by using the following filter:
   • logName="projects/projectId/logs/syslog"
   • labels."compute.googleapis.com/resource_name"="vmName"

Step 5: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Local Accounts.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 6: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the successful brute force attempt.
• Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and desponse solution.
• Consider disabling SSH access to the VM. For information about disabling SSH keys, see Restrict SSH keys from VMs. This step could interrupt authorized access to the VM, so consider the needs of your organization before you proceed.
• Only use SSH authentication with authorized keys.
• Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the quantity of information, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.

Credential Access: External Member Added To Privileged Group

Detects when an external member is added to a privileged Google Group (a group granted sensitive roles or permissions). To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Credential Access: External Member Added To Privileged Group finding, as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, click Source Properties, and then note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • principalEmail: the account that made the changes
   • groupName: the Google Group where the changes were made
   • externalMember: the newly added external member
   • sensitiveRoles: the sensitive roles associated with this group
   • cloudLoggingQueryURI: a link to the Logging page
   • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type

Step 2: Review group members

1. Go to Google Groups.

   Go to Google Groups

2. Click the name of the group you want to review.

3. In the navigation menu, click Members.

4. If the newly added external member should not be in this group, click the checkbox next to the members name, and then select Remove member or Ban member.

   To remove or members, you must be a Google Workspace Admin, or assigned the Owner or Manager role in the Google Group. For more information, see Assign roles to a group's members.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.

2. If necessary, select your project.

   

3. On the page that loads, check logs for Google Group membership changes using the following filters:

   • protoPayload.methodName="google.apps.cloudidentity.groups.v1.MembershipsService.UpdateMembership"
   • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts.
2. To determine if additional remediation steps are necessary, combine your investigation results with MITRE research.

Credential Access: Privileged Group Opened To Public

Detects when a privileged Google Group (a group granted sensitive roles or permissions) is changed to be accessible to the general public. To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Credential Access: Privileged Group Opened To Public finding, as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, click Source Properties, and then note the following fields:

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • principalEmail: the account that made the changes (a potentially compromised account)
   • groupName: the Google Group where the changes were made
   • sensitiveRoles: the sensitive roles associated with this group
   • whoCanJoin: the joinability setting of the group
   • cloudLoggingQueryURI: a link to the Logging page
   • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type

Step 2: Review group access settings

1. Go to the Admin Console for Google Groups. You must be a Google Workspace Admin to sign in to the console.

   Go to Admin Console

2. In the navigation pane, click Directory, and then select Groups.

3. Click the name of the group you want to review.

4. Click Access Settings, and then, under Who can join the group, review the group's joinability setting.

5. In the drop-down menu, if needed, change the joinability setting.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.

2. If necessary, select your project.

   

3. On the page that loads, check logs for Google Group settings changes using the following filters:

   • protoPayload.methodName="google.admin.AdminService.changeGroupSetting"
   • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts.
2. To determine if additional remediation steps are necessary, combine your investigation results with MITRE research.

Credential Access: Sensitive Role Granted To Hybrid Group

Detects when sensitive roles or permissions are granted to a Google Group with external members. To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Credential Access: Sensitive Role Granted To Hybrid Group finding, as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, click Source Properties, and then note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • principalEmail: the account that made the changes (a potentially compromised account)
   • groupName: the Google Group where the changes were made
   • bindingDeltas: the sensitive roles that are newly granted to this group
   • resourceName: the resource where the new role was granted
   • cloudLoggingQueryURI: a link to the Logging page
   • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type

Step 2: Review group permissions

1. Go to the IAM page in the Google Cloud console.

   Go to IAM

2. In the Filter field, enter the account name listed in groupName.

3. Review the sensitive roles granted to the group.

4. If the newly added sensitive role is not needed, revoke the role.

   You need specific permissions to manage roles in your organization. For more information, see Required permissions.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.

2. If necessary, select your project.

   

3. On the page that loads, check logs for Google Group settings changes using the following filters:

   • protoPayload.methodName="SetIamPolicy"
   • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts.
2. To determine if additional remediation steps are necessary, combine your investigation results with MITRE research.

Cryptomining

Malware is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known command and control domains and IP addresses. To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Cryptomining: Bad IP finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • connections
     • sourceIp: the suspected cryptomining IP address
     • sourcePort: the source port of the connection
     • destinationIp: the target IP address
     • destinationPort: the destination port of the connection
     • protocol: the IANA protocol number associated with the connection

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   • properties_sourceInstance: the affected Compute Engine VM instance
   • properties_project_id: the project for the Compute Engine instance

Step 2: Review permissions and settings

1. In the Google Cloud console, go to the Dashboard page.

   Go to the Dashboard

2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in properties_project_id.

3. Navigate to the Resources card and click Compute Engine.

4. Click the VM instance that matches properties_sourceInstance. Investigate the potentially compromised instance for malware.

5. In the navigation pane, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select your project.

3. On the page that loads, find VPC Flow Logs related to Properties_ip_0 by using the following filter:

   • logName="projects/properties_project_id/logs/compute.googleapis.com%2Fvpc_flows"
   • (jsonPayload.connection.src_ip="Properties_ip_0" OR jsonPayload.connection.dest_ip="Properties_ip_0")

Step 4: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Resource Hijacking.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project containing malware.
• Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and response solution.
• If necessary, stop the compromised instance and replace it with a new instance.
• Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.

Initial Access: Log4j Compromise Attempt

This finding is generated when Java Naming and Directory Interface (JNDI) lookups within headers or URL parameters are detected. These lookups may indicate attempts at Log4Shell exploitation. To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Initial Access: Log4j Compromise Attempt finding, as directed in Reviewing finding details.

2. In the Finding Details pane, under Additional Information, click Source Properties, and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • properties
     • loadBalancerName: the name of the load balancer that received the JNDI lookup
     • requestUrl: the request URL of the HTTP request. If present, this contains a JNDI lookup.
     • requestUserAgent: the user agent that sent the HTTP request. If present, this contains a JNDI lookup.
     • refererUrl: the URL of the page that sent the HTTP request. If present, this contains a JNDI lookup.
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings of the same type

Step 2: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in the cloudLoggingQueryURI field from step 1.

2. On the page that loads, check the httpRequest fields for string tokens like ${jndi:ldap:// that may indicate possible exploitation attempts.

   See CVE-2021-44228: Detecting Log4Shell exploit in the Logging documentation for example strings to search for and for an example query.

Step 3: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exploit Public-Facing Application.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 4: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Upgrade to the latest version of Log4j2.
• Follow Google Cloud's recommendations for investigating and responding to the "Apache Log4j 2" vulnerability {: .external target="blog" track-type="tasks" track-name="blogLink"}
• Implement the recommended mitigation techniques in Apache Log4j Security Vulnerabilities
• If you use Google Cloud Armor, deploy the cve-canary rule into a new or existing Google Cloud Armor security policy. For more information, see Google Cloud Armor WAF rule to help mitigate Apache Log4j vulnerability

Active Scan: Log4j Vulnerable to RCE

Supported Log4j vulnerability scanners inject obfuscated JNDI lookups in HTTP parameters, URLs, and text fields with callbacks to domains controlled by the scanners. This finding is generated when DNS queries for the unobfuscated domains are found. Such queries only occur if a JNDI lookup was successful, indicating an active Log4j vulnerability. To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Active Scan: Log4j Vulnerable to RCE finding, as directed in Reviewing finding details.

2. In the Finding Details pane, click the Attributes tab and note the following field.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • resource.name: the full resource name of the Compute Engine instance that is vulnerable to the Log4j RCE.

3. In the Finding Details pane, under Additional Information, click Source Properties, and note the following fields:

   • properties
     • scannerDomain: the domain used by the scanner as part of the JNDI lookup. This tells you which scanner identified the vulnerability.
     • sourceIp: the IP address used to make the DNS query
     • vpcName: the name of the network on the instance where the DNS query was made
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings of the same type

Step 2: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in the cloudLoggingQueryURI field from step 1.

2. On the page that loads, check the httpRequest fields for string tokens like ${jndi:ldap:// that may indicate possible exploitation attempts.

   See CVE-2021-44228: Detecting Log4Shell exploit in the Logging documentation for example strings to search for and for an example query.

Step 3: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Exploitation of Remote Services.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 4: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Upgrade to the latest version of Log4j2.
• Follow Google Cloud's recommendations for investigating and responding to the "Apache Log4j 2" vulnerability {: .external target="blog" track-type="tasks" track-name="blogLink"}
• Implement the recommended mitigation techniques in Apache Log4j Security Vulnerabilities
• If you use Google Cloud Armor, deploy the cve-canary rule into a new or existing Google Cloud Armor security policy. For more information, see Google Cloud Armor WAF rule to help mitigate Apache Log4j vulnerability

Leaked credentials

This finding is generated when Google Cloud service account credentials are accidentally leaked online or compromised. To respond to this finding, do the following:

Step 1: Review finding details

1. Open an account_has_leaked_credentials finding, as directed in Reviewing finding details.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • Compromised_account: the potentially compromised service account
   • Project_identifier: the project that contains the potentially leaked account credentials
   • URL: the link to the GitHub repository

Step 2: Review project and service account permissions

1. In the Google Cloud console, go to the IAM page.

   Go to IAM

2. If necessary, select the project listed in Project_identifier.

3. On the page that appears, in the Filter box, enter the account name listed in Compromised_account and check assigned permissions.

4. In the Google Cloud console, go to the Service Accounts page.

   Go to Service Accounts

5. On the page that appears, in the Filter box, enter the name of the compromised service account and check the service account's keys and key creation dates.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select your project.

3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:

   • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
   • protoPayload.methodName="SetIamPolicy"
   • resource.type="gce_instance" AND log_name="projects/Project_identifier/logs/cloudaudit.googleapis.com%2Factivity"
   • protoPayload.methodName="InsertProjectOwnershipInvite"
   • protoPayload.authenticationInfo.principalEmail="Compromised_account"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Cloud Accounts.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with leaked credentials.
• Consider deleting the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access. Before proceeding, your security team should identify all impacted resources and work with resource owners to ensure business continuity.
• Work with your security team to identify unfamiliar resources, including Compute Engine instances, snapshots, service accounts, and IAM users. Delete resources not created with authorized accounts.
• Respond to notifications from Google Cloud Support.
• To limit who can create service accounts, use the Organization Policy Service.
• To identify and fix overly permissive roles, use IAM Recommender.
• Open the URL link and delete the leaked credentials. Gather more information about the compromised account and contact the owner.

Malware

Malware is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known command and control domains and IP addresses. Currently, Event Threat Detection provides general malware detection ( Malware: Bad IP and Malware: Bad Domain) and detectors particularly for Log4j-related malware (Log4j Malware: Bad IP and Log4j Malware: Bad Domain). The following steps describe how to investigate and respond to IP-based findings. The remediation steps are similar for domain-based findings.

Step 1: Review finding details

1. Open a Malware: Bad IP finding, as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • connections
     • sourceIp: a known malware command and control IP address
     • sourcePort: the source port of the connection
     • destinationIp: the target IP address of the malware
     • destinationPort: the destination port of the connection
     • protocol: the IANA protocol number associated with the connection

3. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   • sourceLogId
     • projectId: the project that contains the finding
   • InstanceDetails: the resource address for the Compute Engine instance
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • virustotalIndicatorQueryUI: link to the VirusTotal analysis page
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings of the same type

Step 2: Investigate in Chronicle

You can use Chronicle to investigate this finding. Chronicle is a Google Cloud service that lets you investigate threats and pivot through related entities in an easy to use timeline. Chronicle enriches findings data, letting you identify indicators of interest and simplify investigations.

1. Go to the Security Command Center Findings page in the Google Cloud console.

   Go to Findings

2. Next to View by, click Source Type.

3. In the Source type list, select Event Threat Detection.

   The table populates with findings for the source type you selected.

4. In the table, under category, click on a Malware: Bad IP finding.

5. In the Finding Details pane, click Investigate in Chronicle.

6. Follow the instructions in Chronicle's guided user interface.

Use the following guides to conduct investigations in Chronicle:

• Investigating an IP address
• Investigating an asset

Step 3: Review permissions and settings

1. In the Google Cloud console, go to the Dashboard page.

   Go to the Dashboard

2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in projectId.

3. Navigate to the Resources card and click Compute Engine.

4. Click the VM instance that matches the name and zone in instanceDetails. Review instance details, including network and access settings.

5. In the navigation pane, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. On the page that loads, find VPC Flow Logs related to the IP address in srcIP by using the following filter:
   • logName="projects/projectId/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="srcIP" OR jsonPayload.connection.dest_ip="destIP")

Step 5: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Dynamic Resolution and Command and Control.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
3. Check flagged URLs and domains on VirusTotal by clicking the link in virustotalIndicatorQueryUI. VirusTotal is an Alphabet-owned service that provides context on potentially malicious files, URLs, domains, and IP addresses.
4. To develop a response plan, combine your investigation results with MITRE research.

Step 6: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project containing malware.
• Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and response solution.
• To track activity and vulnerabilities that allowed the insertion of malware, check audit logs and syslogs associated with the compromised instance.
• If necessary, stop the compromised instance and replace it with a new instance.
• Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.
• To control access and use of VM images, use Shielded VM and Trusted Images IAM policy.

Outgoing DoS

Event Threat Detection detects potential use of an instance to launch a denial of service (DoS) attack by analyzing VPC Flow Logs. To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Malware: Outgoing DoS finding as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • sourceInstanceDetails: the affected Compute Engine VM instance
   • ipConnection
     • srcIP: the source IP address of the DoS activity
     • srcPort: the source port of the connection
     • destIP: the target IP address of the DoS activity
     • destPort: the destination port of the connection
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings of the same type

Step 2: Review permissions and settings

1. In the Google Cloud console, go to the Dashboard page.

   Go to the Dashboard

2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project ID listed in sourceInstanceDetails.

3. Navigate to the Resources card and click Compute Engine.

4. Click the VM instance that matches the instance name and zone in sourceInstanceDetails. Review instance details, including network and access settings.

5. In the navigation pane, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. On the page that loads, find VPC Flow Logs related to the IP address in srcIP by using the following filter:
   • logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="srcIP" OR jsonPayload.connection.dest_ip="destIP")

Step 4: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Network Denial of Service.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project experiencing outgoing DoS traffic.
• Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and response solution.
• To track activity and vulnerabilities that allowed the insertion of malware, check audit logs and syslogs associated with the compromised instance.
• If necessary, stop the compromised instance and replace it with a new instance.
• Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.
• To control access and use of VM images, use Shielded VM and Trusted Images IAM policy.

Persistence: IAM Anomalous Grant

Audit logs are examined to detect the addition of IAM (IAM) role grants that might be considered suspicious. The following are examples of anomalous grants:

• Inviting a gmail.com user as a project owner from the Google Cloud Console
• A service account granting sensitive permissions
• A custom role granting sensitive permissions
• A service account added from outside your organization

To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Persistence: IAM Anomalous Grant finding as directed in Reviewing findings.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • projectId: the project that contains the finding
   • principalEmail: email address for the user or service account that assigned the role
   • bindingDeltas
     • Action: the action taken by the user
     • Role: the role assigned to the user
     • member: the email address of the user that received the role
   • contextUris: internal and external resources to add context to findings
     • mitreURI: a link to the MITRE ATT&CK framework entry for this finding type
     • virustotalIndicatorQueryUI: link to the VirusTotal analysis page
     • cloudLoggingQueryURI: a link to the Logging page
     • relatedFindingURI: a link to related findings of the same type

Step 2: Investigate in Chronicle

You can use Chronicle to investigate this finding. Chronicle is a Google Cloud service that lets you investigate threats and pivot through related entities in an easy to use timeline. Chronicle enriches findings data, letting you identify indicators of interest and simplify investigations.

1. Go to the Security Command Center Findings page in the Google Cloud console.

   Go to Findings

2. Next to View by, click Source Type.

3. In the Source type list, select Event Threat Detection.

   The table populates with findings for the source type you selected.

4. In the table, under category, click on a Persistence: IAM Anomalous Grant finding.

5. In the Finding Details pane, click Investigate in Chronicle.

6. Follow the instructions in Chronicle's guided user interface.

Use the following guides to conduct investigations in Chronicle:

• Investigating a user
• Investigating an asset

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. On the page that loads, look for new or updated IAM resources using the following filters:
   • protoPayload.methodName="SetIamPolicy"
   • protoPayload.methodName="google.iam.admin.v1.UpdateRole"
   • protoPayload.methodName="google.iam.admin.v1.CreateRole"

Step 4: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Valid Accounts: Cloud Accounts.
2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised account.
• Delete the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access.
• Delete project resources created by unauthorized accounts, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.
• To restrict adding gmail.com users, use the Organization Policy.
• To identify and fix overly permissive roles, use IAM Recommender.

Persistence: New Geography

An IAM user or service account is accessing Google Cloud from an anomalous location, based on the geolocation of the requesting IP address.

Step 1: Review finding details

1. Open a Persistence: New Geography finding, as directed in Reviewing finding details earlier on this page.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • principalEmail: the potentially compromised user account
   • projectId: the project that contains the potentially compromised user account
   • callerIp: the external IP address
   • anomalousLocation: the estimated current location of the user
   • typicalGeolocations: the locations where the user usually accesses Google Cloud resources
   • notSeenInLast: the time period used to establish a baseline for normal behavior
   • gcpResourceName: the affected resource
   • cloudLoggingQueryURI: the link to Cloud Logging entries
   • mitreUri: the link to the MITRE ATT&CK framework entry

Step 2: Review project and account permissions

1. In the Google Cloud console, go to the IAM page.

   Go to IAM

2. If necessary, select the project listed in projectId.

3. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check granted roles.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. If necessary, select your project.
3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:
   • protoPayload.methodName="SetIamPolicy"
   • protoPayload.methodName="google.iam.admin.v1.UpdateRole"
   • protoPayload.methodName="google.iam.admin.v1.CreateRole"
   • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Cloud Accounts.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised account.
• Review the anomalousLocation, typicalGeolocations, and notSeenInLast fields to verify whether the access is abnormal and if the account has been compromised.
• Delete project resources created by unauthorized accounts, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.
• To restrict the creation of new resources to specific regions, see Restricting Resource Locations.
• To identify and fix overly permissive roles, use IAM Recommender.

Persistence: New User Agent

An IAM service account is accessing Google Cloud using suspicious software, as indicated by an anomalous user agent.

Step 1: Review finding details

1. Open a Persistence: New User Agent finding, as directed in Reviewing finding details earlier on this page.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • principalEmail: the potentially compromised service account
   • projectId: the project that contains the potentially compromised service account
   • callerUserAgent: the anomalous user agent
   • anomalousSoftwareClassification: the type of software
   • notSeenInLast: the time period used to establish a baseline for normal behavior
   • cloudLoggingQueryURI: the link to Cloud Logging entries
   • mitreUri: the link to the MITRE ATT&CK framework entry

Step 2: Review project and account permissions

1. In the Google Cloud console, go to the IAM page.

   Go to IAM

2. If necessary, select the project listed in projectId.

3. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check granted roles.

4. In the Google Cloud console, go to the Service Accounts page.

   Go to Service Accounts

5. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check the service account's keys and key creation dates.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. If necessary, select your project.
3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:
   • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
   • protoPayload.methodName="SetIamPolicy"
   • protoPayload.methodName="google.iam.admin.v1.UpdateRole"
   • protoPayload.methodName="google.iam.admin.v1.CreateRole"
   • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Cloud Accounts.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised account.
• Review the anomalousSoftwareClassification, callerUserAgent, and behaviorPeriod fields to verify whether the access is abnormal and if the account has been compromised.
• Delete project resources created by unauthorized accounts, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.
• To restrict the creation of new resources to specific regions, see Restricting Resource Locations.
• To identify and fix overly permissive roles, use IAM Recommender.

Privilege Escalation: Changes to sensitive Kubernetes RBAC objects

To escalate privilege, a potentially malicious actor attempted to modify a ClusterRole or ClusterRoleBinding role-based access control (RBAC) object of the sensitive cluster-admin role by using a PUT or PATCH request.

Step 1: Review finding details

1. Open the Privilege Escalation: Changes to sensitive Kubernetes RBAC objects finding as directed in Reviewing findings.

2. In the finding details, on the Summary tab, note the values of following fields:

   • Under What was detected:
     • Kubernetes roles: the sensitive Kubernetes role or ClusterRole that was modified.
     • Kubernetes bindings: the sensitive Kubernetes binding or ClusterRoleBinding that was modified.
     • Principal email: the account that made the call.
     • Method name: the method that was called.
   • Under Affected resource:
     • Resource display name: the Kubernetes cluster where the action occurred.
   • Under Related links:
     • Cloud Logging URI: link to Logging entries.

Step 2: Check logs

1. On the Summary tab of the finding details in the Google Cloud console, go to Logs Explorer by clicking the link in the Cloud Logging URI field.

2. If the value in Method name was a PATCH method, check the request body to see what properties were modified.

   In update (PUT) calls, the whole object is sent in the request, so the changes aren't as clear.

3. Check for other actions taken by the principal by using the following filters:

   • resource.labels.cluster_name="CLUSTER_NAME"

   • protoPayload.authenticationInfo.principalEmail="PRINCIPAL_EMAIL"

     Replace the following:

   • CLUSTER_NAME: the value that you noted in the Resource display name field in the finding details.

   • PRINCIPAL_EMAIL: the value that you noted in the Principal email field in the finding details.

Step 3: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Privilege Escalation.
2. Confirm the sensitivity of the object and if the modification is warranted.
3. For bindings, you can check the subject and investigate whether the subject needs the role it is binded to.
4. Determine whether there are other signs of malicious activity by the principal in the logs.

5. If the principal email is not a service account, contact the owner of the account to confirm whether the legitimate owner conducted the action.

   If the principal email is a service account (IAM or Kubernetes), identify the source of the modification to determine its legitimacy.

6. To develop a response plan, combine your investigation results with MITRE research.

Privilege Escalation: Create Kubernetes CSR for master cert

To escalate privilege, a potentially malicious actor created a Kubernetes master certificate signing request (CSR), which gives them cluster-admin access.

Step 1: Review finding details

1. Open the Privilege Escalation: Create Kubernetes CSR for master cert finding as directed in Reviewing findings.

2. In the finding details, on the Summary tab, note the values of following fields:

   • Under What was detected:
     • Principal email: the account that made the call.
     • Method name: the method that was called.
   • Under Affected resource:
     • Resource display name: the Kubernetes cluster where the action occurred.
   • Under Related links:
     • Cloud Logging URI: link to Logging entries.

Step 2: Check logs

1. On the Summary tab of the finding details in the Google Cloud console, go to Logs Explorer by clicking the link in the Cloud Logging URI field.
2. Check the value in the protoPayload.resourceName field to identify the specific certificate signing request.

3. Check for other actions taken by the principal by using the following filters:

   • resource.labels.cluster_name="CLUSTER_NAME"

   • protoPayload.authenticationInfo.principalEmail="PRINCIPAL_EMAIL"

     Replace the following:

   • CLUSTER_NAME: the value that you noted in the Resource display name field in the finding details.

   • PRINCIPAL_EMAIL: the value that you noted in the Principal email field in the finding details.

Step 3: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Privilege Escalation.
2. Investigate whether giving cluster-admin access was warranted.

3. If the principal email is not a service account, contact the owner of the account to confirm whether the legitimate owner conducted the action.

   If the principal email is a service account (IAM or Kubernetes), identify the source of the action to determine its legitimacy.

4. To develop a response plan, combine your investigation results with MITRE research.

Privilege Escalation: Creation of sensitive Kubernetes bindings

To escalate privilege, a potentially malicious actor attempted to create a new RoleBinding or ClusterRoleBinding object for the cluster-admin role.

Step 1: Review finding details

1. Open the Privilege Escalation: Creation of sensitive Kubernetes bindings finding as directed in Reviewing findings.

2. In the finding details, on the Summary tab, note the values of following fields:

   • Under What was detected:
     • Kubernetes bindings: the sensitive Kubernetes binding or ClusterRoleBinding that was created.
     • Principal email: the account that made the call.
   • Under Affected resource:
     • Resource display name: the Kubernetes cluster where the action occurred.
   • Under Related links:
     • Cloud Logging URI: link to Logging entries.

Step 2: Check logs

1. On the Summary tab of the finding details in the Google Cloud console, go to Logs Explorer by clicking the link in the Cloud Logging URI field.

2. Check for other actions taken by the principal by using the following filters:

   • resource.labels.cluster_name="CLUSTER_NAME"

   • protoPayload.authenticationInfo.principalEmail="PRINCIPAL_EMAIL"

     Replace the following:

   • CLUSTER_NAME: the value that you noted in the Resource display name field in the finding details.

   • PRINCIPAL_EMAIL: the value that you noted in the Principal email field in the finding details.

Step 3: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Privilege Escalation.
2. Confirm the sensitivity of the binding created and if the roles are necessary for the subjects.
3. For bindings, you can check the subject and investigate whether the subject needs the role it is binded to.
4. Determine whether there are other signs of malicious activity by the principal in the logs.

5. If the principal email is not a service account, contact the owner of the account to confirm whether the legitimate owner conducted the action.

   If the principal email is a service account (IAM or Kubernetes), identify the source of the action to determine its legitimacy.

6. To develop a response plan, combine your investigation results with MITRE research.

Privilege Escalation: Get Kubernetes CSR with compromised bootstrap credentials

To escalate privilege, a potentially malicious actor queried for a certificate signing request (CSR), with the kubectl command, using compromised bootstrap credentials.

The following is an example of a command that this rule detects:

kubectl --client-certificate kubelet.crt --client-key kubelet.key --server YOUR_SERVER get csr CSR_NAME

Step 1: Review finding details

1. Open the Privilege Escalation: Get Kubernetes CSR with compromised bootstrap credentials finding as directed in Reviewing findings.

2. In the finding details, on the Summary tab, note the values of following fields:

   • Under What was detected:
     • Principal email: the account that made the call.
     • Method name: the method that was called.
   • Under Affected resource:
     • Resource display name: the Kubernetes cluster where the action occurred.
   • Under Related links:
     • Cloud Logging URI: link to Logging entries.

Step 2: Check logs

If the method name, which you noted in the Method name field in the finding details, is a GET method, do the following:

1. On the Summary tab of the finding details in the Google Cloud console, go to Logs Explorer by clicking the link in the Cloud Logging URI field.
2. Check the value in the protoPayload.resourceName field to identify the specific certificate signing request.

Step 3: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Privilege Escalation.
2. If the specific CSR is available in the log entry, investigate the sensitivity of the certificate and whether the action was warranted.
3. To develop a response plan, combine your investigation results with MITRE research.

Privilege Escalation: Launch of privileged Kubernetes container

A potentially malicious actor created a Pod that contains privileged containers or containers with privilege escalation capabilities.

A privileged container has the privileged field set to true. A container with privilege escalation capabilities has the allowPrivilegeEscalation field set to true. For more information, see the SecurityContext v1 core API reference in the Kubernetes documentation.

Step 1: Review finding details

1. Open the Privilege Escalation: Launch of privileged Kubernetes container finding as directed in Reviewing findings.

2. In the finding details, on the Summary tab, note the values of the following fields:

   • Under What was detected:
     • Kubernetes pods: the newly created Pod with privileged containers.
     • Principal email: the account that made the call.
   • Under Affected resource:
     • Resource display name: the Kubernetes cluster where the action occurred.
   • Under Related links:
     • Cloud Logging URI: link to Logging entries.

3. On the JSON tab, note the values of the finding fields:

   • findings.kubernetes.pods[].containers: the privileged container turned up within the Pod.

Step 2: Check logs

1. On the Summary tab of the finding details in the Google Cloud console, go to Logs Explorer by clicking the link in the Cloud Logging URI field.

2. Check for other actions taken by the principal by using the following filters:

   • resource.labels.cluster_name="CLUSTER_NAME"

   • protoPayload.authenticationInfo.principalEmail="PRINCIPAL_EMAIL"

     Replace the following:

   • CLUSTER_NAME: the value that you noted in the Resource display name field in the finding details.

   • PRINCIPAL_EMAIL: the value that you noted in the Principal email field in the finding details.

Step 3: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Privilege Escalation.
2. Confirm that the container created requires access to host resources and kernel capabilities.
3. Determine whether there are other signs of malicious activity by the principal in the logs.

4. If the principal email is not a service account, contact the owner of the account to confirm whether the legitimate owner conducted the action.

   If the principal email is a service account (IAM or Kubernetes), identify the source of the action to determine its legitimacy.

5. To develop a response plan, combine your investigation results with MITRE research.

Service account self-investigation

A service account credential is being used to investigate the roles and permissions associated with that same service account. This finding indicates that service account credentials might be compromised and immediate action should be taken.

Step 1: Review finding details

1. Open a Discovery: Service Account Self-Investigation finding, as directed in Reviewing finding details earlier on this page.

2. In the Finding Details pane, under Additional Information, click Source Properties and note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • principalEmail: the potentially compromised service account
   • projectId: the project that contains the potentially leaked account credentials
   • callerIp: the internal or external IP address
   • cloudLoggingQueryURI: the link to Cloud Logging entries
   • mitreUri: the link to the MITRE ATT&CK framework entry
   • severity: the risk level assigned to the finding; severity is HIGH if the API call that triggered this finding was unauthorized—the service account doesn't have permission to query its own IAM permissions with the projects.getIamPolicy API.

Step 2: Review project and service account permissions

1. In the Google Cloud console, go to the IAM page.

   Go to IAM

2. If necessary, select the project listed in projectId.

3. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check assigned permissions.

4. In the Google Cloud console, go to the Service Accounts page.

   Go to Service Accounts

5. On the page that appears, in the Filter box, enter the name of the compromised service account and check the service account's keys and key creation dates.

Step 3: Check logs

1. In the Google Cloud console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
2. If necessary, select your project.
3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:
   • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
   • protoPayload.methodName="SetIamPolicy"
   • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

1. Review the MITRE ATT&CK framework entry for this finding type: Permission Groups Discovery: Cloud Groups.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised account.
• Delete the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access.
• Delete project resources created by the compromised account, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.

Compute Engine Admin Metadata detections

Unexpected modifications to Compute Engine VM instance metadata can be an indication of suspicious activity.

Persistence: Compute Engine Admin Added SSH Key

Description	Actions
The ssh-keys Compute Engine instance metadata key was changed on an established instance.	The Compute Engine instance metadata key ssh-keys was modified on an instance that was created more than seven days ago. Verify whether the change was done intentionally by a member, or if it was implemented by an adversary to introduce new access to your organization.	Check logs using the following filters: protopayload.resource.labels.instance_id=INSTANCE_ID protoPayload.serviceName="compute.googleapis.com" (protoPayload.metadata.instanceMetaData.addedMetadataKey : "ssh-keys" OR protoPayload.metadata.instanceMetaData.modifiedMetadataKey : "ssh-keys" ) logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity Replace the following: INSTANCE_ID: the gceInstanceId listed in the finding ORGANIZATION_ID: your organization ID
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1098/004/

Persistence: Compute Engine Admin Added Startup Script

Description	Actions
The startup-script or startup-script-url Compute Engine instance metadata key was changed on an established instance.	The Compute Engine instance metadata key startup-script or startup-script-url was modified on an instance that was created more than seven days ago. Verify whether the change was done intentionally by a member, or if it was implemented by an adversary to introduce new access to your organization.	Check logs using the following filters: protopayload.resource.labels.instance_id=INSTANCE_ID protoPayload.serviceName="compute.googleapis.com" ((protoPayload.metadata.instanceMetaData.addedMetadataKey : "startup-script" OR protoPayload.metadata.instanceMetaData.modifiedMetadataKey : "startup-script" ) OR (protoPayload.metadata.instanceMetaData.addedMetadataKey : "startup-script-url" OR protoPayload.metadata.instanceMetaData.modifiedMetadataKey : "startup-script-url" )) logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity Replace the following: INSTANCE_ID: the gceInstanceId listed in the finding ORGANIZATION_ID: your organization ID
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1098/004/

Google Workspace logs detections

If you share your Google Workspace logs with Cloud Logging, Event Threat Detection generates findings for several Google Workspace threats.

Event Threat Detection enriches log events and writes findings to Security Command Center. The following table contains Google Workspace threats, relevant MITRE ATT&CK framework entries, and details about the events that trigger findings. You can also check logs using specific filters, and combine all of the information to respond to Google Workspace threats.

Initial Access: Disabled Password Leak

Description	Actions
A member's account is disabled because a password leak was detected.	Reset passwords for affected accounts, and advise members to use strong, unique passwords for corporate accounts.	Check logs using the following filters: protopayload.resource.labels.service="login.googleapis.com" logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access Replace ORGANIZATION_ID with your organization ID.
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1078/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/login#account_disabled_password_leak

Initial Access: Suspicious Login Blocked

Description	Actions
A suspicious login to a member's account was detected and blocked.	This account might be targeted by adversaries. Ensure the user account follows your organization's security guidelines for strong passwords and multifactor authentication.	Check logs using the following filters: protopayload.resource.labels.service="login.googleapis.com" logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access Replace ORGANIZATION_ID with your organization ID.
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1078/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/login#suspicious_login

Initial Access: Account Disabled Hijacked

Description	Actions
A member's account was suspended due to suspicious activity.	This account was hijacked. Reset the account password and encourage users to create strong, unique passwords for corporate accounts.	Check logs using the following filters: protopayload.resource.labels.service="login.googleapis.com" logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access Replace ORGANIZATION_ID with your organization ID.
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1078/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/login#account_disabled_hijacked

Impair Defenses: Two Step Verification Disabled

Description	Actions
A member disabled 2-step verification.	Verify if the user intended to disable 2-step verification. If your organization requires 2-step verification, ensure that the user enables it promptly.	Check logs using the following filters: protopayload.resource.labels.service="login.googleapis.com" logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access Replace ORGANIZATION_ID with your organization ID.
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1562/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/login#2sv_disable

Initial Access: Government Based Attack

Description	Actions
Government-backed attackers might have tried to compromise a member account or computer.	This account might be targeted by adversaries. Ensure the user account follows your organization's security guidelines for strong passwords and multifactor authentication.	Check logs using the following filters: protopayload.resource.labels.service="login.googleapis.com" logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access Replace ORGANIZATION_ID with your organization ID.
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1078/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/login#gov_attack_warning

Persistence: SSO Enablement Toggle

Description	Actions
The Enable SSO (single sign-on) setting on the admin account was disabled.	SSO settings for your organization were changed. Verify whether the change was done intentionally by a member or if it was implemented by an adversary to introduce new access to your organization.	Check logs using the following filters: protopayload.resource.labels.service="admin.googleapis.com" protopayload.metadata.event.parameter.value=DOMAIN_NAME logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity Replace the following: DOMAIN_NAME: the domainName listed in the finding ORGANIZATION_ID: your organization ID
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1098/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/admin-domain-settings#TOGGLE_SSO_ENABLED

Persistence: SSO Settings Changed

Description	Actions
The SSO settings for the admin account were changed.	SSO settings for your organization were changed. Verify whether the change was done intentionally by a member or if it was implemented by an adversary to introduce new access to your organization.	Check logs using the following filters: protopayload.resource.labels.service="admin.googleapis.com" protopayload.metadata.event.parameter.value=DOMAIN_NAME logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity Replace the following: DOMAIN_NAME: the domainName listed in the finding ORGANIZATION_ID: your organization ID
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1098/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/admin-domain-settings#CHANGE_SSO_SETTINGS

Impair Defenses: Strong Authentication Disabled

Description	Actions
2-step verification was disabled for the organization.	2-step verification is no longer required for your organization. Find out if this was an intentional policy change by an administrator, or if this is an attempt by an adversary to make account hijacking easier.	Check logs using the following filters: protopayload.resource.labels.service="admin.googleapis.com" logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity Replace ORGANIZATION_ID with your organization ID.
		Research events that trigger this finding: MITRE: https://attack.mitre.org/techniques/T1562/ Workspace event details: https://developers.google.com/admin-sdk/reports/v1/appendix/activity/admin-security-settings#ENFORCE_STRONG_AUTHENTICATION

Respond to Google Workspace threats

If you're a Google Workspace Admin, you can use the service's security tools to resolve these threats:

• Google Workspace Alert Center
• Google Workspace Security Center

The tools include alerts, a security dashboard, security recommendations, and can help you investigate and remediate threats.

If you're not a Google Workspace Admin, do the following:

• If you still have access to your account, change or reset your password and turn on 2-Step verification.
• Reach out to your Google Workspace Admin or the team at your company that manages your Google Workspace account. Use these findings as an indication that an account might be compromised.

Container Threat Detection responses

To learn more about Container Threat Detection, see how Container Threat Detection works.

Added Binary Executed

A binary that was not part of the original container image was executed. Attackers commonly install exploitation tooling and malware after the initial compromise. To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Added Binary Executed finding as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • resource.name: the full resource name of the cluster including the project number, location, and cluster name
   • resource.project_display_name: the name of the project that contains the cluster

   • processes

     • binary.path: the absolute path of the added binary
     • args: the arguments provided when invoking the added binary

3. Click the Source Properties tab and note the following fields:

   • Pod_Namespace: the name of the Pod's Kubernetes namespace
   • Pod_Name: the name of the GKE Pod
   • Container_Name: the name of the affected container
   • Container_Image_Uri: the name of the container image being deployed
   • VM_Instance_Name: the name of the GKE node where the Pod executed

Step 2: Review cluster and node

1. In the Google Cloud console, go to the Kubernetes clusters page.

   Go to Kubernetes clusters

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

1. In the Google Cloud console, go to the Kubernetes Workloads page.

   Go to Kubernetes Workloads

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Set Select time range to the period of interest.

4. On the page that loads, do the following:

   1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
   2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
   3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

1. Go to the Google Cloud console.

   Open Google Cloud console

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Click Activate Cloud Shell

4. Obtain GKE credentials for your cluster by running the following commands.

   For zonal clusters, run:

     gcloud container clusters get-credentials cluster_name --zone location --project project_name
   

   For regional clusters, run:

     gcloud container clusters get-credentials cluster_name --region location --project project_name
   

   Replace the following:

   • cluster_name: the cluster listed in resource.labels.cluster_name
   • location: the location listed in resource.labels.location
   • project_name: the project name listed in resource.project_display_name

5. Retrieve the added binary by running:

     kubectl cp Pod_Namespace/Pod_Name:Process_Binary_Fullpath -c Container_Name  local_file
   

   Replace local_file with a local file path to store the added binary.

6. Connect to the container environment by running:

     kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh
   

   This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Ingress Tool Transfer, Native API.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised container.
• Stop or delete the compromised container and replace it with a new container.

Added Library Loaded

A library that was not part of the original container image was loaded. Attackers might load malicious libraries into existing programs in order to bypass code execution protections and to hide malicious code. To respond to this finding, do the following:

Step 1: Review finding details

1. Open an Added Library Loaded finding as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • resource.name: the full resource name of the cluster
   • resource.project_display_name: the name of the project that contains the asset

   • processes

     • libraries: details about the added library
     • binary.path: the full path of the process binary that loaded the library
     • args: the arguments provided when invoking the process binary

3. Click the Source Properties tab and note the following fields:

   • Pod_Namespace: the name of the Pod's Kubernetes namespace
   • Pod_Name: the name of the GKE Pod
   • Container_Name: the name of the affected container
   • Container_Image_Uri: the name of the container image being executed
   • VM_Instance_Name: the name of the GKE node where the Pod executed

Step 2: Review cluster and node

1. In the Google Cloud console, go to the Kubernetes clusters page.

   Go to Kubernetes clusters

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

1. In the Google Cloud console, go to the Kubernetes Workloads page.

   Go to Kubernetes Workloads

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Set Select time range to the period of interest.

4. On the page that loads, do the following:

   1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
   2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
   3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

1. Go to the Google Cloud console.

   Open Google Cloud console

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Click Activate Cloud Shell.

4. Obtain GKE credentials for your cluster by running the following commands.

   For zonal clusters, run:

     gcloud container clusters get-credentials cluster_name --zone location --project resource.project_display_name
   

   For regional clusters, run:

     gcloud container clusters get-credentials cluster_name --region location --project resource.project_display_name
   

5. Retrieve the added library by running:

     kubectl cp Pod_Namespace/Pod_Name: Added_Library_Fullpath -c Container_Name  local_file
   

   Replace local_file with a local file path to store the added library.

6. Connect to the container environment by running:

     kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh
   

   This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Ingress Tool Transfer, Shared Modules.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised container.
• Stop or delete the compromised container and replace it with a new container.

Malicious Script Executed

A machine learning model identified an executed bash script as malicious. Attackers can use bash to transfer tools and execute commands without binaries.

To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Malicious Script Detected finding as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • resource.name: the full resource name of the cluster, including the project number, location, and cluster name
   • resource.project_display_name: the name of the project that contains the asset

   • processes

     • binary: details about the interpreter that invoked the script
     • args: the arguments provided when invoking the script
     • script.contents: Prefix of the executed script's file contents, which can aid in your investigation; the contents might be truncated for performance reasons
     • script.sha256: the SHA-256 hash of script.contents
     • script.path: absolute path of the name of the script on disk; this attribute only appears for scripts written to disk, not for literal script execution, for example, bash -c

3. Click the Source Properties tab and note the following fields:

   • Pod_Namespace: the name of the Pod's Kubernetes namespace
   • Pod_Name: the name of the GKE Pod
   • Container_Name: the name of the affected container
   • Container_Image_Uri: the name of the container image being executed
   • VM_Instance_Name: the name of the GKE node where the Pod was executed

Step 2: Review cluster and node

1. In the Google Cloud console, go to the Kubernetes clusters page.

   Go to Kubernetes clusters

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

1. In the Google Cloud console, go to the Kubernetes Workloads page.

   Go to Kubernetes Workloads

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Set Select time range to the period of interest.

4. On the page that loads, do the following:

   1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
   2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
   3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

1. In the Google Cloud console, go to the Kubernetes clusters page.

   Go to Kubernetes clusters

2. Click the name of the cluster shown in resource.labels.cluster_name.

3. On the Clusters page, click Connect, and then click Run in Cloud Shell.

   Cloud Shell launches and adds commands for the cluster in the terminal.

4. Press enter and, if the Authorize Cloud Shell dialog appears, click Authorize.

5. Connect to the container environment by running the following command:

     kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh
   

   This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Command and Scripting Interpreter, Ingress Tool Transfer.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised container.
• Stop or delete the compromised container and replace it with a new container.

Malicious URL Observed

Container Threat Detection observed a malicious URL in the argument list of an executable process. Attackers can load malware or malicious libraries through malicious URLs.

To respond to this finding, perform the following steps.

Step 1: Review finding details

1. Open a Malicious URL Observed finding as directed in Reviewing findings.

2. In the finding details, on the Summary tab, note the values of following fields:

   • Under What was detected:
     • URI: the malicious URI observed.
     • Added binary: the full path of the process binary that received the arguments that contain the malicious URL.
     • Arguments: the arguments provided when invoking the process binary.
     • Environment variables: the environment variables that were in effect when the process binary was invoked.
     • Containers: the name of the container.
     • Kubernetes pods: the pod name and namespace.
   • Under Affected resource:
     • Resource display name: the name of the affected resource.
     • Resource full name: the full resource name of the cluster. The full resource name includes the following information:
       • The project that contains the cluster: projects/PROJECT_ID
       • The location in which the cluster is located: either zone/ZONE or locations/LOCATION
       • The name of the cluster: projects/CLUSTER_NAME

3. On the JSON tab, in the sourceProperties attribute, note the value of the VM_Instance_Name property.

Step 2: Review cluster and node

1. In the Google Cloud console, go to the Kubernetes clusters page.

   Go to Kubernetes clusters

2. On the Google Cloud console toolbar, select the project that appears in Resource full name (resource.name), if necessary. The project name appears after /projects/ in the full resource name.

3. Click on the cluster name that you noted in Resource display name (resource.display_name) of the finding summary. The Clusters page opens.

4. In the Metadata section on the **Cluster details page, note any of the user-defined information that might be helfpul in resolving the threat, such as information that identifies the cluster owner.

5. Click the Nodes tab.

6. From the listed nodes, select the node that matches the value of VM_Instance_Name that you noted in the finding JSON earlier.

7. On the Details tab of the Node details page, in the Annotations section, note the value of the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

1. In the Google Cloud console, go to the Kubernetes Workloads page.

   Go to Kubernetes Workloads

2. On the Google Cloud console toolbar, select the project that you noted in the Resource full name (resource.name) of the cluster in the finding summary, if necessary.

3. Click Show system workloads.

4. Filter the list of workloads by the cluster name that you noted in Resource full name (resource.name)of the finding summary and, if necessary, the pod Namespace (kubernetes.pods.ns) that you noted.

5. Click on the workload name that matches the value of the VM_Instance_Name property that you noted in the finding JSON earlier. The Pod details page opens.

6. On the Pod details page, note any information about the Pod that might help you resolve the threat.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project that appears in Resource full name (resource.name), if necessary.

3. Set Select time range to the period of interest.

4. On the page that loads, do the following:

   1. Find Pod logs for your pod (kubernetes.pods.name) by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="PROJECT_ID"
      • resource.labels.location="LOCATION"
      • resource.labels.cluster_name="CLUSTER_NAME"
      • resource.labels.namespace_name="NAMESPACE_NAME"
      • resource.labels.pod_name="POD_NAME"
   2. Find cluster audit logs by using the following filter:
      • logName="projects/PROJECT_NAME/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="PROJECT_ID"
      • resource.labels.location="LOCATION_OR_ZONE"
      • resource.labels.cluster_name="CLUSTER_NAME/var>"
      • POD_NAME
   3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="INSTANCE_ID"

Step 5: Investigate the running container

If the container is still running, it might be possible to investigate the container environment directly.

1. In the Google Cloud console, go to the Kubernetes clusters page.

   Go to Kubernetes clusters

2. Click the name of the cluster shown in resource.labels.cluster_name.

3. On the Clusters page, click Connect, and then click Run in Cloud Shell.

   Cloud Shell launches and adds commands for the cluster in the terminal.

4. Press enter and, if the Authorize Cloud Shell dialog appears, click Authorize.

5. Connect to the container environment by running the following command:

     kubectl exec --namespace=POD_NAMESPACE -ti POD_NAME -c CONTAINER_NAME -- /bin/sh
   

   Replace CONTAINER_NAME with the name of the container that you noted in the finding summary earlier.

   This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

1. Check Safe Browsing site status to get details on why the URL is classified as malicious.
2. Review MITRE ATT&CK framework entries for this finding type: Ingress Tool Transfer.
3. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised container.
• Stop or delete the compromised container and replace it with a new container.

Reverse Shell

A process started with stream redirection to a remote connected socket. Spawning a network-connected shell can allow an attacker to perform arbitrary actions after a limited initial compromise. To respond to this finding, do the following:

Step 1: Review finding details

1. Open a Reverse Shell finding as directed in Reviewing findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • resource.name: the full resource name of the cluster
   • resource.project_display_name: the name of the project that contains the asset

   • processes

     • binary.path: the absolute path of the process started with stream redirection to a remote socket
     • args: the arguments provided when invoking the process binary

3. Click the Source Properties tab and note the following fields:

   • Pod_Namespace: the name of the Pod's Kubernetes namespace
   • Pod_Name: the name of the GKE Pod
   • Container_Name: the name of the affected container
   • Container_Image_Uri: the name of the container image being executed
   • Reverse_Shell_Stdin_Redirection_Dst_Ip: the remote IP address of the connection
   • Reverse_Shell_Stdin_Redirection_Dst_Port: the remote port
   • Reverse_Shell_Stdin_Redirection_Src_Ip: the local IP address of the connection
   • Reverse_Shell_Stdin_Redirection_Src_Port: the local port
   • VM_Instance_Name: the name of the GKE node where the Pod executed

Step 2: Review cluster and node

1. In the Google Cloud console, go to the Kubernetes clusters page.

   Go to Kubernetes clusters

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

1. In the Google Cloud console, go to the Kubernetes Workloads page.

   Go to Kubernetes Workloads

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Set Select time range to the period of interest.

4. On the page that loads, do the following:

   1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
   2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
   3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

1. Go to the Google Cloud console.

   Open Google Cloud console

2. On the Google Cloud console toolbar, select the project listed in resource.project_display_name, if necessary.

3. Click Activate Cloud Shell.

4. Obtain GKE credentials for your cluster by running the following commands.

   For zonal clusters, run:

     gcloud container clusters get-credentials cluster_name --zone location --project resource.project_display_name
   

   For regional clusters, run:

     gcloud container clusters get-credentials cluster_name --region location --project resource.project_display_name
   

5. Launch a shell within the container environment by running:

     kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh
   

   This command requires the container to have a shell installed at /bin/sh.

   To view all processes running in the container, run the following command in the container shell:

     ps axjf
   

   This command requires the container to have /bin/ps installed.

Step 6: Research attack and response methods

1. Review MITRE ATT&CK framework entries for this finding type: Command and Scripting Interpreter, Ingress Tool Transfer.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

• Contact the owner of the project with the compromised container.
• Stop or delete the compromised container and replace it with a new container.

VM Threat Detection response

To learn more about VM Threat Detection, see VM Threat Detection overview.

Execution: Cryptocurrency Mining Hash Match

VM Threat Detection detected cryptocurrency mining activities by matching memory hashes of running programs against known memory hashes of cryptocurrency mining software.

To respond to these findings, do the following:

Step 1: Review finding details

1. Open an Execution: Cryptocurrency Mining Hash Match finding, as directed in Review findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • resourceName: the full resource name of the affected VM instance, including the ID of the project that contains it

   • processes

     VM Threat Detection can recognize kernel builds from major Linux distributions. If it can recognize the affected VM's kernel build, it can identify the application's process details and populate the processes field of the finding. If VM Threat Detection can't regognize the kernel—for example, if the kernel is custom built—the finding's processes field isn't populated.

     • binary.path: the absolute path of the process
     • name: the name of the process running in the VM instance that is associated with the detected signature matches
     • args: the arguments provided when invoking the process binary

   • indicator

     • signatures: a list of signatures that matched the process

       • memory_hash_signature: a signature corresponding to memory page hashes
         • binary_family: the cryptocurrency application that was detected
         • detections
           • binary: the name of the cryptocurrency application's binary—for example, linux--x86-64_ethminer_0.19.0_alpha.0_cuda10.0
           • percent_pages_matched: the percentage of pages in memory that match pages in known cryptocurrency applications in the page-hash database

Step 2: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project that contains the VM instance, as specified in resourceName.

3. Check the logs for signs of intrusion on the affected VM instance. For example, check for suspicious or unknown activities and signs of compromised credentials.

Step 3: Review permissions and settings

1. In the Google Cloud console, go to the Dashboard page.

   Go to the Dashboard

2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project specified in resourceName.

3. Navigate to the Resources card and click Compute Engine.

4. Click the VM instance that matches resourceName. Review instance details, including network and access settings.

Step 4: Research attack and response methods

1. Review MITRE ATT&CK framework entries for Execution.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

To assist with detection and removal, use an endpoint detection and response solution.

1. Contact the owner of the VM.

2. Confirm whether the application is a mining application:

   • If the detected application's process name and binary path are available, consider the values in the processes fields and subfields in your investigation.

   • If the process details aren't available, check if the binary name from the memory hash signature can provide clues. Consider a binary called linux-x86-64_xmrig_2.14.1. You can use the grep command to search for notable files in storage. Use a meaningful portion of the binary name in your search pattern, in this case, xmrig. Examine the search results.

   • Examine the running processes, especially the processes with high CPU usage, to see if there are any that you don't recognize. Determine whether the associated applications are miner applications.

   • Search the files in storage for common strings that mining applications use, such as btc.com, ethminer, xmrig, cpuminer, and randomx. For more examples of strings you can search for, see Software names and YARA rules and the related documentation for each software listed.

3. If you determine that the application is a miner application, and its process is still running, terminate the process. Locate the application's executable binary in the VM's storage, and delete it.

4. If necessary, stop the compromised instance and replace it with a new instance.

Execution: Cryptocurrency Mining YARA Rule

VM Threat Detection detected cryptocurrency mining activities by matching memory patterns, such as proof-of-work constants, known to be used by cryptocurrency mining software.

To respond to these findings, do the following:

Step 1: Review finding details

1. Open an Execution: Cryptocurrency Mining YARA Rule finding, as directed in Review findings.

2. In the Finding Details pane, on the Attributes tab, note the following fields.

   Note: If you opted to upgrade to the Findings Workflow Improvements (in Preview), you can find the attributes and source properties in the JSON tab of the finding's details pane.
   • resourceName: the full resource name of the affected VM instance, including the ID of the project that contains it

   • processes

     VM Threat Detection can recognize kernel builds from major Linux distributions. If it can recognize the affected VM's kernel build, it can identify the application's process details and populate the processes field of the finding. If VM Threat Detection can't regognize the kernel—for example, if the kernel is custom built—the finding's processes field isn't populated.

     • binary.path: the absolute path of the process
     • name: the name of the process running in the VM instance that is associated with the detected signature matches
     • args: the arguments provided when invoking the process binary

   • indicator

     • signatures: a list of signatures that matched the process

       • yara_rule_signature: a signature corresponding to a YARA rule
         • yara_rule_name: the rule triggered for YARA detectors

Step 2: Check logs

1. In the Google Cloud console, go to Logs Explorer.

   Go to Logs Explorer

2. On the Google Cloud console toolbar, select the project that contains the VM instance, as specified in resourceName.

3. Check the logs for signs of intrusion on the affected VM instance. For example, check for suspicious or unknown activities and signs of compromised credentials.

Step 3: Review permissions and settings

1. In the Google Cloud console, go to the Dashboard page.

   Go to the Dashboard

2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project specified in resourceName.

3. Navigate to the Resources card and click Compute Engine.

4. Click the VM instance that matches resourceName. Review instance details, including network and access settings.

Step 4: Research attack and response methods

1. Review MITRE ATT&CK framework entries for Execution.
2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization, but may impact operations. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

To assist with detection and removal, use an endpoint detection and response solution.

1. Contact the owner of the VM.

2. Confirm whether the application is a mining application:

   • If the detected application's process name and binary path are available, consider the values in the processes fields and subfields in your investigation.

   • Examine the running processes, especially the processes with high CPU usage, to see if there are any that you don't recognize. Determine whether the associated applications are miner applications.

   • Search the files in storage for common strings that mining applications use, such as btc.com, ethminer, xmrig, cpuminer, and randomx. For more examples of strings you can search for, see Software names and YARA rules and the related documentation for each software listed.

3. If you determine that the application is a miner application, and its process is still running, terminate the process. Locate the application's executable binary in the VM's storage, and delete it.

4. If necessary, stop the compromised instance and replace it with a new instance.

Fix related vulnerabilities

To help keep threats from reoccurring, review and fix vulnerabilities identified by Security Health Analytics.

1. In the Security Command Center dashboard, go to the Assets page.

   Go to Assets

2. Next to View by, select Project.

3. In the Project list, select the project that contains the threat finding.

4. In the Filter box, enter "securityCenterProperties.resourceType:google.compute.Instance". The table lists instances in the project.

5. Under the resourceProperties.name column, select the instance that contains the finding.

6. In the pane that loads, under Additional Information, select the Findings tab. If findings are listed, review and follow the remediation guidance for each finding.

In addition to Google-provided indicators of compromise, users who are customers of Palo Alto Networks can integrate Palo Alto Networks' AutoFocus Threat Intelligence with Event Threat Detection. AutoFocus is a threat intelligence service that provides information about network threats. To learn more, visit the AutoFocus page in Google Cloud console.

Remediating threats

Remediating Event Threat Detection and Container Threat Detection findings is not as simple as fixing misconfigurations and vulnerabilities identified by Security Command Center.

Misconfigurations and compliance violations identify weaknesses in resources that could be exploited. Typically, misconfigurations have known, easily implemented fixes, like enabling a firewall or rotating an encryption key.

Threats differ from vulnerabilities in that they are dynamic and indicate a possible active exploit against one or more resources. A remediation recommendation might not be effective in securing your resources because the exact methods used to achieve the exploit might not be known.

For example, an Added Binary Executed finding indicates that an unauthorized binary was launched in a container. A basic remediation recommendation might advise you to quarantine the container and delete the binary, but that might not resolve the underlying root cause that allowed the attacker access to execute the binary. You need to find out how the container image was corrupted to fix the exploit. Determining whether the file was added through a misconfigured port or by some other means requires a thorough investigation. An analyst with expert-level knowledge of your system might need to review it for weaknesses.

Bad actors attack resources using different techniques, so applying a fix for a specific exploit might not be effective against variations of that attack. For example, in response to a Brute Force: SSH finding, you might lower permission levels for some user accounts to limit access to resources. However, weak passwords might still provide an attack path.

The breadth of attack vectors makes it difficult to provide remediation steps that work in all situations. Security Command Center's role in your cloud security plan is to identify impacted resources in near-real time, tell you what threats you face, and provide evidence and context to aid your investigations. However, your security personnel must use the extensive information in Security Command Center findings to determine the best ways to remediate issues and secure resources against future attacks.

What's next

• See Event Threat Detection overview to learn more about the service and the threats it detects.

• See Container Threat Detection overview to learn how the service works.

• See VM Threat Detection overview to learn more about the service and the threats it detects.