Investigating and responding to threats

This topic offers informal guidance to help you investigate and respond to threats, and use additional resources to add context to Security Command Center findings. Following these steps helps you understand what happened during a potential attack and develop possible responses for affected resources.

The techniques on this page are not guaranteed to be effective against any previous, current, or future threats you face. See Remediating threats to understand why Security Command Center does not provide official remediation guidance for threats.

Before you begin

You need adequate Identity and Access Management (IAM) roles to view or edit findings and logs, and modify Google Cloud resources. If you encounter access errors in Security Command Center, ask your administrator for assistance and see Access control to learn about roles. To resolve resource errors, read documentation for affected products.

Understanding threat findings

Event Threat Detection produces security findings by matching events in your Cloud Logging and Google Workspace log streams to known indicators of compromise (IoC). IoCs, developed by internal Google security sources, identify potential vulnerabilities and attacks. Event Threat Detection also detects threats by identifying known adversarial tactics, techniques, and procedures in your logging stream, and by detecting deviations from historically observed behavior of your organization.

Container Threat Detection generates findings by collecting and analyzing low-level observed behavior in the guest kernel of containers.

Findings are written to Security Command Center and Cloud Logging.

Reviewing findings

To review threat findings, do the following:

  1. In the Google Cloud Console, go to the Security Command Center Findings page.

    Go to Findings

  2. If necessary, select your Google Cloud project or organization.

  3. On the findings page, next to View by, click Source type.

  4. Select Event Threat Detection or Container Threat Detection. The table is populated with findings for the source you selected.

  5. To view details for a specific finding, click the finding name under category in the table. The Finding Details panel opens.

Findings provide the names and numeric identifiers of resources involved in an incident, along with environment variables and asset properties. You can use that information to quickly isolate affected resources and determine the potential scope of an event.

Threat findings also contain links to external resources to aid in your investigations. These resources include:

  • MITRE ATT&CK framework entries. The framework explains techniques for attacks against cloud resources and provides remediation guidance.
  • VirusTotal, an Alphabet-owned service that provides context on potentially malicious files, URLs, domains, and IP addresses.

The following sections outline potential responses to threat findings.

Event Threat Detection responses

To learn more about Event Threat Detection, see how Event Threat Detection works.

Exfiltration: BigQuery Data Exfiltration

Data exfiltration from BigQuery is detected by examining audit logs for three scenarios:

  • A resource is saved outside of your organization.
  • VPC Service Controls blocks a copy operation.
  • An attempt is made to access BigQuery resources that are protected by VPC Service Controls.

To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an Exfiltration: BigQuery Data Exfiltration finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • properties
      • projectId: the finding involves an asset in this project
      • jobLink: the link to the BigQuery job that exfiltrated data
      • Query: the SQL query run on the BigQuery dataset
      • SourceTables: the tables from which data was exfiltrated
      • DestinationTables: the tables where exfiltrated data was stored
      • userEmail: the account used to exfiltrate the data
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the IAM page .

    Go to IAM

  2. If necessary, select the project listed in projectID.

  3. On the page that appears, in the Filter box, enter the email address listed inuserEmail and check what permissions are assigned to the account.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. Find admin activity logs related to BigQuery jobs by using the following filters:
    • protoPayload.methodName="Jobservice.insert"
    • protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service: Exfiltration to Cloud Storage.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type on the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

Brute Force: SSH

Detection of successful brute force of SSH on a host. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Brute Force: SSH finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • sourceLogId: the project ID and timestamp to identify the log entry
      • projectId: the project that contains the finding
    • Attempts: the number of login attempts
      • sourceIP: the IP address that launched the attack
      • username: the account that logged in
      • vmName: the name of the virtual machine
      • authResult: the SSH authentication result
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Review permissions and settings

  1. In the Google Cloud Console, go to the Dashboard.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in projectId.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the name and zone in vmName. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules on port 22.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, find VPC Flow Logs related to srcIP by using the following filter:
    • logName="projects/projectId/logs/syslog"
    • labels."compute.googleapis.com/resource_name"="vmName"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Local Accounts.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the successful brute force attempt. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and desponse solution.
  • Consider disabling SSH access to the VM. This step could interrupt authorized access to the VM, so consider the needs of your organization before you proceed.
  • Only use SSH authentication with authorized keys.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the quantity of information, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.

Cryptomining

Malware is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known command and control domains and IP addresses. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Cryptomining: Bad IP finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • Properties_ip_0: the suspected cryptomining IP address
    • properties_sourceInstance: the affected Compute Engine VM instance
    • properties_project_id: the project for the Compute Engine instance

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in properties_project_id.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches properties_sourceInstance. Investigate the potentially compromised instance for malware.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Cloud Console toolbar, select your project.

  3. On the page that loads, find VPC Flow Logs related to Properties_ip_0 by using the following filter:

    • logName="projects/properties_project_id/logs/compute.googleapis.com%2Fvpc_flows"
    • (jsonPayload.connection.src_ip="Properties_ip_0" OR jsonPayload.connection.dest_ip="Properties_ip_0")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Resource Hijacking.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project containing malware. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and response solution.
  • If necessary, stop the compromised instance and replace it with a new instance.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.

Leaked credentials

This finding is generated when Google Cloud service account credentials are accidentally leaked online or compromised. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an account_has_leaked_credentials finding, as directed in Reviewing finding details.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • Compromised_account: the potentially compromised service account
    • Project_identifier: the project that contains the potentially leaked account credentials
    • URL: the link to the GitHub repository

Step 2: Review project and service account permissions

  1. In the Cloud Console, go to the IAM page.

    Go to IAM

  2. If necessary, select the project listed in Project_identifier.

  3. On the page that appears, in the Filter box, enter the account name listed in Compromised_account and check assigned permissions.

  4. In the Cloud Console, go to the Service Accounts page.

    Go to Service Accounts

  5. On the page that appears, in the Filter box, enter the name of the compromised service account and check the service account's keys and key creation dates.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Cloud Console toolbar, select your project.

  3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:

    • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
    • protoPayload.methodName="SetIamPolicy"
    • resource.type="gce_instance" AND log_name="projects/Project_identifier/logs/cloudaudit.googleapis.com%2Factivity"
    • protoPayload.methodName="InsertProjectOwnershipInvite"
    • protoPayload.authenticationInfo.principalEmail="Compromised_account"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Cloud Accounts.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with leaked credentials. Following these suggested steps might impact operations.
  • Consider deleting the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access. Before proceeding, your security team should identify all impacted resources and work with resource owners to ensure business continuity.
  • Work with your security team to identify unfamiliar resources, including Compute Engine instances, snapshots, service accounts, and IAM users. Delete resources not created with authorized accounts.
  • Respond to notifications from Google Cloud Support.
  • To limit who can create service accounts, use the Organization Policy Service.
  • To identify and fix overly permissive roles, use IAM Recommender.
  • Open the URL link and delete the leaked credentials. Gather more information about the compromised account and contact the owner.

Malware

Malware is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known command and control domains and IP addresses. To respond to these findings, do the following:

Step 1: Review finding details

  1. Open a Malware: Bad IP finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • sourceLogId
      • projectId: the project that contains the finding
    • ipConnection
      • srcIP: a known malware command and control IP address
      • srcPort: the source port of the connection
      • destIP: the target IP address of the malware
      • destPort: the destination port of the connection
    • InstanceDetails: the resource address for the Compute Engine instance
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • virustotalIndicatorQueryUI: link to the VirusTotal analysis page
      • cloudLoggingQueryURI: link to the Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in projectId.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the name and zone in instanceDetails. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, find VPC Flow Logs related to the IP address in srcIP by using the following filter:
    • logName="projects/projectId/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="srcIP" OR jsonPayload.connection.dest_ip="destIP")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Dynamic Resolution and Command and Control.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. Check flagged URLs and domains on VirusTotal by clicking the link in virustotalIndicatorQueryUI. VirusTotal is an Alphabet-owned service that provides context on potentially malicious files, URLs, domains, and IP addresses.
  4. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project containing malware. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and response solution.
  • To track activity and vulnerabilities that allowed the insertion of malware, check audit logs and syslogs associated with the compromised instance.
  • If necessary, stop the compromised instance and replace it with a new instance.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.
  • To control access and use of VM images, use Shielded VM and Trusted Images IAM policy.

Outgoing DoS

Event Threat Detection detects potential use of an instance to launch a denial of service (DoS) attack by analyzing VPC Flow Logs. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Malware: Outgoing DoS finding as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • sourceInstanceDetails: the affected Compute Engine VM instance
    • ipConnection
      • srcIP: a known malware command and control IP address
      • srcPort: the source port of the connection
      • destIP: the target IP address of the malware
      • destPort: the destination port of the connection
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project ID listed in sourceInstanceDetails.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the instance name and zone in sourceInstanceDetails. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, find VPC Flow Logs related to the IP address in srcIP by using the following filter:
    • logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="srcIP" OR jsonPayload.connection.dest_ip="destIP")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Network Denial of Service.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project experiencing outgoing DoS traffic. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and response solution.
  • To track activity and vulnerabilities that allowed the insertion of malware, check audit logs and syslogs associated with the compromised instance.
  • If necessary, stop the compromised instance and replace it with a new instance.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.
  • To control access and use of VM images, use Shielded VM and Trusted Images IAM policy.

Persistence: IAM Anomalous Grant

Audit logs are examined to detect the addition of IAM (IAM) role grants that might be considered suspicious. The following are examples of anomalous grants:

  • Inviting a gmail.com user as a project owner from the Google Cloud Console
  • A service account granting sensitive permissions
  • A custom role granting sensitive permissions
  • A service account added from outside your organization

To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Persistence: IAM Anomalous Grant finding as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • projectId: the project that contains the finding
    • principalEmail: email address for the user or service account that assigned the role
    • bindingDeltas
      • Action: the action taken by the user
      • Role: the role assigned to the user
      • member: the email address of the user that received the role
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • virustotalIndicatorQueryUI: link to the VirusTotal analysis page
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, look for new or updated IAM resources using the following filters:
    • protoPayload.methodName="SetIamPolicy"
    • protoPayload.methodName="google.iam.admin.v1.UpdateRole"
    • protoPayload.methodName="google.iam.admin.v1.CreateRole"

Step 3: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Valid Accounts: Cloud Accounts.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 4: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised account. Following these suggested steps might impact operations.
  • Delete the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access.
  • Delete project resources created by unauthorized accounts, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.
  • To restrict adding gmail.com users, use the Organization Policy.
  • To identify and fix overly permissive roles, use IAM Recommender.

Persistence: New Geography

An IAM user or service account is accessing Google Cloud from an anomalous location, based on the geolocation of the requesting IP address.

Step 1: Review finding details

  1. Open a Persistence: New Geography finding, as directed in Reviewing finding details earlier on this page.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • principalEmail: the potentially compromised user account
    • projectId: the project that contains the potentially compromised user account
    • callerIp: the external IP address
    • anomalousLocation: the estimated current location of the user
    • typicalGeolocations: the locations where the user usually accesses Google Cloud resources
    • notSeenInLast: the time period used to establish a baseline for normal behavior
    • gcpResourceName: the affected resource
    • cloudLoggingQueryURI: the link to Cloud Logging entries
    • mitreUri: the link to the MITRE ATT&CK framework entry

Step 2: Review project and account permissions

  1. In the Cloud Console, go to the IAM page.

    Go to IAM

  2. If necessary, select the project listed in projectId.

  3. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check granted roles.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. If necessary, select your project.
  3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:
    • protoPayload.methodName="SetIamPolicy"
    • protoPayload.methodName="google.iam.admin.v1.UpdateRole"
    • protoPayload.methodName="google.iam.admin.v1.CreateRole"
    • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Cloud Accounts.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised account. Following these suggested steps might impact operations.
  • Review the anomalousLocation, typicalGeolocations, and notSeenInLast fields to verify whether the access is abnormal and if the account has been compromised.
  • Delete project resources created by unauthorized accounts, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.
  • To restrict the creation of new resources to specific regions, see Restricting Resource Locations.
  • To identify and fix overly permissive roles, use IAM Recommender.

Persistence: New User Agent

An IAM service account is accessing Google Cloud using suspicious software, as indicated by an anomalous user agent.

Step 1: Review finding details

  1. Open a Persistence: New User Agent finding, as directed in Reviewing finding details earlier on this page.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • principalEmail: the potentially compromised service account
    • projectId: the project that contains the potentially compromised service account
    • callerUserAgent: the anomalous user agent
    • anomalousSoftwareClassification: the type of software
    • notSeenInLast: the time period used to establish a baseline for normal behavior
    • cloudLoggingQueryURI: the link to Cloud Logging entries
    • mitreUri: the link to the MITRE ATT&CK framework entry

Step 2: Review project and account permissions

  1. In the Cloud Console, go to the IAM page.

    Go to IAM

  2. If necessary, select the project listed in projectId.

  3. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check granted roles.

  4. In the Cloud Console, go to the Service Accounts page.

    Go to Service Accounts

  5. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check the service account's keys and key creation dates.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. If necessary, select your project.
  3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:
    • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
    • protoPayload.methodName="SetIamPolicy"
    • protoPayload.methodName="google.iam.admin.v1.UpdateRole"
    • protoPayload.methodName="google.iam.admin.v1.CreateRole"
    • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Cloud Accounts.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised account. Following these suggested steps might impact operations.
  • Review the anomalousSoftwareClassification, callerUserAgent, and behaviorPeriod fields to verify whether the access is abnormal and if the account has been compromised.
  • Delete project resources created by unauthorized accounts, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.
  • To restrict the creation of new resources to specific regions, see Restricting Resource Locations.
  • To identify and fix overly permissive roles, use IAM Recommender.

Phishing

Phishing is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known phishing domains and IP addresses. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a resource_used_for_phishing finding as directed in Reviewing finding details.

  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:

    • resourceName: the project that contains the finding
    • urls: the phishing URL
    • vm_host_and_zone_names: the name of the Compute Engine VM that is hosting the phishing URL
    • vm_ips: the IP address of the Compute Engine VM that is hosting the phishing URL

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in resourceName.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the name and zone in InstanceDetails. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Cloud Console toolbar, select your project.

  3. On the page that loads, find VPC Flow Logs related to vm_ips by using the following filter:

    • logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="vm_ips" OR jsonPayload.connection.dest_ip="vm_ips")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Phishing.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised VM, vm_host_and_zone_names. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an endpoint detection and desponse solution.
  • If necessary, stop the compromised instance and replace it with a new instance.

Service account self-investigation

A service account credential is being used to investigate the roles and permissions associated with that same service account. This finding indicates that service account credentials might be compromised and immediate action should be taken.

Step 1: Review finding details

  1. Open a Discovery: Service Account Self-Investigation finding, as directed in Reviewing finding details earlier on this page.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • principalEmail: the potentially compromised service account
    • projectId: the project that contains the potentially leaked account credentials
    • callerIp: the internal or external IP address
    • cloudLoggingQueryURI: the link to Cloud Logging entries
    • mitreUri: the link to the MITRE ATT&CK framework entry
    • severity: the risk level assigned to the finding; severity is HIGH if the API call that triggered this finding was unauthorized—the service account doesn't have permission to query its own IAM permissions with the projects.getIamPolicy API.

Step 2: Review project and service account permissions

  1. In the Cloud Console, go to the IAM page.

    Go to IAM

  2. If necessary, select the project listed in projectId.

  3. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check assigned permissions.

  4. In the Cloud Console, go to the Service Accounts page.

    Go to Service Accounts

  5. On the page that appears, in the Filter box, enter the name of the compromised service account and check the service account's keys and key creation dates.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. If necessary, select your project.
  3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:
    • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
    • protoPayload.methodName="SetIamPolicy"
    • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Permission Groups Discovery: Cloud Groups.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised account. Following these suggested steps might impact operations.
  • Delete the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access.
  • Delete project resources created by the compromised account, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.

Google Workspace detector responses

If you share your Google Workspace logs with Cloud Logging, Event Threat Detection generates findings for several Google Workspace threats.

Event Threat Detection enriches log events and writes findings to Security Command Center. The following table contains Google Workspace threats, relevant MITRE ATT&CK framework entries, and details about the events that trigger findings. You can also check logs using specific filters, and combine all of the information to respond to Google Workspace threats.

Initial Access: Disabled Password Leak

Description Actions
A member's account is disabled because a password leak was detected. Reset passwords for affected accounts, and advise members to use strong, unique passwords for corporate accounts.

Check logs using the following filters:

protopayload.resource.labels.service="login.googleapis.com"

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access

Replace ORGANIZATION_ID with your organization ID.

Research events that trigger this finding:

Initial Access: Suspicious Login Blocked

Description Actions
A suspicious login to a member's account was detected and blocked. This account might be targeted by adversaries. Ensure the user account follows your organization's security guidelines for strong passwords and multifactor authentication.

Check logs using the following filters:

protopayload.resource.labels.service="login.googleapis.com"

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access

Replace ORGANIZATION_ID with your organization ID.

Research events that trigger this finding:

Initial Access: Account Disabled Hijacked

Description Actions
A member's account was suspended due to suspicious activity. This account was hijacked. Reset the account password and encourage users to create strong, unique passwords for corporate accounts.

Check logs using the following filters:

protopayload.resource.labels.service="login.googleapis.com"

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access

Replace ORGANIZATION_ID with your organization ID.

Research events that trigger this finding:

Impair Defenses: Two Step Verification Disabled

Description Actions
A member disabled 2-step verification. Verify if the user intended to disable 2-step verification. If your organization requires 2-step verification, ensure that the user enables it promptly.

Check logs using the following filters:

protopayload.resource.labels.service="login.googleapis.com"

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access

Replace ORGANIZATION_ID with your organization ID.

Research events that trigger this finding:

Initial Access: Government Based Attack

Description Actions
Government-backed attackers might have tried to compromise a member account or computer. This account might be targeted by adversaries. Ensure the user account follows your organization's security guidelines for strong passwords and multifactor authentication.

Check logs using the following filters:

protopayload.resource.labels.service="login.googleapis.com"

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Fdata_access

Replace ORGANIZATION_ID with your organization ID.

Research events that trigger this finding:

Persistence: SSO Enablement Toggle

Description Actions
The Enable SSO (single sign-on) setting on the admin account was disabled. SSO settings for your organization were changed. Verify whether the change was done intentionally by a member or if it was implemented by an adversary to introduce new access to your organization.

Check logs using the following filters:

protopayload.resource.labels.service="admin.googleapis.com"

protopayload.metadata.event.parameter.value=DOMAIN_NAME

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity

Replace the following:

  • DOMAIN_NAME: the domainName listed in the finding
  • ORGANIZATION_ID: your organization ID

Research events that trigger this finding:

Persistence: SSO Settings Changed

Description Actions
The SSO settings for the admin account were changed. SSO settings for your organization were changed. Verify whether the change was done intentionally by a member or if it was implemented by an adversary to introduce new access to your organization.

Check logs using the following filters:

protopayload.resource.labels.service="admin.googleapis.com"

protopayload.metadata.event.parameter.value=DOMAIN_NAME

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity

Replace the following:

  • DOMAIN_NAME: the domainName listed in the finding
  • ORGANIZATION_ID: your organization ID

Research events that trigger this finding:

Impair Defenses: Strong Authentication Disabled

Description Actions
2-step verification was disabled for the organization. 2-step verification is no longer required for your organization. Find out if this was an intentional policy change by an administrator, or if this is an attempt by an adversary to make account hijacking easier.

Check logs using the following filters:

protopayload.resource.labels.service="admin.googleapis.com"

logName="organizations/ORGANIZATION_ID/logs/cloudaudit.googleapis.com%2Factivity

Replace ORGANIZATION_ID with your organization ID.

Research events that trigger this finding:

Respond to Google Workspace threats

If you're a Google Workspace Admin, you can use the service's security tools to resolve these threats:

The tools include alerts, a security dashboard, security recommendations, and can help you investigate and remediate threats.

If you're not a Google Workspace Admin, do the following:

Container Threat Detection responses

To learn more about Container Threat Detection, see how Container Threat Detection works.

Added Binary Executed

A binary that was not part of the original container image was executed. Attackers commonly install exploitation tooling and malware after the initial compromise. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an Added Binary Executed finding as directed in Reviewing findings.
  2. In the Finding Details panel, click the Attributes tab and note the following fields:
    • resource.name: the full resource name of the cluster including the project number, location, and cluster name
    • resource.project_display_name: the name of the project that contains the cluster
  3. Click the Source Properties tab and note the following fields:
    • Process_Binary_Fullpath: the full path of the added binary
    • Process_Arguments: the arguments provided when invoking the added binary
    • Pod_Namespace: the name of the Pod's Kubernetes namespace
    • Pod_Name: the name of the GKE Pod
    • Container_Name: the name of the affected container
    • Container_Image_Uri: the name of the container image being deployed
    • VM_Instance_Name: the name of the GKE node where the Pod executed

Step 2: Review cluster and node

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

  4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

  5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

  1. In the Cloud Console, go to the Kubernetes Workloads page.

    Go to Kubernetes Workloads

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

  4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Set Select time range to the period of interest.

  4. On the page that loads, do the following:

    1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
    2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
    3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

  1. Go to the Cloud Console.

    Open Cloud Console

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Click Activate Cloud Shell

  4. Obtain GKE credentials for your cluster by running the following commands.

    For zonal clusters, run:

      gcloud container clusters get-credentials cluster_name --zone location --project project_name

    For regional clusters, run:

      gcloud container clusters get-credentials cluster_name --region location --project project_name

    Replace the following:

    • cluster_name: the cluster listed in resource.labels.cluster_name
    • location: the location listed in resource.labels.location
    • project_name: the project name listed in resource.project_display_name

  5. Retrieve the added binary by running:

      kubectl cp Pod_Namespace/Pod_Name:Process_Binary_Fullpath -c Container_Name  local_file

    Replace local_file with a local file path to store the added binary.

  6. Connect to the container environment by running:

      kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh

    This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Ingress Tool Transfer, Native API.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised container. Following these suggested steps might impact operations.
  • Stop or delete the compromised container and replace it with a new container.

Added Library Loaded

A library that was not part of the original container image was loaded. Attackers might load malicious libraries into existing programs in order to bypass code execution protections and to hide malicious code. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an Added Library Loaded finding as directed in Reviewing findings.
  2. In the Finding Details panel, click the Attributes tab and take note of the following fields:
    • resource.name: the full resource name of the cluster
    • resource.project_display_name: the name of the project that contains the asset
  3. Click the Source Properties tab and note the following fields:
    • Added_Library_Fullpath: the full path of the added library
    • Process_Binary_Fullpath: the full path of the process binary that loaded the library
    • Process_Arguments: the arguments provided when invoking the process binary
    • Pod_Namespace: the name of the Pod's Kubernetes namespace
    • Pod_Name: the name of the GKE Pod
    • Container_Name: the name of the affected container
    • Container_Image_Uri: the name of the container image being executed
    • VM_Instance_Name: the name of the GKE node where the Pod executed

Step 2: Review cluster and node

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

  4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

  5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

  1. In the Cloud Console, go to the Kubernetes Workloads page.

    Go to Kubernetes Workloads

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

  4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Set Select time range to the period of interest.

  4. On the page that loads, do the following:

    1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
    2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
    3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

  1. Go to the Cloud Console.

    Open Cloud Console

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Click Activate Cloud Shell.

  4. Obtain GKE credentials for your cluster by running the following commands.

    For zonal clusters, run:

      gcloud container clusters get-credentials cluster_name --zone location --project resource.project_display_name

    For regional clusters, run:

      gcloud container clusters get-credentials cluster_name --region location --project resource.project_display_name

  5. Retrieve the added library by running:

      kubectl cp Pod_Namespace/Pod_Name: Added_Library_Fullpath -c Container_Name  local_file

    Replace local_file with a local file path to store the added library.

  6. Connect to the container environment by running:

      kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh

    This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Ingress Tool Transfer, Shared Modules.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised container. Following these suggested steps might impact operations.
  • Stop or delete the compromised container and replace it with a new container.

Malicious Script Executed

A machine learning model identified an executed bash script as malicious. Attackers can use bash to transfer tools and execute commands without binaries.

To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Malicious Script Detected finding as directed in Reviewing findings.
  2. In the Finding Details panel, click the Attributes tab and take note of the following fields:
    • resource.name: the full resource name of the cluster, including the project number, location, and cluster name
    • resource.project_display_name: the name of the project that contains the asset
  3. Click the Source Properties tab and note the following fields:
    • Script_Content: the content of the executed script, which can aid in your investigation; the script might be truncated for performance reasons
    • Script_SHA256: the SHA-256 hash of the script
    • Script_Filename: the name of the script on disk; this attribute only appears for scripts written to disk, not for literal script execution, for example, bash -c
    • Process_Binary_Fullpath: the interpreter that invoked the script, for example, /bin/bash or /bin/dash
    • Process_Arguments: the arguments provided when invoking the script; this attribute indicates whether the script was invoked as a literal or from disk
    • Pod_Namespace: the name of the Pod's Kubernetes namespace
    • Pod_Name: the name of the GKE Pod
    • Container_Name: the name of the affected container
    • Container_Image_Uri: the name of the container image being executed
    • VM_Instance_Name: the name of the GKE node where the Pod was executed

Step 2: Review cluster and node

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

  4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

  5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

  1. In the Cloud Console, go to the Kubernetes Workloads page.

    Go to Kubernetes Workloads

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

  4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Set Select time range to the period of interest.

  4. On the page that loads, do the following:

    1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
    2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
    3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. Click the name of the cluster shown in resource.labels.cluster_name.

  3. On the Clusters page, click Connect, and then click Run in Cloud Shell.

    Cloud Shell launches and adds commands for the cluster in the terminal.

  4. Press enter and, if the Authorize Cloud Shell dialog appears, click Authorize.

  5. Connect to the container environment by running the following command:

      kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh

    This command requires the container to have a shell installed at /bin/sh.

Step 6: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Command and Scripting Interpreter, Ingress Tool Transfer.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised container. Following these suggested steps might impact operations.
  • Stop or delete the compromised container and replace it with a new container.

Reverse Shell

A process started with stream redirection to a remote connected socket. Spawning a network-connected shell can allow an attacker to perform arbitrary actions after a limited initial compromise. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Reverse Shell finding as directed in Reviewing findings.
  2. In the Finding Details panel, click the Attributes tab and take note of the following fields:
    • resource.name: the full resource name of the cluster
    • resource.project_display_name: the name of the project that contains the asset
  3. Click the Source Properties tab and note the following fields:
    • Process_Binary_Fullpath: the full path of the process started with stream redirection to a remote socket
    • Process_Arguments: the arguments provided when invoking the process binary
    • Pod_Namespace: the name of the Pod's Kubernetes namespace
    • Pod_Name: the name of the GKE Pod
    • Container_Name: the name of the affected container
    • Container_Image_Uri: the name of the container image being executed
    • Reverse_Shell_Stdin_Redirection_Dst_Ip: the remote IP address of the connection
    • Reverse_Shell_Stdin_Redirection_Dst_Port: the remote port
    • Reverse_Shell_Stdin_Redirection_Src_Ip: the local IP address of the connection
    • Reverse_Shell_Stdin_Redirection_Src_Port: the local port
    • VM_Instance_Name: the name of the GKE node where the Pod executed

Step 2: Review cluster and node

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Select the cluster listed in resource.name. Note any metadata about the cluster and its owner.

  4. Click the Nodes tab. Select the node listed in VM_Instance_Name.

  5. Click the Details tab and note the container.googleapis.com/instance_id annotation.

Step 3: Review Pod

  1. In the Cloud Console, go to the Kubernetes Workloads page.

    Go to Kubernetes Workloads

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Filter on the cluster listed in resource.name and the Pod namespace listed in Pod_Namespace, if necessary.

  4. Select the Pod listed in Pod_Name. Note any metadata about the Pod and its owner.

Step 4: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Set Select time range to the period of interest.

  4. On the page that loads, do the following:

    1. Find Pod logs for Pod_Name by using the following filter:
      • resource.type="k8s_container"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • resource.labels.namespace_name="Pod_Namespace"
      • resource.labels.pod_name="Pod_Name"
    2. Find cluster audit logs by using the following filter:
      • logName="projects/resource.project_display_name/logs/cloudaudit.googleapis.com%2Factivity"
      • resource.type="k8s_cluster"
      • resource.labels.project_id="resource.project_display_name"
      • resource.labels.location="location"
      • resource.labels.cluster_name="cluster_name"
      • Pod_Name
    3. Find GKE node console logs by using the following filter:
      • resource.type="gce_instance"
      • resource.labels.instance_id="instance_id"

Step 5: Investigate running container

If the container is still running, it might be possible to investigate the container environment directly.

  1. Go to the Cloud Console.

    Open Cloud Console

  2. On the Cloud Console toolbar, select the project listed in resource.project_display_name, if necessary.

  3. Click Activate Cloud Shell.

  4. Obtain GKE credentials for your cluster by running the following commands.

    For zonal clusters, run:

      gcloud container clusters get-credentials cluster_name --zone location --project resource.project_display_name

    For regional clusters, run:

      gcloud container clusters get-credentials cluster_name --region location --project resource.project_display_name

  5. Launch a shell within the container environment by running:

      kubectl exec --namespace=Pod_Namespace -ti Pod_Name -c Container_Name -- /bin/sh

    This command requires the container to have a shell installed at /bin/sh.

    To view all processes running in the container, run the following command in the container shell:

      ps axjf

    This command requires the container to have /bin/ps installed.

Step 6: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Command and Scripting Interpreter, Ingress Tool Transfer.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 7: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised container. Following these suggested steps might impact operations.
  • Stop or delete the compromised container and replace it with a new container.

To help keep threats from reoccurring, review and fix vulnerabilities identified by Security Health Analytics.

  1. In the Security Command Center dashboard, go to the Assets page.

    Go to Assets

  2. Next to View by, select Project.

  3. In the Project list, select the project that contains the threat finding.

  4. In the Filter box, enter "securityCenterProperties.resourceType:google.compute.Instance". The table lists instances in the project.

  5. Under the resourceProperties.name column, select the instance that contains the finding.

  6. In the panel that loads, under Additional Information, select the Findings tab. If findings are listed, review and follow the remediation guidance for each finding.

In addition to Google-provided indicators of compromise, users who are customers of Palo Alto Networks can integrate Palo Alto Networks' AutoFocus Threat Intelligence with Event Threat Detection. AutoFocus is a threat intelligence service that provides information about network threats. To learn more, visit the AutoFocus page in Cloud Console.

Remediating threats

Remediating Event Threat Detection and Container Threat Detection findings is not as simple as fixing misconfigurations and vulnerabilities identified by Security Command Center.

Misconfigurations and compliance violations identify weaknesses in resources that could be exploited. Typically, misconfigurations have known, easily implemented fixes, like enabling a firewall or rotating an encryption key.

Threats differ from vulnerabilities in that they are dynamic and indicate a possible active exploit against one or more resources. A remediation recommendation might not be effective in securing your resources because the exact methods used to achieve the exploit might not be known.

For example, an Added Binary Executed finding indicates that an unauthorized binary was launched in a container. A basic remediation recommendation might advise you to quarantine the container and delete the binary, but that might not resolve the underlying root cause that allowed the attacker access to execute the binary. You need to find out how the container image was corrupted to fix the exploit. Determining whether the file was added through a misconfigured port or by some other means requires a thorough investigation. An analyst with expert-level knowledge of your system might need to review it for weaknesses.

Bad actors attack resources using different techniques, so applying a fix for a specific exploit might not be effective against variations of that attack. For example, in response to a Brute Force: SSH finding, you might lower permission levels for some user accounts to limit access to resources. However, weak passwords might still provide an attack path.

The breadth of attack vectors makes it difficult to provide remediation steps that work in all situations. Security Command Center's role in your cloud security plan is to identify impacted resources in near-real time, tell you what threats you face, and provide evidence and context to aid your investigations. However, your security personnel must use the extensive information in Security Command Center findings to determine the best ways to remediate issues and secure resources against future attacks.

What's next