Investigating and responding to threats

This topic offers informal guidance to help you investigate and respond to threats, and use additional resources to add context to Security Command Center findings. Following these steps helps you understand what happened during a potential attack and develop possible responses for affected resources.

The techniques on this page are not guaranteed to be effective against any previous, current, or future threats you face. See Remediating threats to understand why Security Command Center does not provide official remediation guidance for threats.

Before you begin

You need adequate Identity and Access Management (IAM) roles to view or edit findings and logs, and modify Google Cloud resources. If you encounter access errors in Security Command Center, ask your administrator for assistance and see Access control to learn about roles. To resolve resource errors, read documentation for affected products.

Understanding threat findings

Event Threat Detection produces security findings by matching events in your Cloud Logging stream to known indicators of compromise (IoC). IoCs, developed by internal Google security sources, identify potential vulnerabilities and attacks. Event Threat Detection also detects threats by identifying known adversarial tactics, techniques, and procedures in your logging stream, and by detecting deviations from historically observed behavior of your organization.

Container Threat Detection generates findings by collecting and analyzing low-level observed behavior in the guest kernel of containers.

Findings are written to Security Command Center and Cloud Logging.

Reviewing findings

To review threat findings, do the following:

  1. In the Google Cloud Console, go to the Security Command Center Findings page.

    Go to Findings

  2. If necessary, select your Google Cloud project or organization.

  3. On the findings page, next to View by, click Source type.

  4. Select Event Threat Detection or Container Threat Detection. The table is populated with findings for the source you selected.

  5. To view details for a specific finding, click the finding name under category in the table. The Finding Details panel opens.

Findings provide the names and numeric identifiers of resources involved in an incident, along with environment variables and asset properties. You can use that information to quickly isolate affected resources and determine the potential scope of an event.

Threat findings also contain links to external resources to aid in your investigations. These resources include:

  • MITRE ATT&CK framework entries. The framework explains techniques for attacks against cloud resources and provides remediation guidance.
  • VirusTotal, an Alphabet-owned service that provides context on potentially malicious files, URLs, domains, and IP addresses.

The following sections outline potential responses to threat findings.

Event Threat Detection responses

To learn more about Event Threat Detection, see how Event Threat Detection works.

Exfiltration: BigQuery Data Exfiltration

Data exfiltration from BigQuery is detected by examining audit logs for three scenarios:

  • A resource is saved outside of your organization.
  • VPC Service Controls blocks a copy operation.
  • An attempt is made to access BigQuery resources that are protected by VPC Service Controls.

To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an Exfiltration: BigQuery Data Exfiltration finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • properties
      • projectId: the finding involves an asset in this project
      • jobLink: the link to the BigQuery job that exfiltrated data
      • Query: the SQL query run on the BigQuery dataset
      • SourceTables: the tables from which data was exfiltrated
      • DestinationTables: the tables where exfiltrated data was stored
      • userEmail: the account used to exfiltrate the data
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the IAM page .

    Go to IAM

  2. If necessary, select the project listed in projectID.

  3. On the page that appears, in the Filter box, enter the email address listed inuserEmail and check what permissions are assigned to the account.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. Find admin activity logs related to BigQuery jobs by using the following filters:
    • protoPayload.methodName="Jobservice.insert"
    • protoPayload.methodName="google.cloud.bigquery.v2.JobService.InsertJob"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Exfiltration Over Web Service: Exfiltration to Cloud Storage.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type on the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

Brute Force: SSH

Detection of successful brute force of SSH on a host. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Brute Force: SSH finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • sourceLogId: the project ID and timestamp to identify the log entry
      • projectId: the project that contains the finding
    • Attempts: the number of login attempts
      • sourceIP: the IP address that launched the attack
      • username: the account that logged in
      • vmName: the name of the virtual machine
      • authResult: the SSH authentication result
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Review permissions and settings

  1. In the Google Cloud Console, go to the Dashboard.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in projectId.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the name and zone in vmName. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules on port 22.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, find VPC Flow Logs related to srcIP by using the following filter:
    • logName="projects/projectId/logs/syslog"
    • labels."compute.googleapis.com/resource_name"="vmName"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Local Accounts.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the successful brute force attempt. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an Endpoint Detection and Response solution.
  • Consider disabling SSH access to the VM. This step could interrupt authorized access to the VM, so consider the needs of your organization before you proceed.
  • Only use SSH authentication with authorized keys.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the quantity of information, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.

Cryptomining

Malware is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known command and control domains and IP addresses. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Cryptomining: Bad IP finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • Properties_ip_0: the suspected cryptomining IP address
    • properties_sourceInstance: the affected Compute Engine VM instance
    • properties_project_id: the project for the Compute Engine instance

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in properties_project_id.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches properties_sourceInstance. Investigate the potentially compromised instance for malware.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. Select your project.

  3. On the page that loads, find VPC Flow Logs related to Properties_ip_0 by using the following filter:

    • logName="projects/properties_project_id/logs/compute.googleapis.com%2Fvpc_flows"
    • (jsonPayload.connection.src_ip="Properties_ip_0" OR jsonPayload.connection.dest_ip="Properties_ip_0")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Resource Hijacking.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project containing malware. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an Endpoint Detection and Response solution.
  • If necessary, stop the compromised instance and replace it with a new instance.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.

Leaked credentials

This finding is generated when Google Cloud service account credentials are accidentally leaked online or compromised. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an account_has_leaked_credentials finding, as directed in Reviewing finding details.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • Compromised_account: the potentially compromised service account
    • Project_identifier: the project that contains the potentially leaked account credentials
    • URL: the link to the GitHub repository

Step 2: Review project and service account permissions

  1. In the Cloud Console, go to the IAM page.

    Go to IAM

  2. If necessary, select the project listed in Project_identifier.

  3. On the page that appears, in the Filter box, enter the account name listed in Compromised_account and check assigned permissions.

  4. In the Cloud Console, go to the Service Accounts page.

    Go to Service Accounts

  5. On the page that appears, in the Filter box, enter the name of the compromised service account and check the service account's keys and key creation dates.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. Select your project.

  3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:

    • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
    • protoPayload.methodName="SetIamPolicy"
    • resource.type="gce_instance" AND log_name="projects/Project_identifier/logs/cloudaudit.googleapis.com%2Factivity"
    • protoPayload.methodName="InsertProjectOwnershipInvite"
    • protoPayload.authenticationInfo.principalEmail="Compromised_account"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Valid Accounts: Cloud Accounts.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with leaked credentials. Following these suggested steps might impact operations.
  • Consider deleting the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access. Before proceeding, your security team should identify all impacted resources and work with resource owners to ensure business continuity.
  • Work with your security team to identify unfamiliar resources, including Compute Engine instances, snapshots, service accounts, and IAM users. Delete resources not created with authorized accounts.
  • Respond to notifications from Google Cloud Support.
  • To limit who can create service accounts, use the Organization Policy Service.
  • To identify and fix overly permissive roles, use IAM Recommender.
  • Open the URL link and delete the leaked credentials. Gather more information about the compromised account and contact the owner.

Malware

Malware is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known command and control domains and IP addresses. To respond to these findings, do the following:

Step 1: Review finding details

  1. Open a Malware: Bad IP finding, as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • sourceLogId
      • projectId: the project that contains the finding
    • ipConnection
      • srcIP: a known malware command and control IP address
      • srcPort: the source port of the connection
      • destIP: the target IP address of the malware
      • destPort: the destination port of the connection
    • InstanceDetails: the resource address for the Compute Engine instance
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • virustotalIndicatorQueryUI: link to the VirusTotal analysis page
      • cloudLoggingQueryURI: link to the Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in projectId.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the name and zone in instanceDetails. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, find VPC Flow Logs related to the IP address in srcIP by using the following filter:
    • logName="projects/projectId/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="srcIP" OR jsonPayload.connection.dest_ip="destIP")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Dynamic Resolution and Command and Control.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. Check flagged URLs and domains on VirusTotal by clicking the link in virustotalIndicatorQueryUI. VirusTotal is an Alphabet- owned service that provides context on potentially malicious files, URLs, domains, and IP addresses.
  4. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project containing malware. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an Endpoint Detection and Response solution.
  • To track activity and vulnerabilities that allowed the insertion of malware, check audit logs and syslogs associated with the compromised instance.
  • If necessary, stop the compromised instance and replace it with a new instance.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.
  • To control access and use of VM images, use Shielded VM and Trusted Images IAM policy .

Outgoing DoS

Event Threat Detection detects potential use of an instance to launch a denial of service (DoS) attack by analyzing VPC Flow Logs. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Malware: Outgoing DoS finding as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • sourceInstanceDetails: the affected Compute Engine VM instance
    • ipConnection
      • srcIP: a known malware command and control IP address
      • srcPort: the source port of the connection
      • destIP: the target IP address of the malware
      • destPort: the destination port of the connection
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project ID listed in sourceInstanceDetails.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the instance name and zone in sourceInstanceDetails. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, find VPC Flow Logs related to the IP address in srcIP by using the following filter:
    • logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="srcIP" OR jsonPayload.connection.dest_ip="destIP")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Network Denial of Service.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project experiencing outgoing DoS traffic. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an Endpoint Detection and Response solution.
  • To track activity and vulnerabilities that allowed the insertion of malware, check audit logs and syslogs associated with the compromised instance.
  • If necessary, stop the compromised instance and replace it with a new instance.
  • Block the malicious IP addresses by updating firewall rules or by using Google Cloud Armor. You can enable Google Cloud Armor on the Security Command Center Integrated Services page. Depending on the data volume, Google Cloud Armor costs can be significant. See the Google Cloud Armor pricing guide for more information.
  • To control access and use of VM images, use Shielded VM and Trusted Images IAM policy .

Persistence: IAM Anomalous Grant

Audit logs are examined to detect the addition of IAM (IAM) role grants that might be considered suspicious. The following are examples of anomalous grants:

  • Inviting a gmail.com user as a project owner from the Google Cloud Console
  • A service account granting sensitive permissions
  • A custom role granting sensitive permissions
  • A service account added from outside your organization

To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Persistence: IAM Anomalous Grant finding as directed in Reviewing findings.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • projectId: the project that contains the finding
    • principalEmail: email address for the user or service account that assigned the role
    • bindingDeltas
      • Action: the action taken by the user
      • Role: the role assigned to the user
      • member: the email address of the user that received the role
    • contextUris: internal and external resources to add context to findings
      • mitreURI: link to the MITRE ATT&CK framework entry for this finding type
      • virustotalIndicatorQueryUI: link to the VirusTotal analysis page
      • cloudLoggingQueryURI: link to Logging page
      • relatedFindingURI: link to related findings of the same type

Step 2: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. On the page that loads, look for new or updated IAM resources using the following filters:
    • protoPayload.methodName="SetIamPolicy"
    • protoPayload.methodName="google.iam.admin.v1.UpdateRole"
    • protoPayload.methodName="google.iam.admin.v1.CreateRole"

Step 3: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Valid Accounts: Cloud Accounts.
  2. Review related findings by clicking the link in relatedFindingURI. Related findings are the same finding type and the same instance and network.
  3. To develop a response plan, combine your investigation results with MITRE research.

Step 4: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised account. Following these suggested steps might impact operations.
  • Delete the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access.
  • Delete project resources created by unauthorized accounts, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.
  • To restrict adding gmail.com users, use the Organization Policy.
  • To identify and fix overly permissive roles, use IAM Recommender.

Phishing

Phishing is detected by examining VPC Flow Logs and Cloud DNS logs for connections to known phishing domains and IP addresses. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a resource_used_for_phishing finding as directed in Reviewing finding details.

  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:

    • resourceName: the project that contains the finding
    • urls: the phishing URL
    • vm_host_and_zone_names: the name of the Compute Engine VM that is hosting the phishing URL
    • vm_ips: the IP address of the Compute Engine VM that is hosting the phishing URL

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Dashboard page.

    Go to the Dashboard

  2. Click the Select from drop-down list at the top of the page. In the Select from window that appears, select the project listed in resourceName.

  3. Navigate to the Resources card and click Compute Engine.

  4. Click the VM instance that matches the name and zone in InstanceDetails. Review instance details, including network and access settings.

  5. In the navigation panel, click VPC Network, then click Firewall. Remove or disable overly permissive firewall rules.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. Select your project.

  3. On the page that loads, find VPC Flow Logs related to vm_ips by using the following filter:

    • logName="projects/PROJECT_ID/logs/compute.googleapis.com%2Fvpc_flows" AND (jsonPayload.connection.src_ip="vm_ips" OR jsonPayload.connection.dest_ip="vm_ips")

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Phishing.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised VM, vm_host_and_zone_names. Following these suggested steps might impact operations.
  • Investigate the potentially compromised instance and remove any discovered malware. To assist with detection and removal, use an Endpoint Detection and Response solution.
  • If necessary, stop the compromised instance and replace it with a new instance.

Service account self-investigation

A service account credential is being used to investigate the roles and permissions associated with that same service account. This finding indicates that service account credentials might be compromised and immediate action should be taken.

Step 1: Review finding details

  1. Open a Discovery: Service Account Self-Investigation finding, as directed in Reviewing finding details earlier on this page.
  2. In the Finding Details panel, under Additional Information, click Source Properties and note the following fields:
    • principalEmail: the potentially compromised service account
    • projectId: the project that contains the potentially leaked account credentials
    • callerIp: the internal or external IP address
    • cloudLoggingQueryURI: the link to Cloud Logging entries
    • mitreUri: the link to the MITRE ATT&CK framework entry

Step 2: Review project and service account permissions

  1. In the Cloud Console, go to the IAM page.

    Go to IAM

  2. If necessary, select the project listed in projectId.

  3. On the page that appears, in the Filter box, enter the account name listed in principalEmail and check assigned permissions.

  4. In the Cloud Console, go to the Service Accounts page.

    Go to Service Accounts

  5. On the page that appears, in the Filter box, enter the name of the compromised service account and check the service account's keys and key creation dates.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer by clicking the link in cloudLoggingQueryURI.
  2. If necessary, select your project.
  3. On the page that loads, check logs for activity from new or updated IAM resources using the following filters:
    • proto_payload.method_name="google.iam.admin.v1.CreateServiceAccount"
    • protoPayload.methodName="SetIamPolicy"
    • protoPayload.authenticationInfo.principalEmail="principalEmail"

Step 4: Research attack and response methods

  1. Review the MITRE ATT&CK framework entry for this finding type: Permission Groups Discovery: Cloud Groups.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised account. Following these suggested steps might impact operations.
  • Delete the compromised service account and rotate and delete all service account access keys for the compromised project. After deletion, resources that use the service account for authentication lose access.
  • Delete project resources created by the compromised account, like unfamiliar Compute Engine instances, snapshots, service accounts, and IAM users.

Container Threat Detection responses

To learn more about Container Threat Detection, see how Container Threat Detection works.

Added Binary Executed

A binary that was not part of the original container image was executed. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an Added Binary Executed finding as directed in Reviewing findings.
  2. In the Finding Details panel, click the Attributes tab and note the following fields:
    • assetDisplayName: the user-provided asset name
    • Resource.project_display_name: the name of the project that contains the asset
  3. Click the Source Properties tab and note the following fields:
    • Process_Arguments: the paths for the added binary
    • Container_Image_Uri: the name of the container image being executed
    • Pod_Name: the name of the Google Kubernetes Engine pod
    • Container_Name: the name of the affected container

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. Select the project listed in Resource.project_display_name, if necessary, then select the cluster listed in assetDisplayName. Note information about the service running on the cluster and the service owner.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. Select your project.

  3. On the page that loads, find syslog entries for Container_Name by using the following filter:

    • lprotoPayload.authorizationInfo.permission="container.clusters.get"

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Implant Container Image.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised container. Following these suggested steps might impact operations.
  • Stop or delete the compromised container and replace it with a new container.

Added Library Loaded

A library that was not part of the original container image was loaded. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open an Added Library Loaded finding as directed in Reviewing findings.
  2. In the Finding Details panel, click the Attributes tab and take note of the following fields:
    • assetDisplayName: the user-provided asset name
    • Resource.project_display_name: the name of the project that contains the asset
  3. Click the Source Properties tab and note the following fields:
    • Process_Arguments: the paths for the added library
    • Container_Image_Uri: the name of the container image being executed
    • Pod_Name: the name of the GKE pod
    • Container_Name: the name of the affected container

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. Select the project listed in Resource.project_display_name, if necessary, then select the cluster listed in assetDisplayName. Note information about the service running on the cluster and the service owner.

Step 3: Check logs

  1. In the Cloud Console, go to Logs Explorer.

    Go to Logs Explorer

  2. Select your project.

  3. On the page that loads, find syslog entries for Container_Name by using the following filter:

    • lprotoPayload.authorizationInfo.permission="container.clusters.get"

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Implant Container Image.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised container. Following these suggested steps might impact operations.
  • Stop or delete the compromised container and replace it with a new container.

Reverse Shell

A process started with stream redirection to a remote connected socket. To respond to this finding, do the following:

Step 1: Review finding details

  1. Open a Reverse Shell finding as directed in Reviewing findings.
  2. In the Finding Details panel, click the Attributes tab and take note of the following fields:
    • assetDisplayName: the user-provided asset name
    • Resource.project_display_name: the name of the project that contains the asset
  3. Click the Source Properties tab and note the following fields:
    • Process_Arguments: the path for the binary
    • Container_Image_Uri: the name of the container image being executed
    • Pod_Name: the name of the GKE pod
    • Container_Name: the name of the affected container
    • Reverse_Shell_Stdin_Redirection_Dst_Ip: the known command and control IP address
    • Reverse_Shell_Stdin_Redirection_Dst_Port: the destination port
    • Reverse_Shell_Stdin_Redirection_Src_Ip: the source IP address of the connection
    • Reverse_Shell_Stdin_Redirection_Src_Port: the source port

Step 2: Review permissions and settings

  1. In the Cloud Console, go to the Kubernetes clusters page.

    Go to Kubernetes clusters

  2. Select the project listed in Resource.project_display_name, if necessary, then select the cluster listed in assetDisplayName. Note information about the service running on the cluster and the service owner.

Step 3: Check logs

  1. In the Cloud Console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Select your project.

  3. On the page that loads, find syslog entries for Container_Name by using the following filter:

    • lprotoPayload.authorizationInfo.permission="container.clusters.get"

Step 4: Research attack and response methods

  1. Review MITRE ATT&CK framework entries for this finding type: Remote Services.
  2. To develop a response plan, combine your investigation results with MITRE research.

Step 5: Implement your response

The following response plan might be appropriate for your organization. Carefully evaluate the information you gather in your investigation to determine the best way to resolve findings.

  • Contact the owner of the project with the compromised container. Following these suggested steps might impact operations.
  • Stop or delete the compromised container and replace it with a new container.

To help keep threats from reoccurring, review and fix vulnerabilities identified by Security Health Analytics.

  1. In the Security Command Center dashboard, go to the Assets page.

    Go to Assets

  2. Next to View by, select Project.
  3. In the Project list, select the project that contains the threat finding.
  4. In the Filter box, enter "securityCenterProperties.resourceType:google.compute.Instance". The table lists instances in the project.
  5. Under the resourceProperties.name column, select the instance that contains the finding.
  6. In the panel that loads, under Additional Information, select the Findings tab. If findings are listed, review and follow the remediation guidance for each finding.

In addition to Google-provided indicators of compromise, users who are customers of Palo Alto Networks can integrate Palo Alto Networks' AutoFocus Threat Intelligence with Event Threat Detection. AutoFocus is a threat intelligence service that provides information about network threats. To learn more, visit the AutoFocus page in Cloud Console.

Remediating threats

Remediating Event Threat Detection and Container Threat Detection findings is not as simple as fixing misconfigurations and vulnerabilities identified by Security Command Center.

Misconfigurations and compliance violations identify weaknesses in resources that could be exploited. Typically, misconfigurations have known, easily implemented fixes, like enabling a firewall or rotating an encryption key.

Threats differ from vulnerabilities in that they are dynamic and indicate a possible active exploit against one or more resources. A remediation recommendation might not be effective in securing your resources because the exact methods used to achieve the exploit might not be known.

For example, an Added Binary Executed finding indicates that an unauthorized binary was launched in a container. A basic remediation recommendation might advise you to quarantine the container and delete the binary, but that might not resolve the underlying root cause that allowed the attacker access to execute the binary. You need to find out how the container image was corrupted to fix the exploit. Determining whether the file was added through a misconfigured port or by some other means requires a thorough investigation. An analyst with expert-level knowledge of your system might need to review it for weaknesses.

Bad actors attack resources using different techniques, so applying a fix for a specific exploit might not be effective against variations of that attack. For example, in response to a Brute Force: SSH finding, you might lower permission levels for some user accounts to limit access to resources. However, weak passwords might still provide an attack path,

The breadth of attack vectors makes it difficult to provide remediation steps that work in all situations. Security Command Center's role in your cloud security plan is to identify impacted resources in near-real time, tell you what threats you face, and provide evidence and context to aid your investigations. However, your security personnel must use the extensive information in Security Command Center findings to determine the best ways to remediate issues and secure resources against future attacks.

What's next