Respond to Google Kubernetes Engine threat findings

This document offers informal guidance on how you can respond to findings of suspicious activities in your Google Kubernetes Engine resources. The recommended steps might not be appropriate for all findings and might impact your operations. Before you take any action, you should investigate the findings; assess the information that you gather; and decide how to respond.

The techniques in this document aren't guaranteed to be effective against any previous, current, or future threats that you face. To understand why Security Command Center does not provide official remediation guidance for threats, see Remediating threats.

Before you begin

  1. Review the finding. Review the affected Google Kubernetes Engine resource, the detected principal email, and the caller IP address (if present). Also review the finding for indicators of compromise (IP, domain, file hash, or signature).
  2. To learn more about the finding that you're investigating, search for the finding in the Threat findings index.

General recommendations

  • Contact the owner of the project that contains the potentially compromised resource.
  • Determine whether there are other signs of malicious activity related to the affected GKE resource in the audit logs in Cloud Logging.
  • Stop or delete the compromised GKE resource and replace it with a new one.
  • Determine whether there are other signs of malicious activity by the principal in the audit logs in Cloud Logging.
  • If the principal is a service account (IAM or Kubernetes), identify the source of the modification to determine its legitimacy.
  • If the principal that conducted the action isn't a service account, contact the owner of the account to confirm whether the legitimate owner conducted the action.
  • Review the guidance on the principle of least privilege for the RBAC roles and cluster roles.

In addition, consider the recommendations in the following sections.

Added binary or library

If the added binary, script, or library was intended to be included in the container, rebuild the container image with the binary, script, or library included. For information about immutable container images, see Container images in the Kubernetes documentation.

Threats related to Kubernetes certificate signing requests (CSR)

  • Review the audit logs in Cloud Logging and additional alerts for other CSR-related events. Determine whether the CSR was approved and issued and whether the CSR-related actions are expected from the principal.
  • If a CSR approval was unexpected or is determined to be malicious, the cluster requires a credential rotation to invalidate the certificate. Review the guidance for rotating your cluster credentials.

Threat findings for Pods

  • Review the manifest file of the Pod and its purpose. Verify that the Pod is legitimate and necessary.
  • If the Pod is illegitimate, remove it along with any associated RBAC bindings and service accounts that the workload used.

What's next