Using Container Threat Detection

Review Container Threat Detection findings in the Security Command Center dashboard, and see examples of Container Threat Detection findings. Container Threat Detection is a built-in service for the Security Command Center Premium tier. To view Container Threat Detection findings, it must be enabled in Security Command Center Services settings.

The following video shows the steps to set up Container Threat Detection and provides information about how to use the dashboard. Learn more about viewing and managing Container Threat Detection findings in Reviewing findings on this page.

Using a supported GKE version

To detect potential threats to your containers, make sure that your clusters are on a supported version of Google Kubernetes Engine (GKE). Container Threat Detection currently supports the following GKE versions on the Stable, Regular, and Rapid channels:

  • >= 1.15.9-gke.12
  • >= 1.16.5-gke.2
  • >= 1.17

Enabling Container Threat Detection

To use a supported GKE version and detect threats to your containers, do the following:

  1. Complete the steps in the guide to upgrade a cluster.
  2. Make sure that Container Threat Detection is enabled for the cluster:

    1. Go to the Security Command Center Settings page in the Cloud Console.

      Go to Settings

    2. Navigate to Advanced settings and expand the menu. You see a list of your organization's resources.

    3. Under the Container Threat Detection column, select Enabled by default for each cluster you upgraded. The service is automatically enabled for child resources in folders if they are set to inherit. Manually enable Container Threat Detection for child resources that are not set to inherit.

    4. A dialog appears to confirm your choices. Read the message, then click Yes, I Understand.

For more information, see configuring Security Command Center resources.

Checking GKE cluster configuration

For Container Threat Detection to work, if your cluster is in a Virtual Private Cloud (VPC), its network must meet the routing, firewall, and DNS requirements to communicate with Google APIs and services. To access Google APIs, review the following guides:

Also, your GKE cluster configuration or organization policy constraints must not block the creation or use of any objects that Container Threat Detection needs to function. The following sections include a list of GKE objects that Container Threat Detection creates and explain how to configure essential GKE components to work with Container Threat Detection.

Kubernetes objects

After onboarding, Container Threat Detection creates several GKE objects in your enabled clusters. The objects are used to monitor container images, manage privileged containers and pods, and evaluate state to generate findings. The following table lists the objects, their properties, and essential functions.

Object Name1 Properties Function
ClusterRole pod-reader Grants get, watch, and list permissions on pods Preserves functionality when PodSecurityPolicy is enabled
ClusterRoleBinding container-watcher-pod-reader

gce:podsecuritypolicy:container-watcher

Grants pod-reader and gce:podsecuritypolicy:privileged roles to container-watcher-pod-reader ServiceAccount
RoleBinding gce:podsecuritypolicy:container-watcher Grants gce:podsecuritypolicy:privileged role to container-watcher-pod-reader ServiceAccount
DaemonSet container-watcher Privileged Interactions with Linux Security Module and container engine
Mounts /host/ as read and write Communication with Linux Security Module
Mounts /etc/container-watcher/secrets as read-only to access container-watcher-token Authentication
Uses hostNetwork Finding generation
Image
gke.gcr.io/watcher-daemonset
Enablement and upgrade
Backend
containerthreatdetection-region.googleapis.com:443
Finding generation
ServiceAccount container-watcher-pod-reader Enablement, upgrade, and disablement
Secret container-watcher-token Authentication

1 All objects are in the kube-system namespace, except container-watcher-pod-reader and gce:podsecuritypolicy:container-watcher.

PodSecurityPolicy and Admission Controllers

A PodSecurityPolicy is an admission controller resource you set up that validates requests to create and update pods on your cluster. Container Threat Detection is compatible with PodSecurityPolicies that are automatically applied when creating or updating a cluster with the enable-pod-security-policy flag. Specifically, Container Threat Detection uses the gce.privileged policy when PodSecurityPolicy is enabled.

If you use custom PodSecurityPolicies or other admission controllers, they must not block the creation or use of objects Container Threat Detection needs to function. For example, a webhook-based admission controller that rejects or overrides privileged deployments could prevent Container Threat Detection from functioning properly.

For more information, see Using PodSecurityPolicies.

Required IAM permissions

Container Threat Detection's Google-managed service account, which uses the format service-PROJECT_NUMBER@gcp-sa-ktd-control.iam.gserviceaccount.com, requires the Container Threat Detection Service Agent role (roles/containerthreatdetection.serviceAgent) to enable and disable Container Threat Detection and manage the Container Threat Detection Agent on GKE clusters. Removing this default role from the service account could stop Container Threat Detection from functioning properly.

Container Threat Detection API

Container Threat Detection automatically enables the containerthreatdetection API during onboarding to allow finding generation. You should not interact directly with this required API. Disabling this API would damage Container Threat Detection's ability to generate new findings. If you want to stop receiving Container Threat Detection findings, disable Container Threat Detection in Security Command Center Services settings.

Reviewing findings

When Container Threat Detection generates findings, you can view them in Security Command Center. If you configured Continuous Exports to write logs, you can also view findings in Cloud Logging. To generate a finding and verify your configuration, you can intentionally trigger a detector and test Container Threat Detection.

Container Threat Detection has the following latencies:

  • Activation latency of 3.5 hours for newly onboarded organizations.
  • Activation latency of minutes for newly created clusters.
  • Detection latency of minutes for threats in clusters that have been activated.

Reviewing findings in Security Command Center

Security Command Center roles are granted at the organization, folder, or project level. Your ability to view, edit, create, or update findings, assets, security sources, and security marks depends on the level for which you are granted access. To learn more about Security Command Center roles, see Access control.

To review Container Threat Detection findings in Security Command Center:

  1. Go to the Security Command Center Findings tab in the Google Cloud Console.
    Go to the Findings tab
  2. Next to View by, click Source Type.
  3. In the Source type list, select Container Threat Detection.
  4. To view details about a specific finding, click the finding name under category. The finding details panel expands to display information, including the following:
    • The type of finding, like "Added Binary Executed"
    • Source: "Container Threat Detection"
    • Event time: when the finding occurred
    • Finding ID: a unique identifier for the finding
    • Resource name: the GKE cluster that is affected
    • Finding properties with more information, including the following:
      • Container name
      • Container creation time
      • Container image URI and ID
      • Additional fields based on the detector. For example, Reverse Shell findings include the IP address of the remote host.

Viewing findings in Cloud Logging

To view Container Threat Detection findings in Cloud Logging, do the following:

  1. Go to the Logs Explorer page for Cloud Logging in the Cloud Console.

    Go to Logs Explorer

  2. In the Project selector at the top of the page, select the project where you are storing your Container Threat Detection logs.

  3. Click the Query builder tab.

  4. In the Resource drop-down list, select Threat Detector.

    • To view findings from all detectors, select all detector_name.
    • To view findings from a specific detector, select its name.
  5. Alternatively, enter resource.type="threat_detector" in the query builder text box, and then click Run Query.

  6. The table is updated with the logs you selected.

  7. Create advanced log queries to specify a set of log entries from any number of logs.

Finding categories

Container Threat Detection findings include the following:

Detector Description Inputs to detection
Added Binary Executed

A binary that was not part of the original container image was executed.

If an added binary is executed by an attacker, it's a possible sign that an attacker has control of the workload and they are executing arbitrary commands.

The detector looks for a binary being executed that was not part of the original container image, or was modified from the original container image.
Added Library Loaded

A library that was not part of the original container image was loaded.

If an added library is loaded, it's a possible sign that an attacker has control of the workload and they are executing arbitrary code.

The detector looks for a library being loaded that was not part of the original container image, or was modified from the original container image.
Reverse Shell

A process started with stream redirection to a remote connected socket.

With a reverse shell, an attacker can communicate from a compromised workload to an attacker-controlled machine. The attacker can then command and control the workload to perform desired actions, for example, as part of a botnet.

The detector looks for stdin bound to a remote socket.

Example finding formats

This section includes the JSON output formats for Container Threat Detection findings as they appear when you create exports from the Security Command Center dashboard or run list methods in Security Command Center API.

The output examples contain the fields most common to all findings. However, all fields may not appear in every finding. The actual output you see depends on a resource's configuration and the type and state of findings.

Added Binary Executed

{
  "finding": {
    "name": "organizations/ORGANIZATION_ID/sources/SOURCE_ID/findings/FINDING_ID",
    "parent": "organizations/ORGANIZATION_ID/sources/SOURCE_ID",
    "resourceName": "//container.googleapis.com/projects/PROJECT_ID/zones/ZONE/clusters/CLUSTER_ID",
    "state": "ACTIVE",
    "category": "Added Binary Executed",
    "sourceProperties": {
      "VM_Instance_Name": "INSTANCE_ID",
      "Added_Binary_Kind": "Added",
      "Container_Image_Id": "CONTAINER_IMAGE_ID",
      "Container_Name": "CONTAINER_NAME",
      "Parent_Pid": 1.0,
      "Container_Image_Uri": "CONTAINER_IMAGE_URI",
      "Process_Creation_Timestamp": {
        "seconds": 1.617989997E9,
        "nanos": 1.17396995E8
      },
      "Pid": 53.0,
      "Pod_Namespace": "default",
      "Process_Binary_Fullpath": "BINARY_PATH",
      "Process_Arguments": ["BINARY_PATH"],
      "Pod_Name": "CONTAINER_NAME",
      "description": "A binary that was not part of the original container image
      was executed. If an added binary is executed by an attacker, this is a
      possible sign that an attacker has control of the workload and they are
      executing arbitrary commands.",
      "Environment_Variables": ["KUBERNETES_PORT\u003dtcp://IP_ADDRESS:PORT",
      "KUBERNETES_SERVICE_PORT\u003d443", "HOSTNAME\u003dreconnect-
      test-4af235e12be6f9d9", "HOME\u003d/root",
      "KUBERNETES_PORT_443_TCP_ADDR\u003dIP_ADDRESS",
      "PATH\u003d/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "KUBERNETES_PORT_443_TCP_PORT\u003d443",
      "KUBERNETES_PORT_443_TCP_PROTO\u003dtcp",
      "DEBIAN_FRONTEND\u003dnoninteractive",
      "KUBERNETES_PORT_443_TCP\u003dtcp://IP_ADDRESS:PORT",
      "KUBERNETES_SERVICE_PORT_HTTPS\u003d443",
      "KUBERNETES_SERVICE_HOST\u003dIP_ADDRESS", "PWD\u003d/"],
      "Container_Creation_Timestamp": {
        "seconds": 1.617989918E9,
        "nanos": 0.0
      }
    },
    "securityMarks": {
      "name": "organizations/ORGANIZATION_ID/sources/SOURCE_ID/findings/FINDING_ID/securityMarks"
    },
    "eventTime": "2021-04-09T17:39:57.527Z",
    "createTime": "2021-04-09T17:39:57.625Z",
    "propertyDataTypes": {
      "Container_Image_Id": {
        "primitiveDataType": "STRING"
      },
      "Pod_Namespace": {
        "primitiveDataType": "STRING"
      },
      "Container_Creation_Timestamp": {
        "dataType": "TIMESTAMP",
        "structValue": {
          "fields": {
            "seconds": {
              "primitiveDataType": "NUMBER"
            },
            "nanos": {
              "primitiveDataType": "NUMBER"
            }
          }
        }
      },
      "Environment_Variables": {
        "listValues": {
          "propertyDataTypes": [{
            "primitiveDataType": "STRING"
          }]
        }
      },
      "Added_Binary_Kind": {
        "primitiveDataType": "STRING"
      },
      "description": {
        "primitiveDataType": "STRING"
      },
      "Pid": {
        "primitiveDataType": "NUMBER"
      },
      "Process_Arguments": {
        "listValues": {
          "propertyDataTypes": [{
            "primitiveDataType": "STRING"
          }]
        }
      },
      "Container_Image_Uri": {
        "primitiveDataType": "STRING"
      },
      "Pod_Name": {
        "primitiveDataType": "STRING"
      },
      "Process_Creation_Timestamp": {
        "dataType": "TIMESTAMP",
        "structValue": {
          "fields": {
            "seconds": {
              "primitiveDataType": "NUMBER"
            },
            "nanos": {
              "primitiveDataType": "NUMBER"
            }
          }
        }
      },
      "Parent_Pid": {
        "primitiveDataType": "NUMBER"
      },
      "VM_Instance_Name": {
        "primitiveDataType": "STRING"
      },
      "Container_Name": {
        "primitiveDataType": "STRING"
      },
      "Process_Binary_Fullpath": {
        "primitiveDataType": "STRING"
      }
    },
    "severity": "CRITICAL",
    "workflowState": "NEW",
    "canonicalName": "projects/PROJECT_NUMBER/sources/SOURCE_ID/findings/FINDING_ID"
  },
  "resource": {
    "name": "//container.googleapis.com/projects/PROJECT_ID/zones/ZONE/clusters/CLUSTER_ID",
    "projectName": "//cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER",
    "projectDisplayName": "PROJECT_ID",
    "parentName": "//cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER",
    "parentDisplayName": "PROJECT_ID",
    "type": "google.container.Cluster"
  }
}
    

Added Library Loaded

{
  "finding": {
    "name": "organizations/ORGANIZATION_ID/sources/SOURCE_ID/findingsFINDING_ID",
    "parent": "organizations/ORGANIZATION_ID/sources/SOURCE_ID",
    "resourceName": "//container.googleapis.com/projects/PROJECT_ID/zones/ZONE/clusters/CLUSTER_ID",
    "state": "ACTIVE",
    "category": "Added Library Loaded",
    "sourceProperties": {
      "Process_Arguments": ["BINARY_PATH", "ADDED_LIBRARY_NAME"],
      "Parent_Pid": 1.0,
      "Container_Name": "suspicious-library",
      "Added_Library_Fullpath": "ADDED_LIBRARY_PATH",
      "Container_Image_Id": "CONTAINER_IMAGE_ID",
      "Container_Creation_Timestamp": {
        "seconds": 1.618004144E9,
        "nanos": 0.0
      },
      "Pod_Name": "suspicious-library",
      "Pid": 7.0,
      "description": "A library that was not part of the original container
      image was loaded. If an added library is loaded, this is a possible sign
      that an attacker has control of the workload and they are executing
      arbitrary code.",
      "VM_Instance_Name": "INSTANCE_ID",
      "Pod_Namespace": "default",
      "Environment_Variables": ["KUBERNETES_SERVICE_PORT\u003d443",
      "KUBERNETES_PORT\u003dtcp://IP_ADDRESS:PORT", "HOSTNAME\u003dsuspicious-
      library", "LD_LIBRARY_PATH\u003d/tmp", "PORT\u003d8080",
      "HOME\u003d/root", "PYTHONUNBUFFERED\u003d1",
      "KUBERNETES_PORT_443_TCP_ADDR\u003dIP_ADDRESS",
      "PATH\u003d/opt/python3.7/bin:/opt/python3.6/bin:/opt/python3.5/bin:/opt/p
      ython3.4/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      , "KUBERNETES_PORT_443_TCP_PORT\u003d443",
      "KUBERNETES_PORT_443_TCP_PROTO\u003dtcp", "LANG\u003dC.UTF-8",
      "DEBIAN_FRONTEND\u003dnoninteractive",
      "KUBERNETES_SERVICE_PORT_HTTPS\u003d443",
      "KUBERNETES_PORT_443_TCP\u003dtcp://IP_ADDRESS:PORT",
      "KUBERNETES_SERVICE_HOST\u003dIP_ADDRESS", "PWD\u003d/home/vmagent/app"],
      "Process_Binary_Fullpath": "BINARY_PATH",
      "Added_Library_Kind": "Added",
      "Container_Image_Uri": "CONTAINER_IMAGE_uri"
    },
    "securityMarks": {
      "name": "organizations/ORGANIZATION_ID/sources/SOURCE_ID/findings/FINDING_ID/securityMarks"
    },
    "eventTime": "2021-04-09T21:36:13.069Z",
    "createTime": "2021-04-09T21:36:13.267Z",
    "propertyDataTypes": {
      "Container_Image_Id": {
        "primitiveDataType": "STRING"
      },
      "Added_Library_Fullpath": {
        "primitiveDataType": "STRING"
      },
      "Container_Creation_Timestamp": {
        "dataType": "TIMESTAMP",
        "structValue": {
          "fields": {
            "seconds": {
              "primitiveDataType": "NUMBER"
            },
            "nanos": {
              "primitiveDataType": "NUMBER"
            }
          }
        }
      },
      "Pod_Namespace": {
        "primitiveDataType": "STRING"
      },
      "Environment_Variables": {
        "listValues": {
          "propertyDataTypes": [{
            "primitiveDataType": "STRING"
          }]
        }
      },
      "description": {
        "primitiveDataType": "STRING"
      },
      "Process_Arguments": {
        "listValues": {
          "propertyDataTypes": [{
            "primitiveDataType": "STRING"
          }]
        }
      },
      "Pid": {
        "primitiveDataType": "NUMBER"
      },
      "Container_Image_Uri": {
        "primitiveDataType": "STRING"
      },
      "Pod_Name": {
        "primitiveDataType": "STRING"
      },
      "Added_Library_Kind": {
        "primitiveDataType": "STRING"
      },
      "Parent_Pid": {
        "primitiveDataType": "NUMBER"
      },
      "VM_Instance_Name": {
        "primitiveDataType": "STRING"
      },
      "Container_Name": {
        "primitiveDataType": "STRING"
      },
      "Process_Binary_Fullpath": {
        "primitiveDataType": "STRING"
      }
    },
    "severity": "CRITICAL",
    "workflowState": "NEW",
    "canonicalName": "projects/PROJECT_NUMBER/sources/SOURCE_ID/findings/FINDING_ID"
  },
  "resource": {
    "name": "//container.googleapis.com/projects/PROJECT_ID/zones/ZONE/clusters/CLUSTER_ID",
    "projectName": "//cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER",
    "projectDisplayName": "PROJECT_ID",
    "parentName": "//cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER",
    "parentDisplayName": "PROJECT_ID",
    "type": "google.container.Cluster"
  }
}
  

Reverse Shell

{
  "finding": {
    "name": "organizations/ORGANIZATION_ID/sources/SOURCE_ID/findings/FINDING_ID",
    "parent": "organizations/ORGANIZATION_ID/sources/SOURCE_ID",
    "resourceName": "//container.googleapis.com/projects/PROJECT_ID/zones/ZONE/clusters/CLUSTER_ID",
    "state": "ACTIVE",
    "category": "Reverse Shell",
    "sourceProperties": {
      "Reverse_Shell_Stdin_Redirection_Src_Ip": "SOURCE_IP_ADDRESS",
      "Environment_Variables": ["HOSTNAME\u003dreverse-shell",
      "KUBERNETES_PORT\u003dtcp://IP_ADDRESS:PORT",
      "KUBERNETES_PORT_443_TCP_PORT\u003d443", "PYTHONUNBUFFERED\u003d1",
      "KUBERNETES_SERVICE_PORT\u003d443",
      "KUBERNETES_SERVICE_HOST\u003dIP_ADDRESS",
      "PATH\u003d/opt/python3.7/bin:/opt/python3.6/bin:/opt/python3.5/bin:/opt/p
      ython3.4/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      , "PWD\u003d/home/vmagent/app", "LANG\u003dC.UTF-8", "SHLVL\u003d1",
      "HOME\u003d/root", "KUBERNETES_PORT_443_TCP_PROTO\u003dtcp",
      "KUBERNETES_SERVICE_PORT_HTTPS\u003d443",
      "DEBIAN_FRONTEND\u003dnoninteractive", "PORT\u003d8080",
      "KUBERNETES_PORT_443_TCP_ADDR\u003dIP_ADDRESS",
      "KUBERNETES_PORT_443_TCP\u003dtcp://IP_ADDRESS:PORT", "_\u003d/bin/echo"],
      "Container_Image_Uri": "CONTAINER_IMAGE_URI",
      "Process_Binary_Fullpath": "BINARY_PATH",
      "Container_Creation_Timestamp": {
        "seconds": 1.617989861E9,
        "nanos": 0.0
      },
      "Pod_Name": "reverse-shell",
      "Container_Name": "reverse-shell",
      "Process_Arguments": ["BINARY_PATH", "BINARY_NAME"],
      "Pid": 15.0,
      "Reverse_Shell_Stdin_Redirection_Dst_Port": DESTINATION_PORT,
      "Container_Image_Id": "CONTAINER_IMAGE_ID",
      "Reverse_Shell_Stdin_Redirection_Dst_Ip": "DESTINATION_IP_ADDRESS",
      "Pod_Namespace": "default",
      "VM_Instance_Name": "INSTANCE_ID",
      "Reverse_Shell_Stdin_Redirection_Src_Port": SOURCE_PORT,
      "description": "A process started with stream redirection to a remote
      connected socket. With a reverse shell, an attacker can communicate from a
      compromised workload to an attacker-controlled machine. The attacker can
      then command and control the workload to perform desired actions, for
      example as part of a botnet.",
      "Parent_Pid": 1.0,
      "Process_Creation_Timestamp": {
        "seconds": 1.61798989E9,
        "nanos": 6.16573691E8
      }
    },
    "securityMarks": {
      "name": "organizations/ORGANIZATION_ID/sources/SOURCE_ID/findings/FINDING_ID/securityMarks"
    },
    "eventTime": "2021-04-09T17:38:10.904Z",
    "createTime": "2021-04-09T17:38:15.486Z",
    "propertyDataTypes": {
      "Container_Image_Id": {
        "primitiveDataType": "STRING"
      },
      "Container_Creation_Timestamp": {
        "dataType": "TIMESTAMP",
        "structValue": {
          "fields": {
            "seconds": {
              "primitiveDataType": "NUMBER"
            },
            "nanos": {
              "primitiveDataType": "NUMBER"
            }
          }
        }
      },
      "Pod_Namespace": {
        "primitiveDataType": "STRING"
      },
      "Environment_Variables": {
        "listValues": {
          "propertyDataTypes": [{
            "primitiveDataType": "STRING"
          }]
        }
      },
      "Reverse_Shell_Stdin_Redirection_Dst_Ip": {
        "primitiveDataType": "STRING"
      },
      "description": {
        "primitiveDataType": "STRING"
      },
      "Process_Arguments": {
        "listValues": {
          "propertyDataTypes": [{
            "primitiveDataType": "STRING"
          }]
        }
      },
      "Pid": {
        "primitiveDataType": "NUMBER"
      },
      "Reverse_Shell_Stdin_Redirection_Src_Ip": {
        "primitiveDataType": "STRING"
      },
      "Container_Image_Uri": {
        "primitiveDataType": "STRING"
      },
      "Reverse_Shell_Stdin_Redirection_Dst_Port": {
        "primitiveDataType": "NUMBER"
      },
      "Pod_Name": {
        "primitiveDataType": "STRING"
      },
      "Process_Creation_Timestamp": {
        "dataType": "TIMESTAMP",
        "structValue": {
          "fields": {
            "seconds": {
              "primitiveDataType": "NUMBER"
            },
            "nanos": {
              "primitiveDataType": "NUMBER"
            }
          }
        }
      },
      "Reverse_Shell_Stdin_Redirection_Src_Port": {
        "primitiveDataType": "NUMBER"
      },
      "Parent_Pid": {
        "primitiveDataType": "NUMBER"
      },
      "VM_Instance_Name": {
        "primitiveDataType": "STRING"
      },
      "Container_Name": {
        "primitiveDataType": "STRING"
      },
      "Process_Binary_Fullpath": {
        "primitiveDataType": "STRING"
      }
    },
    "severity": "CRITICAL",
    "workflowState": "NEW",
    "canonicalName": "projects/PROJECT_NUMBER/sources/SOURCE_ID/findings/FINDING_ID"
  },
  "resource": {
    "name": "//container.googleapis.com/projects/PROJECT_ID/zones/ZONE/clusters/CLUSTER_ID",
    "projectName": "//cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER",
    "projectDisplayName": "PROJECT_ID",
    "parentName": "//cloudresourcemanager.googleapis.com/projects/PROJECT_NUMBER",
    "parentDisplayName": "PROJECT_ID",
    "type": "google.container.Cluster"
  }
}
  

What's next