Ingestion metrics BigQuery schema

This document describes Chronicle ingestion_metric table in BigQuery.

Chronicle uses BigQuery as a data layer for its dashboards. The fields in the ingestion_metric table represent the dimensions that appear in the Ingestion metrics Explore interface. You use the Ingestion metrics Explore interface to create a new dashboard.

The following Ingestion metrics schema helps you understand the data types and format of the dimension fields available in the Ingestion metrics Explore interface:

Fields Type Description
component STRING The service or pipeline to which the metric applies, which includes the Forwarder, Ingestion API, Out-of-Band Processor (Chronicle API feed), and Normalizer.
collector_id STRING The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
feed_id STRING The ID for the specific feed which the log belongs to.
log_type STRING The source of the log which identifies the log entries in the batch. For example, WINDOWS_DNS.
start_time TIMESTAMP The start time associated with the metric in microseconds.
end_time TIMESTAMP The end time associated with the metric in microseconds.
input_type STRING This field is populated if the ingestion source is the Chronicle forwarder. Based on the data that the forwarder sends, this field contains pcap, syslog, or splunk.
namespace STRING Namespace that the log belongs to.
event_type STRING The event type determines which fields are included with the event. The event type includes values such as PROCESS_OPEN, FILE_CREATION, USER_CREATION, and NETWORK_DNS.
drop_reason_code STRING This field is populated if the ingestion source is the Chronicle forwarder. and indicates the reason why a log was dropped during normalization.
last_heartbeat_time TIMESTAMP

This field is populated if the ingestion source is the Chronicle forwarder or Chronicle API feed. The last timestamp when the forwarder or API feed was active in microseconds.

When the feed is active, it sends the data through the last_heartbeat_time field, and the log_count and log_volume fields remain empty.
log_volume FLOAT64

The volume of logs during the interval in bytes.

The log_volume field remains empty or is populated in the following cases:

  • When Chronicle forwarder or feed sends data, this field is populated. The last_heartbeat_time field remains empty.
  • If the feed is inactive, no entry is made in the ingestion metrics table.
  • When a feed is active, the last_heartbeat_time, log_count, or log_volume field is populated.
drop_count FLOAT64 This field is populated if the ingestion source is the Chronicle forwarder. and indicates the number of logs dropped during the interval.
log_count FLOAT64

The number of logs ingested during the interval.

The log_count field remains empty or is populated in the following cases:

  • When Chronicle forwarder or feed sends data, this field is populated. The last_heartbeat_time field remains empty.
  • If the feed is inactive, no entry is made in the ingestion metrics table.
  • When a feed is active, the last_heartbeat_time, log_count, or log_volume field is populated.
event_count FLOAT64 The number of events generated during the interval.
state STRING The final status of the event or log. The status is one of the following:
  • parsed. The log is successfully parsed.
  • validated. The log is successfully validated.
  • failed_parsing. The log has parsing errors.
  • failed_validation. The log has validation errors.
memory_used FLOAT64 The percentage of memory used by the forwarder container.
disk_used FLOAT64 The percentage of disk storage used by the forwarder container along with disk identifier.
cpu_used FLOAT64 The percentage of CPU used by the forwarder container.
buffer_used FLOAT64 The percentage of buffer used by the forwarder buffer type per forwarder input type.
buffer_type STRING The type of buffer used by the forwarder.
ingestion_source STRING The ingestion source present in ingestion label when the logs are ingested through ingestion private API.

You can filter ingestion metrics based on the field values. For example, Out-of-Band Processor feeds have collector_id as aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa. Here is an example query to filter Out-of-Band feeds:

SELECT
  component,
  collector_id,
  count(component)
FROM
  `chronicle-CUSTOMER_CODE.ingestion_metrics`
WHERE
  DATE(start_time) = DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY)
  AND component IN ("Out-of-Band Processor","Ingestion API", "Forwarder")
  AND (collector_id != "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  OR collector_id is null)
group by 1,2

Replace CUSTOMER_CODE with the customer code.

The following fields that appear in the Ingestion metrics Explore interface are reserved for future use:

  • Bucketer growth factor
  • Bucketer num finite buckets
  • Bucketer scale factor
  • Bucketer width
  • Error code
  • Latency count
  • Latency overflow
  • Latency time
  • Latency underflow
  • Regex filter

Ingestion metrics examples

The following table lists the metrics and example values that help you understand the fields in the ingestion_metric schema:

Metrics component collector_id feed_id log_type start_time end_time input_type last_heartbeat_time log_volume drop_count log_count memory_used cpu_used disk_used buffer_used ingestion_source drop_reason_code
Heartbeat Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 syslog 2022-04-21T13:18:55.000+00:00
Log Bytes Count Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 pcap 149.0
Log Record Count Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 pcap 154.0
Drop Count (Backlog) Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 pcap 4.0 backlog
Drop Count (Invalid Config) Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 pcap 4.0 invalid_config
Drop Count (Regex) Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 pcap 4.0 regex
Log Record Count Ingestion API xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DHCP 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 3578.0
Log Bytes Count Ingestion API xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DHCP 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 2802.0
Log Record Count Out-of-Band Processor feeds/aaaaaaaaaaaaaa ARUBA_IPS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 3578.0
Log Bytes Count Out-of-Band Processor feeds/aaaaaaaaaaaaaa ARUBA_IPS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 319563.0
Last Ingested Timestamp Out-of-Band Processor feeds/aaaaaaaaaaaaaa ARUBA_IPS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 2022-04-21T13:18:55.000+00:00
Log Count Normalizer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00
Log Size Normalizer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00
Event Count Normalizer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00
Container Memory Used Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 0.32
Container Disk Used Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 0.5
Container CPU Used Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 0.545
Buffer Used Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 0.562
Ingestion Source Forwarder xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx PCAP_DNS 2022-04-21T13:14:50.924+00:00 2022-04-21T13:19:50.924+00:00 windows-spain-dc-1

What's next