Ingestion metrics schema
This document describes the Google Security Operations ingestion metrics fields and their related UDM events for supported ingestion components (collection mechanisms).
Ingestion metrics components
Ingestion components are services or pipelines that ingest logs into the platform from source log feeds. Each ingestion component collects a different set of log fields into its own ingestion metrics schema. These log fields are the dimension fields that appear in the ingestion metrics Explore interface, when creating new dashboards.
The following sections describe the ingestion metrics schemas and dimension fields for the following ingestion components: Forwarder, Ingestion API, Collection agent, Normalizer, and Out-of-band processor (Chronicle API feed).
Forwarder ingestion schema
Fields | Type | Description |
---|---|---|
component
|
STRING | Forwarder , is the ingestion service or pipeline type
ingesting log entities into the platform. |
collector_id
|
STRING | The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. |
log_type
|
STRING | The log source type identifying log entries in a batch. For example,
WINDOWS_DNS .
|
input_type
|
STRING | This field is populated if the ingestion source is the Google Security Operations
forwarder. Based on the data that the forwarder sends, this field
contains pcap ,
syslog , or splunk .
|
drop_reason_code
|
STRING | The reason for dropping the log. |
last_heartbeat_time
|
TIMESTAMP | The last timestamp at which the forwarder or API feed was active, in microseconds. This field is populated if the ingestion source is the Google Security Operations forwarder or Chronicle API feed. When the feed is active, it populates the |
log_volume
|
FLOAT64 | The volume of logs during the interval, in bytes. The
|
drop_count
|
FLOAT64 | The number of logs dropped for the customer. |
log_count
|
FLOAT64 | The number of logs ingested during the interval. The
|
memory_used |
FLOAT64 | The percentage of memory used by the forwarder container. |
disk_used |
FLOAT64 | Percentage of disk used by the forwarder container. |
cpu_used |
FLOAT64 | The percentage of CPU used by the forwarder container. |
buffer_used |
FLOAT64 | Percentage of buffer used by the forwarder buffer type. |
buffer_type |
STRING | The type of buffer used in the forwarder. |
Ingestion API schema
Fields | Type | Description |
---|---|---|
component
|
STRING | Ingestion API , is the ingestion service or pipeline type
ingesting log entities into the platform. |
collector_id
|
STRING | The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. |
log_type
|
STRING | The log source type identifying log entries in a batch. For
example, WINDOWS_DNS .
|
namespace
|
STRING | The namespace that the log belongs to. |
log_volume
|
FLOAT64 | The size of the logs received for the customer by the Ingestion API, in bytes. |
log_count
|
FLOAT64 | The number of logs received for the customer by the Ingestion API. |
quota_limit_per_second
|
FLOAT64 | The quota limits set by the customer, enforced by the Ingestion API. |
quota_rejected_long_term_log_volume |
FLOAT64 | The size of the logs rejected by the Ingestion API due to insufficient quota, for the LONG_TERM_DAILY_LIMIT quota type, in bytes. |
quota_rejected_short_term_log_volumed |
FLOAT64 | The size of the logs rejected by the Ingestion API due to insufficient quota, for the SHORT_TERM_DAILY_LIMIT quota type, in bytes. |
ingestion_source |
STRING | The ingestion source in the **ingestion label** when the logs are ingested through ingestion private API. |
Collection agent
Fields | Type | Description |
---|---|---|
component
|
STRING | Collection Agent , is the ingestion service or pipeline type
ingesting log entities into the platform. |
collector_id
|
STRING | The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. |
log_type
|
STRING | The log source type identifying log entries in a batch. For
example, WINDOWS_DNS .
|
input_type
|
STRING | This field is populated if the ingestion source is the Google Security Operations
forwarder. Based on the data that the forwarder sends, this field
contains pcap ,
syslog , or splunk .
|
drop_count
|
FLOAT64 | The number of spans refused by the agent exporter. |
log_count
|
FLOAT64 | The number of spans accepted by the agent exporter. |
memory_used |
FLOAT64 | Memory occupied by the agent process, in kilobytes. |
cpu_used |
FLOAT64 | CPU time spent on the agent process, in seconds. |
buffer_used |
FLOAT64 | Queue size of the agent exporter. |
buffer_capacity |
FLOAT64 | Queue capacity of the agent exporter. |
process_uptime |
FLOAT64 | The number of seconds that the agent process has been running. |
Normalizer ingestion schema
Fields | Type | Description |
---|---|---|
component
|
STRING | Normalizer , is the ingestion service or pipeline type
ingesting log entities into the platform. |
collector_id
|
STRING | The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. |
state
|
STRING | The final status of the event or log. The status is one of the following:
|
log_type
|
STRING | The log source type identifying log entries in a batch. For example,
WINDOWS_DNS .
|
event_type
|
STRING | The event type determines which fields are included with the event. The
event type includes values such as PROCESS_OPEN ,
FILE_CREATION , USER_CREATION , and
NETWORK_DNS .
|
drop_reason_code
|
STRING | The reason for dropping the log. |
log_volume
|
FLOAT64 | The total size of the log entries received for parsing, in bytes. |
log_count
|
FLOAT64 | The number of log entries received for parsing. |
event_count
|
FLOAT64 | The number of events generated during the interval. |
latency_count |
FLOAT64 | The number of values in a latency distribution, of the difference in time between ingestion and normalization. |
buckets |
FLOAT64 | The number of values in each bucket in a latency distribution, of the difference in time between ingestion and normalization. |
bucketer_num_finite_buckets |
FLOAT64 | The number of buckets in a latency distribution, of the difference in time between ingestion and normalization. |
bucketer_growth_factor |
FLOAT64 | The bucketer growth factor in a latency distribution, of the difference in time between ingestion and normalization. |
bucketer_scale_factor |
FLOAT64 | The bucketer scale factor in a latency distribution, of the difference in time between ingestion and normalization. |
latency_overflow |
FLOAT64 | The overflow bucket in a latency distribution, of the difference in time between ingestion and normalization. |
latency_underflow |
FLOAT64 | The underflow bucket in a latency distribution, of the difference in time between ingestion and normalization. |
Out-of-band processor (Chronicle API feed) ingestion schema
Fields | Type | Description |
---|---|---|
component
|
STRING | Out-of-band processor (Chronicle API feed), is the
ingestion service or pipeline type ingesting log entities into the platform. |
feed_id
|
STRING | The id of the specific Xenon/Gopher feed that a log belongs to. |
log_type
|
STRING | The log source type identifying log entries in the batch. For
example, WINDOWS_DNS .
|
last_heartbeat_time
|
TIMESTAMP | The epoch timestamp of the successful ingestion of the log entry, in seconds. When the feed is active, the |
log_volume
|
FLOAT64 | The size of the logs received in the out-of-band processor, in bytes. The
|
log_count
|
FLOAT64 | The number of logs processed in the out-of-band processor. The
|
Filtering ingestion metrics
You can filter ingestion metrics based on the field values. For example, out-of-band processor
feeds have collector_id
as aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
. Here is an example
query to filter out-of-band feeds:
SELECT component, collector_id, count(component) FROM chronicle-tla.datalake.ingestion-metrics WHERE DATE(start_time) = DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY) AND component IN ("Out-of-Band Processor","Ingestion API", "Forwarder") AND (collector_id != "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa" OR collector_id is null) group by 1,2
Ingestion metrics examples
The following table shows some metrics and example values to help you understand the ingestion_metrics schema fields:
Metrics | component | collector_id | feed_id | log_type | start_time | end_time | input_type | last_heartbeat_time | log_volume | drop_count | log_count | memory_used | cpu_used | disk_used | buffer_used | ingestion_source | drop_reason_code |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Heartbeat | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | syslog | 2022-04-21T13:18:55.000+00:00 | ||||||||||
Log Bytes Count | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 149.0 | ||||||||||
Log Record Count | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 154.0 | ||||||||||
Drop Count (Backlog) | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 4.0 | backlog | |||||||||
Drop Count (Invalid Config) | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 4.0 | invalid_config | |||||||||
Drop Count (Regex) | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | pcap | 4.0 | regex | |||||||||
Log Record Count | Ingestion API | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DHCP | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 3578.0 | |||||||||||
Log Bytes Count | Ingestion API | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DHCP | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 2802.0 | |||||||||||
Log Record Count | Out-of-Band Processor | feeds/aaaaaaaaaaaaaa | ARUBA_IPS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 3578.0 | |||||||||||
Log Bytes Count | Out-of-Band Processor | feeds/aaaaaaaaaaaaaa | ARUBA_IPS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 319563.0 | |||||||||||
Last Ingested Timestamp | Out-of-Band Processor | feeds/aaaaaaaaaaaaaa | ARUBA_IPS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 2022-04-21T13:18:55.000+00:00 | |||||||||||
Log Count | Normalizer | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | ||||||||||||
Log Size | Normalizer | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | ||||||||||||
Event Count | Normalizer | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | ||||||||||||
Container Memory Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.32 | |||||||||||
Container Disk Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.5 | |||||||||||
Container CPU Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.545 | |||||||||||
Buffer Used | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | 0.562 | |||||||||||
Ingestion Source | Forwarder | xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | PCAP_DNS | 2022-04-21T13:14:50.924+00:00 | 2022-04-21T13:19:50.924+00:00 | windows-spain-dc-1 |
What's next
- Read about Ingestion metrics field reference for dashboards.