Ingestion metrics schema

This document describes the Google Security Operations ingestion metrics fields and their related UDM events for supported ingestion components (collection mechanisms).

Ingestion metrics components

Ingestion components are services or pipelines that ingest logs into the platform from source log feeds. Each ingestion component collects a different set of log fields into its own ingestion metrics schema. These log fields are the dimension fields that appear in the ingestion metrics Explore interface, when creating new dashboards.

The following sections describe the ingestion metrics schemas and dimension fields for the following ingestion components: Forwarder, Ingestion API, Collection agent, Normalizer, and Out-of-band processor (Chronicle API feed).

Note: All log flows use the Ingestion API. Therefore, to get an accurate measure of the total volume ingested, refer to the component="Ingestion API".
The UDM events explorer provides exact counts for events.

Note: Aggregate ingestion metrics cannot be used as a true source of exact counts for any given time range, across components. For example, event_count and log_count are aggregate metrics pulled from the processing components at regular intervals. Such values should only be treated as indicators of the approximate volume processed by the components.

Forwarder ingestion schema

Fields	Type	Description
component	STRING	Forwarder, is the ingestion service or pipeline type ingesting log entities into the platform.
collector_id	STRING	The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
log_type	STRING	The log source type identifying log entries in a batch. For example, WINDOWS_DNS.
input_type	STRING	This field is populated if the ingestion source is the Google Security Operations forwarder. Based on the data that the forwarder sends, this field contains pcap, syslog, or splunk.
drop_reason_code	STRING	The reason for dropping the log.
last_heartbeat_time	TIMESTAMP	The last timestamp at which the forwarder or API feed was active, in microseconds. This field is populated if the ingestion source is the Google Security Operations forwarder or Chronicle API feed. When the feed is active, it populates the last_heartbeat_time field, and the log_count and log_volume fields remain empty.
log_volume	FLOAT64	The volume of logs during the interval, in bytes. The log_volume field remains empty or is populated in the following cases: This field is populated when the Google Security Operations forwarder or the feed sends data. The last_heartbeat_time field remains empty. If the feed is inactive, no entry is made in the ingestion metrics table. When a feed is active, the last_heartbeat_time, log_count, or log_volume field is populated.
drop_count	FLOAT64	The number of logs dropped for the customer.
log_count	FLOAT64	The number of logs ingested during the interval. The log_count field remains empty or is populated in the following cases: This field is populated when the Google Security Operations forwarder or the feed sends data. The last_heartbeat_time field remains empty. If the feed is inactive, no entry is made in the ingestion metrics table. When a feed is active, the last_heartbeat_time, log_count, or log_volume field is populated.
memory_used	FLOAT64	The percentage of memory used by the forwarder container.
disk_used	FLOAT64	Percentage of disk used by the forwarder container.
cpu_used	FLOAT64	The percentage of CPU used by the forwarder container.
buffer_used	FLOAT64	Percentage of buffer used by the forwarder buffer type.
buffer_type	STRING	The type of buffer used in the forwarder.

Ingestion API schema

Fields	Type	Description
component	STRING	Ingestion API, is the ingestion service or pipeline type ingesting log entities into the platform.
collector_id	STRING	The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
log_type	STRING	The log source type identifying log entries in a batch. For example, WINDOWS_DNS.
namespace	STRING	The namespace that the log belongs to.
log_volume	FLOAT64	The size of the logs received for the customer by the Ingestion API, in bytes.
log_count	FLOAT64	The number of logs received for the customer by the Ingestion API.
quota_limit_per_second	FLOAT64	The quota limits set by the customer, enforced by the Ingestion API.
quota_rejected_long_term_log_volume	FLOAT64	The size of the logs rejected by the Ingestion API due to insufficient quota, for the LONG_TERM_DAILY_LIMIT quota type, in bytes.
quota_rejected_short_term_log_volumed	FLOAT64	The size of the logs rejected by the Ingestion API due to insufficient quota, for the SHORT_TERM_DAILY_LIMIT quota type, in bytes.
ingestion_source	STRING	The ingestion source in the **ingestion label** when the logs are ingested through ingestion private API.

Collection agent

Fields	Type	Description
component	STRING	Collection Agent, is the ingestion service or pipeline type ingesting log entities into the platform.
collector_id	STRING	The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
log_type	STRING	The log source type identifying log entries in a batch. For example, WINDOWS_DNS.
input_type	STRING	This field is populated if the ingestion source is the Google Security Operations forwarder. Based on the data that the forwarder sends, this field contains pcap, syslog, or splunk.
drop_count	FLOAT64	The number of spans refused by the agent exporter.
log_count	FLOAT64	The number of spans accepted by the agent exporter.
memory_used	FLOAT64	Memory occupied by the agent process, in kilobytes.
cpu_used	FLOAT64	CPU time spent on the agent process, in seconds.
buffer_used	FLOAT64	Queue size of the agent exporter.
buffer_capacity	FLOAT64	Queue capacity of the agent exporter.
process_uptime	FLOAT64	The number of seconds that the agent process has been running.

Normalizer ingestion schema

Fields	Type	Description
component	STRING	Normalizer, is the ingestion service or pipeline type ingesting log entities into the platform.
collector_id	STRING	The unique identifier of the collection mechanism. For push sources, the forwarder ID or generated ID is used. For Chronicle API or Chronicle API feed, the ID has the following format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.
state	STRING	The final status of the event or log. The status is one of the following: parsed. The log is successfully parsed. validated. The log is successfully validated. failed_parsing. The log has parsing errors. failed_validation. The log has validation errors. failed_indexing. The log has batch indexing errors.
log_type	STRING	The log source type identifying log entries in a batch. For example, WINDOWS_DNS.
event_type	STRING	The event type determines which fields are included with the event. The event type includes values such as PROCESS_OPEN, FILE_CREATION, USER_CREATION, and NETWORK_DNS.
drop_reason_code	STRING	The reason for dropping the log.
log_volume	FLOAT64	The total size of the log entries received for parsing, in bytes.
log_count	FLOAT64	The number of log entries received for parsing.
event_count	FLOAT64	The number of events generated during the interval.
latency_count	FLOAT64	The number of values in a latency distribution, of the difference in time between ingestion and normalization.
buckets	FLOAT64	The number of values in each bucket in a latency distribution, of the difference in time between ingestion and normalization.
bucketer_num_finite_buckets	FLOAT64	The number of buckets in a latency distribution, of the difference in time between ingestion and normalization.
bucketer_growth_factor	FLOAT64	The bucketer growth factor in a latency distribution, of the difference in time between ingestion and normalization.
bucketer_scale_factor	FLOAT64	The bucketer scale factor in a latency distribution, of the difference in time between ingestion and normalization.
latency_overflow	FLOAT64	The overflow bucket in a latency distribution, of the difference in time between ingestion and normalization.
latency_underflow	FLOAT64	The underflow bucket in a latency distribution, of the difference in time between ingestion and normalization.

Out-of-band processor (Chronicle API feed) ingestion schema

Fields	Type	Description
component	STRING	Out-of-band processor (Chronicle API feed), is the ingestion service or pipeline type ingesting log entities into the platform.
feed_id	STRING	The id of the specific Xenon/Gopher feed that a log belongs to.
log_type	STRING	The log source type identifying log entries in the batch. For example, WINDOWS_DNS.
last_heartbeat_time	TIMESTAMP	The epoch timestamp of the successful ingestion of the log entry, in seconds. When the feed is active, the last_heartbeat_time field is populated, and the log_count and log_volume fields remain empty.
log_volume	FLOAT64	The size of the logs received in the out-of-band processor, in bytes. The log_volume field remains empty or is populated in the following cases: This field is populated when the feed sends data. The last_heartbeat_time field remains empty. If the feed is inactive, no entry is made in the ingestion metrics table. When the feed is active, the last_heartbeat_time, log_count, or log_volume field is populated.
log_count	FLOAT64	The number of logs processed in the out-of-band processor. The log_count field remains empty or is populated in the following cases: This field is populated when the feed sends data. The last_heartbeat_time field remains empty. If the feed is inactive, no entry is made in the ingestion metrics table. When a feed is active, the last_heartbeat_time, log_count, or log_volume field is populated.

Filtering ingestion metrics

You can filter ingestion metrics based on the field values. For example, out-of-band processor feeds have collector_id as aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa. Here is an example query to filter out-of-band feeds:

SELECT
  component,
  collector_id,
  count(component)
FROM
  chronicle-tla.datalake.ingestion-metrics
WHERE
  DATE(start_time) = DATE_SUB(CURRENT_DATE(), INTERVAL 60 DAY)
  AND component IN ("Out-of-Band Processor","Ingestion API", "Forwarder")
  AND (collector_id != "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
  OR collector_id is null)
group by 1,2

Note: start_time and end_time indicate the time interval for which the metrics were pulled.

Ingestion metrics examples

The following table shows some metrics and example values to help you understand the ingestion_metrics schema fields:

Metrics	component	collector_id	feed_id	log_type	start_time	end_time	input_type	last_heartbeat_time	log_volume	drop_count	log_count	memory_used	cpu_used	disk_used	buffer_used	ingestion_source	drop_reason_code
Heartbeat	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00	syslog	2022-04-21T13:18:55.000+00:00
Log Bytes Count	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00	pcap		149.0
Log Record Count	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00	pcap				154.0
Drop Count (Backlog)	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00	pcap			4.0							backlog
Drop Count (Invalid Config)	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00	pcap			4.0							invalid_config
Drop Count (Regex)	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00	pcap			4.0							regex
Log Record Count	Ingestion API	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DHCP	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00					3578.0
Log Bytes Count	Ingestion API	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DHCP	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00			2802.0
Log Record Count	Out-of-Band Processor		feeds/aaaaaaaaaaaaaa	ARUBA_IPS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00					3578.0
Log Bytes Count	Out-of-Band Processor		feeds/aaaaaaaaaaaaaa	ARUBA_IPS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00			319563.0
Last Ingested Timestamp	Out-of-Band Processor		feeds/aaaaaaaaaaaaaa	ARUBA_IPS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00		2022-04-21T13:18:55.000+00:00
Log Count	Normalizer	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00
Log Size	Normalizer	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00
Event Count	Normalizer	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00
Container Memory Used	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00						0.32
Container Disk Used	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00								0.5
Container CPU Used	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00							0.545
Buffer Used	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00									0.562
Ingestion Source	Forwarder	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx		PCAP_DNS	2022-04-21T13:14:50.924+00:00	2022-04-21T13:19:50.924+00:00										windows-spain-dc-1

What's next

• Read about Ingestion metrics field reference for dashboards.