Collect Netgate pfSense logs

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

Supported in:

Google SecOps SIEM

This document describes how you can collect the Netgate pfSense logs by using a Google Security Operations forwarder.

For more information, see Data ingestion to Google SecOps.

An ingestion label identifies the parser that normalizes raw log data to structured UDM format. The information in this document applies to the parser with the PFSENSE ingestion label.

Configure Netgate pfSense

1. Sign in to the pfSense interface with administrator credentials.
2. In the Global log settings, do the following:
   1. Select Status > System logs > Settings.
   2. On the Settings tab, select General logging options and do the following:
      1. In the Log message format field, select syslog (RFC 5424, with RFC 3339 microsecond-precision timestamps).
      2. In the Log firewall default blocks field, select the following checkboxes:
         • Log packets matched from the default block rules in the ruleset
         • Log packet matched from the default pass rules put in the ruleset
         • Log packets blocked by 'Block bogon networks' rules
         • Log packets blocked by 'Block private networks' rules
3. To save the new settings, click Save.

Send the firewall logs to Google SecOps

1. In the Global log settings, select Status > System logs > Settings.
2. On the Settings tab, select Remote logging options and do the following:
   • In the Enable remote logging field, select the Send log messages to remote syslog server checkbox.
   • In the Source address field, keep the default value as Any.
   • In the IP Protocol field, if the source address is selected as the default value, you don't need to specify the IP address. Otherwise, enter the IP address.
   • In the Remote log servers, specify Google SecOps forwarder IP address.
   • In Remote syslog contents, select the Firewall events checkbox.
3. To save the changes, click Save.

Configure the Google SecOps forwarder to ingest Netgate pfSense logs

1. Select SIEM Settings > Forwarders.
2. Click Add new forwarder.
3. In the Forwarder name field, enter a unique name for the forwarder.
4. Click Submit and then click Confirm. The forwarder is added and the Add collector configuration window appears.
5. In the Collector name field, type a unique name for the collector.
6. Select pfSense as the Log type.
7. In the Collector type field, select Syslog.
8. Configure the following mandatory input parameters:
   • Protocol: specify the connection protocol that the collector uses to listen for syslog data.
   • Address: specify the target IP address or hostname where the collector resides and listens to syslog data.
   • Port: specify the target port where the collector resides and listens to syslog data.
9. Click Submit.

For more information about the Google SecOps forwarders, see Manage forwarder configurations through the Google SecOps UI.

If you encounter issues when you create forwarders, contact Google SecOps support.

Field mapping reference

This pfSense firewall parser extracts fields from syslog messages using grok patterns, handling various log formats including DHCP and firewall events. It then maps these extracted fields to the UDM, enriching the data with contextual information like IP addresses, MAC addresses, usernames, and network details.

UDM mapping table

Log Field	UDM Mapping	Logic
application	principal.application	The value is extracted from the log message using grok patterns. For syslog messages, the application name is typically found after the hostname and timestamp.
command	principal.process.command_line	Extracted from the description field when the log indicates a command execution.
description	metadata.description	The description field is mapped to the UDM metadata description, except for syslog-ng application logs where it's mapped to metadata.description. For DHCP events, the dhcp_type is prepended to the description.
dhcp_type	metadata.product_event_type	The DHCP message type (e.g., DHCPDISCOVER, DHCPOFFER) is extracted and mapped.
host	intermediary.hostname OR intermediary.ip	If the host value is a valid IP address, it's mapped to intermediary.ip. Otherwise, it's mapped to intermediary.hostname.
host	principal.hostname, principal.asset.hostname	When no principal IP is present, the host is treated as the principal hostname.
mac	principal.mac, network.dhcp.chaddr	The MAC address associated with a DHCP request is extracted and mapped.
src_ip	principal.ip, principal.asset.ip	Extracted from specific log formats using a grok pattern.
src_mac	principal.mac	Extracted from specific log formats using a grok pattern.
dst_mac	target.mac	Extracted from specific log formats using a grok pattern.
timestamp	metadata.event_timestamp	The timestamp is extracted from the log message and converted to a UDM timestamp format. The timezone information (tz) is appended to the timestamp before conversion if available.
timestamp_no_year	metadata.event_timestamp	If a timestamp without a year is present, it's parsed, and the current year is added during the parsing process.
user	principal.user.userid	The username associated with an event is extracted and mapped.
column1	security_result.rule_id	Mapped from the first CSV column if the description is in CSV format.
column6	security_result.rule_type	Mapped from the sixth CSV column if the description is in CSV format.
column7	security_result.action	Mapped from the seventh CSV column if the description is in CSV format. Converted to "BLOCK" or "ALLOW".
column8	network.direction	Mapped from the eighth CSV column if the description is in CSV format. Converted to "INBOUND" or "OUTBOUND".
column13	network.ip_protocol (if UDP or ICMP)	Mapped from the thirteenth CSV column if the description is in CSV format and the protocol is UDP or ICMP. For TCP/UDP events, it's used to create an additional field with key "Id".
column16	principal.ip, principal.asset.ip (if IPv6 and column9 is 6)	Mapped from the sixteenth CSV column if the description is in CSV format and column9 is 6. For TCP/UDP events, it's used for protocol identification if column9 is 4.
column17	target.ip, target.asset.ip (if IPv6 and not ip_failure)	Mapped from the seventeenth CSV column if the description is in CSV format, column9 is 6, and the value is a valid IP. For TCP/UDP events, it's used for protocol identification.
column18	principal.port (if UDP)	Mapped from the eighteenth CSV column if the description is in CSV format and the protocol is UDP. For TCP/UDP events, it's mapped to network.received_bytes.
column19	target.port (if UDP)	Mapped from the nineteenth CSV column if the description is in CSV format and the protocol is UDP. For DHCP events, it's mapped to network.dhcp.yiaddr. For other events, it's mapped to principal.ip, principal.asset.ip.
column20	additional.fields (key: "data_length") (if UDP)	Mapped from the twentieth CSV column if the description is in CSV format and the protocol is UDP. For other events, it's mapped to target.ip, target.asset.ip.
column21	principal.port (if TCP/UDP)	Mapped from the twenty-first CSV column if the description is in CSV format and the protocol is TCP or UDP.
column22	target.port (if TCP/UDP)	Mapped from the twenty-second CSV column if the description is in CSV format and the protocol is TCP or UDP.
column23	additional.fields (key: "data_length") (if TCP/UDP)	Mapped from the twenty-third CSV column if the description is in CSV format and the protocol is TCP or UDP.
column24	additional.fields (key: "tcp_flags") (if TCP)	Mapped from the twenty-fourth CSV column if the description is in CSV format and the protocol is TCP.
column25	additional.fields (key: "sequence_number") (if TCP/UDP)	Mapped from the twenty-fifth CSV column if the description is in CSV format and the protocol is TCP or UDP.
column29	additional.fields (key: "tcp_options") (if TCP)	Mapped from the twenty-ninth CSV column if the description is in CSV format and the protocol is TCP.
compression_algo	additional.fields (key: "Compression Algorithm")	Extracted from the description field and added as an additional field.
desc	metadata.description	Extracted from the message field and used as the description.
principal_ip	principal.ip, principal.asset.ip	Extracted from the description field and represents the principal IP address.
principal_username	principal.user.userid	Extracted from the description field and represents the principal username.
status	security_result.detection_fields (key: "status")	Extracted from the description field and added as a detection field within the security result.
target_host	target.hostname, target.asset.hostname	Extracted from the description field and represents the target hostname.
src_port	principal.port	Extracted from the description field and represents the source port. Determined based on various log fields and parser logic. Can be NETWORK_CONNECTION, NETWORK_DHCP, STATUS_UPDATE, or GENERIC_EVENT. Hardcoded to "PFSENSE". Hardcoded to "PFSENSE". Hardcoded to "PFSENSE". Set to "DHCP" for DHCP events. Set to "BOOTREQUEST" for DHCPDISCOVER and DHCPREQUEST, and "BOOTREPLY" for DHCPOFFER and DHCPACK. Set to "DISCOVER", "REQUEST", "OFFER", or "ACK" based on the dhcp_type field.

Changes

2024-05-08

• Added Grok patterns to parse field "description".
• Mapped "principal_ip" to "principal.ip" and "principal.asset.ip".
• Mapped "src_port" to "principal.port".
• Mapped "compression_algo" to "additional.fields".
• Mapped "status" to "security_result.detection_fields".
• Mapped "principal_username" to "principal.user.userid".
• Mapped "target_host" to "target.hostname" and "target.asset.hostname".

2023-05-05

• Added convert to uppercase before mapping "network.ip_protocol".
• Mapped "column18" to "principal.port" when protocol is present in "column13".
• Mapped "column19" to "target.port" when protocol is present in "column13".
• Mapped "column20" to "additional.fields" as "data-length" when protocol is present in "column13".

2023-02-20

• Added Grok pattern to support new filter-log format and syslog-ng format.

2022-10-04

• Remapped firewall device name to intermediary.hostname instead of principal.hostname for logs where event_types is "NETWORK_CONNECTION".

2022-09-05

• Enhancement-
• for csv format logs mapped following fields.
• Added grok pattern to retrieve "IP" and "MAC".
• Mapped "column19" which is "source-address" to "network.dhcp.yiaddr".
• Mapped "security_result.action" to "ALLOW" when "column7" is equal to "pass".
• When "column9" is equals to "6" which indicates "IPV6", then following fields are mapped:
• Mapped "column17" which is "destination-address" to "target.ip".
• Mapped "column16" which is "source-address" to "principal.ip".
• Mapped "event_type" to "NETWORK_CONNECTION" when "column16" and "column17" is not null.
• Mapped "column12" which is "hop_limit" to "additional.fields".
• Mapped "column13" which is "ip_protocol" to "network.ip_protocol".
• Migrated the custom parsers into default parser.
• Added conditional check to set "event_type" to "STATUS_UPDATE"

2022-06-30

• Mapped "ttl" to "additional.fields".
• Mapped "Id" to "additional.fields".
• Mapped "Offset" to "additional.fields".
• Mapped "Data length" to "additional.fields".
• Mapped "Length" to "additional.fields".
• Mapped "Sequence-number" to "additional.fields".

2022-04-11

• Newly created parser