Collect Linux auditd and AIX systems logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This parser handles Linux audit logs in SYSLOG format, transforming them into UDM. It processes both JSON-formatted and plain text log messages, extracting fields using grok, XML parsing, and JSON parsing techniques, and mapping them to appropriate UDM fields based on the event type. The parser also handles specific audit log formats from AIX systems and enriches the UDM with additional fields like security_result and intermediary details.

Before you begin

• Ensure that you have a Google Security Operations instance.
• Ensure that you have root access to the Auditd host.
• Ensure that you installed rsyslog on the Auditd host.
• Ensure that you have a Windows 2012 SP2 or later or Linux host with systemd.
• If running behind a proxy, ensure that the firewall ports are open.

Get Google SecOps ingestion authentication file

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Collection Agents.
3. Download the Ingestion Authentication File.

Get Google SecOps customer ID

1. Sign in to the Google SecOps console.
2. Go to SIEM Settings > Profile.
3. Copy and save the Customer ID from the Organization Details section.

Install BindPlane Agent

1. For Windows installation, run the following script:
   msiexec /i "https://github.com/observIQ/bindplane-agent/releases/latest/download/observiq-otel-collector.msi" /quiet
2. For Linux installation, run the following script:
   sudo sh -c "$(curl -fsSlL https://github.com/observiq/bindplane-agent/releases/latest/download/install_unix.sh)" install_unix.sh
3. Additional installation options can be found in this installation guide.

Configure BindPlane Agent to ingest Syslog and send to Google SecOps

1. Access the machine where BindPlane is installed.

2. Edit the config.yaml file as follows:

   receivers:
     tcplog:
       # Replace the below port <54525> and IP <0.0.0.0> with your specific values
       listen_address: "0.0.0.0:54525" 
   
   exporters:
       chronicle/chronicle_w_labels:
           compression: gzip
           # Adjust the creds location below according the placement of the credentials file you downloaded
           creds: '{ json file for creds }'
           # Replace <customer_id> below with your actual ID that you copied
           customer_id: <customer_id>
           endpoint: malachiteingestion-pa.googleapis.com
           # You can apply ingestion labels below as preferred
           ingestion_labels:
           log_type: SYSLOG
           namespace: auditd
           raw_log_field: body
   service:
       pipelines:
           logs/source0__chronicle_w_labels-0:
               receivers:
                   - tcplog
               exporters:
                   - chronicle/chronicle_w_labels
   

3. Restart BindPlane Agent to apply the changes using the following command: sudo systemctl bindplane restart

Exporting Syslog from Auditd

1. Access the machine from which you want to export audit logs.

2. Open the Auditd configuration file (typically located at /etc/audit/auditd.conf).

   sudo vi /etc/audit/auditd.conf
   

3. Find or add the following line(s) to configure auditd:

   active = yes
   output = syslog
   log_format = ENRICHED
   dispatcher = /sbin/audispd
   

Optional: Specify Syslog Facility: Add or modify the following line in auditd.conf:

```none
syslog_facility = LOG_AUTHPRIV
```

Note: Common facilities include LOG_AUTHPRIV (for security/authorization messages), LOG_LOCAL0 through LOG_LOCAL7 (for custom applications).

1. Open audispd configuration file (typically located at /etc/audisp/plugins.d/syslog.conf):

   sudo vi /etc/audisp/plugins.d/syslog.conf
   

2. Find or add the following line(s) to configure audispd:

   active = yes
   direction = out
   path = builtin_syslog
   type = builtin
   args = LOG_INFO
   format = string
   

3. Restart Auditd service to apply the changes:

   sudo systemctl restart auditd
   

4. Use a tool like tail to monitor the syslog and verify that Auditd logs are being sent:

   tail -f /var/log/syslog | grep auditd # Follow syslog and filter for auditd messages (path may vary depending on your system)
   

5. Edit the rsyslog.conf or create a custom configuration:

   sudo vi /etc/rsyslog.d/50-audit-forwarding.conf
   

6. Add a rule to forward logs:

   • if $programname == 'auditd' then @@<BindPlane_Agent>:<BindPlane_Port>
   • Use @for UDP or @@ for TCP
   • Replace <BindPlane_Agent> with your server's IP/hostname.
   • Replace <BindPlane_Port> with your server's Port.

7. Restart rsyslog service to apply the changes:

   sudo systemctl restart rsyslog
   

UDM Mapping Table

Log Field	UDM Mapping	Logic
acct	target.user.user_display_name	The value of acct from the raw log is mapped to the target.user.user_display_name field in the UDM. This represents the account associated with the event.
addr	principal.ip	The value of addr from the raw log is mapped to the principal.ip field in the UDM. This represents the IP address of the principal involved in the event.
additional.fields	additional.fields	Additional fields from parsed key-value pairs or labels are added to the additional.fields array in the UDM.
agent.googleapis.com/log_file_path	(Not Mapped)	This label is present in some raw logs but is not mapped to the IDM object in the UDM.
algo	(Not used in this example)	Although present in the parser and some raw logs, this field isn't used in the provided example and doesn't appear in the final UDM.
application	principal.application	Derived from the terminal field in the raw log or other fields like exe depending on the log type. Represents the application involved.
arch	security_result.about.platform_version	The architecture from the raw log's arch field is mapped to security_result.about.platform_version.
auid	about.user.userid, security_result.detection_fields.auid	The audit user ID (auid) is mapped to about.user.userid and added as a detection field in security_result.
cmd	target.process.command_line	The command from the raw log's cmd field is mapped to target.process.command_line.
collection_time	(Not Mapped)	This field is the log collection time and is not mapped to the IDM object in the UDM.
comm	principal.application	The command name (comm) is mapped to principal.application.
compute.googleapis.com/resource_name	principal.hostname	The resource name from this label is mapped to principal.hostname.
create_time	(Not Mapped)	This field is not mapped to the IDM object in the UDM.
cwd	security_result.detection_fields.cwd	The current working directory (cwd) is added as a detection field in security_result.
data	(Processed)	The data field contains the main log message and is processed by the parser to extract various fields. It is not directly mapped to a single UDM field.
exe	principal.process.file.full_path, target.process.file.full_path	The executable path (exe) is mapped to either principal.process.file.full_path or target.process.file.full_path depending on the context.
extensions.auth.type	extensions.auth.type	The authentication type is set by the parser logic based on the event type. Often set to MACHINE or AUTHTYPE_UNSPECIFIED.
fp	network.tls.client.certificate.sha256	The fingerprint (fp) is parsed to extract the SHA256 hash and mapped to network.tls.client.certificate.sha256.
insertId	(Not Mapped)	This field is not mapped to the IDM object in the UDM.
jsonPayload.message	(Processed)	This field contains the main log message in JSON format and is processed by the parser.
key	security_result.about.registry.registry_key	The key field is mapped to security_result.about.registry.registry_key.
labels	(Processed)	Labels from the raw log are processed and mapped to various UDM fields or added to additional.fields.
logName	(Not Mapped)	This field is not mapped to the IDM object in the UDM.
msg	security_result.summary	The message (msg) is often used to populate the security_result.summary field.
network.application_protocol	network.application_protocol	Set by the parser logic based on the event type (e.g., SSH, HTTP).
network.direction	network.direction	Set by the parser logic based on the event type (e.g., INBOUND, OUTBOUND).
network.ip_protocol	network.ip_protocol	Set by the parser logic, usually to TCP for SSH events.
network.session_id	network.session_id	Mapped from the ses field or derived from other fields.
network.tls.cipher	network.tls.cipher	The cipher information is extracted from the raw log and mapped to this field.
network.tls.curve	network.tls.curve	The key exchange curve is extracted from the raw log and mapped to this field.
pid	principal.process.pid, target.process.pid	The process ID (pid) is mapped to either principal.process.pid or target.process.pid depending on the context.
ppid	principal.process.parent_process.pid, target.process.parent_process.pid	The parent process ID (ppid) is mapped to either principal.process.parent_process.pid or target.process.parent_process.pid depending on the context.
principal.asset.hostname	principal.asset.hostname	Copied from principal.hostname.
principal.asset.ip	principal.asset.ip	Copied from principal.ip.
principal.platform	principal.platform	Set by the parser logic based on the operating system (e.g., LINUX).
principal.port	principal.port	The port number associated with the principal.
principal.user.group_identifiers	principal.user.group_identifiers	Group IDs associated with the principal user.
receiveTimestamp	(Not Mapped)	This field is the log receive timestamp and is not mapped to the IDM object in the UDM.
res	security_result.action_details	The result (res) is mapped to security_result.action_details.
resource.labels	(Not Mapped)	These labels are present in some raw logs but are not mapped to the IDM object in the UDM.
resource.type	(Not Mapped)	This field is present in some raw logs but is not mapped to the IDM object in the UDM.
security_result.action	security_result.action	Set by the parser logic based on the res field (e.g., ALLOW, BLOCK).
security_result.detection_fields	security_result.detection_fields	Various fields from the raw log are added as key-value pairs to this array for context.
security_result.rule_id	security_result.rule_id	Set by the parser logic, often to the type_name for syscall events.
security_result.severity	security_result.severity	Set by the parser logic based on the severity level in the raw log.
security_result.summary	security_result.summary	A summary of the event, often derived from the msg field or other relevant fields.
ses	network.session_id	The session ID (ses) is mapped to network.session_id.
source	(Not Mapped)	This field contains metadata about the log source and is not mapped to the IDM object in the UDM.
subj	(Processed)	The subject field (subj) is processed to extract user and security context information.
syscall	security_result.about.labels.Syscall	The syscall number is added as a label within security_result.about.
target.administrative_domain	target.administrative_domain	The domain of the target user.
target.group.group_display_name	target.group.group_display_name	The name of the target group.
target.ip	target.ip	The IP address of the target.
target.port	target.port	The port number associated with the target.
target.process.command_line	target.process.command_line	The command line of the target process.
target.resource.type	target.resource.type	The type of the target resource, set by the parser logic (e.g., CREDENTIAL, SETTING).
target.user.attribute.permissions	target.user.attribute.permissions	Permissions related to the target user.
target.user.group_identifiers	target.user.group_identifiers	Group IDs associated with the target user.
target.user.userid	target.user.userid	The user ID of the target.
textPayload	(Processed)	The text payload of the log, processed by the parser to extract various fields.
timestamp	metadata.event_timestamp	The timestamp of the event.
tty	security_result.about.labels.tty	The tty is added as a label within security_result.about.
type	metadata.product_event_type	The event type (type) is mapped to metadata.product_event_type.
uid	principal.user.userid, target.user.userid	The user ID (uid) is mapped to either principal.user.userid or target.user.userid depending on the context.

Changes

2024-05-08

• When the value is not "?", then mapped "field" to "field33" to "security_result.detection_fields".
• When "type_name" is "CRYPTO_KEY_USER", then mapped "exe" to "principal.process.file.full_path".
• When "type_name" is "CRYPTO_KEY_USER", then mapped "fp" to "network.tls.client.certificate.sha256".
• When "type_name" is "CRYPTO_KEY_USER", then mapped "pid" to "principal.process.pid".
• Added Grok patterns to parse new pattern of logs.
• Mapped "syslog-tag" to "security_result.detection_fields".
• Mapped "inter_ip" to "intermediary.ip".
• Mapped "inter_hostname" to "intermediary.hostname".

2024-05-02

• When "type_name" is "USER_MGMT", then mapped "grp" to "target.group.group_display_name".
• When "type_name" is "USER_MGMT", then changed mapping of "uid" from "principal.user.userid" to "target.user.userid".
• When "type_name" is "USER_MGMT" and "op" is equal to "deleting-user-from-group", then set "metadata.event_type" to "GROUP_MODIFICATION".
• When "type_name" is "USER_MGMT", then changed mapping of "exe" from "target.process.file.full_path" to "principal.process.file.full_path".
• When "type_name" is "USER_MGMT", then mapped "id" to "about.user.userid".

2024-04-08

• When "type_name" is "ADD_USER", principal_user_present is "true", target_user_present is "true", and has_principal is "true", then set "metadata.event_type" to "USER_CREATION".
• When "type_name" is "USER_AUTH", then mapped "acct" to "target.user.user_display_name".
• When "type_name" is "USER_AUTH", then mapped "uid" to "principal.user.userid".
• When "type_name" is not in "ADD_USER","USER_AUTH","CRED_ACQ", and "USER_MGMT", then mapped "auid" to "about.user.userid".
• When "type_name" is "ADD_USER", then mapped "auid" to "target.user.userid".
• When "type_name" is "ADD_USER" or "USER_AUTH" then mapped "exe" to "principal.process.file.full_path".
• When "type_name" is "ADD_USER", then mapped "op" and "id" to "security_result.summary".
• When "type_name" is "USER_AUTH", then mapped "op" and "acct" to "security_result.summary".

2024-03-22

• Added support for new pattern of JSON logs.
• Mapped "labels.compute.googleapis.com/resource_name","jsonPayload._HOSTNAME" , "CollectorHostName", "HOSTNAME", and "Computer" to "principal.hostname".
• Mapped "HostIP" to "principal.ip".
• Mapped "ProcessID" and "jsonPayload._PID" to "principal.process.pid".
• Mapped "SyslogMessage" to "metadata.description".
• Mapped "TenantId", "_ItemId", "_Internal_WorkspaceResourceId", "_ResourceId", and "Facility" to "additional.fields".
• Mapped "SeverityLevel" to "security_result.severity".
• Mapped "SourceSystem" to "principal.platform".
• Mapped "jsonPayload._COMM" to "principal.application".
• Mapped "jsonPayload._EXE" to "target.process.file.full_path".
• Mapped "jsonPayload._AUDIT_FIELD_FILE" to "target.file.full_path".
• Mapped "jsonPayload._AUDIT_FIELD_HASH" to "target.file.hash".
• Mapped "jsonPayload._AUDIT_SESSION" to "network.session_id".
• Mapped "jsonPayload._PPID" to "principal.process.parent_process.pid".
• Mapped "jsonPayload._AUDIT_FIELD_A0", "jsonPayload._AUDIT_FIELD_A1", "jsonPayload._AUDIT_FIELD_A2", "jsonPayload._AUDIT_FIELD_A3", "jsonPayload._BOOT_ID", and "jsonPayload._AUDIT_FIELD_EXIT" to "security_result.detection_fields".

2023-11-27

• Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_LOGIN".
• Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_LOGOUT".
• Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_CREATION".
• Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_DELETION".
• Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_UNCATEGORIZED".
• Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_RESOURCE_ACCESS".
• Added validation check to ensure either "principal_user_present", "target_user_present" or "has_principal" is true before setting "metadata.event_type" to "USER_CHANGE_PERMISSIONS".
• When user details are present and principal machine details are not present then changed mapping of "metadata.event_type" from "USER_CREATION" to "USER_UNCATEGORIZED".
• When user details are present and principal machine details are not present then changed mapping of "metadata.event_type" from "USER_DELETION" to "USER_UNCATEGORIZED".

2023-09-06

• Added mapping of "CMD" to "target.process.command_line" for "cron daemon(CROND)".

2023-06-20

• Enhancement - Added or modified the following mappings when type="ADD_USER" and "DEL_USER"-
• Modified the mapping of "uid" from "target.user.userid" to "principal.user.userid".
• Mapped "id" to "target.user.userid".
• Mapped "ID" to "target.user.user_display_name".
• Modified the mapping of "UID" from "principal.user.userid" to "principal.user.user_display_name".
• Modified the mapping of "acct" from "principal.user.user_display_name" to "target.user.user_display_name" and "target.user.userid".

2023-06-09

• Enhancement - Modified "event_type" from "USER_LOGIN" to "USER_CREATION" when "type=ADD_USER".

2023-04-17

• Enhancement
• Added gsub function to replace "GS - Group separator" character which is breaking the JSON construction.

2023-04-10

• Enhancement
• Added 'gid','euid','egid','suid','fsuid','sgid','fsgid','tty','items' fields to security_result.detection_fields.
• Additionally mapped 'gid' to 'principal.user.group_identifiers'.
• Mapped 'euid' to 'target.user.userid'.
• Mapped 'egid' to 'target.user.group_identifiers'.

2023-03-27

• Enhancement - Added support for "jsonPayload" containing logs.

2023-02-28

• Bug-fix - Enhanced parser to convert hex encoded string to ASCII.

2023-02-09

• Enhancement - Modified grok for logs containing "type=PATH" to fetch the correct hostname from logs.

2023-01-24

• Enhancement -
• Parsed log with eventType as "tac_plus".
• Added conditions for mapping different event_types "NETWORK_CONNECTION", "NETWORK_HTTP", "USER_LOGIN".

2022-12-02

• Enhancement -
• Mapped "user_name" to "principal.user.userid".
• added conditional check for "dst_ip", "dst_port".

2022-10-31

• Enhancement -
• Enhanced the parser to parse the log with type=ADD_USER, USER_MGMT, DEL_USER.
• Added null checks for "principal_hostname".
• Added on_error checks for "principal.process.file.full_path", "type_syscall_props.key", "type_syscall_props.arch", "msg2".
• Added conditional checks for mapping to event_type="FILE_OPEN", "USER_UNCATEGORIZED", "STATUS_UPDATE", "USER_DELETION".
• Mapped "principal_user_userid" to "principal.user.userid".

2022-11-16

• Enhancement -
• Improved "GENERIC_EVENT" to "STATUS_UPDATE" for log types containing "Access Logs".

2022-10-14

• Enhancement -
• Migrated customer parser to default parser.

2022-10-13

• Enhancement - Mapped "vendor_name" to "Linux".
• Mapped "product_name" to "AuditD".
• Parsed the logs containing "ProxySG" and mapped "ip" to "target.ip", "port" to "target.port" wherever possible.
• Modified "event_type" from "GENERIC_EVENT" to "STATUS_UPDATE".
• Modified mapping for "intermediary.hostname" to "principal.hostname".

2022-07-28

• Enhancement -
• Mapped the field 'auid' to about.user.userid'.
• Mapped the field 'AUID' to 'about.user.user_display_name'.
• Mapped the field 'proctitle' to 'target.process.file.full_path'.
• Enhanced the parser to parse the log with type=DAEMON_END, CRYPTO_SESSION, CONFIG_CHANGE, PROCTITLE, USER_ERR, CRYPTO_KEY_USER.
• Added conditional check for laddr, addr, cipher, pfs, direction, acct, pid, ppid, cmd, exe, ses.

2022-06-17

• Enhancement - Mapped/Modified the following fields :
• Changed mapping of "auid" from "security_result.about.user.userid" to "about.user.userid".
• Changed "event_type" for type=SYSCALL from "SYSTEM_AUDIT_LOG_UNCATEGORIZED" to "USER_UNCATEGORIZED".
• Mapped "success" to "security_result.summary".
• Mapped "syscall", "exit", "tty", "a0", "a1", "a2", "a3" to "security_result.about.labels".
• Dropped the logs in ASCII format.

2022-06-14

• enhancement
• Enhanced the parser to parse the USER_CMD type of logs.
• Mapped the field 'cmd' to 'principal.process.command_line'.
• Mapped the field 'ses' to 'network.session_id'.
• Mapped the field 'res' to 'security_result.action' and 'security_result.action_details'.
• Mapped the fields 'auid' and 'cwd' to 'security_result.detection_fields'.

2022-04-26

• enhancement
• Increased the parsing percentage by parsing all the unparsed logs.