Collect Cisco Secure Email Gateway logs
This document describes how you can collect the Cisco Secure Email Gateway logs by using a Google Security Operations forwarder.
For more information, see Data ingestion to Google SecOps.
An ingestion label identifies the parser that normalizes raw log data to structured
UDM format. The information in this document applies to the parser with the
CISCO-EMAIL-SECURITY
ingestion label.
Configure Cisco Secure Email Gateway
- In the Cisco Secure Email Gateway console, select System administration > Log subscriptions.
- In the New log subscription window, do the following to add
log subscription:
- In the Log type field, select Consolidated event logs.
- In the Available log fields section, select all the available fields, and then click Add to move them to the Selected log fields.
- To select a log retrieval method for the log subscription, select
Syslog push and do the following:
- In the Hostname field, specify the Google SecOps forwarder IP address.
- In the Protocol field, select the TCP checkbox.
- In the Facility field, use default value.
- To save your configuration changes, click Submit.
Configure the Google SecOps forwarder to ingest Cisco Secure Email Gateway
- In the Google SecOps menu, click Settings > Forwarders > Add new forwarder.
- In the Forwarder name field, enter a unique name for the forwarder.
- Click Submit. The forwarder is added and the Add collector configuration window appears.
- In the Collector name field, enter a unique name for the collector.
- In the Log type field, specify
Cisco Email Security
. - In the Collector type field, select Syslog.
- Configure the following input parameters:
- Protocol: specify the connection protocol that the collector uses to listen to syslog data.
- Address: specify the target IP address or hostname where the collector resides and listens to syslog data.
- Port: specify the target port where the collector resides and listens to syslog data.
- Click Submit.
For more information about the Google SecOps forwarders, see Manage forwarder configurations through the Google SecOps UI.
If you encounter issues when you create forwarders, contact Google SecOps support.
Field mapping reference
This parser handles both structured (JSON, key-value pairs) and unstructured (syslog) Cisco Email Security logs. It normalizes diverse log formats into UDM by leveraging grok
patterns, key-value extraction, and conditional logic based on the product_event
field to map relevant Cisco ESA fields to UDM. It also performs data enrichment, such as converting timestamps and handling repeated messages.
UDM Mapping Table
Log field | UDM mapping | Logic |
---|---|---|
acl_decision_tag |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "ACL Decision Tag". |
access_or_decryption_policy_group |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "AccessOrDecryptionPolicyGroup". |
act |
read_only_udm.security_result.action_details |
Directly mapped. |
authenticated_user |
read_only_udm.principal.user.userid |
Directly mapped if not empty, "-", or "NONE". |
cache_hierarchy_retrieval |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "Cache Hierarchy Retrieval". |
cipher |
read_only_udm.network.tls.cipher |
Directly mapped. |
country |
read_only_udm.principal.location.country_or_region |
Directly mapped. |
data_security_policy_group |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "DataSecurityPolicyGroup". |
description |
read_only_udm.metadata.description |
Directly mapped for syslog messages. For CEF messages, it becomes the overall product description. Various grok patterns extract specific descriptions based on the product_event . Some descriptions are modified by gsub to remove leading/trailing spaces and colons. |
deviceDirection |
read_only_udm.network.direction |
If '0', maps to 'INBOUND'. If '1', maps to 'OUTBOUND'. Used to determine which TLS cipher and protocol to map directly and which to map as labels. |
deviceExternalId |
read_only_udm.principal.asset.asset_id |
Mapped as "Device ID: |
domain |
read_only_udm.target.administrative_domain |
Directly mapped from JSON logs. |
domain_age |
read_only_udm.security_result.about.labels.value |
Directly mapped. Key is "YoungestDomainAge". |
duser |
read_only_udm.target.user.email_addresses , read_only_udm.network.email.to |
If contains ";", split into multiple email addresses and map each to both UDM fields. Otherwise, directly map to both UDM fields if a valid email address. Also used to populate network_to if it's empty. |
dvc |
read_only_udm.target.ip |
Directly mapped. |
entries.collection_time.nanos , entries.collection_time.seconds |
read_only_udm.metadata.event_timestamp.nanos , read_only_udm.metadata.event_timestamp.seconds |
Used to construct the event timestamp. |
env-from |
read_only_udm.additional.fields.value.string_value |
Directly mapped. Key is "Env-From". |
ESAAttachmentDetails |
read_only_udm.security_result.about.file.full_path , read_only_udm.security_result.about.file.sha256 |
Parsed to extract file names and SHA256 hashes. Multiple files and hashes can be extracted. |
ESADCID |
read_only_udm.security_result.about.labels.value |
Directly mapped. Key is "ESADCID". |
ESAFriendlyFrom |
read_only_udm.principal.user.user_display_name , read_only_udm.network.email.from |
Parsed to extract the display name and email address. |
ESAHeloDomain |
read_only_udm.intermediary.administrative_domain |
Directly mapped. |
ESAHeloIP |
read_only_udm.intermediary.ip |
Directly mapped. |
ESAICID |
read_only_udm.security_result.about.labels.value |
Directly mapped. Key is "ESAICID". |
ESAMailFlowPolicy |
read_only_udm.security_result.rule_name |
Directly mapped. |
ESAMID |
read_only_udm.security_result.about.labels.value |
Directly mapped. Key is "ESAMID". |
ESAReplyTo |
read_only_udm.network.email.reply_to |
Directly mapped if a valid email address. Also used to populate network_to . |
ESASDRDomainAge |
read_only_udm.security_result.about.labels.value |
Directly mapped. Key is "ESASDRDomainAge". |
ESASenderGroup |
read_only_udm.principal.group.group_display_name |
Directly mapped. |
ESAStatus |
read_only_udm.security_result.about.labels.value |
Directly mapped. Key is "ESAStatus". |
ESATLSInCipher |
read_only_udm.network.tls.cipher or read_only_udm.security_result.about.labels.value |
Mapped directly to cipher if deviceDirection is '0'. Otherwise, mapped as a label with key "ESATLSInCipher". |
ESATLSInProtocol |
read_only_udm.network.tls.version or read_only_udm.security_result.about.labels.value |
TLS version extracted and mapped directly if deviceDirection is '0'. Otherwise, mapped as a label with key "ESATLSInProtocol". |
ESATLSOutCipher |
read_only_udm.network.tls.cipher or read_only_udm.security_result.about.labels.value |
Mapped directly to cipher if deviceDirection is '1'. Otherwise, mapped as a label with key "ESATLSOutCipher". |
ESATLSOutProtocol |
read_only_udm.network.tls.version or read_only_udm.security_result.about.labels.value |
TLS version extracted and mapped directly if deviceDirection is '1'. Otherwise, mapped as a label with key "ESATLSOutProtocol". |
ESAURLDetails |
read_only_udm.target.url |
Parsed to extract URLs. Only the first URL is mapped because the field is not repeated. |
external_dlp_policy_group |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "ExternalDlpPolicyGroup". |
ExternalMsgID |
read_only_udm.security_result.about.labels.value |
Directly mapped after removing single quotes and angle brackets. Key is "ExternalMsgID". |
from |
read_only_udm.network.email.from |
Directly mapped if a valid email address. Also used to populate network_from . |
host.hostname |
read_only_udm.principal.hostname or read_only_udm.intermediary.hostname |
Mapped to principal hostname if host field is invalid. Also mapped to intermediary hostname. |
host.ip |
read_only_udm.principal.ip or read_only_udm.intermediary.ip |
Mapped to principal IP if ip field is not set in JSON logs. Also mapped to intermediary IP. |
hostname |
read_only_udm.target.hostname |
Directly mapped. |
http_method |
read_only_udm.network.http.method |
Directly mapped. |
http_response_code |
read_only_udm.network.http.response_code |
Directly mapped and converted to integer. |
identity_policy_group |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "IdentityPolicyGroup". |
ip |
read_only_udm.principal.ip |
Directly mapped. Overwritten by source_ip if present. |
kv_msg |
Various | Parsed using kv filter. Pre-processing includes replacing spaces before keys with "#" and swapping csLabel values. |
log_type |
read_only_udm.metadata.log_type |
Hardcoded to "CISCO_EMAIL_SECURITY". |
loglevel |
read_only_udm.security_result.severity , read_only_udm.security_result.action |
Used to determine severity and action. "Info", "", "Debug", "Trace" map to "INFORMATIONAL" and "ALLOW". "Warning" maps to "MEDIUM" and "ALLOW". "High" maps to "HIGH" and "BLOCK". "Critical" and "Alert" map to "CRITICAL", "BLOCK", and set is_alert to true. |
mail_id |
read_only_udm.network.email.mail_id |
Directly mapped from JSON logs. |
mailto |
read_only_udm.target.user.email_addresses , read_only_udm.network.email.to |
Directly mapped to both UDM fields if a valid email address. |
MailPolicy |
read_only_udm.security_result.about.labels.value |
Directly mapped. Key is "MailPolicy". |
message |
Various | Parsed as JSON if possible. Otherwise, processed as a syslog message. |
message_id |
read_only_udm.network.email.mail_id |
Directly mapped. Also used to populate network_data . |
msg |
read_only_udm.network.email.subject |
Directly mapped after UTF-8 decoding and removing carriage returns, newlines, and extra quotes. Also used to populate network_data . |
msg1 |
Various | Parsed using kv filter. Used to extract Hostname , helo , env-from , and reply-to . |
outbound_malware_scanning_policy_group |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "DataSecurityPolicyGroup". |
port |
read_only_udm.target.port |
Directly mapped and converted to integer. |
principalMail |
read_only_udm.principal.user.email_addresses |
Directly mapped. |
principalUrl |
read_only_udm.principal.url |
Directly mapped. |
product_event |
read_only_udm.metadata.product_event_type |
Directly mapped. Used to determine which grok patterns to apply. Leading "%" characters are removed. "amp" is replaced with "SIEM_AMPenginelogs". |
product_version |
read_only_udm.metadata.product_version |
Directly mapped. |
protocol |
read_only_udm.network.tls.version |
Directly mapped. |
received_bytes |
read_only_udm.network.received_bytes |
Directly mapped and converted to unsigned integer. |
reply-to |
read_only_udm.additional.fields.value.string_value |
Directly mapped. Key is "Reply-To". |
reputation |
read_only_udm.security_result.confidence_details |
Directly mapped. |
request_method_uri |
read_only_udm.target.url |
Directly mapped. |
result_code |
read_only_udm.security_result.detection_fields.value |
Directly mapped. Key is "Result Code". |
routing_policy_group |
read_only_udm.security_result.detection_fields.value |
Directly mapped if not empty, "-", or "NONE". Key is "RoutingPolicyGroup". |
rule |
read_only_udm.security_result.detection_fields.value |
Directly mapped. Key is "Matched Condition". |
SDRThreatCategory |
read_only_udm.security_result.threat_name |
Directly mapped if not empty or "N/A". |
SenderCountry |
read_only_udm.principal.location.country_or_region |
Directly mapped. |
senderGroup |
read_only_udm.principal.group.group_display_name |
Directly mapped. |
security_description |
read_only_udm.security_result.description |
Directly mapped. |
security_email |
read_only_udm.security_result.about.email or read_only_udm.principal.hostname |
Mapped to email if a valid email address. Otherwise, mapped to hostname after extracting with grok. |
source |
read_only_udm.network.ip_protocol |
If contains "tcp", maps to "TCP". |
sourceAddress |
read_only_udm.principal.ip |
Directly mapped. |
sourceHostName |
read_only_udm.principal.administrative_domain |
Directly mapped if not "unknown". |
source_ip |
read_only_udm.principal.ip |
Directly mapped. Overwrites ip if present. |
Subject |
read_only_udm.network.email.subject |
Directly mapped after removing trailing periods. Also used to populate network_data . |
suser |
read_only_udm.principal.user.email_addresses , read_only_udm.network.email.bounce_address |
Directly mapped to both UDM fields if a valid email address. |
target_ip |
read_only_udm.target.ip |
Directly mapped. |
to |
read_only_udm.network.email.to |
Directly mapped if a valid email address. Also used to populate network_to . |
total_bytes |
read_only_udm.network.sent_bytes |
Directly mapped and converted to unsigned integer. |
trackerHeader |
read_only_udm.additional.fields.value.string_value |
Directly mapped. Key is "Tracker Header". |
ts , ts1 , year |
read_only_udm.metadata.event_timestamp.seconds |
Used to construct the event timestamp. ts1 and year are combined if ts1 is present. Various formats are supported, with and without the year. If the year is not present, the current year is used. Hardcoded to "Cisco". Hardcoded to "Cisco Email Security". Defaults to "ALLOW". Set to "BLOCK" based on loglevel or description . Defaults to "INBOUND" if application_protocol is present. Set based on deviceDirection for CEF messages. Determined based on a combination of fields including network_from , network_to , target_ip , ip , description , event_type , principal_host , Hostname , user_id , and sourceAddress . Defaults to "GENERIC_EVENT". Set to "SMTP" if application_protocol is "SMTP" or "smtp", or if target_ip and ip are present. Set to "AUTHTYPE_UNSPECIFIED" if login_status and user_id are present in sshd logs. Set to true if loglevel is "Critical" or "Alert". |
Changes
2023-10-05
- Bug-Fix:
- Renamed the 'product_event' from 'amp' to 'SIEM_AMPenginelogs'.
2023-09-15
- Added support for "SIEM_proxylogs","SIEM_webrootlogs","SIEM_AMPenginelogs" of json logs.
2023-09-04
- Enhancement
- Added a Grok pattern to parse unparsed logs and mapped the fields accordingly.
- Added support for new pattern of JSON logs.
2022-12-16
- Enhancement
- Modified conditional checks for the fields mapped to 'network.email.to', 'network.email.from', 'principal.user.email_addresses', 'target.user.email_addresses' and 'network.email.reply_to'.
- Added support for json logs :
- Mapped the field 'host' to 'principal.hostname'.
- Mapped the field 'domain' to 'target.administrative_domain'.
- Mapped the field 'mail_id' to 'network.email.mail_id'.
- Mapped the field 'mailto' to 'network.email.to' and 'target.user.email_addresses'.
- Mapped the field 'source' to 'network.ip_protocol'.
- Mapped the field 'reputation' to 'security_result.confidence_details'.
- Mapped the field 'log_type' to 'security_result.severity' and 'security_result.severity_details'.
- Mapped the field 'cribl_pipe' to 'additional.fields'.
2022-09-22
- Enhancement
- Added a grok pattern for unparsed logs, having the field "product_event" as empty.
2022-08-02
- Enhancement
- Added conditions for newly added event_type "STATUS_UPDATE", "USER_UNCATEGORIZED", "SCAN_PROCESS"
- Mapped "attack" to "security_result.category_details"
- Enahanced parser to parse "ESAAttachmentDetails" field of different types of logs.
2022-06-09
- Enhancement- Mapped "from_user" to "principal.user.user_display_name".
- Updated "metadata.product_event_type" from "Consolidated Log Event" to "ESA_CONSOLIDATED_LOG_EVENT".
2022-06-07
- Enhancement- Mapped suser to network.email.bounce_address.
2022-05-17
- Enhancement - Mapped duser to network.email.to.
- Added on_error for product_version and product_description fields to avoid null value mapping to UDM.
- Added additional logic to parse logs starting with "DAY TIMESTAMP YEAR" format, for example: Wed Feb 18 00:34:12 2021.
2022-05-05
- Enhancement-Used grok for network.email.from
2022-03-31
- Enhancement-Added mappings for new fields.
- ESAReplyTo mapped to network.email.reply_to.
- duser mapped to network.email.to.