Collect Cisco Secure Email Gateway logs

Supported in:

This document describes how you can collect the Cisco Secure Email Gateway logs by using a Google Security Operations forwarder.

For more information, see Data ingestion to Google SecOps.

An ingestion label identifies the parser that normalizes raw log data to structured UDM format. The information in this document applies to the parser with the CISCO-EMAIL-SECURITY ingestion label.

Configure Cisco Secure Email Gateway

  1. In the Cisco Secure Email Gateway console, select System administration > Log subscriptions.
  2. In the New log subscription window, do the following to add log subscription:
    1. In the Log type field, select Consolidated event logs.
    2. In the Available log fields section, select all the available fields, and then click Add to move them to the Selected log fields.
    3. To select a log retrieval method for the log subscription, select Syslog push and do the following:
      1. In the Hostname field, specify the Google SecOps forwarder IP address.
      2. In the Protocol field, select the TCP checkbox.
      3. In the Facility field, use default value.
  3. To save your configuration changes, click Submit.

Configure the Google SecOps forwarder to ingest Cisco Secure Email Gateway

  1. In the Google SecOps menu, click Settings > Forwarders > Add new forwarder.
  2. In the Forwarder name field, enter a unique name for the forwarder.
  3. Click Submit. The forwarder is added and the Add collector configuration window appears.
  4. In the Collector name field, enter a unique name for the collector.
  5. In the Log type field, specify Cisco Email Security.
  6. In the Collector type field, select Syslog.
  7. Configure the following input parameters:
    • Protocol: specify the connection protocol that the collector uses to listen to syslog data.
    • Address: specify the target IP address or hostname where the collector resides and listens to syslog data.
    • Port: specify the target port where the collector resides and listens to syslog data.
  8. Click Submit.

For more information about the Google SecOps forwarders, see Manage forwarder configurations through the Google SecOps UI.

If you encounter issues when you create forwarders, contact Google SecOps support.

Field mapping reference

This parser handles both structured (JSON, key-value pairs) and unstructured (syslog) Cisco Email Security logs. It normalizes diverse log formats into UDM by leveraging grok patterns, key-value extraction, and conditional logic based on the product_event field to map relevant Cisco ESA fields to UDM. It also performs data enrichment, such as converting timestamps and handling repeated messages.

UDM Mapping Table

Log field UDM mapping Logic
acl_decision_tag read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "ACL Decision Tag".
access_or_decryption_policy_group read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "AccessOrDecryptionPolicyGroup".
act read_only_udm.security_result.action_details Directly mapped.
authenticated_user read_only_udm.principal.user.userid Directly mapped if not empty, "-", or "NONE".
cache_hierarchy_retrieval read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "Cache Hierarchy Retrieval".
cipher read_only_udm.network.tls.cipher Directly mapped.
country read_only_udm.principal.location.country_or_region Directly mapped.
data_security_policy_group read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "DataSecurityPolicyGroup".
description read_only_udm.metadata.description Directly mapped for syslog messages. For CEF messages, it becomes the overall product description. Various grok patterns extract specific descriptions based on the product_event. Some descriptions are modified by gsub to remove leading/trailing spaces and colons.
deviceDirection read_only_udm.network.direction If '0', maps to 'INBOUND'. If '1', maps to 'OUTBOUND'. Used to determine which TLS cipher and protocol to map directly and which to map as labels.
deviceExternalId read_only_udm.principal.asset.asset_id Mapped as "Device ID:".
domain read_only_udm.target.administrative_domain Directly mapped from JSON logs.
domain_age read_only_udm.security_result.about.labels.value Directly mapped. Key is "YoungestDomainAge".
duser read_only_udm.target.user.email_addresses, read_only_udm.network.email.to If contains ";", split into multiple email addresses and map each to both UDM fields. Otherwise, directly map to both UDM fields if a valid email address. Also used to populate network_to if it's empty.
dvc read_only_udm.target.ip Directly mapped.
entries.collection_time.nanos, entries.collection_time.seconds read_only_udm.metadata.event_timestamp.nanos, read_only_udm.metadata.event_timestamp.seconds Used to construct the event timestamp.
env-from read_only_udm.additional.fields.value.string_value Directly mapped. Key is "Env-From".
ESAAttachmentDetails read_only_udm.security_result.about.file.full_path, read_only_udm.security_result.about.file.sha256 Parsed to extract file names and SHA256 hashes. Multiple files and hashes can be extracted.
ESADCID read_only_udm.security_result.about.labels.value Directly mapped. Key is "ESADCID".
ESAFriendlyFrom read_only_udm.principal.user.user_display_name, read_only_udm.network.email.from Parsed to extract the display name and email address.
ESAHeloDomain read_only_udm.intermediary.administrative_domain Directly mapped.
ESAHeloIP read_only_udm.intermediary.ip Directly mapped.
ESAICID read_only_udm.security_result.about.labels.value Directly mapped. Key is "ESAICID".
ESAMailFlowPolicy read_only_udm.security_result.rule_name Directly mapped.
ESAMID read_only_udm.security_result.about.labels.value Directly mapped. Key is "ESAMID".
ESAReplyTo read_only_udm.network.email.reply_to Directly mapped if a valid email address. Also used to populate network_to.
ESASDRDomainAge read_only_udm.security_result.about.labels.value Directly mapped. Key is "ESASDRDomainAge".
ESASenderGroup read_only_udm.principal.group.group_display_name Directly mapped.
ESAStatus read_only_udm.security_result.about.labels.value Directly mapped. Key is "ESAStatus".
ESATLSInCipher read_only_udm.network.tls.cipher or read_only_udm.security_result.about.labels.value Mapped directly to cipher if deviceDirection is '0'. Otherwise, mapped as a label with key "ESATLSInCipher".
ESATLSInProtocol read_only_udm.network.tls.version or read_only_udm.security_result.about.labels.value TLS version extracted and mapped directly if deviceDirection is '0'. Otherwise, mapped as a label with key "ESATLSInProtocol".
ESATLSOutCipher read_only_udm.network.tls.cipher or read_only_udm.security_result.about.labels.value Mapped directly to cipher if deviceDirection is '1'. Otherwise, mapped as a label with key "ESATLSOutCipher".
ESATLSOutProtocol read_only_udm.network.tls.version or read_only_udm.security_result.about.labels.value TLS version extracted and mapped directly if deviceDirection is '1'. Otherwise, mapped as a label with key "ESATLSOutProtocol".
ESAURLDetails read_only_udm.target.url Parsed to extract URLs. Only the first URL is mapped because the field is not repeated.
external_dlp_policy_group read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "ExternalDlpPolicyGroup".
ExternalMsgID read_only_udm.security_result.about.labels.value Directly mapped after removing single quotes and angle brackets. Key is "ExternalMsgID".
from read_only_udm.network.email.from Directly mapped if a valid email address. Also used to populate network_from.
host.hostname read_only_udm.principal.hostname or read_only_udm.intermediary.hostname Mapped to principal hostname if host field is invalid. Also mapped to intermediary hostname.
host.ip read_only_udm.principal.ip or read_only_udm.intermediary.ip Mapped to principal IP if ip field is not set in JSON logs. Also mapped to intermediary IP.
hostname read_only_udm.target.hostname Directly mapped.
http_method read_only_udm.network.http.method Directly mapped.
http_response_code read_only_udm.network.http.response_code Directly mapped and converted to integer.
identity_policy_group read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "IdentityPolicyGroup".
ip read_only_udm.principal.ip Directly mapped. Overwritten by source_ip if present.
kv_msg Various Parsed using kv filter. Pre-processing includes replacing spaces before keys with "#" and swapping csLabel values.
log_type read_only_udm.metadata.log_type Hardcoded to "CISCO_EMAIL_SECURITY".
loglevel read_only_udm.security_result.severity, read_only_udm.security_result.action Used to determine severity and action. "Info", "", "Debug", "Trace" map to "INFORMATIONAL" and "ALLOW". "Warning" maps to "MEDIUM" and "ALLOW". "High" maps to "HIGH" and "BLOCK". "Critical" and "Alert" map to "CRITICAL", "BLOCK", and set is_alert to true.
mail_id read_only_udm.network.email.mail_id Directly mapped from JSON logs.
mailto read_only_udm.target.user.email_addresses, read_only_udm.network.email.to Directly mapped to both UDM fields if a valid email address.
MailPolicy read_only_udm.security_result.about.labels.value Directly mapped. Key is "MailPolicy".
message Various Parsed as JSON if possible. Otherwise, processed as a syslog message.
message_id read_only_udm.network.email.mail_id Directly mapped. Also used to populate network_data.
msg read_only_udm.network.email.subject Directly mapped after UTF-8 decoding and removing carriage returns, newlines, and extra quotes. Also used to populate network_data.
msg1 Various Parsed using kv filter. Used to extract Hostname, helo, env-from, and reply-to.
outbound_malware_scanning_policy_group read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "DataSecurityPolicyGroup".
port read_only_udm.target.port Directly mapped and converted to integer.
principalMail read_only_udm.principal.user.email_addresses Directly mapped.
principalUrl read_only_udm.principal.url Directly mapped.
product_event read_only_udm.metadata.product_event_type Directly mapped. Used to determine which grok patterns to apply. Leading "%" characters are removed. "amp" is replaced with "SIEM_AMPenginelogs".
product_version read_only_udm.metadata.product_version Directly mapped.
protocol read_only_udm.network.tls.version Directly mapped.
received_bytes read_only_udm.network.received_bytes Directly mapped and converted to unsigned integer.
reply-to read_only_udm.additional.fields.value.string_value Directly mapped. Key is "Reply-To".
reputation read_only_udm.security_result.confidence_details Directly mapped.
request_method_uri read_only_udm.target.url Directly mapped.
result_code read_only_udm.security_result.detection_fields.value Directly mapped. Key is "Result Code".
routing_policy_group read_only_udm.security_result.detection_fields.value Directly mapped if not empty, "-", or "NONE". Key is "RoutingPolicyGroup".
rule read_only_udm.security_result.detection_fields.value Directly mapped. Key is "Matched Condition".
SDRThreatCategory read_only_udm.security_result.threat_name Directly mapped if not empty or "N/A".
SenderCountry read_only_udm.principal.location.country_or_region Directly mapped.
senderGroup read_only_udm.principal.group.group_display_name Directly mapped.
security_description read_only_udm.security_result.description Directly mapped.
security_email read_only_udm.security_result.about.email or read_only_udm.principal.hostname Mapped to email if a valid email address. Otherwise, mapped to hostname after extracting with grok.
source read_only_udm.network.ip_protocol If contains "tcp", maps to "TCP".
sourceAddress read_only_udm.principal.ip Directly mapped.
sourceHostName read_only_udm.principal.administrative_domain Directly mapped if not "unknown".
source_ip read_only_udm.principal.ip Directly mapped. Overwrites ip if present.
Subject read_only_udm.network.email.subject Directly mapped after removing trailing periods. Also used to populate network_data.
suser read_only_udm.principal.user.email_addresses, read_only_udm.network.email.bounce_address Directly mapped to both UDM fields if a valid email address.
target_ip read_only_udm.target.ip Directly mapped.
to read_only_udm.network.email.to Directly mapped if a valid email address. Also used to populate network_to.
total_bytes read_only_udm.network.sent_bytes Directly mapped and converted to unsigned integer.
trackerHeader read_only_udm.additional.fields.value.string_value Directly mapped. Key is "Tracker Header".
ts, ts1, year read_only_udm.metadata.event_timestamp.seconds Used to construct the event timestamp. ts1 and year are combined if ts1 is present. Various formats are supported, with and without the year. If the year is not present, the current year is used. Hardcoded to "Cisco". Hardcoded to "Cisco Email Security". Defaults to "ALLOW". Set to "BLOCK" based on loglevel or description. Defaults to "INBOUND" if application_protocol is present. Set based on deviceDirection for CEF messages. Determined based on a combination of fields including network_from, network_to, target_ip, ip, description, event_type, principal_host, Hostname, user_id, and sourceAddress. Defaults to "GENERIC_EVENT". Set to "SMTP" if application_protocol is "SMTP" or "smtp", or if target_ip and ip are present. Set to "AUTHTYPE_UNSPECIFIED" if login_status and user_id are present in sshd logs. Set to true if loglevel is "Critical" or "Alert".

Changes

2023-10-05

  • Bug-Fix:
  • Renamed the 'product_event' from 'amp' to 'SIEM_AMPenginelogs'.

2023-09-15

  • Added support for "SIEM_proxylogs","SIEM_webrootlogs","SIEM_AMPenginelogs" of json logs.

2023-09-04

  • Enhancement
  • Added a Grok pattern to parse unparsed logs and mapped the fields accordingly.
  • Added support for new pattern of JSON logs.

2022-12-16

  • Enhancement
  • Modified conditional checks for the fields mapped to 'network.email.to', 'network.email.from', 'principal.user.email_addresses', 'target.user.email_addresses' and 'network.email.reply_to'.
  • Added support for json logs :
  • Mapped the field 'host' to 'principal.hostname'.
  • Mapped the field 'domain' to 'target.administrative_domain'.
  • Mapped the field 'mail_id' to 'network.email.mail_id'.
  • Mapped the field 'mailto' to 'network.email.to' and 'target.user.email_addresses'.
  • Mapped the field 'source' to 'network.ip_protocol'.
  • Mapped the field 'reputation' to 'security_result.confidence_details'.
  • Mapped the field 'log_type' to 'security_result.severity' and 'security_result.severity_details'.
  • Mapped the field 'cribl_pipe' to 'additional.fields'.

2022-09-22

  • Enhancement
  • Added a grok pattern for unparsed logs, having the field "product_event" as empty.

2022-08-02

  • Enhancement
  • Added conditions for newly added event_type "STATUS_UPDATE", "USER_UNCATEGORIZED", "SCAN_PROCESS"
  • Mapped "attack" to "security_result.category_details"
  • Enahanced parser to parse "ESAAttachmentDetails" field of different types of logs.

2022-06-09

  • Enhancement- Mapped "from_user" to "principal.user.user_display_name".
  • Updated "metadata.product_event_type" from "Consolidated Log Event" to "ESA_CONSOLIDATED_LOG_EVENT".

2022-06-07

  • Enhancement- Mapped suser to network.email.bounce_address.

2022-05-17

  • Enhancement - Mapped duser to network.email.to.
  • Added on_error for product_version and product_description fields to avoid null value mapping to UDM.
  • Added additional logic to parse logs starting with "DAY TIMESTAMP YEAR" format, for example: Wed Feb 18 00:34:12 2021.

2022-05-05

  • Enhancement-Used grok for network.email.from

2022-03-31

  • Enhancement-Added mappings for new fields.
  • ESAReplyTo mapped to network.email.reply_to.
  • duser mapped to network.email.to.