Collect Rapid7 InsightIDR logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

This parser handles both JSON and SYSLOG formatted logs from Rapid7 InsightIDR. It extracts fields, normalizes them to the UDM, and performs specific logic for vulnerability data, including CVSS scores and exploit information, handling both JSON and syslog formats separately. It also maps authentication attempts and session events to appropriate UDM event types.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to the InsightIDR management console.

Configuring an API key in Rapid7 InsightIDR

1. Sign in to the InsightIDR Command Platform.
2. Click Administration.
3. Click API Keys.
4. Go to the Organization Keys tab.
5. Click New Organization Key.
6. Select an organization and provide a name for the key (for example, Google SecOps).
7. Generate the key.

8. Copy the key from a new window that displays the generated key.

   Note: You will not be able to view it again after you close the window.

Configure a feed in Google SecOps to ingest Rapid7 InsightIDR logs

1. Click Add new.
2. In the Feed name field, enter a name for the feed (for example, InsightIDR Logs).
3. Select Third party API as the Source type.
4. Select Rapid7 Insight as the Log type.
5. Click Next.
6. Specify values for the following input parameters:
   • Authentication HTTP Header: token previously generated in X-Api-Key:<value> format (for example, X-Api-Key:AAAABBBBCCCC111122223333).
   • API Endpoint: enter vulnerabilities or assets.
   • API Hostname: the FQDN (fully qualified domain name) of Rapid7 API endpoint in the [region].api.insight.rapid7.com format.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label applied to the events from this feed.
7. Click Next.
8. Review the feed configuration in the Finalize screen, and then click Submit.

UDM Mapping Table

Log Field	UDM Mapping	Logic
added	vulnerabilities.first_found	The added field is converted to a timestamp and mapped to vulnerabilities.first_found.
Authentication	security_result.detection_fields.value	The value of Authentication from the raw log is mapped to the value field within security_result.detection_fields. The corresponding key is set to "Authentication".
critical_vulnerabilities	asset.attribute.labels.value	The value of critical_vulnerabilities is mapped to the value field within asset.attribute.labels. The corresponding key is set to "Critical Vulnerabilities".
cves	vulnerabilities.cve_id	The value of cves is mapped to vulnerabilities.cve_id.
cvss_v2_access_complexity	asset.attribute.labels.value	The value of cvss_v2_access_complexity is mapped to the value field within asset.attribute.labels. The corresponding key is set to "Access Complexity(Ac)".
cvss_v2_availability_impact	asset.attribute.labels.value	The value of cvss_v2_availability_impact is mapped to the value field within asset.attribute.labels. The corresponding key is set to "Availability Impact (A)".
cvss_v2_confidentiality_impact	asset.attribute.labels.value	The value of cvss_v2_confidentiality_impact is mapped to the value field within asset.attribute.labels. The corresponding key is set to "Confidentiality Impact (C)".
cvss_v2_integrity_impact	asset.attribute.labels.value	The value of cvss_v2_integrity_impact is mapped to the value field within asset.attribute.labels. The corresponding key is set to "Integrity Impact (I)".
cvss_v2_score	vulnerabilities.cvss_base_score	The value of cvss_v2_score is converted to a string, then to a float, and mapped to vulnerabilities.cvss_base_score.
cvss_v2_vector	vulnerabilities.cvss_vector	The value of cvss_v2_vector is mapped to vulnerabilities.cvss_vector.
cvss_v3_availability_impact	asset.attribute.labels.value	The value of cvss_v3_availability_impact is mapped to the value field within asset.attribute.labels. The corresponding key is set to "Availability Impact (A)".
cvss_v3_score	vulnerabilities.cvss_base_score	The value of cvss_v3_score is converted to a string, then to a float, and mapped to vulnerabilities.cvss_base_score.
cvss_v3_vector	vulnerabilities.cvss_vector	The value of cvss_v3_vector is mapped to vulnerabilities.cvss_vector.
description	vulnerabilities.description	The value of description from the raw log is mapped to vulnerabilities.description.
exploits	asset.attribute.labels.value	The value of exploits is converted to a string and mapped to the value field within asset.attribute.labels. The corresponding key is either "Number of Exploits" or "Rank of Exploit" based on the presence of the "rank" field within the exploits object.
host_name	asset.hostname	The value of host_name is mapped to asset.hostname. If host_name is empty and both ip and mac are empty, the value of id is used instead.
id	asset.product_object_id	The value of id is mapped to asset.product_object_id. If host_name is empty and both ip and mac are empty, the value of id is used for asset.hostname.
ip	asset.ip, entity.asset.ip	The value of ip is mapped to both asset.ip and entity.asset.ip.
last_assessed_for_vulnerabilities	vulnerabilities.scan_end_time	The last_assessed_for_vulnerabilities field is converted to a timestamp and mapped to vulnerabilities.scan_end_time.
last_scan_end	vulnerabilities.last_found	The last_scan_end field is converted to a timestamp and mapped to vulnerabilities.last_found.
last_scan_start	vulnerabilities.first_found	The last_scan_start field is converted to a timestamp and mapped to vulnerabilities.first_found.
links	vulnerabilities.cve_id, vulnerabilities.vendor_knowledge_base_article_id	The id field within links is mapped to vulnerabilities.cve_id, and the href field within links is mapped to vulnerabilities.vendor_knowledge_base_article_id.
mac	asset.mac, entity.asset.mac	The value of mac is converted to lowercase and mapped to both asset.mac and entity.asset.mac.
MessageSourceAddress	principal.ip, principal.asset.ip	The IP address extracted from MessageSourceAddress is mapped to principal.ip and principal.asset.ip.
Method	network.http.method	The value of Method is mapped to network.http.method.
moderate_vulnerabilities	asset.attribute.labels.value	The value of moderate_vulnerabilities is converted to a string and mapped to the value field within asset.attribute.labels. The corresponding key is set to "Moderate Vulnerabilities".
os_architecture	asset.hardware.cpu_platform	The value of os_architecture is mapped to asset.hardware.cpu_platform.
os_description	asset.platform_software.platform_version	The value of os_description is mapped to asset.platform_software.platform_version.
os_family	asset.platform_software.platform	The value of os_family is converted to uppercase and mapped to asset.platform_software.platform. Special handling is done for "MAC OS X", "IOS", "WINDOWS", "MAC", and "LINUX". If it doesn't match any of these, it's set to "UNKNOWN_PLATFORM".
Port	principal.port	The value of Port is mapped to principal.port and converted to an integer.
Principal	principal.user.email_addresses	If Principal is an email address, it's mapped to principal.user.email_addresses.
product_event_type	metadata.product_event_type	The value of product_event_type is mapped to metadata.product_event_type.
Protocol	network.application_protocol	If Protocol is "HTTP" or "HTTPS", it's mapped to network.application_protocol.
published	vulnerabilities.last_found	The published field is converted to a timestamp and mapped to vulnerabilities.last_found.
Referer	network.http.referral_url	The value of Referer is mapped to network.http.referral_url.
risk_score	asset.attribute.labels.value	The value of risk_score is converted to a string and mapped to the value field within asset.attribute.labels. The corresponding key is set to "Risk Score".
security_result_summary	security_result.summary	The value of security_result_summary is mapped to security_result.summary. If it matches the pattern "Total sessions for principal: ", the number is extracted and mapped to a separate label with key "Session Count" within security_result.detection_fields.
Session	network.session_id	The value of Session is mapped to network.session_id.
severe_vulnerabilities	asset.attribute.labels.value	The value of severe_vulnerabilities is converted to a string and mapped to the value field within asset.attribute.labels. The corresponding key is set to "Severe Vulnerabilities".
severity	vulnerabilities.severity, security_result.severity	The value of severity is converted to uppercase. If it's "HIGH", "LOW", "CRITICAL", or "MEDIUM", it's mapped to vulnerabilities.severity. For syslog messages, if it's "Info", it's mapped to "INFORMATIONAL" in security_result.severity. If it's "Error", it's mapped to "ERROR" in security_result.severity.
severity_score	asset.attribute.labels.value	The value of severity_score is converted to a string and mapped to the value field within asset.attribute.labels. The corresponding key is set to "Severity Score".
SiloID	security_result.detection_fields.value	The value of SiloID is mapped to the value field within security_result.detection_fields. The corresponding key is set to "Silo ID".
SourceModuleName	target.resource.name	The value of SourceModuleName with quotes removed is mapped to target.resource.name.
SourceModuleType	observer.application	The value of SourceModuleType with quotes and closing brackets removed is mapped to observer.application.
Status	network.http.response_code	The value of Status is mapped to network.http.response_code and converted to an integer.
tags	asset.attribute.labels	For each element in the tags array, the type field is mapped to key and the name field is mapped to value within asset.attribute.labels.
Thread	security_result.detection_fields.value	The value of Thread is mapped to the value field within security_result.detection_fields. The corresponding key is set to "Thread".
timestamp	event.timestamp, metadata.collected_timestamp, read_only_udm.metadata.event_timestamp	The timestamp field is converted to a timestamp and mapped to event.timestamp for JSON logs and metadata.collected_timestamp for entity events. For syslog messages, it's mapped to read_only_udm.metadata.event_timestamp.
title	vulnerabilities.description	The value of title is mapped to vulnerabilities.description.
total_vulnerabilities	asset.attribute.labels.value	The value of total_vulnerabilities is converted to a string and mapped to the value field within asset.attribute.labels. The corresponding key is set to "Total Vulnerabilities".
URI	security_result.detection_fields.value	The value of URI is mapped to the value field within security_result.detection_fields. The corresponding key is set to "URI".
User-Agent	network.http.user_agent, network.http.parsed_user_agent	The value of User-Agent is mapped to network.http.user_agent. It's also mapped to network.http.parsed_user_agent and converted to a parsed user agent object. Hardcoded to "Rapid7 Insight". Hardcoded to "Rapid7 Insight". Hardcoded to "ASSET" for JSON logs. Set to "GENERIC_EVENT" initially, then potentially changed to "PROCESS_UNCATEGORIZED", "STATUS_UPDATE", or "USER_LOGIN" based on other fields. Set to "AUTHTYPE_UNSPECIFIED" for "USER_LOGIN" events. Set to "ALLOW" or "BLOCK" based on product_event_type. Hardcoded to "RAPID7_INSIGHT" for syslog messages.
username	principal.user.user_display_name	The value of username, with quotes removed and potentially parsed for email address, is mapped to principal.user.user_display_name. The extracted email address, if present, is mapped to principal.user.email_addresses.

Changes

2024-05-13

• Added support for the new format of SYSLOG+KV logs.

2023-05-05

• Enhancement
• Mapped "tags.type" to "asset.attribute.labels.key".
• Mapped "tags.name" to "asset.attribute.labels.value".

2022-12-15

• Enhancement
• Enhanced parser to parse application logs that are of Syslog format.
• Added Grok patterns for the logs with type "Session created", "Session destroyed", "Authentication attempt succeeded", and "Authentication attempt failed".