Collect MISP IOC logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

Overview

This parser extracts IOCs from MISP data formatted as JSON or CSV. It parses the input, maps fields to the UDM, handles various IOC types (IP, domain, file hashes, etc.), and enriches the data with threat intelligence context like confidence and severity. The parser also performs specific logic for different data formats and handles cases with missing or unsupported fields.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to MISP.

Configure a feed in Google SecOps to ingest the MISP Threat Intelligence logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed (for example, MISP Logs).
4. Select Webhook as the Source type.
5. Select MISP Threat Intelligence as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label applied to the events from this feed.
8. Click Next.
9. Review the feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key. You cannot view this secret key again. If needed, you can regenerate a new secret key, but this action makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need to specify this endpoint URL in your client application.
13. Click Done.

Create an API key for the webhook feed

1. Go to Google Cloud console > Credentials.

   Go to Credentials

2. Click Create credentials, and then select API key.

3. Restrict the API key access to the Google Security Operations API.

Specify the endpoint URL

1. In your client application, specify the HTTPS endpoint URL provided in the webhook feed.

2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

   X-goog-api-key = API_KEY
   X-Webhook-Access-Key = SECRET
   

   Recommendation: Specify the API key as a header instead of specifying it in the URL.

3. If your webhook client doesn't support custom headers, you can specify the API key and secret key using query parameters in the following format:

   ENDPOINT_URL?key=API_KEY&secret=SECRET
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google Security Operations.
   • SECRET: the secret key that you generated to authenticate the feed.

Enable send HTTP requests to external URL in MISP

1. Access the machine where MISP is deployed.
2. Edit the file config.php located in app/Config/.
3. Find the rest_client_enable_arbitrary_urls configuration and set it to True. none 'rest_client_enable_arbitrary_urls' => true
4. Save the file.

Configure webhook in MISP for Google SecOps

1. Sign in to MISP Threat Intelligence.
2. Go to Administration > Workflows > List Modules.
3. Select Webhook.
4. Go to Action column and click Enable.
5. Go to Workflows > List triggers.
6. Select the webhook trigger of your choice.
7. Select Webhook from the modules list.
8. Drag and Drop it into the workspace.
9. Connect the input with the trigger's output.
10. In the Webhook module, fill in the following fields:
    • URL: Enter the ENDPOINT_URL, followed by the API_KEY and SECRET.
    • HTTP Request Method: Select POST
11. Click Save.

UDM Mapping Table

Log Field	UDM Mapping	Logic
Attribute.category	event.idm.entity.metadata.threat.category_details	Directly mapped from Attribute.category in the nested JSON object within the "data" field. Used in the JSON parsing path.
Attribute.comment	event.idm.entity.metadata.threat.summary	Directly mapped from Attribute.comment in the nested JSON object within the "data" field. Used in the JSON parsing path.
Attribute.deleted	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Attribute.deleted and added as a detection field with key "Attribute deleted". Used in the JSON parsing path.
Attribute.event_id	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Attribute.event_id and added as a detection field with key "Attribute event_id". Used in the JSON parsing path.
Attribute.first_seen	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Attribute.first_seen and added as a detection field with key "Attribute first_seen". Used in the JSON parsing path.
Attribute.id	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Attribute.id and added as a detection field with key "Attribute id" or "Attribute id $$" depending on the parsing path. Used in both CSV and JSON parsing paths.
Attribute.timestamp	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Attribute.timestamp and added as a detection field with key "Attribute timestamp". Used in the JSON parsing path.
Attribute.to_ids	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Attribute.to_ids and added as a detection field with key "Attribute to_ids". Used in the JSON parsing path.
Attribute.type	log_type	Directly mapped from Attribute.type in the nested JSON object within the "data" field. Used as an interim field, and later used to populate other UDM fields. Used in the JSON parsing path.
Attribute.uuid	event.idm.entity.metadata.product_entity_id	Directly mapped from Attribute.uuid in the nested JSON object within the "data" field. Used in the JSON parsing path.
Attribute.value	Multiple	The value of this field is used to populate several UDM fields depending on the Attribute.type (or log_type if derived from Attribute.type): - event.idm.entity.entity.hostname if type is "domain". - event.idm.entity.entity.file.md5 if type is "md5". - event.idm.entity.entity.file.sha1 if type is "sha1". - event.idm.entity.entity.file.sha256 if type is "sha256". - event.idm.entity.entity.resource.name if type is "mutex". - event.idm.entity.entity.registry.registry_key if type is "regkey". - event.idm.entity.entity.user.email_addresses if type is "threat-actor". - event.idm.entity.entity.url if type is "uri" or "url". - event.idm.entity.entity.file.full_path if type is "filename". - Parsed for IP and port if type is "ip-dst|port", "ip-dst", or "ip-src". Used in the JSON parsing path.
column1	event.idm.entity.metadata.product_entity_id	Directly mapped from column1 in the CSV parsing path.
column14	Part of event.idm.entity.metadata.threat.description	Concatenated with description to form the final description in the threat metadata. Used in the CSV parsing path.
column16	event.idm.entity.metadata.threat.threat_feed_name, event.ioc.feed_name	Directly mapped from column16. Used in the CSV parsing path.
column18	event.idm.entity.metadata.threat.severity_details, event.ioc.raw_severity	Directly mapped from column18. Used in the CSV parsing path.
column21	Part of event.idm.entity.metadata.threat.description, event.ioc.description	Used as the base for the description, later concatenated with event_info. Used in the CSV parsing path.
column3	Part of event.ioc.categorization	Directly mapped from column3 and concatenated with " IOCs" to form the final categorization. Used in the CSV parsing path.
column4	event.idm.entity.metadata.description	Directly mapped from column4. Used in the CSV parsing path.
column5	Multiple	The value of this field is used to populate several UDM fields depending on the column4 field (which maps to type): - event.idm.entity.entity.hostname if type is "domain". - Parsed for IP and port if type is "ip-dst|port", "ip-dst", or "ip-src". - event.idm.entity.entity.file.md5 if type is "md5". - event.idm.entity.entity.file.sha1 if type is "sha1". - event.idm.entity.entity.file.sha256 if type is "sha256". - event.idm.entity.entity.resource.name if type is "mutex". - event.idm.entity.entity.registry.registry_key if type is "regkey". - event.idm.entity.entity.user.email_addresses if type is "threat-actor". - event.idm.entity.entity.url if type is "uri" or "url". - event.idm.entity.entity.file.full_path if type is "filename". Used in the CSV parsing path.
column6	event.idm.entity.metadata.threat.summary	Directly mapped from column6. Used in the CSV parsing path.
column8	event.ioc.active_timerange.start, event.idm.entity.metadata.interval.start_time	Parsed as a UNIX timestamp. Used in the CSV parsing path.
date description	event.idm.entity.metadata.threat.description	Directly mapped from description in the nested JSON object within the "data" field. Used in the JSON parsing path.
event_creator_email	event.idm.entity.entity.labels.value	Directly mapped from event_creator_email and added as a label with key "event_creator_email". Used in the JSON parsing path.
event_id Feed.publish	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Feed.publish and added as a detection field with key "Feed publish". Used in the JSON parsing path.
first_seen	event.ioc.active_timerange.start, event.idm.entity.metadata.interval.start_time	Parsed as a timestamp in the format "yyyy-MM-ddTHH:mm:ssZZ". Used in the JSON parsing path.
id info	event.idm.entity.metadata.description	Directly mapped from info in the nested JSON object within the "data" field. Used in the JSON parsing path.
last_seen	event.ioc.active_timerange.end	Parsed as a timestamp in the format "yyyy-MM-ddTHH:mm:ssZZ". Used in the JSON parsing path.
Org.name	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Org.name and added as a detection field with key "Org name". Used in the JSON parsing path.
published	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from published and added as a detection field with key "published". Used in the JSON parsing path.
Tag.colour	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.colour and added as a detection field with key "tag colour". Used in the JSON parsing path.
Tag.exportable	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.exportable and added as a detection field with key "tag exportable". Used in the JSON parsing path.
Tag.hide_tag	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.hide_tag and added as a detection field with key "tag hide_tag". Used in the JSON parsing path.
Tag.id	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.id and added as a detection field with key "tag id". Used in the JSON parsing path.
Tag.is_custom_galaxy	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.is_custom_galaxy and added as a detection field with key "tag is_custom_galaxy". Used in the JSON parsing path.
Tag.is_galaxy	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.is_galaxy and added as a detection field with key "tag is_galaxy". Used in the JSON parsing path.
Tag.isinherited	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.isinherited and added as a detection field with key "tag isinherited". Used in the JSON parsing path.
Tag.name	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.name and added as a detection field with key "tag name". Used in the JSON parsing path.
Tag.numerical_value	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.numerical_value and added as a detection field with key "tag numerical_value". Used in the JSON parsing path.
Tag.user_id	event.idm.entity.metadata.threat.detection_fields.value	Directly mapped from Tag.user_id and added as a detection field with key "tag user_id". Used in the JSON parsing path.
threat_level_id	event.idm.entity.entity.labels.value	Directly mapped from threat_level_id and added as a label with key "threat_level_id". Used in the JSON parsing path.
timestamp	event.idm.entity.metadata.collected_timestamp, event.idm.entity.metadata.interval.start_time	Parsed as a UNIX timestamp. Used in the CSV parsing path.
uuid	event.idm.entity.metadata.vendor_name	Set to "MISP" by the parser. Set to "MISP" by the parser. Set to a default value far in the future (253402300799 seconds since epoch). Determined by the parser based on the type or log_type field. Can be "FILE", "DOMAIN_NAME", "IP_ADDRESS", "MUTEX", "RESOURCE", or "USER". Can be derived from Attribute.comment or Attribute.value depending on the Attribute.type. Parsed from Attribute.value or column5 if the type is IP related. Parsed from Attribute.value or column5 if the type is "ip-dst|port". Derived from column3 in CSV parsing or Attribute.category in JSON parsing. Derived from column21 and column14 in CSV parsing. Derived from column8 or first_seen. Derived from last_seen. Derived from description using a grok pattern. Derived from column16 or set to "MISP". Derived from column18. Parsed from Attribute.value or column5 if the type is IP related. Parsed from Attribute.value or column5 if the type is "ip-dst|port". Derived from Attribute.value or column5 if the type is "domain". Derived from the confidence field, which is extracted from the description field. Values can be "HIGH_CONFIDENCE", "MEDIUM_CONFIDENCE", "LOW_CONFIDENCE", or "UNKNOWN_CONFIDENCE". Directly mapped from the confidence field, which is extracted from the description field. Directly mapped from the threat_level field, which is derived from column18 in the CSV parsing path. Directly mapped from the feed_name field, which is derived from column16 in the CSV parsing path. Derived from column21 and column14 in CSV parsing. Derived from column6 in CSV parsing or Attribute.comment in JSON parsing. Several fields are added as detection fields with their corresponding keys. Several fields are added as labels with their corresponding keys. Copied from the top-level timestamp field in the raw log.

Changes

2023-09-26

• Mapped "published", "Feed.publish", "Org.name", "Attribute.id", "Attribute.event_id", "Attribute.to_ids", "Attribute.timestamp", "Attribute.comment", "Attribute.deleted", "Attribute.first_seen", all "tag.names" to "threat.detection_fields".

2023-08-17

• Bug-Fix :
• Added a condition to perform a 'gsub' operation, that removes extra backslash, only when log is not JSON.

2023-07-20

• Bug-Fix :
• Changed 'metadata.entity_type' to 'MUTEX' when log is of type mutex.

2023-07-04

• Newly created parser.