Collect OpenCanary logs

Supported in:

Google SecOps SIEM

Note: This feature is covered by Pre-GA Offerings Terms of the Google Security Operations Service Specific Terms. Pre-GA features might have limited support, and changes to pre-GA features might not be compatible with other pre-GA versions. For more information, see the Google Security Operations Technical Support Service guidelines and the Google Security Operations Service Specific Terms.

Overview

This parser extracts fields from OpenCanary SYSLOG and JSON logs, normalizes them into the UDM format, and enriches the data with derived fields like metadata.event_type and security_result.severity. It handles various log formats, performs IP address validation, and maps fields to appropriate UDM objects like principal, target, and network.

Before you begin

• Ensure that you have a Google SecOps instance.
• Ensure that you have privileged access to OpenCanary.

Configure a feed in Google SecOps to ingest the OpenCanary logs

1. Go to SIEM Settings > Feeds.
2. Click Add new.
3. In the Feed name field, enter a name for the feed (for example, OpenCanary Logs).
4. Select Webhook as the Source type.
5. Select OpenCanary as the Log type.
6. Click Next.
7. Optional: Specify values for the following input parameters:
   • Split delimiter: the delimiter that is used to separate log lines, such as \n.
   • Asset namespace: the asset namespace.
   • Ingestion labels: the label applied to the events from this feed.
8. Click Next.
9. Review the feed configuration in the Finalize screen, and then click Submit.
10. Click Generate Secret Key to generate a secret key to authenticate this feed.
11. Copy and store the secret key. You cannot view this secret key again. If needed, you can regenerate a new secret key, but this action makes the previous secret key obsolete.
12. From the Details tab, copy the feed endpoint URL from the Endpoint Information field. You need to specify this endpoint URL in your client application.
13. Click Done.

Create an API key for the webhook feed

1. Go to Google Cloud console > Credentials.

   Go to Credentials

2. Click Create credentials, and then select API key.

3. Restrict the API key access to the Google Security Operations API.

Specify the endpoint URL

1. In your client application, specify the HTTPS endpoint URL provided in the webhook feed.

2. Enable authentication by specifying the API key and secret key as part of the custom header in the following format:

   X-goog-api-key = API_KEY
   X-Webhook-Access-Key = SECRET
   

   Recommendation: Specify the API key as a header instead of specifying it in the URL.

3. If your webhook client doesn't support custom headers, you can specify the API key and secret key using query parameters in the following format:

   ENDPOINT_URL?key=API_KEY&secret=SECRET
   

   Replace the following:

   • ENDPOINT_URL: the feed endpoint URL.
   • API_KEY: the API key to authenticate to Google Security Operations.
   • SECRET: the secret key that you generated to authenticate the feed.

Setting up an OpenCanary Webhook for Google SecOps

1. Find the OpenCanary configuration file, config.json.

   Note: This file is usually found in the OpenCanary installation directory.

2. Open config.json file with a text editor.

3. Find the section labeled alerters within the configuration file.

4. If a webhook alerter already exists, modify it. Otherwise, add a new entry for the webhook alerter.

5. Use the following configuration (replace ENDPOINT_URL, SECRET and API_KEY with your values):

"handlers": {
    "Webhook": {
        "class": "opencanary.logger.WebhookHandler",
        "url": "<ENDPOINT_URL>",
        "method": "POST",
        "data": {"message": "%(message)s"},
        "status_code": 200,
        "headers": {
            "X-Webhook-Access-Key": "<SECRET>",
            "X-goog-api-key": "<API_KEY>"
         }
    }
}

1. Save the config.json file.
2. Restart the OpenCanary service to apply the changes. (for example, sudo systemctl restart opencanary).

Note: For more details about webhook configuration refer to OpenCanary documentation

UDM Mapping Table

Log Field	UDM Mapping	Logic
dst_host	target.asset.ip	The raw log's dst_host field is mapped to the UDM. Also mapped to target.ip.
dst_host	target.ip	The raw log's dst_host field is mapped to the UDM. Also mapped to target.asset.ip.
dst_port	target.port	The raw log's dst_port field is converted to a string and then to an integer and mapped to the UDM.
local_time	metadata.event_timestamp	The raw log's local_time field is used to populate the metadata.event_timestamp in the UDM. The parser uses the create_time from the batch object if the local_time field is not present.
local_time_adjusted	security_result.detection_fields	The raw log's local_time_adjusted field is added as a key-value pair to the security_result.detection_fields array in the UDM.
logdata.COMMUNITY_STRING	security_result.detection_fields	The raw log's logdata.COMMUNITY_STRING field is added as a key-value pair to the security_result.detection_fields array in the UDM.
logdata.DOMAIN	principal.administrative_domain	The raw log's logdata.DOMAIN field is mapped to the UDM.
logdata.FILENAME	target.file.full_path	The raw log's logdata.FILENAME field is mapped to the UDM.
logdata.HOSTNAME	principal.asset.hostname	If the logdata.HOSTNAME field is not an IP address, it's mapped to the UDM. Also mapped to principal.hostname.
logdata.HOSTNAME	principal.asset.ip	If the logdata.HOSTNAME field is an IP address, it's mapped to the UDM. Also mapped to principal.ip.
logdata.HOSTNAME	principal.hostname	If the logdata.HOSTNAME field is not an IP address, it's mapped to the UDM. Also mapped to principal.asset.hostname.
logdata.HOSTNAME	principal.ip	If the logdata.HOSTNAME field is an IP address, it's mapped to the UDM. Also mapped to principal.asset.ip.
logdata.LOCALNAME	principal.asset.hostname	The raw log's logdata.LOCALNAME field is mapped to the UDM. Also mapped to principal.hostname.
logdata.LOCALNAME	principal.hostname	The raw log's logdata.LOCALNAME field is mapped to the UDM. Also mapped to principal.asset.hostname.
logdata.LOCALVERSION	principal.platform_version	The raw log's logdata.LOCALVERSION field is mapped to the UDM.
logdata.PASSWORD	extensions.auth.mechanism	The presence of the logdata.PASSWORD field triggers the parser to set the extensions.auth.mechanism to USERNAME_PASSWORD in the UDM.
logdata.PATH	network.http.referral_url	The raw log's logdata.PATH field is mapped to the UDM.
logdata.REMOTENAME	target.asset.hostname	The raw log's logdata.REMOTENAME field is mapped to the UDM. Also mapped to target.hostname.
logdata.REMOTENAME	target.hostname	The raw log's logdata.REMOTENAME field is mapped to the UDM. Also mapped to target.asset.hostname.
logdata.REMOTEVERSION	target.platform_version	The raw log's logdata.REMOTEVERSION field is mapped to the UDM.
logdata.SMBVER	network.application_protocol	The presence of the logdata.SMBVER field triggers the parser to set the network.application_protocol to SMB in the UDM.
logdata.USERAGENT	network.http.parsed_user_agent	The raw log's logdata.USERAGENT field is converted to a parsed user agent and mapped to the UDM.
logdata.USERAGENT	network.http.user_agent	The raw log's logdata.USERAGENT field is mapped to the UDM.
logdata.USERNAME	target.user.userid	The raw log's logdata.USERNAME field is mapped to the UDM.
loglevel	security_result.severity	The raw log's loglevel field determines the security_result.severity in the UDM. WARNING maps to HIGH, INFO/INFORMATION maps to LOW.
logtype	security_result.detection_fields	The raw log's logtype field is added as a key-value pair to the security_result.detection_fields array in the UDM.
node_id	principal.asset.asset_id	The raw log's node_id field is prepended with "id:" and mapped to the UDM.
src_host	principal.asset.ip	The raw log's src_host field is mapped to the UDM. Also mapped to principal.ip.
src_host	principal.ip	The raw log's src_host field is mapped to the UDM. Also mapped to principal.asset.ip.
src_port	principal.port	The raw log's src_port field is converted to an integer and mapped to the UDM.
utc_time	security_result.detection_fields	The raw log's utc_time field is added as a key-value pair to the security_result.detection_fields array in the UDM.

Changes

2024-03-11

• Removed invalid character at the end of JSON logs to parse unparsed logs.
• Aligned "principal.ip" and "principal.asset.ip" mappings.
• Aligned "target.ip" and "target.asset.ip" mappings.

2024-02-08

• Newly created parser.