Manage prebuilt and custom parsers
This document describes how you can use the parser management feature to create custom parsers or opt in to or out of prebuilt parser updates initiated by Google Security Operations.
Changes to prebuilt parsers are released on a regular cadence as release candidates. During the release candidate window, you can choose to update one or more parsers with the pending changes. Every 4 weeks, pending updates automatically become active when the pending parser changes are promoted to default. The amount of time to evaluate a change depends on when the change was released during the release candidate window.
The parser management feature lets you inspect and test the update during the release candidate window. You can view a list of past changes to a prebuilt parser and also view upcoming changes in the release cadence. You can then opt in to or out of the update.
Google Security Operations also gives you the flexibility to create a custom parser for a log type that does not have a prebuilt parser. You can either create an entirely new parser directly from the raw log or use an existing parser as a base for a new custom parser. You can extend mapping instructions by creating a parser extension for either a prebuilt or custom parser.
The various types of parsers are as follows:
| Parser type | Description |
|---|---|
| Prebuilt | Parsers that are created by Google Security Operations and contain built-in data mapping instructions for transforming original log data to UDM fields. |
| Prebuilt extended | A prebuilt parser created by customers with additional mapping instructions to extract additional data from an original raw log and insert it into the UDM record. |
| Custom | Parsers that are created by customers with custom data mapping instructions for transforming original log data to UDM fields. |
| Custom extended | A custom parser created by customers with additional mapping instructions using a parser extension to extract additional data from an original raw log and insert it into the UDM record. |
Before you begin
The following documents explain prerequisite concepts that are important for managing parser updates:
Create a custom parser based on mapping instructions
You can create a custom parser by writing code that converts the original raw log to a UDM record. For the structure of a parser, see Overview of log parsing and Parser syntax reference for information about the syntax. When creating a parser, make sure the data mapping instructions populate as many important UDM fields as possible.
In the navigation bar, select Settings > SIEM Settings.
Click Create Parser.
Select an appropriate log source from the Log Source list.
Select Start with Raw Logs Only to create a new parser according to your requirements.
Click Create.
Type the code in the Parser Code Terminal. For more information, see Create a code snippet mapping instruction.
Optional: Click to edit the existing raw log or copy.
Optional: Click to load the latest raw log.
Click Preview to view the UDM output. An error message is displayed if the code is incorrect.
In the preview, you can use the statedump filter plugin to validate the internal state of a parser. For more information, see Validate data using statedump plugin.
Click Validate to validate the custom parser.
The validation process may take a few minutes, so we recommend that you preview the custom parser first, make changes if required, and then validate the custom parser.
Click Submit.
The parser is picked for normalization after 20 minutes.
Create a custom parser from an existing parser
You can use an existing parser as a template to create a new parser. You can create a custom parser using the code approach only. To create a custom parser from an existing parser, follow these steps:
From the application menu, select Settings > Parsers.
Click Create Parser.
Select an appropriate log source from the Log Source list.
Select Start with an Existing Prebuilt Parser to use an existing parser as a base to create a new custom parser.
Click Create.
Edit the code in the Parser Code Terminal. For more information, see Create a code snippet mapping instruction.
Optional: Click to edit the raw log.
Optional: Click to refresh the raw log.
As you add code to build the parser, click Preview to view the UDM output. An error message is displayed if the code is incorrect.
In the preview, you can use the statedump filter plugin to validate the internal state of a parser. For more information, see Validate data using statedump plugin.
Click Validate to validate the custom parser.
The validation process may take a few minutes, so we recommend that you preview the custom parser first, make changes if required, and then validate the custom parser.
Click Submit.
The parser is picked for normalization after 20 minutes.
Manage prebuilt parser updates
When Google Security Operations releases an update to a parser, the updates are in pending state for 15 days. To opt in to or out of a parser update, examine the difference between the earlier and the newer parser versions by doing the following:
Log in to your Google Security Operations instance.
From the application menu, select Settings > Parsers.
Click Filter.
Select Prebuilt, Active, and Prebuilt Extended from the list.
Your active prebuilt parsers are displayed. Prebuilt parsers are default parsers released by Google Security Operations. If the Update column contains Pending as the status, it indicates that the parser has an update that you can inspect.
Click Menu and select View pending update from the list.
The Compare parsers page appears. Here, you can view the following:
The code difference between current and the upcoming parser version
The changelogs in the Change logs tab
The generated UDM event for the sampled raw log
The date and time when the parser was created
The date and time when the parser code was last updated
You can either opt in early for the update, wait for it to get auto-applied in 15 days, or opt out of it.
Opt in early for parser updates
The parser management feature lets you opt in early to a parser update and test it. You can opt in early for parser updates only if you're using a prebuilt parser. Once you opt in early, you can revert the parser to its earlier version within 15 days of the update release. To opt in early for the update, follow these steps:
On the Compare parsers page, click Make parser update active.
The Confirm parser update dialog appears.
Click Confirm.
The parser is picked for normalization after 20 minutes.
Opt out of parser updates
To opt out of the current and future parser updates, create a custom parser. You can use your current or the updated version of the parser as a custom parser. All future updates to a custom parser will be visible to you, but won't get applied unless you opt in for them. To opt out of the current or future updates, follow these steps:
On the Compare parsers page, click Skip update.
The Skip update and create custom parser window appears.
Click Create custom parser.
To set the default parser version as the custom parser, select Prebuilt parser. To set the updated version as the custom parser, select Pending Parser Update.
Click Create.
The selected version is picked for normalization after 20 minutes. It appears as Custom and Active in the parsers list on the Parsers page. The earlier prebuilt version appears as Prebuilt and Inactive.
Manage custom parser updates
When you opt out of prebuilt parser updates, a custom parser is created. A custom parser is visible in the parsers list as a new entry.
Make a custom parser inactive
From the application menu, select Settings > Parsers.
Click Menu against the parser that you want to make inactive and select Make inactive from the list.
The Make parser inactive dialog appears.
Click Make inactive.
The custom parser is deactivated and the default parser version is activated after 20 minutes. That is, the custom parser becomes a prebuilt parser. If you have created a custom parser from a prebuilt parser with updates, then the updates are lost when you revert the custom parser to a prebuilt parser. You must again opt in for parser updates.
Delete a custom parser
From the application menu, select Settings > Parsers.
Click Menu against the parser that you want to delete and select Delete from the list.
The Delete custom parser dialog appears.
Click Delete.
The custom parser is deleted and the default parser version is activated after 20 minutes. That is, the custom parser becomes a prebuilt parser. If you have created a custom parser from a prebuilt parser with updates, then the updates are lost when you revert the custom parser to a prebuilt parser. You must again opt in for parser updates.
Create an extension
You can extend a custom or a prebuilt parser by defining custom mapping instructions to extract additional data from an original raw log. You can insert the data into the UDM record generated by a custom parser. You cannot create a new parser using parser extensions.
For information on creating parser extensions, see Using parser extensions.
Revert an early update of a prebuilt parser
If you have opted in early for a parser update, you can revert to the previous version within the 15-day window. To switch back to the previous parser version, follow these steps:
From the application menu, select Settings > Parsers.
Click Menu against the parser that you want to revert.
Click View.
The View prebuilt parser page appears.
Click Revert to previous version.
The Revert to previous dialog appears. You can click Compare Parsers on the dialog to see the difference between the current and the previous versions.
Click Confirm to revert the parser to its previous version.
The parser is reverted to its previous version after 20 minutes.
Control access to parser management
By default, parser updates can be managed by users with Administrator and Editor roles. New permissions can be granted to control who can view and manage parser updates. For more information about managing users and groups, or assigning roles, see the role-based access control user guide.