Parser Rules TabParser Rules Tab
Note: The information in this topic applies to RSA NetWitness Platform Version 11.1 and later.
This tab contains details about the rules for the default log parser, as well as any other custom rules and log parsers that have been defined.
To access this tab, go to (Configure) > Log Parser Rules.
This workflow shows processes available from the Log Parser Rules view.
What do you want to do?
|Role||I want to...||Documentation|
*View log parser rules.
*Add, edit or delete a log parser rule (version 11.2 and later)
*Add or remove a log parser (version 11.2 and later)
*You can perform this task here.
Note: The list of log parsers is based on the first Log Decoder that is installed or registered by the Orchestration Server. If you have more than one Log Decoder, this tab only lists log parsers that have been configured on the first one.
The Log Parser Rules tab organizes and displays information about the configured log parsers in your system.
If you select the Dynamic Rules entry for a parser, the screen displays the dynamic rules:
Note: The icon indicates that there is some uncompleted or unsaved work. For example, next to a parser name, it means that the parser has not yet been deployed.
If you select JSON Mappings, the screen displays JSON mappings for the parser.
Log Parsers PanelLog Parsers Panel
The Log Parsers Panel lists the configured log parsers.
The Add Dynamic Log Parser dialog box allows you to add a custom log parser.
When you are adding a log parser, the following parameters are available.
|select log parser||
Select NEW, or choose an existing log parser.
By choosing an existing log parser, you can add rules to that parser, essentially extending its parsing capabilities.
Note: If you select an existing log parser, the remaining fields are auto-filled based on the values for selected log parser.
Enter a string to define the device type. The name must be between 3 and 30 alphanumeric characters (including underscores), and must not match the name of any existing log parsers.
|device display name||
Enter the display name for the log parser.
Note: The display name must be 64 characters or fewer, and must not match the name of any other device display name.
Select a device class.
|clone dynamic parser rules from||
Leave blank to start with no rules, or select one of the existing log parsers to clone its rules.
Dynamic RulesDynamic Rules
If you select the Dynamic Rules entry for a parser, you see the following panes:
Details PaneDetails Pane
The details pane shows the three pieces for the selected rule:
- Tokens: one or more tokens to match in the message. For example, the Any Port rule looks for the following strings to match against: port , port:, port=, and others.
Values: the value that follows the token. This is a string that is captured as meta. For example, assume a log contains the following string:
The Any Port rule has a token that matches "port ". When it encounters that string, it assigns the token value, "12345" to a meta key.
- Meta: the meta keys to which the value is mapped. For example, the Any Port rule maps the port value to the port meta key.
Essentially, a rule says, "when you are parsing a message, if you match one of my tokens, assign the value that follows the token to the meta key that I want it stored as."
The bottom section of the Details panel contains sample log messages, and how they would be parsed for the selected log parser.
Displays the name of the selected log parser, and the buttons for deploying, saving, and discarding changes. This value changes when you select a different parser.
Displays the name of the selected rule. This value changes when you select a different rule for this parser.
Displays the list of tokens defined for the selected rule.
Displays the type and pattern of the value matching for the selected parser. The values here are determined by the type of the selected value. You can also use the Regex option to define a custom regular expression.
Displays the NetWitness meta to which the selected rule maps any matched tokens. The values here are determined by the selected Rule.
Displays a sample log message, and highlights strings that match tokens in the selected log parser. You can edit this field, and add in your own logs to preview how the selected parser will parse your logs.
Note: The sample section refreshes whenever a rule is changed or updated, as well as when you paste in samples from your logs.
For example, consider the following scenario:
- The default parser is selected.
- The Any Domain rule is selected.
- The Tokens matching list displays all of the tokens that are matched when found in a log message: Domain, Domain Name, domain, ADMIN_DOMAIN, and so on.
- The Meta list displays the NetWitness meta to which the value for the token is mapped: domain.
So, let's say the sample log message area has the following text:
Below are sample log messages:
May 5 2010 15:55:49 switch : %ACE-4-400000: IDS:1000 IP Option Bad Option List by user firstname.lastname@example.org from 10.100.229.59 to 126.96.36.199 on port 12345.
Apr 29 2010 03:15:34 pvg1-ace02: %ACE-3-251008: Health probe failed for server 188.8.131.52:81, connectivity error: server open timeout (no SYN ACK) domain google.com with mac 06-00-00-00-00-00.
In this case, the Sample Log Message area looks like this:
Note that some strings are highlighted, and that there are two "pairs" of highlight colors:
Dark blue and light blue highlighting is applied to the strings that match the currently selected rule.
- Dark Blue highlighted strings match a token in the selected rule. In this case, domain is the token that is matched for the Any Domain rule.
- Light Blue highlighted strings are the values that correspond to the tokens in dark blue. For example, google.com is highlighted in light blue, because it corresponds to the domain token.
Orange and yellow highlighting is applied to the strings that match rules for the current parser that are not currently selected.
- Orange highlighted strings match a token in a rule that is not currently selected.
- Yellow highlighted strings are the values that correspond to the tokens in orange. For example, the user token matches the Username rule (which is not currently selected).
In this example, the domain meta would be assigned a value of google.com for this log message, if it was parsed using the default log parser.
Rules PaneRules Pane
The Rules pane displays the list of rules used by the selected log parser. When you select a rule, you change the values that are displayed in both the Tokens and Values areas of the panel.
Note the highlighted rules:
Other notes for the Rules panel:
JSON MappingsJSON Mappings
If you select the JSON Mappings entry for a parser, you see the following panes:
- Sample JSON Message
- Meta Mappings
- Mapping Details
Sample JSON MessageSample JSON Message
You can enter or paste sample JSON text. Click the Render JSON button to automatically format the text into JSON code. If the text is not valid JSON, you receive a message and the text is not formatted.
The following screen shows the Sample JSON Message area with some JSON that has been pasted in:
Since this is valid JSON, clicking Render JSON produces the following:
Note that you can see the tree mode of the sample JSON if you click Mapping Mode:
Note: You can edit the sample JSON in Editing mode, but not in Mapping mode. Mapping mode is read only.
On the other hand, if you enter text that is not valid JSON, the screen looks as follows:
Auto Discover JSON MappingsAuto Discover JSON Mappings
Beginning with NetWitness Platform version 11.5.1, you can automatically create the mappings without the need to manually enter the name and path of the mapping. For details, see .Auto Discover JSON Mappings.
When the system auto discovers mappings, the path is filled in automatically, based on the structure of the mappings. For simple name-value pairs, this is straightforward. For example, for this pair, "host_name": "WIN-OT2OAJHG9NN" , the path is set to /host_name.
However, the rules for nested mappings and arrays are as follows:
For a nested structure, names are separated with a forward slash (/).
The path is set to /parent/child.
Arrays are accessed by omitting the index.
The path is set to /array/.
The path is set to /array//name.
Variable names are accessed by omitting them from the path.
The paths are set to /root/x/name and /root/y/name.
Remove Unmapped EntriesRemove Unmapped Entries
If there are mappings that you do not want to save, you can remove them. After you validate all of the mappings that you want to keep, you can click Remove Unmapped to remove all mappings that you have not yet validated.
For example, assume you have auto-discovered some mappings as shown here:
After you click Remove Unmapped, you can see that only mapped entries remain:
Note: You cannot save your work until all of the entries have either been mapped or removed from the list.
Meta MappingsMeta Mappings
This panel lists the mappings that exist for this parser. You can add a mapping by using the Add New button, or delete an existing mapping by selecting it and clicking Delete.
Mapping DetailsMapping Details
The Mapping Details pane displays the following information.
This name corresponds to the name displayed in the JSON Mappings panel.
The path to where the values for this portion of the log are stored.
Optionally, you can enter a text description for this mapping.
Select a meta key to which this value from the log is mapped. Select a value from the drop-down menu.
Optional if you choose a Value Format.
Choose a value format parser onto which to pass this JSON value. You can either select an existing meta or Custom Regex Type. If you select custom regex type, you must define the regex and capture to fine parse the value in the meta.
Optional if you choose a Meta.
custom regex type
Select Custom Regex Type from the Value Format drop-down, which allows you to add new custom regex type.
|regex pattern||Specify a regex to identify different pieces of data contained within a JSON node value.|
Select a meta key that should be captured first based on the value defined in the Regex pattern.
add a capture
New capture field is added. By default, it is loaded with meta keys in the drop-down. You can add maximum of 20 captures and this option will be disabled once it reaches maximum.
Note: You need to select a meta or enter a Value Format, but you do not need to fill in values for both settings.
Disable log Parser RulesDisable log Parser Rules
You can disable log parser rules, so that none of them are processed by the Log Decoder. You might have your log parsers working as you like, and do not want any extra processing that you do not need.
You disable them from the reference Log Decoder.
- Go to (Admin) > Services.
In the Administration Services view, select the Decoder and > View > Config.
The Services Config view is displayed with the General tab open.
Under Parsers Configuration, look at the Config Value for PARSERULESCAN.
If it is Enabled, log parser rules are processed. If it is Disabled, they are not processed.
If the rules are Enabled, click Enabled and select Disabled to disable the log parser rules.
To save the changes, click Apply.