Log Parser Customize: Extend an Existing Log Parser

Document created by RSA Information Design and Development on Jul 25, 2018Last modified by RSA Information Design and Development on Sep 20, 2018
Version 4Show Document
  • View in full screen mode
 

Note: The information in this topic applies to RSA NetWitness® Platform Version 11.2 and later.

One typical use case is for extending the capabilities of an existing log parser. In RSA NetWitness® Platform 11.2, you can add rules to a log parser to extend its parsing capabilities. In this topic, we walk through an example of this.

Task Overview

In this example, a customer wants to parse some items from the logs that are not currently being parsed by the existing log parser. Perform the following tasks:

  1. For your event source, get examples of the logs.
  2. In the CONFIGURE > Log Parser Rules view, Add the Log Parser
  3. From your sample logs, paste applicable sections into the Sample Log Messages section of the Log Parser Rules screen.
  4. Use the sample area to understand which items are being parsed by the current parser, and note the items that are not being parsed.
  5. For anything that is not currently being parsed, add rules as described in Add Rules and Deploy.
  6. Save the new rules, and deploy them to all Log Decoders.

Notes

Note: All the procedures in the topic use the CONFIGURE > Log Parser Rules view.

In the Log Parser Rules tab, you may see the Refresh icon () next to an item. This indicates that the item has undeployed changes.

Add the Log Parser

The first step in the process is to add a log parser, based on an existing log parser that you want to customize.

To add a log parser

  1. In the RSA NetWitness® Platform menu, navigate to CONFIGURE > Log Parser Rules.
  2. In the Log Parsers panel, click Add Parser.

    The Add Dynamic Log Parser dialog box is displayed.

  3. In the SELECT LOG PARSER field, select the existing parser to extend. In this example, we use Cisco Pix Firewall.
  4. You can clone the rules from any of your existing parsers, including the default parser. For simplicity, in this example we leave this field blank: thus, only the rules we create are added to the new parser.
  5. Click Add Parser to create the new parser.

The new parser is listed in the Log Parsers panel. Note the symbol next to the new parser—this indicates that your changes have not yet been saved.

About Custom Rules

When you create a new log parser rule, it is saved to an XML definition file for the parser. These files are known as token files. This is important, since the out-of-the-box rules are overwritten if you update the parser through RSA Live, but any custom log parser rules are not overwritten, since Live does not update the token files for log parsers.

Add Rules and Deploy

Once you have added the parser, the next step is to add one or more rules.

Let's say you know that your log messages have some email addresses that follow a "source_mail" string. You could add the following rule to parse these strings:

IMPORTANT: If you click on another parser in the Log Parsers panel, before you save your rule, your changes will be lost.

  1. Make sure the Cisco Pix Firewall parser is selected.

  2. In the Rules panel, click Add Rule.

    The Add New Rule dialog box is displayed.

  3. Enter a name for the rule, and click Add New Rule.

    The center panel is updated to reflect that you are working on a new rule.

  4. In the TOKENS section, enter a string for the token that you want to match, then click +.

    In this example, we entered email .

    Note: Make sure to include a delimiter for your token. For example, in this case, the token consists of 6 characters: the string "email," and then a space. Some tokens might use a colon, semicolon, or some other character as the delimiter, but it can be easy to forget to add the space character when that is the delimiter.

    You can enter more tokens, or continue to add values.
  5. In the VALUES section, choose the value for the rule. If you choose to match a Regex Pattern, you need to enter the pattern in the PATTERN field. Other values do not require any options.

    In this example, we selected Email.

  6. In the META section, click to select a meta key to which the rule stores its information. Some notes:

    • Enter characters to filter the list of available meta keys.
    • For Regex values, you can select "pieces" of the value, and store each piece to its own meta key.

    Note: If any new meta keys are added to the Log Decoder, they do not appear in the list of Meta immediately. They appear automatically after 24 hours, or you can restart the content server service to view them.

    In this example, we selected the email.src meta key.

    The following image shows an example rule:

  7. Click Save to save the rule. Repeat this procedure to continue adding rules.

  8. Once you have added all of your rules, click Deploy to deploy the new parser to your Log Decoders. Some notes about deploying rules:

    • You deploy an entire set of rules for a parser. That is, you can continue adding rules for a specific parser until you have all of your rules, and then you can deploy them all at once.
    • Once you deploy a custom parser, you can no longer delete it. You can only delete parsers that you have not yet deployed.

Note: In this example, we extended an existing log parser. However, if you are creating a new log parser for a new event source, make sure to map the new log parser to the IP address of the event source, as described in "Acknowledging and Mapping Event Sources" in the Event Source Management User Guide.

Regex Values

Custom Log Parser Rules can match regular expression patterns. If you select a Regex pattern for your Value, you can capture the entire matched token, or sections of it:

  • Full capture: the entire matched string is stored to your selected meta key.
  • First capture: the first portion of the string, up to the period character, is stored to the meta key.
  • Second capture: the second portion of the string, starting after the first period character, is stored to the meta key.
  • Third capture: the third portion of the string, starting after the second period character, is stored to the meta key.

You can choose any or all four of these captures, depending on the token you are matching.

For example, we examine the Source IP or IP:Port RSA rule:

  • Regex Pattern: \s*(\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b):?(\d*)
  • Full capture: none
  • First capture: ip.src
  • Second capture: port.src
  • Third capture: none
  • Assume example string of "src=192.168.24.4:8080", where src is one of the tokens defined for this rule:

    • 192.168.24.4 is saved to the ip.src meta key.
    • 8080 is saved to the port.src meta key.

For more details, see any online reference that describes PERL regular expressions. There are many tutorials available online.

IMPORTANT: Be careful when constructing regular expressions in your custom rules. Badly constructed regular expressions could impact your performance.

Previous Topic:Use Cases
You are here
Table of Contents > Extend a Log Parser Example

Attachments

    Outcomes