William Hart

Email Parsing Options

Blog Post created by William Hart Employee on Jul 1, 2016

We have got several requests to generate additional parser updates to some of the parsers we distribute in Live. One approach we have taken to make these optional changes available for customers is to allow for an external configuration file to be read by the original parser. This provides for some customization depending on environment requirements while eliminating the need for customers to be able to write a parser. The reason for them being optional is some configuration changes may generate more meta requiring further storage or may require additional parsing resources or not be appropriate to an environment.


As an example the default email parser, MAIL_Lua, reads email messages regardless of transport protocol (e.g. SMTP, IMAP, POP3) and registers all email addresses into the email meta key. In the options file a user can enable to register the email sender to email.src and the recipient to email.dst instead of registering them all to email. There are additional options available in the attached MAIL_lua_options.lua file that can be enabled/disabled as well.


To have these options take effect the following steps are required.


1) Upload the MAIL_lua_options.lua file into the /etc/netwitness/ng/parsers folder on the appropriate decoder where the MAIL_Lua parser is applied.


2) To enable the source and destination email change mentioned above modify the word false to true on line 24. This is the last line before the end of the function named registerEmailSrcDst


3) Validate that the default email source and destination meta keys are included in the appropriate concentrator default index file (e.g. index-concentrator.xml) located in /etc/netwitness/ng on the concentrator.  This file can be viewed by command line after logging in using secure shell or in administration section of the user interface at Administration > Services > <concentrator name> > config > Files tab. The lines that should be in there and that will be populated by this change are:


<key description="Source E-mail Address" level="IndexValues" name="email.src" format="Text" valueMax="2500000" />

<key description="Destination E-mail Address" level="IndexValues" name="email.dst" format="Text" valueMax="2500000" />


4) For this to take effect, a decoder service restart is required. This will cause a service interruption so recommend making this change during a maintenance window.


I welcome any suggestions on importance of these types of options for this scenario as well as others. As well if it would be better to have these options available in the Security Analytics user interface.


Note: I am just a conduit for this information and have to give credit for the creation of these parser options to RSA Content Engineers.