Using Regular Expressions within Investigator

Discussion created by RSA Admin

on Oct 30, 2012
Latest reply on Apr 24, 2014 by RSA Admin

Like • Show 0 Likes0
Comment • 12

Hello,

I recently blogged about my experiences around detecting domains in alias.host field created by Domain Generating algorithms within investigator by using some regex. I'm not sure if the community allows link outs but I have screenshots and such there that would be a bit cumbersome to recreate here. Also I wanted to get the word out about the home version of Investigator for people toying with info sec at home to the 3 or so people that stumble upon my blog a month.

I'm interested in any more information around the syntax for regex within investigator. I view it as powerful as the regex abilities of an IDS signature, which I guess informer alerts would qualify as.

https://scottfromsecurity.com/blog/2012/10/28/rsa-netwitness-investigator-regular-expressions

If there is an issue around posting links, just let me know. I didn't really see a "community rules" sticky.

Outcomes

12 Replies

RSA Admin Nov 16, 2012 4:57 PM
Regex content searching or queries within Investigator follows the Perl regex conventions.

In addition, when using it as part of a query, you should place quotes around your expression and escape any quotes in your expression using a backslash.
Like • Show 0 Likes0
Actions
- RSA Admin @ RSA Admin on Nov 19, 2012 2:44 PM
  Thanks Scott,
  
  Do you know of any documentation around use of Regex within Investigator?
  
  It was at Netwitness Analyst I and II training I attended in Reston, VA where I was told by the instructor that Investigator uses BOOST regex engine as opposed to perl's interpreter, so I was just going by that. Looking at the NW Investigator User Guide 9.7, I see the below..
  
  NetWitness uses the Boost Perl regular expression engine. All regular expressions must be formatted in the
  appropriate syntax. More information about the Boost Perl regular expression library and syntax can be found at
  the Boost Homepage (pg 85)
  
  I'm not familiar enough with the difference to know if it even really matters for the non-complicated Regex search strings, but any clarification you can provide is welcome. I've just been using trial and error to until the regex is accepted by the custom drill at this point as well as using this link for syntax reference.
  
  http://www.boost.org/doc/libs/1_34_0/libs/regex/doc/syntax_perl.html
  Like • Show 0 Likes0
  Actions
  - RSA Admin @ RSA Admin on Nov 30, 2012 1:45 PM
    AFAIK, it's completely compatible with perl syntax, but I admit I'm not a perl regex expert by any stretch. The current version of boost we use is 1.48, so this link would be better:
    
    http://www.boost.org/doc/libs/1_48_0/libs/regex/doc/html/boost_regex/syntax/perl_syntax.html
    
    Scott
    Like • Show 0 Likes0
    Actions
    - RSA Admin @ RSA Admin on Jan 15, 2013 1:04 AM
      Scott,
      Is there a way to export regular expression patterns, so other users in my organization can easily import them to their worksations and start using them?
      Like • Show 0 Likes0
      Actions
      - RSA Admin @ RSA Admin on Jan 15, 2013 8:10 AM
        Unfortunately, there isn't a way via the UI to import new searches. However, you can copy your expressions out of the settings file under your profile and paste them into another. The Investigator settings file is an XML file and can be found under your profile Roaming\NetWitness\NwInvestigator9.settings
        
        Copy the full <Queries>...</Queries> node into another user's settings file. Make sure Investigator is not running before editing the file.
        
        HTH,
        Scott
        Like • Show 0 Likes0
        Actions
NTRSPhil Apr 24, 2014 11:00 AM
This syntax does not appear to work when I put a comma between {}, I get the following error:

This is what I used to get this this error:
query regex \/[0-9a-zA-Z]{7,}\/\?[0-9a-zA-Z]{5,};

The below query patterns I am trying to match:

Running my regex through an online regex tester this works fine. I have checked my syntax against Perl, Boost Perl, Ruby, and python. All of them accept this command; why does investigator fail to understand this?
Like • Show 0 Likes0
Actions
- RSA Admin @ NTRSPhil on Apr 24, 2014 11:03 AM
  You need to quote your value. Quoting strings is mandatory within the query language. If you use a quote within the regex, you will need to escape it \'.
  
  Also, I recommend upgrading your Investigator client to 9.8.5.19, which has a fix to prevent mangling the regex when it's submitted, assuming it's properly quoted.
  Like • Show 0 Likes0
  Actions
  - NTRSPhil @ RSA Admin on Apr 24, 2014 11:11 AM
    So the above Regex should look like:
    query regex "\/[0-9a-zA-Z]{7,}\/\?[0-9a-zA-Z]{5,};"
    
    Even if I use quotes, I will need to upgrade to 9.8.5.19 for the quotes to even work?
    Like • Show 0 Likes0
    Actions
    - RSA Admin @ NTRSPhil on Apr 24, 2014 11:16 AM
      Yes, you need to quote it, just like that.
      
      It might or might not work with a version earlier than 9.8.5.19. In particular, it didn't like parenthesis before 9.8.5.19 and possibly a few other characters.
      
      9.8.5.19 should resolve any regex issues you have though.
      
      If you want to make sure the expression you sent is received on the concentrator/broker correctly, you will need to look at the audit log on the service to make sure the query is formatted correctly. If it isn't, then you know Investigator mangled it.
      Like • Show 0 Likes0
      Actions
      - NTRSPhil @ RSA Admin on Apr 24, 2014 11:19 AM
        Thanks that worked.
        Like • Show 0 Likes0
        Actions
        NTRSPhil @ NTRSPhil on Apr 24, 2014 11:38 AM
        Spoke to soon, looking at the concentrator shows that the regex query is being mangled with extra escape characters:
        \"\\/[0-9a-zA-Z]{7}\\/\\?[09-a-zA-Z]{5,};\"
        Like • Show 0 Likes0
        Actions
        RSA Admin @ NTRSPhil on Apr 24, 2014 11:42 AM
        You didn't pass the full audit log, so I can't tell, but that may be legit. There are outer quotes passed for the full query and the inner quotes need to be escaped as part of the actual query. After the first level parsing, the second level would return to regular quotes. Try using single quotes ' instead of ".
        
        Regardless, if you got the results you were expecting, it probably parsed fine.
        Like • Show 0 Likes0
        Actions

Using Regular Expressions within Investigator

Outcomes

Related Content

Recommended Content