Seems I spend more time troubleshooting regexs instead of investigating events....for all the money spent on the tool its pretty primitive and time consuming to narrow the data down to what you want.
I figured simple things like ip.dst != 10.0.0.0/8 would simply work....or src.org !='org name" would work....
How about a better regex guide....I really dont have the time to figure out what works and what doesn't....
Scott,
RegExLib.com Regular Expression Cheat Sheet (.NET Framework)
here is some regex.
However ip.src, ip.dst, etc are meta tags and not regex's.
The tool isn't difficult to use.
Scott,
Under ideal circumstances, where would you like such guidance presented within the interface? Regex can obviously be quite complex, so that has to be considered. I know right now you'd probably be happy with a cheat sheet, and I'll see what I can do to find such a beast. Ping me at Michael.shreve@rsa.com
Welcome to the fundamental problem with "Big Data." The problem isn't getting the data, the problem is making sense of it when you've got a lot of it - finding the needle in the haystack. NW/SA is just a tool to help you make sense of it and hunt through the haystack, but it's not necessarily a "find evil" button.
As far as what isn't working, you'll have to be more specific. ip.dst != 10.0.0.0/8 and src.org != 'org name' should both work fine as netwitness/SA queries -- but they just may take a long time to run given the configuration of your infrastructure (i.e. indexing) and the amount of data you're dealing with. In fact there are timeouts built into the system for performance concerns so if those queries take long enough to run they may reach that timeout and be unceremoniously killed, leading to the impression that they "don't work."
Not-equals comparisons are very expensive, try to think about writing queries in a way that does equivalence comparisons to try to accomplish the same thing. In other words, = is in general likely to be a lot faster than != when doing queries against the database (the same doesn't necessarily hold true if you're doing them in parsers or app rules).
I have worked on SIEM and HADOOP for about 10 years. I have worked on envision, q1 and arcsight extensively. I didn't want to mention the dreaded "arcsight" word...but they have made it easy for filtering data quickly to get to the data you need. I don't want to play developer with a security platform, just have it do its job.
I do not want my analyst to go out and buy regex buddy to build the filter strings they want. The whole regex aspects just smacks of developer laziness. I want to drill and refine events in seconds without worrying about "will this regex work in this mode" or not.
The regexs work in some areas and not in others, perhaps its the web interface that's challenged, regardless. ip.src/ip.dst excluding rfc 1918 should work period overhead or not, or excluding orc.src/org.dst chained together should work. Why do I have spend a lot of hours figuring out which regex works in which instance. That's inconsistent behavior and doesn't belong in a SIEM platform. I have viewed comments across many of the blogs complaining about this behavior and RSA needs to address it.
I never use Regex. I don't think you should either since it seems you havent mastered the standard query language and app rule interface/custom query just yet.
Excluding RFC 1918 addresses is easy. For outbound traffic, it is org.dst exists. For inbound traffic, its org.src exists.
To reiterate, I never use regex. It's nice that it is available, but there has never been a use case that exists that cannot be addressed by using the standard query meta language.
Sorry, I do not mean regex, working on a problem while I am venting my frustration with the RSA interface, standard queries...or advanced as your interface calls them....
when you have 20 + orgs, and the query works "mostly" but not fully....I have a problem with that, please look at others in your forum struggling with standard queries NOT working. Suggesting work arounds to issues is skirting the problem, why are standard queries not working...
perhaps more importantly, you are totally missing my point, the interface is less than ideal. I have two major customers (not to mention a few dozen others) on RSA Envision. Do you really think I can recommend going to analytics with this interface? This is not 2002....I have had great hopes for so long and have been very patient. It seems we are taking steps backwards.
as for regex, try finding a dga...
Scott, I'm a product manager for Security Analytics. Specifically I'm the product manager for threat detection and intelligence. If you're feeling pain attempting to put together queries to assist you in looking for threats to your network, that absolutely concerns me. I'd like to be able to formally capture your specific pain points so we can make it less painful going forward. I get the general gist of functional inconsistency, but I'd much rather get as specific as I can in order to capture the necessary amount of information to formulate requirements against actual use cases.
I'd like to know why you think the standard queries do not work. In other words, please give a specific example along with the results you are receiving that are incorrect. Also, what version are you running?
For troubleshooting purposes, it would be even better if you could capture a NWD, pcap or log file that exhibits the problem with a specific session.
Thanks
Hi
May I ask whether the likes of the sample below is legal in SA REGEX?
alias.host regex "sam\d\dv\d\d\d\d\d\d","sam\d\dp\d\d\d\d\d","ph\d\dc\d\dn\d\d","ph\d\dc\d\dn\d\d","ph\d\dc\d\dn\d\d"
Thank you.
I haven't tested this, but I believe you need to double-escape the backslashes \\.
The parameter passing interface to NextGen services uses the backslash to escape certain characters, so if you double escape them, it should get passed to regex correctly (as a single \).
Let me know if that fixes the issue.
Hi Scott,
Thanks for the reply, I did some test and the following sample yielded the one below
(from investigator)
alias.host regex "sam\\d\\dv\\d\\d\\d\\d\\d\\d","sam\\d\\dp\\d\\d\\d\\d\\d","ph\\d\\dc\\d\\dn\\d\\d","de\\d\\dc\\d\\dn\\d\\d"
(when i look at the rest sdk stat queries it got the following)
flags=2305 id2=2484766586040 id1=1336423529586 where="(alias.host regex \"sam\\\\d\\\\dv\\\\d\\\\d\\\\d\\\\d\\\\d\\\\d\",\"sam\\\\d\\\\dp\\\\d\\\\d\\\\d\\\\d\\\\d\",\"ph\\\\d\\\\dc\\\\d\\\\dn\\\\d\\\\d\",\"de\\\\d\\\\dc\\\\d\\\\dn\\\\d\\\\d\") && time=\"2014-04-02 00:00:00\"-\"2014-04-03 00:00:59\"" fieldName=did expiry=3900000 threshold=100000 size=20
I guess it auto adds the extra "\"
So I just ran the following query on a concentrator:
client regex 'mozilla/\d\.\d'
which (on Investigator 9.8) turned into this query in the audit log:
id1=1 id2=318957729 size=20 flags=sessions,sort-total,order-descending threshold=100000 fieldName=client where="time=\"2014-Mar-06 23:23:00\"-\"2014-Mar-07 23:22:59\" && (client regex mozilla/\\\\d\\\\.\\\\d)"
And my results are correct:
id1=2249947 id2=3924797 count=8011 value=mozilla/5.0 type=client flags=0 group=0
id1=2515501 id2=3923474 count=5041 value=mozilla/5.0 (windows nt 6.1; wow64; rv:27.0) gecko/20100101 firefox/27.0 type=client flags=0 group=0
id1=2628520 id2=3922757 count=665 value=mozilla/5.0 (windows nt 6.1; wow64; trident/7.0; rv:11.0) like gecko type=client flags=0 group=0
id1=2250546 id2=3924889 count=347 value=mozilla/4.0 type=client flags=0 group=0
id1=2628152 id2=3924797 count=306 value=mozilla/5.0 (compatible; msie 10.0; windows nt 6.1; trident/7.0) type=client flags=0 group=0
id1=2734419 id2=3921778 count=271 value=mozilla/5.0 (linux; android 4.4.2; nexus 7 build/kot49h) applewebkit/537.36 (khtml, like gecko) version/4.0 chrome/30.0.0.0 safari/537.36 type=client flags=0 group=0
id1=3922309 id2=3924889 count=245 value=mozilla/4.0 (compatible; msie 7.0; windows nt 5.2; trident/4.0; .net clr 1.1.4322; .net clr 2.0.50727; .net clr 3.0.4506.2152; .net clr 3.5.30729) type=client flags=0 group=0
id1=2517374 id2=2618595 count=181 value=mozilla/5.0 (macintosh; intel mac os x 10_9_2) applewebkit/537.36 (khtml, like gecko) chrome/33.0.1750.146 safari/537.36 type=client flags=0 group=0
id1=2720497 id2=2733918 count=87 value=mozilla/5.0 (windows nt 6.1; trident/7.0; rv:11.0) like gecko type=client flags=0 group=0
id1=2517356 id2=2576611 count=64 value=mozilla/5.0 (macintosh; intel mac os x 10.9; rv:24.0) gecko/20100101 thunderbird/24.3.0 lightning/2.6.4 type=client flags=0 group=0
id1=2519422 id2=3922826 count=64 value=mozilla/5.0 (compatible; msie 10.0; windows nt 6.1; wow64; trident/7.0) type=client flags=0 group=0
id1=2550850 id2=3923095 count=64 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; win64; x64; trident/7.0; .net clr 2.0.50727; slcc2; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; infopath.3; tablet pc 2.0; microsoft outlook 14.0.7113; ms-office; msoff type=client flags=0 group=0
id1=2550094 id2=3922932 count=16 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; wow64; trident/7.0; slcc2; .net clr 2.0.50727; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e) type=client flags=0 group=0
id1=3922637 id2=3924782 count=10 value=mozilla/5.0 (windows nt 6.1; wow64) applewebkit/537.36 (khtml, like gecko) chrome/33.0.1750.146 safari/537.36 type=client flags=0 group=0
id1=2550052 id2=3922801 count=10 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; win64; x64; trident/7.0; .net clr 2.0.50727; slcc2; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; infopath.3; tablet pc 2.0) type=client flags=0 group=0
id1=2734897 id2=3920385 count=4 value=mozilla/5.0 (windows nt 6.1; win64; x64; trident/7.0; rv:11.0) like gecko type=client flags=0 group=0
id1=3920400 id2=3920403 count=4 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; win64; x64; trident/7.0; .net clr 2.0.50727; slcc2; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; infopath.3; tablet pc 2.0; ms-office; msoffice 14) type=client flags=0 group=0
id1=3923854 id2=3923856 count=2 value=mozilla/5.0 (windows nt 6.1; wow64; rv:24.0) gecko/20100101 thunderbird/24.3.0 lightning/2.6.4 type=client flags=0 group=0
id1=2720496 id2=2720504 count=2 value=mozilla/5.0 (windows nt 6.1; trident/7.0; rv:11.0) like gecko/20100101 firefox/12.0 type=client flags=0 group=0
id1=2628350 id2=2628353 count=2 value=mozilla/4.0 (compatible; msie 7.0; windows nt 6.1; trident/7.0; slcc2; .net clr 2.0.50727; .net clr 3.5.30729; .net clr 3.0.30729; media center pc 6.0; .net4.0c; .net4.0e; bri/2) type=client flags=0 group=0
Thank you very much for the clarification.
After looking at your use case in closer detail, I think it would serve you better to create a custom feed to tag your server infrastructure better. I imagine you already have a spreadsheet of some kind that identifies your KNOWN hostnames. This could serve as the basis of your feed.
Create a new feed called known infrastructure.
Column 1 would be all your hostnames. It is a non-ip feed. The meta callback would be alias.host for column 1.
Column 2 would be filled top to bottom with the feed.name of Known Infrastructure
Column 3 would be feed.category listed as physical or virtual or desktop, or whatever
Column 4 would be feed.description where you could further label what they do- mail, dns, domain controller, backup server, etc.
Once it is deployed, you should create a rule to look for new hosts that were not identified by your feed. (History shows this will be a bunch). That rule would be alias.host begins sam,ph where feed.name != "known infrastructure"
Once you find new hosts, get them added to your feed. Then wash, rinse, repeat.
And once you have your infrastructure properly labeled and tracked in SA, you can implement great use cases. such as:
Are there any unusual UA strings connecting to my mail servers?
Are windows desktops using the correct PDC or are they failing over to the BDC?
Is my VPN pool over-utilizing my virtual servers?
And the list could go on and on.
Thank you for that.
We've been seeing a lot of malware lately that uses long hex strings in filenames, directories, and sometimes in POSTs- ie:
GET /c83e2e10c9721fcbdadbefb5184397c7/pace-traditional.php
or
directory = /c83e2e10c9721fcbdadbefb5184397c7/
or
filename = 6d5ce4ac1c678e97a3cb45db2e0e2681.php
So I think that regex is the perfect (and maybe the only) way to search for these strings.
So far, after the better part of 2 days, I've got this regex, which works perfectly in several "regex testers", but does not produce the correct results in SA 10.3.2
directory regex '([A-Fa-f0-9]{20,})'
or
filename regex '([A-Fa-f0-9]{20,})'
Try removing the parenthesis, as they have a special meaning in the query language and might be tripping up the results.
Also, keep in mind that directory and filename likely have multiple values per session. All it takes is a single meta in a session to hit in order to return that session. This means you will likely see values in the result set which *do not* match your regex, but if you view all the meta for each session, you will see at least one meta value that matches for each session.
Okay, I've been testing this against a 10.3 Concentrator using Investigator 9.8 and I can definitely say that the regex is working correctly on the Concentrator.
However, Investigator appears to be modifying the regex when parenthesis are in the expression to the point that what you want to submit is not what is received by the Concentrator. When that happens, the query correctly returns no results. Investigator is placing backslashes around the parens and this causes the regex to become invalid.
However, when I use the REST python tool to submit this query:
/sdk values
id1=32355915 id2=33228508 size=20 flags=2305 threshold=100000 fieldName=alias.host where="time=\"2014-Apr-03 20:46:00\"-\"2014-Apr-04 20:45:59\" && (alias.host regex '([a-zA-Z0-9]{20,})')"
I get correct results back. Every session I've received has at least one alias.host with a string of at least 20 alphanumerics in a row.
Sample result:
id1=853016 id2=853236 count=3 format=65 value=comcastresidentialservices.tt.omtrdc.net type=alias.host flags=0 group=0
Since the parens are not necessary, if you use Investigator, I would just remove them and the regex should work fine. At least that particular regex. I have not tested SA and it may not have this issue.
The python app is here: NetWitness NextGen RESTful Python Test App
You can also just use curl or NwConsole.
Thanks Scott!
My post was partly in response to the person who said that you never need to use regex, and partly my frustration in getting it to work. I found that I needed the parentheses when testing in several web-based regex testers, but I'll delete them from now on. Great catch to find the behavior by the fat client.
I'm a great fan of Python, so I"ll def use the sample REST app in the future.
regards
Irimi,
I've found a workaround for you. If you click the "Custom Drill" button on the toolbar, you can enter your query like this:
filename regex '([0-9a-zA-Z]{20,})'
And it will submit the query correctly without modifying the parens. I've also fixed Investigator 9.8 so that it will not modify the regex. Look for the fix in a future service pack.
Scott
Instead of using an app rule or custom drill to detect hex directories and filenames, try this as a parser:
local hexPath = nw.createParser("Hex_Path", "hex character directory and filename detection")
hexPath:setKeys({
nwlanguagekey.create("risk.suspicious"),
})
function hexPath:checkDir(idx, vlu)
if vlu then
if string.find(vlu, "%X") then
return
end
nw.createMeta(self.keys["risk.suspicious"], "hexadecimal directory name")
end
end
function hexPath:checkFile(idx, vlu)
if vlu then
local dotPosition = string.find(vlu, "%.")
if dotPosition then
repeat
local loopControl = 0
local tmpPosition = string.find(vlu, "%.", dotPosition + 1)
if tmpPosition then
dotPosition = tmpPosition
loopControl = 1
end
until loopControl == 0
vlu = string.sub(vlu, 1, dotPosition - 1)
end
if string.find(vlu, "%X") then
return
end
nw.createMeta(self.keys["risk.suspicious"], "hexadecimal file name")
end
end
hexPath:setCallbacks({
[nwlanguagekey.create("directory")] = hexPath.checkDir,
[nwlanguagekey.create("filename"] = hexPath.checkFile
})
What is the malware that you see that makes these get requests? Do you have a hash? I might have better rules for detecting this that I could share.
on Mar 27, 2014
Irimi,
I've found a workaround for you. If you click the "Custom Drill" button on the toolbar, you can enter your query like this:
filename regex '([0-9a-zA-Z]{20,})'
And it will submit the query correctly without modifying the parens. I've also fixed Investigator 9.8 so that it will not modify the regex. Look for the fix in a future service pack.
Scott