Hi everyone,
I was curious to know if anyone has found a way to search for text inside files. I didn't think the SEARCH parser actually checked inside attachments, although maybe I'm wrong.
The best way I could think of was to use Visualize to process a collection then try to use the "post process" to search for strings, but on all my systems, post processing doesn't work.
Thanks!
John
Yes, you can use the /sdk msearch command to search sessions (only in v10.4) for a particular string or regex. If you pass the flag "ds" to the flags parameter, it will decode any sessions for protocols (SMB, HTTP, SMTP, POP, etc) it understands and then search the files that were transmitted thru the protocol.
Example
/sdk msearch where="service=80,139,25,110" limit=10000 flags=sp,ds,pre,post search="My search term"
The flags are:
sp - search packets
ds - decode sessions
pre - show text before hit
post - show text after hit
You can also use NwConsole's sdk content command to extract files and do the search yourself.
Awesome, thanks Scott! This is too cool
Thanks again
John
So I think I am experiencing some "user error" here
So I tried the following:
xxx.xxx.xxx.xxx:50105/sdk?msg=msearch&force-content-type=text/plain&expiry=600&s%2Fsdk=&msearch=&where=service%3D80%2C139%2C25%2C110&limit=10000&flags=sp%2Cds%2Cpre%2Cpost&search=My%20search%20term
....just to mess around with it. It returns id1 and id2, but nothing else. Also, whenever I change the search term it still gives the same id1 and id2.
Thoughts?
Thanks!
John
If it returns nothing but id1 and id2, then nothing matched your search term. Also, the limit parameter was just an example, you should probably search more than just 10k sessions or adjust your where clause to include a time period and then avoid the limit parameter. Either way, it's probably not a good idea to search weeks of data on a decoder.
Yep, that was it. I re-formatted it with a timeframe and it worked great. Time to stress-test this I guess
Thanks again!
John
Hi,
This may seem like an obvious point to make, but this search will only work if the files to be searched are in a format that is easily readable. For example, office documents are really gzipped XML, so won't allow you to search inside them in this way. The search won't unzip the files to make them readable. Thanks
It doesn't do that today, but I'll add that to the road map. Should be very easy to do.
Hi,
Just a follow up question. I've been loving this feature, but it was brought to my attention that a similar function exists within the NwInvestigator client. Just curious as I'm not as familiar with the older gear. Is the Investigator client doing a REST "search" call, while this is using the "msearch?"
Thanks!
Investigator 9.8 uses the /sdk search API (it predates REST). "search" and "msearch" offer roughly the same capabilities, but msearch is faster because it can search multiple sessions based on a where clause and stream the results back.
"search" can only search one session at a time, so there's a fair amount of latency between the client and server, as the client must make a request for each session to search. But ultimately, the results will be the same.
Awesome. I thought it might be something along these lines.
Thanks!
on Jun 3, 2015
Yes, you can use the /sdk msearch command to search sessions (only in v10.4) for a particular string or regex. If you pass the flag "ds" to the flags parameter, it will decode any sessions for protocols (SMB, HTTP, SMTP, POP, etc) it understands and then search the files that were transmitted thru the protocol.
Example
/sdk msearch where="service=80,139,25,110" limit=10000 flags=sp,ds,pre,post search="My search term"
The flags are:
sp - search packets
ds - decode sessions
pre - show text before hit
post - show text after hit
You can also use NwConsole's sdk content command to extract files and do the search yourself.