RSA Admin

Extracting Files using NwConsole

Discussion created by RSA Admin Employee on Sep 21, 2012
Latest reply on Nov 18, 2014 by akK9JNBsR9GgjAuGPlN6hHzzwqL4VSJksLHcVvRlXPk=

A little known feature in NextGen, that has been around since NextGen v9.0, is a console command that will run unattended and extract files from packet sessions based on criteria you enter.

 

The enhancements to NwConsole are in the sdk content calls. So run NwConsole, and login to a collection:


sdk open nw://username:password@concentrator:50005


sdk info - will give you some stats about the system including the session IDs


Set your output directory:


sdk output somedirectory


Then issue your content call. Type "sdk content" to get the options:


sdk content [options]

options:

session=[#]|[#-#] The session or session range (optional) to extract content from

where={where clause} The where clause (optional), used to determine sessions to extract

render=# The render type (defined in NwSDK.h) for content or render name

flags=# The content flags, zero is default

size=# The maximum session size to retrieve, unlimited is default

dir={pathname} Directory path where content files will be placed

maxDirSize=# Max directory size in MBs, default is unlimited

includeFileTypes=.ext1;.ext2 Semicolon separated list of file extensions that will be extracted

excludeFileTypes=.ext1;.ext2 Semicolon separated list of file extensions that will be excluded

renameFileTypes=.ext1|.ext2 Semicolon and pipe separated list of file extensions that will be renamed


render can be a number (defined in NwSDK.h) or one of the following render types:

text, hex, packets, web, mail, raw, rtp, voip, meta, im or files.


renameFileTypes is used to rename certain files from one or more extensions to another. For example:

renameFileTypes=.download|.octet-stream|.program|.exe;.jpeg|.jpg

For the above example, all files ending in .download, .stream or .program will be renamed to .exe

All files ending in .jpeg will be renamed to .jpg


WARNING: Setting maxDirSize will scan the output directory every 20 minutes and will

indiscriminately delete the oldest files that exceed the threshold. Please do not use a directory

with existing files that should not be deleted!


To run continuously, you must provide a where clause and do not provide a session range.

Or you can provide a lower bound session id but leave the upper session id unbounded like:

sessions=1000-u Start at session 1000 and continue nonstop

sessions=now-u Means extract only new sessions as they come in

sessions=2000-3000 Extract sessions between 2000 and 3000 then quit



So a command such as this...


sdk content session=now-u where="extension=exe,dll" render=files includeFileTypes=.exe;.dll; maxDirSize=1000


...will extract all exe and dll files from any session where there is a registered extension of exe or dll. Obviously - we are looking for executables that are not always properly named. So you could use a combination of the filetypes.parser file, located in the content pack. Then issue the following:


sdk content session=now-u where="alert=file_signature_windows_executable" render=files includeFileTypes=.exe;.dll; maxDirSize=1000



Some other examples:


PDFs

sdk content session=now-u where="extension=pdf" render=files includeFileTypes=.pdf maxDirSize=1000


MP3s

sdk content session=now-u where="extension=mp3" render=files includeFileTypes=.mp3; maxDirSize=1000 renameFileTypes=.octet-stream|.mp3


Images

sdk content session=now-u where="extension=jpg,jpeg,png,bmp,gif" render=files includeFileTypes=.jpg;.jpeg;.png;.bmp;.gif maxDirSize=1000


Documents

sdk content session=now-u where="extension exists" render=files includeFileTypes=.doc;.docx;.xls;.xlsx;.ppt;.pptx; .pdf;.zip maxDirSize=1000


You can throw all the commands in a text file, and just launch NwConsole with the -f file option to automate.


Now - for the obligatory warnings:


First - this uses the EXISTING content reconstruction. If the file fails to render in Investigator, it will probably not be extracted correctly. What is happening, is that the SDK uses the index to find matching sessions, and then asks the decoder to reconstruct that session, along with ALL files. It then sends that reconstructed session back to the client, where only the requested file types are extracted and saved. This has the unintended effect of caching that session on the decoder. The same process happens behind the scenes in investigator. This just does it a lot quicker, and at higher volumes.


 

If you have any questions, please respond to this post.

 

Enjoy!

Scott

Outcomes