on Apr 22, 202022APR2020 - UPDATE: Naushad Kasu has posted a video blog of this process and I have posted the template.xml and NweAgentPolicyDetails_x64.exe files from his blog here.
08APR2020 - UPDATE: adding a couple notes and example typespecs after some additional experimenting over the past week
- You may find the process easier to simply copy an existing 11.4 typespec in the /var/netwitness/source-server/content/collection/file directory on the Admin Server and modify it for the custom collection source you need
- example using IIS typespec:
- comparison of the XML from the Log Collector/Log Decoder to the version I created on the Admin Server
- comparison of the XML from the Log Collector/Log Decoder to the version I created on the Admin Server
- another example using a custom typespec to collect Endpoint v4.4 (A.K.A. legacy ECAT) server logs
- two different typespecs to collect the exact same set of logs, we can see exactly how the values in the typespec affect the raw log that ultimately gets ingested by NetWitness
- two different typespecs to collect the exact same set of logs, we can see exactly how the values in the typespec affect the raw log that ultimately gets ingested by NetWitness
**END UPDATE**
The NetWitness 11.4 release included a number of features and enhancements for NetWitness Endpoint, one of which was the ability to collect flat file logs (https://community.rsa.com/docs/DOC-110149#Endpoint_Configuration), with the intent that this collection method would allow organizations to replace existing SFTP agents with the Endpoint Agent.
Flat file collection via the 11.4 Endpoint agent allows for a much easier management compared to the SFTP agent, in addition to the multitude of additional investigative and forensic benefits available with both the free version of the Endpoint agent and the advanced version (NetWitness Endpoint User Guide for NetWitness Platform 11.x - Table of Contents).
The 11.4 release included a number of OOTB, supported Flat File collection sources, with support for additional OOTB, as well as custom, sources planned for future releases. However, because I am both impatient and willing to experiment in my lab where there are zero consequences if I break something, I decided to see whether I could port my existing, custom SFTP-based flat file collections to the new 11.4 Endpoint collection.
The process ended up being quite simple and easy. Assuming you already have your Endpoint Server installed and configured, as well as custom flat file typespecs and parsers that you are using, all you need to do is:
- install an 11.4+ endpoint agent onto the host(s) that have the flat file logs
- ...then copy the custom typespec from the Log Decoder/Log Collector filesystem (/etc/netwitness/ng/logcollection/content/collection/file)
- ...to the Node0/Admin Server filesystem (/var/netwitness/source-server/content/collection/file)
- ...if your typespec does not already include a <defaults.filePath> element in the XML, go ahead and add one (you can modify the path later in the UI)
- ...for example:
- ...after your typespec is copied (and modifed as necessary), restart the source-server on the Node0/Admin Server
- ...now open the NetWitness UI and navigate to Admin/Endpoint Sources and create a new (or modify an existing) Agent File Logs policy (more details and instructions on that here: Endpoint Config: About Endpoint Sources)
- ...find your custom Flat File log source in the dropdown and add it to the Endpoint Policy
- ...modify the Log File Path, if necessary:
- ...then simply publish your newly modified policy
- ...find your custom Flat File log source in the dropdown and add it to the Endpoint Policy
- ...and once you have confirmed Collection via the Endpoint Agent, you can stop the SFTP agent on the log source (https://community.rsa.com/docs/DOC-101743#Replace)
And that's it. Happy logging.
Thank you for putting this together Josh!