Custom Flat File Log Collection with NW-Endpoint 11.4

Blog Post created by Josh Randall

on Apr 22, 2020

Like • Show 4 Likes4
Comment • 7

22APR2020 - UPDATE: Naushad Kasu has posted a video blog of this process and I have posted the template.xml and NweAgentPolicyDetails_x64.exe files from his blog here.

08APR2020 - UPDATE: adding a couple notes and example typespecs after some additional experimenting over the past week

You may find the process easier to simply copy an existing 11.4 typespec in the /var/netwitness/source-server/content/collection/file directory on the Admin Server and modify it for the custom collection source you need
example using IIS typespec:
- comparison of the XML from the Log Collector/Log Decoder to the version I created on the Admin Server
another example using a custom typespec to collect Endpoint v4.4 (A.K.A. legacy ECAT) server logs
- two different typespecs to collect the exact same set of logs, we can see exactly how the values in the typespec affect the raw log that ultimately gets ingested by NetWitness

**END UPDATE**

The NetWitness 11.4 release included a number of features and enhancements for NetWitness Endpoint, one of which was the ability to collect flat file logs (https://community.rsa.com/docs/DOC-110149#Endpoint_Configuration), with the intent that this collection method would allow organizations to replace existing SFTP agents with the Endpoint Agent.

Flat file collection via the 11.4 Endpoint agent allows for a much easier management compared to the SFTP agent, in addition to the multitude of additional investigative and forensic benefits available with both the free version of the Endpoint agent and the advanced version (NetWitness Endpoint User Guide for NetWitness Platform 11.x - Table of Contents).

The 11.4 release included a number of OOTB, supported Flat File collection sources, with support for additional OOTB, as well as custom, sources planned for future releases. However, because I am both impatient and willing to experiment in my lab where there are zero consequences if I break something, I decided to see whether I could port my existing, custom SFTP-based flat file collections to the new 11.4 Endpoint collection.

The process ended up being quite simple and easy. Assuming you already have your Endpoint Server installed and configured, as well as custom flat file typespecs and parsers that you are using, all you need to do is:

install an 11.4+ endpoint agent onto the host(s) that have the flat file logs
...then copy the custom typespec from the Log Decoder/Log Collector filesystem (/etc/netwitness/ng/logcollection/content/collection/file)
...to the Node0/Admin Server filesystem (/var/netwitness/source-server/content/collection/file)
1. ...if your typespec does not already include a <defaults.filePath> element in the XML, go ahead and add one (you can modify the path later in the UI)
2. ...for example:
...after your typespec is copied (and modifed as necessary), restart the source-server on the Node0/Admin Server
...now open the NetWitness UI and navigate to Admin/Endpoint Sources and create a new (or modify an existing) Agent File Logs policy (more details and instructions on that here: Endpoint Config: About Endpoint Sources)
1. ...find your custom Flat File log source in the dropdown and add it to the Endpoint Policy
2. ...modify the Log File Path, if necessary:
3. ...then simply publish your newly modified policy
...and once you have confirmed Collection via the Endpoint Agent, you can stop the SFTP agent on the log source (https://community.rsa.com/docs/DOC-101743#Replace)

And that's it. Happy logging.

Attachments

NweAgentPolicyDetails_x64.exe.zip62.3 KB
template.xml.zip588 bytes

Outcomes

Naushad Kasu

Mar 31, 2020 1:19 PM

Thank you for putting this together Josh!

1 person found this helpful

Actions

Gergely Litavszki Apr 29, 2020 3:39 AM

Hi Josh!

Thanks for the article, this is really helpful!

I put it together in a test enviroment, I get the logs for Microsoft IIS however the parsing is not working properly. I got the latest parser from live.

Example of raw logs in Netwitness:

<13>1 - WIN-9S84OATDFF8 microsoftiis - - [lc@36807 lc.uuid="02C70B8A-DAE5-077C-5899-16CAD2C20667" lc.src="192.168.97.243" lc.ctime="1588069655446"] %IIS-4: 2020-04-28 10:21:51 192.168.97.243 GET /helloguys - 80 - 192.168.97.179 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/81.0.4044.122+Safari/537.36 - 404 0 2 15

Do u have any idea why is that?

Thanks for your help in advance!

Best Regards,

Gergely Litavszki

Actions

Dave Glover

@ Gergely Litavszki on Apr 29, 2020 7:13 AM

In order for IIS to properly work using this method a new IIS message definition needs to be built for your specific logging options. The default IIS parser will not parse the messages properly.

Please reach out to me directly if you would like assistance in building this message definition

Dave

1 person found this helpful

Actions

Gergely Litavszki @ Dave Glover on Apr 30, 2020 3:22 AM

Hi Dave,

Thanks for your help and the quick reply!

Is it possible for Josh to send out his already modified parser? If I see it right, in the article, it is working for him properly.

Thanks again!

Best Regards,

Gergely Litavszki

Actions

Josh Randall

@ Gergely Litavszki on Apr 30, 2020 1:14 PM

Gergely Litavszki

The easiest way to fully parse everything

make sure you have all of the IIS logging fields enabled:

add this text as filename "v20_microsoftiismsg-custom.xml" to the "/etc/netwitness/ng/envision/etc/devices/microsoftiis" directory on your log decoder

<?xml version="1.0" encoding="UTF-8"?>
<DEVICEMESSAGES
      name="microsoftiis"
      displayname="Microsoft IIS"
      group="Web Logs"
      type="55">
<VERSION
      xml="45"
      revision="129"
      checksum="c8e2b0c5fc3bd3dce9e76eacd8378460"
      device="2.0"/>
<MESSAGE
      id1="IIS_LOG_Level:02"
      id2="IIS_LOG_Level"
      insertBefore="IIS_LOG_Level"
      eventcategory="1204010000"
      functions="&lt;@bytes:*CALC(sbytes,+,rbytes)&gt;&lt;@web_domain:*URL($DOMAIN,web_host)&gt;&lt;@tbdstr2:*PARMVAL(web_referer)&gt;&lt;@tbdstr3:*PARMVAL(web_query)&gt;&lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld20,fld21)&gt;"
      content="&lt;fld20&gt;&lt;fld21&gt;&lt;service&gt;&lt;dhost&gt;&lt;daddr&gt;&lt;web_method&gt;&lt;webpage&gt;&lt;web_query&gt;&lt;dport&gt;-&lt;saddr&gt;&lt;protocol&gt;&lt;user_agent&gt;-&lt;web_referer&gt;&lt;web_host&gt;&lt;resultcode&gt;&lt;result&gt;&lt;fld22&gt;&lt;sbytes&gt;&lt;rbytes&gt;&lt;duration&gt;"/>
<MESSAGE
      id1="IIS_LOG_Level:03"
      id2="IIS_LOG_Level"
      insertBefore="IIS_LOG_Level"
      eventcategory="1204010000"
      functions="&lt;@bytes:*CALC(sbytes,+,rbytes)&gt;&lt;@web_domain:*URL($DOMAIN,web_host)&gt;&lt;@tbdstr2:*PARMVAL(web_referer)&gt;&lt;@tbdstr3:*PARMVAL(web_query)&gt;&lt;@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld20,fld21)&gt;"
      content="&lt;fld20&gt;&lt;fld21&gt;&lt;service&gt;&lt;dhost&gt;&lt;daddr&gt;&lt;web_method&gt;&lt;webpage&gt;&lt;web_query&gt;&lt;dport&gt;-&lt;saddr&gt;&lt;protocol&gt;&lt;user_agent&gt;&lt;fld23&gt;&lt;web_referer&gt;&lt;web_host&gt;&lt;resultcode&gt;&lt;result&gt;&lt;fld22&gt;&lt;sbytes&gt;&lt;rbytes&gt;&lt;duration&gt;"/>
</DEVICEMESSAGES>

...and make sure to reload the parsers on your log decoder (or restart the service...but just reloading the parsers is quicker).

The custom XML simply fixes the source and destination IP fields, which Dave Glover found are backwards in the OOTB message id's that handle the log when all the IIS logging fields are selected.

1 person found this helpful

Actions

Gergely Litavszki @ Josh Randall on May 7, 2020 9:55 AM

Thank you both for your help, it works like a charm! I have only one more question if it's not a problem. It is more a cosmetic issue, when we collect logs there are some rows that are commented out on raw log. I would like to drop these messages, is there an easy way for this? Like modifying the file collection XML or something like that.

Example raw log:

#Software: Microsoft Internet Information Services 8.0

Thank you very much again!

BR,

Gergely

Actions

Josh Randall

@ Gergely Litavszki on Dec 28, 2020 3:09 PM

Apologies for the very delayed response...I was just looking back over the comments and saw your question about filtering.

At the time I wrote the blog, there was no filtering capability. That has since changed in the current release (my lab is at 11.5.1...but I'm not sure which release included this feature. I'll have to dig through the release notes to figure that out).

This looks to be a pretty straightforward capability - that main thing to consider when creating/testing your regex is that it will be looking at the original raw log (exactly as shown in your comment) and not what the log looks like after the NWE agent has picked it up.

So for your use case, the regex filter could look like any of these (really just depends on how specific/generic you want to get with the regex):

Actions

7 Comments

Naushad Kasu Mar 31, 2020 1:19 PM
Thank you for putting this together Josh!
1 person found this helpful
Like • Show 1 Like1
Actions
Gergely Litavszki Apr 29, 2020 3:39 AM
Hi Josh!
Thanks for the article, this is really helpful!
I put it together in a test enviroment, I get the logs for Microsoft IIS however the parsing is not working properly. I got the latest parser from live.

Example of raw logs in Netwitness:
<13>1 - WIN-9S84OATDFF8 microsoftiis - - [lc@36807 lc.uuid="02C70B8A-DAE5-077C-5899-16CAD2C20667" lc.src="192.168.97.243" lc.ctime="1588069655446"] %IIS-4: 2020-04-28 10:21:51 192.168.97.243 GET /helloguys - 80 - 192.168.97.179 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/81.0.4044.122+Safari/537.36 - 404 0 2 15

Do u have any idea why is that?

Thanks for your help in advance!
Best Regards,
Gergely Litavszki
Like • Show 0 Likes0
Actions
- Dave Glover @ Gergely Litavszki on Apr 29, 2020 7:13 AM
  In order for IIS to properly work using this method a new IIS message definition needs to be built for your specific logging options. The default IIS parser will not parse the messages properly.
  
  Please reach out to me directly if you would like assistance in building this message definition
  
  Dave
  1 person found this helpful
  Like • Show 1 Like1
  Actions
  - Gergely Litavszki @ Dave Glover on Apr 30, 2020 3:22 AM
    Hi Dave,
    
    Thanks for your help and the quick reply!
    
    Is it possible for Josh to send out his already modified parser? If I see it right, in the article, it is working for him properly.
    
    Thanks again!
    Best Regards,
    Gergely Litavszki
    Like • Show 0 Likes0
    Actions
    - Josh Randall @ Gergely Litavszki on Apr 30, 2020 1:14 PM
      Gergely Litavszki
      The easiest way to fully parse everything
      make sure you have all of the IIS logging fields enabled:
      add this text as filename "v20_microsoftiismsg-custom.xml" to the "/etc/netwitness/ng/envision/etc/devices/microsoftiis" directory on your log decoder
      <?xml version="1.0" encoding="UTF-8"?> <DEVICEMESSAGES name="microsoftiis" displayname="Microsoft IIS" group="Web Logs" type="55"> <VERSION xml="45" revision="129" checksum="c8e2b0c5fc3bd3dce9e76eacd8378460" device="2.0"/> <MESSAGE id1="IIS_LOG_Level:02" id2="IIS_LOG_Level" insertBefore="IIS_LOG_Level" eventcategory="1204010000" functions="<@bytes:*CALC(sbytes,+,rbytes)><@web_domain:*URL($DOMAIN,web_host)><@tbdstr2:*PARMVAL(web_referer)><@tbdstr3:*PARMVAL(web_query)><@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld20,fld21)>" content="<fld20><fld21><service><dhost><daddr><web_method><webpage><web_query><dport>-<saddr><protocol><user_agent>-<web_referer><web_host><resultcode><result><fld22><sbytes><rbytes><duration>"/> <MESSAGE id1="IIS_LOG_Level:03" id2="IIS_LOG_Level" insertBefore="IIS_LOG_Level" eventcategory="1204010000" functions="<@bytes:*CALC(sbytes,+,rbytes)><@web_domain:*URL($DOMAIN,web_host)><@tbdstr2:*PARMVAL(web_referer)><@tbdstr3:*PARMVAL(web_query)><@event_time:*EVNTTIME($MSG,'%W-%G-%F %N:%U:%O',fld20,fld21)>" content="<fld20><fld21><service><dhost><daddr><web_method><webpage><web_query><dport>-<saddr><protocol><user_agent><fld23><web_referer><web_host><resultcode><result><fld22><sbytes><rbytes><duration>"/> </DEVICEMESSAGES>
      
      ...and make sure to reload the parsers on your log decoder (or restart the service...but just reloading the parsers is quicker).
      
      The custom XML simply fixes the source and destination IP fields, which Dave Glover found are backwards in the OOTB message id's that handle the log when all the IIS logging fields are selected.
      
      1 person found this helpful
      Like • Show 1 Like1
      Actions
      - Gergely Litavszki @ Josh Randall on May 7, 2020 9:55 AM
        Thank you both for your help, it works like a charm! I have only one more question if it's not a problem. It is more a cosmetic issue, when we collect logs there are some rows that are commented out on raw log. I would like to drop these messages, is there an easy way for this? Like modifying the file collection XML or something like that.
        
        Example raw log:
        #Software: Microsoft Internet Information Services 8.0
        
        Thank you very much again!
        BR,
        Gergely
        Like • Show 0 Likes0
        Actions
        Josh Randall @ Gergely Litavszki on Dec 28, 2020 3:09 PM
        Apologies for the very delayed response...I was just looking back over the comments and saw your question about filtering.
        At the time I wrote the blog, there was no filtering capability. That has since changed in the current release (my lab is at 11.5.1...but I'm not sure which release included this feature. I'll have to dig through the release notes to figure that out).
        
        This looks to be a pretty straightforward capability - that main thing to consider when creating/testing your regex is that it will be looking at the original raw log (exactly as shown in your comment) and not what the log looks like after the NWE agent has picked it up.
        
        So for your use case, the regex filter could look like any of these (really just depends on how specific/generic you want to get with the regex):
        
        Like • Show 0 Likes0
        Actions

Custom Flat File Log Collection with NW-Endpoint 11.4

Attachments

Outcomes

Related Content

Recommended Content