Dave Glover

Exporting and Re-Injecting Logs and Maintain Original Date/Time

Discussion created by Dave Glover Employee on Sep 12, 2018

There are times when you would like to export data from a log decoder and then re inject it into a new log decoder.

 

Typically you would do this through the investigator interface, save the file and then upload this file into the new log decoder.  The problem with this is that the original date and times are lost, as a log decoder stamps the time at the point of injest or import.

 

There was a change made to the log decoder code for other reasons that we leveraged to help our use case.

 

Collection time stamping has always been done at the Decoder, however a change was made to allow the VLC to time stamp it instead..  We are going to use that capability for our use case.

 

The process is fairly simple and is as follows:

 

Enable the VLC time stamping in the Log Decoder:

 

Log Decoder->Explore-> decoder->parsers->config   change lc.lctime.meta to "session"

 

This will tell the decoder to use the VLC timestamp as the authoritative time stamp

 

The next task is to export the data from the log decoder that houses the data.  

The only filtering capabilities that are possible are start and end time.  You can not filter on meta values.

 

For example I would like to export all logs today between 2 and 3 pm No problem,  I would like to export all logs today between 2 and 3 pm where the device type = Palo Alto ... This is NOT POSSIBLE

 

The process is in 2 parts

First we need to extract and transform the syslog header data

 

Second we need to inject the data..  This is done using Syslog-NG, Rsyslog or NXLog or NWLogplayer.

 

 

 

The following shows the process and a sample of the raw log:

 

 

Sep 10 10:14:49 XTM_2_Series (2018-09-13T14:14:49) firewall: msg_id="3000-0148" Allow 1-Trusted 0-External 68 udp 20 63 192.168.30.211 8.8.8.8 44505 53 (dhcp machines-00)

 

After extraction and transformation using the script the log now looks like

 

[][][192.168.30.2][1536762299000000][] Sep 10 10:14:49 XTM_2_Series (2018-09-13T14:14:49) firewall: msg_id="3000-0148" Allow 1-Trusted 0-External 68 udp 20 63 192.168.30.211 8.8.8.8 44505 53 (dhcp machines-00)

 

The log message being sent in CANNOT have a priority field preceding the message, if it does then the message is treated differently and the bold header is ignored.

 

The following log file would be processed as normal, the original date time would be lost and the bold header would become part of the log file and would not parse.

 

<10> [][][192.168.30.2][1536762299000000][] Sep 10 10:14:49 XTM_2_Series (2018-09-13T14:14:49) firewall: msg_id="3000-0148" Allow 1-Trusted 0-External 68 udp 20 63 192.168.30.211 8.8.8.8 44505 53 (dhcp machines-00)

 

 

A breakdown of the header is as follows:

 

First [] and second [] are blanks

Third [] is the ip address or hostname of the original sending system

Fourth [] is the originally collected time stamp in unix time INCLUDING the microseconds 

Fifth [] is blank

 

 

 

 

On any linux box (or the log decoder) you need to copy the script.py file to the /var/log directory.  Technically you could run this script anywhere but for simplicity we shall copy it to /var/log 

 

Then you run the script as follows

 

2. Run script ->

 

python script.py -d 192.168.1.112 -u admin -p netwitness -s 2018-Apr-19 --t1 11:00:00 -e 2018-Apr-30 --t2 11:00:00
-d is the log decoder from which you want to pull the data from
-s is the start date of the logs you want to extract
--t1 is the start time of the logs you want to extract
-e is the end date of the logs you want to extract
--t2 is the end time of the logs you want to extract


 

This will generate two files:

Extract.txt -- Raw log export

Output.txt --Modified header appended to raw

 

Extract.log can be deleted as it will not be used

 

The output.txt is the file that need to be read and syslog streamed to the Log Decoder# that had the lc.lctime.meta value changed to "session"

 

You can use Syslog-NG, Rsyslog, NXLog or NWLogplayer to stream the data in.  However it is MANDATORY that the contents/lines not be modified in anyway.

 

Please note that you can NOT use the Rsyslog server on any of the Netwitness Components, this must be used on a stand alone system.

 

In the Rsyslog conf file below we can see the template being used->

 

 ____________________________

## This section monitors the log file /var/log/output.txt
$InputFileName /var/log/output.txt
$InputFileTag MYAPP
$InputFileStateFile stat-MYAPP
$InputFileSeverity debug
$InputFileFacility local0
$InputFilePersistStateInterval 20000
$InputRunFileMonitor

$template NWReplay,"%rawmsg%\n"

#if $programname == 'MYAPP'
#then @@192.168.31.26:514;NWReplay

@@192.168.31.26:514;NWReplay
& ~

____________________________

 

This conf file directs Rsyslog to read the file and stream the contents to NWLD over a TCP syslog connection

 

If we wanted to do the same using NXlog on Windows (providing we had the file in the 'C:\logs' directory)  we would define the conf file similarly

 

_______________________

<Input in>
Module im_file
File 'C:\logs\output.txt'
ReadFromLast False
SavePos False
PollInterval 1
Exec $Message = $raw_event; $SyslogFacilityValue = 22;
</Input>

 

<Output out1>
Module om_tcp
Host 192.168.26.514
Port 514

 

_______________________

 

NWLogplayer can be used with the switch of  “-r 1”

 

 

 

 

 

Once the data has been read and injected,it will be available in investigator in the original dates and time.

 

Hope that helps

 

Dave

Attachments

Outcomes