Chris Thomas

Investigating Endpoints with NWE Tracking Data

Blog Post created by Chris Thomas Employee on Mar 10, 2017

There has been a lot of great information published about Sysmon since Mark Russinovich's presentation at RSA Conference. Eric Partington posted a great blog showing how to use Sysmon data with RSA NetWitness for Logs: Log - Sysmon 6 Windows Event Collection. This prompted RSA’s IR Team to publish details on how to get the rich tracking information generated by RSA NetWitness Endpoint that they use everyday for their incident investigations into a SIEM Here.

 

The aim of this blog is to show you how to collect this tracking data from RSA NetWitness Endpoint with RSA NetWitness for Logs. The collection is done via the Log Collector using a custom ODBC typespec.

 

*** DISCLAIMER - this is a field developed Proof of Concept, shared with the Community. It is not endorsed by RSA Engineering. The database structure used by NWE may change at any time. No testing has been done to measure the impact on performance for a production NWE Server. This has been developed and tested using RSA NetWitness Endpoint v4.3.0.1 and RSA NetWitness for Logs v10.6.2.1. /DISCLAIMER ***

 

***DISCLAIMER 2 - for this Proof of Concept, we have disabled the requirement on the NWE SQL Server to Force Encryption.  /DISCLAIMER 2 ***

 

The objective of this integration is to get the tracking data from NWE as it is being collected into NWL, so we can index it and use it for Investigations. Tracking data in NWE can only be viewed on a per machine basis - this integration allows us to get a global view of tracking data across all of our endpoints. Here's the high level summary of what we need to do (if you want to skip to the end, all files are attached as a zip):

  1. Create a new ODBC typespec definition (XML file) to query the NWE Database and get the data we want,
  2. Create a new Log Parser to map the results of the SQL query into metadata,
  3. Add the meta we are using to the table-map-custom.xml so it is persistent,
  4. Add the meta we want to index to the index-concentrator-custom.xml file,
  5. Configure a new ODBC DSN definition,
  6. Configure a new ODBC Event Collector,
  7. Configure a new Meta Group to show our data for investigations,
  8. Configure a new Column Group to show the data we want in Events view,
  9. Configure some Report Rules and Charts to visualise the data,
  10. Configure a new RSA NetWitness Endpoint Dashboard to keep track of our environment.

Here we go!

 

1. Create ODBC Definition

Thanks to Andreas Funk and his blog Integrating a MySQL (community) database with NetWitness for Logs for giving us a primer on how to create a new ODBC connection. We need to create a new Filespec to tell the ODBC collector how to query the NWE database and get the data we want. 

On the Log Collector (either the one on the Log Decoder, or a separate VLC - whichever you are going to use to collect these logs) the ODBC collection definitions are stored here: 

/etc/netwitness/ng/logcollection/content/collection/odbc/

 

We need to add a new file for our NWE tracking data - 

vi /etc/netwitness/ng/logcollection/content/collection/odbc/nwe_tracking.xml

 

Here is the query from Rui Ataide's blog, modified to work for NWL, included in our definition:

<?xml version="1.0" encoding="UTF-8"?>
<typespec>
   <name>nwe_tracking</name>
   <type>odbc</type>
   <prettyName>NetWitness Endpoint Tracking</prettyName>
   <version>2.0</version>
   <author>Chris Thomas</author>
   <description>Import NWE Tracking data</description>
   <device>
      <name>nwe_tracking</name>
   </device>
   <configuration>
   </configuration>
   <collection>
      <odbc>
         <query>
            <tag>nwe_tracking</tag>
            <outputDelimiter>||</outputDelimiter>
            <interval>30</interval>
            <dataQuery>              
           
(SELECT
      SE.PK_WinTrackingEvents,
      SE.EventUTCTIme,
      MA.MacAddress as src_mac,
      MA.LocalIp as src_ip,
      MA.MachineName,
      LOWER(PA.Path),
      LOWER(FN.FileName),
      LOWER(PA.Path + FN.FileName) AS Source,
      MO.HashSHA256,
      LA.LaunchArguments AS SLA,
      CASE      
            WHEN SE.BehaviorFileOpenPhysicalDrive = 1 THEN 'OpenPhysicalDrive'
            WHEN SE.BehaviorFileReadDocument = 1 THEN 'ReadDocument'
            WHEN SE.BehaviorFileWriteExecutable = 1 THEN 'WriteExecutable'
            WHEN SE.BehaviorFileRenameToExecutable = 1 THEN 'RenameExecutable'
            WHEN SE.BehaviorProcessCreateProcess = 1 THEN 'CreateProcess'
            WHEN SE.BehaviorProcessCreateRemoteThread = 1 THEN 'CreateRemoteThread'
            WHEN SE.BehaviorProcessOpenOSProcess = 1 THEN 'OpenOSProcess'
            WHEN SE.BehaviorProcessOpenProcess = 1 THEN 'OpenProcess'
            WHEN SE.BehaviorFileSelfDeleteExecutable = 1 THEN 'SelfDelete'
            WHEN SE.BehaviorFileDeleteExecutable = 1 THEN 'DeleteExecutable'
            WHEN SE.BehaviorRegistryModifyBadCertificateWarningSetting = 1 THEN 'ModifyBadCertificateWarningSetting'
            WHEN SE.BehaviorRegistryModifyFirewallPolicy = 1 THEN 'ModifyFirewallPolicy'
            WHEN SE.BehaviorRegistryModifyInternetZoneSettings = 1 THEN 'ModifyInternetZoneSettings'
            WHEN SE.BehaviorRegistryModifyIntranetZoneBrowsingNotificationSetting = 1 THEN 'ModifyIntranetZoneBrowsingNotificationSetting'
            WHEN SE.BehaviorRegistryModifyLUASetting = 1 THEN 'ModifyLUASetting'
            WHEN SE.BehaviorRegistryModifyRegistryEditorSetting = 1 THEN 'ModifyRegistryEditorSetting'
            WHEN SE.BehaviorRegistryModifyRunKey = 1 THEN 'ModifyRunKey '
            WHEN SE.BehaviorRegistryModifySecurityCenterConfiguration = 1 THEN 'ModifySecurityCenterConfiguration'
            WHEN SE.BehaviorRegistryModifyServicesImagePath = 1 THEN 'ModifyServicesImagePath'
            WHEN SE.BehaviorRegistryModifyTaskManagerSetting = 1 THEN 'ModifyTaskManagerSetting'
            WHEN SE.BehaviorRegistryModifyWindowsSystemPolicy = 1 THEN 'ModifyWindowsSystemPolicy'
            WHEN SE.BehaviorRegistryModifyZoneCrossingWarningSetting = 1 THEN 'ModifyZoneCrossingWarningSetting'
      END AS Action,
      LOWER(SE.Path_Target),
      LOWER(SE.FileName_Target),
      LOWER(SE.Path_Target + SE.FileName_Target) AS Destination,
      SE.LaunchArguments_Target AS TLA,
      se.HashSHA256_Target
FROM
      dbo.WinTrackingEvents_P1 AS SE WITH(NOLOCK)
      INNER JOIN dbo.Machines AS MA WITH(NOLOCK) ON MA.PK_Machines = SE.FK_Machines
      INNER JOIN dbo.MachineModulePaths AS MP WITH(NOLOCK) ON MP.PK_MachineModulePaths = SE.FK_MachineModulePaths
      INNER JOIN dbo.Modules AS MO WITH(NOLOCK) ON MO.PK_Modules = MP.FK_Modules
      INNER JOIN dbo.FileNames AS FN WITH(NOLOCK) ON FN.PK_FileNames = MP.FK_FileNames
      INNER JOIN dbo.Paths AS PA WITH(NOLOCK) ON PA.PK_Paths = MP.FK_Paths
      INNER JOIN dbo.LaunchArguments AS LA WITH(NOLOCK) ON LA.PK_LaunchArguments = SE.FK_LaunchArguments__SourceCommandLine
WHERE PK_WinTrackingEvents > '%TRACKING%'
UNION
SELECT
      SE.PK_WinTrackingEvents,
      SE.EventUTCTIme,
      MA.MacAddress as src_mac,
      MA.LocalIp as src_ip,
      MA.MachineName,
      LOWER(PA.Path),
      LOWER(FN.FileName),
      LOWER(PA.Path + FN.FileName) AS Source,
      MO.HashSHA256,
      LA.LaunchArguments AS SLA,
      CASE      
            WHEN SE.BehaviorFileOpenPhysicalDrive = 1 THEN 'OpenPhysicalDrive'
            WHEN SE.BehaviorFileReadDocument = 1 THEN 'ReadDocument'
            WHEN SE.BehaviorFileWriteExecutable = 1 THEN 'WriteExecutable'
            WHEN SE.BehaviorFileRenameToExecutable = 1 THEN 'RenameExecutable'
            WHEN SE.BehaviorProcessCreateProcess = 1 THEN 'CreateProcess'
            WHEN SE.BehaviorProcessCreateRemoteThread = 1 THEN 'CreateRemoteThread'
            WHEN SE.BehaviorProcessOpenOSProcess = 1 THEN 'OpenOSProcess'
            WHEN SE.BehaviorProcessOpenProcess = 1 THEN 'OpenProcess'
            WHEN SE.BehaviorFileSelfDeleteExecutable = 1 THEN 'SelfDelete'
            WHEN SE.BehaviorFileDeleteExecutable = 1 THEN 'DeleteExecutable'
            WHEN SE.BehaviorRegistryModifyBadCertificateWarningSetting = 1 THEN 'ModifyBadCertificateWarningSetting'
            WHEN SE.BehaviorRegistryModifyFirewallPolicy = 1 THEN 'ModifyFirewallPolicy'
            WHEN SE.BehaviorRegistryModifyInternetZoneSettings = 1 THEN 'ModifyInternetZoneSettings'
            WHEN SE.BehaviorRegistryModifyIntranetZoneBrowsingNotificationSetting = 1 THEN 'ModifyIntranetZoneBrowsingNotificationSetting'
            WHEN SE.BehaviorRegistryModifyLUASetting = 1 THEN 'ModifyLUASetting'
            WHEN SE.BehaviorRegistryModifyRegistryEditorSetting = 1 THEN 'ModifyRegistryEditorSetting'
            WHEN SE.BehaviorRegistryModifyRunKey = 1 THEN 'ModifyRunKey '
            WHEN SE.BehaviorRegistryModifySecurityCenterConfiguration = 1 THEN 'ModifySecurityCenterConfiguration'
            WHEN SE.BehaviorRegistryModifyServicesImagePath = 1 THEN 'ModifyServicesImagePath'
            WHEN SE.BehaviorRegistryModifyTaskManagerSetting = 1 THEN 'ModifyTaskManagerSetting'
            WHEN SE.BehaviorRegistryModifyWindowsSystemPolicy = 1 THEN 'ModifyWindowsSystemPolicy'
            WHEN SE.BehaviorRegistryModifyZoneCrossingWarningSetting = 1 THEN 'ModifyZoneCrossingWarningSetting'
      END AS Action,
      LOWER(SE.Path_Target),
      LOWER(SE.FileName_Target),
      LOWER(SE.Path_Target + SE.FileName_Target) AS Destination,
      SE.LaunchArguments_Target AS TLA,
      se.HashSHA256_Target
FROM
      dbo.WinTrackingEvents_P0 AS SE WITH(NOLOCK)
      INNER JOIN dbo.Machines AS MA WITH(NOLOCK) ON MA.PK_Machines = SE.FK_Machines
      INNER JOIN dbo.MachineModulePaths AS MP WITH(NOLOCK) ON MP.PK_MachineModulePaths = SE.FK_MachineModulePaths
      INNER JOIN dbo.Modules AS MO WITH(NOLOCK) ON MO.PK_Modules = MP.FK_Modules
      INNER JOIN dbo.FileNames AS FN WITH(NOLOCK) ON FN.PK_FileNames = MP.FK_FileNames
      INNER JOIN dbo.Paths AS PA WITH(NOLOCK) ON PA.PK_Paths = MP.FK_Paths
      INNER JOIN dbo.LaunchArguments AS LA WITH(NOLOCK) ON LA.PK_LaunchArguments = SE.FK_LaunchArguments__SourceCommandLine
WHERE PK_WinTrackingEvents > '%TRACKING%' )

ORDER By SE.PK_WinTrackingEvents ASC
            </dataQuery>

            <trackingColumn>PK_WinTrackingEvents</trackingColumn>
     <maxTrackingQuery> SELECT MAX(PK_WinTrackingEvents) FROM dbo.WinTrackingEvents_P0</maxTrackingQuery>
         </query>
      </odbc>
   </collection>
</typespec>

This creates a log entry with a static format, that is delimited by a double pipe ||:

This makes it easy for us to create a new log parser.

 

2. Create a new Log Parser

For information on how to create a new log parser using the new Log Parser Tool, head over here: The specified item was not found.We need to create a new directory where the Log Decoder parsers are kept, and add our ini and xml parser files

mkdir /etc/netwitness/ng/envision/etc/devices/nwe_tracking/

 

Here is the ini file that describes our parser: nwe_tracking.ini

DatabaseName=nwe_tracking

DisplayName=NetWitness Endpoint Tracking

DeviceGroup=

DeviceType=7104

 

And here is the Log Parser: v20_nwe_trackingmsg.xml - the meta keys to use were chosen to line up with where the data from sysmon gets mapped to, as shown here: Log - Sysmon 6 Windows Event Collection

<?xml version="1.0" encoding="UTF-8"?>
<DEVICEMESSAGES
        name="nwe_tracking"
        displayname="NetWitness Endpoint Tracking"
        group=""
        type="7104">

<VERSION
      xml="1"
      revision="1"
        device="2.0"/>

<HEADER
        id1="HDR1"
        id2="HDR1"
        messageid="STRCAT('NWEPMSG')"
        content="%nwe_tracking:&lt;trans_id&gt;||&lt;event_time&gt;||&lt;!payload:trans_id&gt;"/>

<MESSAGE
        id1="NWEPMSG"
        id2="NWEPMSG"
        eventcategory="1612000000"      content="&lt;trans_id&gt;||&lt;event_time&gt;||&lt;smacaddr&gt;||&lt;saddr&gt;||&lt;event_computer&gt;||&lt;directory&gt;||&lt;filename&gt;||&lt;parent_process&gt;||&lt;checksum&gt;||&lt;parent_params&gt;||&lt;category&gt;||&lt;directory&gt;||&lt;filename&gt;||&lt;process&gt;||&lt;params&gt;||&lt;checksum&gt;"/>

</DEVICEMESSAGES>

There should be 2 files in the new directory:

[root@RSAANZSCSA nwe_tracking]# pwd

/etc/netwitness/ng/envision/etc/devices/nwe_tracking

[root@RSAANZSCSA nwe_tracking]# ls -l

total 8

-rw-r--r--. 1 root root  96 Mar  9 10:01 nwe_tracking.ini

-rw-r--r--. 1 root root 761 Mar 10 02:59 v20_nwe_trackingmsg.xml

[root@RSAANZSCSA nwe_tracking]#

 

3. Add meta to table-map-custom.xml

This step can be done using the Web GUI, but since we're already on the command line we'll do it there. It's always a good idea to make a back up copy of the file first!

cp /etc/netwitness/ng/envision/etc/table-map-custom.xml /etc/netwitness/ng/envision/etc/table-map-custom.xml.old

 

Then edit the table-map-custom.xml file:

vi /etc/netwitness/ng/envision/etc/table-map-custom.xml

 

We can add the meta we are using (that is not already set as persistent (flags="None") at the end of the file:

        <!-- NWE Tracking Data
-->

<mapping envisionName="smacaddr" nwName="eth.src" flags="None" format="MAC" envisionDisplayName="SourceMacAddress" nullTokens="Unknown"/>
<mapping envisionName="checksum" nwName="checksum" flags="None"/>
<mapping envisionName="parent_params" nwName="parent.params" flags="None"/>
<mapping envisionName="process" nwName="process" flags="None"/>
<mapping envisionName="parent_process" nwName="parent.process" flags="None"/>
<mapping envisionName="params" nwName="params" flags="None"/>
<mapping envisionName="directory" nwName="directory" flags="None"/>
<mapping envisionName="category" nwName="category" flags="None"/>

Now that we've finished the modifications for the Log Collector and Log Decoder, restart those services so that the changes get loaded.

 

4. Add meta for indexing to index-concentrator-custom.xml

Again, you can do this in the GUI, but since we're on the command line already we'll do it there. Just make sure you switch to your Concentrator first! (I'm on a hybrid ). Again - make a backup first

cp /etc/netwitness/ng/index-concentrator-custom.xml /etc/netwitness/ng/index-concentrator-custom.xml.old

 

Then edit the file

vi /etc/netwitness/ng/index-concentrator-custom.xml

 

Add the new meta to index at the end of the file - you may need to add more keys depending on your existing index settings:

<!-- NWE Tracking Data -->
<key description="Checksum" format="Text" level="IndexValues" name="checksum" valueMax="1000000" defaultAction="Open"/>
<key description="Parent Process" format="Text" level="IndexValues" name="parent.process" valueMax="1000000" defaultAction="Open"/>
<key description="Parent Process Parameters" format="Text" level="IndexValues" name="parent.params" valueMax="1000000" defaultAction="Open"/>
<key description="Process Parameters" format="Text" level="IndexValues" name="params" valueMax="1000000" defaultAction="Open"/>
<key description="Category" format="Text" level="IndexValues" name="category" valueMax="1000000" defaultAction="Open"/>

Restart the Concentrator service so that the changes get loaded.

 

5. Configure new ODBC DSN definition.

Now we can switch to the GUI for our configuration. Go to your Log Collector Config page, and create a new DSN. 

 

Enter the details to connect to your NWE Database (do not use a template) and click Save:

 

6. Configure a new ODBC Event Collector

On the Log Collector Config page, create a new ODBC Event Category by selecting our new nwe_tracking source from the list:

 

Now add a new Event Source and enter the details for your NWE SQL Database:

 

Click Test Connection to see that it all works ...

 

If it's not turned on already, start ODBC collection method (and set it to auto-start). Now you should be collecting NWE Tracking events! run a query for device.type = 'nwe_tracking' to see:

 

 

The remaining steps go through ways to use the NWE Tracking data.

 

7. Configure a Meta Group to show the NWE Meta in Investigations

If you have a favourite Meta Group you use, just add these Meta keys to it. Otherwise, create a new Meta Group called NW Endpoint Tracking. Here's what I have in mine:

 

Note - I have all my Meta Keys set to open for testing purposes. Best Practice is to set did to open, and all other keys to closed. This gives better performance with large datasets by not sending 22 queries to the Concentrator at the same time.

Here's what you should be able to see:

 

By mapping the process name into the filename meta key, the data will trigger any feeds that are looking for matches on filename. The Investigation and Hunting feeds match this data:

8. Configure a new Column Group

The default view for reviewing logs in the Event Viewer is very simple:

 

We can change this view to show the meta extracted from our NWE Tracking logs. Create a new Column Group

 

Note that you can change the "Display Name" to something you like - this will be used for the column heading:

 

9. Configure Report Engine Rules and Charts

Use the new device.type = 'nwe_tracking' to create rules to use for reports and charts. Here's a rule to query on the Source Process (parent.process):

 

results:

 

We can then use the rule as a basis for a chart:

 

 

10. Create a RSA NetWitness Endpoint Dashboard

One you create the charts that you want, you can create a new Dashboard to keep track of your environment. Simply create a new Dashboard, and add your charts as Dashlets using "Reports Realtime Chart"

 

All the files mentioned in this post are available for download in the zip below.

 

Happy Hunting!

 

Thanks to Rui Ataide & Eric Partington for their contributions to this integration.

Attachments

Outcomes