Prerequisite: Export Parsers
Before using the IPDB Export tool the environment should be at 11.x. If still on 10.6.x and upgrading to 11.x, copy any log parsers currently in your environment, particularly any custom-built parsers.
Perform the following steps:
- On your 10.6.x Log Decoder, copy the existing parsers from /etc/netwitness/ng/envision/etc/devices into a backup location.
Note: You can choose to copy only your custom parsers or as many of the parsers as wanted. RSA recommends copying the parsers somewhere external to the NetWitness Platform devices. This is to ensure that no damage or loss occurs to the parsers during an upgrade.
- Upgrade your NetWitness environment from 10.6.x to an appropriate version of 11.x.
- Copy the parsers from the backup location to /etc/netwitness/ng/envision/etc/devices on the recently upgraded Log Decoder.
- Restart the Log Decoder service and verify that the parsers copied are enabled.
Continue with the remaining procedures in this document to extract the data from your IPDB database and inject it into your NetWitness Platform.
Since the IPDB Export utility must be ran on a Windows system, to use the IPDB Export utility one of the following is needed (physical or virtual):
- Windows 2008 R2 SP1 64-bit Server
- Windows 2012 Server
- Windows 2016 Server
- Windows 2019 Server
A minimum of 20% free disk space is required if using an existing data partition. For example, at least 20 GB of free space is needed if the system drive is 100 GB in size. It is recommended that a completely separate and empty partition be used to capture the exported data. The actual size of the space that is required will depend on the amount of data being extracted from the IPDB enVision database. If unsure of the space requirements, perform a small data export to help calculate the actual space requirements needed.
IPDB Export Tool Details
The IPDBExportUtility.exe file is available for download from the IPDB Export Utility page on RSA Link. You can run the tool using either a command-line interface or a graphical UI.
Note: RSA recommends using a read-only IPDB database with this tool.
- To run the tool using the GUI, double-click the executable file.
- To use the command-line interface, run the tool from the command prompt. You can use the -h flag to list the available options.
- Expected Rate for export is approximately 30 GB per hour but can vary depending on your environment (number of CPUs, network latency, so forth)
- This tool should not be ran on an RSA NetWitness Platform node. A virtual or physical Windows server with access to the IPDB database should be used. (See the Setup Requirements section for more details)
Warning: There is currently no bookmark support. If the data export process is running for a long period of time (over a day, for example), if there is a crash, or a system reboot occurs while the extractor utility is running, then it is recommended restarting the export process from the beginning. Due to this limitation, it is recommended using smaller export periods to avoid this potential issue.
Using the Graphical User Interface
After double-clicking the executable file on a Windows server, the following window will appear:
The following table describes the available options:
|IPDB Data Folder||Specify a folder containing the IPDB database. You must provide the path to a valid IPDB database.|
|Device Selection||Select a list of devices to be exported. By default, all devices under a selected folder are exported.|
|Regex Filter||(Optional) Export selected events matching a regex pattern.|
|Exclude messages based on Regex||(Optional) If selected and used with the RegEx Filter box, the events matching the RegEx Filter are excluded from the export.|
|Time Filter||Export events generated within a specified date range. (Optional) A date can be entered in the following format YYYYMMDD.|
|Output Folder||Specify a directory to store exported event data.|
|Message Format||Select the output event format from the following:|
- zconnector (default): use for importing into NetWitness Platform
- simple: syslog like format, used for simple viewing.
- syslog: RFC5424 format, used for advanced viewing
|Work Threads||Sets the number of processing threads between 1 and 64 to use for exporting. If export hardware supports it, more threads will export faster. The default value is 8.|
Note: The work is divided by event source. Thus more threads speed up the exporting when there are a large number of event sources to export.
|Limit Output File Sizes to 4 GB||Limits the output file size to 4 GB, which makes it easier to manage importing into NetWitness Platform.|
- If this option is checked, the export process will create multiple files for a single event source that has more than 4 GB of data saved. Each export file will be no bigger than 4 GB in size.
- If this option is unchecked (default), a single file per event source is created. The export file can be larger than 4 GB in size per event source.
|Replace Contents of Output folder||(Optional) When this option is checked, running the tool will delete all files in the target output directory before starting the export. If unchecked (default), and the folder contains previously exported data, the export could append information to existing export files.|
This option is useful if incorrect options were chosen during a previous export and wiping out the previous export data is preferred.
In most cases, RSA recommends using an empty output folder.
|Export||Click the Export button to start the export process.|
|Pause||If the tool is running, the export process can be paused by clicking this button.|
|Message Preview||Enables sample preview of up to 10,000 events depending on the size of the events.|
Note: The preview window size is 2,000,000 (2 million) characters.
The relatively small result set available in the preview area means you could use this output window to see the result of a search on the data for a particular username or IP address via regex string.
Using the Command Line
Below is a screenshot of running the tool with the -h (help) flag.
Note: The -input and -output flags require an absolute path to directories.
The following are some example commands:
- Export logs in the simple format:
IPDBExportUtility.exe -input c:\lsnode\lsnode -output c:\Exported_CMD\1_Simple -simple -export
Note the absolute paths that are specified for the input and output folders.
- Export logs generated between a date range in YYYYMMDD date format.
IPDBExportUtility.exe -input c:\lsnode\lsnode -output c:\Exported_CMD\1_Simple -simple -starttime 20000101 -endtime 20200124 -export
Export data between January 1, 2000 and January 24, 2020
- Export log data in zconnector format:
IPDBExportUtility.exe -input c:\lsnode\lsnode -output c:\Exported_CMD\2_zconnector -zconnector -export
- Export log data in zconnector format, and auto exit after the export process completes:
IPDBExportUtility.exe -input c:\lsnode\lsnode -output c:\Exported_CMD\2_zconnector -zconnector -export -exit
Import Data into NetWitness Platform
RSA highly recommends using a dedicated Log Decoder and Archiver (preferably a virtual machine) for importing the IPDB event data. The setup process below makes significant changes to the Archiver's configuration which could cause longer term storage issues if used for anything other than IPDB data use. The Log Decoder will only be used to import the data into the NetWitness Platform environment, and the Archiver will be used for long-term storage of the exported events. Once the events are on the Archiver the Log Decoder can be decommissioned as it is no longer required.
Configure the Log Decoder
Before injecting the IPDB data into a Log Decoder, the log collection time must be set to the value in the TCP Connector header of the IPDB-extracted event data.
- Connect to the Log Decoder REST API: http(s)://LogDecoder_IP-Address:50102
- Go to decoder > parsers > config.
- Change the lc.ctime.meta value from off to session.
- Click Set and verify the Success appears.
Configure the Archiver
Before consuming the IPDB data from the Log Decoder by an Archiver for use in reporting, the correct meta key information must be setup.
To add the meta keys to the Log Decoder on the Archiver's Configuration:
- Go to Admin > Services.
- Select the appropriate Archiver service and select Actions > View > Config.
- Select the Log Decoder that is aggregating the IPDB Data. Edit the service to include the meta keys by selecting all the keys.
- Click Save.
- Ensure that the included meta keys are seen after the Save and Apply have been clicked.
To correctly index all the meta keys the Archiver index XML file needs to be adjusted. To get the default indexes that were just set up, the index-concentrator.xml file will must be copied from an existing Concentrator to the Archiver.
- Go to Admin > Services.
- In the Services view, select a Concentrator, and click Actions > View > Config.
- Click the Files tab and select index-concentrator.xml file from the drop-down menu.
- Copy the contents of the file.
- Go back to Services, select the Archiver that will be used for the IPDB data and click Actions > View > Config.
- Click the Files tab and select the index-archiver-custom.xml file from the drop-down menu.
- Copy the contents from the index-concentrator.xml file into the index-archiver-custom.xml file and click Save.
- Restart the Archiver service.
- Ensure the following:
- Use the Log Player to inject the IPDB data exported into the Log Decoder. See the section below for how to use the Log Player tool.
- Once the data is imported into the Log Decoder and consumed by the Archiver, go to Monitor > Reports to create reports for the IPDB exported data.
Importing Data into NetWitness Platform with NwLogPlayer
To import the exported IPDB data into the NetWitness Platform Log Decoder, the NwLogPlayer utility must be used. The NwLogPlayer is installed by default on all 11.x Log Decoders and can be found under /usr/bin. The following is an example of how to use this command. Using a -h option only with the NwLogPlayer will print out a list of all NwLogPlayer options.
/usr/bin/NwLogPlayer -s LD-IP -r 3 -f filenameExportedData
- LD_IP: The IP address of the NetWitness Platform Log Decoder being used for this import process. In this example, it could be 127.0.0.1 (loopback) or the normal IP address of the Log Decoder.
- filenameExportedData: The path and name of the file which contains the exported IPDB data. An example would be /root/ipdbexporteddatafile.