Grok Filter Plugin
You can use Grok to parse incoming logs and extract valuable meta information. For details, see Grok filter plugin: match. The meta extracted using grok is part of the full event package sent to NetWitness Platform, where it can be mapped to NetWitness meta.
- List of Grok patterns: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
- Grok debugger: https://grokdebug.herokuapp.com/
See the following URLs:
You can use the beats input plugin for Logstash to receive events from beats sources, including Filebeat & Auditbeat: Beats Input Plugin
You can use the drop filter plugin for Logstash to filter out unwanted logs being passed through Logstash: Drop Filter Plugin
You can also filter out unwanted logs using the drop-event processor for Filebeat & Auditbeat.
Examples for Filebeat:
Examples for Auditbeat:
Configure heartbeat plugin to send test logs to NetWitness
The Heartbeat plugin can be used to send a test message to verify connectivity between logstash and NetWitness. For details, see the Heartbeat Plugin configuration guide: Heartbeat Input Plugin.
Configure logstash to persist events in case of failure (Recommended)
By default, Logstash uses in-memory queues to buffer events. The size of these in-memory queues is not configurable. If there is a machine failure, or if the service is forcibly stopped, the contents of these queues are lost. To protect against data loss in these situations, Logstash supports persistent queues that are stored on disk and thus can survive failures. For details, see Logstash Persistent Queues.
By default, the NetWitness codec sends the complete JSON event as payload to the NetWitness Log Decoder. If the NetWitness nw_type device parser type has a custom payload format, you must configure the NetWitness codec plugin to recognize this custom format.
The payload_format and payload_format_failover mappings use nw_type as the key. The payload_format mapping is searched first for the device type (nw_type). If the device type is not set, or no format is specified for nw_type, or the system fails to make all configured variable substitutions, the complete JSON output is used as the payload. If the primary format from the payload_format mapping fails, the payload_format_failover mapping is tried. If that also fails, the complete JSON output is used as the payload. You can use The format can use Logstash event field syntax for this custom configuration.
The following code snippet shows example of adding meta with custom payload formats.
Troubleshoot Installation Issues
If you encounter any issues during installation of Logstash, see Logstash Installation and Setup.
Configure NetWitness Platform to Collect Events
You need to start capture on the Log Decoder to which your are sending your Logstash data.
To start or restart network capture on a Log Decoder:
- Log in to NetWitness Platform and click the ADMIN icon > Services.
The Services view is displayed.
- Select a Log Decoder service.
- Under (actions), select View > System.
In the toolbar, click .
By default, Log Decoders support events that are up to 32 KB in size. If your events are getting truncated on the Log Decoder, use the following procedure to change the event size:
- Change LogDecoder REST config at http://LogDecoder_IP:50102/decoder/config, where LogDecoder_IP is the IP address of your Log Decoder.
- Set pool.packet.page.size to 64 KB.
- Restart the Log Decoder: this is required after you change the pool.packet.page value.