Nikolay Klender

Logs deduplication

Discussion created by Nikolay Klender on Oct 9, 2017
Latest reply on Oct 10, 2017 by Dave Glover

Hi! Is there any way to deduplicate events? Say we have two firewalls (intrnal and external) and if some host tries to connect to internet site we will have two log records with same ip.src, ip.dst, ip.dstport. So It will be good to have ability to deduplicate logs in following way: 

1) define deduplication key like set of metas

2) define time period during which logs will be throtled in case of dedup key is equal

So I am looking some thing like logstash throttle filter.


I think it could be done with help of lua parser but there is question about thread safety.