Hi! Is there any way to deduplicate events? Say we have two firewalls (intrnal and external) and if some host tries to connect to internet site we will have two log records with same ip.src, ip.dst, ip.dstport. So It will be good to have ability to deduplicate logs in following way:
1) define deduplication key like set of metas
2) define time period during which logs will be throtled in case of dedup key is equal
So I am looking some thing like logstash throttle filter.
I think it could be done with help of lua parser but there is question about thread safety.