SA/NW Version: 10.6.1.0
We are having an odd issue that just started happening last week (after we updated some parsers from Live). During the day(s) our aggregation from our log decoder on our concentrator gets ridiculously behind. Never catches up. We've observed this is only happening during the day. Around 6PM , the problem goes away. Never reaching >1M sessions behind. This is a new issue, nothing has changed from what I can tell in our collections.
The health and wellness alerts say "check for noisy parser" but searching through /var/log/messages i see no errors. Somewhere else to check? Has anyone ran into an issue like this?
To rule out data -
- Turned off syslog listener (514udp/tcp) on log collector . Left it off for 5 mins - no affect, behind still grows.
- Turned off Windows Collections. Left off for 5 mins - no affect, behind still grows.
- Turned off VMware Collections. Left off for 5 mins - no affect, behind still grows.
- Restarted nwlogdecoder service
We tried tuning sdk/config settings per: 000034117 - Performance of Concentrator Service on Hybrid Appliance is slow in RSA Security Analytics ; hasn't helped.
We have a support case open with RSA and they have been helpful. However I thought to post the issue here in case someone has ran into this before. Or has any ideas.