000037995 - NwBroker service continuously killed by Out of memory killer when any alerts are enabled in reporting engine in RSA NetWitness Platform

Document created by RSA Customer Support Employee on Oct 14, 2019
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000037995
Applies ToRSA Product Set: NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 10.6.6.0 , 11.0.x, 11.1.x and 11.2.0
 
Issue

NwBroker service continuously killed due to OOM when any alerts are enabled in the reporting engine.




Aug 28 14:49:46 RSA-HEAD kernel: service thread invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Aug 28 14:49:46 RSA-HEAD kernel: service thread cpuset=/ mems_allowed=0-1
Aug 28 14:49:46 RSA-HEAD kernel: Pid: 7947, comm: service thread Not tainted 2.6.32-696.28.1.el6.x86_64 #1

# egrep 'Out of memory:' messages | tail -n 10
Aug 28 14:01:40 RSA-HEAD kernel: Out of memory: Kill process 31504 (NwBroker) score 636 or sacrifice child
Aug 28 14:17:49 RSA-HEAD kernel: Out of memory: Kill process 23039 (NwBroker) score 636 or sacrifice child
Aug 28 14:33:52 RSA-HEAD kernel: Out of memory: Kill process 8856 (NwBroker) score 636 or sacrifice child
Aug 28 14:33:52 RSA-HEAD kernel: Out of memory: Kill process 8858 (stat updates) score 636 or sacrifice child
Aug 28 14:49:46 RSA-HEAD kernel: Out of memory: Kill process 7908 (NwBroker) score 636 or sacrifice child
Aug 28 14:49:46 RSA-HEAD kernel: Out of memory: Kill process 7921 (stat updates) score 636 or sacrifice child
Aug 28 14:49:46 RSA-HEAD kernel: Out of memory: Kill process 18292 (SDK-Query) score 636 or sacrifice child
Aug 28 15:05:47 RSA-HEAD kernel: Out of memory: Kill process 19040 (NwBroker) score 635 or sacrifice child
Aug 28 15:21:54 RSA-HEAD kernel: Out of memory: Kill process 32645 (NwBroker) score 635 or sacrifice child
Aug 28 15:37:43 RSA-HEAD kernel: Out of memory: Kill process 29366 (NwBroker) score 635 or sacrifice child

 

And found that the SDK queries by alerts cause Select * query.




Sep 17 14:13:37 RSA-HEAD NwBroker[28182]: [SDK-Query] [audit] User admin (session 33331, 127.0.0.1:50794) has finished query (channel 34603, queued 00:00:00, execute 00:00:00, 10.180.2.57:50005=00:00:00 168.78.204.175:50005=00:00:00): id1=6766405865459 id2=6766420528306 threshold=0 query="select * where udp.dstport exists"
Sep 17 14:14:37 RSA-HEAD NwBroker[28182]: [SDK-Query] [audit] User admin (session 33331, 127.0.0.1:50794) has issued query (channel 34617) (thread 28520): id1=6766420528307 id2=6766434885991 threshold=0 query="select * where udp.dstport exists"
CauseEach alert execution in the Reporting Engine causes Select * Query to be executed on the Core side. This causes the Reporting Engine to pull a LOT of data for each query in Memory. Also, as an additional headache, in case the Alert is producing a lot of results, the Query can take substantial time to complete.
Resolution

This issue was resolved at 10.6.5.1 HF and 11.2.1 and later.

The Alert Execution to fetch only the sessionids first, then we read the session data by reading 50 Sessions at a time. This flow will make sure that we are not causing the Reporting Engine to launch Heavy Queries on the Core Device.

Attachments

    Outcomes