Slow Batch Processing(BPM) due to MachineID propagation related to Blocking commands in RSA NetWitness Endpoint
RSA Product Set: NetWitness Endpoint RSA Product/Service Type: NetWitness Endpoint RSA Version/Condition: 184.108.40.206 and prior versions Platform: Windows
BPM query is showing reduced processing of scans which seems to coincide with a dramatic increase in MachineID commands in the P0 queue. The machine commands table has a large number of blocking hash commands queued.
The database is queuing up blocking updates for ALL endpoints whether blocking is enabled or not, which fills the MachineCommands table with updates that reduce the systems ability to process batch data instead until these commands clear. An example is when a module is blacklisted or unblacklisted, we generate machine commands for all Windows agents regardless of the need to actually propagate these commands across the environment.
This issue is defined by a large number of blocking updates created, and two ways to resolve them exist depending on two primary factors:
Changes to the base code that impact this issue include only updating machines that require these changes rather than impacting ALL machines, such as excluding machines that do not have blocking updates enabled. Upgrade version is 220.127.116.11
There is a second scenario that can produce this issue: the multiple agentID rotation. When more than one endpoint has the same AgentID, their data overwrites the database constantly. This, in turn, spins up new blocking updates to that agent since it assumes it hasn't been updated, but then gets overwritten almost immediately by the next agent's data in the set, and keeps going in an infinite loop causing massive numbers of blocking updates to appear. The resolution is to find and eliminate all machines that have duplicate agentID's.
Multiple AgentID's are normally created only under circumstances where images of machines are pre-generated with the AgentID already installed, typically VDI/VM machines. Since the agentID is identical due to the way the Agent UID is calculated under specific deployment types, this can generate agents with the same ID. Check kb article 000034741 for methods of detecting and resolving this issue.
See kb article 000034741 for a fix to the duplicate AgentID issue specifically.