000036341 - Modules tab is not opening/hanging in RSA NetWitness Endpoint Thick client

Document created by RSA Customer Support Employee on May 11, 2018
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000036341
Applies ToRSA Product Set: NetWitness Endpoint
RSA Product/Service Type: NetWitness Endpoint
RSA Version/Condition: 4.3, 4.4.0.0, 4.4.0.1, 4.4.0.2
Platform: Windows
 
IssueThere are some different symptoms associated with this issue:
  1. Modules tab turns grey and never finishes loading.
  2. Modules tab does finish loading but takes a long period of time to load the list, and the number of modules is fairly massive, much more than expected to be seen in the environment
  3. The job runtimes for a number of jobs are very long, including excessive Database_cleanup runtimes that span hours, ProcessIOCEvaluations that span long periods of time, including on the order of hours, and a number of other jobs related to modules that run for excessive lengths of time.
CauseThe MachineModulePaths and Modules tables have grown to a very large size due to a bug in the cleanup jobs that fail to remove entries that are MarkedAsDeleted=1. These tables grow to be over 100Gb in size, possibly much larger, and processing modules becomes so process intensive that the job runtimes are heavily impacted.

Additionally, the UI becomes unable to cache such a large amount of module data and the Modules tab starts to become inaccessible attempting to load so many modules and the associated meta.

SELECT COUNT(*) FROM MachineModulePaths

//MachineModulePaths count
(No column name)
294338222

NOTE: Attempting to run a select * statement on these tables may take long periods of time to complete, generally it is better when it starts to hang to simply cancel the query and run a select count(*) to verify the number of entries in these tables
SELECT COUNT(*) FROM Modules

//Number of total modules
40300103
ResolutionThis issue was fixed as part of a patch that was applied to 4.4.0.3 and all later versions of Netwitness Endpoint. Upgrading to 4.4.0.3+ will allow the cleanup jobs to begin rolling off this excess data but may take weeks to bring the MachineModulesPath and Modules tables down to a small enough number of entries that performance is considerably improved.

Verification


  1. Run the SQL Agent Stats query to gather the runtimes of the SQL Agent jobs, you should see a number of jobs that are running very long:
    • DatabaseCleanup     6.12 hours
    • ProcessGlobalAggregation 45 minutes
    • SyncIOCEvaluations 62 minutes
    • ProcessModuleAggregation 112 minutes
    • ProcessIOCEvaluations 116 minutes
  2. Run the table sizes query to verify the size of the tables in the database. Notably, the MachineModulePaths table will be huge:
    1. MachineModulePaths    dbo    168 Gb total size
  3. Verify the version is 4.4.0.2 or older

Attachments

    Outcomes