000032710 - RSA NetWitness Logs & Network UI temporarily unavailable after upgrading to NW 10.6

Document created by RSA Customer Support Employee on Jun 30, 2016•Last modified by RSA Customer Support on Apr 18, 2019
Version 3Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000032710
Applies ToRSA Product Set: NetWitness Logs & Network
RSA Product/Service Type: NetWitness Server, NetWitness UI
RSA Version/Condition: 10.6.x
Platform: CentOS
O/S Version: EL6
IssueAfter upgrading to NetWitness Logs & Network 10.6.x, the NetWitness UI (web interface which listens on port 443/TCP) may not be immediately available and performance of the NetWitness head-unit server may be very slow.
CauseThis can be caused if the collectd service is performing a migration of the rsa-sms RRD files.
This is seen as multiple python processes in a 'top' output with moderate CPU usage on the NetWitness head-unit server.
 

# top
top - 05:35:59 up 45 min,  3 users,  load average: 15.87, 11.95, 5.79
Tasks: 767 total,  15 running, 752 sleeping,   0 stopped,   0 zombie
Cpu(s): 53.3%us,  3.2%sy,  0.1%ni, 43.2%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  99154660k total, 27216736k used, 71937924k free,    73060k buffers
Swap:  4194296k total,        0k used,  4194296k free,  2627660k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15250 rabbitmq  20   0 5613m 971m 3880 S 229.2  1.0  34:52.75 beam.smp
13715 root      20   0  454m 195m 1776 R 100.2  0.2   5:51.48 python
13716 root      20   0  454m 195m 1776 R 100.2  0.2   5:47.27 python
27866 root      20   0  454m 195m 1776 R 100.2  0.2   2:39.51 python
28616 root      20   0  454m 195m 1776 R 100.2  0.2   2:38.86 python
28617 root      20   0  374m 115m 1776 R 99.9  0.1   2:40.53 python
27864 root      20   0  374m 115m 1776 R 96.6  0.1   2:39.64 python
28615 root      20   0  445m 186m 1776 R 95.9  0.2   2:40.78 python
27862 root      20   0  454m 195m 1776 R 94.9  0.2   2:41.09 python
13713 root      20   0  374m 115m 1776 R 94.3  0.1   5:51.10 python
28621 root      20   0  454m 195m 1776 R 94.3  0.2   2:39.48 python
27865 root      20   0  454m 195m 1776 R 93.9  0.2   2:40.74 python
28613 root      20   0  454m 195m 1776 R 93.3  0.2   2:41.22 python
13712 root      20   0  454m 195m 1776 R 92.6  0.2   5:50.79 python
13714 root      20   0  415m 156m 1776 R 91.9  0.2   5:47.98 python
25617 root      20   0 20.9g 1.5g  20m S 69.5  1.6   3:43.53 java

 


# ps eax | grep python
28613 ?        R      0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28615 ?        Rl     0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28616 ?        Rl     0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28617 ?        Rl     0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28621 ?        Rl     0:58 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3


 
ResolutionThis process is automatic and will complete eventually.  Once the rsa-sms service has started, then the jettysrv service can fully load.
Progress can be tracked through the collectd's SMS_RRD_migration logs.
 

# cd /var/lib/netwitness/collectd
# ls -ltr
total 525916
-rw-r--r--.  1 root root 532351827 Jan 19 16:54 stats.tgz
drwxr-xr-x. 83 root root      4096 Feb  3 23:15 rrd
-rw-r--r--.  1 root root   2054193 Mar  2 05:50 SMS_RRD_migration_20160302052940.log
-rw-r--r--.  1 root root   2040073 Mar  2 05:50 SMS_RRD_migration_20160302053311.log
-rw-r--r--.  1 root root   2031828 Mar  2 05:50 SMS_RRD_migration_20160302053312.log
NotesThe /etc/init/jettysrv.conf file shows the service dependencies for the jettysrv upstart service.

In NetWitness 10.6.x, the jettysrv service depends on the following services to be running before the NetWitness UI will be available: rabbitmq-server, tokumx and rsa-sms
These Linux System V services are located in /etc/rc.d/init.d and can be checked using:
 

service rabbitmq-server status
service tokumx status
service sa-sms status


 

Attachments

    Outcomes