000032710 - RSA Security Analytics UI temporarily unavailable after upgrading to SA 10.5.1.0

Document created by RSA Customer Support Employee on Jun 30, 2016•Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000032710
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Security Analytics Server, SA Security Analytics UI
RSA Version/Condition: 10.5.1.0
Platform: CentOS
O/S Version: EL6
IssueAfter upgrading to Security Analytics 10.5.1.0, the SA UI (web interface which listens on port 443/TCP) may not be immediately available and performance of the SA server may be very slow.
CauseThe cause is collectd performing a migration of the rsa-sms RRD files. This is seen as multiple python processes in 'top' output with moderate CPU usage.
 
# top
top - 05:35:59 up 45 min,  3 users,  load average: 15.87, 11.95, 5.79
Tasks: 767 total,  15 running, 752 sleeping,   0 stopped,   0 zombie
Cpu(s): 53.3%us,  3.2%sy,  0.1%ni, 43.2%id,  0.1%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:  99154660k total, 27216736k used, 71937924k free,    73060k buffers
Swap:  4194296k total,        0k used,  4194296k free,  2627660k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
15250 rabbitmq  20   0 5613m 971m 3880 S 229.2  1.0  34:52.75 beam.smp
13715 root      20   0  454m 195m 1776 R 100.2  0.2   5:51.48 python
13716 root      20   0  454m 195m 1776 R 100.2  0.2   5:47.27 python
27866 root      20   0  454m 195m 1776 R 100.2  0.2   2:39.51 python
28616 root      20   0  454m 195m 1776 R 100.2  0.2   2:38.86 python
28617 root      20   0  374m 115m 1776 R 99.9  0.1   2:40.53 python
27864 root      20   0  374m 115m 1776 R 96.6  0.1   2:39.64 python
28615 root      20   0  445m 186m 1776 R 95.9  0.2   2:40.78 python
27862 root      20   0  454m 195m 1776 R 94.9  0.2   2:41.09 python
13713 root      20   0  374m 115m 1776 R 94.3  0.1   5:51.10 python
28621 root      20   0  454m 195m 1776 R 94.3  0.2   2:39.48 python
27865 root      20   0  454m 195m 1776 R 93.9  0.2   2:40.74 python
28613 root      20   0  454m 195m 1776 R 93.3  0.2   2:41.22 python
13712 root      20   0  454m 195m 1776 R 92.6  0.2   5:50.79 python
13714 root      20   0  415m 156m 1776 R 91.9  0.2   5:47.98 python
25617 root      20   0 20.9g 1.5g  20m S 69.5  1.6   3:43.53 java

 
# ps eax | grep python
28613 ?        R      0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28615 ?        Rl     0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28616 ?        Rl     0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28617 ?        Rl     0:59 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3
28621 ?        Rl     0:58 /usr/bin/python /usr/lib/collectd/python/migrate_SMS_RRD_files.py TERM=linux PATH=/sbin:/usr/sbin:/bin:/usr/bin _=/usr/bin/python PWD=/var/lib/collectd LANG=en_US.UTF-8 SHLVL=3

 
ResolutionThis process is automatic and will complete eventually. Once the rsa-sms service has started then jettysrv service can fully load.
Progress can be tracked through collectd's SMS_RRD_migration logs.
 
# cd /var/lib/netwitness/collectd
# ls -ltr
total 525916
-rw-r--r--.  1 root root 532351827 Jan 19 16:54 stats.tgz
drwxr-xr-x. 83 root root      4096 Feb  3 23:15 rrd
-rw-r--r--.  1 root root   2054193 Mar  2 05:50 SMS_RRD_migration_20160302052940.log
-rw-r--r--.  1 root root   2040073 Mar  2 05:50 SMS_RRD_migration_20160302053311.log
-rw-r--r--.  1 root root   2031828 Mar  2 05:50 SMS_RRD_migration_20160302053312.log
Notes/etc/init/jettysrv.conf shows the service dependencies for the jettysrv upstart service.
In SA 10.5.X, the jettysrv service depends on the following services to be running before the SA UI will be available: rabbitmq-server, tokumx and rsa-sms
These Linux System V services located in /etc/rc.d/init.d can be checked using:
service rabbitmq-server status
service tokumx status
service sa-sms status
 

Attachments

    Outcomes