000034849 - Cannot log in to RSA Security Analytics UI after upgrading to 10.6.2.

Document created by RSA Customer Support Employee on Mar 16, 2017Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000034849
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Security Analytics Server
RSA Version/Condition: 10.6.2
Platform: CentOS
O/S Version: 6
 
IssueAfter upgrading to Security Analytics 10.6.2 from 10.3.5 by following the supported steps, the SA UI returns the following error.
RSA Security Analytics Service Unavailable
Error Description:
Security Analytics server is unable to start. Please make sure all services are running and restart the Security Analytics server.
View online documentation for details about the error.


It is also noticed that the rabbitmq-server service continues to crash in every few hours to few days.
 
CauseThe issue can be caused by logstash consuming all of the available RabbitMQ sockets due to an incorrect setting in /etc/rsyslog.conf.
The root cause can be identified by following the steps below.
The rabbitmq-server can crash when sockets_used reaches sockets_limit that is found in service rabbitmq-server status as shown below.
 {file_descriptors,
     [{total_limit,3996},
      {total_used,3596},
      {sockets_limit,3594},
      {sockets_used,3594}]},
rabbitmqctl report
shows many connection attempts from 127.0.0.1:xxx -> to 127.0.0.1:5672 where xxx is a random high port.
netstat -anp |grep xxx shows that the connection is from java process.
ps -ef |grep <PID of java> shows references to logstash as below
... /opt/logstash -f /etc/logstash.conf.d -l /var/log/logstash/logstash.log
tail /var/log/logstash/logstash.log confirms many instances of RabbitMQ connection error as below.
{:timestamp=>"2017-02-13T16:50:30.366000+1100", :message=>"RabbitMQ connection error: . Will reconnect in 10", :level=>:error}
Upon reviewing /var/log/messages, an error on rsyslogd is noticed.
Feb 13 17:11:04 SASERVER rsyslogd-2040: error during parsing file /etc/rsyslog.conf, on or before line 22: parameter 'PollingInterval' not known -- typ in config file? [try http://www.rsyslog.com/e/2207 ]
An incorrect/invalid setting is found in /etc/rsyslog.conf.
ResolutionIn order to resolve the issue, back up the current /etc/rsyslog.conf and replace it with the default rsyslog.conf file that can be found from rsyslog-8.4.1-1.el6.x86_64 and restart rsyslog, rabbitmq-server and jettysrv.
cp /etc/rsyslog.conf /root/rsyslog.conf_bak
service rsyslog restart
service rabbitmq-server restart
restart jettysrv

Note: The file is automatically updated with the new one when the SA server is upgraded to a newer version unless it is customized.
If the default /etc/rsyslog.conf is modified to contain an incorrect setting, the future upgrade will not replace it so the problematic setting remains in the file.

Attachments

    Outcomes