000031318 - LogCollector service unavailalable and device not showing up to date on RSA Security Analytics 10.4.1.0 User Interface

Document created by RSA Customer Support Employee on Jul 7, 2016•Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000031318
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Security Analytics Server
RSA Version/Condition: 10.4.1.0
Platform: CentOS
O/S Version: 6
 
Issue
This issue describes when customers try to  perform an SA upgrade from 10.4.1 to 10.5.0 and  they run into issues with rabbitmq after Log Hybrid  is upgraded. LogCollector device doesn't show up to date on SA UI and also appears unavailable. 
 
[root@igteblrsiemhyb1 tmp]# service rabbitmq-server status 
Status of node sa@localhost ... 
Error: unable to connect to node sa@localhost: nodedown

 
DIAGNOSTICS 
=========== 
attempted to contact: [sa@localhost] 
sa@localhost: 
* connected to epmd (port 4369) on localhost 
* epmd reports: node 'sa' not running at all 
other nodes on localhost: ['rabbitmqctl-35484'] 
* suggestion: start the node 
current node details: 
- node name: 'rabbitmqctl-35484@igteblrsiemhyb1' 
- home dir: /var/lib/rabbitmq 
- cookie hash: K2qzPBHLJ1HEpkGE+faD2g==

Also /var/log/rabbitmq/startup_log file shows following error: 

 
[{app_utils,load_applications,2,[]}, 
{app_utils,load_applications,1,[]}, 
{rabbit,start_apps,1,[]}, 
{rabbit,broker_start,0,[]}, 
{rabbit,start_it,1,[]}, 
{init,start_it,1,[]}, 
{init,start_em,1,[]}] 
{"init terminating in do_boot",{rabbit,failure_during_boot,{error,{"no such file or directory","nw_admin.app"}}}} 
[FAILED]


 
CauseWe've seen a few issues like this that stem from the nw_admin plugin.This is sometimes fixed by editing the /etc/rabbitmq/rsa_enabled_plugins file, removing nw_admin from it, then running puppet agent -t. 
WorkaroundSteps for the workaround in order to fix this issue are listed below:
1- Check RabbitMQ port is opened: 
 
"netstat -ntpl |grep 4369" 
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 3533/epmd

2 - Take a backup of the /etc/rabbitmq/rsa_enabled_plugins file
3- Edit the /etc/rabbitmq/rsa_enabled_plugins file
4- Remove nw_admin  from it
5- Then run puppet agent -t
6 - Check RabbitMQ service:
 
# service rabbitmq-server status

7- If RabbitMQ is up, restart jettysrv on the SA server:
 
# stop jettysrv
# start jettysrv

8 - Restart the nwlogcollector service from the Log Hybrid device:
 
# stop nwlogcollector 
# start nwlogcollector


9- If this doesn't work, try reinstalling the rsa-puppet-modules rpm:
 
# yum reinstall rsa-puppet-modules


10- If there is no relief from either of these, run the command manually and capture the output to attach to a Jira case for further troubleshooting: 
 
# rabbitmq-plugins enable rabbitmq_federation


 

Attachments

    Outcomes