000033018 - Rabbitmq service is not starting post 10.6.0.0 upgrade due to incorrect rabbitmq soft link in RSA Security Analytics

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000033018
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Security Analytics Server, Security Analytics Core hosts
RSA Version/Condition: 10.6.0.0
Platform: CentOS
O/S Version: EL6
IssueAfter upgrading RSA Security Analytics hosts to 10.6.0.0, rabbitmq service may not be starting with below errors.
 
[root@XXXX ~]# service rabbitmq-server status
/usr/sbin/rabbitmqctl: line 28: cd: /var/lib/rabbitmq: No such file or directory
{error_logger,{{2016,4,28},{14,43,59}},"Failed to create cookie file '/var/lib/rabbitmq/.erlang.cookie': enoent",[]}

[root@XXXX ~]#  tail -f /var/log/rabbitmq/startup_err
/usr/sbin/rabbitmq-server: line 28: cd: /var/lib/rabbitmq: No such file or directory
Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,auth,{"Failed to create cookie file '/var/lib/rabbitmq/.erlang.cookie'

 
CauseThis issue is due to an invalid soft link is still being used for /var/lib/rabbitmq which is required for the rabbitmq service.
 
[root@XXXX ~]# ll /var/lib/
total 96
drwxr-xr-x.  2 root     root     4096 Feb  1 17:49 alternatives
drwx------.  3 root     root     4096 Oct 15  2014 authconfig
drwxr-xr-x.  2 root     root     4096 Jan 29 17:06 collectd
drwxr-xr-x.  2 root     root     4096 Jul 24  2015 dhclient
drwxr-xr-x.  2 root     root     4096 Sep 23  2011 games
drwxr-xr-x.  2 root     root     4096 Jun  6  2014 hiera
-rw-r--r--.  1 root     root      814 Apr 28 14:01 logrotate.status
drwxr-xr-x.  2 root     root     4096 Sep 23  2011 misc
drwxr-x---.  2 root     slocate  4096 Apr 28 04:02 mlocate
drwxr-xr-x.  2 root     root     4096 Aug 17  2015 net-snmp
drwxr-xr-x.  5 root     root     4096 Apr 27 17:01 nfs
drwxr-xr-x.  2 ntp      ntp      4096 Apr 28 14:38 ntp
drwxr-xr-x.  2 root     root     4096 Aug 11  2014 plymouth
drwx------.  2 postfix  root     4096 Nov 10 09:59 postfix
drwxr-x---. 11 puppet   puppet   4096 Oct 21  2015 puppet
lrwxrwxrwx.  1 root     root       37 Dec 30 21:48 rabbitmq -> /var/netwitness/logcollector/rabbitmq
drwxr-xr-x.  2 rabbitmq rabbitmq 4096 Jun 24  2014 rabbitmq.2015-01-13T23:08:58
drwxr-xr-x.  3 rabbitmq rabbitmq 4096 Dec 30 21:46 rabbitmq.2015-12-30T21:48:58
-rw-------.  1 root     root     4096 Apr 27 20:38 random-seed
drwxr-xr-x.  2 root     root     4096 Apr 27 20:38 rpm
drwx------.  2 root     root     4096 Sep 29  2014 rsyslog
drwxr-xr-x.  4 root     root     4096 Dec 15 15:55 stateless
drwxr-xr-x.  3 root     root     4096 Jul 24  2015 udev
drwxr-xr-x.  3 root     root     4096 Feb 26  2015 xfsdump
drwxr-xr-x.  6 root     root     4096 Apr 27 17:17 yum

In Security Analytics 10.4 and older, a symbolic link was created for /var/lib/rabbitmq to use /var/netwitness/logcollector/rabbitmq or /var/netwitness/rabbitmq. However, from 10.5, there is a separate mount point for /var/lib/rabbitmq. However, in some cases, the symbolic link (to a non-existing path) is not removed causing the problem.
ResolutionPlease follow below steps for resolution.
  1. SSH into the host (e.g. log collector) as a root user.
  2. Remove the soft link using below commands.
    • cd /var/lib
    • rm rabbitmq
  3. Reinstall rabbitmq-server rpm again using below command.
    • yum reinstall rabbitmq-server-3.4.2-1.noarch
  4. Run puppet agent -t which starts rabbitmq service automatically.
  5. Verify the service rabbitmq-server status command output to make sure the rabbitmq service is running.

Attachments

    Outcomes