000030934 - RabbitMQ service does not start due to corrupted mnesia database in RSA Security Analytics

Document created by RSA Customer Support Employee on Jun 14, 2016•Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000030934
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: Log Collector
RSA Version/Condition: 10.3.x, 10.4.x, 10.5.x
Platform: CentOS
Platform (Other): RabbitMQ
O/S Version: EL6
IssueRabbitMQ service does not start on the Log Collector after upgrading as the mnesia database has been corrupted.
As soon as the RabbitMQ service is started, it stops and shows the following error in the /var/log/rabbitmq/sa@localhost.log file:
 
starting file handle cache server                                     ...done
starting worker pool                                                  ...done
starting database                                                     ...BOOT 
ERROR: FAILED
Reason: {error,previous_upgrade_failed}
Stacktrace: [{rabbit_upgrade,apply_upgrades,1},
             {rabbit_upgrade,with_upgrade_graph,1},
             {rabbit_mnesia,init_db,2},
             {rabbit_mnesia,init,0},
             {rabbit,'-run_boot_step/1-lc$^1/1-1-',1},
             {rabbit,run_boot_step,1},
             {rabbit,'-start/2-lc$^0/1-0-',1},
             {rabbit,start,2}]
{"Kernel pid 
terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{rabbit,failure_during_boot}}}}}"}
WorkaroundAn interim workaround that fixes the problem would be to move the contents of /var/lib/rabbitmq/mnesia/sa@localhost and the sa@localhost-upgrade-backup folder to another location.
CAUTION:  All persistent messages and broker configurations (such as persistent queues, exchanges, and users) will be lost through this workaround.
Follow the steps below to perform the workaround.
  1. Make sure there is no rabbitmq process running in the system and kill any that are found.
    service rabbitmq-server stop
    ps aux | grep rabbitmq

  2. Move the directories and files into another folder (such as /tmp).
    mv /var/lib/rabbitmq/mnesia/sa@localhost /tmp
    mv /var/lib/rabbitmq/mnesia/sa@localhost-upgrade-backup /tmp

    NOTE:  Do not move the /var/lib/rabbitmq/mnesia/sa@localhost-plugins-expand directory.
  3. Start the RabbitMQ service.
    service rabbitmq-server start

  4. Run the puppet configuration to make sure all the certificates and CA trusts are in place.
    puppet agent -t

  5. In the Security Analytics UI, navigate to the Config view for the Log Collector and configure the destination again.  Refer to the Security Analytics User Guide for more information.
If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.   

Attachments

    Outcomes