000035358 - Error "Unable to connect to node sa@localhost: node down" appears after moving RDQ files manually to another directory in RSA Security Analytics

Document created by RSA Customer Support Employee on Aug 19, 2017
Version 1Show Document
  • Comment • 0
  • View in full screen mode

Article Content

Article Number000035358
Applies ToRSA Product Set: Security Analytics, NetWitness Logs & Packets
RSA Product/Service Type: Log Collector
RSA Version/Condition:10.4.x and above 
Platform: CentOS
IssueAfter moving the rdq files manually from /var/lib/rabbitmq/mnesia/sa\@localhost/msg_store_persistent/ to any other temp directory in order to prevent it to reach the threshold of 80% is causing rabbitmq server not to start.
User-added image 
 
Resolution

IMPORTANT: After following the below workaround, the user will not be able to re-inject the RDQ files back in to the Log Decoder for processing. If this data must be reprocessed, contact RSA Customer Support for assistance before continuing. 


  1. Connect to the Log Collector via SSH as the root user.
  2. Change to the /var/lib/rabbitmq/mnesia/sa@localhost/queues directory.
    [root@LOGDECODER queues]# ll
    total 24
    drwxr-x---. 2 rabbitmq rabbitmq 4096 Jul  4 07:35 2T16GPLM6GOAXPFRG41XPFSSA
    drwxr-x---. 2 rabbitmq rabbitmq 4096 Jul  4 08:01 59LPG0HYYP81PHP7YGZ8XTSF5
    drwxr-x---. 2 rabbitmq rabbitmq 4096 Jul  3 12:20 5FZKF04O6BORNU1WMXASJYUYT
    drwxr-x---. 2 rabbitmq rabbitmq 4096 Jul  4 08:11 8CG4JO8XCLTF2WVB5OOF8NLTY
    drwxr-x---. 2 rabbitmq rabbitmq 4096 Mar  1 16:41 9ULRAPZV46OWKATP3WXPJLK4H
    drwxr-x---. 2 rabbitmq rabbitmq 4096 Jul  3 13:14 E5UW9XH1GMXAYEBE1JF8KZKB9

  3. Remove all the files from this directory.
    rm -rf ./*

  4. Stop all of the associated services.
    service puppet stop
    service collectd stop
    service mcollective stop
    stop nwlogcollector

  5. Kill the "beam" and "empd" processes for rabbitmq server.
    ps -ef | grep -i "beam"
    ps -ef | grep -i "empd"

  6. Trigger a puppet catalog run which should start all of the services including rabbitmq-server successfully.
    puppet agent -t

Attachments

    Outcomes