000030331 - Connectivity issues are seen with RSA Security Analytics 10.4.1 services due to RabbitMQ errors

Document created by RSA Customer Support Employee on Jun 17, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 4Show Document
  • View in full screen mode

Article Content

Article Number000030331
Applies ToRSA Product Set: Security Analytics
RSA Version/Condition: 10.4.1.0
Platform: CentOS
O/S Version: EL6
IssueThe error message "Service is Unreachable" is displayed when attempting to view a service within the Security Analytics UI.
Issuing the command netstat -ano on the affected appliance shows many ports open in a CLOSE_WAIT state.
The /var/log/messages file on the affected appliance has many collectd errors similar to the following:
May 19 16:22:50 PDecoder collectd[20033]: An error occurred publishing a statistic for plugin 1dea3f1f-76eb-4431-908a-ab03c549995e/sms_collectd.MessageBusWriteModule-counter-published.  Error: An error occurred publishing an AMQP Message.  Exchange name: carlos.sms.collectd; error: a socket error occurred; message size: 317
May 19 16:22:50 PDecoder collectd[20033]: An error occurred publishing a statistic for plugin 1dea3f1f-76eb-4431-908a-ab03c549995e/decoder_processinfo-string-service_status.  Error: An error occurred publishing an AMQP Message.  Exchange name: carlos.sms.collectd; error: An error occurred creating an AMQP Channel.  Configuration: {#012    "urn": "carlos.sms.collectd",#012    "connection":#012    {#012        "vhost": "\/rsa\/system"#012    }#012}#012; error: a socket error occurred; message size: 262

The /var/log/rabbitmq/sa@localhost.log file on the affected appliance reports errors similar to the example below.
May 19 14:57:39 LDecoder nw[714]: [MessageBroker] [failure] error 2015-05-01T14.57.39Z ** Generic server rabbit_mgmt_external_stats terminating ** Last message in was emit_update** When Server state == {state,1024}** Reason for termination == ** {badarg, [{erlang,list_to_binary,[['tlsv1.2','tlsv1.1',tlsv1]],
 

The /var/log/messages file also shows many instances of a "Stopping RabbitMQ" message, as shown in the screenshot below.
Stopping RabbitMQ messages
CauseThe connectivity issues occur because of a known issue with RabbitMQ in Security Analytics 10.4.1 that is propagated to the appliances via puppet.
ResolutionIn order to resolve the issue, perform the steps below.
  1. Download the init.pp file that is attached to this article and transfer it to the /root directory on the Security Analytics Server appliance.
  2. Connect to the Security Analytics Server appliance via SSH as the root user and issue the following commands:
    1. mv /etc/puppet/modules/rabbitmq/manifests/init.pp /etc/puppet/modules/rabbitmq/manifests/init.pp.orig
    2. mv /root/init.pp /etc/puppet/modules/rabbitmq/manifests/init.pp
    3. service puppetmaster restart
    4. service puppet restart
    5. service rabbitmq-server restart
  3. Connect to the affected appliance(s) via SSH as the root user and issue the command below to apply the change.
    1. puppet agent -t
If you are unsure of any of the steps above or experience any issues, contact RSA Support and quote this article number for further assistance.

Attachments

Outcomes