000030657 - Connection refused error when collectd is making a JMX connection to service in RSA Security Analytics 10.4.x and above

Document created by RSA Customer Support Employee on Aug 21, 2018
Version 1Show Document
  • View in full screen mode

Article Content

Article Number000030657
Applies ToRSA Product Set: RSA NetWitness Logs and Network (formerly RSA Security Analytics)
RSA Product/Service Type: Malware Analysis (MA), Event Stream Analysis (ESA), Head Unit / NetWitness Server [Incident Management, Health & Wellness (rsa-sms), Context Hub, Reporting Engine]
RSA Version/Condition: 10.4.x, 10.5.x, 10.6.x
Platform: CentOS 6
Platform (Other): collectd
O/S Version: EL6
IssueA message similar to the following can be seen repeating in /var/log/messages:

Apr 14 17:00:51 prodmalwareanalysis collectd[2478]: GenericJMXConfConnection: Creating MBean server connection failed: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is: #012#011java.net.ConnectException: Connection refused]

Note: Above message is from an RSA Malware Analysis host but could also have occurred on a RSA Event Stream Analysis (ESA) host or for a service on the RSA NetWitness Server.
CauseThe service collectd is having issues in communicating with Java based service using Java Management Extensions (JMX).
This may be normal and expected if the source service is stopped (and so is not accepting JMX connections).
Workaround1. Check if the source service is running (easy to guess on a dedicated MA or ESA host)
RSA Malware Analysis

# status rsaMalwareDevice

Example normal output:

rsaMalwareDevice start/running, process 16561

RSA Event Stream Analysis

# service rsa-esa status

Example normal output:

RSA NetWitness ESA :: Server is running (2847).

Note: If the service that collectd is collecting from is experiencing issues, proceed no further and work on correcting the service issue first.

For the rest of the example will assume issue is on a dedicated RSA Malware Analysis (MA) host:
2. Restart the data source service and collectd service after re-downloading collectd configuration from puppet recipe
2a. Stop the Malware Analysis service

# stop rsaMalwareDevice

Example Output:

rsaMalwareDevice stop/waiting

2b. Stop collectd service

# service collectd stop

Example Output:

Stopping collectd:                                         [  OK  ]

2c. Rename service config file

# find /etc/collectd.d -type f -name "jmx-*.conf" -exec mv {} {}.old \;

2d. Run puppet agent manually to download of collectd config file and restart of rsaMalwareDevice & collectd services:

# puppet agent -t

Example Output should include the following lines (excerpt):

Info: /Stage[main]/Malware-analysis/File[jmx-MalwareAnalysis.conf]: Scheduling refresh of Service[collectd]
Notice: /Stage[main]/Fips/Notify[FIPS Mode =  false]/message: defined 'message' as 'FIPS Mode =  false'
Notice: /Stage[main]/Yumconfig/Exec[disable-Centos-Repos]/returns: executed successfully
Notice: /Stage[main]/Malware-analysis/Service[rsaMalwareDevice]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Malware-analysis/Service[rsaMalwareDevice]: Unscheduling refresh on Service[rsaMalwareDevice]
Notice: /Stage[main]/Collectd/Service[collectd]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Collectd/Service[collectd]: Unscheduling refresh on Service[collectd]

3. If issue remains try restarting the OS of the appliance.

shutdown -r now   # or simply restart

If you require any further assistance please contact RSA Support