|Applies To||RSA Product Set: Security Analytics|
RSA Product/Service Type: Security Analytics UI, Health & Wellness, Security Analytics Server
RSA Version/Condition: 10.4.1.0, 10.4.1.1, 10.5.0.1, 10.5.0.2, 10.5.1
O/S Version: EL6
|Issue||Multiple CLOSE_WAIT connections to the Jetty web server from various hosts eventually leads to connectivity issues that cause the Security Analytics UI to become unreachable, as shown in the example below.|
[root@SA-Server ~]# netstat -anp | grep 443
tcp 982 0 192.168.1.1:443 192.168.1.2:62403 CLOSE_WAIT 30163/java
tcp 523 0 192.168.1.1:443 192.168.1.3:50466 CLOSE_WAIT 30163/java
tcp 1 0 192.168.1.1:443 192.168.1.4:54124 CLOSE_WAIT 30163/java
tcp 1 0 192.168.1.1:443 192.168.1.4:52478 CLOSE_WAIT 30163/java
tcp 982 0 192.168.1.1:443 192.168.1.5:62312 CLOSE_WAIT 30163/java
tcp 2054 0 192.168.1.1:443 192.168.1.4:52521 CLOSE_WAIT 30163/java
The /var/lib/netwitness/uax/logs/sa.log file streams the following errors when the issue is occurring:
2015-05-07 17:33:10,457 [pool-3-thread-31304] ERROR com.rsa.smc.sa.admin.util.monitoring.MessageBusReader - System Monitoring message queue is full
Restarting the rsa-sms, collectd, and rabbitmq-server services on the appliance have no effect on the issue.
|Resolution||A hotfix to resolve the issue has been prepared by the Engineering team for Security Analytics 10.4.1.1.|
The issue is being investigated by the Engineering team for the other affected versions.
If you are experiencing this issue, contact RSA Support and quote this article number for further assistance.
|Workaround||In order to temporarily resolve the issue, the jettysrv service on the Security Analytics server appliance can be restarted by issuing the commands below.|
Another workaround to prevent the issue from occurring in the future is to modify the SMS polling intervals in the puppet recipes to occur every 60 seconds rather than every 10 seconds, and by modifying the collectd interval to occur every 180 seconds rather than every 60 seconds. This can be done by issuing the command below on the Security Analytics server.
[root@SA-Server ~]# updatedb && for files in $(locate --regex ".conf.erb$");do sed -i 's/interval "60"/interval "180"/g' $files; sed -i 's/interval "10"/interval "60"/g' $files;done && sed -i 's/interval 60/interval 180/g' /etc/puppet/modules/rsa-sms-server/files/_collectd_java.conf && sed -i 's/interval "5"/interval "60"/g' /etc/puppet/modules/broker/templates/NwBroker.conf.erb && service puppet restart