000034816 - RSA Security Operations Management: Incident Management Endpoint between UCF and NetWitness intermittently goes down

Document created by RSA Customer Support Employee on Mar 2, 2017Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 3Show Document
  • View in full screen mode

Article Content

Article Number000034816
Applies ToRSA Product Set: Security Management
RSA Product/Service Type: SecOps
RSA Version/Condition: 1.2 and Newer
Platform: Windows/Linux
 
IssueSecurity Analytics Incident Management (SAIM) endpoint between the Windows Host UCF and NetWitness keeps going down intermittently. 
 

24 Oct 2016 00:12:46,501 | ERROR - SaimAmqpServiceJob.tryStartRabbitConnection(171) | Failed to start the SAIM Service Job with error: java.net.SocketException: Software caused connection abort: recv failed.  Please verify if the rabbitmq service is running in the Security Analytics box. Will retry again later.


29 Oct 2016 00:03:45,942 | ERROR - TcpNioConnection.readPacket(489) | Exception on Read s0adcqualys6.us.royalahold.net:36133:1515:d18bf8c0-ced0-450e-9b6c-e5fdafd55bbe An established connection was aborted by the software in your host machine
java.io.IOException: An established connection was aborted by the software in your host machine
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(Unknown Source)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
    at sun.nio.ch.IOUtil.read(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
    at org.springframework.integration.ip.tcp.connection.TcpNioConnection.doRead(TcpNioConnection.java:404)
    at org.springframework.integration.ip.tcp.connection.TcpNioConnection.readPacket(TcpNioConnection.java:477)
    at org.springframework.integration.ip.tcp.connection.AbstractConnectionFactory$1.run(AbstractConnectionFactory.java:633)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)


 

29 Oct 2016 00:03:45,942 | WARN - TcpConnectionSupport.doPublish(383) | No publisher available to publish TcpConnectionCloseEvent [source=org.springframework.integration.ip.tcp.connection.TcpNioSSLConnection@53498ed2], [factory=unknown, connectionId=s0adcqualys6.us.royalahold.net:36133:1515:d18bf8c0-ced0-450e-9b6c-e5fdafd55bbe] **CLOSED**


 

29 Oct 2016 00:03:47,206 | WARN - SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.logConsumerException(1208) | Consumer raised exception, processing can restart if the connection factory supports it
org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:63)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:369)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:80)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:130)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:401)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1054)
    at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.recvAlert(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
    at sun.security.ssl.AppOutputStream.write(Unknown Source)
    at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
    at java.io.BufferedOutputStream.flush(Unknown Source)
    at java.io.DataOutputStream.flush(Unknown Source)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:129)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:134)
    at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:276)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:590)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:624)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)
    ... 7 more



 
CauseUCF incorrectly appends its certificate to the "truststore.pem" certificate of RabbitMQ on the NetWitness head unit.  It should be appending to "ca.pem" instead. Puppet Agent could push updates to the "truststore.pem" certificate for many reasons. Some examples are (but not limited to):
  1. Adding a new Windows Legacy Log Collector (WLC)
  2. Restarting services (or rebooting) WLC
  3. Updating WLC to a newer version
ResolutionLong term fix with be a code change in a future release.
Workaround
  1. Locate the rootcastore.crt.pem, not keystore.crt.pem (same location at C:\Program Files\RSA\SA IM integration service\cert-tool\certs). See attached screenshot for the location.
User-added image

  1. Copy the rootcastore.crt.pem to the SA server (the /root folder).
  2. Remove the DOS formatting for line break. A convenient way to do is using command “vi rootcastore.crt.pem" to open the file then run the command:
    :%s/\r//g

  3. Then proceed to save the file using:
    :wq

  4. See attached screenshots on how the DOS formatting ^M is removed by using the command
User-added image
User-added image

  1. Make a backup of the ca.pem file, this is the correct location where the UCF certificate will be:
    cp /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/ssl/certs/ca.org.pem

  2. After that, proceed to append the certificate to the ca.pem (not truststore.pem) using: 
    cat /root/rootcastore.crt.pem >> /var/lib/puppet/ssl/certs/ca.pem

  3. Run puppet agent -t to have the certificate populate to the truststore.pem

Attachments

    Outcomes