000034816 - RSA Security Operations Management (SecOps) Incident Management Endpoint between UCF and the RSA NetWitness Platform intermittently goes down

24 Oct 2016 00:12:46,501 | ERROR - SaimAmqpServiceJob.tryStartRabbitConnection(171) | Failed to start the SAIM Service Job with error: java.net.SocketException: Software caused connection abort: recv failed.  Please verify if the rabbitmq service is running in the Security Analytics box. Will retry again later.

29 Oct 2016 00:03:45,942 | ERROR - TcpNioConnection.readPacket(489) | Exception on Read rsaconnection.example.com:36133:1515:d18bf8c0-ced0-450e-9b6c-e5fdafd55bbe An established connection was aborted by the software in your host machine
java.io.IOException: An established connection was aborted by the software in your host machine
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(Unknown Source)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
    at sun.nio.ch.IOUtil.read(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
    at org.springframework.integration.ip.tcp.connection.TcpNioConnection.doRead(TcpNioConnection.java:404)
    at org.springframework.integration.ip.tcp.connection.TcpNioConnection.readPacket(TcpNioConnection.java:477)
    at org.springframework.integration.ip.tcp.connection.AbstractConnectionFactory$1.run(AbstractConnectionFactory.java:633)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

29 Oct 2016 00:03:45,942 | WARN - TcpConnectionSupport.doPublish(383) | No publisher available to publish TcpConnectionCloseEvent [source=org.springframework.integration.ip.tcp.connection.TcpNioSSLConnection@53498ed2], [factory=unknown, connectionId=rsaconnection.example.com:36133:1515:d18bf8c0-ced0-450e-9b6c-e5fdafd55bbe] **CLOSED**

29 Oct 2016 00:03:47,206 | WARN - SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.logConsumerException(1208) | Consumer raised exception, processing can restart if the connection factory supports it
org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:63)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:369)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:80)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:130)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:401)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1054)
    at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.recvAlert(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
    at sun.security.ssl.AppOutputStream.write(Unknown Source)
    at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
    at java.io.BufferedOutputStream.flush(Unknown Source)
    at java.io.DataOutputStream.flush(Unknown Source)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:129)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:134)
    at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:276)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:590)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:624)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)
    ... 7 more

• Adding a new Windows Legacy Log Collector (WLC)
• Restarting services (or rebooting) WLC
• Updating WLC to a newer version

cat keystore.crt.pem >> /etc/puppet/modules/rabbitmq/files/truststore.pem

1. Locate the rootcastore.crt.pem, not keystore.crt.pem (same location at C:\Program Files\RSA\SA IM integration service\cert-tool\certs). See attached screenshot for the location.

2. Copy the rootcastore.crt.pem to the SA server (the /root folder).
3. Check whether the format of the copied file is a Windows Text file (with Carriage Return & Line Feed [CR LF] characters)
   

   
   # file rootcastore.crt.pem
   rootcastore.crt.pem: ASCII text, with CRLF line terminators

   
   
   
   1. 
       
      If you see 'CRLF line terminators' output from the file command please perform the following:
      
      
       

      
      # vi rootcastore.crt.pem

      
         
      
       
      File Before (Windows Text format with ^M characters denoting LF line terminators).
       
       
      
         
   2. 
       
      3b - Convert text file from Windows Text format to Unix Format (LF only)
      
      
       

      
      :%s/\r//g
      set ff=unix
      

      
       
   3. 
       
      3c- Then proceed to save the file using
      
      
       

      
      :wq

      
         
      
       
      See below screenshot on how the DOS formatting ^M characters have been removed
      
         
      
       
      File After (Unix Text format)
       
      
       

4. Make a backup of the ca.pem file, this is the correct location where the UCF certificate will be:
   

   
   cp /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/ssl/certs/ca.pem.$(date +"%Y%m%d_%H%M")

   
   
5. After that, proceed to append the certificate to the ca.pem (not truststore.pem) using: 
   

   
   cat /root/rootcastore.crt.pem >> /var/lib/puppet/ssl/certs/ca.pem

   
   
6. Remove all blank lines from the file (optional but recommended)

sed -ri '/^\s*$/d' /var/lib/puppet/ssl/certs/ca.pem 

7. Run puppet agent -t to have the certificate populate to the truststore.pem. In NetWitness 10.6.x can check truststore.pem using the command:

cat /etc/rabbitmq/ssl/truststore.pem

• Make sure you edit /var/lib/puppet/ssl/certs/ca.pem and NOT /etc/rabbitmq/ssl/truststore.pem directly.
• Do NOT update the puppet recipe /etc/puppet/modules/rabbitmq/files/truststore.pem as SA server may overwrite the content in its operations removing the UCF certificate, causing broken connection between UCF and SA server. Actions such as NetWitness software upgrades may overwrite this file.