000034816 - RSA Security Operations Management (SecOps) Incident Management Endpoint between UCF and the RSA NetWitness Platform intermittently goes down

Document created by RSA Customer Support Employee on Mar 2, 2017Last modified by RSA Customer Support on Feb 27, 2019
Version 5Show Document
  • View in full screen mode

Article Content

Article Number000034816
Applies ToRSA Product Set: Archer, Security Management, NetWitness
RSA Product/Service Type: Security Operations Management (SecOps) & NetWitness 10.6.x
RSA Version/Condition: 1.2 and Newer
Platform: Windows/CentOS
IssueRSA Security Analytics Incident Management (SAIM) endpoint between the Windows Host UCF and the RSA NetWitness Platform keeps going down intermittently. 
 

24 Oct 2016 00:12:46,501 | ERROR - SaimAmqpServiceJob.tryStartRabbitConnection(171) | Failed to start the SAIM Service Job with error: java.net.SocketException: Software caused connection abort: recv failed.  Please verify if the rabbitmq service is running in the Security Analytics box. Will retry again later.



29 Oct 2016 00:03:45,942 | ERROR - TcpNioConnection.readPacket(489) | Exception on Read rsaconnection.example.com:36133:1515:d18bf8c0-ced0-450e-9b6c-e5fdafd55bbe An established connection was aborted by the software in your host machine
java.io.IOException: An established connection was aborted by the software in your host machine
    at sun.nio.ch.SocketDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(Unknown Source)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
    at sun.nio.ch.IOUtil.read(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
    at org.springframework.integration.ip.tcp.connection.TcpNioConnection.doRead(TcpNioConnection.java:404)
    at org.springframework.integration.ip.tcp.connection.TcpNioConnection.readPacket(TcpNioConnection.java:477)
    at org.springframework.integration.ip.tcp.connection.AbstractConnectionFactory$1.run(AbstractConnectionFactory.java:633)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)


 

29 Oct 2016 00:03:45,942 | WARN - TcpConnectionSupport.doPublish(383) | No publisher available to publish TcpConnectionCloseEvent [source=org.springframework.integration.ip.tcp.connection.TcpNioSSLConnection@53498ed2], [factory=unknown, connectionId=rsaconnection.example.com:36133:1515:d18bf8c0-ced0-450e-9b6c-e5fdafd55bbe] **CLOSED**


 

29 Oct 2016 00:03:47,206 | WARN - SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.logConsumerException(1208) | Consumer raised exception, processing can restart if the connection factory supports it
org.springframework.amqp.AmqpIOException: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:63)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:369)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createConnection(ConnectionFactoryUtils.java:80)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:130)
    at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:401)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1054)
    at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.Alerts.getSSLException(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.recvAlert(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
    at sun.security.ssl.SSLSocketImpl.writeRecord(Unknown Source)
    at sun.security.ssl.AppOutputStream.write(Unknown Source)
    at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
    at java.io.BufferedOutputStream.flush(Unknown Source)
    at java.io.DataOutputStream.flush(Unknown Source)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:129)
    at com.rabbitmq.client.impl.SocketFrameHandler.sendHeader(SocketFrameHandler.java:134)
    at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:276)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:590)
    at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:624)
    at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)
    ... 7 more

CauseUCF incorrectly appends its certificate to the "truststore.pem" certificate of RabbitMQ on the NetWitness head unit.  It should be appending to "ca.pem" instead. Puppet Agent could push updates to the "truststore.pem" certificate for many reasons. Some examples are (but not limited to):
  • Adding a new Windows Legacy Log Collector (WLC)
  • Restarting services (or rebooting) WLC
  • Updating WLC to a newer version

Note: The following KB corrects the guidance given in the RSA SecOps documentation (RSA Archer Security Operations Management Installation and Configuration Guide/RSA Archer Integration Guide) will lead to issues.
The file referenced keystore.crt.pem should have been rootcastore.crt.pem
The real problem command is this one which will likely append a Windows text file to a Linux file and update the truststore.pem in the puppet recipe used by all 10.6.x NetWitness Hosts:

cat keystore.crt.pem >> /etc/puppet/modules/rabbitmq/files/truststore.pem

WorkaroundIf you are running NetWitness 11.x then please refer to the KB #000036450 - The command "orchestration-cli-client --update-admin-node" fails while trying to import certificates in RSA NetWitness Logs & Network

For NetWitness 10.6.x follow the steps below to resolve the issue.
  1. Locate the rootcastore.crt.pem, not keystore.crt.pem (same location at C:\Program Files\RSA\SA IM integration service\cert-tool\certs). See attached screenshot for the location.

User-added image


  1. Copy the rootcastore.crt.pem to the SA server (the /root folder).
  2. Check whether the format of the copied file is a Windows Text file (with Carriage Return & Line Feed [CR LF] characters)

    # file rootcastore.crt.pem
    rootcastore.crt.pem: ASCII text, with CRLF line terminators



    1.  
      If you see 'CRLF line terminators' output from the file command please perform the following:


       

      # vi rootcastore.crt.pem

         

       
      File Before (Windows Text format with ^M characters denoting LF line terminators).
        User-added image
        dos text file in vi

         

    2.  
      3b - Convert text file from Windows Text format to Unix Format (LF only)


       

      :%s/\r//g
      set ff=unix


       

    3.  
      3c- Then proceed to save the file using


       

      :wq

         

       
      See below screenshot on how the DOS formatting ^M characters have been removed

         

       
      File After (Unix Text format)
        User-added image

       

 


  1. Make a backup of the ca.pem file, this is the correct location where the UCF certificate will be:

    cp /var/lib/puppet/ssl/certs/ca.pem /var/lib/puppet/ssl/certs/ca.pem.$(date +"%Y%m%d_%H%M")

  2. After that, proceed to append the certificate to the ca.pem (not truststore.pem) using: 

    cat /root/rootcastore.crt.pem >> /var/lib/puppet/ssl/certs/ca.pem

  3. Remove all blank lines from the file (optional but recommended)


sed -ri '/^\s*$/d' /var/lib/puppet/ssl/certs/ca.pem


  1. Run puppet agent -t to have the certificate populate to the truststore.pem. In NetWitness 10.6.x can check truststore.pem using the command:


cat /etc/rabbitmq/ssl/truststore.pem

 









Notes
  • Make sure you edit /var/lib/puppet/ssl/certs/ca.pem and NOT /etc/rabbitmq/ssl/truststore.pem directly.
  • Do NOT update the puppet recipe /etc/puppet/modules/rabbitmq/files/truststore.pem as SA server may overwrite the content in its operations removing the UCF certificate, causing broken connection between UCF and SA server. Actions such as NetWitness software upgrades may overwrite this file.

Attachments

    Outcomes