000038783 - After an upgrade to 11.4, RSA NetWitness is inaccessible and unable to access page

Document created by RSA Customer Support Employee on May 6, 2020Last modified by RSA Customer Support Employee on Jul 29, 2020
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000038783
Applies ToRSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: Core Appliance
RSA Version/Condition: 11.4, 11.4.1
IssueFrom sa.log, you can see error "Request to admin-server.any./rsa/process/ready timed out"

at java.base/java.lang.Thread.run(Thread.java:834)
[taskScheduler-5] ERROR com.netwitness.platform.server.common.atmosphere.WebSocketSessionExpiry - Error retrieving idle session timeout settings from admin-server
com.rsa.asoc.launch.api.transport.client.RequestTimeoutException: Request to admin-server.any./rsa/process/ready timed out
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClientHelper.requestTimeoutException(AmqpTransportClientHelper.java:51)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClientHelper.throwRequestTimeoutException(AmqpTransportClientHelper.java:44)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.doSendAndReceive(AmqpTransportClient.java:115)
at com.rsa.asoc.launch.api.transport.client.AmqpTransportClient.send(AmqpTransportClient.java:69)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.makeRemoteCall(TransportClientInvocationHandler.java:68)
at com.rsa.asoc.launch.api.transport.client.TransportClientInvocationHandler.invoke(TransportClientInvocationHandler.java:50)

From admin-server log, we can see that certificate is untrusted.


[ main] WARN Security|Certificate for CN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx,OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US issued by CN=Puppet CA: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx is not trusted

 


 

CauseDuring the upgrade, the trust certificate in /etc/netwitness/admin-server/ was modified.  The upgrade happened in this case on Feb 10, something tried changing the certificate of keystore.p12 during the time of the update Feb 10 15:56:
 

[root@xxxx admin-server]# ls -alh
total 84K
drwxr-xr-x. 2 netwitness netwitness 4.0K Feb 10 15:56 .
drwxr-xr-x. 24 netwitness netwitness 4.0K Feb 11 17:43 ..
r-------. 1 netwitness netwitness 576 Jan 11 06:47 admin-server.conf
rw-r----. 1 netwitness netwitness 14K Feb 10 15:56 keystore.p12
rw-rr-. 1 root root 13K Jan 6 18:47 keystore.p12.good
rw-rr-. 1 root root 12K Jan 6 18:47 keystore.p12.good.new
rw-rr-. 1 root root 13K Jan 6 18:47 keystore.p12.orig
rw-rr-. 1 netwitness netwitness 986 May 23 2018 lockbox.ss
rw-rr-. 1 netwitness netwitness 0 May 23 2018 lockbox.ss.lock
rw-rr-. 1 netwitness netwitness 240 Feb 10 15:56 modules.yml
rw-rr-. 1 netwitness netwitness 36 May 23 2018 service-id


 
Workaround
  1. Stop rsa-nw-admin-server service:
    systemctl stop rsa-nw-admin-server.service
  2. Change directory to admin-server:
    cd /var/log/netwitness/admin-server
  3. Backup existing keystore file:
    mv keystore.p12  keystore.p12.backup
  4. Replace keystore.p12 with the last known working keystore.p12.good.new:
    cp keystore.p12.good.new  keystore.p12
  5. Set permissions to keystore.p12:
    chmod 640 keystore.p12
    chown netwitness:netwitness keystore.p12
  6. Start rsa-nw-admin-server service:
    systemctl start rsa-nw-admin-server.service
  7. Restart the web application server service jetty:
    systemctl restart jetty
If replacing the keystore.p12 with the last known working keystore.p12.good.new does not work, you may have to try running a fix-keystore command in nw-shell.

For more information about using nw-shell and the fix-keystore command go to the following link: https://community.rsa.com/docs/DOC-110593.

NotesIf the last known backup keystore.p12 is not working, you may have to regenerate the certificates. To do so please see article below: 

  

Reissue root CA security certificates on RSA NetWitness Platform 11.x


   https://community.rsa.com/docs/DOC-107280

Attachments

    Outcomes