000027904 - Certs used for trusted connections in RSA Security Analytics are stale after "Remove and Repurpose"

Document created by RSA Customer Support Employee on Jun 15, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000027904
Applies ToRSA Product Set: Security Analytics
RSA Product/Service Type: SA Core Appliance
RSA Version/Condition: 10.4.0.1,10.4.0
Platform: Linux
IssueErrors similar to this below are observed in /var/log/messages:
Oct 16 18:45:13 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: client not initialized--cannot accept stat /sys/license/stats/license.status
Oct 16 18:45:14 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: nwsdk failure: NwOpen returned 0; code 0; error: Could not find trusted session id in hello response
Oct 16 18:45:14 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: failed to connect to device: failed to connect to nws://admin@localhost:56003/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem
Oct 16 18:45:14 xxxxx collectd[22183]: NgNativeReader_NwBroker-SlowUpdate: client not initialized--cannot accept stat /sys/license/stats/license.version Oct 16 18:45:18 gsodc1loganal01 collectd[22183]: NgNativeReader_NwBroker-FastUpdate: nwsdk failure: NwOpen returned 0; code 0; error: Could not find trusted session id in hello response
Oct 16 18:45:18 xxxxx collectd[22183]: NgNativeReader_NwBroker-FastUpdate: failed to connect to device: failed to connect to nws://admin@localhost:56003/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem
Oct 16 18:45:18 xxxxx collectd[22183]: NgNativeReader_NwIPDBExtractor-FastUpdate: nwsdk failure: NwOpen returned 0; code 0; error: Could not find trusted session id in hello response
Oct 16 18:45:18 xxxxx collectd[22183]: NgNativeReader_NwIPDBExtractor-FastUpdate: failed to connect to device: failed to connect to nws://admin@localhost:56025/?group=Administrators&cert=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fcerts%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem&key=%2Fvar%2Flib%2Fpuppet%2Fssl%2Fprivate_keys%2F3bcfd8ce-e7f6-4bbe-be9e-52462992a533.pem
Cause

When the appliance is re-added (and re-provisioned) through the Security Analytics UI, the puppet recipes only check whether the key and certificate material used for trusted connections (/etc/netwitness/ng/<svc>/trust(peers|certs) and /opt/rsa/carlos/keystore) exist.  They are not checked for whether they exist and are synchronized with the values in /var/lib/puppet/ssl.
Examples of <svc> include the following:


  • appliance
  • logdecoder


As a consequence, trusted connections do not work after re-provisioning the appliance.

When this occurs, the only remedy is to manually remove the trusted certs and re-run puppet.

Resolution

The Steps below can be performed to correct this issue.


  1. For Security Analytics classic nodes (decoder, logdecoder, concentrator,broker,etc)
    1. Connect to appliance via SSH and login with root user
    2. Create temp folder under /tmp directory for each service to store backup certificate, example, #mkdir /tmp/logdecoder
    3. Navigate to appropriate directory with following command, #cd /etc/netwitness/ng/<svc>/trustpeers
    4. Issue following command to backup and remove certificate for each service, # mv * /tmp/logdecoder/
    5. Re-run puppet in the foreground, #puppet agent -t, or wait up to 30 minutes for puppet to run automatically.
    6. Verify that new trusted peer certificate are created. 
    7. Restart the collectd service with the command service collectd restart
  2. For Security Analytics appliance that has Carlos service (ESA, Malware,etc)
    1. Connect to appliance via SSH and login with root user
    2. Create temp folder under /tmp directory for appliance to store backup keystore, example, #mkdir /tmp/esa
    3. Navigate to appropriate directory with following command, #cd /opt/rsa/carlos
    4. Issue following command to backup and remove keystore file, # mv * /tmp/esa/
    5. Re-run puppet in the foreground, #puppet agent -t, or wait up to 30 minutes for puppet to run automatically.
    6. Verify that new keystore file is created. 
    7. Restart the collectd service with the command service collectd restart

Attachments

    Outcomes