000038001 - Reissue root CA security certificates on RSA NetWitness Platform 11.x

Document created by RSA Customer Support Employee on Sep 26, 2019Last modified by RSA Customer Support Employee on Nov 25, 2019
Version 22Show Document
  • View in full screen mode

Article Content

Article Number000038001
Applies To

RSA Product Set: RSA NetWitness Platform
RSA Product/Service Type: All RSA NetWitness Platform hosts (appliances)
RSA Version/Condition: 11.x (This article applies to customers who have upgraded RSA NetWitness Platform hosts from older versions of 10.x to 11.x.)
Platform: CentOS 7




    IssueCustomers who have upgraded their RSA NetWitness Platform hosts from older versions of 10.x to 11.x may have internal RSA-issued security certificates that are close to expiring or which have already expired.
    CauseWhen upgrading the RSA NetWitness Platform from 10.6.x to 11.x, the older Certificate Authority (CA) along with other internal security certificates are still in use, even though the original expiration dates of these certificates were not updated and maybe close to their expiration date.
    Resolution

    Caution: Do not run the script to update the Root CA certificates in these situations:




     



    Consider the following guidelines before the Root CA Update:



    • Prepare for adequate downtime of your environment. Depending on the number of hosts in your deployment, this procedure can take several hours to complete. Plan your change window accordingly.
    • Services will be restarted during the reissue process, causing data capture interruptions. After reissuing the root CA certificate, the root CA reissue script restarts all the services.
    • Renew certificates for all components in the same change window. Do not break up the process across multiple change windows.
    • If you have Windows Legacy Collectors (WLC) in your deployment, renew certificates for them within the same change window. See the separate instructions below for Windows Legacy Collector (WLC) hosts.

     


    If you updated from an older version of RSA NetWitness Platform 10.x to 11.x, check and reissue the internal RSA-issued security certificates.

    Prerequisites



    1. Download root-ca-update.zip from the Certificate Re-issue Tool for RSA NetWitness 11.x page on RSA Link and upload the contents to your RSA NetWitness (Admin) server /root directory using any available file transfer software (FileZilla, WinSCP, and so on).
    2. SSH to the RSA NetWitness server, login as root, and extract the root-ca-update.zip content.
    3. Install the RPM on the NetWitness server using  # yum install -y rsa-nw-root-ca-update-<version number>.noarch.rpm:

    If 11.3.2 or later is already installed on your NetWitness Platform, skip this step and go to the Procedures section below. A version of the RPM mentioned in this step is already installed.




    # yum install -y rsa-nw-root-ca-update-1.0.3.0-1910171844.1.030c45b.el7.noarch.rpm


     



    Procedures



    Determine if the systems may have internal RSA NetWitness Platform security certificates that are about to expire or have already expired



    1. Locate the ca-expire-test.sh attached to this article.
    2. Run the certificate expiration test on the RSA NetWitness server.

      # sh ca-expire-test.sh --version 11.x

    3. If the test runs successfully, the following output will be displayed. The highlighted number below shows the number of days until the certificates are due to expire. 

      NW Platform CA Certificate Expiration Date:  Aug  8 20:03:27 2020 GMT
      NW SSCA Expiration Date:                     Aug 12 15:17:48 2028 GMT
      NW Node Certificate Expiration Date:         Jul 11 20:45:35 2022 GMT

      #########################################################

      You must re-issue certificate within 297 days

      #########################################################
      RSA highly recommends you update your certificate if it is expired, or about to expire. Failure to do so will result in a system wide outage.
      Please review the Knowledge Base Article this script was attached to for what steps to take.
      You can contact Netwitness Support at the following address https://community.rsa.com/community/support

    4. RSA highly recommends you update your certificate if the Platform CA or SSCA CA certificate is expired, or about to expire. Failure to do so will result in a system-wide outage.


    Update the Certificates


    In the following procedure, you will run a script that updates the Root CA and node certificates on the NetWitness Server and the other hosts and then restarts the services. Before you run the script, you will perform a pre-check, which prints the validity and integrity of the certificates.

    1. Precheck: SSH to the RSA NetWitness (Admin) server and run the following pre-check and save the output to a file:


    # keytool -printcert -file /etc/rabbitmq/ssl/truststore.pem | grep -Ei "owner|valid"  >> /var/log/netwitness/cert-renew-precheck.out


    1. Execute the nw-root-ca-update script to clean up the Puppet CA in truststore.pem by running the following command:


    # nw-root-ca-update --clean-imports


    1. Execute the nw-root-ca-update script to create the new CA certificate by running the following commands:

    1. To update the root CA certificate on the RSA NetWitness Server:


      # nw-root-ca-update --renew-ca


      • This prompts you for your deploy_admin password.
      • This may take a few minutes to complete. Please wait until it completes before running the next command.
      • If you see the following message, your root CA certificate updated successfully on the NetWitness Server:
         

        *** root CA update successful, run --synch-host to update component hosts ***

         

    1. To synchronize the reissued root CA certificate with all the other NetWitness hosts:


    # nw-root-ca-update --synch-host --host-all


    It may take a few minutes per host to complete. Until then, the hosts may show offline on the NetWitness Platform user interface (Admin > Hosts). Estimate the required time to complete based on the size of your deployment and wait until the above-issued command completes.



    You can use --synch-host with multiple variations. See the following option examples.
    Option 1: Run the --synch-host for a set of nodes to run in sequence mode.




    nw-root-ca-update --synch-host --host-id <node-id1> --host-id <node-id2>

    Option 2:  Run the --synch-host for a set of nodes to run in parallel mode.


    nw-root-ca-update --use-parallel --synch-host --host-id <node-id1> --host-id <node-id2>



    The node-id of any host can be found in the /etc/salt/minion file or by executing the upgrade-cli-client --list command on the NetWitness server.



    1. Verify the certificate expiration dates:

      1. After reissuing the certificates, run the certification test again on the NetWitness server using the ca-expire-test.sh script.
      2. Verify that the RSA NetWitness Platform CA certificate expiration date is extended for ten years.
      3. Verify that all your system services are running. In NetWitness Platform (Admin > Services), the status of all the services except WLC should show as running (Green Circle).
    2. Custom or ECAT feeds that are configured with HTTPS must be reconfigured. For additional details on how to configure them, see article 000029414 - Integrate ECAT Feed with RSA Security Analytics.
    3. If you run into any issues check the troubleshooting section later in this document. If the issue is not addressed, contact RSA Customer Support.



    Renew certificates that have expired or not expired for Windows Legacy Collector (WLC) hosts




    You must use the wlc-cert-renew-11.x.sh script to renew the Windows Legacy Collector certificates. This and dependent scripts are part of the ZIP file that was downloaded and extracted on the RSA NetWitness server. (node-zero)



    Prerequisites



    Ensure all WLCs are running with RSA NetWitness Platform version 11.2 or above.



    1. Ensure that the RSA NetWitness Platform 11.x root CA update is completed all for non-WLC systems.
    2. The script will prompt for the Security Server username and password. Enter the admin credentials for the RSA NetWitness Platform.
    3. Create a folder by running the following command and changing into it:



    # mkdir /root/wlc-certreissue
    # cd /root/wlc-certreissue



    1. From the location where the root-ca-update.zip was extracted, copy the following files to the /root/wlc-certreissue directory:

    • 11.3-wlc-cli-client.sh
    • wlc-cert-renew-11.x.sh
    • update-WLC-truststore.bat

    1. Run the following command to make the scripts executable:



    # chmod +x /root/wlc-certreissue/*.sh



    1. On the RSA NetWitness Server (node-zero), create a wlc-systems file. This will be used to enter the details of the WLC being updated. The file should use the format of <wlc_ip_address>,<wlc_REST_admin_username>,<wlc_REST_admin_password>.  For example,



    10.1.10.231,admin,netwitness



    Renewing certs for each WLC takes approximately five minutes each, so it is recommended to create the wlc-systems file with one entry at a time and make sure that there are no spaces or blank lines in the wlc-systems file.



    1. There are manual steps needed on EACH Windows Legacy Collector system. Be prepared to log in to each WLC host using Windows Administrator credentials.


    To renew the certificates for Windows Legacy Collector (WLC) hosts



    In a multi Windows Legacy Collector environment, this procedure must be repeated for each one. The process should only be performed on a single Windows Legacy Collector at a time.



    1. Check the certificate details. To view the details of the current certificate in use on the WLC, run the following command on the RSA NetWitness server (node-zero):



    # ./wlc-cert-renew-11.x.sh wlc-systems



    1. Renew the certificates. To renew the certificates for the configured WLC host, run the following command on the NetWitness Server (node-zero):



    # ./wlc-cert-renew-11.x.sh --renew wlc-systems



    1. If the certificates are successfully renewed, the following message is displayed. If they are not successful, see the troubleshooting section later in this document.



    Wed Oct 14 11:23:47 UTC 2026: WLC_CERT_RENEW: [INFO] ---------------------------------------
    Wed Oct 14 11:23:47 UTC 2026: WLC_CERT_RENEW: [INFO] Renew completed successfully for 2 of 2
    Wed Oct 14 11:23:47 UTC 2026: WLC_CERT_RENEW: [INFO] Done
    Wed Oct 14 11:23:47 UTC 2026: WLC_CERT_RENEW: [INFO] --------------------------------------



    The next steps need to be performed on the Windows Legacy Collector. Perform the following manual steps only once per WLC:



    1. Log into the WLC host using Windows administrator credentials.
    2. Copy over the update-WLC-truststore.bat file from step 4 in the Prerequisites section to a suitable directory on the WLC host.
    3. Start the command prompt and change the directory to where the update-WLC-truststore.bat was copied.
    4. Run the update-WLC-truststore.bat file. When it finishes it will make the needed changes and stop/start the RabbitMQ, NwLogCollector, and nwStatCollector services.



    # update-WLC-truststore.bat



    1. In the RSA NetWitness Platform user interface, confirm that the WLC service status shows as running (green filled circle) and the shovel status is running (green filled circle). If not, see the WLC Troubleshooting in the Troubleshooting section later in this document.



    Troubleshooting Information



    If you have any issues, look at the following log files to try to determine the root cause:



    • /var/log/netwitness/root-ca-update/netwitness-key-repair.log
    • /var/log/netwitness/root-ca-update/update.log


    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------




    Issue: An error like the one below is seen when running nw-root-ca-update –synch-host –host-all:





    ba914d35-5edd-47a8-93a9-1d9c5f3437f6:
    Minion did not return. [Not connected]
    Verifying : rsa-nw-root-ca-update-1.0.2.1-1910160114.1.71403e6.el7.noarch



    Resolution:




    1. Determine the salt minion id of the host that failed by running the command upgrade-cli-client -l |grep <local minion id that failed>



    # upgrade-cli-client -l | grep ba914d35-5edd-47a8-93a9-1d9c5f3437f6
    Host: ID=ba914d35-5edd-47a8-93a9-1d9c5f3437f6, ADDR=10.1.11.12, NAME=upgradeesa-up, VERSION=11.2.0.0



    1. Get the host IP address from the above call and see if that node is active using the NetWitness Platform user interface or by using SSH to the NetWitness Server host. In the above example, the host address is 10.1.11.12.
    2. If the host is reachable, ensure that salt-minion is running:



    # service status salt-minion



    1. If it does not return anything, then it is probably a host that you previously orchestrated and removed, which can be safely ignored.

    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------




    Issue: Error messages like the ones listed below are seen while installing the rsa-nw-root-ca-update RPM





    mkdir: cannot create directory '/var/log/netwitness/root-ca-update': No space left on device




    ERROR: Minions returned with non-zero exit code




    [2019-10-11T17:57:13+00:00] <25809> (WARN) Failed to complete setup on one or more component hosts!




    [2019-10-11T17:57:13+00:00] <25809> (WARN) Please rerun --post-install once failed node(s) are healthy...



    Resolution: Address the issue mentioned in the error message and the run the following command to finish the certificate update:





    # nw-root-ca-update --post-install



    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------



    This issue and resolution applies only to the Event Stream Analysis server.




    Issue: The ESA Correlation, ESA Analytics, or Context Hub services do not come up after the root CA update and an error message similar to the example will appear in the service log:





    2019-10-10 15:41:32,793 [main] WARN Security|Certificate for CN=f332065c-2fb2-4a24-9e59-f0781dda7fe1,
    OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US issued by CN=Puppet CA: f332065c-2fb2-4a24-9e59-f0781dda7fe1
    is not trusted



    Resolution: Fix the issue by running the command java -Dre-construct-keystore=true -Dservices=<servicename-server> -jar /var/netwitness/download/root-ca-update/fix-launch-keystores.jar from the SSH terminal on the ESA host where the service is installed. Servicename-server must be contexthub-server, esa-analytics-server, or correlation-server. For example,





    # java -Dre-construct-keystore=true -Dservices=esa-analytics-server -jar /var/netwitness/download/root-ca-update/fix-launch-keystores.jar



    Do not use the above command for any other service than the three specifically mentioned in this step. (esa-analytics-server, correlation-server, contexthub-server)



    ----------------------------------------------------------------------------------------------------------------------------------------------------------------------



    This issue and resolution apply only to Windows Legacy Collectors.




    Issue: Windows Legacy Collector Troubleshooting




    Resolution: Review the following items to help with troubleshooting WLC issues.



    • After completing the steps to renew certificates for WLC hosts, if shovel issues persist, make sure the RabbitMQ, NeLogCollector, and NwStatCollector services are running on the WLC host.
    • On the user interface, if the shovel is still in an error state (Red Circle), edit the shovel and save it without making any changes.
    • If the Health & Wellness statistics for the WLC do not appear, stop/start RabbitMQ and the NwStatCollector on the WLC host.

    Attachments

    Outcomes