000036450 - The command "orchestration-cli-client --update-admin-node" fails while trying to import certificates in RSA NetWitness Logs & Network

Document created by RSA Customer Support Employee on Jun 19, 2018Last modified by RSA Customer Support Employee on Jul 14, 2018
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000036450
Applies ToRSA Product Set: RSA NetWitness Logs & Network
RSA Product/Service Type: RSA NetWitness Host
RSA Version/Condition: 11.x
Platform: CentOS
O/S Version: EL7
IssueWhen attempting to integrate the RSA Archer Platform or RSA NetWitness Endpoint with the RSA NetWitness Respond Server, you may encounter an error when trying to import certificates into the RSA NetWitness Admin Server RabbitMQ keystore. After placing your .pem file(s) into the /etc/pki/nw/trust/import folder and running orchestration-cli-client --update-admin-node, you may encounter an error like the following:

2018-06-13 16:51:35,313 [                          main] INFO             Bootstrap|Service logs will be written to /var/log/netwitness/orchestration-client
2018-06-13 16:51:35,319 [                          main] INFO             Bootstrap|Service configuration will be read from /etc/netwitness/orchestration-client
2018-06-13 16:51:35,696 [                          main] INFO             Bootstrap|Starting orchestration-client.87d919c0-20d3-4397-8d50-728bda6ae8ff (v0.0.0.0)
2018-06-13 16:51:36,191 [                          main] INFO             Bootstrap|Initialized service cryptography with 4 providers (BSAFE=CRYPTOJ 6.2.2 20161215 0745, FIPS-140=true).
2018-06-13 16:51:36,973 [                          main] INFO             Bootstrap|A version change was detected and an upgrade is not required.
2018-06-13 16:51:37,133 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|Starting OrchestrationApplication on netwitnesssa with PID 190674 (/usr/bin/orchestration-cli-client.jar started by root in /etc/pki/nw/trust/import)
2018-06-13 16:51:37,133 [                          main] INFO  c.r.n.i.o.c.OrchestrationApplication|The following profiles are active: standard
2018-06-13 16:51:37,212 [                          main] INFO             Bootstrap|Service will accept AMQP requests at broker localhost:5672/rsa/system
2018-06-13 16:51:37,215 [                          main] INFO             Bootstrap|Service will use the deployment security-server
2018-06-13 16:51:38,483 [    Notify Handshake Completed] INFO              Security|Accepted new connection with CN=d4ee2ca7-b16a-48c0-8f14-7f14d1c4d6cf,OU=NetWitness Platform,O=RSA,L=Reston,ST=VA,C=US from using TLS_DHE_RSA_WITH_AES_128_GCM_SHA2
2018-06-13 16:51:39,275 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 599 more times)...
2018-06-13 16:51:45,289 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 598 more times)...
2018-06-13 16:51:51,302 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 597 more times)...
2018-06-13 16:51:57,313 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 596 more times)...
2018-06-13 16:52:03,322 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 595 more times)...
2018-06-13 16:52:09,335 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 594 more times)...
2018-06-13 16:52:15,347 [                          main] INFO  c.r.n.i.o.c.LaunchHelper|Task [Refresh Host] running (polling 593 more times)...
2018-06-13 16:52:21,367 [                          main] ERROR c.r.n.i.o.c.OrchestrationClient|Task [Refresh Host] stopped with errors!
2018-06-13 16:52:21,368 [                          main] ERROR c.r.n.i.o.c.OrchestrationApplication|Requested operation failed, aborting...
CauseA number of issues could cause this particular operation to fail. For instance, the files that exist in /etc/pki/nw/trust/import MUST be in .pem format. Anything else will cause the process to fail. You can review the following logs files on the Netwitness Admin Server to see which file(s) could be causing the problem:

  • /var/log/netwitness/config-management/chef-solo.log
  • /var/lib/netwitness/config-management/cache/chef-stacktrace.out   (if present)
Here is an example from the chef-stacktrace.out file:

>>>> Caused by Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of openssl pkcs12 -in /etc/pki/nw/trust/truststore.p12 -out /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -passin pass:changeit && openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -nokeys -name rootcastore.crt -in /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -certfile /etc/pki/nw/trust/import/rootcastore.crt.pem -out /etc/pki/nw/trust/truststore.p12 -passout pass:changeit && rm -f /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c && chown netwitness:nwpki /etc/pki/nw/trust/truststore.p12 && chmod 640 /etc/pki/nw/trust/truststore.p12 && echo '/etc/pki/nw/trust/import/rootcastore.crt.pem' >> /etc/pki/nw/trust/truststore.p12.idx ----
STDERR: 139904017254304:error:0D07207B:asn1 encoding routines:ASN1_get_object:header too long:asn1_lib.c:157:
---- End output of openssl pkcs12 -in /etc/pki/nw/trust/truststore.p12 -out /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -passin pass:changeit && openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -nokeys -name rootcastore.crt -in /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -certfile /etc/pki/nw/trust/import/rootcastore.crt.pem -out /etc/pki/nw/trust/truststore.p12 -passout pass:changeit && rm -f /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c && chown netwitness:nwpki /etc/pki/nw/trust/truststore.p12 && chmod 640 /etc/pki/nw/trust/truststore.p12 && echo '/etc/pki/nw/trust/import/rootcastore.crt.pem' >> /etc/pki/nw/trust/truststore.p12.idx ----
Ran openssl pkcs12 -in /etc/pki/nw/trust/truststore.p12 -out /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -passin pass:changeit && openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -nokeys -name rootcastore.crt -in /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c -certfile /etc/pki/nw/trust/import/rootcastore.crt.pem -out /etc/pki/nw/trust/truststore.p12 -passout pass:changeit && rm -f /tmp/openssl-trust.daf3e46f-38b9-4fc4-8d93-73e7ba45644c && chown netwitness:nwpki /etc/pki/nw/trust/truststore.p12 && chmod 640 /etc/pki/nw/trust/truststore.p12 && echo '/etc/pki/nw/trust/import/rootcastore.crt.pem' >> /etc/pki/nw/trust/truststore.p12.idx returned 1

From the above, we can extrapolate that it is having a problem with the /etc/pki/nw/trust/import/rootcastore.crt.pem file. 

Generally, this is a result of badly formatted .pem files for one reason or another. Here is an example of one such file:

+O3pTczC3wYau167ADb0lGppzHWTD1UN8S8SHduj0gHfrtTJKptsoWdyPYFEkeTo+bBSWyyez+WX-----END CERTIFICATE-----

The above file has two problems. If these files originated from a Microsoft Windows device, they may be filled with Windows endline characters ("^M") which violates the structure of the .pem file. This can also happen if you open a file in Windows before moving it over to the NetWitness Admin Server. These Windows endlines must be removed before the import can be completed successfully. In addition, the "------END CERTIFICATE-----" must be on its own line. You may need to open the file in vi to see them if they are not visible to you.
ResolutionYou can now open the file in vi to check/modify it's content

vi keystore.crt.pem

Delete the "^M" characters from the end of the lines. Ensure that the "------END CERTIFICATE-----" is on it's own line. I would also suggest you run the following in vi to ensure that we are only using Unix endlines at this point.

Your finished file will look similar to the following:


When you are done, we can set all endlines to be Unix just for sanity checking. Then, you can type ":wq" to exit vi and save the file.

:set ff=unix
WorkaroundIMPORTANT: Once you are finished applying the changes to the files, you MUST do the following to allow for orchestration-cli-client --update-admin-node to run again. When you look inside of /etc/pki/nw/trust after your failed run, you will see something like the following:

[root@nwadmin1 trust]# ls /etc/pki/nw/trust -alh
total 32K
drwxr-x---.  3 netwitness nwpki 4.0K Apr 16 03:18 .
drwxr-x---. 16 netwitness nwpki 4.0K Mar 26 22:56 ..
drwxr-x---.  2 netwitness nwpki    6 Feb  1 22:55 import
-rw-r-----.  1 netwitness nwpki 3.9K Feb  1 22:54 truststore.jks
-rw-r--r--.  1 root       root    99 Feb  1 22:54 truststore.jks.idx
-rw-r-----.  1 netwitness nwpki    0 Apr 16 03:18 truststore.p12
-rw-r--r--.  1 root       root    58 Apr 16 03:18 truststore.p12.idx
-rw-r-----.  1 netwitness nwpki 4.0K Nov  1  2017 truststore.pem
-rw-r--r--.  1 root       root    58 Nov  1  2017 truststore.pem.idx

You will notice that the truststore.p12 file is of size 0. This happens when a failed run occurs. You will need to move this and it's sister file out of this directory to allow for them to be recreated with the appropriate certs. No data is loss when you make this move:

mv /etc/pki/nw/trust/truststore.p12* /root

After this is done and you are sure you have made the necessary changes for the .pem files to be accepted, you may run the orchestration process again:

orchestration-cli-client --update-admin-node

If these steps did not help you and you are still running into the same error, please contact RSA Customer Support and reference this article for further assistance. When creating a case revolving around this issue, it would speed up the troubleshooting process if you also attach the following files to the case:

  • /var/log/netwitness/config-management/chef-solo.log
  • /var/lib/netwitness/config-management/cache/chef-stacktrace.out   (if present)