000032408 - RSA Authentication Manager 8.1 webtier showing offline after hard shutdown. Error: System fingerprint encrypted key is missing and Failed to reload password database

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000032408
Applies ToRSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager Webtier
RSA Version/Condition: 8.1
IssueThe webtier server running under Red Hat Linux was shut down hard and when the system came back up the webtier services are no longer running.  When checking /opt/RSASecurity/RSAAUthenticationManagerWebtier/server/logs/AdminServer.log and AdminServerWrapper.log, the following error is seen:
Errors Logged are System-fingerprint encrypted key is missing and Failed to reload password database
CauseThis can happen if the systemfields.properties file is being written to when the webtier server is shut down hard vs. a normal shutdown. The file becomes corrupted and cannot be read when the services try to start again on reboot. Currently the file is being written to every two minutes by the bootstrapper service so there is a good chance this file can become corrupted if the webtier server is shut down hard (e.g. power is lost, or somebody unplugged the server while it was running). 
The
/opt/RSASecurity/RSAAUthenticationManagerWebtier/server/logs/AdminServer.log and /opt/RSASecurity/RSAAUthenticationManagerWebtier/server/logs/AdminServerWrapper.log on the webtier server show the following errors highlighted below:
In the Admin Server log:
####<Jan 5, 2016 3:19:49 PM EST> <Info> <Security> <rh81wt.vcloud.local> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default 
(self-tuning)'> <<WLS Kernel>> <> <> <1452025189310> <BEA-090511> <The following exception has occurred:
com.bea.common.engine.ServiceInitializationException: java.lang.RuntimeException: Failed to reload password database
at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:365)
at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:299)
at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:221)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1790)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:446)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:871)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1034)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.RuntimeException: Failed to reload password database
at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:401)
at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.<init>(IMSAuthenticatorDatabase.java:156)
at com.rsa.ims.wls.security.IMSCertificateAuthenticationProviderImpl.initialize(IMSCertificateAuthenticationProviderImpl.java:128)
at com.bea.common.security.internal.legacy.service.SecurityProviderImpl.init(SecurityProviderImpl.java:60)
at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:363)
at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:299)
at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:221)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1790)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:446)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:871)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1034)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: com.rsa.ims.security.keymanager.sys.MissingSystemKeysException: System fingerprint encrypted key is missing
at com.rsa.ims.security.lockbox.crypto.h.b(h.java:57)
at com.rsa.ims.security.lockbox.b.loadFields(b.java:119)
at com.rsa.ims.security.lockbox.h.loadFields(h.java:9)
at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.getLoader(IMSAuthenticatorDatabase.java:270)
at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:373)
at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.<init>(IMSAuthenticatorDatabase.java:156)
at com.rsa.ims.wls.security.IMSCertificateAuthenticationProviderImpl.initialize(IMSCertificateAuthenticationProviderImpl.java:128)
at com.bea.common.security.internal.legacy.service.SecurityProviderImpl.init(SecurityProviderImpl.java:60)
at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:363)
at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:299)
at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:221)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1790)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:446)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:871)
at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1034)
at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
at weblogic.security.SecurityService.start(SecurityService.java:148)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

In the AdminServerWrapper.log,
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | <Jan 5, 2016 3:19:49 PM EST> <Error> <Security> <BEA-090870> <The realm "rsa" failed to be loaded: weblogic.security.service.SecurityServiceException: 
com.bea.common.engine.ServiceInitializationException: java.lang.RuntimeException: Failed to reload password database.

INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: java.lang.RuntimeException:
Failed to reload password database

INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:341)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:220)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1789)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:443)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  Truncated. see log file for complete stacktrace
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | Caused By: com.bea.common.engine.ServiceInitializationException: java.lang.RuntimeException: Failed to reload password database
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:365)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  Truncated. see log file for complete stacktrace
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | Caused By: java.lang.RuntimeException: Failed to reload password database
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:401)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.<init>(IMSAuthenticatorDatabase.java:156)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.wls.security.IMSCertificateAuthenticationProviderImpl.initialize(IMSCertificateAuthenticationProviderImpl.java:128)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.bea.common.security.internal.legacy.service.SecurityProviderImpl.init(SecurityProviderImpl.java:60)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:363)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  Truncated. see log file for complete stacktrace
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | Caused By: com.rsa.ims.security.keymanager.sys.MissingSystemKeysException: System fingerprint encrypted key is missing
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.security.lockbox.crypto.h.b(h.java:57)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.security.lockbox.b.loadFields(b.java:119)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.security.lockbox.h.loadFields(h.java:9)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.getLoader(IMSAuthenticatorDatabase.java:270)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |  at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:373)

 
Resolution
Solution 1
Uninstall the webtier then reinstall it to resolve the issue.  If you do not want to uninstall then reinstall please refer to the second solution below. 
Solution 2
1.  SSH to the primary Authentication Manager server as the rsaadmin user then run the following commands:

cd /opt/rsa/am/utils/etc/
ls -alh systemfields.properties

-rw------- 1 rsaadmin rsaadmin 11K Jan 15 16:22 systemfields.properties
cp systemfields.properties /tmp/

2.  Use WinSCP or another program to connect to the primary Authentication Manager server as the rsaadmin user.
3.  Download the systemfields.properties file you just copied in the /tmp folder to your local PC then disconnect from WinSCP.
4.  SSH into the Linux or Windows webtier server then sudo to the root user (Linux) or administrator (Windows). Use the same password you used for rsaadmin when entering the command below:

sudo su -

5.  Upload the systemfields.properties file to /tmp on the Linux webtier servers.
6.  Via SSH on the webtier, type the following commands (make adjustments if your install location is different from the default):

cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/utils/etc
mv systemfields.properties systemfields.properties.orig
cp /tmp/systemfields.properties ./

7.  Now that the file has been copied, set the correct permissions on the file.  First run the following command to check permissions on both of the systemfields.properties files:
ls -alh systemfields.properties*
-rw-------. 1 root root 11K Jan 21 08:42 systemfields.properties
-rw-------. 1 webtier webtier   0 Jan  4 05:36 systemfields.properties.orig

8.  In this case, when the webtier was installed, the user picked during the install was called webtier and if we look at the file permissions for the original systemfields.properties file we can see the owner and group are webtier. We need to make sure permissions on the new systemfields.properties file match the one we renamed. The user and group will be different than what is in this example so use that vs what you see here.
chmod 600 systemfields.properties
chown webtier:webtier systemfields.properties

The above commands set the new file permissions and owner/group to match the original file.
9.  Next, run the following command to update the systemfields.properties file for the webtier server OS and hardware.  Until now we have been doing everything as root, you need to switch to the webtier user to run this last command. Again the user you picked during install will be different than the example user. 

sudo su - webtier
cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/utils/
./rsautil manage-sec

Invalid argument. Multiple CLU's starting with 'manage-sec' found:
    manage-secrets-8.1.1.10.0
    manage-secrets-8.1.1.2.0
    manage-secrets-8.1.1.9.0

When you run the above command you will see an error, take note of the patch level of your webtier in the Operations Console.  In the example below, we are on SP1 P10 so you want to use the same manage-secrets version via the following command.
./rsautil manage-secrets-8.1.1.10.0 -a recover
Please enter OC Administrator username: <enter name of Operations Console admin user>
Please enter OC Administrator password: <enter password for Operations Console admin user>
Machine fingerprint restored successfully.

NOTE:  The above command requires the Operations Console username and password from the primary Authentication Manager server from which you copied the systemfields.properties file.
If all commands have been run without issue, you can now start the webtier services via the following command:
cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/
./rsaserv start

You will get a green status, then in about five minutes the server will show up as online again in the Operations Console.

Attachments

    Outcomes