Webtier showing offline after hard shutdown. Error: System fingerprint encrypted key is missing and Failed to reload password database in RSA Authentication Manager 8.x
2 years ago
Originally Published: 2016-01-21
Article Number
000049834
Applies To
RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.x
Issue
The webtier server running under Red Hat Linux was shut down hard and when the system came back up the webtier services are no longer running.  When checking the /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/logs/AdminServer.log and the AdminServerWrapper.log, the following error is seen:

Errors Logged are System-fingerprint encrypted key is missing and Failed to reload password database

Cause
This can happen if the systemfields.properties file is being written to when the webtier server is shut down hard, as opposed to a normal shutdown. The file becomes corrupted and cannot be read when the services try to start again on reboot. Currently the file is being written to every two minutes by the bootstrapper service so there is a good chance this file can become corrupted if the webtier server is shut down hard (e.g. power is lost, or somebody unplugged the server while it was running). 

The /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/logs/AdminServer.log and /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/logs/AdminServerWrapper.log on the webtier server show the following errors highlighted below:

In the Admin Server log: 
####<Jan 5, 2016 3:19:49 PM EST> <Info> <Security> <rh81wt.vcloud.local> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default 
(self-tuning)'> <<WLS Kernel>> <> <> <1452025189310> <BEA-090511> <The following exception has occurred: 

com.bea.common.engine.ServiceInitializationException: 
java.lang.RuntimeException: 
Failed to reload password database
	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:365)
	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
	at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
	at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
	at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
	at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:299)
	at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:221)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1790)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:446)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:871)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1034)
	at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
	at weblogic.security.SecurityService.start(SecurityService.java:148)
	at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: java.lang.RuntimeException: 
Failed to reload password database
	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:401)
	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.<init>(IMSAuthenticatorDatabase.java:156)
	at com.rsa.ims.wls.security.IMSCertificateAuthenticationProviderImpl.initialize(IMSCertificateAuthenticationProviderImpl.java:128)
	at com.bea.common.security.internal.legacy.service.SecurityProviderImpl.init(SecurityProviderImpl.java:60)
	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:363)
	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
	at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
	at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
	at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
	at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:299)
	at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:221)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1790)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:446)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:871)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1034)
	at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
	at weblogic.security.SecurityService.start(SecurityService.java:148)
	at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused By: com.rsa.ims.security.keymanager.sys.MissingSystemKeysException: System fingerprint encrypted key is missing
	at com.rsa.ims.security.lockbox.crypto.h.b(h.java:57)
	at com.rsa.ims.security.lockbox.b.loadFields(b.java:119)
	at com.rsa.ims.security.lockbox.h.loadFields(h.java:9)
	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.getLoader(IMSAuthenticatorDatabase.java:270)
	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:373)
	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.<init>(IMSAuthenticatorDatabase.java:156)
	at com.rsa.ims.wls.security.IMSCertificateAuthenticationProviderImpl.initialize(IMSCertificateAuthenticationProviderImpl.java:128)
	at com.bea.common.security.internal.legacy.service.SecurityProviderImpl.init(SecurityProviderImpl.java:60)
	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:363)
	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
	at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
	at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
	at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
	at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:299)
	at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:221)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1790)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:446)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealms(CommonSecurityServiceManagerDelegateImpl.java:871)
	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1034)
	at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:894)
	at weblogic.security.SecurityService.start(SecurityService.java:148)
	at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

In the AdminServerWrapper.log, 
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | <Jan 5, 2016 3:19:49 PM EST> <Error> <Security> <BEA-090870> <The realm "rsa" failed to be loaded: 
weblogic.security.service.SecurityServiceException: 
com.bea.common.engine.ServiceInitializationException: java.lang.RuntimeException: Failed to reload password database.
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | weblogic.security.service.SecurityServiceException: com.bea.common.engine.ServiceInitializationException: 
java.lang.RuntimeException: 
Failed to reload password database
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at weblogic.security.service.CSSWLSDelegateImpl.initializeServiceEngine(CSSWLSDelegateImpl.java:341)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at weblogic.security.service.CSSWLSDelegateImpl.initialize(CSSWLSDelegateImpl.java:220)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.InitializeServiceEngine(CommonSecurityServiceManagerDelegateImpl.java:1789)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initializeRealm(CommonSecurityServiceManagerDelegateImpl.java:443)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadRealm(CommonSecurityServiceManagerDelegateImpl.java:841)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | 	Truncated. see log file for complete stacktrace
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | Caused By: com.bea.common.engine.ServiceInitializationException: java.lang.RuntimeException: Failed to reload password database
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:365)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:315)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.bea.common.engine.internal.ServiceEngineImpl.lookupService(ServiceEngineImpl.java:257)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.bea.common.engine.internal.ServicesImpl.getService(ServicesImpl.java:72)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at weblogic.security.service.internal.WLSIdentityServiceImpl.initialize(WLSIdentityServiceImpl.java:46)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | 	Truncated. see log file for complete stacktrace
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | 
Caused By: java.lang.RuntimeException: Failed to reload password database
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:401)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.<init>(IMSAuthenticatorDatabase.java:156)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.wls.security.IMSCertificateAuthenticationProviderImpl.initialize(IMSCertificateAuthenticationProviderImpl.java:128)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.bea.common.security.internal.legacy.service.SecurityProviderImpl.init(SecurityProviderImpl.java:60)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.bea.common.engine.internal.ServiceEngineImpl.findOrStartService(ServiceEngineImpl.java:363)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | 	Truncated. see log file for complete stacktrace
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 | Caused By: com.rsa.ims.security.keymanager.sys.MissingSystemKeysException: 
System fingerprint encrypted key is missing
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.security.lockbox.crypto.h.b(h.java:57)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.security.lockbox.b.loadFields(b.java:119)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.security.lockbox.h.loadFields(h.java:9)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.getLoader(IMSAuthenticatorDatabase.java:270)
INFO   | jvm 1    | main    | 2016/01/05 15:19:49 |
 	at com.rsa.ims.common.security.utils.IMSAuthenticatorDatabase.reloadProps(IMSAuthenticatorDatabase.java:373)
Resolution

Solution 1

Uninstall the webtier then reinstall it to resolve the issue.  If you do not want to uninstall and reinstall, please refer to the second solution below. 


Solution 2

  1. SSH to the primary RSA Authentication Manager server as the rsaadmin user then run the following commands:
login as: rsaadmin
Using keyboard-interactive authentication.
Password: <enter operating system password>
Last login: Fri Jan 10 12:33:27 2020 from jumphost.vcloud.local
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am82p:~> cd /opt/rsa/am/utils/etc/
rsaadmin@am82p:~> ls -alh systemfields.properties
-rw------- 1 rsaadmin rsaadmin 11K Jan 15 16:22 systemfields.properties
rsaadmin@am82p:~> cp systemfields.properties /tmp/
  1. Use WinSCP or another program to connect to the primary RSA Authentication Manager server as the rsaadmin user.
  2. Download the systemfields.properties file you just copied to /tmp to your local PC then disconnect from WinSCP.
  3. SSH into the Linux or Windows webtier server.
  4. Run sudo to the root user (Linux) or administrator (Windows). Use the same password you used for rsaadmin when entering the command below:
rsaadmin@am82p:~> sudo su -
  1. Upload the systemfields.properties file to /tmp on the Linux webtier servers.
  2. Via SSH on the webtier, type the following commands (make adjustments if your install location is different from the default):
cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/utils/etc
mv systemfields.properties systemfields.properties.orig
cp /tmp/systemfields.properties ./
  1. Now that the file has been copied, set the correct permissions on the file.  First run the following command to check permissions on both of the systemfields.properties files:
ls -alh systemfields.properties*
-rw-------. 1 root root 11K Jan 21 08:42 systemfields.properties
-rw-------. 1 webtier webtier   0 Jan  4 05:36 systemfields.properties.orig
  1. In this case, when the webtier was installed, the user defined during the install was called webtier and if we look at the file permissions for the original systemfields.properties file we can see the owner and group are both webtier. We need to make sure permissions on the new systemfields.properties file match the one we renamed. The user and group will be different than what is in this example, so use that as opposed to what is shown here. The commands below set the new file permissions and owner/group to match the original file.
chmod 600 systemfields.properties
chown webtier:webtier systemfields.properties
  1. Run the following command to update the systemfields.properties file for the webtier server OS and hardware.  Until now we have been doing everything as root, but you need to switch to the webtier user to run this last command. Again the user you picked during install will be different than the example user. 
sudo su - webtier
cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/utils/
./rsautil manage-sec
Invalid argument. Multiple CLU's starting with 'manage-sec' found:
    manage-secrets-8.1.1.10.0
    manage-secrets-8.1.1.2.0
    manage-secrets-8.1.1.9.0
When you run the above command you will see an error, take note of the patch level of your webtier in the Operations Console.  In the example below, we are on SP1 patch 10, so you want to use the same manage-secrets version via the following command. 
./rsautil manage-secrets-8.1.1.10.0 -a recover
Please enter OC Administrator username: <enter name of Operations Console admin user>
Please enter OC Administrator password: <enter password for Operations Console admin user>
Machine fingerprint restored successfully.
The above command requires the Operations Console username and password from the primary Authentication Manager server from which you copied the systemfields.properties file.

If all commands have been run without issue, you can now start the webtier services via the following command: 
cd /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/
./rsaserv start
You will get a green status, then in about five minutes the server will show up as online again in the Operations Console.

Related Articles

Trending Articles

Don't see what you're looking for?