|Applies To||RSA Product Set: SecurID|
RSA Product/Service Type: Authentication Manager Webtier
RSA Version/Condition: 8.1
|Issue||The webtier server running under Red Hat Linux was shut down hard and when the system came back up the webtier services are no longer running. When checking /opt/RSASecurity/RSAAUthenticationManagerWebtier/server/logs/AdminServer.log and AdminServerWrapper.log, the following error is seen:|
Errors Logged are System-fingerprint encrypted key is missing and Failed to reload password database
|Cause||This can happen if the systemfields.properties file is being written to when the webtier server is shut down hard vs. a normal shutdown. The file becomes corrupted and cannot be read when the services try to start again on reboot. Currently the file is being written to every two minutes by the bootstrapper service so there is a good chance this file can become corrupted if the webtier server is shut down hard (e.g. power is lost, or somebody unplugged the server while it was running). |
The /opt/RSASecurity/RSAAUthenticationManagerWebtier/server/logs/AdminServer.log and /opt/RSASecurity/RSAAUthenticationManagerWebtier/server/logs/AdminServerWrapper.log on the webtier server show the following errors highlighted below:
In the Admin Server log:
####<Jan 5, 2016 3:19:49 PM EST> <Info> <Security> <rh81wt.vcloud.local> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default
In the AdminServerWrapper.log,
INFO | jvm 1 | main | 2016/01/05 15:19:49 | <Jan 5, 2016 3:19:49 PM EST> <Error> <Security> <BEA-090870> <The realm "rsa" failed to be loaded: weblogic.security.service.SecurityServiceException:
Uninstall the webtier then reinstall it to resolve the issue. If you do not want to uninstall then reinstall please refer to the second solution below.
1. SSH to the primary Authentication Manager server as the rsaadmin user then run the following commands:
2. Use WinSCP or another program to connect to the primary Authentication Manager server as the rsaadmin user.
3. Download the systemfields.properties file you just copied in the /tmp folder to your local PC then disconnect from WinSCP.
4. SSH into the Linux or Windows webtier server then sudo to the root user (Linux) or administrator (Windows). Use the same password you used for rsaadmin when entering the command below:
sudo su -
5. Upload the systemfields.properties file to /tmp on the Linux webtier servers.
6. Via SSH on the webtier, type the following commands (make adjustments if your install location is different from the default):
7. Now that the file has been copied, set the correct permissions on the file. First run the following command to check permissions on both of the systemfields.properties files:
ls -alh systemfields.properties*
8. In this case, when the webtier was installed, the user picked during the install was called webtier and if we look at the file permissions for the original systemfields.properties file we can see the owner and group are webtier. We need to make sure permissions on the new systemfields.properties file match the one we renamed. The user and group will be different than what is in this example so use that vs what you see here.
chmod 600 systemfields.properties
The above commands set the new file permissions and owner/group to match the original file.
9. Next, run the following command to update the systemfields.properties file for the webtier server OS and hardware. Until now we have been doing everything as root, you need to switch to the webtier user to run this last command. Again the user you picked during install will be different than the example user.
sudo su - webtier
When you run the above command you will see an error, take note of the patch level of your webtier in the Operations Console. In the example below, we are on SP1 P10 so you want to use the same manage-secrets version via the following command.
./rsautil manage-secrets-18.104.22.168.0 -a recover
NOTE: The above command requires the Operations Console username and password from the primary Authentication Manager server from which you copied the systemfields.properties file.
If all commands have been run without issue, you can now start the webtier services via the following command:
You will get a green status, then in about five minutes the server will show up as online again in the Operations Console.