Web Tier status offline/Reinstall status changes to pending connection for RSA Authentication Manager 8.4

[#1]

Article Number

000039196

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.4.0, 8.4.0.7.0, 8.4.0.13.0, 8.3, 8.5, 8.6, 8.x
Platform: Linux
Platform (Other): Web Tier
O/S Version: SUSE Linux 11.4, RHEL 7.x on Web Tier

Issue

The RSA Authentication Manager Web Tier status is changed to offline, while some Web Tiers still work.
 

Image description

Other symptoms show in the AdminServer, biztier and console logs on RSA Authentication Manager, as shown in the log snippets below:

2020-08-01 17:54:33,032, [[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'], 
(WebTierConfigurationAdministrationImpl.java:367), 
trace.com.rsa.authmgr.internal.admin.webtier.impl.WebTierConfigurationAdministrationImpl, 
ERROR, <Primary.com>,,,,Fail to Pack Webtier Customization to latest versioncom.rsa.authmgr.internal.admin.webtier.WebtierConfigurationsPackageException: 
Fail to Pack Webtier Customization to latest version

Aug 1, 2020 5:22:35,436 PM EDT> <Notice> <Security> <'primary'> <biztier> <[ACTIVE] ExecuteThread: '3' for queue: 
'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-2215EA2996AC4262E80E> <6a0372a1-bc44-4226-81b9-4a0b61d65179-00000055> 
<1596316955436> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-090171>* 
<Loading the identity certificate and private key stored under the alias server_identity_key_webserver 
from the jks keystore file /opt/rsa/am/server/security/biztier-identity.jks.>*

Aug 1, 2020 5:22:35,436 PM EDT> <Notice> <Security> <'primary'> <biztier> <[ACTIVE] ExecuteThread: '3' for queue: 
'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-2215EA2996AC4262E80E> <6a0372a1-bc44-4226-81b9-4a0b61d65179-00000055> 
<1596316955436> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-090171>* 
<Loading the identity certificate and private key stored under the alias server_identity_key_webserver 
from the jks keystore file /opt/rsa/am/server/security/biztier-identity.jks.>*

2020-08-01 18:42:27,540, [[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'], (WebTierConfigurationAdministrationImpl.java:543), 
trace.com.rsa.authmgr.internal.admin.webtier.impl.WebTierConfigurationAdministrationImpl, INFO, <Primary.com>,,,,Exception in thread "main" : 
error running fixcrlf on file /opt/rsa/am/config/src/scripts/Config.groovy.orig

2020-08-01 18:42:27,552, [[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'], (WebTierConfigurationAdministrationImpl.java:543), 
trace.com.rsa.authmgr.internal.admin.webtier.impl.WebTierConfigurationAdministrationImpl, INFO, <Primary.com>,,,,
Caused by: java.io.FileNotFoundException: /opt/rsa/am/config/src/scripts/Config.groovy.orig (Permission denied)

Also, the Web Tier directory /opt/RSASecurity/RSAAuthenticationManagerWebTier/server does not exist. It is created during Web Tier update.

Cause

When following article 000037358 - Increase biztier and console heapsizes to address console memory allocation errors for RSA Authentication Manager 8.3 and higher, the user made a backup copy of /opt/rsa/am/config/src/scripts/config.groovy as the root user rather than as the rsaadmin user. A permissions issue on files in /opt/rsa/am/config/src/scripts/ prevents an update of the Web Tiers, and causes the Web Tiers to be offline or have a connection status of Pending.

Image description

The cause of the Web Tiers failing to update is that the file Config.groovy.orig file, which is owned by root, therefore, it cannot be read by rsaadmin.  Even though this is a backup file, it is still found in this /opt/rsa/am/config/src/scripts/ directory, and causes this particular problem.

Resolution

To correct the issue,

1. Elevate to the root user.
2. Delete or move the Config.groovy.orig file to a different directory path. 

mv Config.groovy.orig /tmp

3. Optionally, change ownership and group on the file to rsaadmin.

chown rsaadmin:rsaadmin Config.groovy.orig

​​​​​​
  Image description

Immediately after /opt/rsa/am/config/src/scripts/config.groovy.orig (owned by root, root) was removed from the RSA Authentication Manager primary server, all the Web Tiers started to change status to online, 
 

Image description

The /opt/RSASecurity/RSAAuthenticationManagerWebTier/server directory was created on Web Tiers:

Notes

Other problems that cause pending connection status in Web Tiers are
- blocked TCP ports 7036 or 7030 internally
@@@2021-07-14 11:38:41,396, [WrapperSimpleAppMain], (ConfigServiceUtils.java:82), trace.com.rsa.tool.webtierbootstrapper.utils.ConfigServiceUtils, INFO
, <server_name>,,,, [java] WLSTException: Error occurred while performing connect : Cannot connect via t3s or https. If using demo certs, verify that the -Dweblogic.security.TrustKeyStore=DemoTrust system property is set. : Failed to initialize JNDI context, tried 2 time or times to tally, the interval of each time is 0ms.
@@@2021-07-14 11:38:41,397, [WrapperSimpleAppMain], (ConfigServiceUtils.java:82), trace.com.rsa.tool.webtierbootstrapper.utils.ConfigServiceUtils, INFO
, <server_name>,,,, [java] t3s://<server_name>:7036: Destination 10.251.65.100, 7036 unreachable.; nested exception is:

- name resolution, Web Tier package name not spelled same as the Web Tier DNS name
systemd[1]: [/run/systemd/generator.late/rsabootstrapperservmgr.service:14] Failed to add dependency on +memorycontrol.service, ignoring: Invalid argument