The RSA Authentication Manager Web Tier status is changed to offline, while some Web Tiers still work.
Image description
Other symptoms show in the AdminServer, biztier and console logs on RSA Authentication Manager, as shown in the log snippets below:
2020-08-01 17:54:33,032, [[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'],
(WebTierConfigurationAdministrationImpl.java:367),
trace.com.rsa.authmgr.internal.admin.webtier.impl.WebTierConfigurationAdministrationImpl,
ERROR, <Primary.com>,,,,Fail to Pack Webtier Customization to latest versioncom.rsa.authmgr.internal.admin.webtier.WebtierConfigurationsPackageException:
Fail to Pack Webtier Customization to latest version
Aug 1, 2020 5:22:35,436 PM EDT> <Notice> <Security> <'primary'> <biztier> <[ACTIVE] ExecuteThread: '3' for queue:
'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-2215EA2996AC4262E80E> <6a0372a1-bc44-4226-81b9-4a0b61d65179-00000055>
<1596316955436> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-090171>*
<Loading the identity certificate and private key stored under the alias server_identity_key_webserver
from the jks keystore file /opt/rsa/am/server/security/biztier-identity.jks.>*
Aug 1, 2020 5:22:35,436 PM EDT> <Notice> <Security> <'primary'> <biztier> <[ACTIVE] ExecuteThread: '3' for queue:
'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <BEA1-2215EA2996AC4262E80E> <6a0372a1-bc44-4226-81b9-4a0b61d65179-00000055>
<1596316955436> <[severity-value: 32] [rid: 0] [partition-id: 0] [partition-name: DOMAIN] > <BEA-090171>*
<Loading the identity certificate and private key stored under the alias server_identity_key_webserver
from the jks keystore file /opt/rsa/am/server/security/biztier-identity.jks.>*
2020-08-01 18:42:27,540, [[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'], (WebTierConfigurationAdministrationImpl.java:543),
trace.com.rsa.authmgr.internal.admin.webtier.impl.WebTierConfigurationAdministrationImpl, INFO, <Primary.com>,,,,Exception in thread "main" :
error running fixcrlf on file /opt/rsa/am/config/src/scripts/Config.groovy.orig
2020-08-01 18:42:27,552, [[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'], (WebTierConfigurationAdministrationImpl.java:543),
trace.com.rsa.authmgr.internal.admin.webtier.impl.WebTierConfigurationAdministrationImpl, INFO, <Primary.com>,,,,
Caused by: java.io.FileNotFoundException: /opt/rsa/am/config/src/scripts/Config.groovy.orig (Permission denied)
Also, the Web Tier directory /opt/RSASecurity/RSAAuthenticationManagerWebTier/server does not exist. It is created during Web Tier update.
When following article
000037358 - Increase biztier and console heapsizes to address console memory allocation errors for RSA Authentication Manager 8.3 and higher, the user made a backup copy of /opt/rsa/am/config/src/scripts/config.groovy as the root user rather than as the rsaadmin user. A permissions issue on files in /opt/rsa/am/config/src/scripts/ prevents an update of the Web Tiers, and causes the Web Tiers to be offline or have a connection status of Pending.
Image descriptionThe cause of the Web Tiers failing to update is that the file Config.groovy.orig file, which is owned by root, therefore, it cannot be read by rsaadmin. Even though this is a backup file, it is still found in this /opt/rsa/am/config/src/scripts/ directory, and causes this particular problem.
To correct the issue,
- Elevate to the root user.
- Delete or move the Config.groovy.orig file to a different directory path.
mv Config.groovy.orig /tmp
- Optionally, change ownership and group on the file to rsaadmin.
chown rsaadmin:rsaadmin Config.groovy.orig
Image descriptionImmediately after /opt/rsa/am/config/src/scripts/config.groovy.orig (owned by root, root) was removed from the RSA Authentication Manager primary server, all the Web Tiers started to change status to online,
Image descriptionThe /opt/RSASecurity/RSAAuthenticationManagerWebTier/server directory was created on Web Tiers:
Other problems that cause pending connection status in Web Tiers are
-
blocked TCP ports 7036 or 7030 internally@@@2021-07-14 11:38:41,396, [WrapperSimpleAppMain], (ConfigServiceUtils.java:82), trace.com.rsa.tool.webtierbootstrapper.utils.ConfigServiceUtils, INFO
, <server_name>
,,,, [java] WLSTException: Error occurred while performing connect : Cannot connect via t3s or https. If using demo certs, verify that the -Dweblogic.security.TrustKeyStore=DemoTrust system property is set. : Failed to initialize JNDI context, tried 2 time or times to tally, the interval of each time is 0ms.
@@@2021-07-14 11:38:41,397, [WrapperSimpleAppMain], (ConfigServiceUtils.java:82), trace.com.rsa.tool.webtierbootstrapper.utils.ConfigServiceUtils, INFO
, <server_name>
,,,, [java] t3s://<server_name>:7036: Destination 10.251.65.100, 7036 unreachable.; nested exception is:-
name resolution, Web Tier package name not spelled same as the Web Tier DNS namesystemd[1]: [/run/systemd/generator.late/rsabootstrapperservmgr.service:14] Failed to add dependency on +memorycontrol.service, ignoring: Invalid argument