Troubleshooting Web Tier deployments on Red Hat Enterprise Linux for RSA Authentication Manager 8.1

[#1]

Article Number

000031585

Applies To

RSA Product Set: SecurID
RSA Product/Service Type:  Authentication Manager
RSA Version/Condition:  8.1 SP1

Issue

The RSA Authentication Manager 8.1 Web Tier software has been deployed and the Web Tier is reporting the following message in the Operations Console:

Offline, Reinstall Required

Resolution

For a deployed Authentication Manager Web Tier, the general rule of thumb is that the fully-qualified hostname of the Authentication Manager instances and virtual hostname defined in the deployment are resolvable via a Domain Name Service (or at the very least a local host file on the operating system hosting the Web Tier software). The Authentication Manager instances in the deployment must also be able to resolve the Web Tier hostname,as defined in the Operations Console under Deployment Configuration > Web-Tier Deployments > Manage Existing.

Where RSA Authentication Manager 8.1 Service Pack 1 Patch 2 or later is used to update the Authentication Manager instances in the deployment, the RSA Authentication Manager 8.1 Service Pack 1 Patch 2 Web Tier software must be installed for the Web Tier deployment. 
 

This involves uninstalling the existing Web Tier software and installing the new RSA Authentication Manager 8.1 Service Pack 1 Patch 2 web software.

Troubleshooting a Web Tier installation (on a supported Red Hat Enterprise Linux server)

Here is an example of the message displayed for a Web Tier where it is stating it is offline and a reinstall is required:
 

Image description

The goal is to get the Web Tier online, as in the screen shot below. 
 

The Update button is where a custom logo was added and the Web Tier is waiting for the change to be pushed to the Web Tier deployment.

 

Image description

Troubleshooting Steps

1. Use the Operations Console to verify name resolution of the Authentication Manager hostname(s), Web Tier hostname(s) and virtual hostname.
   1. Logon to the Operations Console with an administrative account.
   2. Navigate to Administration > Network > Network Tools.
   3. For Select Command, select the Verify Name Resolution command.
   4. Enter either an Authentication Manager hostname, Web Tier hostname or virtual hostname.
   5. Click Run Command.

​Image description

2. Logon to the Red Hat Enterprise Linux operating system with the user account specified when installing the RSA Authentication Manager 8.1 Web Tier software (in these examples, rsaadmin is the Unix account for the Web Tier).
3. Check that the hostname in /etc/sysconfig/network matches the output from the hostname command.  For example,

[rsaadmin@webtierRH6 ~]$ hostname
webtierRH6.vcloud.local
[rsaadmin@webtierRH6 ~]$ cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=webtierRH6.vcloud.local
[rsaadmin@webtierRH6 ~]$​ ​

4. Check that port 7022 is open to the Authentication Manager instances using the command openssl s_client –connect <Authentication_Manager_hostname>:7022, substituting <Authentication_Manager_hostname> with the fully qualified hostname of the Authentication Manager instance.  

Using the fully qualified hostnames of the Authentication Manager server and Web Tier with the openssl command verifies name resolution.  Below is an example where a connection is made to port 7022 and a certificate is returned with the common name (CN) of the Authentication Manager instance:

[rsaadmin@webtierRH6 ~]# openssl s_client -connect am81p.vcloud.private:7022
CONNECTED(00000003)
depth=1 CN = RSA root CA for am81p.vcloud.private, serialNumber = 81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
verify error:num=19:self signed certificate in certificate chain
verify return:0

139939655591752:error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate:s3_pkt.c:1256:SSL alert number 42
139939655591752:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/CN=am81p.vcloud.private/serialNumber=7df7b8d5f50e4dbf77c12bfae19172ed957c4f2e544c6553e65fc8733bf61b9f
   i:/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
1 s:/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
   i:/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDfjCCAmagAwIBAgIQizOPOB70ObvHBrpVJJ00eDANBgkqhkiG9w0BAQUFADB6
MS0wKwYDVQQDDCRSU0Egcm9vdCBDQSBmb3IgYW04MXAudmNsb3VkLnByaXZhdGUx
STBHBgNVBAUTQDgxZTJmY2FjOGFlM2FhNDQyNzY3MGU2NTNmMGU4NDIxNGViOTc0
ZjllNjgxMGJhYzgwYTljOGI5ZmNmMTcwOTEwHhcNMTUwOTAxMDcxMDUzWhcNMzUw
OTAyMDcxMDUzWjBqMR0wGwYDVQQDDBRhbTgxcC52Y2xvdWQucHJpdmF0ZTFJMEcG
A1UEBRNAN2RmN2I4ZDVmNTBlNGRiZjc3YzEyYmZhZTE5MTcyZWQ5NTdjNGYyZTU0
NGM2NTUzZTY1ZmM4NzMzYmY2MWI5ZjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAOb+f36bczN7xRnkIXkjvq6SRsKPgFMHg24Lpp4MAe7DwLXjGR7jYaly
d1KKXfzUV+qXhiytixMVBFnTHuHO61aCSXCJeIPOeqvb4FifK+LKPTUEo3SBcaum
xf2fQr7wVTDRKu8LSZtvD7JPwf+eUU97duKV9i0RtyDzdN8qwAmJ42C+qJkhzWvE
OIUosDWq8iM6ERvg80kDHZ0VD15yhLJIJ58O3vVz7161G5Vks+uXP/OMxmkEG9Rj
dm0nc2ptmZ4LHnUMkeB3rGBAaOQmGm/6bhwqbbWuFfYdOPo7i5H+xAfV55yP0kFa
Z+P76ovcX+lPqskETTfXbJ84/UmEOtcCAwEAAaMQMA4wDAYDVR0TAQH/BAIwADAN
BgkqhkiG9w0BAQUFAAOCAQEAgZQ9hOrNx/C4Mq8F1t3x/VaTLHIjJCntDa/5oa6T
LzUz9PF5zsIFvIC8qi/WgZuDCIE+ztUj4+wBLohqB9JYabLYWUVMTk4EkgN3G9Em
o580rfLOacqM7JEZt2wGiea9S6FaWjJtDaRC8GWoU1M3iFWqm1EyrMc+i000wg8x
DhnyizHhncuIv5VVCxsezsAf1EuhQJszzV4hCDErM4O0dgQw85lex9m+y6xnUKBn
uP9NI1PZ0xTRmO0rCEVUFx1+SgynuEJLBN3z3JBRtQPzDS9NpCFacSdrNEKp2FCb
U9vGxSkLqosB/wGjSu0LV75sw3pSb8ycYJ4c1xYmYnbMsQ==
-----END CERTIFICATE-----
subject=/CN=am81p.vcloud.private/serialNumber=7df7b8d5f50e4dbf77c12bfae19172ed957c4f2e544c6553e65fc8733bf61b9f
issuer=/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
---
Acceptable client certificate CA names
/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
---
SSL handshake has read 2850 bytes and written 370 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES256-SHA256
    Session-ID: 562D60E13C1C69BD4A7A4E0F6A7E72DD6BFBA15B5253412053534C4A20202020
    Session-ID-ctx:
    Master-Key: B81A873D380177C28C13970E1D5C8FEABC38C7597A276CF140A3673FA638D023F8DC782BFDB3F5F0A4B98E40C825436E
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1445814472
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
[rsaadmin@webtierRH6 ~]#

5. Check that port 443 is open to the Web Tier using the command openssl s_client –connect [wt_hostname]:433, (substitute [WT_hostname] with the fully qualified hostname of the Web Tier).  

• The following is an example where port 443 is not available because the Web Tier services were not running:

[rsaadmin@webtierRH6 ~]# openssl s_client -connect webtierrh6.vcloud.local:443
socket: Connection refused
connect:errno=111
[rsaadmin@webtierRH6 ~]#

• Here is an example of a connection that can be made to port 443 and the common name (CN) of the certificate contains the virtual hostname:

[rsaadmin@webtierRH6 ~]# openssl s_client -connect webtierrh6.vcloud.local:443
socket: Connection refused
connect:errno=111
[rsaadmin@webtierRH6 ~]#

• Another example of a connection made to port 443 and the Common Name (CN) of the certificate contains the virtual hostname:

[rsaadmin@webtierRH6 ~]$ openssl s_client -connect webtierrh6.vcloud.local:443
CONNECTED(00000003)
depth=1 CN = RSA root CA for am81p.vcloud.private, serialNumber = 81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
verify error:num=19:self signed certificate in certificate chain
verify return:0
140662454237000:error:100AE081:elliptic curve routines:EC_GROUP_new_by_curve_name:unknown group:ec_curve.c:316:
140662454237000:error:1408D010:SSL routines:SSL3_GET_KEY_EXCHANGE:EC lib:s3_clnt.c:1641:
---
Certificate chain
0 s:/CN=selfservice.company.com/serialNumber=7da369177dd998d5034e0c7e05af2d1c2136daf0677e2b0d264142d4fb66fa05
   i:/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
1 s:/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
   i:/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDgTCCAmmgAwIBAgIQDQIWPYO12DxKxVMHGG2AdDANBgkqhkiG9w0BAQUFADB6
MS0wKwYDVQQDDCRSU0Egcm9vdCBDQSBmb3IgYW04MXAudmNsb3VkLnByaXZhdGUx
STBHBgNVBAUTQDgxZTJmY2FjOGFlM2FhNDQyNzY3MGU2NTNmMGU4NDIxNGViOTc0
ZjllNjgxMGJhYzgwYTljOGI5ZmNmMTcwOTEwHhcNMTUwOTA2MDQzNDE5WhcNMjUw
OTA3MDQzNDE5WjBtMSAwHgYDVQQDDBdzZWxmc2VydmljZS5jb21wYW55LmNvbTFJ
MEcGA1UEBRNAN2RhMzY5MTc3ZGQ5OThkNTAzNGUwYzdlMDVhZjJkMWMyMTM2ZGFm
MDY3N2UyYjBkMjY0MTQyZDRmYjY2ZmEwNTCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAMThBYFh5OurINaldszXad8b2rMZxPmbiSv8wAyAzpBRUkMnmYt6
GUwZMqHMQrJag1rrZpJjiy0sI50H50q+WkIg9E1O64NSBu7OMX2CdkKrSSvcR8Kx
f4+FKg9FudvwZQD6F+c7h8gdWx3WnYM5ycnzeqEJGRL9Bk4b5Rvif+sPn/903tyK
k8sLC+QbNBXY0tUq/WQ6fj4HIhhmBzbJxNyye31f4NRo/KLSSCGiSu7EgkWDZiqU
PlDbJy90JiioUj8TXBnlyTM91AchzUAL3BUvKUOKmV/VZB2aWJ7LI+W1u/R+1wUS
HAEhwd8u4E3MQHeiUw4/eCmGPyK0ZWp53JUCAwEAAaMQMA4wDAYDVR0TAQH/BAIw
ADANBgkqhkiG9w0BAQUFAAOCAQEAdjkCdHDFhPpt8M0asHUqMoqpyr/SQhSgSiRB
JMmdHYi/YOy0GBu0boTHa5uy9qMoqK4ZXL1yppigPpP3gO9UBETtH0DU2rbY7Q32
ou+OhuQcPpiM8ymESm9wojICZjnXX2RTNOPOsyeNmJCi/3XVTAF5S/rb6+6HFN2s
UuPI7r8az/CgWkevJ9zH0bBriE+zX9B4ieWMm6izOhZHVCvpk0dJkMsiVXXj5bT6
RzlGzVI6k1cMPvg5S8S6DcQDlPgrTuzffLFmOV3vz8zk1b6pIZjufjV4SOFJeR/r
gQqXu7eTW2suaXn99SmMwO3dVBMmqsch1qRtKWKGtx8B2ONlVQ==
-----END CERTIFICATE-----
subject=/CN=selfservice.company.com/serialNumber=7da369177dd998d5034e0c7e05af2d1c2136daf0677e2b0d264142d4fb66fa05
issuer=/CN=RSA root CA for am81p.vcloud.private/serialNumber=81e2fcac8ae3aa4427670e653f0e84214eb974f9e6810bac80a9c8b9fcf17091
---
No client certificate CA names sent
---
SSL handshake has read 2343 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 562D6B7D4A301DFE1DA88DB044A0A8F758E0ABAB5253412053534C4A20202020
    Session-ID-ctx:
    Master-Key:
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1445817213
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---
[rsaadmin@webtierRH6 ~]$

7. Check the status of the RSA Web Tier Bootstrapper Server.  By default, the Web Tier software is installed in /opt/RSASecurity/RSAAuthenticationManagerWebtier/.  In the following steps this will be referred to as [WT_Home].
   
   Check the status of the RSA Webtier Bootstrapper Server with the command [WT_Home]/webtierBootstrapper/server/rsaserv status.  For example, 

[rsaadmin@webtierRH6 ~]# /opt/RSASecurity/RSAAuthenticationManagerWebtier/webtierBootstrapper/server/rsaserv status
Running as rsaadmin...
RSA Webtier Bootstrapper Server                            [RUNNING]

[rsaadmin@webtierRH6 ~]#

8. Should this service be shutdown, start the RSA Webtier Bootstrapper Server with the followingcommand:

​[rsaadmin@webtierRH6 ~]# webtierBootstrapper/server/rsaserv start

9. Should the RSA Webtier Bootstrapper Server fail to start then look into the [WT_Home]/ webtierBootstrapper/n.n.n.n.n/logs/WebtierBootstrapperServerWrapper.log file (where n.n.n.n.n is the version number relating to the version of web tier software (e. g., 8.1.1.2.0).

An example of a typical hostname lookup error is as follows:

INFO   | jvm 1    | main    | 2015/10/26 10:26:04 | ERROR: Failed to get local host name!
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 | WrapperSimpleApp: Encountered an error running main: java.lang.Error: Failure to initialize UniqueIDGenerator
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 | java.lang.Error: Failure to initialize UniqueIDGenerator
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.common.UniqueIDGenerator.<clinit>(UniqueIDGenerator.java:58)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.common.IMSGUID.<init>(IMSGUID.java:40)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.logging.BasicLogEventDetails.<init>(BasicLogEventDetails.java:119)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.logging.log4j.Log4jLoggerWrapper.log(Log4jLoggerWrapper.java:278)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.logging.log4j.Log4jLoggerWrapper.log(Log4jLoggerWrapper.java:255)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.logging.log4j.Log4jLoggerWrapper.log(Log4jLoggerWrapper.java:114)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.logging.trace.TraceLog.log(TraceLog.java:297)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.logging.trace.TraceLog.log(TraceLog.java:282)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.tool.webtierbootstrapper.WebtierBootstrapper.start(WebtierBootstrapper.java:65)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.tool.webtierbootstrapper.WebtierBootstrapper.main(WebtierBootstrapper.java:87)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at java.lang.reflect.Method.invoke(Method.java:597)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:248)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at java.lang.Thread.run(Thread.java:680)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 | Caused by: java.net.UnknownHostException: webtierRH6.vcloud.dmz: webtierRH6.vcloud.dmz
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at java.net.InetAddress.getLocalHost(InetAddress.java:1409)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.common.UniqueIDGenerator.init(UniqueIDGenerator.java:111)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     at com.rsa.ims.common.UniqueIDGenerator.<clinit>(UniqueIDGenerator.java:56)
INFO   | jvm 1    | main    | 2015/10/26 10:26:04 |     ... 15 more
STATUS | wrapper  | main    | 2015/10/26 10:26:06 | <-- Wrapper Stopped

10. Check the status of the RSA Webtier Server  By default the web tier software is installed into/opt/RSASecurity/RSAAuthenticationManagerWebtier/, referred to below as [WT_Home].
11. Check the status of the RSA Webtier Bootstrapper Server with the command [wt_home]/server/rsaserv status.  For example ,

[rsaadmin@webtierRH6 ~]# /opt/RSASecurity/RSAAuthenticationManagerWebtier/server/rsaserv start
Running as rsaadmin...
RSA Webtier Server                                         [RUNNING]
 
[rsaadmin@webtierRH6 ~]#

12. The log files for this RSA Webtier Server are located in[WT_Home]/server/logs.  The log files of interest will be AdminServer.log, AdminServerWrapper.log, imsConsoleTrace.log and rsaserv.log.n.  Look at the AdminServer.log file first to check for any exceptions or errors should there be an issue starting the RSA Webtier Server.
13. Use article 000017402  Enabling debug for troubleshooting an RSA Authentication Manager 8.x deployment to enable debug and determine what the RSA Web Tier Server is actually doing when tasks are passed to the Web Tier.