Collections fail with NullPointerException on Remote Agent in SecurID Governance & Lifecycle

[#1]

Article Number

000067995

Applies To

This issue can occur in the following versions:

• RSA Identity Governance & Lifecycle 7.2.1
• RSA Identity Governance & Lifecycle 7.5.0
• SecurID Governance & Lifecycle 7.5.2

Issue

RSA Remote Agent shows as status "Is Running=Yes" but all Collectors for this Agent are failing. 

The Remote Agent's aveksaAgent.log file shows the following INFO level log messages with a warning:

05/20/2020 13:34:46.468 INFO  (ApplyChangesRegularThread-3) [com.aveksa.client.datacollector.framework.DataCollectorManager] WARNING: Hostname is not matched for session.

The aveksaAgent.log file shows a variety of other ERROR level messages related to the failure of the collectors to load JAVA classes:

05/20/2020 13:34:46.733 ERROR (ApplyChangesRegularThread-3) [com.aveksa.server.agent.message.ExceptionMessage] 
com.aveksa.common.ConfigException: Please ensure that all required classes are there in following urls: 
[https://{acm host}:8444/aveksa/DBIdentityCollector/, https://{acm host}:8444/aveksa/DBIdentityCollector1/classes/]

Testing a collector fails with a NullPointerException.  The aveksaAgent.log file shows an ERROR level log message related to the test:

05/20/2020 13:34:46.819 ERROR (ApplyChangesRegularThread-3) [com.aveksa.client.datacollector.framework.DataCollectorManager] FAILED method=Collect CollectionMetaInfo[{ID=1, run_id=381, collector_id=3, test-run=true, collector_name=IDC}] 
com.aveksa.common.ConfigException: java.lang.NullPointerException

There are no errors in the aveksaServer.log file and normal INFO level log messages indicate the remote agent is running:

05/20/2020 13:34:24.320 INFO  (default task-42) [com.aveksa.server.agent.core.AveksaAgent] Contact initiated from agent: OnPremise Integration Agent.Version: 2.0, Agent Runtime Data{dataTimestamp='1589981664320', hostname='{agent}', internalIp='10.10.10.10', externalIp='10.10.10.10', javaInstanceId='8692@{agent}', agentStartTime='1589981617657'}

Cause

This issue can occur on any version of SecurID Governance & Lifecycle if the system was upgraded from an older version AND certificates were NOT regenerated (Admin > System > Security > Change Certificate Store) since the upgrade.

The issue only occurs if the JAVA version on the SecurID Governance & Lifecycle application server has been upgraded to 1.8.0_191 or higher.

JAVA 1.8.0_191 introduced additional checks for TLS/SSL communication that requires verifying the hostname of the server against the DNS names configured in Subject Alternative Name (SAN) extension of the server certificate.  Certificates generated on older versions of SecurID Governance & Lifecycle may not have the required attributes.

This issue is commonly seen when SecurID Governance & Lifecycle application server is hosted on AWS platform but may occur for any type of deployment where proxies obfuscate the hostname that is resolved for DNS checks.

Note that the Remote Agent, as a client, initiates an HTTPS connection to the SecurID Governance & Lifecycle application server over port 8444 (in a default configuration), as a server.

Resolution

This issue is resolved in the following versions, where a custom setting DomainNames can be used to specify additional comma separated hostnames or aliases which are then added, as DNS Names, to the Subject Alternative Name (SAN) extension of a regenerated server certificate for SecurID Governance & Lifecycle application server (port 8444):

• RSA Identity Governance & Lifecycle 7.2.1 P03
• RSA Identity Governance & Lifecycle 7.5.0
• SecurID Governance & Lifecycle 7.5.2

Note that the DNS names added to the SAN extension of the SecurID Governance & Lifecyle's server certificate (for port 8444) must match exactly with the hostname that the Remote Agent uses to contact the server.  The Remote Agent's configuration file <RemoteAgent-install-dir>/conf/config.properties lists the URL it uses to contact the server in a parameter server_url.  You must ensure that the hostname in the server URL used by the Remote Agent must match with one of the DNS names in the SecurID Governance & Lifecycle application server certificate's SAN extension.

This is particularly important for SecurID Governance & Lifecycle deployments on AWS or other deployments with a proxy server where the hostname of SecurID Governance & Lifecycle's server instance is different for external clients (Remote Agent and Remote AFX) than it is for internal connections. It is also important for clustered deployments where the individual nodes have different hostnames than the front end load balancer.

In these instances, you must ensure that correct hostname(s) are added to the SAN certificate extension using the custom system parameter "DomainNames" as per instructions in the SecurID Governance & Lifecycle 7.5.2 Installation Guide, section "Using Domain Name for Certificate Validation".

Refer to the following article when generating new server certificate for SecurID Governance & Lifecycle:  How to Update the Root (Server) and Client Certificates in RSA Identity Governance & Lifecycle

Workaround

Revert back to a JAVA version older than 1.8.0_191 on the Remote Agent host.

Notes

Note that generating new server certificate (Admin > System > Security > Change Certificate Store) for SecurID Governance & Lifecycle invalidates any existing Remote Agents or AFX Servers.  You must generate new client certificates for each of the Remote Agents and AFX Servers IF you have generated a new server certificate.