Upgrade to RSA Authentication Manager 8.4.0 breaks LDAPS and other TLS/SSL connections

[#1]

Article Number

000037225

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.4.0

Issue

It has been reported that Active Directory LDAPS connections fail after an upgrade to RSA Authentication Manager 8.4.0.
Symptoms include;

1. A network packet capture shows the Authentication Manager server sending an SSL/TLS client hello request to the domain controller requesting a connection on TLSv1.2 using any of 31 ciphers listed. The DC responds that it will use the TLS_DHE_RSA_WITH_AES_256_GCM_SHA_384 Cipher Suite, which was one of the ciphers in the Authentication Manager.
   Image description

The Authentication Manager server ACKs, followed immediately by an Alert (Level: Fatal, Description: Internal Error), followed by a FIN.

2. The verbose /opt/rsa/am/server/logs/imsTrace.log shows the Diffie-Hellman DH Key negotiation fails

2019-02-18 14:52:27,635, [[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'], (LDAPConnectionTesterImpl.java:231), trace.com.rsa.ims.ldapslotmgt.impl.LDAPConnectionTesterImpl, ERROR, am82p.vcloud.local,,,,LDAP Server connection test failed javax.naming.CommunicationException: 2k12-dc1.2k12-vcloud.local:636 [Root exception is javax.net.ssl.SSLException: *Could not generate DH key pairs*]
Caused by: java.security.InvalidAlgorithmParameterException: Accepted DH prime length is 2048 or higher

Cause

Authentication Manager 8.4 is FIPS compliance for cryptographic operations (see the RSA SecurID Access Release Notes for RSA Authentication Manager 8.4), which means that 1024-bit certificates are no longer supported.  However, in these customer cases the domain controller and the Authentication Manager 8.4 server both have 2048-bit certificates.

However, within the negotiation of an SSL/TLS connection, the cipher suites that use Diffie-Hellman for key exchange must use a 2048-bit prime when generating a key pair.

In this case, the responding SSL server (which was an F5 load balancer in front of the domain controller) responded that the TLS_DHE_RSA_WITH_AES_256_GCM_SHA_384 cipher suite would be using a 1024 prime to generate a DHE key pair.

Image description

Because the F5 is using a DHE key size that is not FIPS 140-2 compliant, the Authentication Manager server errors out and the LDAPS connection fails.

Resolution

While it may be possible to insist that an F5 load balancer or a Microsoft domain controller use 2048 prime for DHE key generation within the TLS_DHE_RSA_WITH_AES_256_GCM_SHA_384 cipher suite, RSA Engineering has developed a fix that allows avoiding cipher suites such as TLS_DHE_RSA_WITH_AES_256_GCM_SHA_384, that depend on negotiation of other cipher components such as DHE key size.  This fix involves both a patch and a configurable GLOBAL variable.  
 

This hotfix is planned for release in Authentication Manager 8.4 patch 2, or customer support has a hotfix that could be applied to Authentication Manager 8.4 base.

 

Global variable fix to avoid cipher suites that allow negotiation of non-FIPS 140-2 compliant cipher components

1. Open an SSH session on each Authentication Manager server, starting with the primary first. 
2. Login as the rsaadmin user, noting that during Quick Setup another user name may have been selected. If that is the case, that user name to login.
3. Navigate to /opt/rsa/am/utils.
4. Run the command ./rsautil store -a add_config ims.tls.cipher_list.use_via_trust true GLOBAL BOOLEAN.  This global variable prevents Authentication Manager 8.4 from including the TLS_DHE_RSA_WITH_AES_256_GCM_SHA_384 cipher suite in the SSL client hello to the domain controller (F5) when trying to connect using LDAPS.  Internal testing at RSA has been done by Support and QE.

login as: rsaadmin
Using keyboard-interactive authentication.
Password: <enter operating system password>
Last login: Tue FEb 26 10:36:31 2018 from 192.168.2.102
RSA Authentication Manager Installation Directory: /opt/rsa/am
rsaadmin@am82p:~> cd /opt/rsa/am/utils r
saadmin@am82p:/opt/rsa/am/utils> ./rsautil store -a add_config ims.tls.cipher_list.use_via_trust true GLOBAL BOOLEAN Please enter OC Administrator user name: <enter Operations Console administrator name>
Please enter OC Administrator password: <enter Operations Console administrator password>
psql.bin:/tmp/f8e39a3c-a614-41e3-be96-299e670f0a73525273943558510875.sql;0108; NOTICE:  Added the new configuration parameter "ims.tls.cipher_list.use_via_trust" with the value "true"
 add_config 
---------------------

(1 row)
rsaadmin@am82p:/opt/rsa/am/utils>

• Use add_config the first time you run this command because you are creating or adding a new global variable for the code to use.  
• Use update_config any subsequent to undo this change.  For example,

saadmin@am82p:/opt/rsa/am/utils> ./rsautil store -a update_config ims.tls.cipher_list.use_via_trust false GLOBAL BOOLEAN
Please enter OC Administrator user name: <enter Operations Console administrator name>
Please enter OC Administrator password: <enter Operations Console administrator password>
psql.bin:/tmp/e6871864-6126-47cc-af20-0c261a3bbb643013521437038491182.sql;167; NOTICE:  Added the new configuration parameter "ims.tls.cipher_list.use_via_trust" from "true" to "false" for the instance 'GLOBAL'.
 update_config 
---------------------

(1 row)
rsaadmin@am82p:/opt/rsa/am/utils>

Workaround

We did not observe Microsoft domain controllers using the TLS_DHE_RSA_WITH_AES_256_GCM_SHA_384 by default, only the F5 load balancer configured to terminate the SSL connection, and in order to reproduce this, Support had to go through a lot of configuration and manipulation to force a domain controller to use this cipher suite; therefore, a workaround would be to avoid the load balancer or configure the load balancer to pass through LDAPS connections

Notes

This issue does not affect connections to the Authentication Manager server, only SSL/TLS connections from the Authentication Manager server to other devices, such as SMTP and SMS gateways that deliver on-demand tokencodes. 
See related issue AM-33242 (After Authentication Manager 8.4 upgrade SMS HTTPS plugin fails with some cipher suite)/

Also, there is no way to avoid the stricter protocol and cipher suites associated with FIPS 140-2 compliance on the Authentication Manager 8.4 servers.  You cannot turn it off and use these types of cipher suites