000013957 - SSL handshake error in browser phase of RCM installation: 'ssl_error_handshake_failure_alert' on browser  and 'SSL3_SEND_SERVER_KEY_EXCHANGE' in iws-cipher.log

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000013957
Applies ToRSA Certificate Manager 6.9 build 551 through 556
RSA Registration Manager 6.9 build 551 through 556
RHEL 5.4 (64-bit)
Mozilla Firefox 3.0.12 (on RHEL)
Sun Solaris 10 SPARC 64
Mozilla Firefox 2.0.0.19 (on Solaris)
If using Microsoft Internet Explorer, instead of Firefox, this problem does not occur
IssueSSL handshake error in browser phase of RCM installation: "ssl_error_handshake_failure_alert" on browser, and "SSL3_SEND_SERVER_KEY_EXCHANGE" in iws-cipher.log
BSAFE CryptoC ME files were properly installed in folder /usr/lib as per RCM Install guide
The very first step in browser-phase of the installation, on connecting to the install port, fails with SSL handshake error
Mozilla Firefox (on RHEL) shows the following:
An error occurred during a connection to <server-name>:<install-port>
SSL peer was unable to negotiate an acceptable set of security parameters.
(Error code: ssl_error_handshake_failure_alert)

Mozilla Firefox (on Solaris) shows the following:
Alert
<hostname> has received an incorrect or unexpectedmessage. Error Code: -12227
[OK]

The error code -12227 maps to SSL_ERROR_HANDSHAKE_FAILURE_ALERT and means: "SSL peer was unable to negotiate an acceptable set of security parameters."
RSA_CM/iws/logs/iws-cipher.log show the following:
[Fri Sep 14 14:02:19 2012] [info] Loading certificate & private key of SSL-aware server
[Fri Sep 14 14:02:19 2012] [info] Configuring server for SSL protocol
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(674): Creating new SSL context (protocols: SSLv2, SSLv3, TLSv1)
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(866): Configuring permitted SSL ciphers [ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL]
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(1022): Configuring HOSTNAME:3536:RSA server certificate
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(1135): Init: (HOSTNAME:3536:RSA) Configuring RSA server private key
[Fri Sep 14 14:02:19 2012] [info] Configuring server for SSL protocol
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(674): Creating new SSL context (protocols: SSLv2, SSLv3, TLSv1)
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(866): Configuring permitted SSL ciphers [ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL]
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(1022): Configuring HOSTNAME:3536:RSA server certificate
[Fri Sep 14 14:02:19 2012] [debug] ssl_engine_init.c(1135): Init: (HOSTNAME:3536:RSA) Configuring RSA server private key
[Fri Sep 14 14:02:39 2012] [info] Connection to child 0 established (server HOSTNAME:3536, client 10.20.10.101)
[Fri Sep 14 14:02:39 2012] [info] Seeding PRNG with 136 bytes of entropy
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1786): SSL-C: Handshake: start
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1794): SSL-C: Loop: before/accept initialization
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1743): SSL-C: read 7/7 bytes from BIO#a27e9a8 [mem: a289250] (BIO dump follows)
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1690): +-------------------------------------------------------------------------+
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0000: 16 03 01 00 af 01                                ......           |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1719): | 0007 - <SPACES/NULS>
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1721): +-------------------------------------------------------------------------+
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1743): SSL-C: read 173/173 bytes from BIO#a27e9a8 [mem: a289257] (BIO dump follows)
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1690): +-------------------------------------------------------------------------+
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0000: 00 ab 03 01 50 53 38 ff-85 9f df cd 86 20 cc ee  ....PS8...... .. |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0010: de 4c 8c 80 04 c7 71 c3-07 f5 0d 77 37 69 cc 39  .L....q....w7i.9 |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0020: 9f 29 ca 4d 00 00 2c 00-39 00 38 00 35 00 33 00  .).M..,.9.8.5.3. |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0030: 32 00 04 00 05 00 2f 00-16 00 13 fe ff 00 0a 00  2...../......... |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0040: 15 00 12 fe fe 00 09 00-64 00 62 00 03 00 06 00  ........d.b..... |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0050: 02 00 01 01 00 00 56 00-00 00 10 00 0e 00 00 0b  ......V......... |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0060: 67 61 6d 72 73 76 74 31-30 30 34 00 0a 00 34 00  gamrsvt1004...4. |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0070: 32 00 01 00 02 00 03 00-04 00 05 00 06 00 07 00  2............... |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0080: 08 00 09 00 0a 00 0b 00-0c 00 0d 00 0e 00 0f 00  ................ |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 0090: 10 00 11 00 12 00 13 00-14 00 15 00 16 00 17 00  ................ |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1715): | 00a0: 18 00 19 00 0b 00 02 01-00 00 23                 ..........#      |
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1719): | 0173 - <SPACES/NULS>
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_io.c(1721): +-------------------------------------------------------------------------+
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1794): SSL-C: Loop: SSLv3 read client hello A
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1794): SSL-C: Loop: SSLv3 write server hello A
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1794): SSL-C: Loop: SSLv3 write certificate A
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1187): handing out temporary 1 bit DH key
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1187): handing out temporary 139795788 bit DH key
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1804): SSL-C: Write: SSLv3 write key exchange A
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1823): SSL-C: Exit: error in SSLv3 write key exchange A
[Fri Sep 14 14:02:39 2012] [debug] ssl_engine_kernel.c(1823): SSL-C: Exit: error in SSLv3 write key exchange A
[Fri Sep 14 14:02:39 2012] [info] SSL library error 1 in handshake (server HOSTNAME:3536, client 10.20.10.101)
[Fri Sep 14 14:02:39 2012] [info] SSL Library Error: 336183467 error:1409C0AB:SSL routines:SSL3_SEND_SERVER_KEY_EXCHANGE:missing tmp dh key
[Fri Sep 14 14:02:39 2012] [info] Connection to child 0 closed with abortive shutdown(server HOSTNAME:3536, client 10.20.10.101)

SSLCipherSuite
SSLProtocol
CauseThe problem occurs in underlying BSAFE SSL-C libraries.  Based on the error in apache logs, it looks like the server chose a cipher suite that uses Ephemeral Diffie Hellman (EDH). This requires that a temporary DH key be set in the SSL library (or a callback for generating one). This appears to have not happened correctly. Either it couldn't generate one, or didn't get set.
ResolutionContact RSA Customer Support on the status of this issue and whether it has been fixed in a later build for RCM 6.9.
For completing new installation of RCM (or RRM):
For build 551 through build 555, the following steps (also documented in readme for builds 553-556) can be taken to workaround the problem and complete RCM (or RRM) installation on Linux or Solaris platforms:
1. Uninstall the current incomplete RCM installation (kill the processes 'xudad' and 'httpsd' if they are running, then remove RCM installation folder, say /home/<username>/RSA_CM)
2. Untar (e.g.,) RSACM-v6.9build551r-linux.tar file in the RCM installation folder (say, home directory "/home/<username>/")
3. After untarring the RCM (or RRM) tar file, open the following files (one at a time) in a text editor (vi):
iws/dist/iws.conf
WebServer/dist/scep.conf
WebServer/dist/admin.conf
WebServer/dist/enroll.conf

4. Search for the following line:
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:SSLv2:+EXP:+eNULL

5. Comment out the above line and add the following line below to the above line as follows:
SSLCipherSuite AES256-SHA
SSLProtocol +TLSv1

6. 
Then proceed with installation by running ./INSTALL script
7. SSL handshake should be successful and License Agreement page should show on the browser.  Follow the rest of the steps to complete RCM installation.


NOTE: You will not be able to use Windows XP with the settings above as Windows XP do NOT support AES256-sha.


 

For existing / already installed RCM (or RRM) deployments:
For existing deployments of RSA Cerfiticate Manager (or Registration Manager) where the same problem occurs (when Firefox is used to connect to enrollment or admin interface), follow the steps below to work around the problem:

1. Use a text editor and open the following configuration file in the RCM (or RRM) installation:
<install-folder>/WebServer/conf/httpd.conf

2. Search for the following line:
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:SSLv2:+EXP:+eNULL


3. Comment out the above line (by appending # in front of the line) and add the following lines below to the above line as follows:
SSLCipherSuite AES256-SHA
SSLProtocol +TLSv1

6. Save changes to httpd.conf

7. Restart RCM (or RRM) services, SSL handshake should be successful after the restart.


NOTE: You will not be able to use Windows XP with the settings above as Windows XP do NOT support AES256-sha.

WorkaroundInstalling RCM 6.9 on RHEL 5.4 64-bit
NotesThe above workaround to set cipher suite to AES256-SHA works for Firefox but may prevent IE8 on Windows 2003 Server. By default, Windows 2003 Server does not have support for AES256-SHA cipher with IE.  The following update is available to support AES256-SHA cipher on Windows 2003 Server:
http://support.microsoft.com/kb/948963
After applying the above patch, IE8 on Windows 2003 Server works fine.
* There is no issue with IE8 on Windows 2008/Windows 7 operating system (this has been tested with IE8/IE10 in Windows 2008 server machine and IE8 in Windows 7 machine).
Legacy Article IDa59568

Attachments

    Outcomes