000013355 - AAOP- Adapter Siteminder 1.1.4 ssl handshake is breaking on newly upgraded Solaris 10

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000013355
Issue
3144/1][Tue Aug 28 2012 09:54:35][CServer.cpp:5111][INFO] Waiting for messages on thread id 1
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1575][ERROR] Handshake error: Unknown client name 'cfs216pw9htphost' in hello message
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1651][ERROR] Bad security handshake attempt. Handshake error: 3160
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1672][ERROR] Handshake error: Bad hostname in hello message
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1793][ERROR] Failed handshake with 10.64.160.61:64665
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1575][ERROR] Handshake error: Unknown client name 'cfs216pw9htphost' in hello message
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1651][ERROR] Bad security handshake attempt. Handshake error: 3160
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1672][ERROR] Handshake error: Bad hostname in hello message
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1793][ERROR] Failed handshake with 10.64.160.61:64666
[3144/7][Tue Aug 28 2012 09:54:36][CServer.cpp:1575][ERROR] Handshake error: Unknown client name 'cfs216pw9htphost' in hello message

This shows on the stdout of the smps siteminder adapter.
Thread-6, READ: SSLv3 Alert, length = 32
Padded plaintext after DECRYPTION:  len = 32
0000: 8E 27 B1 5C FA 45 96 91   BF 34 2D C4 19 DF F2 E4  .'.\.E...4-.....
0010: CB 19 12 87 75 94 37 D5   F6 88 0F BA 3E C8 06 90  ....u.7.....>...
Thread-6, SEND SSLv3 ALERT:  fatal, [Loaded com.sun.net.ssl.internal.ssl.Alerts from /usr/jdk/jre1.6.0_25/lib/jsse.jar]
description = bad_record_mac

Thread-6, called closeSocket()
Thread-6, handling exception: javax.net.ssl.SSLException: Invalid padding
ResolutionOn sun java java.security file, ensure that you change the order .
The sunpkcs11 was first on the list. this was changed and the handshakign went through. 
The issue was with the key Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA which both sides agreed on but 
there was padding issue.

From :-> security.provider.4=com.sun.crypto.provider.SunJCE

To: -> security.provider.1=com.sun.crypto.provider.SunJCE

Legacy Article IDa59598

Attachments

    Outcomes