AAOP- Adapter Siteminder 1.1.4 ssl handshake is breaking on newly upgraded Solaris 10
2 years ago
Originally Published: 2012-09-20
Article Number
000046889
Issue
3144/1][Tue Aug 28 2012 09:54:35][CServer.cpp:5111][INFO] Waiting for messages on thread id 1
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1575][ERROR] Handshake error: Unknown client name 'cfs216pw9htphost' in hello message
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1651][ERROR] Bad security handshake attempt. Handshake error: 3160
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1672][ERROR] Handshake error: Bad hostname in hello message
[3144/10][Tue Aug 28 2012 09:54:36][CServer.cpp:1793][ERROR] Failed handshake with 10.64.160.61:64665
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1575][ERROR] Handshake error: Unknown client name 'cfs216pw9htphost' in hello message
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1651][ERROR] Bad security handshake attempt. Handshake error: 3160
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1672][ERROR] Handshake error: Bad hostname in hello message
[3144/13][Tue Aug 28 2012 09:54:36][CServer.cpp:1793][ERROR] Failed handshake with 10.64.160.61:64666
[3144/7][Tue Aug 28 2012 09:54:36][CServer.cpp:1575][ERROR] Handshake error: Unknown client name 'cfs216pw9htphost' in hello message

This shows on the stdout of the smps siteminder adapter.
Thread-6, READ: SSLv3 Alert, length = 32
Padded plaintext after DECRYPTION:  len = 32
0000: 8E 27 B1 5C FA 45 96 91   BF 34 2D C4 19 DF F2 E4  .'.\.E...4-.....
0010: CB 19 12 87 75 94 37 D5   F6 88 0F BA 3E C8 06 90  ....u.7.....>...
Thread-6, SEND SSLv3 ALERT:  fatal, [Loaded com.sun.net.ssl.internal.ssl.Alerts from /usr/jdk/jre1.6.0_25/lib/jsse.jar]
description = bad_record_mac

Thread-6, called closeSocket()
Thread-6, handling exception: javax.net.ssl.SSLException: Invalid padding

Resolution
On sun java java.security file, ensure that you change the order .
The sunpkcs11 was first on the list. this was changed and the handshakign went through. 
The issue was with the key Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA which both sides agreed on but 
there was padding issue.

From :-> security.provider.4=com.sun.crypto.provider.SunJCE

To: -> security.provider.1=com.sun.crypto.provider.SunJCE