Error "Key negotiation exchange failed. Server response was CANCELLED" with RSA Authentication Agent API 8.5 and later

[#1]

Article Number

000044614

Applies To

RSA Product Set: RSA SecurID
RSA Product/Service Type:  RSA Authentication Agent API
RSA Version/Condition: 8.5 and later
 

Issue

When testing the TCP Authentication from RSA Authentication API 8.5 and later, or any other agents that uses these APIs for Authentication in the TCP mode this is the error seen:

com.rsa.authagent.authapi.AuthAgentException: Error in initial AuthnReq/Rsp for serverTime.Error in processing Authn request: connect exception processing key negotiation request: com.rsa.authmgr.commonagent.h: Key negotiation exchange failed. Server response was CANCELLED

com.rsa.authagent.authapi.AuthAgentException: com.rsa.authagent.authapi.AuthAgentException: Error in initial AuthnReq/Rsp for serverTime.Error in processing Authn request: connect exception processing key negotiation request: com.rsa.authmgr.commonagent.h: Key negotiation exchange failed. Server response was CANCELLED
        at com.rsa.authagent.authapi.AuthSessionFactory.a(AuthSessionFactory.java)
        at com.rsa.authagent.authapi.AuthSessionFactory.getInstance(AuthSessionFactory.java)
        at sample.AuthUser.<init>(AuthUser.java:32)
        at sample.AuthUser.main(AuthUser.java:62)

 

Cause

If you browse to Security Console –> Setup –> System Settings --> Agents, then click on To configure agents using IPV6, click here.
This is the certificate the AM uses in the communication with the TCP Agents. If this certificate is not correct then the authentication will fail with the above error.

The certificate found in the above page should be the same as the one we can export after accessing https://AM fully qualified domain name:7002
 

Resolution

1. Using Google Chrome, browse to https://AM fully qualified domain name:7002
2. Click on the lock icon in the browser address bar. 
3. Click on Certificate.
4. Click the Certification Path tab.
5. Double-click on the top-level (root, very first) certificate in the list.
6. Click on Details tab, then Copy to File...
7. Click Next, then check the Second Option Base-64 encoded X.509 output format (.CER)
8. Click Next, then click Browse to choose the location and give it any name, such as root then Click Save.
9. Click Next then Finish, you'll find the exported certificate in the location chosen in Step 8
10. Browse to Security Console –> Setup –> System Settings --> Agents, then click on To configure agents using IPV6, click here.
11. Scroll down under Existing Certificate Details, click on the Choose File Option then browse to the certificate we just exported then Click Update.

Notes

If the Authentication Manager is on 8.2 SP1 till 8.2 SP1 Patch 4, you can still see the above errors even after you complete the steps in the Resolution. This is because there is a bug on AM 8.2 SP1 where it doesn't communicate with the certificate we export from the 7002 port that it should use, it communicates with the Console Certificate. The environments that will see this are the environments that have replaced the default self-signed certificate for the consoles with another certificate.

There are 3 workarounds to solve this:

1. Upgrade to AM 8.2 SP1 Patch 5.
2. Revert the Console Certificate to use the default self-signed certificate using below steps or normally from the Operations console:   Reverting back to the RSA self-signed default certificates on Authentication Manager 8.1
3. Follow the exact steps in the resolution here, but in step 1 rather than browsing to https://AM fully qualified domain name:7002, browse to either the Security Console or the Operations Console of the Authentication Manager and export the certificate from there, then complete the same steps as they are.