Importing an SSL console certificate PKCS#12 file to the RSA Authentication Manager 8.2 Operations Console fails with password incorrect

[#1]

Article Number

000034200

Applies To

RSA Product Set: SecurID
RSA Product/Service Type: Authentication Manager
RSA Version/Condition: 8.2

Issue

• Within the RSA Authentication Manager 8.2 Operations Console, if you try to import an SSL console replacement certificate in PKCS#12 format, which indicates the CSR was generated by a third-party tool and requires a password, it fails with the following error:

Your password is incorrect.  Correct your password, and try again.
 

Image description

• The password is known to be correct and the same .pfx or .p12 file imports successfully into Internet Explorer with the same password.
• The /opt/rsa/am/server/logs/ops-console.log will show something like the following error:

GUILog.traceException(587) | exception: 
com.rsa.ims.security.tools.ssl.exception.InvalidCertificateException: Algorithm not allowable in FIPS140 mode: PBE/PKCS12/SHA1/RC2/CBC/40
at com.rsa.ims.security.tools.ssl.ImportSSLCertConsoleHelper.importP12File(ImportSSLCertConsoleHelper.java:573)

• Manually importing this certificate with Java Keytool results in this error:

keytool error: java.lang.SecurityException: Algorithm not allowable in FIPS140 mode: PBE/PKCS12/SHA1/RC2/CBC/40
java.lang.SecurityException: Algorithm not allowable in FIPS140 mode: PBE/PKCS12/SHA1/RC2/CBC/40

Cause

We have seen this behavior mostly with Entrust or Entrust/Venafi certificates.  An RSA console-generated Certificate Signing Request (CSR) also generates a public/private key pair and includes only the public key.  The private key never leaves the RSA console keystore.  The Certificate Authority (CA) response certificate will typically be in PKCS#7 format (i. e., either with a .cer or .p7b file extension).  It will not include the private key and will not be password protected.

If a third-party tool generates the CSR, it will also generate the public/private key pair, and the CA response certificate will typically be in a PKCS#12 format, with either a .pfx or .p12 file extension.  It will be password-protected, because it also includes the private key.

It appears that in this case, the Entrust/Venafi certificate response used an RC2 algorithm to password-protect either the certificate file itself or the private key within the .pfx certificate file.  

The Authentication Manager Operations Console flagged this as an incorrect password, but the /opt/rsa/am/server/logs/ops-console.log and keytool flagged it with the following error:

Algorithm not allowable in FIPS140 mode: PBE/PKCS12/SHA1/RC2/CBC/40

The /opt/rsa/am/server/logs/imsTrace.log will show this error: 

com.rsa.ims.security.tools.ssl.exception.InvalidCertificateException: MacData.MacData: MAC Verification failed

Resolution

Since RC2 is not FIPS-compliant, Entrust or another third-party CA must use another algorithm that is FIPS-compliant.

Workaround

Import this PKCS#12 formatted certificate response file into another tool such as OpenSSL and export it with a password with 3DES or another algorithm that is FIPS 140-2 compliant, such as AES.  One can use OpenSSL that comes in the Authentication Manager installation to do this.

1. First, extract the original certificate and private key into encrypted PEM format:

openssl pkcs12 -in <PKCS#12_filename> -out <encrypted_PEM_filename>

          For example,

rsaadmin@am81p:~> ls /tmp/vmrsa*
/tmp/vmrsa01.pfx
rsaadmin@am81p:~> openssl pkcs12 -in /tmp/vmrsa01.pfx -out /tmp/vmrsa01.pem -nodes
Enter Import Password: <enter password>
MAC verified OK
rsaadmin@am81p:~> ls /tmp/vmrsa*
/tmp/vmrsa01.pfx
rsaadmin@am81p:~> ls vmrsa*
vmrsa01.pem
rsaadmin@am81p:~> 

2. Now generate a FIPS compliant PKCS file with the following command:

openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -in <path to .pem file and file name> -out >path to .pfx file and file name> -name "<machine_name>"

For example,

rsaadmin@am81p:~> openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -in /tmp/vmrsa01.pem -out /tmp/vmrsa01pbe.pfx -name "vmrsa01-2016"
Enter Export Password: <enter export password>
Verifying - Enter Export Password: <enter export password>
rsaadmin@am81p:~>

3. Then generate a FIPS-compliant PKCS file, as shown here:

rsaadmin@am81p:~> openssl pkcs12 -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -export -in <encrypted_PEM_filename> -out <PKCS#12_filename> -name "<display_name>"

Notes

The error message "This certificate or its signing CA is not valid" could indicate that the imported certificate response is not for this particular server, possibly because the fully qualified domain name (FQDN) of the Authentication Manager server should match the Common Name (CN) within the subject field of the certificate response, as shown here:

Image descriptionImage description

The error could also indicate the private key is not in the PKCS#12 file or also that a private key is included in a PKCS#7 certificate response file when it is not expected to be there.