000011778 - A PKCS#10 request through the enrollment server fails with an invalid signature error

Document created by RSA Customer Support Employee on Jun 14, 2016Last modified by RSA Customer Support Employee on Apr 21, 2017
Version 2Show Document
  • View in full screen mode

Article Content

Article Number000011778
Applies ToRSA Certificate Manager (RCM)
RSA Certificate Manager 6.8
RSA Certificate Manager 6.7
Sun Solaris 9
Sun Solaris 10
Microsoft Windows 2003 SP1
nCipher Hardware Security Module (HSM)
FIPS 140-2 Level III Strict mode enabled in nCipher Security World
nCipher cryptographic provider selected as Signature Verification Cryptographic Provider in RSA Certificate Manager
A PKCS#10 request is submitted through the enrollment server, which fails with the following message:

  This certificate request has been refused because it contains an invalid signature.

The request then goes into the refused state, but can be successfully approved from the queue.
If the same request is submitted with Software Cryptographic Provider selected as the signature verification cryptographic provider, the enrollment is successful.

This affects *all* requests (not just cut&past P10) - especially FireFox seems to be affected but I wouldn't bet that Opera or others wouldn't suffer the same fate.

Captured what is sent by FireFox and did a ASN.1 decode of the public key blob that is being send:


    0:d=0  hl=4 l= 576 cons: SEQUENCE         

    4:d=1  hl=4 l= 296 cons: SEQUENCE         

    8:d=2  hl=4 l= 290 cons: SEQUENCE         

   12:d=3  hl=2 l=  13 cons: SEQUENCE         

   14:d=4  hl=2 l=   9 prim: OBJECT            :rsaEncryption

   25:d=4  hl=2 l=   0 prim: NULL             

   27:d=3  hl=4 l= 271 prim: BIT STRING       

  302:d=2  hl=2 l=   0 prim: IA5STRING         :

  304:d=1  hl=2 l=  13 cons: SEQUENCE         

  306:d=2  hl=2 l=   9 prim: OBJECT            :md5WithRSAEncryption

  317:d=2  hl=2 l=   0 prim: NULL             

  319:d=1  hl=4 l= 257 prim: BIT STRING


As long as "md5" appears and the HSM is in strict FIPS mode and it is being used as the signature validation device it'll cause a "invalid signature"


Not a FireFox problem, not a HSM problem, not our problem but this combination will cause issues until FF switches to SHAx for the POP.

IssueA PKCS#10 request through the enrollment server fails with an invalid signature error
CauseWhen the PKCS#10 request has a Digest Algorithm that is not one that conforms to an allowed strict FIPS 140-2 Level III mode mechanism - in this case it was RSA MD5 - an invalid mechanism type message will be sent back to RSA Certificate Manager by nCipher. This will cause the invalid signature message to appear.

FIPS-approved algorithms: The following FIPS-approved Cryptographic algorithms are used: DSA (Cert. #143); Triple-DES (Cert. #378); AES (Cert, #303); RSA (Cert. #96); SHA-1; Diffie-Helman (used for key exchange in SSH2 is allowed in FIPS Mode but not approved).

The following algorithms are not available in FIPS Mode: MD5; Twofish; Blowfish; RC4.

ResolutionEither use an allowed mechanism - nominally SHA1 - or select software cryptographic provider for such PKCS#10 requests.
Legacy Article IDa41922