[openssl-dev] Discrepancy between docs and actual behavior: CMS in 1.0.2

[Blumenthal, Uri - 0553 - MITLL]

I confess I did not test this with 1.1.x. But in 1.0.2h there’s a problem.

CMS man page says:

If the -decrypt option is used without a recipient certificate then an
attempt is made to locate the
recipient by trying each potential recipient in turn using the supplied
private key. To thwart the MMA
attack (Bleichenbacher's attack on PKCS #1 v1.5 RSA padding) all recipients
are tried whether they
succeed or not and if no recipients match the message is "decrypted" using a
random key which will
typically output garbage. The -debug_decrypt option can be used to disable
the MMA attack protection
and return an error if no recipient can be found: this option should be used
with caution.
However, the observed behavior is different:
$ openssl cms -engine pkcs11 -keyform engine -decrypt -debug_decrypt -aes256
-inform SMIME -in Cyph_Bot_test.smime.eml -outform SMIME -out
Cyph_Bot_test.decrypt1.eml -inkey
"pkcs11:object=KEY%20MAN%20key;object-type=private"
engine "pkcs11" set.
PKCS#11 token PIN:
Error decrypting CMS using private key
140735083847760:error:2E072084:CMS routines:CMS_decrypt_set1_pkey:no
matching recipient:cms_smime.c:661:
$

The following proves that the provided private key is correct (and the above
decryption should’ve succeeded):
$ openssl cms -engine pkcs11 -keyform engine -decrypt -aes256 -inform SMIME
-in Cyph_Bot_test.smime.eml -outform SMIME -out Cyph_Bot_test.decrypt.eml
-recip ~/Documents/Certs/me_mouse_yubi_9d_.pem -inkey
"pkcs11:object=KEY%20MAN%20key;object-type=private"
engine "pkcs11" set.
PKCS#11 token PIN:
$ tail Cyph_Bot_test.decrypt.eml
Message-id: <FBD81819-1B51-48CA-939B-***@us.army.mil>

It is either a bug in the man page or a bug in the code. In either case it
should be addressed.

P.S. This is how the CMS message in question was created:
$ openssl cms -engine pkcs11 -encrypt -aes256 -inform SMIME -in
Cyph_Bot_test.eml -outform SMIME -out Cyph_Bot_test.smime.eml -subject
SMIME_ECC ~/Documents/Certs/me_mouse_yubi_9d_.pem
engine "pkcs11" set.
$ tail Cyph_Bot_test.smime.eml
p7qKV4ttuid/6ynNbobYNgSUenzrmnbO0Z03KhglAy1l/om4crfK3+5r2UJ+5daf
9kL1EUrVy6flhE198793YTZJngi3zKFYk+BY2K8wNzLEoXAfJSY6a9z8RINZW9n8

[Dr. Stephen Henson]

I confess I did not test this with 1.1.x. But in 1.0.2h there???s a problem.
If the -decrypt option is used without a recipient certificate then an
attempt is made to locate the
recipient by trying each potential recipient in turn using the supplied
private key. To thwart the MMA
attack (Bleichenbacher's attack on PKCS #1 v1.5 RSA padding) all recipients
are tried whether they
succeed or not and if no recipients match the message is "decrypted" using a
random key which will
typically output garbage. The -debug_decrypt option can be used to disable
the MMA attack protection
and return an error if no recipient can be found: this option should be used
with caution.

That's a bug in the documentation. Currently that only works for RSA keys, not
EC or DH.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org