We've recently updated some old code with a later version of Crypt2 and have found an odd problem. We were exchanging data with a large, difficult company, and when they told us they could not decrypt data that we had encrypted using their public key (and Crypt2.EncryptBytes()) I was surprised.
Digging into it, I discovered that their public key had an illegal serial number - that is, it is a negative number (starts with hex "D3", in fact). That is clearly bad, based on the X509 RFC ("MUST be a positive integer").
Unfortunately, when we were using Crypt2 version 220.127.116.11, this was working. Now that we're on 18.104.22.168, it is not. Looking at the encrypted byte array, it's clear that the serialnumber Crypt2 is including in the header is now prepending the zero byte needed to make the serialnumber positive and "legal", and with version 22.214.171.124, it did not. Of course, their decryption software, being RFC-challenged, vomits.
Can anyone think of a work-around? I really don't want to roll back to an ancient version of Crypt2 and stay with that forever, and the receiving company is being completely obtuse about the problem being in their certificate generation. They claim they've had no such problem with any other trading partner, of course.
asked Nov 26 '14 at 10:28