I'm switching from the pure Python ecdsa library to the much faster coincurve library for signing data. I would also like to switch to coincurve for verifying the signatures (including the old signatures created by the ecdsa library).
It appears that signatures created with ecdsa are not (always?) valid in coincurve. Could someone please explain why this is not working? Also, it seems that cryptography library is able to validate both ecdsa signatures and coincurve signatures without issues, consistently.
What is even more confusing, if you run below script a few times, is that sometimes it prints point 3 and other times it does not. Why would coincurve only occasionally find the signature valid?
pip install ecdsa cryptography coincurve
import ecdsa
import hashlib
import coincurve
from coincurve.ecdsa import deserialize_compact, cdata_to_der
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec
from cryptography.hazmat.primitives.asymmetric.utils import Prehashed
ecdsa_private_key = ecdsa.SigningKey.generate(ecdsa.SECP256k1, None, hashlib.sha256)
ecdsa_pub = ecdsa_private_key.get_verifying_key()
message = b"Hello world!"
digest = hashlib.sha256(message).digest()
serialized_signature = ecdsa_private_key.sign_digest_deterministic(digest, hashfunc=hashlib.sha256)
signature = cdata_to_der(deserialize_compact(serialized_signature))
cc_private_key = coincurve.PrivateKey(ecdsa_private_key.to_string())
cc_pub = cc_private_key.public_key
crypto_pub = ec.EllipticCurvePublicKey.from_encoded_point(ec.SECP256K1(), cc_pub.format(True))
if ecdsa_pub.verify_digest(serialized_signature, digest) is True:
print("1. ecdsa can validate its own signature")
crypto_pub.verify(signature, digest, ec.ECDSA(Prehashed(hashes.SHA256())))
print("2. cryptography can validate ecdsa signature (raises exception if not valid)")
if cc_pub.verify(signature, digest, None) is False:
print("3. coincurve will not validate ecdsa signature")
signature = cc_private_key.sign(digest, None)
crypto_pub.verify(signature, digest, ec.ECDSA(Prehashed(hashes.SHA256())))
print("4. cryptography can validate coincurve signature (raises exception if not valid)")
if cc_pub.verify(signature, digest, None) is True:
print("5. coincurve will validate its own signature")
Bitcoin and the coincurve library use canonical signatures while this is not true for the ecdsa library.
What does canonical signature mean?
In general, if (r,s) is a valid signature, then (r,s') := (r,-s mod n) is also a valid signature (n is the order of the base point).
A canonical signature uses the value s' = -s mod n = n - s instead of s, i.e. the signature (r, n-s), if s > n/2, s. e.g. here.
All signatures from the ecdsa library that were not been successfully validated by the coincurve library in your test program have an s > n/2 and thus are not canonical, whereas those that were successfully validated are canonical.
So the fix is simply to canonize the signature of the ecdsa library, e.g.:
def canonize(s_bytes):
n = 115792089237316195423570985008687907852837564279074904382605163141518161494337
s = int.from_bytes(s_bytes, byteorder='big')
if s > n//2:
s = n - s
return s.to_bytes(32, byteorder='big')
...
serialized_signature = serialized_signature[:32] + canonize(serialized_signature[32:]) # Fix: canonize
signature = cdata_to_der(deserialize_compact(serialized_signature))
...
With this fix, the coincurve library successfully validates all signatures from the ecdsa library in your test program.
Related
I'm playing around with the cryptography library in Python and I have a CRL object. Here is the setup:
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.x509.oid import ExtensionOID, NameOID, AuthorityInformationAccessOID
# Need to set default backend for cryptography as our version is low:
default_backend = default_backend()
crl_file = path_to/some_crl_file.crl
crl = x509.load_pem_x509_crl(crl_file, default_backend)
print(crl.extensions)
I receive the following output of extensions:
<Extensions([<Extension(oid=<ObjectIdentifier(oid=2.5.29.20, name=cRLNumber)>, critical=False, value=<CRLNumber(17)>)>, <Extension(oid=<ObjectIdentifier(oid=2.5.29.35, name=authorityKeyIdentifier)>, critical=False, value=<AuthorityKeyIdentifier(key_identifier=b"\xe4\xaf+&q\x1a+H'\x85/Rf,\xef\xf0\x89\x13q>", authority_cert_issuer=None, authority_cert_serial_number=None)>)>])>
I then get the CRL Number with the following:
print(crl.extensions.get_extension_for_oid(ExtensionOID.CRL_NUMBER).value)
This outputs <CRLNumber(17)>. How do I get it so that I receive the int value of 17 from this?
I swear someone answered this and I responded, but maybe I dreamed it up.
Anywho. I took another look into this and resolved it with the following:
print(crl.extensions.get_extension_for_oid(ExtensionOID.CRL_NUMBER).value.crl_number)
I needed to add the .crl_number at the end of value.
Working with an HSM I can download the public key of a RSA 4096, and can use the HSM to encrypt a blob using the corresponding private key, which I can't download.
I'm trying to find a reference to how I can use the HSM apis, to build the X509 certificate myself. I could find the example below in python, wrapping up openssl libs.
It does mostly all, but
from OpenSSL import crypto, SSL
from socket import gethostname
from pprint import pprint
from time import gmtime, mktime
CERT_FILE = "selfsigned.crt"
KEY_FILE = "private.key"
def create_self_signed_cert():
# create a key pair
k = crypto.PKey()
k.generate_key(crypto.TYPE_RSA, 1024)
# create a self-signed cert
cert = crypto.X509()
cert.get_subject().C = "UK"
cert.get_subject().ST = "London"
cert.get_subject().L = "London"
cert.get_subject().O = "Dummy Company Ltd"
cert.get_subject().OU = "Dummy Company Ltd"
cert.get_subject().CN = gethostname()
cert.set_serial_number(1000)
cert.gmtime_adj_notBefore(0)
cert.gmtime_adj_notAfter(10*365*24*60*60)
cert.set_issuer(cert.get_subject())
cert.set_pubkey(k)
cert.sign(k, 'sha1')
open(CERT_FILE, "wt").write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
open(KEY_FILE, "wt").write(crypto.dump_privatekey(crypto.FILETYPE_PEM, k))
create_self_signed_cert()
Mostly what I would expect to be possible is to replace the line
cert.sign(k, 'sha1')
by the corresponding calls to the APIs of my HSM with "an export" of the structure of "cert". Is it possible at all ?
So after one day of research I found a satisfactory answer at Adding a signature to a certificate . While there seems to be some Python examples out how to build the certificate through Python with pyasn1, the most robust seems to be the ones in java using boucycastle libraries. I also found a strong answer how to incorporate it with openssl here How to generate certificate if private key is in HSM? (mostly points 3 and 4), but the approach is far beyond my capacities.
When you want someone else, like your HSM, to sign a certificate for you the standard way to do that is to generate a Certificate Signing Request (CSR) and send the CSR to the signer. The CSR can contain all of the fields that you can set when making a certificate, and those values will be reflected in the resulting certificate.
While you can do this with the OpenSSL library that you are using, see X509Req, their documentation (and I quote) "strongly suggests" that you use the cryptography library instead. (They're maintained by the same group of people.)
In the cryptography library you use the CertificateSigningRequestBuilder to make a CSR. To quote from their docs:
>>> from cryptography import x509
>>> from cryptography.hazmat.primitives import hashes
>>> from cryptography.hazmat.primitives.asymmetric import rsa
>>> from cryptography.x509.oid import AttributeOID, NameOID
>>> private_key = rsa.generate_private_key(
... public_exponent=65537,
... key_size=2048,
... )
>>> builder = x509.CertificateSigningRequestBuilder()
>>> builder = builder.subject_name(x509.Name([
... x509.NameAttribute(NameOID.COMMON_NAME, u'cryptography.io'),
... ]))
>>> builder = builder.add_extension(
... x509.BasicConstraints(ca=False, path_length=None), critical=True,
... )
>>> builder = builder.add_attribute(
... AttributeOID.CHALLENGE_PASSWORD, b"changeit"
... )
>>> request = builder.sign(
... private_key, hashes.SHA256()
... )
>>> isinstance(request, x509.CertificateSigningRequest)
True
To render out the CSR in the format usually used for sending them around:
>>> from cryptography.hazmat.primitives import serialization
>>> request.public_bytes(serialization.Encoding.PEM)
I am communicating with our clients server. For an api I need to sign a string with my private key. They have the following condition to follow
User SHA 256 algorithm to calculate the hash of the string
Use the private key and RSA (PKCS1_PADDING) algorithm to sign the Hash Value.
Base64 encode the encrypted Hash Value
and I am doing following
from Crypto.Signature import PKCS1_v1_5
from Crypto.Hash import SHA256
from Crypto.PublicKey import RSA
import base64
pkey = RSA.importKey(keystring)
message = "Hello world"
h = SHA256.new(message.encode())
signature = PKCS1_v1_5.new(pkey).sign(h)
result = base64.b64encode(signature).decode()
Here I am getting a string as result. But on the server side my signature is not matching.
Is there anything I am going wrong with ?? Can anyone help me on this ?
I came back to this question recently and noticed it was never resolved. I don't know what was going wrong with the OPs setup but the following code worked for me.
First, the python code that generates the signature of "Hello world":
from Cryptodome.Signature import PKCS1_v1_5
from Cryptodome.Hash import SHA256
from Cryptodome.PublicKey import RSA
import base64
def sign(message: str, private_key_str: str) -> str:
priv_key = RSA.importKey(private_key_str)
h = SHA256.new(message.encode('utf-8'))
signature = PKCS1_v1_5.new(priv_key).sign(h)
result = base64.b64encode(signature).decode()
return result
And now the Java code that verifies it:
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.GeneralSecurityException;
import java.security.KeyFactory;
import java.security.PublicKey;
import java.security.Signature;
import java.security.spec.X509EncodedKeySpec;
import java.util.Base64;
...
...
public static boolean verify(String message, String b64Sig, byte[] pubkey_spki) throws GeneralSecurityException {
var pubKey = (PublicKey) KeyFactory.getInstance("RSA").generatePublic(new X509EncodedKeySpec(pubkey_spki));
var verifier = Signature.getInstance("SHA256withRSA");
verifier.initVerify(pubKey);
verifier.update(message.getBytes(StandardCharsets.UTF_8));
return verifier.verify(Base64.getDecoder().decode(b64Sig));
}
Perhaps the trickiest part of this is specifying the correct padding scheme in each language/library. These signatures use the scheme identified as RSASSA-PKCS1-v1_5 in the PKCS#1 RFC 8017. On the python side this is accomplished by providing the SHA256 hash object to the PKCS1_v1_5 signature object. In Java it is perhaps a little more straightforward in that you ask for Signature object that implements the RSA algorithm with SHA256 as the hash function, but still have to know that this is RSASSA-PKCS1-v1_5 and not some other possibility in RFC 8017.
I think if one is not already something of an expert then understanding that these magic choices in python and Java produce compatible code is going to be difficult.
Currently I have some code that signs a byte string with the SHA256 algorithm using the native OpenSSL binary, the code calls an external process, sends the parameters, and receive the result back into the Python code.
The current code is as follows:
signed_digest_proc = subprocess.Popen(
['openssl', 'dgst', '-sha256', '-sign', tmp_path],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE
)
signed_digest_proc.stdin.write(original_string)
signed_digest, _ = signed_digest_proc.communicate()
base64.encodestring(signed_digest).decode().replace('\n', '')
When original_string is too big, I might have problems with the result (from the communication with an external process I think), that's why I'm trying to change it to a Python only solution:
import hmac, hashlib
h = hmac.new(bytes(key_pem(), 'ASCII'), original_string, hashlib.sha256)
result = base64.encodestring(h).decode().replace('\n', '')
This result in a completely different string than the first one.
What would be the way to implement the original code without calling an external process?
The openssl command you used does three things:
Create a hash of the data, using SHA256
If RSA is used, pad out the message to a specific length, using PKCS#1 1.5
Sign the (padded) hash, using the private key you provided. It'll depend on the type of key what algorithm was used.
The hmac module does not serve the same function.
You'll need to install a cryptography package like cryptography to replicate what openssl dgst -sign does. cryptography uses OpenSSL as a backend, so it will produce the same output.
You can then
load the key with the load_pem_private_key() function. This returns the right type of object for the algorithm used.
use the key to sign the message; each key type has a sign() method, and this method will take care of hashing the message for you if you so wish. See for example the Signing section for RSA.
However, you'll need to provide different kinds of config for the different .sign() methods. Only the RSA, DSA and Elliptic Curve keys can be used to create a signed digest.
You'll have to switch between the types to get the signature right:
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import dsa, ec, rsa, utils
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import padding
# configuration per key type, each lambda takes a hashing algorithm
_signing_configs = (
(dsa.DSAPrivateKey, lambda h: {
'algorithm': h}),
(ec.EllipticCurvePrivateKey, lambda h: {
'signature_algorithm': ec.ECDSA(h)}),
(rsa.RSAPrivateKey, lambda h: {
'padding': padding.PKCS1v15(),
'algorithm': h
}),
)
def _key_singing_config(key, hashing_algorithm):
try:
factory = next(
config
for type_, config in _signing_configs
if isinstance(key, type_)
)
except StopIteration:
raise ValueError('Unsupported key type {!r}'.format(type(key)))
return factory(hashing_algorithm)
def sign(private_key, data, algorithm=hashes.SHA256()):
with open(private_key, 'rb') as private_key:
key = serialization.load_pem_private_key(
private_key.read(), None, default_backend())
return key.sign(data, **_key_singing_config(key, algorithm))
If you need to hash a large amount of data, you can hash the data yourself first, in chunks, before passing in just the digest and the special util.Prehashed() object:
def sign_streaming(private_key, data_iterable, algorithm=hashes.SHA256()):
with open(private_key, 'rb') as private_key:
key = serialization.load_pem_private_key(
private_key.read(), None, default_backend())
hasher = hashes.Hash(algorithm, default_backend())
for chunk in data_iterable:
hasher.update(chunk)
digest = hasher.finalize()
prehashed = utils.Prehashed(algorithm)
return key.sign(digest, **_key_singing_config(key, prehashed))
with open(large_file, 'rb') as large_file:
signature = sign_streaming(private_key_file, iter(lambda: large_file.read(2 ** 16), b''))
This uses the iter() function to read data from a binary file in chunks of 64 kilobytes.
Demo; I'm using an RSA key I generated in /tmp/test_rsa.pem. Using the command-line to produce a signed digest for Hello world!:
$ echo -n 'Hello world!' | openssl dgst -sign /tmp/test_rsa.pem -sha256 | openssl base64
R1bRhzEr+ODNThyYiHbiUackZpx+TCviYR6qPlmiRGd28wpQJZGnOFg9tta0IwkT
HetvITcdggXeiqUqepzzT9rDkIw6CU7mlnDRcRu2g76TA4Uyq+0UzW8Ati8nYCSx
Wyu09YWaKazOQgIQW3no1e1Z4HKdN2LtZfRTvATk7JB9/nReKlXgRjVdwRdE3zl5
x3XSPlaMwnSsCVEhZ8N7Gf1xJf3huV21RKaXZw5zMypHGBIXG5ngyfX0+aznYEve
x1uBrtZQwUGuS7/RuHw67WDIN36aXAK1sRP5Q5CzgeMicD8d9wr8St1w7WtYLXzY
HwzvHWcVy7kPtfIzR4R0vQ==
or using the Python code:
>>> signature = sign(keyfile, b'Hello world!')
>>> import base64
>>> print(base64.encodebytes(signature).decode())
R1bRhzEr+ODNThyYiHbiUackZpx+TCviYR6qPlmiRGd28wpQJZGnOFg9tta0IwkTHetvITcdggXe
iqUqepzzT9rDkIw6CU7mlnDRcRu2g76TA4Uyq+0UzW8Ati8nYCSxWyu09YWaKazOQgIQW3no1e1Z
4HKdN2LtZfRTvATk7JB9/nReKlXgRjVdwRdE3zl5x3XSPlaMwnSsCVEhZ8N7Gf1xJf3huV21RKaX
Zw5zMypHGBIXG5ngyfX0+aznYEvex1uBrtZQwUGuS7/RuHw67WDIN36aXAK1sRP5Q5CzgeMicD8d
9wr8St1w7WtYLXzYHwzvHWcVy7kPtfIzR4R0vQ==
Although the line lengths differ, the base64 data the two output is clearly the same.
Or, using a generated file with random binary data, size 32kb:
$ dd if=/dev/urandom of=/tmp/random_data.bin bs=16k count=2
2+0 records in
2+0 records out
32768 bytes transferred in 0.002227 secs (14713516 bytes/sec)
$ cat /tmp/random_data.bin | openssl dgst -sign /tmp/test_rsa.pem -sha256 | openssl base64
b9sYFdRzpBtJTan7Pnfod0QRon+YfdaQlyhW0aWabia28oTFYKKiC2ksiJq+IhrF
tIMb0Ti60TtBhbdmR3eF5tfRqOfBNHGAzZxSaRMau6BuPf5AWqCIyh8GvqNKpweF
yyzWNaTBYATTt0RF0fkVioE6Q2LdfrOP1q+6zzRvLv4BHC0oW4qg6F6CMPSQqpBy
dU/3P8drJ8XCWiJV/oLhVehPtFeihatMzcZB3IIIDFP6rN0lY1KpFfdBPlXqZlJw
PJQondRBygk3fh+Sd/pGYzjltv7/4mC6CXTKlDQnYUWV+Rqpn6+ojTElGJZXCnn7
Sn0Oh3FidCxIeO/VIhgiuQ==
Processing the same file in Python:
>>> with open('/tmp/random_data.bin', 'rb') as random_data:
... signature = sign_streaming('/tmp/test_rsa.pem', iter(lambda: random_data.read(2 ** 16), b''))
...
>>> print(base64.encodebytes(signature).decode())
b9sYFdRzpBtJTan7Pnfod0QRon+YfdaQlyhW0aWabia28oTFYKKiC2ksiJq+IhrFtIMb0Ti60TtB
hbdmR3eF5tfRqOfBNHGAzZxSaRMau6BuPf5AWqCIyh8GvqNKpweFyyzWNaTBYATTt0RF0fkVioE6
Q2LdfrOP1q+6zzRvLv4BHC0oW4qg6F6CMPSQqpBydU/3P8drJ8XCWiJV/oLhVehPtFeihatMzcZB
3IIIDFP6rN0lY1KpFfdBPlXqZlJwPJQondRBygk3fh+Sd/pGYzjltv7/4mC6CXTKlDQnYUWV+Rqp
n6+ojTElGJZXCnn7Sn0Oh3FidCxIeO/VIhgiuQ==
I'm attempting to use M2Crypto to verify a signature contained in an XML response returned from my SSO/SAML provider in my django/python app, but I can't seem to get it to work.
My XML response looks sort of like the second example here.
ETA: And here's a pastebin of my actual XML.
I'm using some code like this to attempt the verification:
def verify_signature(signed_info, cert, signature):
from M2Crypto import EVP, RSA, X509
x509 = X509.load_cert_string(base64.decodestring(cert), X509.FORMAT_DER)
pubkey = x509.get_pubkey().get_rsa()
verify_EVP = EVP.PKey()
verify_EVP.assign_rsa(pubkey)
verify_EVP.reset_context(md='sha1')
verify_EVP.verify_init()
verify_EVP.verify_update(signature.decode('base64'))
result = verify_EVP.verify_final(signed_info)
return result
I can successfully get the NameID from the response, and I know I'm successfully loading the certificate, because I can pull the issuer, etc. out of it.
As for the signature, though, I've tried hashing the passed in XML, encoding/not encoding various pieces, and passing in various bits of XML for the signed_info parameter (the SignedInfo tag, the Response tag, the whole thing), and I've tried using ElementTree/ElementC14N.py to ensure the XML is exclusively canonicalized, as the Transform implies should be done, but I'm not getting a positive result.
What am I missing here? Am I trying to validate against the wrong XML? Something wrong with my verification technique?
You were so close! You should pass to verify_update the signed_info, and then to verify_final pass the signature.
You do need to make sure that your signed_info is correctly canonicalized before verifying the signature.
Here is the correct method:
def verify_signature(signed_info, cert, signature):
from M2Crypto import EVP, RSA, X509
x509 = X509.load_cert_string(base64.decodestring(cert), X509.FORMAT_DER)
pubkey = x509.get_pubkey().get_rsa()
verify_EVP = EVP.PKey()
verify_EVP.assign_rsa(pubkey)
verify_EVP.reset_context(md='sha1')
verify_EVP.verify_init()
verify_EVP.verify_update(signed_info)
result = verify_EVP.verify_final(signature.decode('base64'))
return result
FYI, I was facing the same problem as you, and found no useful software for validating XML signatures in Python, so I wrote a new library: https://github.com/kislyuk/signxml.
from lxml import etree
from signxml import xmldsig
with open("saml2_idp_metadata.xml", "rb") as fh:
cert = etree.parse(fh).find("//ds:X509Certificate").text
root = ElementTree.fromstring(signature_data)
xmldsig(root).verify(x509_cert=cert)