We Keep Coding

Converting Certificate Signing Request to type cryptography.x509.base.CertificateSigningRequestBuilder to be ready for the signature

Step 1. I have generated a certificate signing request (csr) using the cryptography library.
from cryptography import x509
from cryptography.x509.oid import NameOID
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives import serialization
csr = x509.CertificateSigningRequestBuilder().subject_name(x509.Name([
# Provide various details about who we are.
x509.NameAttribute(NameOID.COUNTRY_NAME, u"US"),
x509.NameAttribute(NameOID.STATE_OR_PROVINCE_NAME, u"California")
])).add_extension(
x509.SubjectAlternativeName([
# Describe what sites we want this certificate for.
x509.DNSName(u"mysite.com"),
]),
critical=False
)
type(csr) = cryptography.x509.base.CertificateSigningRequestBuilder
Step 2. Then I signed the certificate
cert = csr.sign(private_key, hashes.SHA256())
It now has the type:
type(cert) = CertificateSigningRequest
Step 3. The csr needs to be serialized, and the data can be written to a file to be saved or sent over the network
serialized = cert.public_bytes(
serialization.Encoding.PEM
)
type:
type(serialized) = bytes
Save the certificate
with open('cert_name.cert', 'wb') as f:
f.write(serialized)
The issue arises when I need to read the certificate for another signature.
with open('cert_name.cert', 'rb') as f:
load_cert = f.read()
type(load_cert) = byte
I guess, as in step 2, the load_cert needs to be of type cryptography.x509.base.CertificateSigningRequestBuilder to be signed, then, as in step 3, it will be serialized to be saved again.
How could I convert the load_cert to type cryptography.x509.base.CertificateSigningRequestBuilder to be ready for the signature. Or Does anyone have an alternative to the process described here ?

I'm not sure why you are trying to sign a CSR multiple times, it should be signed once by you and then verified by a CA who will give you a certificate. This tutorial might clear things up for you.
To deserialise your CSR you should do something like:
with open('cert_name.cert', 'rb') as f:
data = f.read()
load_cert = cryptography.x509.load_pem_x509_csr(data)
and you can verify that the deserialised CSR already has a signature with:
load_cert.is_signature_valid

base64 HMAC with SHA256 in Python

I am having a hard time creating a signature.
I am needing to make a signature using HMAC with SHA256 using a Checkout Request JSON and a secret key. I need to do it by concatenating signature, pipe character (|) and Checkout Request JSON and then encoding it with BASE64.
This is a formula I found in the documentations:
$signed_checkout_request = base64( hmac_sha256( $checkout_request, $private_key ) + "|" + $checkout_request )
I have made this based on some online code:
import hashlib
import hmac
import base64
checkout_request = '{"charge":{"amount":499,"currency":"EUR"}}'.encode('utf-8');
private_key = b'44444444444';
digest = hmac.new(private_key, msg=checkout_request, digestmod=hashlib.sha256).digest()
signature = base64.b64encode(digest).decode()
However I am not sure how to get the "|" into it. I am also not sure if I am even on the right track if I am honest... I don't have much experience in this section and I have failed at googling.

private_key = 'blahblahblah'
checkout_request = json.dumps({"charge":{"amount":4999,"currency":"EUR"}}, sort_keys=True, separators=(",", ":"))
digest = hmac.new(private_key.encode(), msg=checkout_request.encode(), digestmod=hashlib.sha256,).hexdigest()
signature = base64.b64encode((digest + "|" + checkout_request).encode()).decode()
I was able to get it to work with that :)

How to create x X.509 certificate without direct access to the private key?

Working with an HSM I can download the public key of a RSA 4096, and can use the HSM to encrypt a blob using the corresponding private key, which I can't download.
I'm trying to find a reference to how I can use the HSM apis, to build the X509 certificate myself. I could find the example below in python, wrapping up openssl libs.
It does mostly all, but
from OpenSSL import crypto, SSL
from socket import gethostname
from pprint import pprint
from time import gmtime, mktime
CERT_FILE = "selfsigned.crt"
KEY_FILE = "private.key"
def create_self_signed_cert():
# create a key pair
k = crypto.PKey()
k.generate_key(crypto.TYPE_RSA, 1024)
# create a self-signed cert
cert = crypto.X509()
cert.get_subject().C = "UK"
cert.get_subject().ST = "London"
cert.get_subject().L = "London"
cert.get_subject().O = "Dummy Company Ltd"
cert.get_subject().OU = "Dummy Company Ltd"
cert.get_subject().CN = gethostname()
cert.set_serial_number(1000)
cert.gmtime_adj_notBefore(0)
cert.gmtime_adj_notAfter(10*365*24*60*60)
cert.set_issuer(cert.get_subject())
cert.set_pubkey(k)
cert.sign(k, 'sha1')
open(CERT_FILE, "wt").write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
open(KEY_FILE, "wt").write(crypto.dump_privatekey(crypto.FILETYPE_PEM, k))
create_self_signed_cert()
Mostly what I would expect to be possible is to replace the line
cert.sign(k, 'sha1')
by the corresponding calls to the APIs of my HSM with "an export" of the structure of "cert". Is it possible at all ?

So after one day of research I found a satisfactory answer at Adding a signature to a certificate . While there seems to be some Python examples out how to build the certificate through Python with pyasn1, the most robust seems to be the ones in java using boucycastle libraries. I also found a strong answer how to incorporate it with openssl here How to generate certificate if private key is in HSM? (mostly points 3 and 4), but the approach is far beyond my capacities.

When you want someone else, like your HSM, to sign a certificate for you the standard way to do that is to generate a Certificate Signing Request (CSR) and send the CSR to the signer. The CSR can contain all of the fields that you can set when making a certificate, and those values will be reflected in the resulting certificate.
While you can do this with the OpenSSL library that you are using, see X509Req, their documentation (and I quote) "strongly suggests" that you use the cryptography library instead. (They're maintained by the same group of people.)
In the cryptography library you use the CertificateSigningRequestBuilder to make a CSR. To quote from their docs:
>>> from cryptography import x509
>>> from cryptography.hazmat.primitives import hashes
>>> from cryptography.hazmat.primitives.asymmetric import rsa
>>> from cryptography.x509.oid import AttributeOID, NameOID
>>> private_key = rsa.generate_private_key(
... public_exponent=65537,
... key_size=2048,
... )
>>> builder = x509.CertificateSigningRequestBuilder()
>>> builder = builder.subject_name(x509.Name([
... x509.NameAttribute(NameOID.COMMON_NAME, u'cryptography.io'),
... ]))
>>> builder = builder.add_extension(
... x509.BasicConstraints(ca=False, path_length=None), critical=True,
... )
>>> builder = builder.add_attribute(
... AttributeOID.CHALLENGE_PASSWORD, b"changeit"
... )
>>> request = builder.sign(
... private_key, hashes.SHA256()
... )
>>> isinstance(request, x509.CertificateSigningRequest)
True
To render out the CSR in the format usually used for sending them around:
>>> from cryptography.hazmat.primitives import serialization
>>> request.public_bytes(serialization.Encoding.PEM)

What is the necessary info need by signature verification?

I wrote a test demo of signature & verification complete process base on rsa, which helps me to figure out the logic of the process.
# https://cryptography.io/en/latest/hazmat/primitives/asymmetric/rsa
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives.asymmetric import rsa
# Preparation phase
# Generate key pairs
# private_key contains the both private key and public key
private_key = rsa.generate_private_key(
public_exponent=65537,
key_size=2048,
backend=default_backend()
)
# Serilize the keys
from cryptography.hazmat.primitives import serialization
pem = private_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.PKCS8,
encryption_algorithm=serialization.BestAvailableEncryption(b'mypassword')
)
with open('private-key.pem', 'wb') as f:
f.write(pem)
f.close()
public_key = private_key.public_key()
pem = public_key.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo
)
with open('public-key.pem', 'wb') as f:
f.write(pem)
f.close()
# Signer
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import padding
from cryptography.hazmat.primitives.asymmetric import utils
with open('private-key.pem', 'rb') as f:
private_key = serialization.load_pem_private_key(
f.read(),
password=b'mypassword',
backend=default_backend()
)
chosen_hash = hashes.SHA256()
hasher = hashes.Hash(chosen_hash, default_backend())
hasher.update(b"data & ")
hasher.update(b"more data")
digest = hasher.finalize()
signature = private_key.sign(
digest,
padding.PSS(
mgf=padding.MGF1(hashes.SHA256()),
salt_length=padding.PSS.MAX_LENGTH
),
utils.Prehashed(chosen_hash)
)
with open('signature', 'wb') as f:
f.write(signature)
f.close()
# Verifier
chosen_hash = hashes.SHA256()
hasher = hashes.Hash(chosen_hash, default_backend())
hasher.update(b"data & ")
hasher.update(b"more data")
digest = hasher.finalize()
hasher1 = hashes.Hash(chosen_hash, default_backend())
hasher1.update(b"data & more data")
digest1 = hasher1.finalize()
print(digest == digest1)
with open('signature', 'rb') as f:
signature = f.read()
with open('public-key.pem', 'rb') as f:
public_key = serialization.load_pem_public_key(
f.read(),
backend=default_backend()
)
if isinstance(public_key, rsa.RSAPublicKey):
public_key.verify(
signature,
digest,
padding.PSS(
mgf=padding.MGF1(hashes.SHA256()),
salt_length=padding.PSS.MAX_LENGTH
),
utils.Prehashed(chosen_hash)
)
Question:
Does the padding type(eg. PSS) having to be known as input when verification?
But in openssl CLI Generate EC KeyPair from OpenSSL command line
openssl dgst -sha256 -verify public.pem -signature msg.signature.txt msg.digest.txt
Why here didn't mention the padding? I think no matter the key pairs algorithm(ECC or RSA) is different or not, the inputs parameter of (standard?) verification method shall be the same.
Another question, I saw in python having isinstance(public_key, rsa.RSAPublicKey) can find out the algorithm of the keys.
Does the algorithm type also necessary for the verification method?
Like inside the lib may having such ecc_verify rsa_verify methods.
BTW for my understanding, the verify method parameter(same as openssl CLI):
public key
hash type
signature

Does the padding type(eg. PSS) having to be known as input when verification?
Yes, the padding is a required configuration parameter that should be known beforehand.
Why here didn't mention the padding?
ECDSA signature verification doesn't require padding; it is significantly different from RSA signature generation / verification.
There is only one used standard for it, as was the case previously for RSA (PKCS#1 v1.5 padding). PSS was only added later in the RSA PKCS#1 standards when version 2.0 was published.
Does the algorithm type also necessary for the verification method?
Although there is probably not a direct attack, you should assume that the verification algorithm is known in advance, just like with the padding method. RSA as described in PKCS#1 has at least two signature generation functions and there are more in other standards - although those are not common at all.
BTW for my understanding, the verify method parameter(same as openssl CLI):
As you've already seen, RSA PSS uses two hash functions, one for hashing the input data and one for the internal MGF1 function used for padding. So there is not one hash type, but two. The hash types are not necessarily the same and implementations differ on how the MGF1 hash is determined (specifying it explicitly, as you do, is for the best).

Twisted Python, using ssl.CertificateOptions when switching from plain text to secure connection

Following advice from Jean-Paul Calderone here on SO, I'm trying to modify the twisted "starttls_server" sample below to support the use of ssl.ClientCertificateOptions, to allow me to specify my private key, certificate, and trusted roots, as per http://twistedmatrix.com/documents/14.0.0/api/twisted.internet.ssl.CertificateOptions.html
from twisted.internet import ssl, protocol, defer, task, endpoints
from twisted.protocols.basic import LineReceiver
from twisted.python.modules import getModule
class TLSServer(LineReceiver):
def lineReceived(self, line):
print("received: " + line)
if line == "STARTTLS":
print("-- Switching to TLS")
self.sendLine('READY')
self.transport.startTLS(self.factory.options)
def main(reactor):
certData = getModule(__name__).filePath.sibling('server.pem').getContent()
cert = ssl.PrivateCertificate.loadPEM(certData)
factory = protocol.Factory.forProtocol(TLSServer)
factory.options = cert.options()
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)
endpoint.listen(factory)
return defer.Deferred()
if __name__ == '__main__':
import starttls_server
task.react(starttls_server.main)
My understanding is that I effectively need to replace the cert = ssl.PrivateCertificate... and cert.options = ssl.PrivateCertificate.... lines with something like certopts = ssl.CertificateOptions(privateKey=pKeyData, certificate=certData, trustRoot=caCertsData) (having read the appropriate files in to certData, caCertsData, and pKeyData) and then pass this in to factory.options - but without pasting every variant of code I've tried, I've yet to work this out correctly - my efforts have produced varying results from the classic "OpenSSL.crypto.Error: []" - through to seemingly just dumping the contents of my 3 PEM files to screen and exiting!
Can anyone enlighten me? Thank you :)

cert.options() is already returning a CertificateOptions. The problem is that options takes authorities (as Certificate objects) as positional args, and doesn't let you pass through all the other configuration values through, so you probably want to construct a CertificateOptions directly.
Just change the factory.options = cert.options() line to factory.options = ssl.CertificateOptions(...).
However, CertificateOptions takes a pyOpenSSL PKey object as its privateKey, not the key data. So you'll need to use OpenSSL APIs to load that key, or you can extract it from a PrivateCertificate.
If you read the signature of CertificateOptions very carefully, the required types should be fairly clear. You may also need to consult the pyOpenSSL documentation as well.