I am working on implementing a web application that utilizes an API. During a response, the API server sends over a link to an X509 certificate (in PEM format, composed of a signing certificate and one or more intermediate certificates to a root CA certificate ) that I must download and use to do further verification.
Before using the certificate, I need to ensure that all certificates in the chain combine to create a chain of trust to a trusted root CA certificate (to detect and avoid any malicious requests). I am having a hard time doing this in python and my research into the subject is not yielding anything useful.
The certificate is easily grabbed and loaded using requests and M2Crypto
import requests
from M2Crypto import RSA, X509
mypem = requests.get('https://server.com/my_certificate.pem')
cert = X509.load_cert_string(str(mypem.text), X509.FORMAT_PEM)
However, validating the certificate chain is a problem. It is not feasible for me to write the certificate to disk in order to use a command line utility like openssl through something like subprocess, so it must be done through python. I also do not have any open connections and so using a connection based validation solution (like is mentioned in this answer / thread: https://stackoverflow.com/a/1088224/4984533) will not work either.
On another thread about this problem (at https://stackoverflow.com/a/4427081) abbot explains that m2crypto is incapable of doing this validation and says that he has written an extension to allow validation (using the module m2ext) but his patch never seems work, always returning false even though I know it's valid:
from m2ext import SSL
ctx = SSL.Context()
ctx.load_verify_locations(capath='/etc/ssl/certs/') # I have run c_rehash in this directory to generate a list of cert files with signature based names
if not ctx.validate_certificate(cert): # always happens
print('Invalid certificate!')
There's also this answer on a similar thread here https://stackoverflow.com/a/9007764/4984533 in which John Matthews claims to have a patch written which will do it, but unfortunately the patch link is now dead -- and anyway there is a comment on that thread stating that the patch did not work with openssl 0.9.8e.
All answers relating to validating a certificates chain of trust in python seem to either link to the dead patch or go back to m2ext.
Is there a simple, straightforward way to go about validating my certificates chain of trust in Python?
While the response of Avi Das is valid for the trivial case of verifying a single trust anchor with a single leaf certificate, it places trust in the intermediate certificate. That means that in the case where the intermediate is sent, as well as the client certificate, the entire chain is trusted.
Do not do this. The code found in pyOpenSSL's tests are flawed!
I found this thread on Python's cryptography-dev mailing lists (which links back to this answer): https://mail.python.org/pipermail/cryptography-dev/2016-August/000676.html
We see this code makes no distinction between the root_cert and the
intermediate. If we look at the documentation, add_cert itself adds a
trusted cert (maybe add_trusted_cert would be a better name?).
It includes examples on why this is a terrible idea. I can not stress this enough: verifying your chain by trusting the intermediates is similar to not performing any check at all.
Having said that, how do you verify a certificate chain in Python? The best alternative I found is https://github.com/wbond/certvalidator, which seems to do the job.
There are also some flawed alternatives:
https://github.com/alex/x509-validator, which explicitly states that it is not safe to use.
https://github.com/pyca/cryptography/issues/1660#issuecomment-75075319, which does some sort of checking, but I'm not sure it works as intended
This is the current state for some respectable Python cryptography libraries:
https://github.com/pyca/pyopenssl/pull/473 discusses a pull request to incorporate chain verification in pyopenssl, which has not been merged yet.
https://github.com/pyca/cryptography/issues/2381 is an issue requesting a similar feature to the cryptography module, but it is still unresolved
Both threads seem stale at this time.
I know this does not match the question's case, but: if you are building something using certificate validation in TLS sockets, just use the modules already available in Python. Never re-invent the wheel, especially regarding cryptography. Crypto is hard; the only easy thing about it is messing it up.
I looked into pyopenssl library and found this for certificate chain validation. The following example is from their tests and seem to do what you want, which is validating chain of trust to a trusted root certificate. Here are the relevant docs for X509Store and X509StoreContext
from OpenSSL.crypto import load_certificate, load_privatekey
from OpenSSL.crypto import X509Store, X509StoreContext
from six import u, b, binary_type, PY3
root_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIIC7TCCAlagAwIBAgIIPQzE4MbeufQwDQYJKoZIhvcNAQEFBQAwWDELMAkGA1UE
BhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdU
ZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwIhgPMjAwOTAzMjUxMjM2
NThaGA8yMDE3MDYxMTEyMzY1OFowWDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklM
MRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9U
ZXN0aW5nIFJvb3QgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAPmaQumL
urpE527uSEHdL1pqcDRmWzu+98Y6YHzT/J7KWEamyMCNZ6fRW1JCR782UQ8a07fy
2xXsKy4WdKaxyG8CcatwmXvpvRQ44dSANMihHELpANTdyVp6DCysED6wkQFurHlF
1dshEaJw8b/ypDhmbVIo6Ci1xvCJqivbLFnbAgMBAAGjgbswgbgwHQYDVR0OBBYE
FINVdy1eIfFJDAkk51QJEo3IfgSuMIGIBgNVHSMEgYAwfoAUg1V3LV4h8UkMCSTn
VAkSjch+BK6hXKRaMFgxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJJTDEQMA4GA1UE
BxMHQ2hpY2FnbzEQMA4GA1UEChMHVGVzdGluZzEYMBYGA1UEAxMPVGVzdGluZyBS
b290IENBggg9DMTgxt659DAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GB
AGGCDazMJGoWNBpc03u6+smc95dEead2KlZXBATOdFT1VesY3+nUOqZhEhTGlDMi
hkgaZnzoIq/Uamidegk4hirsCT/R+6vsKAAxNTcBjUeZjlykCJWy5ojShGftXIKY
w/njVbKMXrvc83qmTdGl3TAM0fxQIpqgcglFLveEBgzn
-----END CERTIFICATE-----
""")
intermediate_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICVzCCAcCgAwIBAgIRAMPzhm6//0Y/g2pmnHR2C4cwDQYJKoZIhvcNAQENBQAw
WDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAw
DgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwHhcNMTQw
ODI4MDIwNDA4WhcNMjQwODI1MDIwNDA4WjBmMRUwEwYDVQQDEwxpbnRlcm1lZGlh
dGUxDDAKBgNVBAoTA29yZzERMA8GA1UECxMIb3JnLXVuaXQxCzAJBgNVBAYTAlVT
MQswCQYDVQQIEwJDQTESMBAGA1UEBxMJU2FuIERpZWdvMIGfMA0GCSqGSIb3DQEB
AQUAA4GNADCBiQKBgQDYcEQw5lfbEQRjr5Yy4yxAHGV0b9Al+Lmu7wLHMkZ/ZMmK
FGIbljbviiD1Nz97Oh2cpB91YwOXOTN2vXHq26S+A5xe8z/QJbBsyghMur88CjdT
21H2qwMa+r5dCQwEhuGIiZ3KbzB/n4DTMYI5zy4IYPv0pjxShZn4aZTCCK2IUwID
AQABoxMwETAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBDQUAA4GBAPIWSkLX
QRMApOjjyC+tMxumT5e2pMqChHmxobQK4NMdrf2VCx+cRT6EmY8sK3/Xl/X8UBQ+
9n5zXb1ZwhW/sTWgUvmOceJ4/XVs9FkdWOOn1J0XBch9ZIiFe/s5ASIgG7fUdcUF
9mAWS6FK2ca3xIh5kIupCXOFa0dPvlw/YUFT
-----END CERTIFICATE-----
""")
untrusted_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICWDCCAcGgAwIBAgIRAPQFY9jfskSihdiNSNdt6GswDQYJKoZIhvcNAQENBQAw
ZjEVMBMGA1UEAxMMaW50ZXJtZWRpYXRlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzAeFw0xNDA4MjgwMjEwNDhaFw0yNDA4MjUwMjEwNDhaMG4xHTAbBgNV
BAMTFGludGVybWVkaWF0ZS1zZXJ2aWNlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqpJZygd+w1faLOr1
iOAmbBhx5SZWcTCZ/ZjHQTJM7GuPT624QkqsixFghRKdDROwpwnAP7gMRukLqiy4
+kRuGT5OfyGggL95i2xqA+zehjj08lSTlvGHpePJgCyTavIy5+Ljsj4DKnKyuhxm
biXTRrH83NDgixVkObTEmh/OVK0CAwEAATANBgkqhkiG9w0BAQ0FAAOBgQBa0Npw
UkzjaYEo1OUE1sTI6Mm4riTIHMak4/nswKh9hYup//WVOlr/RBSBtZ7Q/BwbjobN
3bfAtV7eSAqBsfxYXyof7G1ALANQERkq3+oyLP1iVt08W1WOUlIMPhdCF/QuCwy6
x9MJLhUCGLJPM+O2rAPWVD9wCmvq10ALsiH3yA==
-----END CERTIFICATE-----
""")
root_cert = load_certificate(FILETYPE_PEM, root_cert_pem)
intermediate_cert = load_certificate(FILETYPE_PEM, intermediate_cert_pem)
untrusted_cert = load_certificate(FILETYPE_PEM, untrusted_cert_pem)
store = X509Store()
store.add_cert(root_cert)
store.add_cert(intermediate_cert)
store_ctx = X509StoreContext(store, untrusted_cert)
print(store_ctx.verify_certificate())
Used the excellent explanation on https://duo.com/labs/research/chain-of-fools to adapt the previous solution (from avi_das), and added some comments in code. Basically we need to only add certificates to the store when they are trusted (e.g. root certificate) or verified/trusted by another (e.g. intermediate certificate). You cannot add all certificates to the store in one go, as you need to verify each certificate along the chain with the correct certificates in the store at that moment. The code comments below are less abstract.
import OpenSSL
from six import u, b, binary_type, PY3
root_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIIC7TCCAlagAwIBAgIIPQzE4MbeufQwDQYJKoZIhvcNAQEFBQAwWDELMAkGA1UE
BhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdU
ZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwIhgPMjAwOTAzMjUxMjM2
NThaGA8yMDE3MDYxMTEyMzY1OFowWDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklM
MRAwDgYDVQQHEwdDaGljYWdvMRAwDgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9U
ZXN0aW5nIFJvb3QgQ0EwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAPmaQumL
urpE527uSEHdL1pqcDRmWzu+98Y6YHzT/J7KWEamyMCNZ6fRW1JCR782UQ8a07fy
2xXsKy4WdKaxyG8CcatwmXvpvRQ44dSANMihHELpANTdyVp6DCysED6wkQFurHlF
1dshEaJw8b/ypDhmbVIo6Ci1xvCJqivbLFnbAgMBAAGjgbswgbgwHQYDVR0OBBYE
FINVdy1eIfFJDAkk51QJEo3IfgSuMIGIBgNVHSMEgYAwfoAUg1V3LV4h8UkMCSTn
VAkSjch+BK6hXKRaMFgxCzAJBgNVBAYTAlVTMQswCQYDVQQIEwJJTDEQMA4GA1UE
BxMHQ2hpY2FnbzEQMA4GA1UEChMHVGVzdGluZzEYMBYGA1UEAxMPVGVzdGluZyBS
b290IENBggg9DMTgxt659DAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GB
AGGCDazMJGoWNBpc03u6+smc95dEead2KlZXBATOdFT1VesY3+nUOqZhEhTGlDMi
hkgaZnzoIq/Uamidegk4hirsCT/R+6vsKAAxNTcBjUeZjlykCJWy5ojShGftXIKY
w/njVbKMXrvc83qmTdGl3TAM0fxQIpqgcglFLveEBgzn
-----END CERTIFICATE-----
""")
intermediate_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICVzCCAcCgAwIBAgIRAMPzhm6//0Y/g2pmnHR2C4cwDQYJKoZIhvcNAQENBQAw
WDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAklMMRAwDgYDVQQHEwdDaGljYWdvMRAw
DgYDVQQKEwdUZXN0aW5nMRgwFgYDVQQDEw9UZXN0aW5nIFJvb3QgQ0EwHhcNMTQw
ODI4MDIwNDA4WhcNMjQwODI1MDIwNDA4WjBmMRUwEwYDVQQDEwxpbnRlcm1lZGlh
dGUxDDAKBgNVBAoTA29yZzERMA8GA1UECxMIb3JnLXVuaXQxCzAJBgNVBAYTAlVT
MQswCQYDVQQIEwJDQTESMBAGA1UEBxMJU2FuIERpZWdvMIGfMA0GCSqGSIb3DQEB
AQUAA4GNADCBiQKBgQDYcEQw5lfbEQRjr5Yy4yxAHGV0b9Al+Lmu7wLHMkZ/ZMmK
FGIbljbviiD1Nz97Oh2cpB91YwOXOTN2vXHq26S+A5xe8z/QJbBsyghMur88CjdT
21H2qwMa+r5dCQwEhuGIiZ3KbzB/n4DTMYI5zy4IYPv0pjxShZn4aZTCCK2IUwID
AQABoxMwETAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBDQUAA4GBAPIWSkLX
QRMApOjjyC+tMxumT5e2pMqChHmxobQK4NMdrf2VCx+cRT6EmY8sK3/Xl/X8UBQ+
9n5zXb1ZwhW/sTWgUvmOceJ4/XVs9FkdWOOn1J0XBch9ZIiFe/s5ASIgG7fUdcUF
9mAWS6FK2ca3xIh5kIupCXOFa0dPvlw/YUFT
-----END CERTIFICATE-----
""")
untrusted_cert_pem = b("""-----BEGIN CERTIFICATE-----
MIICWDCCAcGgAwIBAgIRAPQFY9jfskSihdiNSNdt6GswDQYJKoZIhvcNAQENBQAw
ZjEVMBMGA1UEAxMMaW50ZXJtZWRpYXRlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzAeFw0xNDA4MjgwMjEwNDhaFw0yNDA4MjUwMjEwNDhaMG4xHTAbBgNV
BAMTFGludGVybWVkaWF0ZS1zZXJ2aWNlMQwwCgYDVQQKEwNvcmcxETAPBgNVBAsT
CG9yZy11bml0MQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExEjAQBgNVBAcTCVNh
biBEaWVnbzCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqpJZygd+w1faLOr1
iOAmbBhx5SZWcTCZ/ZjHQTJM7GuPT624QkqsixFghRKdDROwpwnAP7gMRukLqiy4
+kRuGT5OfyGggL95i2xqA+zehjj08lSTlvGHpePJgCyTavIy5+Ljsj4DKnKyuhxm
biXTRrH83NDgixVkObTEmh/OVK0CAwEAATANBgkqhkiG9w0BAQ0FAAOBgQBa0Npw
UkzjaYEo1OUE1sTI6Mm4riTIHMak4/nswKh9hYup//WVOlr/RBSBtZ7Q/BwbjobN
3bfAtV7eSAqBsfxYXyof7G1ALANQERkq3+oyLP1iVt08W1WOUlIMPhdCF/QuCwy6
x9MJLhUCGLJPM+O2rAPWVD9wCmvq10ALsiH3yA==
-----END CERTIFICATE-----
""")
# load certificates
root_cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, root_cert_pem)
intermediate_cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, intermediate_cert_pem)
untrusted_cert = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, untrusted_cert_pem)
# Trust the root certificate
store = OpenSSL.crypto.X509Store()
store.add_cert(root_cert)
# only add intermediate if it can be verified by the root
store_ctx = OpenSSL.crypto.X509StoreContext(store, intermediate_cert)
print(store_ctx.verify_certificate())
store.add_cert(intermediate_cert)
# now that root and intermediate are trused, you can verify the end certificate using the store
store_ctx = OpenSSL.crypto.X509StoreContext(store, untrusted_cert)
print(store_ctx.verify_certificate())
Awesome answer by Paul-Armand and the chain of fools article. This works for me. Note: I need to set the validation date of the store appropriately.
However I found a way to avoid adding the untrusted certificate as trusted. I checked the pyopenssl code (https://github.com/pyca/pyopenssl/blob/main/src/OpenSSL/crypto.py) and found there's a "chain" parameter on X509StoreContext, so I tried to just put the intermediate certificate in there like so:
# .... paste Paul-Armand's example here ....
store = OpenSSL.crypto.X509Store()
store.add_cert(root_cert)
# Set time to reuse certs from Paul-Armand's example
validation_date = datetime.datetime.strptime("2016-01-01", "%Y-%m-%d")
store.set_time(validation_date)
# This is the change, added param [intermediate_cert]
store_ctx = OpenSSL.crypto.X509StoreContext(store, untrusted_cert, [intermediate_cert])
print("Untrusted chain example")
try:
store_ctx.verify_certificate()
print("Verify - OK")
except OpenSSL.crypto.X509StoreContextError:
print("Verify failed - bad")
It all validates properly for me and is closer to what I use on the command line with OpenSSL (-untrusted intermediate.pem). Sure wish I figured out pyopenssl in January 2019, would have saved me years of using the "certvalidate" python alternative. I revisited because I'm moving to EdDSA25519 keys and certvalidate does not support them.
Related
I have the following certificate, as returned by ssl.enum_certificates:
import ssl
cert = ssl.enum_certificates("MY")[1][0]
print(cert)
Result:
b'0\x82\x02\xa60\x82\x01\x8e\xa0\x03\x02\x01\x02\x02\x11\x00\xff\xe1C\xed\x8c\xa5UA\xad\x870"\x0f\xc3+\xc30\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x000&1$0"\x06\x03U\x04\x03\x13\x1bXBL Client IPsec Issuing CA0\x1e\x17\r211013223833Z\x17\r211014223833Z0\x1b1\x190\x17\x06\x03U\x04\x03\x0c\x10F900E607B75850AE0\x81\x9f0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x81\x8d\x000\x81\x89\x02\x81\x81\x00\xb8\xaeE\x04\x9c\x02\xe7\xe1\xe7g\xd9\xad\xfc\xbeg\x83\xbc\x7f\xd7\xcc\xc7m\x1d\x02\xe3\xff\x93\xe3\x81\xe9%\x8b\xdc]%\xd2\x8b!!;\xd0^\xdc\\\xd47\x07\x05g\xe2\xdeR\x8d\xc8\xea\xd4\xc5\xdc\x1c]\xa7L\xab\x92\x1f9\x02\x0f\x83\x12\x99\xd1^\xe2\xfdm\xcfU_8N\x94T\x92\x94\x8e\xa3\xe9\xa3\xc3\xf6\x04\xd9\x1b\xa1\xa5\t\xce\xa6~_\xc9\xc1DH\xd0j\x95\x84\x8e\xef\xd9o\xd7\xfe\x91Yz\x1a\xa2\xbe\xc5\x06Y\x9bc\x8bU\x02\x03\x01\x00\x01\xa3^0\\0\x0e\x06\x03U\x1d\x0f\x01\x01\xff\x04\x04\x03\x02\x01\xa60)\x06\x03U\x1d%\x04"0 \x06\x08+\x06\x01\x05\x05\x07\x03\x02\x06\x08+\x06\x01\x05\x05\x07\x03\x05\x06\n+\x06\x01\x04\x01\x827x\x04\x040\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14\xc1\xfe7p|Z9\x12&\xec\xc0\x94\xcd\x80\x94\xb1\xa8u\xe4\xd80\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x00\x03\x82\x01\x01\x00.\xaakE7h\x95\xc7\xa8\x17\xce\x1d\xf2\xc5\x9d[y8g\xb44\x12\xcd\xbb\xdak\xdbF9\r\xef\xb8\x15\x01W[\xa0 \xcf\xa8\x19ML\xa5\x12\x00\xcaF\xb1\xa8\x91\xd9\xde\xc6\xe1.\xe4\xa0\xfe\\\xde\x90\x0c#\xaf\x7fS\x13\xd0\xde\xd9Q\xe2\xa6J\x98\x0f\xfd\x11\x7f\xf7\xed\xf8\xfeI-\x89o\x1b\xc1\xfeF\xbb\x05\x8d\xd0{\x9fc2\xc9\xcf\xcfe\x04\x1e+.\xbe\x1b\x90<~##\xa00\xa4\\1`\xd3\x84\x14\xf2\xe5\x16"\xda\x80\xa9\x90d\x92\xf5\x82\xc9\x87\xdf\xc4x\xc6\xde+\x17\x10\xc5j\x96\xed\x8e\xdfp\xb3\x04\x9e\xeb\xc6\xe6{\x1ee\x1c\xeeN\xad\xe4K7xb\xd5\x99\xb9\xf4pb{\x13\xa1\xa4:\x1cF\xef\xe7Hf*\x96!k\x0c\xda\x08*x\x13`\xd9Ju`e\xd8W\xf1;B\xa4\xb8B\xe5Y\xa5\n\x9a\xdb\xc7\xb5\xee+\xfe~\xac\xc1\x85\xfex\x1d\x8f\xc3s\xbc\xdd\xab\x1bo\xe0a\x98\xba\xa5\x98z\x03\xee}c[\xe1\xcb8\x1a\xb4\x80'
How to convert this output to a human-readable format? (dict preferred).
This is pretty much the same question as this one, but here I have the certificate in bytes format. I could call ssl.DER_cert_to_PEM_cert and try to go from there, but I'd rather avoid this additional step.
There is a library called cryptography which can do exactly this:
pip install cryptography
Then you can create an x509 certificate object and print whatever results you are after:
import ssl
from cryptography import x509
cert = ssl.enum_certificates("MY")[1][0]
decoded_cert = x509.load_der_x509_certificate(cert)
print(decoded_cert.subject)
Sometimes you just have to add additional steps/libraries to your code. It's not a reasonable expectation to only use native functionality or be expected to reinvent the wheel.
so I was working with verify a X509 certificate to my IoT Hub in order to send/receive message from my application. However it keeps throwing out the ssl.SSLError: [SSL] PEM lib (_ssl.c:3833) error.
I have the correct certificate, private key and the pass phrase. So I went over to the python github to check out what does the error means, and the line 3833 in the _ssl.c file means
r = SSL_CTX_use_certificate_chain_file(self->ctx,
PyBytes_AS_STRING(certfile_bytes));
PySSL_END_ALLOW_THREADS_S(pw_info.thread_state);
if (r != 1) {
if (pw_info.error) {
ERR_clear_error();
/* the password callback has already set the error information */
}
else if (errno != 0) {
ERR_clear_error();
PyErr_SetFromErrno(PyExc_OSError);
}
else {
_setSSLError(NULL, 0, __FILE__, __LINE__); <--- THIS IS LINE 3833
}
goto error;
}
Does that means my certificate is wrong? My certificate currently is the location like C:/Certificate/MyCertName.pfx
Thank you for reading and any helps is appreciated!
Source of the _ssl.c in python 3.7.4: https://github.com/python/cpython/blob/v3.7.4/Modules/_ssl.c
This means that SSL_CTX_use_certificate_chain_file (which is a part of OpenSSL) returned an error code and neither of the two typical cases describe the situation, so Python's code cannot tell you more about it.
So the only lead is to check what exactly that function receives and read its documentation (and source code if that's not enough) to try to figure out why it fails. If that's not enough, you'll have to run the process under a C debugger to check the same in vivo.
My (blind) guess is that the certificate file might be of an incorrect/unsupported format/cipher, or not contain the correct certificate chain as required by the TLS standard. Specifically, the documentation says:
SSL_CTX_use_certificate_chain_file() loads a certificate chain from
file into ctx. The certificates must be in PEM format and must be
sorted starting with the subject's certificate (actual client or
server certificate), followed by intermediate CA certificates if
applicable, and ending at the highest level (root) CA.
There is already a lot of questions out there but the problem is, none of them have sufficient answers how to do it, especially when using python3.
Basically, I want to read JAR/APK certificates, like this one: Link to ASN1 Decoder, with Android Test Signing Key
There are now several alternatives:
pyasn1: seems to work, but only can parse the raw ASN.1 format
M2Crypto: only works on py2
Chilkat: Not free, although CkCert seems to be free
cryptography: Can not load the certificate, as the X509 certificate is inside the PKCS#7 container
I found a way to use pyasn1 to unpack the cert from the pkcs#7 message, then use cryptography to read it:
from pyasn1.codec.der.decoder import decode
from pyasn1.codec.der.encoder import encode
from cryptography import x509
from cryptography.hazmat.backends import default_backend
cdata = open("CERT.RSA", "rb").read()
cert, rest = decode(cdata)
# The cert should be located there
realcert = encode(cert[1][3])
realcert = realcert[2 + (realcert[1] & 0x7F) if realcert[1] & 0x80 > 1 else 2:] # remove the first DER identifier from the front
x509.load_der_x509_certificate(realcert, default_backend())
which gives
<Certificate(subject=<Name([<NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.6, name=countryName)>, value='US')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.8, name=stateOrProvinceName)>, value='California')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.7, name=localityName)>, value='Mountain View')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.10, name=organizationName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.11, name=organizationalUnitName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.3, name=commonName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=1.2.840.113549.1.9.1, name=emailAddress)>, value='android#android.com')>])>, ...)>
Is there no other way to have it clean and tidy?
There are now libraries to do this in pure python. One is asn1crypto: https://github.com/wbond/asn1crypto#readme
This is also impemented in androguard, including examples how to use it: https://androguard.readthedocs.io/en/latest/intro/certificates.html
I could not find a suitable way to get the name of the signature algorithm used for a certificate that is received by an SSLSocket. I'm aware you can pull the bytes of the peer's cert with SSLSocket.getpeercert(True) but I don't know what to do with it past that. PyOpenSSL doesn't seem to have an easy interface to load an X509 from it's byte content.
I would like to know this information as certificates with SHA1 are not allowed to have expiration times after January 1st, 2017 and this is not checked by Python's SSLSocket.do_handshake() implementation.
After the very helpful comment pointing me in the right direction, you can use the cryptography module to load the DER X509 format that the SSLSocket.getpeercert() function outputs. From this Certificate instance you can also access many other fields about the certificate such as expiration time. Here is the way that I am doing it right now:
import cryptography.x509
import cryptography.hazmat.backends.openssl
import ssl
sock = ssl.SSLSocket()
# Wrap a socket, connect, handshake, etc etc...
cert = cryptography.x509.load_der_x509_certificate(
sock.getpeercert(True),
cryptography.hazmat.backends.openssl.backend
)
print(cert.signature_hash_algorithm.name) # This prints "sha1".
I want to be able to sign some data, and later verify the signature; I also want to use a CRL.
Right now, I'm using a small helper executable (written in C) to do all of this; I would prefer to do the entire thing in Python, since that's what the rest of the system is written in.
In the helper executable, I
Create a store
Load both the system root CA and our root CA into a store
Load the CRL using:
crl = d2i_X509_CRL_fp(crl, NULL);
crl = PEM_read_X509_CRL(crl, NULL, NULL, NULL);
X509_VERIFY_PARAM *param = X509_VERIFY_PARAM_new();
X509_VERIFY_PARAM_set_flags(param, X509_V_FLAG_CRL_CHECK);
X509_STORE_set1_param(store, param);
X509_VERIFY_PARAM_free(param);
X509_STORE_add_crl(store, crl);
Read the certificate I want to verify against
Verify said certificate against the store I set up before
And then verify the signature against the certificate
I see that Python3.4 has an ability to load a CRL in OpenSSL.crypto, but then I don't see how I'd go about using it.
I hate to think I'm missing something obvious, but of course that's possible.
So... am I? If so, what?
Added: It looks like I have an outdated version of pyopenssl -- version 16.1.0 added a add_crl method to the X509Store class, so that may solve my problem. (And looks like, the obvious thing I missed was checking the source repository!)