I am extracting SSL/TLS certificate fields from serverhello packet using scapy-ssl_tls library which I installed using pip.
The problem is, I'm not able to figure out a way to extract values from ASN1 encoded fields:
sign_algo2: <ASN1_OID['.1.2.840.113549.1.1.11']>
sa2_value: <ASN1_NULL[0L]>
not_before: <ASN1_UTC_TIME['170321131500Z']>
not_after: <ASN1_UTC_TIME['200321131500Z']>
pubkey_algo: <ASN1_OID['.1.2.840.113549.1.1.1']>
version: <ASN1_INTEGER[2L]>
sn: <ASN1_INTEGER[6348220899422160075L]>
sign_algo: <ASN1_OID['.1.2.840.113549.1.1.11']>
pubkey: <ASN1_BIT_STRING['\x000\x']>
I've dug out scapy.layers.ssl_tls, ssl_tls_crypto, scapy.layers.x509 modules but couldn't get any hint to decode it. I also tried using Asn1Value.load() from asn1crypto.core package but it fails with following error:
TypeError: encoded_data must be a byte string, not scapy.asn1.asn1.ASN1_UTC_TIME
It'd be great if anyone could help me getting this resolved using scapy's native decoding preferably or any other way possible.
Note: Please note that I've to extract these values from SSL/TLS serverhello packets which I'm reading from a pcap file as I need other fields from packet headers as well. I know many solutions exist on stackoverflow or wireshark/tshark for extracting/reading certificates from .pem files or possibly from .der files (after having it exported from wireshark explicitly), they don't work in my case as I need a solution which works around extracting certificates or certificate fields from packets.
If you have a DER-encoded byte string of the X.509 certificate, the correct way to use asn1crypto would be:
from asn1crypto import x509
cert = x509.Certificate.load(der_byte_string)
print(cert.native)
It would seem from the error message, you have tried to pass some of Python object that represents an ASN.1 value.
Make sure you are passing asn1crypto the byte string, not some internal scapy object. May be you need to cast the latter into a byte string.
Alternatively, this tool is designed to decode X.509 certs into a tree of Python objects. You also need to feed it either a string (Python2) or bytes (Python 3).
This has been discussed in scapy-ssl_tls issue #116 which describes basic handling of scapy ASN.1 fields:
...
# resp holds the raw socket response to a client_hello
tls_response = TLS(resp)
tls_response.show() # show the structure
# iterate all certificates
for cert in tls_response[TLSCertificateList].payload.certificates:
# .payload as the structure is [...][TLSCertificateList][TLS10Certificate].certificates = [x509Cert,x509Cert,...]
# we'll have a TLSCertificateList object at this point; get the scapy X509Cert Object
tlscert = cert[X509Cert]
print repr(str(tlscert.sign_algo)) # raw bytes -> '\x06\t*\x86H\x86\xf7\r\x01\x01\x0b'
print repr(tlscert.sign_algo) # <ASN1_OID['.1.2.840.113549.1.1.11']>
print tlscert.sign_algo.val # 1.2.840.113549.1.1.11
print repr(tlscert.version) # <ASN1_INTEGER[2L]>
print tlscert.version.val # 2
Related
I have the following certificate, as returned by ssl.enum_certificates:
import ssl
cert = ssl.enum_certificates("MY")[1][0]
print(cert)
Result:
b'0\x82\x02\xa60\x82\x01\x8e\xa0\x03\x02\x01\x02\x02\x11\x00\xff\xe1C\xed\x8c\xa5UA\xad\x870"\x0f\xc3+\xc30\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x000&1$0"\x06\x03U\x04\x03\x13\x1bXBL Client IPsec Issuing CA0\x1e\x17\r211013223833Z\x17\r211014223833Z0\x1b1\x190\x17\x06\x03U\x04\x03\x0c\x10F900E607B75850AE0\x81\x9f0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x81\x8d\x000\x81\x89\x02\x81\x81\x00\xb8\xaeE\x04\x9c\x02\xe7\xe1\xe7g\xd9\xad\xfc\xbeg\x83\xbc\x7f\xd7\xcc\xc7m\x1d\x02\xe3\xff\x93\xe3\x81\xe9%\x8b\xdc]%\xd2\x8b!!;\xd0^\xdc\\\xd47\x07\x05g\xe2\xdeR\x8d\xc8\xea\xd4\xc5\xdc\x1c]\xa7L\xab\x92\x1f9\x02\x0f\x83\x12\x99\xd1^\xe2\xfdm\xcfU_8N\x94T\x92\x94\x8e\xa3\xe9\xa3\xc3\xf6\x04\xd9\x1b\xa1\xa5\t\xce\xa6~_\xc9\xc1DH\xd0j\x95\x84\x8e\xef\xd9o\xd7\xfe\x91Yz\x1a\xa2\xbe\xc5\x06Y\x9bc\x8bU\x02\x03\x01\x00\x01\xa3^0\\0\x0e\x06\x03U\x1d\x0f\x01\x01\xff\x04\x04\x03\x02\x01\xa60)\x06\x03U\x1d%\x04"0 \x06\x08+\x06\x01\x05\x05\x07\x03\x02\x06\x08+\x06\x01\x05\x05\x07\x03\x05\x06\n+\x06\x01\x04\x01\x827x\x04\x040\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14\xc1\xfe7p|Z9\x12&\xec\xc0\x94\xcd\x80\x94\xb1\xa8u\xe4\xd80\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x00\x03\x82\x01\x01\x00.\xaakE7h\x95\xc7\xa8\x17\xce\x1d\xf2\xc5\x9d[y8g\xb44\x12\xcd\xbb\xdak\xdbF9\r\xef\xb8\x15\x01W[\xa0 \xcf\xa8\x19ML\xa5\x12\x00\xcaF\xb1\xa8\x91\xd9\xde\xc6\xe1.\xe4\xa0\xfe\\\xde\x90\x0c#\xaf\x7fS\x13\xd0\xde\xd9Q\xe2\xa6J\x98\x0f\xfd\x11\x7f\xf7\xed\xf8\xfeI-\x89o\x1b\xc1\xfeF\xbb\x05\x8d\xd0{\x9fc2\xc9\xcf\xcfe\x04\x1e+.\xbe\x1b\x90<~##\xa00\xa4\\1`\xd3\x84\x14\xf2\xe5\x16"\xda\x80\xa9\x90d\x92\xf5\x82\xc9\x87\xdf\xc4x\xc6\xde+\x17\x10\xc5j\x96\xed\x8e\xdfp\xb3\x04\x9e\xeb\xc6\xe6{\x1ee\x1c\xeeN\xad\xe4K7xb\xd5\x99\xb9\xf4pb{\x13\xa1\xa4:\x1cF\xef\xe7Hf*\x96!k\x0c\xda\x08*x\x13`\xd9Ju`e\xd8W\xf1;B\xa4\xb8B\xe5Y\xa5\n\x9a\xdb\xc7\xb5\xee+\xfe~\xac\xc1\x85\xfex\x1d\x8f\xc3s\xbc\xdd\xab\x1bo\xe0a\x98\xba\xa5\x98z\x03\xee}c[\xe1\xcb8\x1a\xb4\x80'
How to convert this output to a human-readable format? (dict preferred).
This is pretty much the same question as this one, but here I have the certificate in bytes format. I could call ssl.DER_cert_to_PEM_cert and try to go from there, but I'd rather avoid this additional step.
There is a library called cryptography which can do exactly this:
pip install cryptography
Then you can create an x509 certificate object and print whatever results you are after:
import ssl
from cryptography import x509
cert = ssl.enum_certificates("MY")[1][0]
decoded_cert = x509.load_der_x509_certificate(cert)
print(decoded_cert.subject)
Sometimes you just have to add additional steps/libraries to your code. It's not a reasonable expectation to only use native functionality or be expected to reinvent the wheel.
I've imported a valid .pfx certificate into a keyvault in Azure. When I try to download it with Python using the SecretClient object, I get a value with no "BEGIN CERTIFICATE" or "END CERTIFICATE" footer. My understanding is that this value should contain the public certificate and private key, but I can't seem to convert this string value into anything I can then use or read with openssl.
I can download the certificate fine with az keyvault secret download and then read it correctly with openssl
I've tried writing the string to a file and manually adding headers etc. but I feel I am missing something fundamental. The example I've seen here: https://github.com/Azure/azure-sdk-for-js/issues/7647 appears to show the value being directly written to a file and read with openssl. This does not work for me and I get the following error:
error:0D07803A: asn1 encoding routines : ASN1_ITEM_EX_D2I : nested asn1 error
So the fundamental question is: how to convert KeyVaultSecret.value into an x509 object or how to write it to a file in such a way that openssl can succesfully read it
Error was in converting to base64. Code below for future interested parties:
import base64
from azure.keyvault.secrets import SecretClient
secret = SecretClient(keyvaulturl,credentials)
secret_b64 = base64.b64decode(secret.value)
with open('test.pfx','wb') as fopen:
fopen.write(secret_b64)
This can be interrogated with openssl succesfully.
i am dealing with cTrader Trading platform.
My project is written in python 3 on tornado.
And have issue in decoding the prtobuf message from report API Events.
Below will list everything what i achieved and where have the problem.
First cTrader have Rest API for Report
so i got the .proto file and generated it for python 3
proto file is called : cTraderReportingMessages5_9_pb2
from rest Report API getting the protobuf message and able to decode in the following way because i know which descriptor to pass for decoding
from models import cTraderReportingMessages5_9_pb2
from protobuf_to_dict import protobuf_to_dict
raw_response = yield async_client.fetch(base_url, method=method, body=form_data, headers=headers)
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(descriptors[endpoint]['decode'], raw_response.body)
descriptors[endpoint]['decode'] = is my descriptor know exactly which descriptor to pass to decode my message
my content from cTraderReportingMessages5_9_pb2
# here is .proto file generated for python 3 is too big cant paste content here
https://ufile.io/2p2d6
So until here using rest api and know exactly which descriptor to pass, i am able to decode protobuf message and go forward.
2. Now the issue i face
Connecting with python 3 to the tunnel on 127.0.0.:5672
i am listening for events and receiving this kind of data back
b'\x08\x00\x12\x88\x01\x08\xda\xc9\x06\x10\xb6\xc9\x03\x18\xa1\x8b\xb8\x01 \x00*\x00:\x00B\x00J\x00R\x00Z\x00b\x00j\x00r\x00z\x00\x80\x01\xe9\x9b\x8c\xb5\x99-\x90\x01d\x98\x01\xea\x9b\x8c\xb5\x99-\xa2\x01\x00\xaa\x01\x00\xb0\x01\x00\xb8\x01\x01\xc0\x0
1\x00\xd1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xd9\x01\x00\x00\x00\x00\x00\x00\x00\x00\xe1\x01\x00\x00\x00\x00\x00\x00\x00\x00\xea\x01\x00\xf0\x01\x01\xf8\x01\x00\x80\x02\x00\x88\x02\x00\x90\x02\x00\x98\x02\x00\xa8\x02\x00\xb0\x02\x00\xb8\x02\x90N\xc0\x02\x00\xc8\x0
2\x00
as recommendation i got, i need to use same .proto file generated for python that i did in step 1 and decode the message but without any success because i don't know the descriptor need to be passed.
so in 1 step was doing and working perfect this way
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(descriptors[endpoint]['decode'], raw_response.body)
but in second step can not decode the message using in the same way, what i am missing or how to decode the message using same .proto file?
Finally found a workaround by my self, maybe is a primitive way but only this worked for me.
By the answer got from providers need to use same .proto file for both situations
SOLUTION:
1. Did list with all the descriptors from .proto file
here is .proto file generated for python 3 is too big cant paste content here
https://ufile.io/2p2d6
descriptors = [cTraderReportingMessages5_9_pb2.descriptor_1, cTraderReportingMessages5_9_pb2.descriptor_2]
2. Loop throw list and pass one by one
for d in descriptors:
decoded_response = cTraderReportingMessages5_9_pb2._reflection.ParseMessage(d, raw_response.body)
3. Check if decoded_response is not blank
if decoded_response:
# descriptor was found
# response is decoded
else:
# no descriptor
4. After decoded response we go parse it into dict:
from protobuf_to_dict import protobuf_to_dict
decoded_response_to_dict = protobuf_to_dict(decoded_response)
This solution that spent weeks on it finally worked.
There is already a lot of questions out there but the problem is, none of them have sufficient answers how to do it, especially when using python3.
Basically, I want to read JAR/APK certificates, like this one: Link to ASN1 Decoder, with Android Test Signing Key
There are now several alternatives:
pyasn1: seems to work, but only can parse the raw ASN.1 format
M2Crypto: only works on py2
Chilkat: Not free, although CkCert seems to be free
cryptography: Can not load the certificate, as the X509 certificate is inside the PKCS#7 container
I found a way to use pyasn1 to unpack the cert from the pkcs#7 message, then use cryptography to read it:
from pyasn1.codec.der.decoder import decode
from pyasn1.codec.der.encoder import encode
from cryptography import x509
from cryptography.hazmat.backends import default_backend
cdata = open("CERT.RSA", "rb").read()
cert, rest = decode(cdata)
# The cert should be located there
realcert = encode(cert[1][3])
realcert = realcert[2 + (realcert[1] & 0x7F) if realcert[1] & 0x80 > 1 else 2:] # remove the first DER identifier from the front
x509.load_der_x509_certificate(realcert, default_backend())
which gives
<Certificate(subject=<Name([<NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.6, name=countryName)>, value='US')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.8, name=stateOrProvinceName)>, value='California')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.7, name=localityName)>, value='Mountain View')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.10, name=organizationName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.11, name=organizationalUnitName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.3, name=commonName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=1.2.840.113549.1.9.1, name=emailAddress)>, value='android#android.com')>])>, ...)>
Is there no other way to have it clean and tidy?
There are now libraries to do this in pure python. One is asn1crypto: https://github.com/wbond/asn1crypto#readme
This is also impemented in androguard, including examples how to use it: https://androguard.readthedocs.io/en/latest/intro/certificates.html
I could not find a suitable way to get the name of the signature algorithm used for a certificate that is received by an SSLSocket. I'm aware you can pull the bytes of the peer's cert with SSLSocket.getpeercert(True) but I don't know what to do with it past that. PyOpenSSL doesn't seem to have an easy interface to load an X509 from it's byte content.
I would like to know this information as certificates with SHA1 are not allowed to have expiration times after January 1st, 2017 and this is not checked by Python's SSLSocket.do_handshake() implementation.
After the very helpful comment pointing me in the right direction, you can use the cryptography module to load the DER X509 format that the SSLSocket.getpeercert() function outputs. From this Certificate instance you can also access many other fields about the certificate such as expiration time. Here is the way that I am doing it right now:
import cryptography.x509
import cryptography.hazmat.backends.openssl
import ssl
sock = ssl.SSLSocket()
# Wrap a socket, connect, handshake, etc etc...
cert = cryptography.x509.load_der_x509_certificate(
sock.getpeercert(True),
cryptography.hazmat.backends.openssl.backend
)
print(cert.signature_hash_algorithm.name) # This prints "sha1".