We Keep Coding

How to decode a bytes SSL certificate?

I have the following certificate, as returned by ssl.enum_certificates:
import ssl
cert = ssl.enum_certificates("MY")[1][0]
print(cert)
Result:
b'0\x82\x02\xa60\x82\x01\x8e\xa0\x03\x02\x01\x02\x02\x11\x00\xff\xe1C\xed\x8c\xa5UA\xad\x870"\x0f\xc3+\xc30\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x000&1$0"\x06\x03U\x04\x03\x13\x1bXBL Client IPsec Issuing CA0\x1e\x17\r211013223833Z\x17\r211014223833Z0\x1b1\x190\x17\x06\x03U\x04\x03\x0c\x10F900E607B75850AE0\x81\x9f0\r\x06\t*\x86H\x86\xf7\r\x01\x01\x01\x05\x00\x03\x81\x8d\x000\x81\x89\x02\x81\x81\x00\xb8\xaeE\x04\x9c\x02\xe7\xe1\xe7g\xd9\xad\xfc\xbeg\x83\xbc\x7f\xd7\xcc\xc7m\x1d\x02\xe3\xff\x93\xe3\x81\xe9%\x8b\xdc]%\xd2\x8b!!;\xd0^\xdc\\\xd47\x07\x05g\xe2\xdeR\x8d\xc8\xea\xd4\xc5\xdc\x1c]\xa7L\xab\x92\x1f9\x02\x0f\x83\x12\x99\xd1^\xe2\xfdm\xcfU_8N\x94T\x92\x94\x8e\xa3\xe9\xa3\xc3\xf6\x04\xd9\x1b\xa1\xa5\t\xce\xa6~_\xc9\xc1DH\xd0j\x95\x84\x8e\xef\xd9o\xd7\xfe\x91Yz\x1a\xa2\xbe\xc5\x06Y\x9bc\x8bU\x02\x03\x01\x00\x01\xa3^0\\0\x0e\x06\x03U\x1d\x0f\x01\x01\xff\x04\x04\x03\x02\x01\xa60)\x06\x03U\x1d%\x04"0 \x06\x08+\x06\x01\x05\x05\x07\x03\x02\x06\x08+\x06\x01\x05\x05\x07\x03\x05\x06\n+\x06\x01\x04\x01\x827x\x04\x040\x1f\x06\x03U\x1d#\x04\x180\x16\x80\x14\xc1\xfe7p|Z9\x12&\xec\xc0\x94\xcd\x80\x94\xb1\xa8u\xe4\xd80\r\x06\t*\x86H\x86\xf7\r\x01\x01\x0b\x05\x00\x03\x82\x01\x01\x00.\xaakE7h\x95\xc7\xa8\x17\xce\x1d\xf2\xc5\x9d[y8g\xb44\x12\xcd\xbb\xdak\xdbF9\r\xef\xb8\x15\x01W[\xa0 \xcf\xa8\x19ML\xa5\x12\x00\xcaF\xb1\xa8\x91\xd9\xde\xc6\xe1.\xe4\xa0\xfe\\\xde\x90\x0c#\xaf\x7fS\x13\xd0\xde\xd9Q\xe2\xa6J\x98\x0f\xfd\x11\x7f\xf7\xed\xf8\xfeI-\x89o\x1b\xc1\xfeF\xbb\x05\x8d\xd0{\x9fc2\xc9\xcf\xcfe\x04\x1e+.\xbe\x1b\x90<~##\xa00\xa4\\1`\xd3\x84\x14\xf2\xe5\x16"\xda\x80\xa9\x90d\x92\xf5\x82\xc9\x87\xdf\xc4x\xc6\xde+\x17\x10\xc5j\x96\xed\x8e\xdfp\xb3\x04\x9e\xeb\xc6\xe6{\x1ee\x1c\xeeN\xad\xe4K7xb\xd5\x99\xb9\xf4pb{\x13\xa1\xa4:\x1cF\xef\xe7Hf*\x96!k\x0c\xda\x08*x\x13`\xd9Ju`e\xd8W\xf1;B\xa4\xb8B\xe5Y\xa5\n\x9a\xdb\xc7\xb5\xee+\xfe~\xac\xc1\x85\xfex\x1d\x8f\xc3s\xbc\xdd\xab\x1bo\xe0a\x98\xba\xa5\x98z\x03\xee}c[\xe1\xcb8\x1a\xb4\x80'
How to convert this output to a human-readable format? (dict preferred).
This is pretty much the same question as this one, but here I have the certificate in bytes format. I could call ssl.DER_cert_to_PEM_cert and try to go from there, but I'd rather avoid this additional step.

There is a library called cryptography which can do exactly this:
pip install cryptography
Then you can create an x509 certificate object and print whatever results you are after:
import ssl
from cryptography import x509
cert = ssl.enum_certificates("MY")[1][0]
decoded_cert = x509.load_der_x509_certificate(cert)
print(decoded_cert.subject)
Sometimes you just have to add additional steps/libraries to your code. It's not a reasonable expectation to only use native functionality or be expected to reinvent the wheel.

Convert .key formatted private key into .pem format with python3

I am trying to convert .key formatted private key into .pem formatted key using python3. Before using python3 I was able to convert private key into .pem format by simply running openssl command as:
openssl rsa -in <private_key_file> -outform PEM -out <new_file>
I would like to know how people are using python3 to convert it private keys into .pem or other formats. So far I've tried running subprocess.run() command to execute openssl command from python. It would be nice to know if there is python way of converting into .pem format.
I also tried following using pycryptodrome library https://pycryptodome.readthedocs.io/en/latest/index.html as per this thread pyOpenSSL creating a pem file which creates in pem format but wasn't successful.
def convert_key_to_pem_format(key_file):
print("Converting to pem/rsa format")
pv_key_string = key_file.exportKey()
with open ("private.pem", "w") as prv_file:
print("{}".format(pv_key_string.decode()), file=prv_file)
This is my code. Any help would be appreciated.

From the Cryptodome library, you can use the import_key method, which supports the following formats for an RSA private key:
PKCS#1 RSAPrivateKey DER SEQUENCE (binary or PEM encoding)
PKCS#8_ PrivateKeyInfo or EncryptedPrivateKeyInfo DER SEQUENCE (binary or PEM encoding)
Here's how you could use it:
from Cryptodome.PublicKey import RSA
key = RSA.import_key(open('private.key', "rb").read(), passphrase="(passphrase in case you have one)")
pv_key_string = key.exportKey()
with open ("private.pem", "w") as prv_file:
print("{}".format(pv_key_string.decode()), file=prv_file)

Python-GNUPG encrypted file cannot be decrypted with private key

I am trying to encrypt a text file in Python 3.6 using python-gnupg, and a public key provided by a client, for which they have a private key to decrypt it with. I don't have access to that key. Despite python-gnupg appearing to successfully encrypt the file (though with some confusing errors appearing in the log), the client is unable to decrypt it. We're told the error they're getting is gpg: decryption failed: No secret key
When we tested encrypting a file using Cryptophane (different computer, running Windows instead of Ubuntu) and the same public key, they were able to decrypt it. This is how the encryption was successfully done manually for months. When testing the same code with our company public key, we were able to decrypt it using our private key and Cryptophane.
I've googled extensively for the error messages and general problem, and haven't found anything that seemed to be the same problem getting solved.
Here's the relevant code. filepath is the relative path to the file to be encrypted. pgp_key_name is the name of the .asc file containing the public key. pgp_key_dir is the directory it's in.
def pgp_encrypt_file(filepath, pgp_key_name, pgp_key_dir):
gpg = gnupg.GPG()
output_full_filepath = filepath + '.pgp'
try:
with open(pgp_key_dir + pgp_key_name) as file:
key_data = file.read()
import_result = gpg.import_keys(key_data)
logger.info(msg='Public key imported: {}'.format(pgp_key_name))
public_keys = gpg.list_keys()
fingerprint = public_keys[0]['fingerprint']
logger.info(msg='Attempting to encrypt file: ' +
output_full_filepath)
with open(filepath, 'r') as f:
newfile = f.read()
status = gpg.encrypt(newfile, fingerprint,
output=output_full_filepath)
logger.info(msg='status.ok : ' + str(status.ok))
logger.info(msg='status.status : ' + str(status.status))
except FileNotFoundError as e:
logger.error(msg='File not found: ' + str(e))
except TypeError as e:
logger.error(msg='GNUPG TypeError: ' + str(e))
return output_full_filepath
And the relevant section of the logs:
03-01 15:18:58 gnupg INFO Setting homedir to
'/home/[user]/.config/python-gnupg'
03-01 15:18:58 gnupg ERROR Could neither invoke nor terminate a
gpg process... Are you sure you specified the corrent (and full) path to the
gpg binary?
(That error did NOT appear later, and I was unable to find anything relevant on Google or Stack Overflow for it.)
03-04 09:04:39 gnupg WARNING Ignoring '/usr/bin/gpg' (path is a symlink)
03-04 09:04:39 gnupg ERROR Could not find binary for 'gpg'.
03-04 09:04:39 gnupg INFO Setting homedir to
'/home/[user]/.config/python-gnupg'
03-04 09:04:39 gnupg INFO
Initialised settings:
binary: /usr/bin/gpg2
binary version: `2.0.14\ncfg:pubkey:1;16;17\ncfg:cipher:2;3;4;7;8;9;10;11;12;13\ncfg:ciphername:3DES;CAST5;BLOWFISH;AES;AES192;AES256;TWOFISH;CAMELLIA128;CAMELLIA192;CAMELLIA256\ncfg:digest:1;2;3;8;9;10;11\ncfg:digestname:MD5;SHA1;RIPEMD160;SHA256;SHA384;SHA512;SHA224\ncfg:compress:0;1;2;3\n'
homedir: /home/[user]/.config/python-gnupg
ignore_homedir_permissions: False
keyring: /home/[user]/.config/python-gnupg/pubring.gpg
secring: /home/[user]/.config/python-gnupg/secring.gpg
default_preference_list: SHA512 SHA384 SHA256 AES256 CAMELLIA256 TWOFISH
AES192 ZLIB ZIP Uncompressed
keyserver: hkp://wwwkeys.pgp.net
options: None
verbose: False
use_agent: False
03-04 09:04:39 gnupg INFO Importing: [first few lines of public key]
03-04 09:04:39 root INFO Public key imported: [name of key]
03-04 09:04:39 root INFO Attempting to encrypt file: [file]
03-04 09:04:39 gnupg INFO Writing encrypted output to file:
[file.pgp]
03-04 09:04:39 gnupg INFO Encrypted output written successfully.
Some thoughts and things we've tried:
Though there is a gpg binary in /usr/bin/gpg, we're using a conda virtual environment for the project itself, which I think may be messing this up. However, when I ran this code from the command line, with the environment deactivated, I ended up with the same result.
I see that the log file says that it couldn't find the gpg binary, and that it's ignoring a symlink pointing to it, but all of its status messages thereafter seemed to indicate that the encrytion was fine, and again, it worked just fine multiple times with a different public/private key pair.
Examining the pgp object in the IDE once instantiated leads me to think that it found the gpg binary just fine, even without passing any parameters to gnupg.GPG(). Passing in gnupghome='/usr/bin/gpg' leads me to the same place, and passing in gnupghome='not/real/path throws an error.
Setting armor=False on the call to encrypt did not change anything.
I really appreciate any and all thoughts on the matter.
If the answer is that it's just not looking in the right directories for the gpg binary or homedir, due to our virtual environment settings, recommendations on how to work around that would also be appreciated.

Resolved.
In this case, it was the client's error. We later attempted to encrypt the file using a variety of slightly different options, including many done from the command line, and from Python.
They were able to decrypt every single one.
For the sake of helping some others down the line, here are a few things that I've learned since starting on this journey:
There are two distinct packages both named python-gnupg.
The original one (from what I understand): https://pythonhosted.org/python-gnupg/
And a fork of it: https://github.com/isislovecruft/python-gnupg
Since these packages share a name, it is very confusing when googling errors in one or the other. Doing pip install python-gnupg seems to always download the second one. My experience is almost entirely with this second one, so keep that in mind when reading everything else in this post.
On CentOS 6, /usr/bin/gpg is a symlink that points to /usr/bin/gpg2. Python-GNUPG logs errors noting this, but then it seems to find /usr/bin/gpg2 just fine.
Regarding the error Could neither invoke nor terminate a gpg process...: While this concerned me, this also appears to have had no effect at all on any functionality. Your mileage may vary.
Compatibility issues are possible between the Python-GNUPG version and gpg binary version. This can lead to Unknown status message: [SOME-GPG-MESSAGE] errors; for example: Unknown status message: PINENTRY_LAUNCHED which I believe arises when gpg tries to bring up the passphrase prompt (which it does not do in older versions!). If you are NOT trying to make a module with different uses on different OSes (we were), you can try your luck with manually editing the python-gnupg source code once you pip install the package. Specifically, in pretty_bad_protocol._parsers.py in the _handle_status method, there is a tuple of known status messages; just add in any of the "unknown" status messages there, and that error won't trip in the future. I mean, you're on your own after that, but it was something that we tried and it doesn't appear to have harmed anything.
Best of luck to anyone trying to do pgp encryption in the future.

thanks for providing all the details.
I have fixed that issue by
gpg = gnupg.GPG(binary='/usr/bin/gpg2', homedir='/tmp')

Decoding Scapy ASN1 encoded SSL/TLS certificate fields

I am extracting SSL/TLS certificate fields from serverhello packet using scapy-ssl_tls library which I installed using pip.
The problem is, I'm not able to figure out a way to extract values from ASN1 encoded fields:
sign_algo2: <ASN1_OID['.1.2.840.113549.1.1.11']>
sa2_value: <ASN1_NULL[0L]>
not_before: <ASN1_UTC_TIME['170321131500Z']>
not_after: <ASN1_UTC_TIME['200321131500Z']>
pubkey_algo: <ASN1_OID['.1.2.840.113549.1.1.1']>
version: <ASN1_INTEGER[2L]>
sn: <ASN1_INTEGER[6348220899422160075L]>
sign_algo: <ASN1_OID['.1.2.840.113549.1.1.11']>
pubkey: <ASN1_BIT_STRING['\x000\x']>
I've dug out scapy.layers.ssl_tls, ssl_tls_crypto, scapy.layers.x509 modules but couldn't get any hint to decode it. I also tried using Asn1Value.load() from asn1crypto.core package but it fails with following error:
TypeError: encoded_data must be a byte string, not scapy.asn1.asn1.ASN1_UTC_TIME
It'd be great if anyone could help me getting this resolved using scapy's native decoding preferably or any other way possible.
Note: Please note that I've to extract these values from SSL/TLS serverhello packets which I'm reading from a pcap file as I need other fields from packet headers as well. I know many solutions exist on stackoverflow or wireshark/tshark for extracting/reading certificates from .pem files or possibly from .der files (after having it exported from wireshark explicitly), they don't work in my case as I need a solution which works around extracting certificates or certificate fields from packets.

If you have a DER-encoded byte string of the X.509 certificate, the correct way to use asn1crypto would be:
from asn1crypto import x509
cert = x509.Certificate.load(der_byte_string)
print(cert.native)
It would seem from the error message, you have tried to pass some of Python object that represents an ASN.1 value.

Make sure you are passing asn1crypto the byte string, not some internal scapy object. May be you need to cast the latter into a byte string.
Alternatively, this tool is designed to decode X.509 certs into a tree of Python objects. You also need to feed it either a string (Python2) or bytes (Python 3).

This has been discussed in scapy-ssl_tls issue #116 which describes basic handling of scapy ASN.1 fields:
...
# resp holds the raw socket response to a client_hello
tls_response = TLS(resp)
tls_response.show() # show the structure
# iterate all certificates
for cert in tls_response[TLSCertificateList].payload.certificates:
# .payload as the structure is [...][TLSCertificateList][TLS10Certificate].certificates = [x509Cert,x509Cert,...]
# we'll have a TLSCertificateList object at this point; get the scapy X509Cert Object
tlscert = cert[X509Cert]
print repr(str(tlscert.sign_algo)) # raw bytes -> '\x06\t*\x86H\x86\xf7\r\x01\x01\x0b'
print repr(tlscert.sign_algo) # <ASN1_OID['.1.2.840.113549.1.1.11']>
print tlscert.sign_algo.val # 1.2.840.113549.1.1.11
print repr(tlscert.version) # <ASN1_INTEGER[2L]>
print tlscert.version.val # 2

Reading the certificate of PKCS#7 Signed Data using pure python

There is already a lot of questions out there but the problem is, none of them have sufficient answers how to do it, especially when using python3.
Basically, I want to read JAR/APK certificates, like this one: Link to ASN1 Decoder, with Android Test Signing Key
There are now several alternatives:
pyasn1: seems to work, but only can parse the raw ASN.1 format
M2Crypto: only works on py2
Chilkat: Not free, although CkCert seems to be free
cryptography: Can not load the certificate, as the X509 certificate is inside the PKCS#7 container
I found a way to use pyasn1 to unpack the cert from the pkcs#7 message, then use cryptography to read it:
from pyasn1.codec.der.decoder import decode
from pyasn1.codec.der.encoder import encode
from cryptography import x509
from cryptography.hazmat.backends import default_backend
cdata = open("CERT.RSA", "rb").read()
cert, rest = decode(cdata)
# The cert should be located there
realcert = encode(cert[1][3])
realcert = realcert[2 + (realcert[1] & 0x7F) if realcert[1] & 0x80 > 1 else 2:] # remove the first DER identifier from the front
x509.load_der_x509_certificate(realcert, default_backend())
which gives
<Certificate(subject=<Name([<NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.6, name=countryName)>, value='US')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.8, name=stateOrProvinceName)>, value='California')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.7, name=localityName)>, value='Mountain View')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.10, name=organizationName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.11, name=organizationalUnitName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=2.5.4.3, name=commonName)>, value='Android')>, <NameAttribute(oid=<ObjectIdentifier(oid=1.2.840.113549.1.9.1, name=emailAddress)>, value='android#android.com')>])>, ...)>
Is there no other way to have it clean and tidy?

There are now libraries to do this in pure python. One is asn1crypto: https://github.com/wbond/asn1crypto#readme
This is also impemented in androguard, including examples how to use it: https://androguard.readthedocs.io/en/latest/intro/certificates.html