We Keep Coding

Python3 - imaplib - decrypt using pem certificate

As part of a larger workflow, I've been supplied with an email and password, and with an .pfx-certificate and password for that certificate. I need to use those to download a specific e-mail (specifically, its attachment), so it can be processed in other scripts.
I've already managed to transform the .pfx-file into a .pem file. Using the code below, I can download the message:
import email
import imaplib
EMAIL = 'test#test.com'
PASSWORD = 'testpassword'
SERVER = "outlook.office365.com"
# Connect to the server and go to its inbox
mail = imaplib.IMAP4_SSL(SERVER, port=993)
mail.login(EMAIL, PASSWORD)
mail.select('inbox')
# Get the relevant mail ID
status, ids = mail.search(None, '(HEADER Subject "Testmessage_")')
mail_id = ids[0].split()[0]
# Fetch the mail, get the data, write the file
status, contents = mail.fetch(mail_id, '(RFC822)')
data = contents[0][1]
with open("outputfile.txt", "wb") as outputfile:
outputfile.write(data)
I can then decode outputfile.txt with OpenSSL with the following command:
openssl cms -decrypt -in outputfile.txt -inkey cert.pem > outputmessage.txt
I can confirm that in the outputmessage.txt-file, the contents of the attachment are visible, and with some workaround, usable in the rest of my Python-script.
However, this means working with multiple temporary files and (at least) three different commands (python - openssl - python again). I would like to do all of this in Python, so I don't have to create temporary files and can process the results right away in the following scripts.
Additionally, I'd prefer to use as few external dependencies as possible.
I've seen M2Crypto and cryptography mentioned, but I can't seem to get either package working on my machine (something with wheels?). Are there any other options?

Trivially,
import subprocess
# ...
subprocess.run(['openssl', 'cms', '-decrypt',
'-inkey', 'cert.pem', '-out', 'outputmessage.txt'],
input=data, check=True)
I don't have openssl cms locally but I'm guessing from the man page that it will read standard input if you don't pass in an -in option. If that's not true, maybe try something like -in, - or maybe replace - with /dev/stdin or whatever your platform offers.

Python Confluent-Kafka SSL Configuration

A basic Confluent-Kafka producer and consumer have been created to send plaintext messages.
After successfully sending messages from producer to consumer, additional configs were added to use SSL rather than PLAINTEXT.
The following Configs have been implemented, which result in the following error.
"Message Timed Out"
Producer Configs:
bootstrap.servers: localhost9093
security.protocol: SSL
ssl.keystore.location: ../keystore.p12
ssl.keystore.password: [password]
ssl.ca.location: ../CARoot.pem
ssl.key.location: ../key.pem
ssl.certificate.location: ../cert.pem
ssl.key.password: [password]
Server Configs:
ssl.keystore.type= PKCS12
ssl.keystore.location= ../keystore.p12
ssl.keystore.password= [password]
ssl.ca.location= ..\\CARoot.pem
ssl.certificate.location= ..\\cert.pem
ssl.key.password= [password]
ssl.key.location= ../key.pem
security.inter.broker.protocol= SSL
listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093
Are there additional configs required to implement SSL?
Additionally, can anyone summarize the CARoot?
From what I have read, it is a chain of certificates.
Therefore, if there is only one certificate, should CARoot.pem and cert.pem be identical?
This file might be the problem.
The certificate and private key were created in PyOpenSSL.
keystore.p12 was converted from a .jks keystore using keytool.
Is there a way to create a CARoot.pem file using this library?
Thank you.

The producer was timing out due to the formatting of the CA Certificate file.
The solution to the following question was used to resolve the time-out error, which uses OpenSSL rather than PyOpenSSL.
Note: OpenSSL is available in Git Bash.
How to export CA certificate chain from PFX in PEM format without bag attributes
Additionally, there were some changes that were made to the configuration of both the server and producer.
Producer Configurations:
bootstrap.servers: localhost9093
security.protocol: SSL
ssl.ca.location: ../CARoot.pem
ssl.key.location: ../key.pem
ssl.certificate.location: ../cert.pem
Server Configurations:
ssl.protocol= SSL
ssl.keystore.type= JKS
ssl.keystore.location= ../keystore.jks
ssl.keystore.password= [password]
ssl.client.auth= required
security.inter.broker.protocol= SSL
listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093
advertised.listeners = PLAINTEXT://localhost:9092,SSL://localhost:9093

How to Export Certificate on Windows in Python

I'm writing a Python program that needs to export a certificate from a certificate store on Windows. I've tried searching for a code snippet that does this but I'm having trouble finding one that does that. The important thing here is that I need to export the certificate with the private key from certificate stores that belonging to the machine and the current user.
My goal was to use a certificate to authenticate to Azure Key Vault. Based on the accepted answer, there's no way to retrieve a certificate from a cert store on windows. I decided to, instead, write a C# app to authenticate to Azure Key Vault and pass the secrets to the Python application.

You could send a subprocess call to powershell to export the certificates from the certificate store. This script prompts for a user password, then exports the certificates from the user's and localmachine certificate store that have a private key as .pfx files.
import subprocess
import getpass
pw = getpass.getpass()
proc = subprocess.Popen(
[
'powershell.exe',
'-c',
f'&{{ $pw = ConvertTo-SecureString -String "{pw}" -Force -AsPlainText;',
'gci Cert:\\*\\My\\* |',
'?{ $_.HasPrivateKey } |',
'%{ Export-PfxCertificate -cert $_.PSPath',
'-FilePath $env:USERPROFILE\\$($_.thumbprint).pfx -Password $pw}',
'}'
],
stdout = subprocess.PIPE,
stderr = subprocess.PIPE,
shell = True
)
out, err = proc.communicate()

How to use ssl client certificate (p12) with Scrapy?

I need to use client certificate file in format p12(PKCS12) to talk to a webserver with scrapy, is there a way to do that ?

I can't offer you a tested and complete solution here, but I know a few places where some adjustments might give you what you need.
The starting point is scrapy's ContextFactory object which defines the SSL/TLS configuration. The standard implementation ScrapyClientContextFactory doesn't use client certificates and also doesn't do any server certificate verification, it just accepts any certificate. (More details)
When looking into the source code however you see the alternative BrowserLikeContextFactory is creating an optionsForClientTLS object.
This object can also take a clientCertificate parameter to authenticate to the server. (Details)
So in theory you need to subclass BrowserLikeContextFactory, write there your own creatorForNetloc method and make it create optionsForClientTLS that also have a clientCertificate
In a gist:
#implementer(IPolicyForHTTPS)
class ClientCertContextFactory(BrowserLikeContextFactory):
def creatorForNetloc(self, hostname, port):
with open('yourcert.pem') as keyAndCert:
myClientCert = twisted.internet.ssl.PrivateCertificate.load(keyAndCert.read())
return optionsForClientTLS(hostname.decode("ascii"),
trustRoot=platformTrust(),
clientCertificate=myClientCert,
extraCertificateOptions={
'method': self._ssl_method,
})
Activating the context factory in settings.py:
DOWNLOADER_CLIENTCONTEXTFACTORY = 'your.package.ClientCertContextFactory'
According to the docs twisted.internet.ssl.PrivateCertificate can only load pem or asn.1 format keys, means you will have to convert your key into pem format:
openssl pkcs12 -in client_ssl.pfx -out client_ssl.pem -clcerts
(Borrowed from Converting pfx to pem using openssl)
Update Conversion for PKCS12 files in p12 format:
openssl pkcs12 -in client_cert.p12 -out client_cert.pem -clcerts

pyOpenSSL creating a pem file

I've created a key pair using the following code in python with pyOpenSSL:
from OpenSSL import crypto
k = crypto.PKey()
k.generate_key(crypto.TYPE_RSA, 2048)
Now how can I create the private and public key .pem files from the key object?
If there is any tutorial available please let me know. I found none. From the manual, it's difficult to know as I'm new to OpenSSL.
What are the chances that the same code will create two same key pairs is there is no specific unique key is being used in RSA?

I know this is an old question - but as I've just found it I thought I'd add an answer.
The easiest way to do this with Python 3.x is to use PyCryptodome.
The in Python (for a 2048-bit key):
from Cryptodome.PublicKey import RSA
key = RSA.generate(2048)
pv_key_string = key.exportKey()
with open ("private.pem", "w") as prv_file:
print("{}".format(pv_key_string.decode()), file=prv_file)
pb_key_string = key.publickey().exportKey()
with open ("public.pem", "w") as pub_file:
print("{}".format(pb_key_string.decode()), file=pub_file)
If you want to check the private key on the (Linux) command-line use:
$ openssl rsa -check -inform pem -noout -in private.pem
RSA key ok
...

I hope this will help people in the future, because I had this same need and couldn't find an answer so I did it myself. Thought I would share it with you.
1. Creating a PEM file
bio_pub = _new_mem_buf() # Memory buffers to write to
bio_priv = _new_mem_buf()
helper = OpenSSL.crypto._PassphraseHelper(OpenSSL.crypto.FILETYPE_PEM, None)
pk = OpenSSL.crypto.PKey()
pk.generate_key(OpenSSL.crypto.TYPE_RSA, n)
# Convert from EVP_PKEY type to RSA type
rsa_pkey = _lib.EVP_PKEY_get1_RSA(pk._pkey)
result_code = _lib.PEM_write_bio_RSAPublicKey(bio_pub, rsa_pkey)
result_code = _lib.PEM_write_bio_RSAPrivateKey(
bio_priv, rsa_pkey, _ffi.NULL, _ffi.NULL, 0,
helper.callback, helper.callback_args)
After this part you will have the public and private keys in your buffers.
To get it as a string you can call the functions:
_bio_to_string(bio_pub), _bio_to_string(bio_priv)
I used these imports for the special "private" functions of OpenSSL.crypto:
import OpenSSL
from OpenSSL._util import lib as _lib, ffi as _ffi
from OpenSSL.crypto import _new_mem_buf, _bio_to_string

You can create a .pem key by follow this tutorial at:
https://help.ubuntu.com/community/OpenSSL
that suppose you want to create a CA(certificate authority) certificate, that
is little complicate because you already have to get a CA from somewhere
because it's not free.
if you only want to create a key juste for your ssl connection test it
better to create
a self-sign certificate.
then make sure first you have install openssl and you have resolve the CN (Common Name) on your serve. without that you will be in trouble to use the created certificate.
for the Self-sign certificate use this command line:
$ openssl genrsa -des3 -passout pass:x -out server.pass.key 2048
$ openssl rsa -passin pass:x -in server.pass.key -out server.key
$ rm server.pass.key
$ openssl req -new -key server.key -out server.csr (list of question to answer)
$ openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
after you got the certificate create you have to activate your
server mod-ssl and add the line where is locate your certificate.
later you have to insert that certificate in your IE certificate
list to get it work with you apache ssl connection daemon.