We Keep Coding

python ssl (eqivalent of openssl s_client -showcerts ) How to get list of CAs for client certs from server

I have a group of nginx servers, that accept client certificates.
They have the ssl_client_certificate option with a file containing one or more CAs
If I use a web browser, then the web browser seems to receive a list of valid CAs for client certs. The browser shows only client certs signed by one of these CAs.
Following openssl command gives me a list of CA certs:
openssl s_client -showcerts -servername myserver.com -connect myserver.com:443 </dev/null
The lines I am interested in look following way:
---
Acceptable client certificate CA names
/C=XX/O=XX XXXX
/C=YY/O=Y/OU=YY YYYYYL
...
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
How can I get the same information with python?
I do have following code snippet, that allows to obtain a server's certificate, but this code does not return the list of CAs for client certs.
import ssl
def get_server_cert(hostname, port):
conn = ssl.create_connection((hostname, port))
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
sock = context.wrap_socket(conn, server_hostname=hostname)
cert = sock.getpeercert(True)
cert = ssl.DER_cert_to_PEM_cert(cert)
return cerft
I expected to find a functional equivalent of getpeercert(), something like getpeercas() but didn't find anything.
Current workaround:
import os
import subprocess
def get_client_cert_cas(hostname, port):
"""
returns a list of CAs, for which client certs are accepted
"""
cmd = [
"openssl",
"s_client",
"-showcerts",
"-servername", hostname,
"-connect", hostname + ":" + str(port),
]
stdin = open(os.devnull, "r")
stderr = open(os.devnull, "w")
output = subprocess.check_output(cmd, stdin=stdin, stderr=stderr)
ca_signatures = []
state = 0
for line in output.decode().split("\n"):
print(state, line)
if state == 0:
if line == "Acceptable client certificate CA names":
state = 1
elif state == 1:
if line.startswith("Client Certificate Types:"):
break
ca_signatures.append(line)
return ca_signatures
Update:Solution with pyopenssl (Thanks Steffen Ullrich)
#Steffen Ulrich suggested to use pyopenssl, which has a method get_client_ca_list() and this helped me to write a small code snippet.
Below code seems to work. Not sure if it can be improved or whether there are any pit falls.
If nobody is answering within the next days I will post this as a potential answer.
import socket
from OpenSSL import SSL
def get_client_cert_cas(hostname, port):
ctx = SSL.Context(SSL.SSLv23_METHOD)
# If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
# an empty result
ctx.set_options(SSL.OP_NO_TLSv1_3)
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
# next line for SNI
sock.set_tlsext_host_name(hostname.encode("utf-8"))
sock.connect((hostname, port))
# without handshake get_client_ca_list will be empty
sock.do_handshake()
return sock.get_client_ca_list()
Update: 2021-03-31
Above suggested solution using pyopenssl works in most cases.
However sock.get_client_ca_list()) cannot be called immediately after performing a sock.connect((hostname, port))
Some actions seem to be required in between these two commands.
Initially I used sock.send(b"G"), but now I use sock.do_handshake(), which seems a little cleaner.
Even stranger, the solution doesn't work with TLSv1.3 so I had to exclude it.

As a generic example in python
first you need to contact the server to learn which issuer CA subjects it accepts:
from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509Name
from certifi import where
import idna
def get_server_expected_client_subjects(host :str, port :int = 443) -> list[X509Name]:
expected_subjects = []
ctx = SSL.Context(method=SSL.SSLv23_METHOD)
ctx.verify_mode = SSL.VERIFY_NONE
ctx.check_hostname = False
conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
conn.connect((host, port))
conn.settimeout(3)
conn.set_tlsext_host_name(idna.encode(host))
conn.setblocking(1)
conn.set_connect_state()
try:
conn.do_handshake()
expected_subjects :list[X509Name] = conn.get_client_ca_list()
except SSL.Error as err:
raise SSL.Error from err
finally:
conn.close()
return expected_subjects
This did not have the client certificate, so the TLS connection would fail. There are a lot of bad practices here, but unfortunately they are necessary and the only way to gather the message from the server before we actually want to attempt client authentication using hte correct certificate.
Next you load the cert based on the server:
from pathlib import Path
from OpenSSL.crypto import load_certificate, FILETYPE_PEM
from pathlib import Path
def check_client_cert_issuer(client_pem :str, expected_subjects :list) -> str:
client_cert = None
if len(expected_subjects) > 0:
client_cert_path = Path(client_pem)
cert = load_certificate(FILETYPE_PEM, client_cert_path.read_bytes())
issuer_subject = cert.get_issuer()
for check in expected_subjects:
if issuer_subject.commonName == check.commonName:
client_cert = client_pem
break
if client_cert is None or not isinstance(client_cert, str):
raise Exception('X509_V_ERR_SUBJECT_ISSUER_MISMATCH') # OpenSSL error code 29
return client_cert
In a real app (not an example snippet) you would have a database of some sort to take the server subject and lookup the location of the cert to load - this example does it in reverse for demonstration only.
Make the TLS connection, and capture any OpenSSL errors:
from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509, FILETYPE_PEM
from certifi import where
import idna
def openssl_verifier(conn :SSL.Connection, server_cert :X509, errno :int, depth :int, preverify_ok :int):
ok = 1
verifier_errors = conn.get_app_data()
if not isinstance(verifier_errors, list):
verifier_errors = []
if errno in OPENSSL_CODES.keys():
ok = 0
verifier_errors.append((server_cert, OPENSSL_CODES[errno]))
conn.set_app_data(verifier_errors)
return ok
client_pem = '/path/to/client.pem'
client_issuer_ca = '/path/to/ca.pem'
host = 'example.com'
port = 443
ctx = SSL.Context(method=SSL.SSLv23_METHOD) # will negotiate TLS1.3 or lower protocol, what every is highest possible during negotiation
ctx.load_verify_locations(cafile=where())
if client_pem is not None:
ctx.use_certificate_file(certfile=client_pem, filetype=FILETYPE_PEM)
if client_issuer_ca is not None:
ctx.load_client_ca(cafile=client_issuer_ca)
ctx.set_verify(SSL.VERIFY_NONE, openssl_verifier)
ctx.check_hostname = False
conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
conn.connect((host, port))
conn.settimeout(3)
conn.set_tlsext_host_name(idna.encode(host))
conn.setblocking(1)
conn.set_connect_state()
try:
conn.do_handshake()
verifier_errors = conn.get_app_data()
except SSL.Error as err:
raise SSL.Error from err
finally:
conn.close()
# handle your errors in your main app
print(verifier_errors)
Just make sure you handle those OPENSSL_CODES errors if any are encountered, the lookup dictionary is here.
Many validations occur pre verification inside OpenSSL itself and all PyOpenSSL will do is a basic validation. so we need to access these codes from OpenSSL if we want to do Client Authentication, i.e. on the client and throw away the response from an untrusted server if it fails any authentication checks on the client side, per Client Authorisation or rather mutual-TLS dictates

#Stof's solution is more complete than this one.
So I selected his answer as 'official' answer.
This answer predates his, but might still be of some interest.
With #Steffen Ullrich's help I found following solution,
which works for all the (nginx with a ssl_client_certificate setting) servers that I tested with.
It requires to install an external package
pip install pyopenssl
Then following work will work:
import socket
from OpenSSL import SSL
def get_client_cert_cas(hostname, port):
ctx = SSL.Context(SSL.SSLv23_METHOD)
# If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
# an empty result
ctx.set_options(SSL.OP_NO_TLSv1_3)
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
# next line for SNI
sock.set_tlsext_host_name(hostname.encode("utf-8"))
sock.connect((hostname, port))
# without handshake get_client_ca_list will be empty
sock.do_handshake()
return sock.get_client_ca_list()
The line sock.do_handshake() is required to trigger enough of the SSL protocol. Otherwise client_ca_list information doesn't seem to be populated.
At least for the servers, that I tested I had to make sure TLSv1.3 is not used. I don't know whether this is a bug, a feature or whether with TLSv1.3 another function has to be called prior to calling get_client_ca_list()
I am no pyopenssl expert, but could imagine, that there is a more elegant / more explicit way to get the same behavior.
but so far this works for me for all the servers, that I encountered.

Can't pickle an RSA key to send over a socket

I have a list with a public key and a username that I want to send over a socket.
I found
how to send an array over a socket in python but using pickling won't work either.
My code:
private_key = generateRSA()
public_key = private_key.public_key()
host = ''
port = 8000
username = sys.argv[1]
mySocket = socket.socket()
mySocket.connect((host, port))
dataToSend = [public_key, username.encode()]
dataSend = pickle.dumps(dataToSend)
mySocket.send(dataSend)
The error in the console says
dataSend = pickle.dumps(dataToSend)
_pickle.PicklingError: Can't pickle <class '_cffi_backend.CDataGCP'>: attribute lookup CDataGCP on _cffi_backend failed
The key was generated with the cryptography library.

You are trying to send a RSAPublicKey instance, but this is a data structure managed by SSL, not Python. Because you interact with it through a CFFI proxy, you can't pickle one of these.
You'll need to pick a key serialization format to send it instead. You could convert to PEM, for example:
from cryptography.hazmat.primitives import serialization
pem = public_key.public_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PublicFormat.SubjectPublicKeyInfo
)
The above would export your key to a string value you could then convert to bytes and send over the socket directly. The other side would load the key from that string again.
Note that the above uses no encryption whatsoever. Anyone intercepting the traffic over that socket would be able to read your public key. That may or may not be a problem for your application.

How can I perform Two way SSL authentication in python?

I am a beginner of python. I have implemented the one way SSL authentication in python, below is a part of the server side code:
...
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind(('', 12345))
s.listen(5)
while True:
(connection, address) = s.accept()
connstream = ssl.wrap_socket(connection,
server_side=True,
certfile="ssl/server.crt",
keyfile="ssl/server.key",
)
#print repr(connection.recv(65535));
try:
deal_with_client(connstream)
....
below is the client side code:
import socket, ssl, pprint
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket(s,
ca_certs="ssl/server.crt",
cert_reqs=ssl.CERT_REQUIRED)
ssl_sock.connect(('localhost', 12345))
print repr(ssl_sock.getpeername())
print ssl_sock.cipher()
print pprint.pformat(ssl_sock.getpeercert())
while 1:
ssl_sock.write("boo!")
s.close()
Actually I want to perform two way SSL authentication, then I generated the certificates of ca and client and private key of client and ca by using openssl tool.
Now I have below six files:
ca.crt
server.crt
client.crt
ca.key
server.key
client.key
So now how can I modify the server side and client side code to perform two way two way SSL authentication?
Sorry for my english, please help.

If you are client and want to connect a server and send request at the same time, you can use the following code
response = requests.post(url, data=your_data, cert=('path_client_certificate_file', 'path_certificate_key_file'), verify='path_rootCA')

You just need to do the same in the client and in the server:
#server
ssl.wrap_socket(connection,
server_side=True,
certfile="ssl/server.crt",
keyfile="ssl/server.key",
ca_certs="ssl/client.crt"
)
#client
ssl_sock = ssl.wrap_socket(s,
ca_certs="ssl/server.crt",
cert_reqs=ssl.CERT_REQUIRED,
certfile="ssl/client.crt",
keyfile="ssl/client.key"
)
I know this is an old one, but I looked for the same thing and didn't find an answer.

python raw socket: Protocol not supported

I am trying to open a raw socket with Python under linux.
My simple code:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
s.bind((HOST, 5454))
And I got this error:
[ERROR] Protocol not supported
By the way, I am using python 2.7.3 under linux 12.04, and I used root to run the code.
Does anyone have a clue?
Update: The solution given by dstromberg is correct. If you want the whole packet, then use his solution. However, there is another combination:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_TCP)
that also works.
In this case, you will receive a whole TCP packet with IP and TCP headers on it.
If your use dstromberg's solution, you will also see the ethernet header. So it depends on how 'raw' you want your packet to be.

Try socket.AF_PACKET instead of socket.AF_INET.

This runs without error as root:
#!/usr/local/cpython-3.3/bin/python
import socket as socket_mod
#s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
socket = socket_mod.socket(socket_mod.AF_PACKET, socket_mod.SOCK_RAW, socket_mod.IPPROTO_IP)
#socket.bind(('localhost', 5454))
socket.bind(('lo', 5454))

Try socket.AF_UNIX, it can solve your problem, good luck.

Public key not recognized

I am trying to export a public key from openssl using python. I have the actual key information transferred to the client from the server but the PEM encoding is not transferred so the key on the client is useless. I basically send the public key using send all in python but unfortunately this does not send the PEM encoding. Does anyone know how to transfer the encoding?
I didn't know that the encoding would not transfer along with the key.
THe code where the string is read in
import socket
import M2Crypto as m2c
import os
max_transfer_block = 1024
server_addr = "10.1.1.2"
dest_port = 3333
listen_port = 8888
client_addr = "10.1.1.3"
mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysocket.connect((server_addr, dest_port))
#receive the public key from the server
keysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
keysocket.bind((client_addr, listen_port))
keysocket.listen(1)
conn, client_addr = keysocket.accept()
print 'connected by', client_addr
data = conn.recv(max_transfer_block)
#FILE = m2c.RSA.save_pub_key(data, "serverPubKey.pem")
FILE = open("sPub.pem", "w")
FILE.write(data)
keysocket.close()
#transfer encrypted file
key = m2c.RSA.load_pub_key('serverPubKey.pem')
FILE = open("test.txt", "r")
data = FILE.read()
encrypted = key.public_encrypt(data, m2c.RSA.pkcs1_padding)
mysocket.sendall(encrypted)
mysocket.close()
When I use the line key = m2c.RSA.load_pub_key('serverPubKey.pem') I get an error telling me that there is no starting point.
raise RSAError, m2.err_reason_error_string(m2.err_get_error()) M2Crypto.RSA.RSAError: no start line
I have figured out that this is because there is not in PEM format. Unfortunately, I don't know how to put it in that format.

The mistake was that the public/private key pair needs to be created from the same wrapper. What I mean by this is that not all key pairs are the same. My specific problem was that Openssl and the M2Crypto instances of key pairs were not in the same underlying format. Thus creating keys with Openssl and then trying to use M2Crypto to use the keys was wrong. The lesson from all of this is to not import keys from other wrappers. If you do, make sure that they are in the same underlying format like ASCII or Unicode before trying to use them.