Related
I have a group of nginx servers, that accept client certificates.
They have the ssl_client_certificate option with a file containing one or more CAs
If I use a web browser, then the web browser seems to receive a list of valid CAs for client certs. The browser shows only client certs signed by one of these CAs.
Following openssl command gives me a list of CA certs:
openssl s_client -showcerts -servername myserver.com -connect myserver.com:443 </dev/null
The lines I am interested in look following way:
---
Acceptable client certificate CA names
/C=XX/O=XX XXXX
/C=YY/O=Y/OU=YY YYYYYL
...
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
How can I get the same information with python?
I do have following code snippet, that allows to obtain a server's certificate, but this code does not return the list of CAs for client certs.
import ssl
def get_server_cert(hostname, port):
conn = ssl.create_connection((hostname, port))
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
sock = context.wrap_socket(conn, server_hostname=hostname)
cert = sock.getpeercert(True)
cert = ssl.DER_cert_to_PEM_cert(cert)
return cerft
I expected to find a functional equivalent of getpeercert(), something like getpeercas() but didn't find anything.
Current workaround:
import os
import subprocess
def get_client_cert_cas(hostname, port):
"""
returns a list of CAs, for which client certs are accepted
"""
cmd = [
"openssl",
"s_client",
"-showcerts",
"-servername", hostname,
"-connect", hostname + ":" + str(port),
]
stdin = open(os.devnull, "r")
stderr = open(os.devnull, "w")
output = subprocess.check_output(cmd, stdin=stdin, stderr=stderr)
ca_signatures = []
state = 0
for line in output.decode().split("\n"):
print(state, line)
if state == 0:
if line == "Acceptable client certificate CA names":
state = 1
elif state == 1:
if line.startswith("Client Certificate Types:"):
break
ca_signatures.append(line)
return ca_signatures
Update:Solution with pyopenssl (Thanks Steffen Ullrich)
#Steffen Ulrich suggested to use pyopenssl, which has a method get_client_ca_list() and this helped me to write a small code snippet.
Below code seems to work. Not sure if it can be improved or whether there are any pit falls.
If nobody is answering within the next days I will post this as a potential answer.
import socket
from OpenSSL import SSL
def get_client_cert_cas(hostname, port):
ctx = SSL.Context(SSL.SSLv23_METHOD)
# If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
# an empty result
ctx.set_options(SSL.OP_NO_TLSv1_3)
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
# next line for SNI
sock.set_tlsext_host_name(hostname.encode("utf-8"))
sock.connect((hostname, port))
# without handshake get_client_ca_list will be empty
sock.do_handshake()
return sock.get_client_ca_list()
Update: 2021-03-31
Above suggested solution using pyopenssl works in most cases.
However sock.get_client_ca_list()) cannot be called immediately after performing a sock.connect((hostname, port))
Some actions seem to be required in between these two commands.
Initially I used sock.send(b"G"), but now I use sock.do_handshake(), which seems a little cleaner.
Even stranger, the solution doesn't work with TLSv1.3 so I had to exclude it.
As a generic example in python
first you need to contact the server to learn which issuer CA subjects it accepts:
from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509Name
from certifi import where
import idna
def get_server_expected_client_subjects(host :str, port :int = 443) -> list[X509Name]:
expected_subjects = []
ctx = SSL.Context(method=SSL.SSLv23_METHOD)
ctx.verify_mode = SSL.VERIFY_NONE
ctx.check_hostname = False
conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
conn.connect((host, port))
conn.settimeout(3)
conn.set_tlsext_host_name(idna.encode(host))
conn.setblocking(1)
conn.set_connect_state()
try:
conn.do_handshake()
expected_subjects :list[X509Name] = conn.get_client_ca_list()
except SSL.Error as err:
raise SSL.Error from err
finally:
conn.close()
return expected_subjects
This did not have the client certificate, so the TLS connection would fail. There are a lot of bad practices here, but unfortunately they are necessary and the only way to gather the message from the server before we actually want to attempt client authentication using hte correct certificate.
Next you load the cert based on the server:
from pathlib import Path
from OpenSSL.crypto import load_certificate, FILETYPE_PEM
from pathlib import Path
def check_client_cert_issuer(client_pem :str, expected_subjects :list) -> str:
client_cert = None
if len(expected_subjects) > 0:
client_cert_path = Path(client_pem)
cert = load_certificate(FILETYPE_PEM, client_cert_path.read_bytes())
issuer_subject = cert.get_issuer()
for check in expected_subjects:
if issuer_subject.commonName == check.commonName:
client_cert = client_pem
break
if client_cert is None or not isinstance(client_cert, str):
raise Exception('X509_V_ERR_SUBJECT_ISSUER_MISMATCH') # OpenSSL error code 29
return client_cert
In a real app (not an example snippet) you would have a database of some sort to take the server subject and lookup the location of the cert to load - this example does it in reverse for demonstration only.
Make the TLS connection, and capture any OpenSSL errors:
from socket import socket, AF_INET, SOCK_STREAM
from OpenSSL import SSL
from OpenSSL.crypto import X509, FILETYPE_PEM
from certifi import where
import idna
def openssl_verifier(conn :SSL.Connection, server_cert :X509, errno :int, depth :int, preverify_ok :int):
ok = 1
verifier_errors = conn.get_app_data()
if not isinstance(verifier_errors, list):
verifier_errors = []
if errno in OPENSSL_CODES.keys():
ok = 0
verifier_errors.append((server_cert, OPENSSL_CODES[errno]))
conn.set_app_data(verifier_errors)
return ok
client_pem = '/path/to/client.pem'
client_issuer_ca = '/path/to/ca.pem'
host = 'example.com'
port = 443
ctx = SSL.Context(method=SSL.SSLv23_METHOD) # will negotiate TLS1.3 or lower protocol, what every is highest possible during negotiation
ctx.load_verify_locations(cafile=where())
if client_pem is not None:
ctx.use_certificate_file(certfile=client_pem, filetype=FILETYPE_PEM)
if client_issuer_ca is not None:
ctx.load_client_ca(cafile=client_issuer_ca)
ctx.set_verify(SSL.VERIFY_NONE, openssl_verifier)
ctx.check_hostname = False
conn = SSL.Connection(ctx, socket(AF_INET, SOCK_STREAM))
conn.connect((host, port))
conn.settimeout(3)
conn.set_tlsext_host_name(idna.encode(host))
conn.setblocking(1)
conn.set_connect_state()
try:
conn.do_handshake()
verifier_errors = conn.get_app_data()
except SSL.Error as err:
raise SSL.Error from err
finally:
conn.close()
# handle your errors in your main app
print(verifier_errors)
Just make sure you handle those OPENSSL_CODES errors if any are encountered, the lookup dictionary is here.
Many validations occur pre verification inside OpenSSL itself and all PyOpenSSL will do is a basic validation. so we need to access these codes from OpenSSL if we want to do Client Authentication, i.e. on the client and throw away the response from an untrusted server if it fails any authentication checks on the client side, per Client Authorisation or rather mutual-TLS dictates
#Stof's solution is more complete than this one.
So I selected his answer as 'official' answer.
This answer predates his, but might still be of some interest.
With #Steffen Ullrich's help I found following solution,
which works for all the (nginx with a ssl_client_certificate setting) servers that I tested with.
It requires to install an external package
pip install pyopenssl
Then following work will work:
import socket
from OpenSSL import SSL
def get_client_cert_cas(hostname, port):
ctx = SSL.Context(SSL.SSLv23_METHOD)
# If we don't force to NOT use TLSv1.3 get_client_ca_list() returns
# an empty result
ctx.set_options(SSL.OP_NO_TLSv1_3)
sock = SSL.Connection(ctx, socket.socket(socket.AF_INET, socket.SOCK_STREAM))
# next line for SNI
sock.set_tlsext_host_name(hostname.encode("utf-8"))
sock.connect((hostname, port))
# without handshake get_client_ca_list will be empty
sock.do_handshake()
return sock.get_client_ca_list()
The line sock.do_handshake() is required to trigger enough of the SSL protocol. Otherwise client_ca_list information doesn't seem to be populated.
At least for the servers, that I tested I had to make sure TLSv1.3 is not used. I don't know whether this is a bug, a feature or whether with TLSv1.3 another function has to be called prior to calling get_client_ca_list()
I am no pyopenssl expert, but could imagine, that there is a more elegant / more explicit way to get the same behavior.
but so far this works for me for all the servers, that I encountered.
I am trying to override the IP address for the destination host on the fly using urllib3, while I am passing client certificates. Here is my code:
import urllib3
conn = urllib3.connection_from_url('https://MYHOST', ca_certs='ca_crt.pem', key_file='pr.pem', cert_file='crt.pem', cert_reqs='REQUIRED')
response = conn.request('GET', 'https://MYHOST/OBJ', headers={"HOST": "MYHOST"})
print(response.data)
I was thinking to use transport adapters, but I am not quite sure how to do it without using sessions.
Any thoughts or help?
I guess we can follow the solutions presented here: Python 'requests' library - define specific DNS?
Hence a nasty way to do this is to override the hostname to the IP address we want by adding:
from urllib3.util import connection
import urllib3
hostname = "MYHOST"
host_ip = "10.10.10.10"
_orig_create_connection = connection.create_connection
def patched_create_connection(address, *args, **kwargs):
overrides = {
hostname: host_ip
}
host, port = address
if host in overrides:
return _orig_create_connection((overrides[host], port), *args, **kwargs)
else:
return _orig_create_connection((host, port), *args, **kwargs)
connection.create_connection = patched_create_connection
conn = urllib3.connection_from_url('https://MYHOST', ca_certs='ca_crt.pem', key_file='pr.pem', cert_file='crt.pem', cert_reqs='REQUIRED')
response = conn.request('GET', 'https://MYHOST/OBJ', headers={"HOST": "MYHOST"})
print(response.data)
But, again based on the posted link, the better way to implement it is to have a proper adapter to override the IP address.
The client.Agent class has a connection timeout argument:
agent = client.Agent(reactor, connectTimeout=timeout, pool=pool)
How can this timeout be set when using client.ProxyAgent?
auth = base64.b64encode("%s:%s" % (username, password))
headers['Proxy-Authorization'] = ["Basic " + auth.strip()]
endpoint = endpoints.TCP4ClientEndpoint(reactor, host, port)
agent = client.ProxyAgent(endpoint, reactor=reactor, pool=pool)
The TCP4ClientEndpoint you pass to ProxyAgent can be initialized with a timeout.
auth = base64.b64encode("%s:%s" % (username, password))
headers['Proxy-Authorization'] = ["Basic " + auth.strip()]
endpoint = endpoints.TCP4ClientEndpoint(reactor, host, port, timeout=yourTimeout)
agent = client.ProxyAgent(endpoint, reactor=reactor, pool=pool)
This is supposing you want to set the timeout for connecting to the proxy. If you wanted to set the timeout used by the proxy to connect to the upstream HTTP server, you can't control this.
It looks like client.ProxyAgent doesn't have a connectTimeout property:
class ProxyAgent(_AgentBase):
"""
An HTTP agent able to cross HTTP proxies.
#ivar _proxyEndpoint: The endpoint used to connect to the proxy.
#since: 11.1
"""
def __init__(self, endpoint, reactor=None, pool=None):
if reactor is None:
from twisted.internet import reactor
_AgentBase.__init__(self, reactor, pool)
self._proxyEndpoint = endpoint
def request(self, method, uri, headers=None, bodyProducer=None):
"""
Issue a new request via the configured proxy.
"""
# Cache *all* connections under the same key, since we are only
# connecting to a single destination, the proxy:
key = ("http-proxy", self._proxyEndpoint)
# To support proxying HTTPS via CONNECT, we will use key
# ("http-proxy-CONNECT", scheme, host, port), and an endpoint that
# wraps _proxyEndpoint with an additional callback to do the CONNECT.
return self._requestWithEndpoint(key, self._proxyEndpoint, method,
_URI.fromBytes(uri), headers,
bodyProducer, uri)
ProxyAgent inherits from the same class Agent does (_AgentBase) and not from Agent itself.
I just started using Twisted. I want to connect to an FTP server and perform some basic operations (use threading if possible). I am using this example.
Which does the job quite well. The question is how to add a SOCKS4/5 proxy usage to the code? Can somebody please provide a working example? I have tried this link too.
But,
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.
"""
An example of using the FTP client
"""
# Twisted imports
from twisted.protocols.ftp import FTPClient, FTPFileListProtocol
from twisted.internet.protocol import Protocol, ClientCreator
from twisted.python import usage
from twisted.internet import reactor, endpoints
# Socks support test
from socksclient import SOCKSv4ClientProtocol, SOCKSWrapper
from twisted.web import client
# Standard library imports
import string
import sys
try:
from cStringIO import StringIO
except ImportError:
from StringIO import StringIO
class BufferingProtocol(Protocol):
"""Simple utility class that holds all data written to it in a buffer."""
def __init__(self):
self.buffer = StringIO()
def dataReceived(self, data):
self.buffer.write(data)
# Define some callbacks
def success(response):
print 'Success! Got response:'
print '---'
if response is None:
print None
else:
print string.join(response, '\n')
print '---'
def fail(error):
print 'Failed. Error was:'
print error
def showFiles(result, fileListProtocol):
print 'Processed file listing:'
for file in fileListProtocol.files:
print ' %s: %d bytes, %s' \
% (file['filename'], file['size'], file['date'])
print 'Total: %d files' % (len(fileListProtocol.files))
def showBuffer(result, bufferProtocol):
print 'Got data:'
print bufferProtocol.buffer.getvalue()
class Options(usage.Options):
optParameters = [['host', 'h', 'example.com'],
['port', 'p', 21],
['username', 'u', 'webmaster'],
['password', None, 'justapass'],
['passive', None, 0],
['debug', 'd', 1],
]
# Socks support
def wrappercb(proxy):
print "connected to proxy", proxy
pass
def run():
def sockswrapper(proxy, url):
dest = client._parse(url) # scheme, host, port, path
endpoint = endpoints.TCP4ClientEndpoint(reactor, dest[1], dest[2])
return SOCKSWrapper(reactor, proxy[1], proxy[2], endpoint)
# Get config
config = Options()
config.parseOptions()
config.opts['port'] = int(config.opts['port'])
config.opts['passive'] = int(config.opts['passive'])
config.opts['debug'] = int(config.opts['debug'])
# Create the client
FTPClient.debug = config.opts['debug']
creator = ClientCreator(reactor, FTPClient, config.opts['username'],
config.opts['password'], passive=config.opts['passive'])
#creator.connectTCP(config.opts['host'], config.opts['port']).addCallback(connectionMade).addErrback(connectionFailed)
# Socks support
proxy = (None, '1.1.1.1', 1111, True, None, None)
sw = sockswrapper(proxy, "ftp://example.com")
d = sw.connect(creator)
d.addCallback(wrappercb)
reactor.run()
def connectionFailed(f):
print "Connection Failed:", f
reactor.stop()
def connectionMade(ftpClient):
# Get the current working directory
ftpClient.pwd().addCallbacks(success, fail)
# Get a detailed listing of the current directory
fileList = FTPFileListProtocol()
d = ftpClient.list('.', fileList)
d.addCallbacks(showFiles, fail, callbackArgs=(fileList,))
# Change to the parent directory
ftpClient.cdup().addCallbacks(success, fail)
# Create a buffer
proto = BufferingProtocol()
# Get short listing of current directory, and quit when done
d = ftpClient.nlst('.', proto)
d.addCallbacks(showBuffer, fail, callbackArgs=(proto,))
d.addCallback(lambda result: reactor.stop())
# this only runs if the module was *not* imported
if __name__ == '__main__':
run()
I know the code is wrong. I need Solution.
Okay, so here's a solution (gist) that uses python's built-in ftplib, as well as the open source SocksiPy module.
It doesn't use twisted, and it doesn't explicitly use threads, but using and communicting between threads is pretty easily done with threading.Thread and threading.Queue in python's standard threading module
Basically, we need to subclass ftplib.FTP to support substituting our own create_connection method and add proxy configuration semantics.
The "main" logic just configures an FTP client that connects via a localhost socks proxy, such as one created by ssh -D localhost:1080 socksproxy.example.com, and downloads a source snapshot for GNU autoconf to the local disk.
import ftplib
import socket
import socks # socksipy (https://github.com/mikedougherty/SocksiPy)
class FTP(ftplib.FTP):
def __init__(self, host='', user='', passwd='', acct='',
timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
proxyconfig=None):
"""Like ftplib.FTP constructor, but with an added `proxyconfig` kwarg
`proxyconfig` should be a dictionary that may contain the following
keys:
proxytype - The type of the proxy to be used. Three types
are supported: PROXY_TYPE_SOCKS4 (including socks4a),
PROXY_TYPE_SOCKS5 and PROXY_TYPE_HTTP
addr - The address of the server (IP or DNS).
port - The port of the server. Defaults to 1080 for SOCKS
servers and 8080 for HTTP proxy servers.
rdns - Should DNS queries be preformed on the remote side
(rather than the local side). The default is True.
Note: This has no effect with SOCKS4 servers.
username - Username to authenticate with to the server.
The default is no authentication.
password - Password to authenticate with to the server.
Only relevant when username is also provided.
"""
self.proxyconfig = proxyconfig or {}
ftplib.FTP.__init__(self, host, user, passwd, acct, timeout)
def connect(self, host='', port=0, timeout=-999):
'''Connect to host. Arguments are:
- host: hostname to connect to (string, default previous host)
- port: port to connect to (integer, default previous port)
'''
if host != '':
self.host = host
if port > 0:
self.port = port
if timeout != -999:
self.timeout = timeout
self.sock = self.create_connection(self.host, self.port)
self.af = self.sock.family
self.file = self.sock.makefile('rb')
self.welcome = self.getresp()
return self.welcome
def create_connection(self, host=None, port=None):
host, port = host or self.host, port or self.port
if self.proxyconfig:
phost, pport = self.proxyconfig['addr'], self.proxyconfig['port']
err = None
for res in socket.getaddrinfo(phost, pport, 0, socket.SOCK_STREAM):
af, socktype, proto, canonname, sa = res
sock = None
try:
sock = socks.socksocket(af, socktype, proto)
sock.setproxy(**self.proxyconfig)
if self.timeout is not socket._GLOBAL_DEFAULT_TIMEOUT:
sock.settimeout(self.timeout)
sock.connect((host, port))
return sock
except socket.error as _:
err = _
if sock is not None:
sock.close()
if err is not None:
raise err
else:
raise socket.error("getaddrinfo returns an empty list")
else:
sock = socket.create_connection((host, port), self.timeout)
return sock
def ntransfercmd(self, cmd, rest=None):
size = None
if self.passiveserver:
host, port = self.makepasv()
conn = self.create_connection(host, port)
try:
if rest is not None:
self.sendcmd("REST %s" % rest)
resp = self.sendcmd(cmd)
# Some servers apparently send a 200 reply to
# a LIST or STOR command, before the 150 reply
# (and way before the 226 reply). This seems to
# be in violation of the protocol (which only allows
# 1xx or error messages for LIST), so we just discard
# this response.
if resp[0] == '2':
resp = self.getresp()
if resp[0] != '1':
raise ftplib.error_reply, resp
except:
conn.close()
raise
else:
raise Exception("Active transfers not supported")
if resp[:3] == '150':
# this is conditional in case we received a 125
size = ftplib.parse150(resp)
return conn, size
if __name__ == '__main__':
ftp = FTP(host='ftp.gnu.org', user='anonymous', passwd='guest',
proxyconfig=dict(proxytype=socks.PROXY_TYPE_SOCKS5, rdns=False,
addr='localhost', port=1080))
with open('autoconf-2.69.tar.xz', mode='w') as f:
ftp.retrbinary("RETR /gnu/autoconf/autoconf-2.69.tar.xz", f.write)
To elaborate why I asked some of my original questions:
1) Do you need to support active transfers or will PASV transfers be sufficient?
Active transfers are much harder to do via a socks proxy because they require the use of the PORT command. With the PORT command, your ftp client tells the FTP server to connect to you on a specific port (e.g., on your PC) in order to send the data. This is likely to not work for users behind a firewall or NAT/router. If your SOCKS proxy server is not behind a firewall, or has a public IP, it is possible to support active transfers, but it is complicated: It requires your SOCKS server (ssh -D does support this) and client library (socksipy does not) to support remote port binding. It also requires the appropriate hooks in the application (my example throws an exception if passiveserver = False) to do a remote BIND instead of a local one.
2) Does it have to use twisted?
Twisted is great, but I'm not the best at it, and I haven't found a really great SOCKS client implementation. Ideally there would be a library out there that allowed you to define and/or chain proxies together, returning an object that implements the IReactorTCP interface, but I have not yet found anything like this just yet.
3) Is your socks proxy behind a VIP or just a single host directly connected to the Internet?
This matters because of the way PASV transfer security works. In a PASV transfer, the client asks the server to provide a port to connect in order to start a data transfer. When the server accepts a connection on that port, it SHOULD verify the client is connected from the same source IP as the connection that requested the transfer. If your SOCKS server is behind a VIP, it is less likely that the outbound IP of the connection made for the PASV transfers will match the outbound IP of the primary communication connection.
I'm trying to use Twisted's ProxyAgent class to connect to a proxy server and make HTTP requests, however the server requires a username and password. Is it possible to specify these credentials to the server using ProxyAgent?
endpoint = TCP4ClientEndpoint(reactor, host, port)
agent = ProxyAgent(endpoint)
# Maybe need to pass auth credentials in the header here?
body = agent.request("GET", path)
Figured out the problem, the Proxy-Authorization field has to be set in the headers:
endpoint = TCP4ClientEndpoint(reactor, host, port)
agent = ProxyAgent(endpoint)
headers = {}
auth = base64.b64encode("%s:%s" % (username, password))
headers["Proxy-Authorization"] = ["Basic " + auth.strip()]
body = agent.request("GET", path, Headers(headers))