Trying to get the SSL certificate from a response in requests.
What is a good way to do this?
requests deliberately wraps up low-level stuff like this. Normally, the only thing you want to do is to verify that the certs are valid. To do that, just pass verify=True. If you want to use a non-standard cacert bundle, you can pass that too. For example:
resp = requests.get('https://example.com', verify=True, cert=['/path/to/my/ca.crt'])
Also, requests is primarily a set of wrappers around other libraries, mostly urllib3 and the stdlib's http.client (or, for 2.x, httplib) and ssl.
Sometimes, the answer is just to get at the lower-level objects (e.g., resp.raw is the urllib3.response.HTTPResponse), but in many cases that's impossible.
And this is one of those cases. The only objects that ever see the certs are an http.client.HTTPSConnection (or a urllib3.connectionpool.VerifiedHTTPSConnection, but that's just a subclass of the former) and an ssl.SSLSocket, and neither of those exist anymore by the time the request returns. (As the name connectionpool implies, the HTTPSConnection object is stored in a pool, and may be reused as soon as it's done; the SSLSocket is a member of the HTTPSConnection.)
So, you need to patch things so you can copy the data up the chain. It may be as simple as this:
HTTPResponse = requests.packages.urllib3.response.HTTPResponse
orig_HTTPResponse__init__ = HTTPResponse.__init__
def new_HTTPResponse__init__(self, *args, **kwargs):
orig_HTTPResponse__init__(self, *args, **kwargs)
try:
self.peercert = self._connection.sock.getpeercert()
except AttributeError:
pass
HTTPResponse.__init__ = new_HTTPResponse__init__
HTTPAdapter = requests.adapters.HTTPAdapter
orig_HTTPAdapter_build_response = HTTPAdapter.build_response
def new_HTTPAdapter_build_response(self, request, resp):
response = orig_HTTPAdapter_build_response(self, request, resp)
try:
response.peercert = resp.peercert
except AttributeError:
pass
return response
HTTPAdapter.build_response = new_HTTPAdapter_build_response
That's untested, so no guarantees; you may need to patch more than that.
Also, subclassing and overriding would probably be cleaner than monkeypatching (especially since HTTPAdapter was designed to be subclassed).
Or, even better, forking urllib3 and requests, modifying your fork, and (if you think this is legitimately useful) submitting pull requests upstream.
Anyway, now, from your code, you can do this:
resp.peercert
This will give you a dict with 'subject' and 'subjectAltName' keys, as returned by pyopenssl.WrappedSocket.getpeercert. If you instead want more information about the cert, try Christophe Vandeplas's variant of this answer that lets you get an OpenSSL.crypto.X509 object. If you want to get the entire peer certificate chain, see GoldenStake's answer.
Of course you may also want to pass along all the information necessary to verify the cert, but that's even easier, because it already passes through the top level.
To start, abarnert's answer is very complete. While chasing the proposed connection-close issue of Kalkran I actually discovered that the peercert didn't contain detailed information about the SSL Certificate.
I dug deeper in the connection and socket info and extracted the self.sock.connection.get_peer_certificate() function which contains great functions like:
get_subject() for CN
get_notAfter() and get_notBefore() for expiration dates
get_serial_number() and get_signature_algorithm() for crypto related technical details
...
Note that these are only available if you have pyopenssl installed on your system. Under the hood, urllib3 uses pyopenssl if it's available and the standard library's ssl module otherwise. The self.sock.connection attribute shown below only exists if self.sock is a urllib3.contrib.pyopenssl.WrappedSocket, not if it's a ssl.SSLSocket. You can install pyopenssl with pip install pyopenssl.
Once that's done, the code becomes:
import requests
HTTPResponse = requests.packages.urllib3.response.HTTPResponse
orig_HTTPResponse__init__ = HTTPResponse.__init__
def new_HTTPResponse__init__(self, *args, **kwargs):
orig_HTTPResponse__init__(self, *args, **kwargs)
try:
self.peer_certificate = self._connection.peer_certificate
except AttributeError:
pass
HTTPResponse.__init__ = new_HTTPResponse__init__
HTTPAdapter = requests.adapters.HTTPAdapter
orig_HTTPAdapter_build_response = HTTPAdapter.build_response
def new_HTTPAdapter_build_response(self, request, resp):
response = orig_HTTPAdapter_build_response(self, request, resp)
try:
response.peer_certificate = resp.peer_certificate
except AttributeError:
pass
return response
HTTPAdapter.build_response = new_HTTPAdapter_build_response
HTTPSConnection = requests.packages.urllib3.connection.HTTPSConnection
orig_HTTPSConnection_connect = HTTPSConnection.connect
def new_HTTPSConnection_connect(self):
orig_HTTPSConnection_connect(self)
try:
self.peer_certificate = self.sock.connection.get_peer_certificate()
except AttributeError:
pass
HTTPSConnection.connect = new_HTTPSConnection_connect
You will be able to access the result easily:
r = requests.get('https://yourdomain.tld', timeout=0.1)
print('Expires on: {}'.format(r.peer_certificate.get_notAfter()))
print(dir(r.peer_certificate))
If, like me, you want to ignore SSL Certificate warnings just add the following in the top of the file and do not SSL verify:
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
r = requests.get('https://yourdomain.tld', timeout=0.1, verify=False)
print(dir(r.peer_certificate))
Thanks for everyone's awesome answers.
It helped me over engineer an answer to this question:
How to add a custom CA Root certificate to the CA Store used by Python in Windows?
UPDATE 2019-02-12
Please take a look at Cert Human: SSL Certificates for Humans for an impressive rewrite of my https://github.com/neozenith/get-ca-py project by lifehackjim.
I have archived the original repository now.
Stand alone snippet
#! /usr/bin/env python
# -*- coding: utf-8 -*-
"""
Get Certificates from a request and dump them.
"""
import argparse
import sys
import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
"""
Inspired by the answers from this Stackoverflow question:
https://stackoverflow.com/questions/16903528/how-to-get-response-ssl-certificate-from-requests-in-python
What follows is a series of patching the low level libraries in requests.
"""
"""
https://stackoverflow.com/a/47931103/622276
"""
sock_requests = requests.packages.urllib3.contrib.pyopenssl.WrappedSocket
def new_getpeercertchain(self, *args, **kwargs):
x509 = self.connection.get_peer_cert_chain()
return x509
sock_requests.getpeercertchain = new_getpeercertchain
"""
https://stackoverflow.com/a/16904808/622276
"""
HTTPResponse = requests.packages.urllib3.response.HTTPResponse
orig_HTTPResponse__init__ = HTTPResponse.__init__
def new_HTTPResponse__init__(self, *args, **kwargs):
orig_HTTPResponse__init__(self, *args, **kwargs)
try:
self.peercertchain = self._connection.sock.getpeercertchain()
except AttributeError:
pass
HTTPResponse.__init__ = new_HTTPResponse__init__
HTTPAdapter = requests.adapters.HTTPAdapter
orig_HTTPAdapter_build_response = HTTPAdapter.build_response
def new_HTTPAdapter_build_response(self, request, resp):
response = orig_HTTPAdapter_build_response(self, request, resp)
try:
response.peercertchain = resp.peercertchain
except AttributeError:
pass
return response
HTTPAdapter.build_response = new_HTTPAdapter_build_response
"""
Attempt to wrap in a somewhat usable CLI
"""
def cli(args):
parser = argparse.ArgumentParser(description="Request any URL and dump the certificate chain")
parser.add_argument("url", metavar="URL", type=str, nargs=1, help="Valid https URL to be handled by requests")
verify_parser = parser.add_mutually_exclusive_group(required=False)
verify_parser.add_argument("--verify", dest="verify", action="store_true", help="Explicitly set SSL verification")
verify_parser.add_argument(
"--no-verify", dest="verify", action="store_false", help="Explicitly disable SSL verification"
)
parser.set_defaults(verify=True)
return vars(parser.parse_args(args))
def dump_pem(cert, outfile="ca-chain.crt"):
"""Use the CN to dump certificate to PEM format"""
PyOpenSSL = requests.packages.urllib3.contrib.pyopenssl
pem_data = PyOpenSSL.OpenSSL.crypto.dump_certificate(PyOpenSSL.OpenSSL.crypto.FILETYPE_PEM, cert)
issuer = cert.get_issuer().get_components()
print(pem_data.decode("utf-8"))
with open(outfile, "a") as output:
for part in issuer:
output.write(part[0].decode("utf-8"))
output.write("=")
output.write(part[1].decode("utf-8"))
output.write(",\t")
output.write("\n")
output.write(pem_data.decode("utf-8"))
if __name__ == "__main__":
cli_args = cli(sys.argv[1:])
url = cli_args["url"][0]
req = requests.get(url, verify=cli_args["verify"])
for cert in req.peercertchain:
dump_pem(cert)
This, although not pretty at all, works:
import requests
req = requests.get('https://httpbin.org')
pool = req.connection.poolmanager.connection_from_url('https://httpbin.org')
conn = pool.pool.get()
# get() removes it from the pool, so put it back in
pool.pool.put(conn)
print(conn.sock.getpeercert())
To start, abarnert's answer is very complete
But I would like to add, that in the case you're looking for the peer cert chain, you would need to patch yet another piece of code
import requests
sock_requests = requests.packages.urllib3.contrib.pyopenssl.WrappedSocket
def new_getpeercertchain(self,*args, **kwargs):
x509 = self.connection.get_peer_cert_chain()
return x509
sock_requests.getpeercertchain = new_getpeercertchain
after that you can call it in a very similiar manner as the accepted answer
HTTPResponse = requests.packages.urllib3.response.HTTPResponse
orig_HTTPResponse__init__ = HTTPResponse.__init__
def new_HTTPResponse__init__(self, *args, **kwargs):
orig_HTTPResponse__init__(self, *args, **kwargs)
try:
self.peercertchain = self._connection.sock.getpeercertchain()
except AttributeError:
pass
HTTPResponse.__init__ = new_HTTPResponse__init__
HTTPAdapter = requests.adapters.HTTPAdapter
orig_HTTPAdapter_build_response = HTTPAdapter.build_response
def new_HTTPAdapter_build_response(self, request, resp):
response = orig_HTTPAdapter_build_response(self, request, resp)
try:
response.peercertchain = resp.peercertchain
except AttributeError:
pass
return response
HTTPAdapter.build_response = new_HTTPAdapter_build_response
you will get resp.peercertchain which contains a tuple of OpenSSL.crypto.X509 objects
For retrieving the details of a certificate such as CN and expiry date the following script adapted from this example works well. It also avoids some errors I got which I assume were due to incorrect/incompatible versions of requests and urllib3: "AttributeError: 'SSLSocket' object has no attribute 'connection'" and "AttributeError: 'VerifiedHTTPSConnection' object has no attribute 'peer_certificate'"
from OpenSSL.SSL import Connection, Context, SSLv3_METHOD, TLSv1_2_METHOD
from datetime import datetime, time
import socket
host = 'www.google.com'
try:
try:
ssl_connection_setting = Context(SSLv3_METHOD)
except ValueError:
ssl_connection_setting = Context(TLSv1_2_METHOD)
ssl_connection_setting.set_timeout(5)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect((host, 443))
c = Connection(ssl_connection_setting, s)
c.set_tlsext_host_name(str.encode(host))
c.set_connect_state()
c.do_handshake()
cert = c.get_peer_certificate()
print("Is Expired: ", cert.has_expired())
print("Issuer: ", cert.get_issuer())
subject_list = cert.get_subject().get_components()
cert_byte_arr_decoded = {}
for item in subject_list:
cert_byte_arr_decoded.update({item[0].decode('utf-8'): item[1].decode('utf-8')})
print(cert_byte_arr_decoded)
if len(cert_byte_arr_decoded) > 0:
print("Subject: ", cert_byte_arr_decoded)
if cert_byte_arr_decoded["CN"]:
print("Common Name: ", cert_byte_arr_decoded["CN"])
end_date = datetime.strptime(str(cert.get_notAfter().decode('utf-8')), "%Y%m%d%H%M%SZ")
print("Not After (UTC Time): ", end_date)
diff = end_date - datetime.now()
print('Summary: "{}" SSL certificate expires on {} i.e. {} days.'.format(host, end_date, diff.days))
c.shutdown()
s.close()
except:
print("Connection to {} failed.".format(host))
This script requires Python 3 and pyOpenSSL.
Cleaner(-ish) solution, based on previous very good answers !
need to patch requests.Adapter source file before overriding
HTTPResponse class (pending pull request: https://github.com/psf/requests/pull/6039):
add static class variable to class HTTPAdapter(BaseAdapter) : _clsHTTPResponse = HTTPResponse
modify send() method to use _clsHTTPResponse rather than direct HTTPResponse object creation: resp = _clsHTTPResponse.from_httplib(...
use this code:
"""
Subclassing HTTP / requests to get peer_certificate back from lower levels
"""
from typing import Optional, Mapping, Any
from http.client import HTTPSConnection
from requests.adapters import HTTPAdapter, DEFAULT_POOLBLOCK
from urllib3.poolmanager import PoolManager,key_fn_by_scheme
from urllib3.connectionpool import HTTPSConnectionPool,HTTPConnectionPool
from urllib3.connection import HTTPSConnection,HTTPConnection
from urllib3.response import HTTPResponse as URLLIB3_HTTPResponse
#force urllib3 to use pyopenssl
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
class HTTPSConnection_withcert(HTTPSConnection):
def __init__(self, *args, **kw):
self.peer_certificate = None
super().__init__(*args, **kw)
def connect(self):
res = super().connect()
self.peer_certificate = self.sock.connection.get_peer_certificate()
return res
class HTTPResponse_withcert(URLLIB3_HTTPResponse):
def __init__(self, *args, **kwargs):
self.peer_certificate = None
res = super().__init__( *args, **kwargs)
self.peer_certificate = self._connection.peer_certificate
return res
class HTTPSConnectionPool_withcert(HTTPSConnectionPool):
ConnectionCls = HTTPSConnection_withcert
ResponseCls = HTTPResponse_withcert
class PoolManager_withcert(PoolManager):
def __init__(
self,
num_pools: int = 10,
headers: Optional[Mapping[str, str]] = None,
**connection_pool_kw: Any,
) -> None:
super().__init__(num_pools,headers,**connection_pool_kw)
self.pool_classes_by_scheme = {"http": HTTPConnectionPool, "https": HTTPSConnectionPool_withcert}
self.key_fn_by_scheme = key_fn_by_scheme.copy()
class HTTPAdapter_withcert(HTTPAdapter):
_clsHTTPResponse = HTTPResponse_withcert
def build_response(self, request, resp):
response = super().build_response( request, resp)
response.peer_certificate = resp.peer_certificate
return response
def init_poolmanager(self, connections, maxsize, block=DEFAULT_POOLBLOCK, **pool_kwargs):
#do not call super() to not initialize PoolManager twice
# save these values for pickling
self._pool_connections = connections
self._pool_maxsize = maxsize
self._pool_block = block
self.poolmanager = PoolManager_withcert(num_pools=connections,
maxsize=maxsize,
block=block,
strict=True,
**pool_kwargs)
class Session_withcert(Session):
def __init__(self):
super().__init__()
self.mount('https://', HTTPAdapter_withcert())
And thats all !
You can now use your new session Session_withcert() like the base one, but you can also do :
ss= Session_withcert()
resp=ss.get("https://www.google.fr")
resp.peer_certificate.get_subject()
print(resp.peer_certificate.get_subject())
which will output:
<X509Name object '/CN=*.google.fr'>
Just do this:
import requests
with requests.get("https://www.bilibili.com", stream=True) as response:
certificate_info = response.raw.connection.sock.getpeercert()
subject = dict(x[0] for x in certificate_info['subject'])
issuer = dict(x[0] for x in certificate_info['issuer'])
print("commonName:", subject['commonName'])
print("issuer:", issuer['commonName'])
Then the output is :
commonName: *.bilibili.com
issuer: GlobalSign RSA OV SSL CA 2018
Wish to help you.
I have a site that has PKI security enabled. Each client used either a card reader to load their certificate, or the certificate is installed in the IE certificate storage on their box.
So my question are:
How can I use either the card reader certificate or the certificate stored on the system to verify the system?
How do I pass the credentials onto the site to say, hey I'm me and I can access the service? They example can be using soft certificates. I can figure out the card reader part later.
I've been searching around, and I haven't come up with anything to help me in this situation. Django has a bunch of modules, but this isn't an option because I'm only concerned of the client side of things. I'm not creating a site to host the service. I need to just access these services.
I have this code working sort of. I just do not know how to handle the redirect I am getting:
import httplib
KEYFILE = r"C:\cert\my.key"
CERTFILE = r"c:\cert\my.pem"
HOSTNAME = 'machine.com'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = KEYFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/arcgis/sharing/rest?f=json')
conn.endheaders()
response = conn.getresponse()
print response.read()
The result of all of this is:
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved here.</p>
</body></html>
Any help provided would be great!
Software specs: python 2.7.8, Windows 2012 R2
I created a PKI handler to handle the requests so I can use it work urllib2 library.
import httplib, urllib2
class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
def __init__(self, key, cert):
urllib2.HTTPSHandler.__init__(self)
self.key = key
self.cert = cert
def https_open(self, req):
#Rather than pass in a reference to a connection class, we pass in
# a reference to a function which, for all intents and purposes,
# will behave as a constructor
return self.do_open(self.getConnection, req)
def getConnection(self, host, timeout=300):
return httplib.HTTPSConnection(host,
key_file=self.key,
cert_file=self.cert,
timeout=timeout)
To use this, you will need to use a cookiejar with the handler.
from cookielib import CookieJar
cookiejar = CookieJay()
handlers = []
handlers.append(HTTPSClientAuthHandler(somekey, somecert))
handlers.append(urllib2.HTTPCookieProcessor(cookiejar))
opener = urllib2.build_opener(*handlers)
... do other urllib2 calls ....
Hope this helps everyone!
Try this code
#!/usr/bin/env python
import httplib
CERTFILE = '/home/robr/mycert'
HOSTNAME = 'localhost'
conn = httplib.HTTPSConnection(
HOSTNAME,
key_file = CERTFILE,
cert_file = CERTFILE
)
conn.putrequest('GET', '/ssltest/')
conn.endheaders()
response = conn.getresponse()
print response.read()
Is there anyway to get suds returning the SoapRequest (in XML) without sending it?
The idea is that the upper levels of my program can call my API with an additional boolean argument (simulation).
If simulation == false then process the other params and send the request via suds
If simulation == false then process the other params, create the XML using suds (or any other way) and return it to the caller without sending it to the host.
I already implemented a MessagePlugin follwing https://fedorahosted.org/suds/wiki/Documentation#MessagePlugin, but I am not able to get the XML, stop the request and send back the XML to the caller...
Regards
suds uses a "transport" class called HttpAuthenticated by default. That is where the actual send occurs. So theoretically you could try subclassing that:
from suds.client import Client
from suds.transport import Reply
from suds.transport.https import HttpAuthenticated
class HttpAuthenticatedWithSimulation(HttpAuthenticated):
def send(self, request):
is_simulation = request.headers.pop('simulation', False)
if is_simulation:
# don't actually send the SOAP request, just return its XML
return Reply(200, request.headers.dict, request.msg)
return HttpAuthenticated(request)
...
sim_transport = HttpAuthenticatedWithSimulation()
client = Client(url, transport=sim_transport,
headers={'simulation': is_simulation})
It's a little hacky. (For example, this relies on HTTP headers to pass the boolean simulation option down to the transport level.) But I hope this illustrates the idea.
The solution that I implemented is:
class CustomTransportClass(HttpTransport):
def __init__(self, *args, **kwargs):
HttpTransport.__init__(self, *args, **kwargs)
self.opener = MutualSSLHandler() # I use a special opener to enable a mutual SSL authentication
def send(self,request):
print "===================== 1-* request is going ===================="
is_simulation = request.headers['simulation']
if is_simulation == "true":
# don't actually send the SOAP request, just return its XML
print "This is a simulation :"
print request.message
return Reply(200, request.headers, request.message )
return HttpTransport.send(self,request)
sim_transport = CustomTransportClass()
client = Client(url, transport=sim_transport,
headers={'simulation': is_simulation})
Thanks for your help,
I'm using the python suds module and would like to retrieve the response headers (specifically Last-Modified) from a suds response.
With more effort than ought be necessary is the answer.
I've got suds version 0.3.9 here. I had to subclass the transport class in use and wrap the send method to store the last received headers on in the transport class.
import logging
logging.basicConfig(level=logging.INFO)
#logging.getLogger('suds.client').setLevel(logging.DEBUG)
#logging.getLogger('suds.transport').setLevel(logging.DEBUG)
#logging.getLogger('suds.xsd.schema').setLevel(logging.DEBUG)
#logging.getLogger('suds.wsdl').setLevel(logging.DEBUG)
from suds.client import Client
from suds.xsd.doctor import ImportDoctor, Import
from suds.transport.https import HttpAuthenticated
class MyTransport(HttpAuthenticated):
def __init__(self,*args,**kwargs):
HttpAuthenticated.__init__(self, *args, **kwargs)
self.last_headers = None
def send(self,request):
result = HttpAuthenticated.send(self, request)
self.last_headers = result.headers
return result
doctor = ImportDoctor(Import('http://schemas.xmlsoap.org/soap/encoding/'))
svc_url = 'https://server/Service?wsdl'
svc_user = 'username'
svc_pass = 'password'
client = Client(svc_url,doctor=doctor,transport=MyTransport())
# For some reason I can't be bothered to investigate, setting the username and password in
# client kwargs doesn't pass them to the custom transport:
client.set_options(location=svc_url.partition('?')[0],username=svc_user,password=svc_pass)
# call a method
client.service.SomeMethod()
# look at headers
client.options.transport.last_headers
I need to create a secure channel between my server and a remote web service. I'll be using HTTPS with a client certificate. I'll also need to validate the certificate presented by the remote service.
How can I use my own client certificate with urllib2?
What will I need to do in my code to ensure that the remote certificate is correct?
Because alex's answer is a link, and the code on that page is poorly formatted, I'm just going to put this here for posterity:
import urllib2, httplib
class HTTPSClientAuthHandler(urllib2.HTTPSHandler):
def __init__(self, key, cert):
urllib2.HTTPSHandler.__init__(self)
self.key = key
self.cert = cert
def https_open(self, req):
# Rather than pass in a reference to a connection class, we pass in
# a reference to a function which, for all intents and purposes,
# will behave as a constructor
return self.do_open(self.getConnection, req)
def getConnection(self, host, timeout=300):
return httplib.HTTPSConnection(host, key_file=self.key, cert_file=self.cert)
opener = urllib2.build_opener(HTTPSClientAuthHandler('/path/to/file.pem', '/path/to/file.pem.') )
response = opener.open("https://example.org")
print response.read()
Here's a bug in the official Python bugtracker that looks relevant, and has a proposed patch.
Per Antoine Pitrou's response to the issue linked in Hank Gay's answer, this can be simplified somewhat (as of 2011) by using the included ssl library:
import ssl
import urllib.request
context = ssl.create_default_context()
context.load_cert_chain('/path/to/file.pem', '/path/to/file.key')
opener = urllib.request.build_opener(urllib.request.HTTPSHandler(context=context))
response = opener.open('https://example.org')
print(response.read())
(Python 3 code, but the ssl library is also available in Python 2).
The load_cert_chain function also accepts an optional password parameter, allowing the private key to be encrypted.
check http://www.osmonov.com/2009/04/client-certificates-with-urllib2.html