How do I make a simple request for certificate revocation status to an EJBCA OSCP Responder using the Python requests library?
Example:
# Determine if certificate has been revoked
ocsp_url = req_cert.extensions[2].value[0].access_location.value
ocsp_headers = {"whatGoes: here?"}
ocsp_body = {"What goes here?"}
ocsp_response = requests.get(ocsp_url, ocsp_headers, ocsp_body)
if (ocsp_response == 'revoked'):
return func.HttpResponse(
"Certificate is not valid (Revoked)."
)
Basically it involves the following steps:
retrieve the corresponding cert for a hostname
if a corresponding entry is contained in the certificate, you can query the extensions via AuthorityInformationAccessOID.CA_ISSUERS, which will provide you with a link to the issuer certificate if successful
retrieve the issuer cert with this link
similarly you get via AuthorityInformationAccessOID.OCSP the corresponding OCSP server
with this information about the current cert, the issuer_cert and the ocsp server you can feed OCSPRequestBuilder to create an OCSP request
use requests.get to get the OCSP response
from the OCSP response retrieve the certificate_status
To retrieve a cert for a hostname and port, you can use this fine answer: https://stackoverflow.com/a/49132495. The OCSP handling in Python is documented here: https://cryptography.io/en/latest/x509/ocsp.html.
Code
If you convert the above points into a self-contained example, it looks something like this:
import base64
import ssl
import requests
from urllib.parse import urljoin
from cryptography import x509
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.hashes import SHA256
from cryptography.x509 import ocsp
from cryptography.x509.ocsp import OCSPResponseStatus
from cryptography.x509.oid import ExtensionOID, AuthorityInformationAccessOID
def get_cert_for_hostname(hostname, port):
conn = ssl.create_connection((hostname, port))
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
sock = context.wrap_socket(conn, server_hostname=hostname)
certDER = sock.getpeercert(True)
certPEM = ssl.DER_cert_to_PEM_cert(certDER)
return x509.load_pem_x509_certificate(certPEM.encode('ascii'), default_backend())
def get_issuer(cert):
aia = cert.extensions.get_extension_for_oid(ExtensionOID.AUTHORITY_INFORMATION_ACCESS).value
issuers = [ia for ia in aia if ia.access_method == AuthorityInformationAccessOID.CA_ISSUERS]
if not issuers:
raise Exception(f'no issuers entry in AIA')
return issuers[0].access_location.value
def get_ocsp_server(cert):
aia = cert.extensions.get_extension_for_oid(ExtensionOID.AUTHORITY_INFORMATION_ACCESS).value
ocsps = [ia for ia in aia if ia.access_method == AuthorityInformationAccessOID.OCSP]
if not ocsps:
raise Exception(f'no ocsp server entry in AIA')
return ocsps[0].access_location.value
def get_issuer_cert(ca_issuer):
issuer_response = requests.get(ca_issuer)
if issuer_response.ok:
issuerDER = issuer_response.content
issuerPEM = ssl.DER_cert_to_PEM_cert(issuerDER)
return x509.load_pem_x509_certificate(issuerPEM.encode('ascii'), default_backend())
raise Exception(f'fetching issuer cert failed with response status: {issuer_response.status_code}')
def get_oscp_request(ocsp_server, cert, issuer_cert):
builder = ocsp.OCSPRequestBuilder()
builder = builder.add_certificate(cert, issuer_cert, SHA256())
req = builder.build()
req_path = base64.b64encode(req.public_bytes(serialization.Encoding.DER))
return urljoin(ocsp_server + '/', req_path.decode('ascii'))
def get_ocsp_cert_status(ocsp_server, cert, issuer_cert):
ocsp_resp = requests.get(get_oscp_request(ocsp_server, cert, issuer_cert))
if ocsp_resp.ok:
ocsp_decoded = ocsp.load_der_ocsp_response(ocsp_resp.content)
if ocsp_decoded.response_status == OCSPResponseStatus.SUCCESSFUL:
return ocsp_decoded.certificate_status
else:
raise Exception(f'decoding ocsp response failed: {ocsp_decoded.response_status}')
raise Exception(f'fetching ocsp cert status failed with response status: {ocsp_resp.status_code}')
def get_cert_status_for_host(hostname, port):
print(' hostname:', hostname, "port:", port)
cert = get_cert_for_hostname(hostname, port)
ca_issuer = get_issuer(cert)
print(' issuer ->', ca_issuer)
issuer_cert = get_issuer_cert(ca_issuer)
ocsp_server = get_ocsp_server(cert)
print(' ocsp_server ->', ocsp_server)
return get_ocsp_cert_status(ocsp_server, cert, issuer_cert)
Test 1: Good Certificate
A test call like the following with a good certificate
status = get_cert_status_for_host('software7.com', 443)
print('software7.com:', status, '\n')
results in the following output:
hostname: software7.com port: 443
issuer -> http://cacerts.digicert.com/EncryptionEverywhereDVTLSCA-G1.crt
ocsp_server -> http://ocsp.digicert.com
software7.com: OCSPCertStatus.GOOD
Test 2: Revoked Certificate
Of course you also have to do a counter test with a revoked cert. Here revoked.badssl.com is the first choice:
status = get_cert_status_for_host('revoked.badssl.com', 443)
print('revoked.badssl.com:', status, '\n')
This gives as output:
hostname: revoked.badssl.com port: 443
issuer -> http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt
ocsp_server -> http://ocsp.digicert.com
revoked.badssl.com: OCSPCertStatus.REVOKED
AIA Retrieval of the Issuer Certificate
A typical scenario for a certificate relationship looks as follows:
The server provides the server certificate and usually one or more intermediate certificates during the TLS handshake. The word 'usually' is used intentionally: some servers are configured not to deliver intermediate certificates. The browsers then use AIA fetching to build the certification chain.
Up to two entries can be present in the Certificate Authority Information Access extension: The entry for downloading the issuer certificate and the link to the OCSP server.
These entries may also be missing, but a short test script that checks the certs of the 100 most popular servers shows that these entries are usually included in certificates issued by public certification authorities.
The CA Issuers entry may also be missing, but while the information about an OCSP server is available, it can be tested e.g. with OpenSSL using a self-signed certificate:
In this case you would have to determine the issuer certificate from the chain in the TLS handshake, it is the certificate that comes directly after the server certificate in the chain, see also the figure above.
Just for the sake of completeness: There is another case that can sometimes occur especially in conjunction with self-signed certificates: If no intermediate certificates are used, the corresponding root certificate (e.g. available in the local trust store) must be used as issuer certificate.
Related
I am trying to list the certificates that Python SSL is actually using to verify a remote server, but clearly don't understand how this works.
The following code works but seems to contradict itself.
A self-signed cert is installed in Windows: Local Computer\Trusted Root Certification Authorities\Certificates and works nicely in Chrome etc., but is not present in the certificate list that ssl has loaded.
import ssl
import socket
from pprint import pprint, pformat
hostname = 'myhost.mydom.tld' # self-signed cert
port = 443
context = ssl.create_default_context()
print(context.verify_mode) # VerifyMode.CERT_REQUIRED
# connect to server and fetch cert (verified to fail on invalid cert)
with socket.create_connection((hostname, port)) as sock:
with context.wrap_socket(sock, server_hostname=hostname) as ssock:
print(ssock.version())
cert = ssock.getpeercert()
pprint(cert) # the expected self-signed cert
print('-' * 60)
# print certs we verified against
pprint(context.get_ca_certs()) # self-signed cert not present
print('-' * 60)
# just make extra sure
for cert in context.get_ca_certs():
if 'MYHOST' in pformat(cert).upper():
pprint(cert) # never reached
So why am I not finding the expected self-signed cert in the list when it is in fact verified?
I've generated a self-signed certificate in Windows Server 2012 R2 [WSUS Server - 10.66.194.98] [Dec15.cer] and enabled SSL in all 'WSUS Administration' website. Now I want to use this in python code to contact with the server.
Dec15.cer
And I'm running into below error
ERROR: Host not reachable [HTTPSConnectionPool(host='10.66.194.98', port=8531): Max retries exceeded with url: /ApiRemoting30/WebService.asmx (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)'),))]
This is what I tried.
wsusutil.exe configuressl 10.66.194.98
Then I copied the Dec15.cer to python root directory. and ran the below code
from requests import Session
from requests_ntlm import HttpNtlmAuth
user = 'administrator'
password = '******'
session = Session()
session.cert = session.verify = 'Dec15.cer'
# session.verify = False
session.auth = HttpNtlmAuth(user, password)
print(session.get("https://10.66.194.98:8531/ApiRemoting30",
verify=session.verify,
cert=session.cert))
While the certificate in question Dec15.cer is a self-signed certificate it does not have basic constraints CA:true:
$ openssl x509 -text -in Dec15.cer
...
X509v3 extensions:
X509v3 Key Usage:
Key Encipherment, Data Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
But, as I already said in a comment, the certificates given to the verify parameter in requests must be CA certificates, i.e. have basic constraints CA:true.
I am trying to get Certificate issuer information (Common Name), but the code in the link doesn't work with some URLs.
How can i get Certificate issuer information in python?
import ssl, socket
hostname = 'google.com'
ctx = ssl.create_default_context()
s = ctx.wrap_socket(socket.socket(), server_hostname=hostname)
s.connect((hostname, 443))
cert = s.getpeercert()
subject = dict(x[0] for x in cert['subject'])
issued_to = subject['commonName']
>>> issued_to
u'*.google.com'
For example, I tried hostname "cds.ca", it says
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
but I could still get the Common name using Internet Explorer (*.cds.ca)
So I think I should use my own certificate(.cer) instead of using getpeercert(), then how should I change that line?
Or, is there any other way to achieve CN with my own certificate file?
If you just want to get the CN or other certificate details no matter if certificate validation succeeds or not, you have to disable the verification. Unfortunately a simple sock.getpeercert() will by design only return an empty dictionary if certificate validation is disabled. That's why one has to use sock.getpeercert(True) to get the binary representation of the certificate and extract the CN using OpenSSL.crypto from it:
import socket
import ssl
import OpenSSL.crypto as crypto
dst = ('cds.ca',443)
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(dst)
# upgrade the socket to SSL without checking the certificate
# !!!! don't transfer any sensitive data over this socket !!!!
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
s = ctx.wrap_socket(s, server_hostname=dst[0])
# get certificate
cert_bin = s.getpeercert(True)
x509 = crypto.load_certificate(crypto.FILETYPE_ASN1,cert_bin)
print("CN=" + x509.get_subject().CN)
My script
from stmplib import SMTP
con = SMTP(server, port)
con.starttls()
con.login(user, pass)
con.quit()
falls with error:
python2.7/ssl.py", line 847, in do_handshake self._sslobj.do_handshake()
When I run command openssl to this server it falls with error 21: Verify return code: 21 (unable to verify the first certificate).
I would like to know how to specify in smtplib of python option “always accept self-signed certificate when connect is established via tls to e-mail server"?
Like I do in requests.get setting key verify=False.
Update
This variant with custom smtp class and context = ssl._create_unverified_context() return the same error as above:
import smtplib
import ssl
class MySMTP(smtplib.SMTP):
def __init__(self, host='', port=0, timeout=5):
smtplib.SMTP.__init__(self, host, port, timeout=timeout)
self._host = host
def starttls(self, keyfile=None, certfile=None, context=None):
from urllib import _have_ssl
self.ehlo_or_helo_if_needed()
if not self.has_extn("starttls"):
raise SMTPNotSupportedError("STARTTLS extension not supported by server.")
(resp, reply) = self.docmd("STARTTLS")
if resp == 220:
if not _have_ssl:
raise RuntimeError("No SSL support included in this Python")
if context is not None and keyfile is not None:
raise ValueError("context and keyfile arguments are mutually "
"exclusive")
if context is not None and certfile is not None:
raise ValueError("context and certfile arguments are mutually "
"exclusive")
if context is None:
context = ssl._create_stdlib_context(certfile=certfile,
keyfile=keyfile)
self.sock = context.wrap_socket(self.sock,
server_hostname=self._host)
self.file = None
# RFC 3207:
# The client MUST discard any knowledge obtained from
# the server, such as the list of SMTP service extensions,
# which was not obtained from the TLS negotiation itself.
self.helo_resp = None
self.ehlo_resp = None
self.esmtp_features = {}
self.does_esmtp = 0
return (resp, reply)
con= MySMTP(server, port)
context = ssl._create_unverified_context()
con.starttls(context = context)
con.login(user, pass)
con.quit()
Late answer, but I fixed the ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852) on python 3.7 using ssl._create_unverified_context(), i.e.:
import smtplib, ssl
context = ssl._create_unverified_context()
with smtplib.SMTP_SSL("domain.tld", 465, context=context) as server:
server.login(user, password)
server.sendmail(sender_email, receiver_email, message.as_string())
It really seems that your approach with subclassing SMTP class and overriding starttls() method to accept context argument is the way to go - it's actually even introduced in Python 3.x.
However:
con = MySMTP(server, port)
context = ssl.create_default_context(cafile=PATH_TO_CERTIFICATE_AUTHORITY_ROOT_CRT_FILE)
con.starttls(context=context)
con.login(user, pass)
con.quit()
should work instead of unverified context.
When you self signed your certificate you used your own certificate authority, so probably you created root CA certificate before.
This root certificate must be known to your SMTP client in order to verify your SMTP server certificate.
So find that file, put it in location accessible to your SMTP client and set this path as a value of PATH_TO_CERTIFICATE_AUTHORITY_ROOT_CRT_FILE.
What you did instead was:
ssl._create_stdlib_context(certfile=certfile, keyfile=keyfile)
but certfile and keyfile are client certificate and key - some servers won't accept connections from unverified client.
Useful links:
ssl.create_default_context() documentation
In case you can't find your root certificate you can start from scratch (just ignore all MacOS windows on pictures - your SMTP client wants access to root certificate file - the one they add to the browser in this guide)
This is not a duplicate for this post. I tried the solutions there and nothing works in my case.
I am using Windows and Python 3.6.5. I have a python script for a TLS client. The server I need to connect to uses a self-signed certificate. When I try to connect to it using my script, I get this error:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)
I need to parse the certificate. I tried to add my server's certificate .pem content to file named: cacert.pem which is in: C:\Python36\Lib\site-packages\certifi and run the program again. Nothing change. Here is my scripot. Please help me to make the client make exception for this server as I trust its certificate.
import socket, ssl
import itertools
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_OPTIONAL
context.check_hostname = False
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
domain="192.168.56.3" # my server
ssl_sock = context.wrap_socket(s, server_hostname=domain)
ssl_sock.connect((domain, 443))
print("====== peer's certificate ======")
try:
cert = ssl_sock.getpeercert()
print(cert)
except SSLError as e:
print("Error: ",e)
ssl_sock.close()
This question has been idle for a while, but in case somebody is still struggling with connecting to a server with a self-signed certificate via Python ssl library:
You can use the load_verify_locations method of SSLContext to specify custom self-signed root certs (see Python Docs for load_verify_locations).
The forementioned code could be extended as follows:
...
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_OPTIONAL
context.check_hostname = False
context.load_verify_locations(cafile='/path/to/your/cacert.pem')
...
Be aware that you also need to include public root certs in case you want to connect to other servers with public certificates with the same client/context. (you could for example append your cacert.pem content to the certifi root certificates and reference that folder / path instead).
See also this Python docs paragraph for more information: client-side operations.