We Keep Coding

How to detect if a server is using SNI for HTTPS with python

I'm trying to detect if a server is using the Server Name Indication SSL extension for its HTTPS certificate on a website.
Reproducing this command line with python
openssl s_client -servername www.SERVERNAME.com -tlsextdebug -connect www.YOURSERVER.com:443
Python :
context = ssl.create_default_context()
with socket.create_connection((hostname, port)) as sock:
with context.wrap_socket(sock, server_hostname=sni) as sslsock:
Code below return error
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sni'. (_ssl.c:992)
But openssl return the certificate correct 💯
Unfortunately im not that experienced in python
Thanks!

with context.wrap_socket(sock, server_hostname=sni) as sslsock:
Check with server_hostname set to the real server name (to use SNI) and to None (to not use SNI). If the results differ (i.e. connection fails, different certificates) then the server requires SNI.To check the certificate use getpeercert.
Depending on the kind of server you might need to disable certificate validation, i.e. set CERT_NONE for the context. While disabling certificate is a bad idea in general, it is acceptable in this case since all you want is to check the servers capabilities and not actually transfer any application data:
context.check_hostname = False
context.verify_mode = ssl.CERT_NONE
In this case you need to call socket.getpeercert(binary_form=True) since it otherwise will not return information. For comparison this binary form should be sufficient though.

How to allow python to trust my server's TLS self-signed certificate: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

This is not a duplicate for this post. I tried the solutions there and nothing works in my case.
I am using Windows and Python 3.6.5. I have a python script for a TLS client. The server I need to connect to uses a self-signed certificate. When I try to connect to it using my script, I get this error:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)
I need to parse the certificate. I tried to add my server's certificate .pem content to file named: cacert.pem which is in: C:\Python36\Lib\site-packages\certifi and run the program again. Nothing change. Here is my scripot. Please help me to make the client make exception for this server as I trust its certificate.
import socket, ssl
import itertools
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_OPTIONAL
context.check_hostname = False
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
domain="192.168.56.3" # my server
ssl_sock = context.wrap_socket(s, server_hostname=domain)
ssl_sock.connect((domain, 443))
print("====== peer's certificate ======")
try:
cert = ssl_sock.getpeercert()
print(cert)
except SSLError as e:
print("Error: ",e)
ssl_sock.close()

This question has been idle for a while, but in case somebody is still struggling with connecting to a server with a self-signed certificate via Python ssl library:
You can use the load_verify_locations method of SSLContext to specify custom self-signed root certs (see Python Docs for load_verify_locations).
The forementioned code could be extended as follows:
...
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_OPTIONAL
context.check_hostname = False
context.load_verify_locations(cafile='/path/to/your/cacert.pem')
...
Be aware that you also need to include public root certs in case you want to connect to other servers with public certificates with the same client/context. (you could for example append your cacert.pem content to the certifi root certificates and reference that folder / path instead).
See also this Python docs paragraph for more information: client-side operations.

User can't communicate with proxy

I'm implementing an IMAP proxy which securely communicates with a client.However, I have a problem when handshaking.
The code of my proxy is:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.bind((host, port))
ssock, addr = self.sock.accept()
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
self.conn_client = context.wrap_socket(ssock)
And I receive the error:
ssl.SSLError: [SSL: UNEXPECTED_MESSAGE] unexpected message (_ssl.c:833)
The code of my tests is:
M = imaplib.IMAP4_SSL(IP_PROXY)
And I receive the error:
ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:777)
However, when the code of the proxy is:
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.bind((host, port))
ssock, addr = self.sock.accept()
self.conn_client = ssl.wrap_socket(ssock, certfile=CERT, server_side= True)
It correctly works but I don't want to use certificate.
Thank you

It correctly works but I don't want to use certificate.
SSL/TLS is almost everywhere used with a certificate to make sure that the client is talking to the expected server and not to some man in the middle. If you don't want to use a certificate you need to either use a different kind of authentication (like PSK) or use no authentication at all ("anonymous authentication" - very bad idea).
In any way you would need to set the relevant ciphers to enable this alternative authentication on both client and server. This can be done with the ciphers attribute to wrap_socket on the server side and in your client code it could probably be done by constructed a SSLContext with the necessary ciphers and using the ssl_context argument to specific the context to be used in IMAP4_SSL.
But this is only for your specific Python based IMAP client. Don't expect that you will be able to configure commonly used IMAP clients like Thunderbird or Outlook to be usable with a server without certificates. And like I said, it is a bad idea in the first place.

Sending email without keyfile (only certfile) using Python smtplib

Trying to send email with a certificate file using the following script:
import smtplib
client = smtplib.SMTP(myhost, myport)
client.ehlo()
client.starttls(certfile=mycertfile)
client.ehlo()
client.login(myusername, mypassword)
client.sendmail(sender, receiver, Message)
client.quit()
I get the following error:
SSLError: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
I think documentations (smtplib.html and ssl.html) say I need to provide a private key. I only have the certificate file (base64 PEM format). My devops says that a private key is not required in this case because I do not need to identify the local side of the connection.
Question
Is there a way to send the email without providing the private key? If a private key is required, why?

There are two ways to use SSL/TLS: client authenticated and "basic" where the client is unauthenticated. In client authenticated connections, the both the server and the client send a certificate to the other. In "basic" only the server does.
If you pass neither a certificate nor a keyfile, smtplib will use a basic connection, where the client is authenticated.
If you use a certificate, it will be used for a client authenticated connection. In that case, the server will demand that the client shows it owns the certificate by signing a handshake message. For the client to be able to do that it also needs the private key, which can be either in the certificate file or as a separate keyfile.
Long story short, if you want to use a client certificate, you must also use a key. If not, you can just leave both empty.
OTOH, maybe you have a server certificate file or CA list you want to use with the connection?
In that case you need to pass it to ssl.wrap_socket in the ca_certs parameter. Since you use Python 2.6 there's no easy way to do that with smtplib (Python 3.3+ has a context argument to starttls).
How to solve this depends on your application. For example, if you do not need ssl for anything else, a hackish solution would be to monkey-patch ssl.wrap_socket with one that provides your ca_cert (as well as cert_reqs=CERT_REQUIRED, likely).
A more full blown solution would be to extend smtplib.SMTP with your own variant that does allow passing in those parameters.

Here's a monkey-patch taken from this page:
class SMTPExt(smtplib.SMTP):
"""
This class extends smtplib.SMTP and overrides the starttls method
allowing extra parameters and forwarding them to ssl.wrap_socket.
"""
def starttls(self, keyfile=None, certfile=None, **kwargs):
self.ehlo_or_helo_if_needed()
if not self.has_extn("starttls"):
raise SMTPException("STARTTLS extension not supported by server.")
(resp, reply) = self.docmd("STARTTLS")
if resp == 220:
self.sock = ssl.wrap_socket(self.sock, keyfile, certfile, **kwargs)
self.file = SSLFakeFile(self.sock)
# RFC 3207:
# The client MUST discard any knowledge obtained from
# the server, such as the list of SMTP service extensions,
# which was not obtained from the TLS negotiation itself.
self.helo_resp = None
self.ehlo_resp = None
self.esmtp_features = {}
self.does_esmtp = 0
return (resp, reply)
Using the root cert from requests

SSL in Python: Why it doesn't send certificate to server?

I'm writing some software that is supposed to acquire information from SSL-secured web page. Below there's piece of code I use to connect to server.
s = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket (
s,
ca_certs = '/home/stilz/Desktop/Certyfikaty/GLOWNE_CA.cer',
cert_reqs = ssl.CERT_REQUIRED,
ssl_version = ssl.PROTOCOL_SSLv3,
)
ssl_sock.connect ((HOST, PORT))
However, this doesn't work. It throws exception with message "Handshake alert failed". I captured some TCP packets that come out of my script and also corresponding packets from Internet Explorer and figured out that my script doesn't send certificate at all (server returns something like "Fatal: no certificate supplied") while IE sends it normally. As far as I know file ca.cer is x509 certificate (beginning with "-----BEGIN CERTIFICATE-----").
Please help me and tell what I'm doing wrong. If I've supplied to few information, please let me know.
Regards

First of all, you need to determine if you need to authenticate yourself on the server (certificate is sent to the server only in this case) or you need to validate server's authenticity (in which case the certificate is not sent to the server).
Now about issues in your code:
Case 1: you don't need to authenticate on the server. In this case you don't need to specify your certificate and nothing is sent to the server.
Case 2: you need to authenticate yourself on the server. In that case you need to provide your certificate and your private key file in keyfile and certfile parameters. Only then your certificate is sent to the server (and the private key is used in the handshake).
I guess that your have case 1 in fact. So first of all you need to check if the server provides a valid certificate by connecting to it with a web browser and inspecting site's certificate chain. It can happen that the site sends its certificate but omits intermediate CA certificates.
In any case I'd like to remind about the discussion about certificates in Python docs which I suggest you re-read.

If the server requests (and requires) a certificate to be sent by the client, you need to supply ssl.wrap_socket with the path to your certificate (certfile) and its matching private key (keyfile). The ca_certs parameter is only used for the client to verify the server's certificate against known CA certificates, it has nothing to do with sending a client-certificate.
You may need to export your client-certificate and private key from IE (and then convert them to PEM format).