We Keep Coding

getting error using self certificate verification in python ssl

Getting the following error :
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)
I'm using self-signed certificates between many servers, now need to integrate python in the system but unable to verify self-signed certificates.
The code I'm using
context = ssl.create_default_context()
context.load_verify_locations("/var/certs.crt")
context.load_cert_chain(certfile=cert_path, keyfile=key_path)
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_REQ
resp = urllib.request.urlopen(url_string, context=ctx)
var/certs.crt containing the certificate of the specific server I'm starting an ssl connection with.
cert_path & key_path are my own cert and private key to establish 2 way ssl.
Things I've checked :
1.I can see my certs being loaded after load_cert_chain in context.get_ca_certs()
2.I tried context.verify_flags |= 0x80000 but it didn't work.
If ctx.verify_mode = False then I'm able to connect properly but it will not be secured.
Since the best existing answer on StackOverflow is to use ctx.verify = False and it's not the way, I'm hoping this time to find someone who actually fixed it.
Thanks

After checking in wireshark I saw that python throwing the wrong error. the problem wasn't with the self certificate but was "Certificate Unknown" and the SSL handshake failed.
So it can be done with ssl.CERT_REQ

Certificate Generation for Python Twisted SSL

I am trying to figure out how to setup a SSL link using the Python library Twisted. I have managed to create a certificate that works on the server side, but I am totally stuck when it comes to the client side.
The example from the twisted website states:
The following examples rely on the files server.pem (private key and
self-signed certificate together) and public.pem (the server’s public
certificate by itself).
I have generated myself a certificate and key using OpenSSL:
# Generate Private Key:
openssl genrsa -des3 -out certs/server.key 2048
# Generate Certificate Signing Request:
openssl req -new -key certs/server.key -sha256 -out certs/server.csr
# Generate a Self-Signed Certificate:
openssl x509 -req -days 365 -in certs/server.csr -signkey certs/server.key -sha256 -out certs/server.crt
# Convert the CRT to PEM format:
openssl x509 -in certs/server.crt -out certs/server.pem -outform PEM
For the server-side I am combining certs/server.crt and certs/server.key to create server.pem and trying to use server.crt for public.
When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData)
I get an error about not starting line. Which certificate should I be using for the client if it's not server.crt please?

In case you want to have certificate based authentication for the clients as well:
I had that issue some time ago and wrote a blog post about my solution.
It also contains a guide to create certificates and sign them with an own certificate authority. You can find the python example code at GitHub.
It uses Twisted for a simple JSONRPCServer
with certificate based authentication for both, server as well as for the clients.
The main thing is to define an own AltCtxFactory for the clients:
# Use our own context factory to use our certificate to authenticate
# against the server and ensure that we are using a strong SSL/TLS
# encryption method
class AltCtxFactory(ssl.ClientContextFactory):
def getContext(self):
# Used TLS/SSL encryption method
sslMethod = SSL.TLSv1_2_METHOD
# Clients private Key, used for authentication
privKey = "<PATH TO YOUR PRIVATE KEY>"
# Clients certificate, used for authentication
certificate = "<PATH TO YOUR CERTIFICATE>"
# Our trusted Certificate Authority for server connections
accepted_ca = "<PATH TO YOUR ACCEPTED CERTIFICATE AUTHORITY>"
self.method = sslMethod
ctx = ssl.ClientContextFactory.getContext(self)
# Ensure that we verify server's certificate and use our own
# verifyCallback method to get further details of invalid certificates
ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
verifyCallback)
# Ensure that we only trust our CA
ctx.load_verify_locations(accepted_ca)
# Use our own Callback mehtod if a password is needed to decrypt our
# private key
ctx.set_passwd_cb(password_cb)
# Use our certificate for authentication against server
ctx.use_certificate_file(certificate)
# Use our private key for authentication against server
ctx.use_privatekey_file(privKey)
return ctx
Feel free to use the code in your projects.

When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData) I get an error
about not starting line. Which certificate should I be using for the
client if it's not server.crt please?
This should be ssl.Certificate.LoadPEM(certData) if you look at the example on the Twisted howto page.

How to allow python to trust my server's TLS self-signed certificate: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

This is not a duplicate for this post. I tried the solutions there and nothing works in my case.
I am using Windows and Python 3.6.5. I have a python script for a TLS client. The server I need to connect to uses a self-signed certificate. When I try to connect to it using my script, I get this error:
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)
I need to parse the certificate. I tried to add my server's certificate .pem content to file named: cacert.pem which is in: C:\Python36\Lib\site-packages\certifi and run the program again. Nothing change. Here is my scripot. Please help me to make the client make exception for this server as I trust its certificate.
import socket, ssl
import itertools
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_OPTIONAL
context.check_hostname = False
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
domain="192.168.56.3" # my server
ssl_sock = context.wrap_socket(s, server_hostname=domain)
ssl_sock.connect((domain, 443))
print("====== peer's certificate ======")
try:
cert = ssl_sock.getpeercert()
print(cert)
except SSLError as e:
print("Error: ",e)
ssl_sock.close()

This question has been idle for a while, but in case somebody is still struggling with connecting to a server with a self-signed certificate via Python ssl library:
You can use the load_verify_locations method of SSLContext to specify custom self-signed root certs (see Python Docs for load_verify_locations).
The forementioned code could be extended as follows:
...
context = ssl.SSLContext()
context.verify_mode = ssl.CERT_OPTIONAL
context.check_hostname = False
context.load_verify_locations(cafile='/path/to/your/cacert.pem')
...
Be aware that you also need to include public root certs in case you want to connect to other servers with public certificates with the same client/context. (you could for example append your cacert.pem content to the certifi root certificates and reference that folder / path instead).
See also this Python docs paragraph for more information: client-side operations.

SSL client certificate is not sent by python 'requests' library or s_client, but works fine in web browsers

I am trying to write a client script in Python that accesses a web application and uses SSL client certificates for authentication. Though I am able to access the application from both Firefox and Chrome as long as I have the client certificate loaded, I get the following response whenever I send the request via Python 'requests:
400 No required SSL certificate was sent
nginx/1.4.6 (Ubuntu)
I have also tried Python httplib, s_client, and curl, and get the same error message. I am using the same client certificate for all the testing, in pkcs12 format for the web browsers and pulled out into certificate and key PEM files for the command line tools. My python code looks like:
import requests
CERT = r'/path/to/cert.crt' #Client certificate
KEY = r'/path/to/key.key' #Client private key
CACERT = r'/path/to/ca.crt' #Server certificate chain
session = requests.Session()
session.cert = (CERT, KEY)
resp = session.get('https://my.webapp.com/',
verify=CACERT)
print resp.content
session.close()
s_client gives more information about what is happening. Here's an abbreviated version of the output:
$ openssl s_client -cert cert.crt -key key.key -CAfile ca.crt -connect <host>:<port>
CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify return:1
depth=1 /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
verify return:1
depth=0 /serialNumber=tgBIwyM-p18O/aDyvyWNKHDnOezzDJag/OU=GT89519184/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.rexdb.us
verify return:1
---
Certificate chain
...
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=...
---
No client certificate CA names sent
---
SSL handshake has read 3519 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES128-SHA
Session-ID: ...
Session-ID-ctx:
Master-Key: ...
Key-Arg : None
Start Time: 1409269239
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
GET / HTTP/1.1
Host: my.webapp.com
HTTP/1.1 400 Bad Request
Server: nginx/1.4.6 (Ubuntu)
Date: Thu, 28 Aug 2014 23:41:04 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: close
...
<head><title>400 No required SSL certificate was sent</title></head>
...
closed
I'm reasonably sure this is not an issue with the server because authentication works in a browser. However, the s_client output says 'No client certificate CA names sent', which sounds like a server problem (e.g. the client certificate is not being sent on to the server because the server isn't asking for it). Here is the relevant part of the nginx configuration:
ssl on;
ssl_certificate_key /path/to/server/key.pem;
ssl_certificate /path/to/server/certificate.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:AES128:AES256:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK;
ssl_prefer_server_ciphers on;
ssl_session_timeout 5m;
ssl_client_certificate /path/to/client/ca/certificate.crt;
ssl_verify_depth 10;
ssl_verify_client on;
The application uses uwsgi through nginx. There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
The only other potentially relevant difference I can find is that in Firefox the connection is keep-alive and in s_client it is close. I've tried setting Connection: keep-alive in the header, with the same result.

There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
I assume that means you have multiple certificates behind the same IP address and that the client has to use SNI (Server Name Indication) to send the server the expected hostname inside the SSL handshake. openssl s_client does not use SNI by default and I don't know if python does - it might depend on the version of python you use.
Because the client only sends a certificate if the server tells it do to it might be, that because of missing SNI you run into the wrong configuration part, that is the default part with another certificate and without requirement for client certificates.
I would recommend to try with openssl s_client again, but use the command line option -servername (not documented in man page but shown if called with -h) to explicitly set the expected server name. If this works you need to find a way to use SNI in python too. If this does not work please make a packet dump and make sure with wireshark, that the server really requires the client to send a certificate and the client really does not send a certificate.

SSL in Python: Why it doesn't send certificate to server?

I'm writing some software that is supposed to acquire information from SSL-secured web page. Below there's piece of code I use to connect to server.
s = socket.socket (socket.AF_INET, socket.SOCK_STREAM)
ssl_sock = ssl.wrap_socket (
s,
ca_certs = '/home/stilz/Desktop/Certyfikaty/GLOWNE_CA.cer',
cert_reqs = ssl.CERT_REQUIRED,
ssl_version = ssl.PROTOCOL_SSLv3,
)
ssl_sock.connect ((HOST, PORT))
However, this doesn't work. It throws exception with message "Handshake alert failed". I captured some TCP packets that come out of my script and also corresponding packets from Internet Explorer and figured out that my script doesn't send certificate at all (server returns something like "Fatal: no certificate supplied") while IE sends it normally. As far as I know file ca.cer is x509 certificate (beginning with "-----BEGIN CERTIFICATE-----").
Please help me and tell what I'm doing wrong. If I've supplied to few information, please let me know.
Regards

First of all, you need to determine if you need to authenticate yourself on the server (certificate is sent to the server only in this case) or you need to validate server's authenticity (in which case the certificate is not sent to the server).
Now about issues in your code:
Case 1: you don't need to authenticate on the server. In this case you don't need to specify your certificate and nothing is sent to the server.
Case 2: you need to authenticate yourself on the server. In that case you need to provide your certificate and your private key file in keyfile and certfile parameters. Only then your certificate is sent to the server (and the private key is used in the handshake).
I guess that your have case 1 in fact. So first of all you need to check if the server provides a valid certificate by connecting to it with a web browser and inspecting site's certificate chain. It can happen that the site sends its certificate but omits intermediate CA certificates.
In any case I'd like to remind about the discussion about certificates in Python docs which I suggest you re-read.

If the server requests (and requires) a certificate to be sent by the client, you need to supply ssl.wrap_socket with the path to your certificate (certfile) and its matching private key (keyfile). The ca_certs parameter is only used for the client to verify the server's certificate against known CA certificates, it has nothing to do with sending a client-certificate.
You may need to export your client-certificate and private key from IE (and then convert them to PEM format).