I'm writting a Flask app that connects to external soap service that uses TLS v1.2.
I'm using Python 2.7 and requests library in version 2.18.1.
I've contacted server owner and he told me that I need to include multiple client certificates in TLS connection. It's a chain of 3 certificates which I have in separate .pem files. (root + indermediate + my client certificate).
Server won't let me in if I would have just the last one.
I've tested this with SoapUI and Wireshark and it's true. I receive a response only when I provide the whole chain of 3 certificates.
I get an error from the server when passing just my client certificate.
From requests documentation you can read that as client certificate you can pass just one cert using:
session = requests.session()
session.cert = ('/path/client_cert.pem', '/path/private_key.pem')
response = session.post(SERVICE_URL, data=XML_CONTENT, headers=HEADERS)
I get an error even if my "client_cert.pem" file is a bundle of 3 certificates (just like you do it in session.verify with CA certs). I can see on Wireshark that only the first one is used in TLS connection.
Is there any way to include multiple certificates TLS connection in Python's requests library?
Maybe I should use different library or override some of it's code?
I've got it!
I had some legacy library versions installed.
It seems that this issue was fixed by requests library developers in version 1.23. I also had to update urllib3.
My current requirements.txt is:
requests==2.22.0
urllib3==1.25.2 # compatible with requests 2.22
For following spec everything works perfecly. I've checked TLS connection on Wireshark. All certificates from "client_cert.pem" chain are passed.
If you'll have problems like this in the future remember to check if your requests and urllib3 library versions are compatible.
Thank you guys!
Related
I’m trying to establish TLS channel between a client and a web server that are under my control. Both the client and server authenticates themselves using certificates that I’ve created under private PKI scheme. Client key and certificate are stored on usb dongle type HSM. Python is the main application language.
I’m able to do all required crypto operations for my project using python-pkcs11 package such as AES encryption, HMAC signing, RSA signing, and etc. However, I couldn’t find a way to “bind” pkcs11 to any TLS library. What I mean is a “Pythonic” way of calling a function that handles pkcs11 layer and establishes a TLS channel. Requests does not support pkcs11. libcurl has support for pkcs11 but it’s not implemented in pycurl, neither pyopenssl.
I’m able to do it openssl’s s_client CLI tool using engine api:
openssl s_client -engine pkcs11 -verify 2 -CAfile path/to/CA.pem -keyform engine -key "pkcs11:...;object=rsa;type=private" -cert path/to/client-cert.pem -connect localhost:8443
An example of what I’m looking for:
do_tls_with_pkcs(key=’pkcs11:URL’, cert=’cert.pem’, verify=’CA-cert.pem’)
As far as I could search around, no such library exists yet. Now I’m looking for a workaround.
I have read that if openssl, libp11, and python are compiled in such a way it is possible to abstract all of this, hence simple requests calls would go through HSM, transparent to application code. Although, I couldn’t find any material on how to do it.
I faced a similar problem as I wanted to use a PKCS#11 token (YubiKey, PIV applet) along Python requests.
I came up with https://github.com/cedric-dufour/scriptisms/blob/master/misc/m2requests.py
It's imperfect, in the sense that it does not use connections pools and does not support HTTP streams or proxying - like requests's stock HTTPS adapter does - but it does the job for simple connections to backends that require mTLS.
I’m extracting the SSL certificate from a website using the socket + ssl library in python. My understanding that it connects using the preferred method used by the server.
Using this method I’m able to identify what version of SSL is used to connect, but I also need to identify whether the website supports SSL v3, in the case when the default connection is TLS.
Is there a way to identify this information without manually testing multiple SSL connections?
I don't think sites advertise what they support. Rather, it's negotiated between client and server.
You could use the excellent server tester at www.ssllabs.com. It will try lots of configurations and report what the server in question supports. (Hopefully the site doesn't support SSL v3!)
I'm trying to parse the data from this url:
https://www.chemeo.com/search?q=show%3Ahfus+tf%3A275%3B283
But I think this is failing because the website uses SSL TLS 1.3. How can I enable my Python script, below, to connect using SSL in urllib.request?
I've tried using an SSL context but this doesn't seem to work.
This is the Python 3.6 code I have:
import urllib.request
import ssl
from bs4 import BeautifulSoup
scontext = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
chemeo_search_url = "https://www.chemeo.com/search?q=show%3Ahfus+tf%3A275%3B283"
print(chemeo_search_url)
with urllib.request.urlopen(chemeo_search_url, context=scontext) as f:
print(f.read(200))
Try:
ssl.PROTOCOL_TLS
From the docs on "PROTOCOL_SSLv23":
Deprecated since version 2.7.13: Use PROTOCOL_TLS instead.
note:
Be sure to have the CA certificate bundles installed, like on a minimal build of alpine linux, busybox, the certs have to be installed. Also sometimes if python wasn't compiled with SSL support, it might be necessary to to do so. Also depending on which version of OpenSSL has been compiled will determine which features for SSL will be usable.
Also note chemeo site doesn't use TLSv1.3 ... it is still experimental and not all that secure at the time of this writing, they currently support tls 1.0, 1.1, 1.2 using "letsencrypt" as their cert provider.
I'm using Python's requests library to perform client side authentication with certificates. The scenario is the following: CA1 has issued a certificate for an intermediate CA (CA2) and CA2 has issued my client's certificate CLIENT. The server I'm connecting to trusts CA1's cert (but does not have CA2's cert). When I use:
requests.get('https:..', cert=('/path/CLIENT.cert', '/path/CLIENT.key'))
I get an error "certificate verify failed". I assumed that's because the server can not retrieve CA2's cert.
However, I'm unable to find a way to send CA2's cert to the server. If I include it in CLIENT, I get an error about private key and cert mismatch. I have also tried to include the chain of certificates in the verify parameter but there does not seem to be any difference on the result (as far as I understand, certs in the verify parameter are used for server side authentication).
Although I think this must be a quite common scenario, I'm unable to find a solution...
PD: If I verify CLIENT's cert with openssl and the full chain of certificates the validation is successful (so there is no problem with the certificates themselves).
Requests recommends using certifi as a CA bundle. Have you tried installing certifi, adding CA1, and passing the certifi bundle path to requests?
I am writing a Flask Web-Application and use eventlet as the networking library for that application (eventlet is wrapped by Flask-SocketIO to allow asynchronous operation)
Following this guide I have been successfully creating a SSL key- and cert-file which I pass to the WSGI Server
socket_io.run(app,
host=APP_HOST,
port=APP_PORT,
keyfile='ia.key',
certfile='ia.crt')
This works fine but unfortunately Safari / Chrome says that my SSL-Certificate is not trustworthy when I access the page for the first time.
The Chrome-Failure is the following:
NET::ERR_CERT_COMMON_NAME_INVALID
How to I generate a valid SSL Certificate so that the browsers don't show that error when a user connects to the web application the first time!?
That is because it is something called a "Self Signed Certificate", which is not from any trusted company, so any modern browser auto-detects this as an untrusted site. If you are using a UNIX-based operating system, (Linux, or macOS, Fedora, and more), you can use what I am using. You have to generate new certification from a trusted site.
This is what I use to get a TRUSTED certificate that most browsers can use: https://certbot.eff.org/instructions.