I’m trying to establish TLS channel between a client and a web server that are under my control. Both the client and server authenticates themselves using certificates that I’ve created under private PKI scheme. Client key and certificate are stored on usb dongle type HSM. Python is the main application language.
I’m able to do all required crypto operations for my project using python-pkcs11 package such as AES encryption, HMAC signing, RSA signing, and etc. However, I couldn’t find a way to “bind” pkcs11 to any TLS library. What I mean is a “Pythonic” way of calling a function that handles pkcs11 layer and establishes a TLS channel. Requests does not support pkcs11. libcurl has support for pkcs11 but it’s not implemented in pycurl, neither pyopenssl.
I’m able to do it openssl’s s_client CLI tool using engine api:
openssl s_client -engine pkcs11 -verify 2 -CAfile path/to/CA.pem -keyform engine -key "pkcs11:...;object=rsa;type=private" -cert path/to/client-cert.pem -connect localhost:8443
An example of what I’m looking for:
do_tls_with_pkcs(key=’pkcs11:URL’, cert=’cert.pem’, verify=’CA-cert.pem’)
As far as I could search around, no such library exists yet. Now I’m looking for a workaround.
I have read that if openssl, libp11, and python are compiled in such a way it is possible to abstract all of this, hence simple requests calls would go through HSM, transparent to application code. Although, I couldn’t find any material on how to do it.
I faced a similar problem as I wanted to use a PKCS#11 token (YubiKey, PIV applet) along Python requests.
I came up with https://github.com/cedric-dufour/scriptisms/blob/master/misc/m2requests.py
It's imperfect, in the sense that it does not use connections pools and does not support HTTP streams or proxying - like requests's stock HTTPS adapter does - but it does the job for simple connections to backends that require mTLS.
Related
I'm writting a Flask app that connects to external soap service that uses TLS v1.2.
I'm using Python 2.7 and requests library in version 2.18.1.
I've contacted server owner and he told me that I need to include multiple client certificates in TLS connection. It's a chain of 3 certificates which I have in separate .pem files. (root + indermediate + my client certificate).
Server won't let me in if I would have just the last one.
I've tested this with SoapUI and Wireshark and it's true. I receive a response only when I provide the whole chain of 3 certificates.
I get an error from the server when passing just my client certificate.
From requests documentation you can read that as client certificate you can pass just one cert using:
session = requests.session()
session.cert = ('/path/client_cert.pem', '/path/private_key.pem')
response = session.post(SERVICE_URL, data=XML_CONTENT, headers=HEADERS)
I get an error even if my "client_cert.pem" file is a bundle of 3 certificates (just like you do it in session.verify with CA certs). I can see on Wireshark that only the first one is used in TLS connection.
Is there any way to include multiple certificates TLS connection in Python's requests library?
Maybe I should use different library or override some of it's code?
I've got it!
I had some legacy library versions installed.
It seems that this issue was fixed by requests library developers in version 1.23. I also had to update urllib3.
My current requirements.txt is:
requests==2.22.0
urllib3==1.25.2 # compatible with requests 2.22
For following spec everything works perfecly. I've checked TLS connection on Wireshark. All certificates from "client_cert.pem" chain are passed.
If you'll have problems like this in the future remember to check if your requests and urllib3 library versions are compatible.
Thank you guys!
I’m extracting the SSL certificate from a website using the socket + ssl library in python. My understanding that it connects using the preferred method used by the server.
Using this method I’m able to identify what version of SSL is used to connect, but I also need to identify whether the website supports SSL v3, in the case when the default connection is TLS.
Is there a way to identify this information without manually testing multiple SSL connections?
I don't think sites advertise what they support. Rather, it's negotiated between client and server.
You could use the excellent server tester at www.ssllabs.com. It will try lots of configurations and report what the server in question supports. (Hopefully the site doesn't support SSL v3!)
i've written a small OPC-UA-Client in Python which acts as datalogger for PLC's with integrated OPC-UA Server.
The Connection with no security works fine but i want to secure it with a certificate.
I can import trusted certificates to the server and export the server certificate but how can i generate my own certificate ?
Thanks
You can use openssl to generate your own self-Signed certificate.
https://www.openssl.org/source/
Be Carefull with the extensions tho sometimes they demand .der .cer or as .pem and you might get an outform .crt.
How to create them:
https://www.ibm.com/support/knowledgecenter/en/SS8JFY_9.2.0/com.ibm.lmt.doc/Inventory/security/t_ssl_creating_certs.html
https://dzone.com/articles/secure-communication-with-tls-and-the-mosquitto-broker
Types of encryption:
https://blog.storagecraft.com/5-common-encryption-algorithms/
If you show your program maybe I can help you out.
I was wondering why isn't there a simple way to find the ciphers supported by a particular ssl context? (Or did I miss it)
I know we can use socket.cipher() to get the one being used for the connection for that particular socket, but why not something to get all supported ciphers?
Another things is,
I have a server which is running openssl library with the cipher string "HIGH+TLSv1.2:!MD5:!SHA1". The client is a python file using the ssl library, with default options, and after the connection is established the socket.cipher() shows the below tuple
('DHE-RSA-AES256-GCM-SHA384', 'TLSv1/SSLv3', 256)
How is the connection established with TLSv1, when I explicitly mentioned TLSv1.2, and how is it using SHA384, when TLSv1 doesn't have support for anything higher then SHA1?
I was wondering why isn't there a simple way to find the ciphers supported by a particular ssl context? (Or did I miss it)
Only OpenSSL 1.1.0 added a function SSL_CTX_get_ciphers to access this list and this functionality is not yet available in Python.
('DHE-RSA-AES256-GCM-SHA384', 'TLSv1/SSLv3', 256)
How is the connection established with TLSv1, when I explicitly mentioned TLSv1.2, and how is it using SHA384, when TLSv1 doesn't have support for anything higher then SHA1?
According to the source code Python is using SSL_CIPHER_get_version to find out the version string. The matching OpenSSL documentation says for OpenSSL 1.0.2:
SSL_CIPHER_get_version() returns string which indicates the SSL/TLS protocol version that first defined the cipher. This is currently SSLv2 or TLSv1/SSLv3. In some cases it should possibly return "TLSv1.2" but does not; use SSL_CIPHER_description()
Thus it is a bug in OpenSSL which according to the documentation of the same function in OpenSSL 1.1.0 was fixed in the latest OpenSSL version.
I'm trying to write a python (or Java) program that makes an https connection to a website and then returns properties of the https connection. I've been using python's ssl (http://docs.python.org/2/library/ssl.html), specifically the .cipher() method. My main issue with the output is that it isn't very specific:
('RC4-SHA', 'TLSv1/SSLv3', 128)
This is the output for www.amazon.com. But when I go into my browser and manually examine the connection, I can really see that it is: RC4 with SHA1 message authentication, RSA key exchange and TLS v1.0. In fact, the .cipher() method outputs TLSv1/SSLv3 for TLS versions 1.0, 1.1 and 1.2 and outputs SHA for SHA1 and SHA256.
Is there any Python (or Java) API that will give me more information about the https connection?
The Python ssl module is just a wrapper around OpenSSL, and it can't provide any more information than the library provides. But really, I don't think you're missing any information. It is being specific. You're just always getting TLSv1.0 and SHA1.
First, TLSv1/SSLv3 in OpenSSL 0.9 means TLSv1.0. It cannot mean 1.1 or 1.2, because OpenSSL 0.9 does not support those protocols. (And you are probably using OpenSSL 0.9. For example, the 3.3.0 64-bit Mac binary installer I just got off Python.org uses 0.9.8r.) You can check this from Python with ssl.OPENSSL_VERSION.
Second, RC4-SHA means SHA1, not SHA256. RC4-SHA is just the OpenSSL name for the TLSv1.0 cipher suite TLS_RSA_WITH_RC4_128_SHA. That's a complete specification of the cipher; there are different ciphers with SHA256 on the end of their names. You can see the list of cipher suites specified in RFC 2246 and its addenda for TLS 1.0 (it's RFC4346 for 1.1 and RFC 5246 for 1.2). I don't think the mapping from OpenSSL names to RFC names is specified anywhere except inside the code, but if you have the command-line OpenSSL tools you can type openssl ciphers to dump out the list of OpenSSL names for all suggested cipher suites it will send, and then you can match them up to the values sent in the handshake. (To see the handshake, openssl s_client -connect www.amazon.com:443 -msg, or try -debug or other flags instead of/in addition to -msg.)
So, why does your browser show TLSv1.2 or SHA256 for some of those same sites? Because your browser has a completely different SSL library (or a newer OpenSSL), and therefore does a completely different handshake with the server, and ends up agreeing on different cipher suites, and therefore it reports different information.
So, it's not that Python is negotiating TLSv1.2 or SHA256 and just hiding that from you; it's negotiating TLSv1.0 and SHA1 with the same server, and telling you what it's done.
If you want to use a different library that can handle things OpenSSL 0.9 can't, there are lots of choices. If you install OpenSSL 1.0.1 or later and build PyOpenSSL against that, I believe (I haven't tested) that you should be able to negotiate newer protocols and ciphers and find out that you've done so. It might even rework if you rebuild Python, or the standalone ssl module, against it. (If neither of those works, there are a zillion more OpenSSL wrappers and other SSL or TLS modules at PyPI, or you can just ctypes your favorite yourself.)