We Keep Coding

sslv3 alert bad certificate with mosquitto broker and using tls certificates

I have in a Rpi a mosquitto broker with a server TLS certificate signed by a self-signed CA located in the Rpi. I am trying to connect to this broker from a Parrot virtualbox machine using a python script with the following commands:
TLS_CERT_PATH = "/etc/mosquitto/ca.pem"
client_crt = "/etc/mosquitto/VM.pem"
client_key = "/etc/mosquitto/parrot.key"
client.tls_set(ca_certs=TLS_CERT_PATH, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
client.tls_insecure_set(False)
And the following error appears in the broker:
sslv3 alert bad certificate
And in the virtual machine:
certificate verify failed: IP address mismatch
I don't understand the error because if I run in the Virtual machine the following, where 192.168.1.254 is the IP of the Rpi:
mosquitto_pub -h 192.168.1.254 -p 2259 --tls-version tlsv1.2 --cafile /etc/mosquitto/ca.crt --cert /etc/mosquitto/VM.crt --key /etc/mosquitto/parrot.key -t Injecction_moulding/pressure -q 0 -m trying
It doesn't give me any error, even though I am using the same certificate files.
I though that maybe it was something related to an intermediate certificate signing my Virtual machine's client certificate, but it is issued by the same CA that the broker uses. Moreover, I have also added in /etc/ssl/certs, the certificates that I am using just in case the CA was not recognising them as valid certificates.
Any idea of the error? I would appreciate your help because I have spent two days with this and I'm kind of stuck.
What I have in the broker's certificate is:
Certificate:
Data:
Version: 1 (0x0)
Serial Number:
37:4c:1f:f9:cd:80:c7:f4:82:82:04:69:15:5f:25:de:09:60:ae:b9
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = ES, ST = Galicia, L = Vigo, O = TFG, OU = TFG, CN = tfg, emailAddress = myemail
Validity
Not Before: May 25 17:26:29 2022 GMT
Not After : Jul 9 17:26:29 2023 GMT
Subject: C = UK, ST = Galicia, L = Pontevedra, O = Universidad de Vigo, OU = test, CN = 192.168.1.254, emailAddress = myemail
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:b4:85:b7:9b:1e:70:51:9a:0e:af:99:08:26:6f:
55:8e:3a:99:ce:32:c6:8e:23:1b:d4:39:58:75:cc:
c7:9c:5f:3d:a9:f1:f8:85:db:91:9f:25:24:f1:e8:
89:4f:ea:7e:71:23:21:f5:c5:59:77:a6:07:77:53:
e0:9d:12:39:4b:67:b0:10:e0:61:07:d9:fc:fc:f6:
3c:c5:98:f0:70:dd:d1:93:a0:10:64:73:09:8c:63:
de:07:d4:8f:81:18:23:cf:67:60:d6:dd:97:6f:62:
14:ec:0b:20:63:0b:cf:54:ad:78:2d:4d:de:be:29:
01:c1:c1:0f:cb:86:40:e5:2f:0b:29:a3:89:62:03:
6f:f0:ce:b2:43:f6:60:7c:10:46:ea:fd:0e:b4:4b:
ba:64:29:dc:39:50:3d:96:cf:04:c0:43:fa:a6:f3:
bd:e1:da:37:df:19:70:a2:dc:d7:04:2e:2d:b8:27:
97:9c:a3:08:84:be:c6:cf:d6:b6:82:d3:eb:8f:98:
13:eb:39:8b:33:1e:98:72:24:c2:3d:cb:0c:06:3d:
6e:99:fd:a9:f9:02:25:c5:3d:30:76:58:da:33:fa:
6a:33:d1:31:79:15:09:d4:a0:15:df:f2:9a:b9:7e:
68:08:8f:01:73:b7:b5:ac:24:da:bc:0e:d3:0f:b8:
d4:bf
Exponent: 65537 (0x10001)
Signature Algorithm: sha256WithRSAEncryption
25:87:fd:af:41:5a:b2:65:fa:6f:95:12:7f:72:92:ed:ef:8d:
1e:4d:e9:81:ca:75:47:17:a0:b0:0e:20:9e:3e:1a:bf:27:a1:
2f:b4:1f:2b:03:fa:21:ad:2c:da:87:d1:c0:01:12:b4:a0:ce:
29:2e:15:35:50:79:e1:84:3f:05:14:41:f7:e2:17:ee:3c:f7:
6d:32:b7:89:b4:1f:86:e8:89:69:97:a5:33:f3:cd:b5:51:88:
88:41:a5:95:e6:39:44:d4:16:c7:73:19:e1:c5:f1:5e:a7:b3:
a6:d9:9b:16:25:60:6f:ca:3c:0f:1f:85:47:eb:34:24:ad:0b:
74:bb:6b:6f:7a:f1:be:28:4a:fe:6b:55:4b:bd:40:d0:e3:fc:
f4:b7:02:75:86:87:6e:65:f4:91:a4:07:0b:34:a2:f7:88:6a:
3c:d2:67:85:e4:a2:29:5d:02:a3:72:9f:d7:7b:57:8b:c9:ed:
f4:c9:17:00:5b:3d:bc:d4:65:b4:5f:30:12:e1:cb:e2:44:23:
12:05:6f:d4:dd:15:be:56:55:99:d1:7c:f8:8f:34:4e:be:ca:
35:c4:60:03:51:41:ce:98:9a:f4:52:76:b2:69:31:db:3c:1b:
85:b2:04:0b:d8:3d

How to get SHA1 fingerprint of a certificate

My aim is to achieve SHA1 fingerprint of a third party website's certificate. I am able to get it successfully using openssl command line however, it's not getting same when I tried to achieve it using python code. The SHA1 fingerprint obtained using python code is totally different than the one obtained via openssl.
openssl steps -->
openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443
The above command output contains chain and root certificate;
Certificate chain
0 s:/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=*.actions.githubusercontent.com
i:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
-----BEGIN CERTIFICATE-----
MIIG9jCCBd6gAwIBAgIQCFCR4fqbkQJJbzQZsc87qzANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE
aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMjAxMTEwMDAwMDBa
Save the chain certificate with .crt extension as MaingithubOIDC.crt and running below command gives SHA1 fingerprint;
❯ openssl x509 -in MaingithubOIDC.crt -fingerprint -noout
SHA1 Fingerprint=15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31
Reference link - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
Python code (version 3.8/3.9) -->
import ssl
import socket
import hashlib
addr = 'token.actions.githubusercontent.com'
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
print (wrappedSocket)
except:
response = False
else:
der_cert = wrappedSocket.getpeercert(True)
pem_cert = ssl.DER_cert_to_PEM_cert(wrappedSocket.getpeercert(True))
print(pem_cert)
#Print SHA1 Thumbprint
thumb_sha1 = hashlib.sha1(der_cert).hexdigest()
print("SHA1: " + thumb_sha1)
Python code output;
SHA1: 55a7ef500a3a99f64c99c665daaf3f07403cff3d
So, the SHA1 fingerprint doesn't match with the one obtained using openssl. Am I missing something in python code?

The problem is not the wrong fingerprint calculation from the certificate but that you get the wrong certificate. The server in question is a multi-domain setup which will return different certificates based on the server_name given in the TLS handshake - see Server Name Indication.
The following code will not provide a server_name, which results in a certificate returned for *.azureedge.net, not *.actions.githubusercontent.com as the openssl s_client code gets:
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
To fix this the server_name need to be given:
ctx = ssl.create_default_context()
wrappedSocket = ctx.wrap_socket(sock,
server_hostname='token.actions.githubusercontent.com')
try:
wrappedSocket.connect((addr, 443))
With this change the expected certificate is send by the server and the fingerprint is properly calculated on it.

Python `urllib3`: sudden "certificate verify failed: certificate has expired" error

I have an old tool that can no longer successfully communicate with the outside world because it doesn't understand any of the modern TLS protocols or ciphers. I decided to put it behind a forwarding proxy that would bear the responsibility of dealing with HTTPS instead of it. I quickly wrote a working Python proxy based on code someone else had already had going to solve the problem but never finished it, however, in testing I quickly found the Stack Exchange network sites among some others do not open through it because of this error:
"GET https://stackoverflow.com/" HTTPSConnectionPool(host='stackoverflow.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)')))
I did follow the advice on similar questions on manually downloading and adding the missing/expired CA certificates (seems like I lacked the Let's Encrypt Authority X3 (Let's Encrypt R3) in my case) to the CA bundle, but it didn't change a thing. What bothers me the most that I have a non-system up-to-date OpenSSL installation and IT IS ALSO AFFECTED out of the blue.
> openssl s_client -connect stackoverflow.com:443 -servername stackoverflow.com -quiet -CAfile .\cacert.pem
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.stackexchange.com
verify error:num=10:certificate has expired
notAfter=Jan 3 13:02:44 2021 GMT
verify return:1
depth=0 CN = *.stackexchange.com
notAfter=Jan 3 13:02:44 2021 GMT
verify return:1
But at the same time an old version of the proxy that I kept in the executable form turns out to be able to connect and serve content from any of the "expired" domains still just fine, but... WHAT? I'm now completely puzzled at all of this; it also doesn't have an up-to-date CA bundle unlike the new version.
The relevant parts of the code sending the request are here:
self.timeout = urllib3.util.timeout.Timeout(connect=90.0, read=300.0)
self.params = dict(maxsize = 10, block = True, timeout = timeout)
self.sslparams = dict(cert_reqs='CERT_REQUIRED', ca_certs=CA_CERTS,
ssl_version=ssl.PROTOCOL_TLS_CLIENT)
#...
sslparams = {} if not isSSL else {**self.sslparams,
'server_hostname': host,
'assert_hostname': host,
}
params = {**sslparams, **self.params}
pool = urllib3.PoolManager(**params)
r = pool.urlopen(self.command, url, body=body, headers=headers,
enforce_content_length=False, retries=1, redirect=False,
preload_content=False, decode_content=False)
Additional info:
> python -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.1.1g 21 Apr 2020
> python --version
Python 3.9.1
I have pyOpenSSL v20.0.1 and urllib3 v1.26.2. I'm on Windows 10 x64 with the latest updates installed.

Certificate Generation for Python Twisted SSL

I am trying to figure out how to setup a SSL link using the Python library Twisted. I have managed to create a certificate that works on the server side, but I am totally stuck when it comes to the client side.
The example from the twisted website states:
The following examples rely on the files server.pem (private key and
self-signed certificate together) and public.pem (the server’s public
certificate by itself).
I have generated myself a certificate and key using OpenSSL:
# Generate Private Key:
openssl genrsa -des3 -out certs/server.key 2048
# Generate Certificate Signing Request:
openssl req -new -key certs/server.key -sha256 -out certs/server.csr
# Generate a Self-Signed Certificate:
openssl x509 -req -days 365 -in certs/server.csr -signkey certs/server.key -sha256 -out certs/server.crt
# Convert the CRT to PEM format:
openssl x509 -in certs/server.crt -out certs/server.pem -outform PEM
For the server-side I am combining certs/server.crt and certs/server.key to create server.pem and trying to use server.crt for public.
When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData)
I get an error about not starting line. Which certificate should I be using for the client if it's not server.crt please?

In case you want to have certificate based authentication for the clients as well:
I had that issue some time ago and wrote a blog post about my solution.
It also contains a guide to create certificates and sign them with an own certificate authority. You can find the python example code at GitHub.
It uses Twisted for a simple JSONRPCServer
with certificate based authentication for both, server as well as for the clients.
The main thing is to define an own AltCtxFactory for the clients:
# Use our own context factory to use our certificate to authenticate
# against the server and ensure that we are using a strong SSL/TLS
# encryption method
class AltCtxFactory(ssl.ClientContextFactory):
def getContext(self):
# Used TLS/SSL encryption method
sslMethod = SSL.TLSv1_2_METHOD
# Clients private Key, used for authentication
privKey = "<PATH TO YOUR PRIVATE KEY>"
# Clients certificate, used for authentication
certificate = "<PATH TO YOUR CERTIFICATE>"
# Our trusted Certificate Authority for server connections
accepted_ca = "<PATH TO YOUR ACCEPTED CERTIFICATE AUTHORITY>"
self.method = sslMethod
ctx = ssl.ClientContextFactory.getContext(self)
# Ensure that we verify server's certificate and use our own
# verifyCallback method to get further details of invalid certificates
ctx.set_verify(SSL.VERIFY_PEER | SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
verifyCallback)
# Ensure that we only trust our CA
ctx.load_verify_locations(accepted_ca)
# Use our own Callback mehtod if a password is needed to decrypt our
# private key
ctx.set_passwd_cb(password_cb)
# Use our certificate for authentication against server
ctx.use_certificate_file(certificate)
# Use our private key for authentication against server
ctx.use_privatekey_file(privKey)
return ctx
Feel free to use the code in your projects.

When I try and run my test program using:
certificate = ssl.PrivateCertificate.loadPEM(certData) I get an error
about not starting line. Which certificate should I be using for the
client if it's not server.crt please?
This should be ssl.Certificate.LoadPEM(certData) if you look at the example on the Twisted howto page.

SSL client certificate is not sent by python 'requests' library or s_client, but works fine in web browsers

I am trying to write a client script in Python that accesses a web application and uses SSL client certificates for authentication. Though I am able to access the application from both Firefox and Chrome as long as I have the client certificate loaded, I get the following response whenever I send the request via Python 'requests:
400 No required SSL certificate was sent
nginx/1.4.6 (Ubuntu)
I have also tried Python httplib, s_client, and curl, and get the same error message. I am using the same client certificate for all the testing, in pkcs12 format for the web browsers and pulled out into certificate and key PEM files for the command line tools. My python code looks like:
import requests
CERT = r'/path/to/cert.crt' #Client certificate
KEY = r'/path/to/key.key' #Client private key
CACERT = r'/path/to/ca.crt' #Server certificate chain
session = requests.Session()
session.cert = (CERT, KEY)
resp = session.get('https://my.webapp.com/',
verify=CACERT)
print resp.content
session.close()
s_client gives more information about what is happening. Here's an abbreviated version of the output:
$ openssl s_client -cert cert.crt -key key.key -CAfile ca.crt -connect <host>:<port>
CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify return:1
depth=1 /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
verify return:1
depth=0 /serialNumber=tgBIwyM-p18O/aDyvyWNKHDnOezzDJag/OU=GT89519184/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.rexdb.us
verify return:1
---
Certificate chain
...
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=...
---
No client certificate CA names sent
---
SSL handshake has read 3519 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES128-SHA
Session-ID: ...
Session-ID-ctx:
Master-Key: ...
Key-Arg : None
Start Time: 1409269239
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
GET / HTTP/1.1
Host: my.webapp.com
HTTP/1.1 400 Bad Request
Server: nginx/1.4.6 (Ubuntu)
Date: Thu, 28 Aug 2014 23:41:04 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: close
...
<head><title>400 No required SSL certificate was sent</title></head>
...
closed
I'm reasonably sure this is not an issue with the server because authentication works in a browser. However, the s_client output says 'No client certificate CA names sent', which sounds like a server problem (e.g. the client certificate is not being sent on to the server because the server isn't asking for it). Here is the relevant part of the nginx configuration:
ssl on;
ssl_certificate_key /path/to/server/key.pem;
ssl_certificate /path/to/server/certificate.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:AES128:AES256:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK;
ssl_prefer_server_ciphers on;
ssl_session_timeout 5m;
ssl_client_certificate /path/to/client/ca/certificate.crt;
ssl_verify_depth 10;
ssl_verify_client on;
The application uses uwsgi through nginx. There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
The only other potentially relevant difference I can find is that in Firefox the connection is keep-alive and in s_client it is close. I've tried setting Connection: keep-alive in the header, with the same result.

There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
I assume that means you have multiple certificates behind the same IP address and that the client has to use SNI (Server Name Indication) to send the server the expected hostname inside the SSL handshake. openssl s_client does not use SNI by default and I don't know if python does - it might depend on the version of python you use.
Because the client only sends a certificate if the server tells it do to it might be, that because of missing SNI you run into the wrong configuration part, that is the default part with another certificate and without requirement for client certificates.
I would recommend to try with openssl s_client again, but use the command line option -servername (not documented in man page but shown if called with -h) to explicitly set the expected server name. If this works you need to find a way to use SNI in python too. If this does not work please make a packet dump and make sure with wireshark, that the server really requires the client to send a certificate and the client really does not send a certificate.