sslv3 alert bad certificate with mosquitto broker and using tls certificates - python

I have in a Rpi a mosquitto broker with a server TLS certificate signed by a self-signed CA located in the Rpi. I am trying to connect to this broker from a Parrot virtualbox machine using a python script with the following commands:
TLS_CERT_PATH = "/etc/mosquitto/ca.pem"
client_crt = "/etc/mosquitto/VM.pem"
client_key = "/etc/mosquitto/parrot.key"
client.tls_set(ca_certs=TLS_CERT_PATH, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLSv1_2, ciphers=None)
client.tls_insecure_set(False)
And the following error appears in the broker:
sslv3 alert bad certificate
And in the virtual machine:
certificate verify failed: IP address mismatch
I don't understand the error because if I run in the Virtual machine the following, where 192.168.1.254 is the IP of the Rpi:
mosquitto_pub -h 192.168.1.254 -p 2259 --tls-version tlsv1.2 --cafile /etc/mosquitto/ca.crt --cert /etc/mosquitto/VM.crt --key /etc/mosquitto/parrot.key -t Injecction_moulding/pressure -q 0 -m trying
It doesn't give me any error, even though I am using the same certificate files.
I though that maybe it was something related to an intermediate certificate signing my Virtual machine's client certificate, but it is issued by the same CA that the broker uses. Moreover, I have also added in /etc/ssl/certs, the certificates that I am using just in case the CA was not recognising them as valid certificates.
Any idea of the error? I would appreciate your help because I have spent two days with this and I'm kind of stuck.
What I have in the broker's certificate is:
Certificate:
Data:
Version: 1 (0x0)
Serial Number:
37:4c:1f:f9:cd:80:c7:f4:82:82:04:69:15:5f:25:de:09:60:ae:b9
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = ES, ST = Galicia, L = Vigo, O = TFG, OU = TFG, CN = tfg, emailAddress = myemail
Validity
Not Before: May 25 17:26:29 2022 GMT
Not After : Jul 9 17:26:29 2023 GMT
Subject: C = UK, ST = Galicia, L = Pontevedra, O = Universidad de Vigo, OU = test, CN = 192.168.1.254, emailAddress = myemail
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:b4:85:b7:9b:1e:70:51:9a:0e:af:99:08:26:6f:
55:8e:3a:99:ce:32:c6:8e:23:1b:d4:39:58:75:cc:
c7:9c:5f:3d:a9:f1:f8:85:db:91:9f:25:24:f1:e8:
89:4f:ea:7e:71:23:21:f5:c5:59:77:a6:07:77:53:
e0:9d:12:39:4b:67:b0:10:e0:61:07:d9:fc:fc:f6:
3c:c5:98:f0:70:dd:d1:93:a0:10:64:73:09:8c:63:
de:07:d4:8f:81:18:23:cf:67:60:d6:dd:97:6f:62:
14:ec:0b:20:63:0b:cf:54:ad:78:2d:4d:de:be:29:
01:c1:c1:0f:cb:86:40:e5:2f:0b:29:a3:89:62:03:
6f:f0:ce:b2:43:f6:60:7c:10:46:ea:fd:0e:b4:4b:
ba:64:29:dc:39:50:3d:96:cf:04:c0:43:fa:a6:f3:
bd:e1:da:37:df:19:70:a2:dc:d7:04:2e:2d:b8:27:
97:9c:a3:08:84:be:c6:cf:d6:b6:82:d3:eb:8f:98:
13:eb:39:8b:33:1e:98:72:24:c2:3d:cb:0c:06:3d:
6e:99:fd:a9:f9:02:25:c5:3d:30:76:58:da:33:fa:
6a:33:d1:31:79:15:09:d4:a0:15:df:f2:9a:b9:7e:
68:08:8f:01:73:b7:b5:ac:24:da:bc:0e:d3:0f:b8:
d4:bf
Exponent: 65537 (0x10001)
Signature Algorithm: sha256WithRSAEncryption
25:87:fd:af:41:5a:b2:65:fa:6f:95:12:7f:72:92:ed:ef:8d:
1e:4d:e9:81:ca:75:47:17:a0:b0:0e:20:9e:3e:1a:bf:27:a1:
2f:b4:1f:2b:03:fa:21:ad:2c:da:87:d1:c0:01:12:b4:a0:ce:
29:2e:15:35:50:79:e1:84:3f:05:14:41:f7:e2:17:ee:3c:f7:
6d:32:b7:89:b4:1f:86:e8:89:69:97:a5:33:f3:cd:b5:51:88:
88:41:a5:95:e6:39:44:d4:16:c7:73:19:e1:c5:f1:5e:a7:b3:
a6:d9:9b:16:25:60:6f:ca:3c:0f:1f:85:47:eb:34:24:ad:0b:
74:bb:6b:6f:7a:f1:be:28:4a:fe:6b:55:4b:bd:40:d0:e3:fc:
f4:b7:02:75:86:87:6e:65:f4:91:a4:07:0b:34:a2:f7:88:6a:
3c:d2:67:85:e4:a2:29:5d:02:a3:72:9f:d7:7b:57:8b:c9:ed:
f4:c9:17:00:5b:3d:bc:d4:65:b4:5f:30:12:e1:cb:e2:44:23:
12:05:6f:d4:dd:15:be:56:55:99:d1:7c:f8:8f:34:4e:be:ca:
35:c4:60:03:51:41:ce:98:9a:f4:52:76:b2:69:31:db:3c:1b:
85:b2:04:0b:d8:3d

Related

How to get SHA1 fingerprint of a certificate

My aim is to achieve SHA1 fingerprint of a third party website's certificate. I am able to get it successfully using openssl command line however, it's not getting same when I tried to achieve it using python code. The SHA1 fingerprint obtained using python code is totally different than the one obtained via openssl.
openssl steps -->
openssl s_client -servername token.actions.githubusercontent.com -showcerts -connect token.actions.githubusercontent.com:443
The above command output contains chain and root certificate;
Certificate chain
0 s:/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=*.actions.githubusercontent.com
i:/C=US/O=DigiCert Inc/CN=DigiCert TLS RSA SHA256 2020 CA1
-----BEGIN CERTIFICATE-----
MIIG9jCCBd6gAwIBAgIQCFCR4fqbkQJJbzQZsc87qzANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE
aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMjAxMTEwMDAwMDBa
Save the chain certificate with .crt extension as MaingithubOIDC.crt and running below command gives SHA1 fingerprint;
❯ openssl x509 -in MaingithubOIDC.crt -fingerprint -noout
SHA1 Fingerprint=15:E2:91:08:71:81:11:E5:9B:3D:AD:31:95:46:47:E3:C3:44:A2:31
Reference link - https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_oidc_verify-thumbprint.html
Python code (version 3.8/3.9) -->
import ssl
import socket
import hashlib
addr = 'token.actions.githubusercontent.com'
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.settimeout(1)
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
print (wrappedSocket)
except:
response = False
else:
der_cert = wrappedSocket.getpeercert(True)
pem_cert = ssl.DER_cert_to_PEM_cert(wrappedSocket.getpeercert(True))
print(pem_cert)
#Print SHA1 Thumbprint
thumb_sha1 = hashlib.sha1(der_cert).hexdigest()
print("SHA1: " + thumb_sha1)
Python code output;
SHA1: 55a7ef500a3a99f64c99c665daaf3f07403cff3d
So, the SHA1 fingerprint doesn't match with the one obtained using openssl. Am I missing something in python code?
The problem is not the wrong fingerprint calculation from the certificate but that you get the wrong certificate. The server in question is a multi-domain setup which will return different certificates based on the server_name given in the TLS handshake - see Server Name Indication.
The following code will not provide a server_name, which results in a certificate returned for *.azureedge.net, not *.actions.githubusercontent.com as the openssl s_client code gets:
wrappedSocket = ssl.wrap_socket(sock)
try:
wrappedSocket.connect((addr, 443))
To fix this the server_name need to be given:
ctx = ssl.create_default_context()
wrappedSocket = ctx.wrap_socket(sock,
server_hostname='token.actions.githubusercontent.com')
try:
wrappedSocket.connect((addr, 443))
With this change the expected certificate is send by the server and the fingerprint is properly calculated on it.

Python `urllib3`: sudden "certificate verify failed: certificate has expired" error

I have an old tool that can no longer successfully communicate with the outside world because it doesn't understand any of the modern TLS protocols or ciphers. I decided to put it behind a forwarding proxy that would bear the responsibility of dealing with HTTPS instead of it. I quickly wrote a working Python proxy based on code someone else had already had going to solve the problem but never finished it, however, in testing I quickly found the Stack Exchange network sites among some others do not open through it because of this error:
"GET https://stackoverflow.com/" HTTPSConnectionPool(host='stackoverflow.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1123)')))
I did follow the advice on similar questions on manually downloading and adding the missing/expired CA certificates (seems like I lacked the Let's Encrypt Authority X3 (Let's Encrypt R3) in my case) to the CA bundle, but it didn't change a thing. What bothers me the most that I have a non-system up-to-date OpenSSL installation and IT IS ALSO AFFECTED out of the blue.
> openssl s_client -connect stackoverflow.com:443 -servername stackoverflow.com -quiet -CAfile .\cacert.pem
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.stackexchange.com
verify error:num=10:certificate has expired
notAfter=Jan 3 13:02:44 2021 GMT
verify return:1
depth=0 CN = *.stackexchange.com
notAfter=Jan 3 13:02:44 2021 GMT
verify return:1
But at the same time an old version of the proxy that I kept in the executable form turns out to be able to connect and serve content from any of the "expired" domains still just fine, but... WHAT? I'm now completely puzzled at all of this; it also doesn't have an up-to-date CA bundle unlike the new version.
The relevant parts of the code sending the request are here:
self.timeout = urllib3.util.timeout.Timeout(connect=90.0, read=300.0)
self.params = dict(maxsize = 10, block = True, timeout = timeout)
self.sslparams = dict(cert_reqs='CERT_REQUIRED', ca_certs=CA_CERTS,
ssl_version=ssl.PROTOCOL_TLS_CLIENT)
#...
sslparams = {} if not isSSL else {**self.sslparams,
'server_hostname': host,
'assert_hostname': host,
}
params = {**sslparams, **self.params}
pool = urllib3.PoolManager(**params)
r = pool.urlopen(self.command, url, body=body, headers=headers,
enforce_content_length=False, retries=1, redirect=False,
preload_content=False, decode_content=False)
Additional info:
> python -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.1.1g 21 Apr 2020
> python --version
Python 3.9.1
I have pyOpenSSL v20.0.1 and urllib3 v1.26.2. I'm on Windows 10 x64 with the latest updates installed.

connect to rabbitmq SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)

I'm trying to connect rabbitmq using SSL, the encrypted connection on port 61613 works great, but I can't make the connection to encrypted port 61614 using Let's Encrypt .
I tride simply connect to TLS 61614 rabbitmq stomp with this code:
hostname='rabbitmq.DOMAIN.NAME'
context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
with socket.create_connection((hostname, 61614 )) as sock:
with context.wrap_socket(sock, server_hostname=hostname) as ssock:
print(ssock.version())
with result:
SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)
Test with openssl:
openssl s_client -connect rabbitmq.DOMAIN.NAME:61614
has this result:
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = rabbitmq.DOMAIN.NAME
verify return:1
140003270226176:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1544:SSL alert number 40
---
Certificate chain
0 s:CN = rabbitmq.DOMAIN.NAME
i:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
1 s:C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFXzCCBEegAwIBAgISBNLfEcyZu5MmKWiRqiljwOY5MA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDA4MDcwOTA3MTZaFw0y
MDExMDUwOTA3MTZaMB8xHTAbBgNVBAMTFHJhYmJpdG1xLmxhbm9zLmNsb3VkMIIB
.....
GhqnyShKe63Uf/Buxy1gqOpBXRO+Sd8L8ww0IUciamomYoKGkwGkcT6Y+SB+IxCg
pA+3qsUUKjxE2kJ2S+lwsiHxpsHEMyoSXxFnmoELKF3FQEk=
-----END CERTIFICATE-----
subject=CN = rabbitmq.DOMAIN.NAME
issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
---
Acceptable client certificate CA names
C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
CN = ************************
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3169 bytes and written 460 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: CF1*********************************7415F1B61
Session-ID-ctx:
Master-Key: 051*********************************B4
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1597525241
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
---
Any suggestion how to solve it?
The output from openssl s_client shows basically the same error as the output from Python:
.. ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1544:SSL alert number 40
But it also shows that the main part of the TLS handshake worked, i.e. it got certificate, cipher etc. And it also shows that the server seems to ask for a client certificate:
Acceptable client certificate CA names
C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
CN = ************************
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
But neither your openssl s_client nor your Python code provide any client certificate. Very likely this missing client certificate is the reason the server finally abandons the TLS handshake which results in the handshake failure.
Thus, either you need to change the server configuration that no client certificate is requested or you need to match the servers requirement for a client certificate by providing the expected one - whatever this is (check your server).
I just forgot to add to rabbitmq.conf this line:
ssl_options.depth = 2
Now everything forks correctly.
Thank you for help.

SSL client certificate is not sent by python 'requests' library or s_client, but works fine in web browsers

I am trying to write a client script in Python that accesses a web application and uses SSL client certificates for authentication. Though I am able to access the application from both Firefox and Chrome as long as I have the client certificate loaded, I get the following response whenever I send the request via Python 'requests:
400 No required SSL certificate was sent
nginx/1.4.6 (Ubuntu)
I have also tried Python httplib, s_client, and curl, and get the same error message. I am using the same client certificate for all the testing, in pkcs12 format for the web browsers and pulled out into certificate and key PEM files for the command line tools. My python code looks like:
import requests
CERT = r'/path/to/cert.crt' #Client certificate
KEY = r'/path/to/key.key' #Client private key
CACERT = r'/path/to/ca.crt' #Server certificate chain
session = requests.Session()
session.cert = (CERT, KEY)
resp = session.get('https://my.webapp.com/',
verify=CACERT)
print resp.content
session.close()
s_client gives more information about what is happening. Here's an abbreviated version of the output:
$ openssl s_client -cert cert.crt -key key.key -CAfile ca.crt -connect <host>:<port>
CONNECTED(00000003)
depth=2 /C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
verify return:1
depth=1 /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
verify return:1
depth=0 /serialNumber=tgBIwyM-p18O/aDyvyWNKHDnOezzDJag/OU=GT89519184/OU=See www.rapidssl.com/resources/cps (c)13/OU=Domain Control Validated - RapidSSL(R)/CN=*.rexdb.us
verify return:1
---
Certificate chain
...
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=...
---
No client certificate CA names sent
---
SSL handshake has read 3519 bytes and written 328 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES128-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : DHE-RSA-AES128-SHA
Session-ID: ...
Session-ID-ctx:
Master-Key: ...
Key-Arg : None
Start Time: 1409269239
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
GET / HTTP/1.1
Host: my.webapp.com
HTTP/1.1 400 Bad Request
Server: nginx/1.4.6 (Ubuntu)
Date: Thu, 28 Aug 2014 23:41:04 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: close
...
<head><title>400 No required SSL certificate was sent</title></head>
...
closed
I'm reasonably sure this is not an issue with the server because authentication works in a browser. However, the s_client output says 'No client certificate CA names sent', which sounds like a server problem (e.g. the client certificate is not being sent on to the server because the server isn't asking for it). Here is the relevant part of the nginx configuration:
ssl on;
ssl_certificate_key /path/to/server/key.pem;
ssl_certificate /path/to/server/certificate.pem;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:AES128:AES256:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK;
ssl_prefer_server_ciphers on;
ssl_session_timeout 5m;
ssl_client_certificate /path/to/client/ca/certificate.crt;
ssl_verify_depth 10;
ssl_verify_client on;
The application uses uwsgi through nginx. There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
The only other potentially relevant difference I can find is that in Firefox the connection is keep-alive and in s_client it is close. I've tried setting Connection: keep-alive in the header, with the same result.
There are multiple virtual hosts on the vm, and only this one uses certificate authentication.
I assume that means you have multiple certificates behind the same IP address and that the client has to use SNI (Server Name Indication) to send the server the expected hostname inside the SSL handshake. openssl s_client does not use SNI by default and I don't know if python does - it might depend on the version of python you use.
Because the client only sends a certificate if the server tells it do to it might be, that because of missing SNI you run into the wrong configuration part, that is the default part with another certificate and without requirement for client certificates.
I would recommend to try with openssl s_client again, but use the command line option -servername (not documented in man page but shown if called with -h) to explicitly set the expected server name. If this works you need to find a way to use SNI in python too. If this does not work please make a packet dump and make sure with wireshark, that the server really requires the client to send a certificate and the client really does not send a certificate.

openssl, python requests error: "certificate verify failed"

If I run the following command from my development box:
$ openssl s_client -connect github.com:443
I get the following last line of output:
Verify return code: 20 (unable to get local issuer certificate)
If I try to do this with requests I get another failed request:
>>> import requests
>>> r = requests.get('https://github.com/', verify=True)
With an exception raised:
SSLError: [Errno 1] _ssl.c:507: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
I can also run the first command with the verify flag and get similar output:
$ openssl s_client -connect github.com:443 -verify 9
...
Verify return code: 27 (certificate not trusted)
Basically this is telling me that there is a problem with the certificates. I can specify a specific certificate with both methods and it will work:
$ openssl s_client -connect github.com:443 -CAfile /etc/ssl/certs/DigiCert_High_Assurance_EV_Root_CA.pem -verify 9
...
Verify return code: 0 (ok)
and:
>>> r = requests.get('https://github.com/', verify='/etc/ssl/certs/DigiCert...pem')
<Response [200]>
So, to my question, what exactly is wrong here? Shouldn't requests/openssl already know where to find valid certs?
Other Info:
Python==2.7.6
requests==2.2.1
openssl 0.9.8h
Also, I know passing verify=False to the requests.get method will work too, but I do want to verify.
EDIT
I've confirmed that, as #Heikki Toivonen indicated in an answer, specifying the -CAfile flag for the version of openssl that I'm running works.
$ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'`
...
Verify return code: 0 (ok)
So there is nothing wrong with the version of openssl that I'm running, and there is nothing wrong with the default cacert.pem file that requests provides.
Now that I know openssl is meant to work that way, that the CAfile or the place to find certs has to be specified, I'm more concerned about getting requests to work.
If I run:
>>> r = requests.get('https://github.com/', verify='path to cacert.pem file')
I'm still getting the same error as before. I even tried downloading the cacert.pem file from http://curl.haxx.se/ca and it still didn't work. requests only seems to work (on this specific machine) if I specify a specific vendor cert file.
A side note: On my local machine everything is working as expected. There are several difference between the two machines though. I so far haven't been able to determine what the specific difference is that causes this issue.
If I run the following command from my development box:
$ openssl s_client -connect github.com:443
I get the following last line of output:
Verify return code: 20 (unable to get local issuer certificate)
You are missing DigiCert High Assurance EV CA-1 as a root of trust:
$ openssl s_client -connect github.com:443
CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
...
Start Time: 1393392088
Timeout : 300 (sec)
Verify return code: 20 (unable to get local issuer certificate)
Download DigiCert High Assurance EV CA-1 from DigiCert Trusted Root Authority Certificates:
$ wget https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt
--2014-02-26 00:27:50-- https://www.digicert.com/CACerts/DigiCertHighAssuranceEVCA-1.crt
Resolving www.digicert.com (www.digicert.com)... 64.78.193.234
...
Convert the DER encoded certifcate to PEM:
$ openssl x509 -in DigiCertHighAssuranceEVCA-1.crt -inform DER -out DigiCertHighAssuranceEVCA-1.pem -outform PEM
Then, use it with OpenSSL via the -CAfile:
$ openssl s_client -CAfile DigiCertHighAssuranceEVCA-1.pem -connect github.com:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV CA-1
verify return:1
depth=0 businessCategory = Private Organization, 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Delaware, serialNumber = 5157550, street = 548 4th Street, postalCode = 94107, C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
verify return:1
---
Certificate chain
0 s:/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHOjCCBiKgAwIBAgIQBH++LkveAITSyvjj7P5wWDANBgkqhkiG9w0BAQUFADBp
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
d3cuZGlnaWNlcnQuY29tMSgwJgYDVQQDEx9EaWdpQ2VydCBIaWdoIEFzc3VyYW5j
ZSBFViBDQS0xMB4XDTEzMDYxMDAwMDAwMFoXDTE1MDkwMjEyMDAwMFowgfAxHTAb
BgNVBA8MFFByaXZhdGUgT3JnYW5pemF0aW9uMRMwEQYLKwYBBAGCNzwCAQMTAlVT
MRkwFwYLKwYBBAGCNzwCAQITCERlbGF3YXJlMRAwDgYDVQQFEwc1MTU3NTUwMRcw
FQYDVQQJEw41NDggNHRoIFN0cmVldDEOMAwGA1UEERMFOTQxMDcxCzAJBgNVBAYT
AlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQHEw1TYW4gRnJhbmNpc2Nv
MRUwEwYDVQQKEwxHaXRIdWIsIEluYy4xEzARBgNVBAMTCmdpdGh1Yi5jb20wggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDt04nDXXByCfMzTxpydNm2WpVQ
u2hhn/f7Hxnh2gQxrxV8Gn/5c68d5UMrVgkARWlK6MRb38J3UlEZW9Er2TllNqAy
GRxBc/sysj2fmOyCWws3ZDkstxCDcs3w6iRL+tmULsOFFTmpOvaI2vQniaaVT4Si
N058JXg6yYNtAheVeH1HqFWD7hPIGRqzPPFf/jsC4YX7EWarCV2fTEPwxyReKXIo
ztR1aE8kcimuOSj8341PTYNzdAxvEZun3WLe/+LrF+b/DL/ALTE71lmi8t2HSkh7
bTMRFE00nzI49sgZnfG2PcVG71ELisYz7UhhxB0XG718tmfpOc+lUoAK9OrNAgMB
AAGjggNUMIIDUDAfBgNVHSMEGDAWgBRMWMsl8EFPUvQoyIFDm6aooOaS5TAdBgNV
HQ4EFgQUh9GPGW7kh29TjHeRB1Dfo79VRyAwJQYDVR0RBB4wHIIKZ2l0aHViLmNv
bYIOd3d3LmdpdGh1Yi5jb20wDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsG
AQUFBwMBBggrBgEFBQcDAjBjBgNVHR8EXDBaMCugKaAnhiVodHRwOi8vY3JsMy5k
aWdpY2VydC5jb20vZXZjYTEtZzIuY3JsMCugKaAnhiVodHRwOi8vY3JsNC5kaWdp
Y2VydC5jb20vZXZjYTEtZzIuY3JsMIIBxAYDVR0gBIIBuzCCAbcwggGzBglghkgB
hv1sAgEwggGkMDoGCCsGAQUFBwIBFi5odHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9z
c2wtY3BzLXJlcG9zaXRvcnkuaHRtMIIBZAYIKwYBBQUHAgIwggFWHoIBUgBBAG4A
eQAgAHUAcwBlACAAbwBmACAAdABoAGkAcwAgAEMAZQByAHQAaQBmAGkAYwBhAHQA
ZQAgAGMAbwBuAHMAdABpAHQAdQB0AGUAcwAgAGEAYwBjAGUAcAB0AGEAbgBjAGUA
IABvAGYAIAB0AGgAZQAgAEQAaQBnAGkAQwBlAHIAdAAgAEMAUAAvAEMAUABTACAA
YQBuAGQAIAB0AGgAZQAgAFIAZQBsAHkAaQBuAGcAIABQAGEAcgB0AHkAIABBAGcA
cgBlAGUAbQBlAG4AdAAgAHcAaABpAGMAaAAgAGwAaQBtAGkAdAAgAGwAaQBhAGIA
aQBsAGkAdAB5ACAAYQBuAGQAIABhAHIAZQAgAGkAbgBjAG8AcgBwAG8AcgBhAHQA
ZQBkACAAaABlAHIAZQBpAG4AIABiAHkAIAByAGUAZgBlAHIAZQBuAGMAZQAuMH0G
CCsGAQUFBwEBBHEwbzAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuZGlnaWNlcnQu
Y29tMEcGCCsGAQUFBzAChjtodHRwOi8vY2FjZXJ0cy5kaWdpY2VydC5jb20vRGln
aUNlcnRIaWdoQXNzdXJhbmNlRVZDQS0xLmNydDAMBgNVHRMBAf8EAjAAMA0GCSqG
SIb3DQEBBQUAA4IBAQBfFW1nwzrVo94WnEUzJtU9yRZ0NMqHSBsUkG31q0eGufW4
4wFFZWjuqRJ1n3Ym7xF8fTjP3fdKGQnxIHKSsE0nuuh/XbQX5DpBJknHdGFoLwY8
xZ9JPI57vgvzLo8+fwHyZp3Vm/o5IYLEQViSo+nlOSUQ8YAVqu6KcsP/e612UiqS
+UMBmgdx9KPDDzZy4MJZC2hbfUoXj9A54mJN8cuEOPyw3c3yKOcq/h48KzVguQXi
SdJbwfqNIbQ9oJM+YzDjzS62+TCtNSNWzWbwABZCmuQxK0oEOSbTmbhxUF7rND3/
+mx9u8cY//7uAxLWYS5gIZlCbxcf0lkiKSHJB319
-----END CERTIFICATE-----
subject=/businessCategory=Private Organization/1.3.6.1.4.1.311.60.2.1.3=US/1.3.6.1.4.1.311.60.2.1.2=Delaware/serialNumber=5157550/street=548 4th Street/postalCode=94107/C=US/ST=California/L=San Francisco/O=GitHub, Inc./CN=github.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1
---
No client certificate CA names sent
---
SSL handshake has read 4139 bytes and written 446 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 59D2883BBCE8E81E63E5551FAE7D1ACC00C49A9473C1618237BBBB0DD9016B8D
Session-ID-ctx:
Master-Key: B6D2763FF29E77C67AD83296946A4D44CDBA4F37ED6F20BC27602F1B1A2D137FACDEAC862C11279C01095594F9776F79
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1393392673
Timeout : 300 (sec)
Verify return code: 0 (ok)
Shouldn't requests/openssl already know where to find valid certs?
No. OpenSSL trusts nothing by default. Its a polar opposite of a browser's model, where nearly everything is trusted by default.
$ openssl s_client -connect github.com:443 -CAfile `python -c 'import requests; print(requests.certs.where())'`
...
>>> r = requests.get('https://github.com/', verify='path to cacert.pem file')
Why would you trust hundreds of CAs and subordinate CAs (re: cacert.pem) when you know the one CA that is certifying the public key for the site? Trust the one required root and nothing more: DigiCert High Assurance EV CA-1.
Trusting everything - as in the browser's model - is what allowed Comodo Hacker to spoof certificates for Gmail, Hotmail, Yahoo, etc when the Diginotar root was compromised.
From Request 2.4.0 the author recommends using certifi, which is a collection of Root Certificates. There's a python package for it:
pip install certifi
openssl s_client by default will not use the CA certificates file it ships with, but it does try to verify the connection. This is the reason why your test fails without any parameters and works with -CAfile.
Similarly, Requests tries to verify the connection by default, but it seems it doesn't know where the CA certificates are. This might be a configuration issue in your environment when building/installing OpenSSL, Python or Requests. I say this because the Requests website shows your example working against https://github.com without needing to set the CA path.
I use openSUSE tumbleweed and Windows, I could install and download new model files without any issues.
Yesterday I got my Mac mini m2 and I've installed katrain.
Later I tried to download models, then I have also errors for certificate. I am not sure that is an issue with certificate. So I checked cacert.pem under /opt/homebrew/Cellar/ca-certificates/2023-01-10/share/ca-certificates.
I tried to check and display details of this cacert.pem to add to system, I got a message like it is invalid format or too big.
Run openssl s_client -connect github.com:443 no issue.
So I work around this issue by copy *.bin.gz under ~/.katrain on my tumbleweed to /Users/$username/.katrain on Mac mini
Then katrain found the model files :)

Categories