We Keep Coding

Requests cannot connect to server with supplied .pem file, but urrlib3 works even without it

I am trying to send HTTP requests to server (only accessible from specific network, so all names are made up) via requests lib, but I am getting SSL error.
My code:
site_auth = HTTPBasicAuth("user", "password")
response = requests.get("https://my.hidden.site.com", auth=site_auth , verify="location/src/mycertfile.pem")
I have tried also this:
site_auth = HTTPBasicAuth("user", "password")
response = requests.get("https://my.hidden.site.com", auth=site_auth , verify=True, cert="location/src/mycertfile.pem")
But the result was the same stacktrace.
Stacktrace:
Traceback (most recent call last):
File "location\.venv\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "location\.venv\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "location\.venv\lib\site-packages\urllib3\connectionpool.py", line 976, in _validate_conn
conn.connect()
File "location\.venv\lib\site-packages\urllib3\connection.py", line 361, in connect
self.sock = ssl_wrap_socket(
File "location\.venv\lib\site-packages\urllib3\util\ssl_.py", line 377, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\Adam\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\Adam\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Users\Adam\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "location\.venv\lib\site-packages\requests\adapters.py", line 439, in send
resp = conn.urlopen(
File "location\.venv\lib\site-packages\urllib3\connectionpool.py", line 724, in urlopen
retries = retries.increment(
File "location\.venv\lib\site-packages\urllib3\util\retry.py", line 439, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='my.hidden.site.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "location/main_playground.py", line 15, in <module>
response = requests.get("https://my.hidden.site.com", auth=rt_auth, verify="location/mycertfile.pem")
File "location\.venv\lib\site-packages\requests\api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "location\.venv\lib\site-packages\requests\api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "location\.venv\lib\site-packages\requests\sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "location\.venv\lib\site-packages\requests\sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "location\.venv\lib\site-packages\requests\adapters.py", line 514, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='my.hidden.site.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')))
I am using these versions of the libs:
requests==2.23.0
urllib3==1.25.9
I have googled a lot, but nothing works for me. I have manually downloaded the certification chain and created my .pem file which I have checked with openssl verify. I have 3 certificates in it and I have them ordered from the root to my target site in top to bottom fashion like this:
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Weird thing is that urrlib3 works fine even without .pem file and explicitly enabled certificates, code:
http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED')
r = http.request('GET', "https://my.hidden.site.com", fields={'user': "user", 'pass': "password"})
print(r)
print(r.data.decode('utf-8'))
Another weird thing is that curl does not work by default, but works like it should when I supply the .pem file, so the requests should too, right?
curl https://my.hidden.site.com --cacert mycertfile.pem
I tried also Postman, but he does not work even with selected .pem file in the settings.
Could not get any response
There was an error connecting to https://my.hidden.site.com.
I have also tried installing root and intermediate certificates on my machine (Windows 10), but it has no effect. Has anyone any idea what can be wrong or what should I check next?
Thanks for any help
// EDIT 1
I had created the chain of certificates by using this command and extracting the 3 keys from it. I have changed the order in the .pem file like I have already mentioned so the key of the root CA would be first in the .pem file and also the intermediate is second and the key for my.hidden.site.com is third.
I can see there are lines unable to get local issuer certificate and
verify error:num=21:unable to verify the first certificate, but I do not know whether it is a problem or not, or maybe there is something missing.
$ openssl s_client -showcerts -connect my.hidden.site.com:443
CONNECTED(0000017C)
---
Certificate chain
0 s:SECRET LINE my.hidden.site.com
i:SECRET LINE intermediate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
1 s:SECRET LINE intermediate
i:SECRET LINE root ca
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
2 s:SECRET LINE root ca
i:SECRET LINE root ca
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
---
Server certificate
subject=SECRET LINE my.hidden.site.com
issuer=SECRET LINE intermediate
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4618 bytes and written 444 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: <HIDDEN_SESSION_ID>
Session-ID-ctx:
Master-Key: <HIDDEN_MASTER_KEY>
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
<HIDDEN_SESSION_TICKET>
Start Time: 1588693969
Timeout : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
Extended master secret: no
---
closed
depth=0 HIDDEN DETAILS, CN = my.hidden.site.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 HIDDEN DETAILS CN = my.hidden.site.com
verify error:num=21:unable to verify the first certificate
verify return:1

requests.get("https://my.hidden.site.com", auth=site_auth , verify=True, cert="location/src/mycertfile.pem")
According to the docs cert is a path to client certificate, which is used in case of mutual TLS. This is when the server also validates your certificate, so it's not the case here.
verify is the path to a CA bundle.
I have manually downloaded the certification chain and created my .pem
It depends on how you downloaded the cert chain, but servers don't send root CA certificate during TLS handshake. It's your job as a client to build the chain of trust. How to get it then? It depends on how was the my.hidden.site.com certificate generated. You could inspect the chain and see who is the issuer of the top-most certificate. If it's some global CA you can find the certificate online, but be careful with that.
Another weird thing is that curl does not work by default, but works like it should when I supply the .pem file, so the requests should too, right?
That's true ;/.
I tested https://stackoverflow.com using actual CA, which was matched by the browser and the intermediate CA that was send during handshake and intermediate CA didn't work. You might be using the intermediate one, but it's really weird that curl worked.
EDIT: As I said earlier, CA is not sent in the handshake, so you won't see it in openssl s_client output. You can probably verify that by checking if the CA is self-signed. Check if the issuer is the same as subject.
See this answer on why specifying intermediate CA as --cacert won't work with curl below 7.68.0.

Python requests throwing "certificate verify failed: unable to get local issuer certificate" error

When trying to call an API using python Requests I'm getting certificate verify failed: unable to get local issuer certificate error.
I have added the SSL certificate in cacert.pem of certifi. This did not fix the problem.
data = {'foo':'bar'}
url = 'hostname'
r = requests.post(url, data=data)
print(r)
The above code only works when I put verify=False in the above code.
I have also extracted SSL certificate of API using the below code and added it to certifi cacert.pem but that too did not work.
import ssl, socket
myhostname = 'hostname'
myctx = ssl.create_default_context()
myctx.check_hostname = False
myctx.verify_mode = ssl.CERT_NONE
s = myctx.wrap_socket(socket.socket(), server_hostname=myhostname)
s.connect((myhostname, 443))
bcert = s.getpeercert(binary_form=True)
cert = ssl.DER_cert_to_PEM_cert(bcert)
Below is the error that is getting thrown
Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connectionpool.py", line 976, in _validate_conn
conn.connect()
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\connection.py", line 361, in connect
self.sock = ssl_wrap_socket(
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\site-packages\urllib3\util\ssl_.py", line 377, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 1040, in _create
self.do_handshake()
File "C:\Users\user\AppData\Local\Programs\Python\Python38-32\lib\ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)

I have also extracted SSL certificate of API
My guess is that you used leaf server certificate as CA, and that's not how cert validation works. The SSL library on your side is trying to build a trust chain. Typical server certificate chain looks like this:
(CA ->) Intermediate CA -> server certificate
The CA cert is not sent in the TLS handshake with the server. It's your ssl client's job to find a matching trusted CA for the chain.
You have to add CA to CA_BUNDLE not leaf server cert.
How to get the CA cert? Try in your browser and click on the lock, the CA cert will be upmost in the chain if it's found.
You can also inspect the cert and find out who signed it (issuer attribute). Maybe you'll be able to find that issuer's cert in some public CA database.

Try running the program code below:
# install_certifi.py
#
# sample script to install or update a set of default Root Certificates
# for the ssl module. Uses the certificates provided by the certifi package:
# https://pypi.python.org/pypi/certifi
import os
import os.path
import ssl
import stat
import subprocess
import sys
STAT_0o775 = ( stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR
| stat.S_IRGRP | stat.S_IWGRP | stat.S_IXGRP
| stat.S_IROTH | stat.S_IXOTH )
def main():
openssl_dir, openssl_cafile = os.path.split(
ssl.get_default_verify_paths().openssl_cafile)
print(" -- pip install --upgrade certifi")
subprocess.check_call([sys.executable,
"-E", "-s", "-m", "pip", "install", "--upgrade", "certifi"])
import certifi
# change working directory to the default SSL directory
os.chdir(openssl_dir)
relpath_to_certifi_cafile = os.path.relpath(certifi.where())
print(" -- removing any existing file or link")
try:
os.remove(openssl_cafile)
except FileNotFoundError:
pass
print(" -- creating symlink to certifi certificate bundle")
os.symlink(relpath_to_certifi_cafile, openssl_cafile)
print(" -- setting permissions")
os.chmod(openssl_cafile, STAT_0o775)
print(" -- update complete")
if __name__ == '__main__':
main()

How to configure Python to ignore the hostname verification?

We are relatively new to in Python therefore may be the question is too simple.
We are using Python version 2.7.15.
We are trying to use Python over TLS without success.
This is our code:
import ssl,socket
import urllib2
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
context.verify_mode = ssl.CERT_REQUIRED
context.check_hostname = False
context.load_verify_locations("/py-test/python/bin/certificate.pem")
url = "https://10.0.0.12"
request = urllib2.Request(url)
websocket = urllib2.urlopen(request,None,None,None,None,None,context)
pages=websocket.readlines()
print pages
As you see, we have configured context.check_hostname = False
Unfortunately, it fails with the following exception
Traceback (most recent call last):
File "./test.py", line 11, in <module>
websocket = urllib2.urlopen(request,None,None,None,None,None,context)
File "/py-test/python/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/py-test/python/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/py-test/python/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/py-test/python/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/py-test/python/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/py-test/python/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)>
It is definitely the hostname verification.
If we use the correct certificate and correct hostname the request successful.
If we use the wrong certificate it fails with the following exception.
File "./test.py", line 8, in <module>
context.load_verify_locations("/py-test/python/bin/certificate_bad.pem")
ssl.SSLError: [X509] PEM lib (_ssl.c:3027)
Therefore, we need help to understand how to configure Python to ignore the hostname verification.
One more question (can be asked in the separate thread).
Do we have in Python a trustore file that include all known CA? Like cacerts.jks in Java.
Where can we find the trustore?
Added
We “want to verify that the certificate was signed by a valid CA”, but we “don't care whether it identifies the site you're actually connecting to”.
We need help to configure Python to ignore the hostname verification?
What is mistake in our code?
We have tried to create the code according to the documentation https://docs.python.org/2/library/ssl.html
Added 2
We have invested a lot of time but unfortunately we do not have the progress.
Is anyone has the working example in Python 2.7?
I mean is the code works if you access with other URL then appears in a certificate.
May be Python 2.7 cannot be configured to ignore the hostname verification?
What can be our problem?
We use it on CentOS 6.
May be it is related to OpenSSL? We use the latest version openssl-1.0.1e-57.el6.x86_64.
May be we should upgrade to Python 3.x?

As you discovered, you can accomplish this by customizing the SSLContext object used for verifying the connection.
However, to hook this into urllib2.urlopen you'll need to build a custom opener and install that.
Here's an example:
import httplib
import socket
import ssl
import urllib2
import certifi
class InterceptedHttpsConnection(httplib.HTTPSConnection):
def connect(self):
# Open an unencrypted TCP socket first
sock = socket.create_connection((self.host, self.port), self.timeout)
# Build up a custom SSLContext. (Might be better to do this once rather
# than on every request.)
context = ssl.SSLContext(ssl.PROTOCOL_TLS)
# We require the SSL context to verify the certificate.
context.verify_mode = ssl.CERT_REQUIRED
# But we instruct the SSL context to *not* validate the hostname.
context.check_hostname = False
# Load CA certificates from the certifi bundle.
context.load_verify_locations(cafile=certifi.where())
# Use our SSLContext object to wrap the bare socket into an SSL socket.
self.sock = context.wrap_socket(sock, server_hostname=self.host)
class InterceptedHttpsHandler(urllib2.HTTPSHandler):
def https_open(self, req):
return self.do_open(InterceptedHttpsConnection, req)
def main():
opener = urllib2.build_opener(InterceptedHttpsHandler)
urllib2.install_opener(opener)
contents = urllib2.urlopen('https://example.com/').read()
print contents

import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Forces python to skip the validation of certificates by default. Hope it helps.

Python Error SSL: ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:726)

i'm trying to get the SNI from a (IP, destport 443) with python 2.7.15 and i'm using the last version of OpenSSL and the ssl module.
Here there is my code:
import OpenSSL as OSsl #This two modules are imported for the only purpose of getting the SNI using function defined by us down here getSNI
import ssl
ip = "52.85.25.17"
dport = "443"
#With this function we get the Server Name Identification for the trasmissions with Secure Socket Layer identified by the port 443. We only care about the destinationIP and the destinationPort
def getSNI(ip, dport):
if dport != "443":
commonName = "Not SSL"
print commonName
else:
server_certificate = ssl.get_server_certificate((ip, dport))
x509 = OSsl.crypto.load_certificate(OSsl.crypto.FILETYPE_PEM, server_certificate) #x509 is referred to the standard used for PKI (Public Key Infrastructure) used in this case for ciphering our informations about certificate
#FILETYPE_PEM serializes data to a Base64-Encoded
#getting the informations about Certificate
certInfo = x509.get_subject()
commonName = certInfo.commonName
print (commonName)
return commonName
getSNI(ip,dport)
This works, but for the address specified (in the snippet of code i posted here) I get this error:
Traceback (most recent call last):
File "getSNI.py", line 31, in <module>
getSNI(ip,dport)
File "getSNI.py", line 17, in getSNI
server_certificate = ssl.get_server_certificate((ip, dport))
File "/usr/lib/python2.7/ssl.py", line 1023, in get_server_certificate
with closing(context.wrap_socket(sock)) as sslsock:
File "/usr/lib/python2.7/ssl.py", line 369, in wrap_socket
_context=self)
File "/usr/lib/python2.7/ssl.py", line 617, in __init__
self.do_handshake()
File "/usr/lib/python2.7/ssl.py", line 846, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:726)
I have all modules and packages upgraded, i read a lot of issues about this topic and i don't know how to solve this problem
Edit1: Executing whois I found that this IpAddress is connected to Amazon, so are there any particular issues about Amazon and SNI?

The point of SNI is that there could exists several domains which are resolved to concrete IP address. So, the IP you provided (52.85.25.17) is the one of such addresses. Server couldn't decide which domain's cert you are requesting, so it terminates the connection with error.
Appendix 1. Catching SSLError exceptions
You can catch ssl.SSLError this way:
try:
server_certificate = ssl.get_server_certificate((ip, dport))
...
except ssl.SSLError as e:
common_name = "Handshake Failed"

Prevent OpenSSL from using system certificates?

How can I prevent OpenSSL (specifically, Python's ssl module) from using system certificate authorities?
In other words, I would like it to trust only the certificate authorities which I specify, and nothing else:
ssl_socket = ssl.wrap_socket(newsocket, server_side=True, certfile="my_cert.pem",
ca_certs=MY_TRUSTED_CAs, # <<< Only CAs specified here
cert_reqs=ssl.CERT_REQUIRED, ssl_version=ssl.PROTOCOL_TLSv1)

I've just run a few tests, and listing your selection of CAs in the ca_certs parameters is exactly what you need.
The system I've tried it on is Linux with Python 2.6. If you don't use ca_certs, it doesn't let you use cert_reqs=ssl.CERT_REQUIRED:
Traceback (most recent call last):
File "sockettest.py", line 18, in <module>
cert_reqs=ssl.CERT_REQUIRED, ssl_version=ssl.PROTOCOL_TLSv1)
File "/usr/lib/python2.6/ssl.py", line 350, in wrap_socket
suppress_ragged_eofs=suppress_ragged_eofs)
File "/usr/lib/python2.6/ssl.py", line 113, in __init__
cert_reqs, ssl_version, ca_certs)
ssl.SSLError: _ssl.c:317: No root certificates specified for verification of other-side certificates.
I've also tried to use a client to send a certificate that's not from a CA in the ca_certs parameter, and I get ssl_error_unknown_ca_alert (as expected).
Note that either way, there's no client-certificate CA list send (in the certificate_authorities list in the CertificateRequest TLS message), but that wouldn't be required. It's only useful to help the client choose the certificate.