We Keep Coding

can connect to URL with curl but not with requests (i.e. requests ignoring my CA bundle?)

I am able to connect to a certain URL with cURL, after I installed the corresponding SSL certificates:
$ export MY_URL=https://www.infosubvenciones.es/bdnstrans/GE/es/convocatoria/616783
$ curl -vvvv $MY_URL # Fails
$ sudo openssl x509 -inform pem -outform pem -in /tmp/custom-cert.pem -out /usr/local/share/ca-certificates/custom-cert.crt
$ sudo update-ca-certificates
$ curl -vvvv $MY_URL # OK
However, requests (or httpx, or any other library I use) refuses to do so:
In [1]: import os
...: import requests
...: requests.get(os.environ["MY_URL"])
---------------------------------------------------------------------------
SSLCertVerificationError Traceback (most recent call last)
...
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
My understanding is that requests uses certifi and as such these custom certificates are not available here:
In [1]: import certifi
In [2]: certifi.where()
Out[2]: '/tmp/test_ca/.venv/lib/python3.10/site-packages/certifi/cacert.pem'
I have already tried a number of things, like trying to use the system CA bundle:
export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt same error
requests.get(..., verify="/etc/ssl/certs/ca-certificates.crt") same error
switched to httpx + a custom SSL context as explained in the docs, same error
attempted truststore as discussed in this httpx issue, same error
How can I make Python (requests, httpx, raw ssl, anything) use the same certificates that cURL is successfully using?
The only thing that worked so far, inspired by this hackish SO answer, is to do verify=False. But I don't want to do that.
In [9]: requests.get(
...: my_url,
...: verify=False,
...: )
/tmp/test_ca/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'xxx'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

i tried your stuff on my system (Manjaro Linux, python 3.10) i can make a connection. I downloaded the complete certificate chain from the website (with my browser). After that i can use it with:
r = requests.get(url=URL, verify=<path to pem file>)
and with
export REQUESTS_CA_BUNDLE=<path to pem>
r = requests.get(url=URL)
I tried the export within pyCharm.
So the python stuff is working and you may have a problem in your certificates. Without this stuff i get the ssl error (of course), because python does not use the system certs as you mentioned correct. In my pem-file i have 3 certificates. Maybe you have only 1 and the others are in the global store, so that curl does not need the complete chain, instead of python. You should try to download the complete chain with your browser and try again.

How can I solve python openssl load_cert_chain function error?

I'm setting up a new server using Pytoh, tornado.
I'm going to set HTTPS.
But when server loads server certificate files, load_cert_chain function generates an error.
I purchased those certificate files from GoDaddy and it was okay on Apache Server before.
So certificate file and key file match and ca file is also okay.
python version: 3.7.4
tornado version : 4.5.2
openssl version: OpenSSL 1.1.1c 28 May 2019
ssl_ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl_ctx.load_verify_locations("../ca.crt")
ssl_ctx.verify_mode = ssl.CERT_REQUIRED
https_server = tornado.httpserver.HTTPServer(app, ssl_options=ssl_ctx)
This is error message.
ssl_ctx.load_cert_chain("../server.crt", "../server.key")
ssl.SSLError: [SSL] PEM lib (_ssl.c:3854)

I solved this problem. I use NginX server to host python server with digital signed certificates.

SSL Handshake issue with Pymongo on Python3

Trying to connect to Azure CosmosDB mongo server results into an SSL handshake error.
I am using Python3 and Pymongo to connect to my Azure CosmosDB. The connection works fine if I run the code with Python27 but causes the below error when using Python3:
import pymongo
from pymongo import MongoClient
import json
import sys
def check_server_status(client, data):
'''check the server status of the connected endpoint'''
db = client.result_DB
server_status = db.command('serverStatus')
print('Database server status:')
print(json.dumps(server_status, sort_keys=False, indent=2, separators=(',', ': ')))
coll = db.file_result
print (coll)
coll.insert_one(data)
def main():
uri = "mongodb://KEY123#backend.documents.azure.com:10255/?ssl=true&replicaSet=globaldb"
client = pymongo.MongoClient(uri)
emp_rec1 = {
"name":"Mr.Geek",
"eid":24,
"location":"delhi"
}
check_server_status(client, emp_rec1)
if __name__ == "__main__":
main()
Running this on Python3 results into below error:
pymongo.errors.ServerSelectionTimeoutError: SSL handshake failed:
backendstore.documents.azure.com:10255: [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
Here is my successful output when I run the same code with Python27:
Database server status: { "_t": "OKMongoResponse", "ok": 1 }
Collection(Database(MongoClient(host=['backend.documents.azure.com:10255'],
document_class=dict, tz_aware=False, connect=True, ssl=True,
replicaset='globaldb'), u'result_DB'), u'file_result')

On Windows you can do like this
pip install certifi
Then use it in code:
import certifi
ca = certifi.where()
client = pymongo.MongoClient(
"mongodb+srv://username:password#cluster0.xxxxx.mongodb.net/xyzdb?retryWrites=true&w=majority", tlsCAFile=ca)

Solved the problem with this change:
client = pymongo.MongoClient(uri, ssl_cert_reqs=ssl.CERT_NONE)

The section Troubleshooting TLS Errors of the PyMongo offical document `TLS/SSL and PyMongo introduces the issue as below.
TLS errors often fall into two categories, certificate verification failure or protocol version mismatch. An error message similar to the following means that OpenSSL was not able to verify the server’s certificate:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
This often occurs because OpenSSL does not have access to the system’s root certificates or the certificates are out of date. Linux users should ensure that they have the latest root certificate updates installed from their Linux vendor. macOS users using Python 3.6.0 or newer downloaded from python.org may have to run a script included with python to install root certificates:
open "/Applications/Python <YOUR PYTHON VERSION>/Install Certificates.command"
Users of older PyPy and PyPy3 portable versions may have to set an environment variable to tell OpenSSL where to find root certificates. This is easily done using the certifi module from pypi:
$ pypy -m pip install certifi
$ export SSL_CERT_FILE=$(pypy -c "import certifi; print(certifi.where())")
You can try to follow the description above to fix your issue, which seems to be for Linux and Mac Users. On Windows, I can not reproduce your issue in Python 3.7 and 3.6. If you have any concern, please feel free to let me know.

Faced the same issue when trying to connect mongodb from Digital Ocean,
Solved by using this function with params in MongoClient:
def get_client(host,port,username,password,db):
return MongoClient('mongodb://{}:{}/'.format(host,port),
username=username,
password=password,
authSource=db,
ssl=True,ssl_cert_reqs=ssl.CERT_NONE)
client = get_client("host-ip","port","username","password","db-name")

On Mac Mojave 10.14.6 , I used (PyMongo 3.10 and python 3.7), to solve:
flask pymongo pymongo.errors.ServerSelectionTimeoutError [SSL: CERTIFICATE_VERIFY_FAILED]
Execute in terminal:
sudo /Applications/Python\ 3.7/Install\ Certificates.command
If you use other python version, only change versión number (In my case, i have Python 3.7)

cluster = MongoClient(
"url",
ssl=True,
ssl_cert_reqs=ssl.CERT_NONE,
)

By default pymongo relies on the operating system’s root certificates.
It could be that Atlas itself updated its certificates or it could be that something on your OS changed. “certificate verify failed” often occurs because OpenSSL does not have access to the system’s root certificates or the certificates are out of date. For how to troubleshoot see TLS/SSL and PyMongo — PyMongo 3.12.0 documentation 107.
pls Try :
client = pymongo.MongoClient(connection, tlsCAFile=certifi.where())
and dont forget to install certifi

On mac Monterey, I used pymongo 3.12.1 and virtual environment
To solve, use
ssl_cert_reqs=CERT_NONE
with mongodb url

pip: cert failed, but curl works

We installed the our root cert on the client, and the https connection works for curl.
But if we try to use pip, it fails:
Could not fetch URL https://installserver:40443/pypi/simple/pep8/:
There was a problem confirming the ssl certificate:
<urlopen error [Errno 1] _ssl.c:499: error:14090086:SSL
routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed>
The cert is on the client. See:
(foo_fm_qti)foo_fm_qti#vis-work:~$ curl -v https://installserver:40443/pypi/simple/pep8/
* About to connect() to installserver port 40443 (#0)
* Trying 127.0.0.1... connected
* Connected to installserver (127.0.0.1) port 40443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs/
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS alert, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* subject: C=DE; ST=Sachsen; L=Chemnitz; O=FOO-COM GmbH; OU=DV; CN=gray.foo-com.lan; emailAddress=info#foo-com.de
* start date: 2013-09-09 10:47:50 GMT
* expire date: 2019-05-24 10:47:50 GMT
* subjectAltName: installserver matched
* issuer: C=DE; ST=Sachsen; L=Chemnitz; O=FOO-COM GmbH; CN=FOO-COM Root CA; emailAddress=info#foo-com.de
* SSL certificate verify ok.
> GET /pypi/simple/pep8/ HTTP/1.1
Version: pip 1.4.1

Unfortunately pip does not use the system certs, but curl does.
I found a solution:
pip --cert /etc/ssl/certs/FOO_Root_CA.pem install pep8
This is not nice (curl and other libraries find the cert without adding a parameter) but works.
If you don't want to use the command line argument, you can set the cert in ~/.pip/pip.conf:
[global]
cert = /etc/ssl/certs/Foo_Root_CA.pem

My solution is downloading cacert.pem from http://curl.haxx.se/ca/cacert.pem
and add the path for cacert.pem to ~/.pip/pip.conf as guettli suggested
[global]
cert = /path/to/cacert.pem

For me, none of the config-file workarounds worked. I'm using pip 1.5.4 on Ubuntu 14.04
The command posted by #arjenve didn't work on my system either. I get: /usr/bin/python: No module named _vendor.requests
UPDATE
An even better solution than my first workaround is installing the certificate on the system first (for me on ubuntu this would be)
sudo cp ~/my_cert.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
The previous automatically updates the bundle file (checking at the bottom of /etc/ssl/certs/ca-certificates.crt you should now see the same certificate as in my_cert.crt)
Now export that path into PIP_CERT and add it to your .bashrc:
echo export PIP_CERT=/etc/ssl/certs/ca-certificates.crt >> ~/.bashrc
OLDER WORKAROUND
My workaround was to create a bundle file from /etc/ssl/certs/ca-certificates.crt and my corporate's crt (just concatenated both files). And then export a variable (put that on my .bashrc) like this:
export PIP_CERT=/my/path/to/the/bundle.crt

I use:
export PIP_CERT=`python -m pip._vendor.requests.certs`
pip install pep8
PIP always validates the certificate of HTTPS connections (and all pypi packages redirect to HTTPS).
The algorithm for determining the CA file is based on 3 steps:
Look in a list of default locations for different linux distributions
(in my case this file turned out to be out of date, as I am building on a very old linux distribution)
If available, override the value found in (1) from a value in the pip.conf file, the environment or the command-line (in that order),
If both (1) and (2) did not result in a value, use a bundled file
Note that pip does not use the default SSL directories and files (from ssl.get_default_verify_paths()). But only supports a bundled CA file.
PIP does support a command-line action to list the bundled file from step 3 and that is what I use for this answer.

Why am I receiving a low level socket error when using the Fabric python library?

When I run the command:
fab -H localhost host_type
I receive the following error:
[localhost] Executing task 'host_type'
[localhost] run: uname -s
Fatal error: Low level socket error connecting to host localhost: Connection refused
Aborting.
Any thoughts as to why? Thanks.
Fabfile.py
from fabric.api import run
def host_type():
run('uname -s')
Configuration
Fabric 1.0a0 (installed from the most recent Github commit---b8e1b6a)
Paramiko 1.7.4
PyCrypto 2.0.1
Virtualenv ver 1.3.3
Python 2.6.2+ (release26-maint:74924, Sep 18 2009, 16:03:18)
Mac OS X 10.6.1

The important part isn't the "low level error" part of the message - the important part is the "Connection refused" part. You'll get a "connection refused" message when trying to connect to a closed port.
The most likely scenario is that you are not running an ssh server on your machine at the time that Fabric is running. If you do
ssh localhost
you'll probably get a message similar to
ssh: connect to host localhost: Connection refused
So you'll have to go out and set up an SSH server on your computer before you can proceed with Fabric from there.

I had the same problem, but the reason was different: While I could easily log in to the server via SSH (default port 22), fabric tried to connect on a closed port 9090.
Finally I recognized that I had defined "env.port=9090" in my old fabfile for some WSGI server setup; while that was never a problem, I updated my Python installation some weeks before, and fabric now uses env.port for its SSH connection.
I just renamed that config, and all is well again.

This can also happen in OS X 10.11.4 and Fabric 1.10.1, in the case where you are ssh'ing to a VM using Vagrant, which does port forwarding from localhost. In this case, localhost was resolving to the IPv6 ::1 address (not due to /etc/hosts file), and giving this error.
The fix was to force use of IPv4 by using the 127.0.0.1 address in the fabfile instead of the hostname. Using a hostname in /etc/hosts with this address didn't work.
You might also want to try these useful tips for debugging connection issues in Fabric.

env.roledefs = {
'role1': env.hosts[0:5],
'role2':[env.hosts[5],]
}
I encountered the same error if "role" value IS NOT A LIST. for example, the above code works but the following doesn't.
env.roledefs = {
'role1': env.hosts[0:5],
'role2':env.hosts[5],
}