I'm attempting to load a website using the Python requests package via a http/https proxy:
url = 'https://booster-magazine.ch' # Example of failing URL
proxies = {
'http': 'proxy.domain.internal:4321',
'https': 'proxy.domain.internal:4321',
}
r = requests.get(url, proxies=proxies)
Running this triggers the following error message
requests.exceptions.SSLError: ("bad handshake: Error([('SSL Routine', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
Web browsers indicate that the site's certificate is trusted and cURL on the command line using the same webproxy as the python code works just fine.
curl -x proxy.domain.internal:4321 https://booster-magazine.ch # No error
Some sites I have tested fail, while others work fine. As far as I can tell, the SSL Labs report for this failing site doesn't have any issues.
Package versions I'm using:
requests==2.23.0
certifi==2020.4.5.1
I'm aware of the existence of the verify=False option in the requests library. This is generally bad practice and opens the possibility to mitm attacks. The goal is to have a working ssl validation.
I am hosting a site using SSL / HTTPS, and am attempting to make a request to it from a Python 2.7 script on the server (Ubuntu 18.04).
When running the script, I get this error:
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)
However, when I run curl --verbose -X GET -I <url> on the same server, it says the certificate was verified.
I do know that the cert is in fact valid and is not a self signed cert.
Any ideas on what I can do to get python to accept that cert?
Edit: here's the code to trigger the issue. Note that I'm not including the URL as it is not accessible to the general public:
import requests
r = requests.get('https://www.example.org')
print r.status_code
I'm having trouble to set anaconda python 3.7 on pycharm , and i get this error:
CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.anaconda.com/pkgs/r/noarch/repodata.json.bz2
from the excuted command :
C:\Users\Matan\Anaconda3\Scripts\conda.exe create -p C:\Users\Matan\Anaconda3\envs\untitled2 -y python=3.7
Command output:
Collecting package metadata: ...working... failed
WARNING: The conda.compat module is deprecated and will be removed in a future release.
CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://repo.anaconda.com/pkgs/r/noarch/repodata.json.bz2
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.
SSLError(MaxRetryError('HTTPSConnectionPool(host=\'repo.anaconda.com\', port=443): Max retries exceeded with url: /pkgs/r/noarch/repodata.json.bz2 (Caused by SSLError("Can\'t connect to HTTPS URL because the SSL module is not available."))'))
Trying to connect to Azure CosmosDB mongo server results into an SSL handshake error.
I am using Python3 and Pymongo to connect to my Azure CosmosDB. The connection works fine if I run the code with Python27 but causes the below error when using Python3:
import pymongo
from pymongo import MongoClient
import json
import sys
def check_server_status(client, data):
'''check the server status of the connected endpoint'''
db = client.result_DB
server_status = db.command('serverStatus')
print('Database server status:')
print(json.dumps(server_status, sort_keys=False, indent=2, separators=(',', ': ')))
coll = db.file_result
print (coll)
coll.insert_one(data)
def main():
uri = "mongodb://KEY123#backend.documents.azure.com:10255/?ssl=true&replicaSet=globaldb"
client = pymongo.MongoClient(uri)
emp_rec1 = {
"name":"Mr.Geek",
"eid":24,
"location":"delhi"
}
check_server_status(client, emp_rec1)
if __name__ == "__main__":
main()
Running this on Python3 results into below error:
pymongo.errors.ServerSelectionTimeoutError: SSL handshake failed:
backendstore.documents.azure.com:10255: [SSL:
CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
Here is my successful output when I run the same code with Python27:
Database server status: { "_t": "OKMongoResponse", "ok": 1 }
Collection(Database(MongoClient(host=['backend.documents.azure.com:10255'],
document_class=dict, tz_aware=False, connect=True, ssl=True,
replicaset='globaldb'), u'result_DB'), u'file_result')
On Windows you can do like this
pip install certifi
Then use it in code:
import certifi
ca = certifi.where()
client = pymongo.MongoClient(
"mongodb+srv://username:password#cluster0.xxxxx.mongodb.net/xyzdb?retryWrites=true&w=majority", tlsCAFile=ca)
Solved the problem with this change:
client = pymongo.MongoClient(uri, ssl_cert_reqs=ssl.CERT_NONE)
The section Troubleshooting TLS Errors of the PyMongo offical document `TLS/SSL and PyMongo introduces the issue as below.
TLS errors often fall into two categories, certificate verification failure or protocol version mismatch. An error message similar to the following means that OpenSSL was not able to verify the server’s certificate:
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
This often occurs because OpenSSL does not have access to the system’s root certificates or the certificates are out of date. Linux users should ensure that they have the latest root certificate updates installed from their Linux vendor. macOS users using Python 3.6.0 or newer downloaded from python.org may have to run a script included with python to install root certificates:
open "/Applications/Python <YOUR PYTHON VERSION>/Install Certificates.command"
Users of older PyPy and PyPy3 portable versions may have to set an environment variable to tell OpenSSL where to find root certificates. This is easily done using the certifi module from pypi:
$ pypy -m pip install certifi
$ export SSL_CERT_FILE=$(pypy -c "import certifi; print(certifi.where())")
You can try to follow the description above to fix your issue, which seems to be for Linux and Mac Users. On Windows, I can not reproduce your issue in Python 3.7 and 3.6. If you have any concern, please feel free to let me know.
Faced the same issue when trying to connect mongodb from Digital Ocean,
Solved by using this function with params in MongoClient:
def get_client(host,port,username,password,db):
return MongoClient('mongodb://{}:{}/'.format(host,port),
username=username,
password=password,
authSource=db,
ssl=True,ssl_cert_reqs=ssl.CERT_NONE)
client = get_client("host-ip","port","username","password","db-name")
On Mac Mojave 10.14.6 , I used (PyMongo 3.10 and python 3.7), to solve:
flask pymongo pymongo.errors.ServerSelectionTimeoutError [SSL: CERTIFICATE_VERIFY_FAILED]
Execute in terminal:
sudo /Applications/Python\ 3.7/Install\ Certificates.command
If you use other python version, only change versión number (In my case, i have Python 3.7)
cluster = MongoClient(
"url",
ssl=True,
ssl_cert_reqs=ssl.CERT_NONE,
)
By default pymongo relies on the operating system’s root certificates.
It could be that Atlas itself updated its certificates or it could be that something on your OS changed. “certificate verify failed” often occurs because OpenSSL does not have access to the system’s root certificates or the certificates are out of date. For how to troubleshoot see TLS/SSL and PyMongo — PyMongo 3.12.0 documentation 107.
pls Try :
client = pymongo.MongoClient(connection, tlsCAFile=certifi.where())
and dont forget to install certifi
On mac Monterey, I used pymongo 3.12.1 and virtual environment
To solve, use
ssl_cert_reqs=CERT_NONE
with mongodb url
I am using Python Requests library to connect to a REST server by using .pem certificates to authenticate who I am to establish a connection so I can start collecting data, parsing it, etc. When I run my program via Eclipse or the terminal, I get this error:
[('system library', 'fopen', 'Permission denied'), ('BIO routines', 'FILE_CTRL', 'system lib'), ('SSL routines', 'SSL_CTX_use_certificate_file', 'system lib')]
However, if I run as 'sudo' - the requests library works as intended and I am able to retrieve the data. Unfortunately, running as 'sudo' has side effects that where the default Python interpreter is the root interpreter, which is Python2. However, there are a lot of library dependencies that are needed from Anaconda.
For context, here is the function I am using to establish a connection:
PEM_FILE = os.path.expanduser("path/to/pem/file.pem") #Path is to a folder in root level
def set_up_connection(self):
#URL's to request a connection with
rest_auth = 'https://www.restwebsite.com/get/data'
ip_address = self.get_ip_address()
body = json.dumps({'userid':'user', 'password':'pass', 'ip_address':ip_address})
try:
resp = self.session.post(rest_auth, data=body, cert=PEM_FILE, verify=False)
values = resp.json()
token = values['token']
except Exception as e:
print(e)
token = None
return token, ip_address
TLDR; Using 'python rest_connector.py' renders an error. Running that command as sudo works.
Context for the certificates: The .pem cert permissions is set to 600 (rw-------).
To try and solve my problem running as sudo, I have started a terminal and run 'sudo -Es' which sets the terminal to run as root and uses Anaconda as my default interpreter, BUT I end up with a handshake error:
[('SSL routines', 'ssl3_read_bytes', 'tlsv1 alert unknown ca'), ('SSL routines', 'ssl3_read_bytes', 'ssl handshake failure')]
If someone can help me solve it this way, it would be a nice temporary fix, but I still need to be able to run this without sudo.
Thanks, in advance.
The username needs to be able to read the file. You can verify by running ls path_to_file.pem.
If you have changed ownership of the file, you might still be missing executable permissions on the directories containing the file.
You can potentially fix that with chmod -R +x ~/path/to_directory_containing_perm
Ok, so I managed to solve this and I'll post what I did in case anyone else stumbles upon this with a similar problem.
Permissions being set to 600 for the certs and pem file, ownership being set to root, and performing the openssl hashing function, the only problem was where the certs were placed in the sub directories.
While I placed the certs into 'etc/pki/tls/certs', they actually belonged in 'etc/ssl/certs'. The same goes for the .pem file except that .pem would be placed in a restricted folder, 'private'. After moving the files to the correct folder. Then I was able to set the verify param for the request to the cert_path and everything worked like I needed it to.
resp = self.session.post(rest_auth, data=body, cert=PEM_FILE, verify=cert_path)
'etc/pki/tls/certs' is the directory for Fedora distribution of Linux. 'etc/ssl/certs' is the directory for the Ubuntu distribution of Linux.
I hope this helps someone.