We Keep Coding

SSL error on macOS Monterey when using python urllib via RDFLib

So I have been getting the following error when trying to request an HTTPS resource using the python RDFLib module:
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>
What I have tried thus far:
Installed Python 3.10.4 from python.org
Executed the Install Certificates.command successfully tried the request again and I get the same error
Created a virtual environment using python3 -m venv venv which is based on Python 3.10.4, activated it, updated certifi just in case using pip3 install certifi --upgrade as it was suggested in other posts, and tried the request again and I still got the error.
Out of desperation, I did a clean install of Monterey and still get the same error
So it appears that the issue is that the certificates are not actually getting installed even though the command executes successfully:
/Applications/Python\ 3.10/Install\ Certificates.command ; exit;
meh#iMac ~ % /Applications/Python\ 3.10/Install\ Certificates.command ; exit;
-- pip install --upgrade certifi
Requirement already satisfied: certifi in /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages (2021.10.8)
-- removing any existing file or link
-- creating symlink to certifi certificate bundle
-- setting permissions
-- update complete
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
I have actually been trying to solve this for many months, coming back and trying again and again but I can't seem to find a solution.
This seems to be a common issue that is usually resolved by executing the Install Certificates.command but that simply doesn't work for me. It's worth noting that I have the same issue on a brand new MacBook Pro with Monterey.
I would appreciate any suggestions on what I can try next.

It turns out that the target resource has a misconfigured certificate. Since there is no way to disable certificate verification in RDFLib - as far as I can tell - I have worked around the solution by requesting the resource via the python requests module with verify=False instead. Then I load the resource into RDFLib as text.

Can't install tornado

I am setting up tensorflow and i got this output:
ERROR: matplotlib 1.3.1 requires nose, which is not installed.
so i am doing
sudo easy_install tornado
and i got as output:
earching for tornado
Reading https://pypi.python.org/simple/tornado/
Download error on https://pypi.python.org/simple/tornado/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) -- Some packages may not be found!
Couldn't find index page for 'tornado' (maybe misspelled?)
Scanning index of all packages (this may take a while)
Reading https://pypi.python.org/simple/
Download error on https://pypi.python.org/simple/: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590) -- Some packages may not be found!
No local packages or download links found for tornado
error: Could not find suitable distribution for Requirement.parse('tornado')
what should i do?
thanks
francesco
ERROR: matplotlib 1.3.1 requires nose, which is not installed.

It seems that you are running the command behind a proxy or a firewall. If this is the case, then you need to set the corresponding environment variables:
set http_proxy="username:password#host:port"
set https_proxy="username:password#host:port"
Otherwise, you can install tornado using pip by running (if you are not behind a proxy, then ignore --proxy):
pip --proxy http://user:password#proxy-host:port install tornado
EDIT: I would suggest to run
sudo -H pip install --ignore-installed matplotlib
in order to install the required packages and ignore the current version.

Brew Python3.7 install on Mac High OSierra 10.13.6. Help understand openssl install instructions

I am having problems installing python and keep getting openssl errors. So I have reinstalled numerous times and still struggling. I think I am not following the below instructions properly and hope someone can explain what I should do. I have read everything on openssl errors online over the last 3 weeks and am practically at the end of my wits. I will very likely give up learning to program if I can't figure this out.
I am able to install packages using pip3. But get errors if I need to access web pages. Examples of errors I get:
pip3 installed geopy but running below raises a huge error the final part of which I have included below:
import geopy
geocoder=geopy.geocoders.GoogleV3(domain="maps.google.co.uk")
geocoder.geocode('Cambridge', exactly_one=False)
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)'
I can install pandas and pandas_datareader but running below raises error the final part of which follows the code:
import datetime as dt
import matplotlib.pyplot as plt
from matplotlib import style
import pandas as pd
import pandas_datareader.data as web
style.use('ggplot')
start = dt.datetime(2000,1,1)
end = dt.datetime(2016,12,31)
df = web.DataReader('TSLA', 'yahoo', start, end)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 511, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='finance.yahoo.com', port=443): Max retries exceeded with url: /quote/TSLA/history?period1=946699200&period2=1483243199&interval=1d&frequency=1d&filter=history (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1045)')))
So I need to know exactly what steps to take after installing python 3.7 via brew as stated below.
Do I need to add additional certificates and as a result place a .pem file in certs directory? When and why is such an action necessary?
Do I only need to run c_rehash if the answer to question 1 is true?
Do I need to have openssl first in my path?
Do I need compilers to find openssl? Under what circumstances should I set the LDFLAGS and CPPFLAGS as stated?
Do I need pkg-config to find openssl?
Basically, I am a beginner python user and not sure how many of these things are necessary for me if I just want to install packages using pip3 and be able to securely access data and websites online.
$ brew install openssl
Updating Homebrew...
==> Downloading https://homebrew.bintray.com/bottles/openssl-1.0.2p.high_sierra.
Already downloaded: /Users/user/Library/Caches/Homebrew/downloads/f3b0a441f330cc1e6072080557f91b61256c8b0734e8348563d0efcb2af248ce--openssl-1.0.2p.high_sierra.bottle.tar.gz
==> Pouring openssl-1.0.2p.high_sierra.bottle.tar.gz
==> Caveats
A CA file has been bootstrapped using certificates from the SystemRoots
keychain. To add additional certificates (e.g. the certificates added in
the System keychain), place .pem files in
/usr/local/etc/openssl/certs
and run
/usr/local/opt/openssl/bin/c_rehash
openssl is keg-only, which means it was not symlinked into /usr/local,
because Apple has deprecated use of OpenSSL in favor of its own TLS and crypto libraries.
If you need to have openssl first in your PATH run:
echo 'export PATH="/usr/local/opt/openssl/bin:$PATH"' >> ~/.bash_profile
For compilers to find openssl you may need to set:
export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
For pkg-config to find openssl you may need to set:
export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"
==> Summary
🍺 /usr/local/Cellar/openssl/1.0.2p: 1,793 files, 12.3MB

Certificate Verification Failure for youtube-dl

I am trying to get the audio from a YouTube video, using the command:
youtube-dl --extract-audio --audio-format mp3 [video link]
Each time I try to run this command in Terminal on macOS (High Sierra v10.13.2), it gives me this error:
ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)> (caused by URLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)'),))
Note: I have youtube-dl installed, and also have Python 2.7 and Python 3.6 (if that helps)
If anyone can help me resolve that issue, that would be great.

I've had this issue for a while now and was never able to solve it. So I did the temporary fix of
[ Terminal ]
--no-check-certificate
Sadly, that just turns your HTTPS youtube-dl request into a plain text HTTP request.
After some digging I found that it was a Python v3.6 issue.
https://bugs.python.org/issue29065#msg283984
If you use VIM or any edit to check
vim '/Applications/Python 3.6/ReadMe.rtf'
If you look for ( Certificate verification and OpenSSL\ )
You'll see that ...
During installation of Python v3.6 They decide to use their own private version of OpenSSL, unfortunately that doesn't natively work with the default root certificates. It does however, come with a command script to install a curated bundle of default root certificates.
The bug recommended me to the python certifi module. The modules just seems to be good for finding where your certificate is.
[ Python v3.6 ]
import certifi
certifi.where()
'/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/certifi/cacert.pem'
That really just told where it was, but I did it using the python terminal interpreter just in case.
[ Terminal ]
open '/Applications/Python 3.6/Install Certificates.command'
Me personally this was my terminal response...
[ Terminal ]
-- pip install --upgrade certifi
Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
-- removing any existing file or link
-- creating symlink to certifi certificate bundle
-- setting permissions
Traceback (most recent call last):
File "", line 44, in
File "", line 40, in main
PermissionError: [Errno 1] Operation not permitted: 'cert.pem'
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
BUT IT WORKED AND I DON'T HAVE TO DEAL WITH THAT SSL VERIFICATION FAILED ANYMORE! (-.-)\ /(-.-)/

I will try to make a proper approach to the problem rather than just listing what works for me. The following does not require the Install Certificates.command script, which may or may not have been installed along with Python.
Source of the problem
It needs to be pointed out that the source of the error is not Python itself. Looking closely to the error message, you can see that Python just communicates the error produced by the OpenSSL library (called by _ssl.c). If the Python you are using has been installed using homebrew, chances are that more network utilities (e.g. wget) installed using homebrew have similar issues.
This means that in order to fix the problem, we need to make sure that the OpenSSL library used by Python has access to a valid, up-to-date certificates bundle.
Locating the right OpenSSL library
As many OpenSSL libraries may be installed on your system, you need to find the one used by your Python interpreter. The OpenSSL library is loaded by the ssl Python module, so we need to locate that first:
pyssld=$(python3 -c 'import ssl, pathlib; print(pathlib.Path(ssl.__file__).parent)')
echo "$pyssld"
This will print out the directory where we should look for the C library used by the ssl to load OpenSSL. This is done with the following command:
pyssl=$(find "$pyssld" -iname '*ssl*.so')
echo "$pyssl"
Finally, we can check where the OpenSSL library loaded by the Python ssl module is located:
pyopenssl=$(otool -L "$pyssl" | grep libssl | awk '{print $1}')
echo "$pyopenssl"
This shall print something like:
/opt/homebrew/opt/openssl#1.1/lib/libssl.1.1.dylib
This points where the OpenSSL library used by Python is located.
Fixing the problem
Acquiring a certificate bundle
To acquire an up to date certificate bundle to use with OpenSSL library, you can install the Python certifi package.
pip3 install --upgrade certifi
cabundle=$(python3 -c 'import certifi; print(certifi.where())')
echo $cabundle
Adding the new certificate bundle to OpenSSL
Finally, we need to place the certificate bundle where OpenSSL can find it. We know the location of the library. But the bundles are stored in a different directory. To jump to the right directory and link the bundle, use:
cd $(echo "$pyopenssl" | sed -E 's%/opt/(openssl[^/]*).*%/etc/\1%')
ln -sf "$cabundle" cert.pem
Cleanup
After checking that everything works, you can now clean up the environment variables we have used.
unset cabundle pyopenssl pyssl pyssld

Install anaconda library from a local source

I have been trying to install pymc for some time on a Windows PC behind a very complicated proxy; effectively making this an installation on a computer not connected to the internet. I have tried - unsuccessfully - to set a proxy in the condarc file but I still get error messages
conda install -c https://conda.binstar.org/pymc pymc
Fetching package metadata: SSL verification error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

To solve this, you need to download the tar file (even if using Windows) that the installer is trying to fetch and then use offline mode:
Use this command (which will throw an error), to determine the file to be downloaded:
>conda install -c https://conda.binstar.org/pymc pymc
Fetching package metadata: SSL verification error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
....
Solving package specifications: ......................
The following packages will be downloaded:
package | build
---------------------------|-----------------
pymc-2.3.5 | np110py35_0 402 KB defaults
The following NEW packages will be INSTALLED:
pymc: 2.3.5-np110py35_0 defaults
Proceed ([y]/n)? y
Fetching packages ...
Could not connect to https://repo.continuum.io/pkgs/free/win-64/pymc-2.3.5-np110py35_0.tar.bz2
... [error message continues]...
Now download the tar file mentioned in the error message:
https://repo.continuum.io/pkgs/free/win-64/pymc-2.3.5-np110py35_0.tar.bz2
And then run this command with the path to the tar file:
>conda install --offline C:\pymc-2.3.5-np110py35_0.tar.bz2

Just of notes: "-c conda-forge" might be handy for some other packages. For example:
conda install -c conda-forge python-levenshtein

Adding to the solution above. Anyone met "SSL verification error" can just turn the verification step off temporarily in ~/.condarc
channels:
- defaults
ssl_verify: true
In this way, you could install from your local hub.

I was not able to run offline installation in Anaconda (on Win10) because Anaconda always tried to connect to remote channels or collecting metadata process failed.
To solve this I had to:
Download requested module as bz2 file (or convert tar.gz into tar.bz2)
Run Anaconda prompt (ev. navigate to folder with bz2 file)
Run offline installation with conda install path-to-bz2, i.e conda install zeep-4.0.0.tar.bz2