We Keep Coding

Requests (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.") On Linux

This question has been answered here but for Windows users. I am getting this error on a Linux machine on a 3rd party application under a Anaconda environment. I added the following paths to my PATH env:
<path>/anaconda2/lib
<path>/anaconda2/bin
But when I run:
import requests
requests.get("https://api.ipify.org")
I get the error:
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.ipify.org',
port=443): Max retries exceeded with url: / (Caused by SSLError("Can't
connect to HTTPS URL because the SSL module is not available."))
The 3rd party application and a conda shell both include those paths in the PATH env but it only works on the shell.

The error message says "SSL module is not available". Requests uses openssl for its "SSL module" so make sure openssl is installed and working correctly.
First run:
~$ conda list
You should see openssl and pyopenssl in the output
if you don't install openssl and pyopenssl
If openssl is installed run:
~$ openssl
If you get:
openssl: error while loading shared libraries: libcrypto.so.1.0.0: cannot enable executable stack as shared object requires: Invalid argument
Run using the version number from above
~$ sudo find / -name libcrypto.so.[your version]
Then using the path to libcrypto.so.[your version] from the output of the above:
~$ sudo execstack -c [path to]/libcrypto.so.[your version]
Then run to test:
~$ openssl
OpenSSL>
You will need to do this for all environments

It turns out the _ssl.so module in the lib-dynload directory of my python environment was being overriden by the _ssl.so module in the 3rd party software, it was conflicting. So I have made a sym-link to my own module.

Can't connect to HTTPS URL because the SSL module is not available on Mac/OS

Trying to build Python 2.7.17 and then install pip on Mac OS 10.14
Installed OpenSSL using brew
Python's build fails to build _ssl. When I try to build pip, it fails with error "Can't connect to HTTPS URL because the SSL module is not available
OpenSSL is correctly installed at /usr/local/Cellar/openssl#1.1/1.1.1d/

Python looks for ssl at /usr/local/ssl
To get it to build I did the following:
1. Set up a path to OpenSSL that didn't involve any weird characters:
cd /usr/local/Cellar/
ln -s openssl\#1.1 openssl
2: Ran ./configure
3: Edited Makefile replacing all "/usr/local/ssl" with "/usr/local/Cellar/openssl"
4: Run make / make test / make install
This created a Python that had _ssl, and thus pip would run

Certificate Verification Failure for youtube-dl

I am trying to get the audio from a YouTube video, using the command:
youtube-dl --extract-audio --audio-format mp3 [video link]
Each time I try to run this command in Terminal on macOS (High Sierra v10.13.2), it gives me this error:
ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)> (caused by URLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)'),))
Note: I have youtube-dl installed, and also have Python 2.7 and Python 3.6 (if that helps)
If anyone can help me resolve that issue, that would be great.

I've had this issue for a while now and was never able to solve it. So I did the temporary fix of
[ Terminal ]
--no-check-certificate
Sadly, that just turns your HTTPS youtube-dl request into a plain text HTTP request.
After some digging I found that it was a Python v3.6 issue.
https://bugs.python.org/issue29065#msg283984
If you use VIM or any edit to check
vim '/Applications/Python 3.6/ReadMe.rtf'
If you look for ( Certificate verification and OpenSSL\ )
You'll see that ...
During installation of Python v3.6 They decide to use their own private version of OpenSSL, unfortunately that doesn't natively work with the default root certificates. It does however, come with a command script to install a curated bundle of default root certificates.
The bug recommended me to the python certifi module. The modules just seems to be good for finding where your certificate is.
[ Python v3.6 ]
import certifi
certifi.where()
'/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/certifi/cacert.pem'
That really just told where it was, but I did it using the python terminal interpreter just in case.
[ Terminal ]
open '/Applications/Python 3.6/Install Certificates.command'
Me personally this was my terminal response...
[ Terminal ]
-- pip install --upgrade certifi
Requirement already up-to-date: certifi in /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages
-- removing any existing file or link
-- creating symlink to certifi certificate bundle
-- setting permissions
Traceback (most recent call last):
File "", line 44, in
File "", line 40, in main
PermissionError: [Errno 1] Operation not permitted: 'cert.pem'
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
[Process completed]
BUT IT WORKED AND I DON'T HAVE TO DEAL WITH THAT SSL VERIFICATION FAILED ANYMORE! (-.-)\ /(-.-)/

I will try to make a proper approach to the problem rather than just listing what works for me. The following does not require the Install Certificates.command script, which may or may not have been installed along with Python.
Source of the problem
It needs to be pointed out that the source of the error is not Python itself. Looking closely to the error message, you can see that Python just communicates the error produced by the OpenSSL library (called by _ssl.c). If the Python you are using has been installed using homebrew, chances are that more network utilities (e.g. wget) installed using homebrew have similar issues.
This means that in order to fix the problem, we need to make sure that the OpenSSL library used by Python has access to a valid, up-to-date certificates bundle.
Locating the right OpenSSL library
As many OpenSSL libraries may be installed on your system, you need to find the one used by your Python interpreter. The OpenSSL library is loaded by the ssl Python module, so we need to locate that first:
pyssld=$(python3 -c 'import ssl, pathlib; print(pathlib.Path(ssl.__file__).parent)')
echo "$pyssld"
This will print out the directory where we should look for the C library used by the ssl to load OpenSSL. This is done with the following command:
pyssl=$(find "$pyssld" -iname '*ssl*.so')
echo "$pyssl"
Finally, we can check where the OpenSSL library loaded by the Python ssl module is located:
pyopenssl=$(otool -L "$pyssl" | grep libssl | awk '{print $1}')
echo "$pyopenssl"
This shall print something like:
/opt/homebrew/opt/openssl#1.1/lib/libssl.1.1.dylib
This points where the OpenSSL library used by Python is located.
Fixing the problem
Acquiring a certificate bundle
To acquire an up to date certificate bundle to use with OpenSSL library, you can install the Python certifi package.
pip3 install --upgrade certifi
cabundle=$(python3 -c 'import certifi; print(certifi.where())')
echo $cabundle
Adding the new certificate bundle to OpenSSL
Finally, we need to place the certificate bundle where OpenSSL can find it. We know the location of the library. But the bundles are stored in a different directory. To jump to the right directory and link the bundle, use:
cd $(echo "$pyopenssl" | sed -E 's%/opt/(openssl[^/]*).*%/etc/\1%')
ln -sf "$cabundle" cert.pem
Cleanup
After checking that everything works, you can now clean up the environment variables we have used.
unset cabundle pyopenssl pyssl pyssld

Install anaconda library from a local source

I have been trying to install pymc for some time on a Windows PC behind a very complicated proxy; effectively making this an installation on a computer not connected to the internet. I have tried - unsuccessfully - to set a proxy in the condarc file but I still get error messages
conda install -c https://conda.binstar.org/pymc pymc
Fetching package metadata: SSL verification error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

To solve this, you need to download the tar file (even if using Windows) that the installer is trying to fetch and then use offline mode:
Use this command (which will throw an error), to determine the file to be downloaded:
>conda install -c https://conda.binstar.org/pymc pymc
Fetching package metadata: SSL verification error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)
....
Solving package specifications: ......................
The following packages will be downloaded:
package | build
---------------------------|-----------------
pymc-2.3.5 | np110py35_0 402 KB defaults
The following NEW packages will be INSTALLED:
pymc: 2.3.5-np110py35_0 defaults
Proceed ([y]/n)? y
Fetching packages ...
Could not connect to https://repo.continuum.io/pkgs/free/win-64/pymc-2.3.5-np110py35_0.tar.bz2
... [error message continues]...
Now download the tar file mentioned in the error message:
https://repo.continuum.io/pkgs/free/win-64/pymc-2.3.5-np110py35_0.tar.bz2
And then run this command with the path to the tar file:
>conda install --offline C:\pymc-2.3.5-np110py35_0.tar.bz2

Just of notes: "-c conda-forge" might be handy for some other packages. For example:
conda install -c conda-forge python-levenshtein

Adding to the solution above. Anyone met "SSL verification error" can just turn the verification step off temporarily in ~/.condarc
channels:
- defaults
ssl_verify: true
In this way, you could install from your local hub.

I was not able to run offline installation in Anaconda (on Win10) because Anaconda always tried to connect to remote channels or collecting metadata process failed.
To solve this I had to:
Download requested module as bz2 file (or convert tar.gz into tar.bz2)
Run Anaconda prompt (ev. navigate to folder with bz2 file)
Run offline installation with conda install path-to-bz2, i.e conda install zeep-4.0.0.tar.bz2

Python requests throwing SSL errors

This is a followup to SSLError using requests for python:
I have just installed requests on a Mac OSX 10.8.5. My first attempt at doing requests.get failed on missing certificate:
SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
The thread above says to look for /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/re‌​quests/cacert.pem but actually I don't even have a .../site-packages/requests directory. It's not clear to me if this should have been added by the installation (I used pip)
Further threads and the requests docs say to install certifi, so I did. But now I get a different error:
python -c 'import requests; requests.get("https://api.github.com/events")' /usr/lib/anaconda/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Traceback (most recent call last):
...
File "/usr/lib/anaconda/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm
Thanks!

Notice that you're using HTTPS. As mentioned in the Requests manual
To check a host’s SSL certificate, you can use the verify argument [...] By default, verify is set to True
Here are few ways to fix that:
Update OpenSSL (probably will solve your problem)
Taken from here:
If you encounter one of the following errors:
error:0D0890A1:asn1 encoding routines:ASN1_verify:unknown message digest algorithm
error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm
The software you are using might be compiled with a version too old of OpenSSL that does not take certificates signed with sha256WithRSAEncryption into account.
It requires at least OpenSSL 0.9.8o for a total management of SHA256. OpenSSl 0.9.7m only assures a partial management, for server
mode only.
Check your openssl version by
openssl version
OpenSSL 1.0.1k-fips 8 Jan 2015
If you have a smaller version than OpenSSL0.9.8o, you have to update its version (OS X):
brew update
brew install openssl
brew link --force openssl
If that doesn't work, try this way:
brew uninstall openssl
rm -rf /usr/local/openssl
brew install openssl
there's an issue with openssl installed before OS X 10.10.3 and reinstalling it fixes it
these command lines will uninstall openssl, remove its folder from your hard-disk and install it again (the updated version)
Install certifi
Taken from here
By default Requests bundles a set of root CAs that it trusts, sourced
from the Mozilla trust store. However, these are only updated once for
each Requests version. This means that if you pin a Requests version
your certificates can become extremely out of date.
From Requests version 2.4.0 onwards, Requests will attempt to use
certificates from certifi if it is present on the system. This allows
for users to update their trusted certificates without having to
change the code that runs on their system.
For the sake of security we recommend upgrading certifi frequently!
In other word, try to install certifi, if you have Request 2.4.0 or newer:
pip install certifi
Hopefully, this will fix the problem.
Use different version of OpenSSL and Requests
Looking into it using Google, I have found that there is a problem with OpenSSL in Python 2:
https://github.com/docker/docker-py/issues/465#issuecomment-76520363
https://github.com/Homebrew/homebrew/issues/38226
https://github.com/docker/compose/issues/1484
However, I am using Python 2.7.6, Requests 2.2.1 and OpenSSL 1.0.1f 6 Jan 2014 and everything runs correctly.
Pass the certificate
In other cases, you may need to tell requests.get the path to the certificate file, if the host's certificate was signed by you.
requests.get("https://api.github.com/events", verify=True, cert=['/path/to/my/ca.crt'])
Set the verify argument to False (NOT RECOMMENDED!)
In case you want to avoid the certificate verification, you have to pass verify=False to the request.get method.
python -c 'import requests; requests.get("https://api.github.com/events", verify=False)'
or from script.py file:
import requests
res = requests.get("https://api.github.com/events", verify=False)
print res
terminal:
$ python script.py
<Response [200]>
Important: Very bad idea; You can be MITM attacked, which is a critical security vulnerability.