SSL: CERTIFICATE_VERIFY_FAILED -> trouble opening up a web page [duplicate] - python

I need to use curtom root certificates on the company intranet and loading them in the Mac OS TrustStore (KeyChain) does solve the problem for all browsers and GUI apps.
It seems that it works even with the version of curl that ships with Mac OS X but it doesn't work with python, even the version that ships with Mac OS 10.12 Sierra (Python 2.7.10)
Still, it seems that I would be hit by:
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>
How can I solve this?
Because I encounter this issue in lots and lots of Python tools I would really appreciate if I find a way to avoid it without having to patch them.
Providing the custom CA certificate myself is not an option because I cannot patch tens of Python tools that I use.
Most of the tools are using the requests library but there are a few that are using the native ssl support in Python directly.

This is also a problem in Python 3.6 with MacOS Sierrra. I know your use case is different. But I stumbled upon this thread while investigating this problem. So if anyone is also having this article is worth checking out:
http://www.cdotson.com/2017/01/sslerror-with-python-3-6-x-on-macos-sierra/
In a nutshell: Python 3.6 does not rely on MacOS' openSSL anymore. It comes with its own openSSL bundled and doesn't have access on MacOS' root certificates.
You have two options:
Run an install command shipped with Python 3.6
cd /Applications/Python\ 3.6/
./Install\ Certificates.command
or
Install the certifi package with
pip install certifi
I chose the first option and it worked.

Run this to set the appropriate variables. This is a combination of the answers that have already been given here. Put it in your ~/.bash_profile to make it permanent.
CERT_PATH=$(python -m certifi)
export SSL_CERT_FILE=${CERT_PATH}
export REQUESTS_CA_BUNDLE=${CERT_PATH}

Mac brew install python env.
$ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import certifi
>>> certifi.where()
'/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/cacert.pem'
>>>
Or from the command line:
$ python -m certifi
then need link cacert.pem as cert.pem
$ ln -s /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/certifi/cacert.pem cert.pem
$ pwd
/Library/Frameworks/Python.framework/Versions/3.7/etc/openssl
rehash
then work fine.

If you put the additional certificates in a PEM bundle file you can use these two environment variables to overwrite the default cert stores used by Python openssl and requests.
SSL_CERT_FILE=/System/Library/OpenSSL/cert.pem
REQUESTS_CA_BUNDLE=/System/Library/OpenSSL/cert.pem
Please note that this file does not exist, you need to build it yourself.

This was intended as an edit to an existing question, but since the queue was already full, posting as a separate answer.
Tested on MacOS 12.3.1 with python 3.10 installed with MacPorts.
If you prefer to trust root CA according to your OS, export them from System Roots keychain into a single file:
security export -t certs -f pemseq -k /System/Library/Keychains/SystemRootCertificates.keychain -o bundleCA.pem
If, additionally, you want to trust some internal self-signed CAs, export them, too. They are probably stored under the System keychain:
security export -t certs -f pemseq -k /Library/Keychains/System.keychain -o selfSignedCAbundle.pem
Merge the two files:
cat bundleCA.pem selfSignedCAbundle.pem >> allCAbundle.pem
Export as bash variable
export REQUESTS_CA_BUNDLE=/path/to/allCAbundle.pem
Consider adding the last code snippet to your .bash_profile
Note that REQUESTS_CA_BUNDLE works only for a single file, and does not for a directory.

I see a lot of answers out there recommend to turn off certificate validation or to use certifi.where.
While turning off SSL is obvious risk. certifi.where is also a risk, mainly if you intend to make this code a production code that will run in a customer env.
PEP describing why it is wrong.
ssl.create_default_context is well integrate with linux and windows trustStore. the problem is, as in your case with mac.
I solve this by loading the certificates using the integrated security commandline tool
def create_macos_ssl_context():
import subprocess
import ssl
import tempfile
ctx = ssl.create_default_context()
macos_ca_certs = subprocess.run(["security", "find-certificate", "-a", "-p",
"/System/Library/Keychains/SystemRootCertificates.keychain"],
stdout=subprocess.PIPE).stdout
with tempfile.NamedTemporaryFile('w+b') as tmp_file:
tmp_file.write(macos_ca_certs)
ctx.load_verify_locations(tmp_file.name)
print(ctx.get_ca_certs())
Note that this gives you the systemRoot certificates. if you need the user than simply change the value in the security command

As an update and datapoint, I ran into this issue running Python 3.7.0 on macOS 10.13.4:
$ ipython
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.0.1 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import bokeh.sampledata
In [2]: bokeh.sampledata.download()
Using data directory: /Users/me/.bokeh/data
...
SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)
Instructions for solving the problem are in /Applications/Python\ 3.7/ReadMe.rtf
Following the suggestion there and running /Applications/Python\ 3.7/Install\ Certificates.command solved the problem:
From the terminal:
$ /Applications/Python\ 3.7/Install\ Certificates.command
Re-starting IPython...
$ ipython
>>> import bokeh.sampledata
>>> bokeh.sampledata.download()
Using data directory: /Users/me/.bokeh/data
Downloading: CGM.csv (1589982 bytes)
1589982 [100.00%]
...

For me /Applications/Python\ 3.6/./Install\ Certificates command fails on pip certifi install. I am on mac High Sierra and use python3 so pip somewhat fails and I have to use pip3 instead.
So here what I did:
Manually ran pip3 install --update certify in a shell
Remove the install certifi line from the command script
Reran the script and everything was fine.
Note that you will end up with a cert.pem symbolic link in: /Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/

Solution for MacOS or Linux with latest Python versions installed either as standalone or via port or brew
Download certificates from Certifi project at https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem. FYI Certifi is a 3rd party library that provides Mozilla’s curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.
Then add to your ~/.zshrc on the latest MacOS or ~/.bash_profile or similar:
export SSL_CERT_FILE=/pathtodownloadedfile/cacert.pem
export REQUESTS_CA_BUNDLE=/pathtodownloadedfile/cacert.pem

Related

SSL: CERTIFICATE_VERIFY_FAILED in python while installing package [duplicate]

I get the following error when trying to install Punkt for nltk:
nltk.download('punkt')
[nltk_data] Error loading Punkt: <urlopen error [SSL:
[nltk_data] CERTIFICATE_VERIFY_FAILED] certificate verify failed
[nltk_data] (_ssl.c:590)>
False
TLDR: Here is a better solution: https://github.com/gunthercox/ChatterBot/issues/930#issuecomment-322111087
Note that when you run nltk.download(), a window will pop up and let you select which packages to download (Download is not automatically started right away).
To complement the accepted answer, the following is a complete list of directories that will be searched on Mac (not limited to the one mentioned in the accepted answer):
- '/Users/YOUR_USERNAME/nltk_data'
- '/usr/share/nltk_data'
- '/usr/local/share/nltk_data'
- '/usr/lib/nltk_data'
- '/usr/local/lib/nltk_data'
- '/Users/YOUR_USERNAME/YOUR_VIRTUAL_ENV_DIRECTORY/nltk_data'
- '/Users/YOUR_USERNAME/YOUR_VIRTUAL_ENV_DIRECTORY/share/nltk_data'
- '/Users/YOUR_USERNAME/YOUR_VIRTUAL_ENV_DIRECTORY/lib/nltk_data'
In case the link above dies, here is the solution pasted in its entirety:
import nltk
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
pass
else:
ssl._create_default_https_context = _create_unverified_https_context
nltk.download()
Run the above code in your favourite Python IDE or via the command line.
This works by disabling SSL check!
import nltk
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
pass
else:
ssl._create_default_https_context = _create_unverified_https_context
nltk.download()
Run the Python interpreter and type the commands:
import nltk
nltk.download()
from here: http://www.nltk.org/data.html
if you get an SSL/Certificate error, run the following command
bash /Applications/Python 3.6/Install Certificates.command
from here: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)
Search 'Install Certificates.command' in the finder and open it.
Then do the following steps in the terminal:
python3
import nltk
nltk.download()
The downloader script is broken. As a temporal workaround can manually download the punkt tokenizer from here and then place the unzipped folder in the corresponding location. The default folders for each OS are:
Windows: C:\nltk_data\tokenizers
OSX: /usr/local/share/nltk_data/tokenizers
Unix: /usr/share/nltk_data/tokenizers
This is how I solved it for MAC OS.
Initially after installing nltk, I was getting the SSL error.
Solution:
Goto
cd /Applications/Python\ 3.8
Run the command
./Install\ Certificates.command
Now if you try again, it should work!
Thanks a lot to this article!
You just need to Install the certificate doing this simple step
In the python application folder double-click on the file 'Certificates.command'
this will make a prompt window show in your screen and basically will automatically install the certificate for you, close this window and try again.
My solution is:
Download punkt.zip from here and unzip
Create nltk_data/tokenizers folders under home folder
Put punkt folder under tokenizers folder
There is a very simple way to fix all of this as written in the formal bug report for anyone else coming across this problem recently (e.g. 2019) and using MacOS. From the bug report at https://bugs.python.org/issue28150:
...there is a simple double-clickable or command-line-runnable script ("/Applications/Python 3.6/Install Certificates.command") that does two things: 1. uses pip to install certifi and 2. creates a symlink in the OpenSSL directory to certifi's installed bundle location.
Simply running the "Install Certificates.command" script worked for me on MacOS (10.15 beta as of this writing) and I was off and running.
My solution after nothing worked. I navigated, via the GUI to the Python 3.7 folder, opened the 'Certificates.command' file in terminal and the SSL issue was immediately resolved.
A bit late to the party but I just entered Certificates.command into Spotlight which found it and ran it. All fixed in seconds.
I'm running mac Catalina and using python 3.7 installed by Homebrew
It means that you are not using HTTPS to work consistently with other run time dependencies for Python etc.
If you are using Linux (Ubuntu)
~$ sudo apt-get install ca-certificates
Should solve the issue.
If you are using this in a script with a docker file, you have to make sure you have install the the ca-certificates modules in your docker file.
For mac users,
just copy paste the following in the terminal:
/Applications/Python\ 3.10/Install\ Certificates.command ; exit;
First go to the path /Applications/Python 3.6/ and run
Install Certificates.command
You will admin rights for the same.
If you are unable to download it, then as other answer suggest you can download directly and place it. You need to place them in the following directory structure.
> nltk_data
> corpora
> brown
> conll2000
> movie_reviews
> wordnet
> taggers
> averaged_perceptron_tagger
> tokenizers
> punkt
Updating the python certificates worked for me.
At the top of your script, keep:
import nltk
nltk.download('punkt')
In a separate terminal run (Mac):
bash /Applications/Python <version>/Install Certificates.command
For me, the solution was much simpler: I was still connected to my corporate network/VPN which blocks certain types of downloads. Switching the network made the SSL error disappear.

No module named 'stix2' [duplicate]

After installing mechanize, I don't seem to be able to import it.
I have tried installing from pip, easy_install, and via python setup.py install from this repo: https://github.com/abielr/mechanize. All of this to no avail, as each time I enter my Python interactive I get:
Python 2.7.3 (default, Aug 1 2012, 05:14:39)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mechanize
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named mechanize
>>>
The installations I ran previously reported that they had completed successfully, so I expect the import to work. What could be causing this error?
In my case, it is permission problem. The package was somehow installed with root rw permission only, other user just cannot rw to it!
I had the same problem: script with import colorama was throwing an ImportError, but sudo pip install colorama was telling me "package already installed".
My fix: run pip without sudo: pip install colorama. Then pip agreed it needed to be installed, installed it, and my script ran. Or even better, use python -m pip install <package>. The benefit of this is, since you are executing the specific version of python that you want the package in, pip will unequivocally install the package into the "right" python. Again, don't use sudo in this case... then you get the package in the right place, but possibly with (unwanted) root permissions.
My environment is Ubuntu 14.04 32-bit; I think I saw this before and after I activated my virtualenv.
I was able to correct this issue with a combined approach. First, I followed Chris' advice, opened a command line and typed 'pip show packagename'
This provided the location of the installed package.
Next, I opened python and typed 'import sys', then 'sys.path' to show where my python searches for any packages I import. Alas, the location shown in the first step was NOT in the list.
Final step, I typed 'sys.path.append('package_location_seen_in_step_1'). You optionally can repeat step two to see the location is now in the list.
Test step, try to import the package again... it works.
The downside? It is temporary, and you need to add it to the list each time.
It's the python path problem.
In my case, I have python installed in:
/Library/Frameworks/Python.framework/Versions/2.6/bin/python,
and there is no site-packages directory within the python2.6.
The package(SOAPpy) I installed by pip is located
/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
And site-package is not in the python path, all I did is add site-packages to PYTHONPATH permanently.
Open up Terminal
Type open .bash_profile
In the text file that pops up, add this line at the end:
export PYTHONPATH=$PYTHONPATH:/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/
Save the file, restart the Terminal, and you're done
The Python import mechanism works, really, so, either:
Your PYTHONPATH is wrong,
Your library is not installed where you think it is
You have another library with the same name masking this one
I have been banging my head against my monitor on this until a young-hip intern told me the secret is to "python setup.py install" inside the module directory.
For some reason, running the setup from there makes it just work.
To be clear, if your module's name is "foo":
[burnc7 (2016-06-21 15:28:49) git]# ls -l
total 1
drwxr-xr-x 7 root root 118 Jun 21 15:22 foo
[burnc7 (2016-06-21 15:28:51) git]# cd foo
[burnc7 (2016-06-21 15:28:53) foo]# ls -l
total 2
drwxr-xr-x 2 root root 93 Jun 21 15:23 foo
-rw-r--r-- 1 root root 416 May 31 12:26 setup.py
[burnc7 (2016-06-21 15:28:54) foo]# python setup.py install
<--snip-->
If you try to run setup.py from any other directory by calling out its path, you end up with a borked install.
DOES NOT WORK:
python /root/foo/setup.py install
DOES WORK:
cd /root/foo
python setup.py install
I encountered this while trying to use keyring which I installed via sudo pip install keyring. As mentioned in the other answers, it's a permissions issue in my case.
What worked for me:
Uninstalled keyring:
sudo pip uninstall keyring
I used sudo's -H option and reinstalled keyring:
sudo -H pip install keyring
In PyCharm, I fixed this issue by changing the project interpreter path.
File -> Settings -> Project -> Project Interpreter
File -> Invalidate Caches… may be required afterwards.
I couldn't get my PYTHONPATH to work properly. I realized adding export fixed the issue:
(did work)
export PYTHONPATH=$PYTHONPATH:~/test/site-packages
vs.
(did not work)
PYTHONPATH=$PYTHONPATH:~/test/site-packages
This problem can also occur with a relocated virtual environment (venv).
I had a project with a venv set up inside the root directory. Later I created a new user and decided to move the project to this user. Instead of moving only the source files and installing the dependencies freshly, I moved the entire project along with the venv folder to the new user.
After that, the dependencies that I installed were getting added to the global site-packages folder instead of the one inside the venv, so the code running inside this env was not able to access those dependencies.
To solve this problem, just remove the venv folder and recreate it again, like so:
$ deactivate
$ rm -rf venv
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -r requirements.txt
Something that worked for me was:
python -m pip install -user {package name}
The command does not require sudo. This was tested on OSX Mojave.
In my case I had run pip install Django==1.11 and it would not import from the python interpreter.
Browsing through pip's commands I found pip show which looked like this:
> pip show Django
Name: Django
Version: 1.11
...
Location: /usr/lib/python3.4/site-packages
...
Notice the location says '3.4'. I found that the python-command was linked to python2.7
/usr/bin> ls -l python
lrwxrwxrwx 1 root root 9 Mar 14 15:48 python -> python2.7
Right next to that I found a link called python3 so I used that. You could also change the link to python3.4. That would fix it, too.
In my case it was a problem with a missing init.py file in the module, that I wanted to import in a Python 2.7 environment.
Python 3.3+ has Implicit Namespace Packages that allow it to create a packages without an init.py file.
Had this problem too.. the package was installed on Python 3.8.0 but VS Code was running my script using an older version (3.4)
fix in terminal:
py .py
Make sure you're installing the package on the right Python Version
I had colorama installed via pip and I was getting "ImportError: No module named colorama"
So I searched with "find", found the absolute path and added it in the script like this:
import sys
sys.path.append("/usr/local/lib/python3.8/dist-packages/")
import colorama
And it worked.
I had just the same problem, and updating setuptools helped:
python3 -m pip install --upgrade pip setuptools wheel
After that, reinstall the package, and it should work fine :)
The thing is, the package is built incorrectly if setuptools is old.
If the other answers mentioned do not work for you, try deleting your pip cache and reinstalling the package. My machine runs Ubuntu14.04 and it was located under ~/.cache/pip. Deleting this folder did the trick for me.
Also, make sure that you do not confuse pip3 with pip. What I found was that package installed with pip was not working with python3 and vice-versa.
I had similar problem (on Windows) and the root cause in my case was ANTIVIRUS software! It has "Auto-Containment" feature, that wraps running process with some kind of a virtual machine.
Symptoms are: pip install somemodule works fine in one cmd-line window and import somemodule fails when executed from another process with the error
ModuleNotFoundError: No module named 'somemodule'
In my case (an Ubuntu 20.04 VM on WIN10 Host), I have a disordered situation with many version of Python installed and variuos point of Shared Library (installed with pip in many points of the File System). I'm referring to 3.8.10 Python version.
After many tests, I've found a suggestion searching with google (but' I'm sorry, I haven't the link). This is what I've done to resolve the problem :
From shell session on Ubuntu 20.04 VM, (inside the Home, in my case /home/hduser), I've started a Jupyter Notebook session with the command "jupyter notebook".
Then, when jupyter was running I've opened a .ipynb file to give commands.
First : pip list --> give me the list of packages installed, and, sympy
wasn't present (although I had installed it with "sudo pip install sympy"
command.
Last with the command !pip3 install sympy (inside jupyter notebook
session) I've solved the problem, here the screen-shot :
Now, with !pip list the package "sympy" is present, and working :
In my case, I assumed a package was installed because it showed up in the output of pip freeze. However, just the site-packages/*.dist-info folder is enough for pip to list it as installed despite missing the actual package contents (perhaps from an accidental deletion). This happens even when all the path settings are correct, and if you try pip install <pkg> it will say "requirement already satisfied".
The solution is to manually remove the dist-info folder so that pip realizes the package contents are missing. Then, doing a fresh install should re-populate anything that was accidentally removed
When you install via easy_install or pip, is it completing successfully? What is the full output? Which python installation are you using? You may need to use sudo before your installation command, if you are installing modules to a system directory (if you are using the system python installation, perhaps). There's not a lot of useful information in your question to go off of, but some tools that will probably help include:
echo $PYTHONPATH and/or echo $PATH: when importing modules, Python searches one of these environment variables (lists of directories, : delimited) for the module you want. Importing problems are often due to the right directory being absent from these lists
which python, which pip, or which easy_install: these will tell you the location of each executable. It may help to know.
Use virtualenv, like #JesseBriggs suggests. It works very well with pip to help you isolate and manage the modules and environment for separate Python projects.
I had this exact problem, but none of the answers above worked. It drove me crazy until I noticed that sys.path was different after I had imported from the parent project. It turned out that I had used importlib to write a little function in order to import a file not in the project hierarchy. Bad idea: I forgot that I had done this. Even worse, the import process mucked with the sys.path--and left it that way. Very bad idea.
The solution was to stop that, and simply put the file I needed to import into the project. Another approach would have been to put the file into its own project, as it needs to be rebuilt from time to time, and the rebuild may or may not coincide with the rebuild of the main project.
I had this problem with 2.7 and 3.5 installed on my system trying to test a telegram bot with Python-Telegram-Bot.
I couldn't get it to work after installing with pip and pip3, with sudo or without. I always got:
Traceback (most recent call last):
File "telegram.py", line 2, in <module>
from telegram.ext import Updater
File "$USER/telegram.py", line 2, in <module>
from telegram.ext import Updater
ImportError: No module named 'telegram.ext'; 'telegram' is not a package
Reading the error message correctly tells me that python is looking in the current directory for a telegram.py. And right, I had a script lying there called telegram.py and this was loaded by python when I called import.
Conclusion, make sure you don't have any package.py in your current working dir when trying to import. (And read error message thoroughly).
I had a similar problem using Django. In my case, I could import the module from the Django shell, but not from a .py which imported the module.
The problem was that I was running the Django server (therefore, executing the .py) from a different virtualenv from which the module had been installed.
Instead, the shell instance was being run in the correct virtualenv. Hence, why it worked.
This Works!!!
This often happens when module is installed to an older version of python or another directory, no worries as solution is simple.
- import module from directory in which module is installed.
You can do this by first importing the python sys module then importing from the path in which the module is installed
import sys
sys.path.append("directory in which module is installed")
import <module_name>
Most of the possible cases have been already covered in solutions, just sharing my case, it happened to me that I installed a package in one environment (e.g. X) and I was importing the package in another environment (e.g. Y). So, always make sure that you're importing the package from the environment in which you installed the package.
For me it was ensuring the version of the module aligned with the version of Python I was using.. I built the image on a box with Python 3.6 and then injected into a Docker image that happened to have 3.7 installed, and then banging my head when Python was telling me the module wasn't installed...
36m for Python 3.6
bsonnumpy.cpython-36m-x86_64-linux-gnu.so
37m for Python 3.7 bsonnumpy.cpython-37m-x86_64-linux-gnu.so
I know this is a super old post but for me, I had an issue with a 32 bit python and 64 bit python installed. Once I uninstalled the 32 bit python, everything worked as it should.
I have solved my issue that same libraries were working fine in one project(A) but importing those same libraries in another project(B) caused error. I am using Pycharm as IDE at Windows OS.
So, after trying many potential solutions and failing to solve the issue, I did these two things (deleted "Venv" folder, and reconfigured interpreter):
1-In project(B), there was a folder named("venv"), located in External Libraries/. I deleted that folder.
2-Step 1 (deleting "venv" folder) causes error in Python Interpreter Configuration, and
there is a message shown at top of screen saying "Invalid python interpreter selected
for the project" and "configure python interpreter", select that link and it opens a
new window. There in "Project Interpreter" drop-down list, there is a Red colored line
showing previous invalid interpreter. Now, Open this list and select the Python
Interpreter(in my case, it is Python 3.7). Press "Apply" and "OK" at the bottom and you
are good to go.
Note: It was potentially the issue where Virtual Environment of my Project(B) was not recognizing the already installed and working libraries.

How to add a custom CA Root certificate to the CA Store used by pip in Windows?

I just installed Python3 from python.org and am having trouble installing packages with pip. By design, there is a man-in-the-middle packet inspection appliance on the network here that inspects all packets (ssl included) by resigning all ssl connections with its own certificate. Part of the GPO pushes the custom root certificate into the Windows Keystore.
When using Java, if I need to access any external https sites, I need to manually update the cacerts in the JVM to trust the Self-Signed CA certificate.
How do I accomplish that for python? Right now, when I try to install packages using pip, understandably, I get wonderful [SSL: CERTIFICATE_VERIFY_FAILED] errors.
I realize I can ignore them using the --trusted-host parameter, but I don't want to do that for every package I'm trying to install.
Is there a way to update the CA Certificate store that python uses?
Self-Signed Certificate Authorities pip / conda
After extensively documenting a similar problem with Git (How can I make git accept a self signed certificate?), here we are again behind a corporate firewall with a proxy giving us a MitM "attack" that we should trust and:
NEVER disable all SSL verification!
This creates a bad security culture. Don't be that person.
UPDATE 2022 Python 3.10
Can experimentally use the system truststore.
https://pip.pypa.io/en/latest/user_guide/#using-system-trust-stores-for-verifying-https
# Requires Python 3.10 or later
python --version
Python 3.10.4
# Install the 'truststore' package from PyPI
python -m pip install truststore
# Use '--use-feature=truststore' flag to enable
python -m pip install SomePackage --use-feature=truststore
tl;dr: Configuring your own trust store CA bundle
pip config set global.cert path/to/ca-bundle.crt
pip config list
conda config --set ssl_verify path/to/ca-bundle.crt
conda config --show ssl_verify
# Bonus while we are here...
git config --global http.sslVerify true
git config --global http.sslCAInfo path/to/ca-bundle.crt
But where do we get ca-bundle.crt?
Get an up to date CA Bundle
cURL publishes an extract of the Certificate Authorities bundled with Mozilla Firefox
https://curl.haxx.se/docs/caextract.html
Direct Download
SHA256
I recommend you open up this cacert.pem file in a text editor as we will need to add our self-signed CA to this file.
Certificates are a document complying with X.509 but they can be encoded to disk a few ways. The below article is a good read but the short version is that we are dealing with the base64 encoding which is often called PEM in the file extensions. You will see it has the format:
----BEGIN CERTIFICATE----
....
base64 encoded binary data
....
----END CERTIFICATE----
https://support.ssl.com/Knowledgebase/Article/View/19/0/der-vs-crt-vs-cer-vs-pem-certificates-and-how-to-convert-them
Getting our Self Signed Certificate
Below are a few options on how to get our self signed certificate:
Via OpenSSL CLI
Via Browser
Via Python Scripting
Get our Self-Signed Certificate by OpenSSL CLI
https://unix.stackexchange.com/questions/451207/how-to-trust-self-signed-certificate-in-curl-command-line/468360#468360
echo quit | openssl s_client -showcerts -servername "curl.haxx.se" -connect curl.haxx.se:443 > cacert.pem
Get our Self-Signed Certificate Authority via Browser
Acquiring your CA: https://stackoverflow.com/a/50486128/622276
http://blog.majcica.com/2016/12/27/installing-self-signed-certificates-into-git-cert-store/
Thanks to this answer and the linked blog, it shows steps (on Windows) how to view the certificate and then copy to file using the base64 PEM encoding option.
Copy the contents of this exported file and paste it at the end of your cacerts.pem file.
For consistency rename this file cacerts.pem --> ca-bundle.crt and place it somewhere easy like:
# Windows
%USERPROFILE%\certs\ca-bundle.crt
# Linux/macOS
$HOME/certs/cabundle.crt
Get our Self-Signed Certificate Authority via Python
Thanks to all the brilliant answers in:
How to get response SSL certificate from requests in python?
I have put together the following to attempt to take it a step further.
https://github.com/neozenith/get-ca-py
Finally
Set the configuration in pip and conda so that it knows where this CA store resides with our extra self-signed CA.
# Windows
pip config set global.cert %USERPROFILE%\certs\ca-bundle.crt
conda config --set ssl_verify %USERPROFILE%\certs\ca-bundle.crt
OR
# Linux / macOS
pip config set global.cert $HOME/certs/ca-bundle.crt
conda config --set ssl_verify $HOME/certs/ca-bundle.crt
THEN
pip config list
conda config --show ssl_verify
# Hot tip: use -v to show where your pip config file is...
pip config list -v
# Example output for macOS and homebrew installed python
For variant 'global', will try loading '/Library/Application Support/pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.pip/pip.conf'
For variant 'user', will try loading '/Users/jpeak/.config/pip/pip.conf'
For variant 'site', will try loading '/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/Versions/3.7/pip.conf'
Troubleshooting
Based on a great comment below
I've tried this and still get a SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)')) error. Any suggestions?
here is a troubleshooting guide:
This is the normal error message when the certificates authorities are not yet correctly setup.
It could be a variety of factors to check:
The paths to your ca-bundle.crt have the right path separators for your OS (it has stung me),
you may not have the latest CAs to verify normal certificates,
you may not have added your CA in the right encoding.
Python is effectively doing those 3 steps:
Find my CA store,
read all entries,
look up this certificate against my trust store.
If any of those fail you get this error message from experience.
Check this answer linked from below to display and check your ssl_cert_dir using:
python -c "import ssl; print(ssl.get_default_verify_paths())"
References
Pip SSL: https://pip.pypa.io/en/stable/user_guide/#configuration
Conda SSL: https://stackoverflow.com/a/35804869/622276
Acquiring your CA: https://stackoverflow.com/a/50486128/622276
http://blog.majcica.com/2016/12/27/installing-self-signed-certificates-into-git-cert-store/
Using Python to automatically grab your Peer CA: How to get response SSL certificate from requests in python?
Run: python -c "import ssl; print(ssl.get_default_verify_paths())" to check the current paths which are used to verify the certificate. Add your company's root certificate to one of those.
The path openssl_capath_env points to the environment variable: SSL_CERT_DIR.
If SSL_CERT_DIR doesn't exist, you will need to create it and point it to a valid folder within your filesystem. You can then add your certificate to this folder to use it.
Alternative solution on Windows is to install python-certifi-win32 that will allow Python to use Windows Certificate Store.
pip install python-certifi-win32
Not best answer but you can reuse an already created ca bundle using --cert option of pip, for instance:
pip install SQLAlchemy==1.1.15 --cert="C:\Users\myUser\certificates\my_ca-bundle.crt"
On Windows, I solved it by creating a pip.ini file in %APPDATA%\pip\
e.g. C:\Users\asmith\AppData\Roaming\pip\pip.ini
In the pip.ini I put the path to my certificate:
[global]
cert=C:\Users\asmith\SSL\teco-ca.crt
https://pip.pypa.io/en/stable/user_guide/#configuration has more information about the configuration file.
I think nt86's solution is the most appropriate because it leverages the underlying Windows infrastructure (certificate store).
But it doesn't explain how to install python-certifi-win32 to start with since pip is non functional.
The trick is to use --trustedhost to install python-certifi-win32 and then after that, pip will automatically use the windows certificate store to load the certificate used by the proxy.
So in a nutshell, you should do:
pip install python-certifi-win32 -trustedhost pypi.org
and after that you should be good to go
In my scenario, on Ubuntu using python 3.8, I wanted to see what I could change in my local system (rather than code or a installing a new package) that would allow me to utilize my custom CA and my self-signed certs.
So I just traced where python was looking:
strace python my_prog 2> out.txt
Then grepped the output for pem (you may need to look for "crt" "cert")
grep -i pem out.txt
Even though certifi wasn't ever explicitely installed, the file it was looking for was:
/path/to/virtualenv/for/project/lib/python3.8/site-packages/certifi/cacert.pem
So I renamed the old file and linked it to my system's ca bundle (e.g., /etc/ssl/certs/ca-certificates.crt)
And now it worked.
Open Anaconda Navigator.
Go to File\Preferences.
Enable SSL verification Disable (not recommended)
or Enable and indicate SSL certificate path(Optional)
Update a package to a specific version:
Select Install on Top-Right
Select package click on tick
Mark for update
Mark for specific version installation
Click Apply

ImportError: No module named _ssl

Ubuntu Maverick w/Python 2.7:
I can't figure out what to do to resolve the following import error:
>>> import ssl
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/ssl.py", line 60, in <module>
import _ssl # if we can't import it, let the error propagate
ImportError: No module named _ssl
UPDATE:
I recompiled the source. I was unable to figure out how to add the --with-ssl option the answers below mention, instead I got this to work by editing the lines regarding SSL in /Modules/Setup.dist.
Unrelated to the original question, but because this is the first Google result... I hit this on Google AppEngine and had to add:
libraries:
- name: ssl
version: latest
to app.yaml per: https://cloud.google.com/appengine/docs/python/sockets/ssl_support
Please NOTE: This seems to work upto Python version 2.7.9 but not for 2.7.10 or 2.7.11.
Did you build the Python from source? If so, you need the --with-ssl option while building.
Since --with-ssl is not recognized anymore I just installed the libssl-dev.
For debian based systems:
sudo apt-get install libssl-dev
For CentOS and RHEL
sudo yum install openssl-devel
To restart the make first clean up by:
make clean
Then start again and execute the following commands one after the other:
./configure
make
make test
make install
For further information on OpenSSL visit the Ubuntu Help Page on OpenSSL.
If you built Python from source, this is just a matter of dependencies: since you miss OpenSSL lib installed, python silently fails installing the _ssl module. You can see it in the final report of the make command:
Python build finished, but the necessary bits to build these modules were not found:
_bsddb _sqlite3 _ssl
_tkinter bsddb185 dbm
dl gdbm imageop
sunaudiodev
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
Installing OpenSSL lib in any of the standard lib paths (/usr/lib, /usr/local/lib...) should do the trick. Anyway this is how I did :-)
I had exactly the same problem. I fixed it without rebuilding python, as follows:
Find another server with the same architecture (i386 or x86_64) and the same python version (example: 2.7.5). Yes, this is the hard part. You can try installing python from sources into another server if you can't find any server with the same python version.
In this another server, check if import ssl works. It should work.
If it works, then try to find the _ssl lilbrary as follows:
[root#myserver]# find / -iname _ssl.so
/usr/local/python27/lib/python2.7/lib-dynload/_ssl.so
Copy this file into the original server. Use the same destination folder:
/usr/local/python27/lib/python2.7/lib-dynload/
Double check owner and permissions:
[root#myserver]# chown root:root _ssl.so
[root#myserver]# chmod 755 _ssl.so
Now you should be able to import ssl.
This worked for me in a CentOS 6.3 x86_64 environment with python 2.7.3. Also I had python 2.6.6 installed, but with ssl working fine.
The underscore usually means a C module (i.e. DLL), and Python can't find it. Did you build python yourself? If so, you need to include SSL support.
I am writing this solution for those who are still facing such issue and cant find the solution.
in my case, I am using
shared hosting (Cpanel Access) Linux CentOS.
I was facing this issue
No module named '_ssl'
I tried for all possible solutions but as you know sometimes things don't work for you and in hosting you don't have access to fully root and run queries.
even my hosting provider did for me.. but NO GOOD RESULT.
so how I solved if you are using shared hosting and you have deployed your Django App using
Setup Python App
You only have to downgrade your Python Version, I downgraded from
Python 3.7.3
(As Python 3.7 does not have SSL module in it)
To
Python 3.6.8
through Setup Python App.
Hope it will be helpful for someone with the same issue,
Either install the supplementary packages for python-ssl using your package manager or
recompile Python using -with-ssl (requires OpenSSL headers/libs installed).
On Solaris 11, I had to modify setup.py to include /opt/csw/include/openssl in the SSL include search path.
Uwe

How do I install GASP for Python 2.6.2 on a Mac

I'm currently trying to learn Python and am going through How to Think Like a Computer Scientist: Learning With Python. I have installed Python 2.6.2 on Mac OSX 10.4.11 and am using the IDLE.
At the end of chapter 4 Elkner et al. refer to GASP. However their instructions don't work as when I enter:
>>> from gasp import*
I get:
Traceback (most recent call last):
File "<pyshell#17>", line 1, in <module>
from gasp import*
ImportError: No module named gasp
I've had a look around on google and can only find outdated methods of installation or pages of gobbledegook. I believe I have to install PyObjC first and haven't been able to accomplish this either.
Can anyone please help me out with some plain English instructions?
Well everybody, sorry for the incomplete sentences and overall poor English but I wanted to make this simple to read and understand for someone who is completely inexperienced in any sort of programming, as I am (very first day messing with this stuff, e.g., terminal). This is the result of hours of Googling that was all done in one day. Perhaps someone who is familiar with the commands below (in bold) wouldn’t mind explaining what exactly is taking place. Additionally, this was all done in terminal on a MacBook Pro running Mac OS Lion.
Install macport binary (comes with installer; easy)
sudo port install py-game
not sure if this is necessary, as it doesn’t appear to cause pygame to be functional for python version 2.7.1 (stock python on lion)
sudo port select --set python python 2.7
I believe this set the default python version to 2.7.2 which I also believe was
downloaded during step 2 (therefore why I think this ends up being a necessary step)
Download setuptools-0.6c11-py2.7.tar
In folder gasp-0.3.4, which appears after clicking on the .tar, place setup.py in the
gasp folder
sudo python gasp/setup.py install
make sure your directory is the folder gasp-0.3.4
sudo port –v install py27-pygtk
takes about an hour for this step to complete
sudo port uninstall py-game
this step is not necessary for gasp to work; I simply didn’t want any unnecessary
stuff on my computer that was downloaded during the second step; however, this step
put python 2.7.2 on my computer; could install 2.7.2 separately I guess but this way
worked for me; a lot of other unnecessary stuff is installed during this step too but
I think it’ll remain even after this command, oh well
This is actually somewhat of a coincidence; I'm one of the packagers of GASP. On our download page, which is linked by our main project page, there are instructions on how to install it on most major platforms. Hadn't considered OSX, however. Will write something up shortly.
Essentially, install the Official Python from the PSF. Then add MacPorts and run
sudo ports install py-game
Extract the source tarball from the download page linked above to your site-packages directory.
There are also some alternative instructions I found that might work better, as I have not tested the above.
This is an interesting problem faced by most of the readers using "How to Think Like a Computer Scientist : Learning with Python", when they reach 4th chapter.
Now to install GASP, you need to check whether you have python installed on your machine.
Assumption: I am going to assume that you are using Mac.
Type the following command on your terminal,
`$ python -V`
{If your system has python installed on it, you will get an answer like Python 2.7.1 ( if the version is 2.7.1).}
Once you have python available on your system, you should install PyObjC.
An easy way out is to type on to your terminal prompt,
`$ easy_install pyobjc==2.2`
{this will install the version 2.2}
Next step is to install pygame package , you can do this in two ways either by downloading the .dmg file or using "homebrew". I prefer the second method.
to install pygame package using "brew" you have to install mercurial first. It's a simple procedure, just type in
$ brew install mercurial
Then type in the following commands,
$ brew install sdl sdl_image sdl_mixer sdl_ttf smpeg portmidi
$ sudo pip install hg+http://bitbucket.org/pygame/pygame
Next step is to install GASP. Download gasp from https://launchpad.net/gasp-core/+download
Extract the .tar file, you will get a folder structure. Our aim is to copy the folder named "gasp" to the Systems Library folder. To check which folder or version of python is used and to know their correct path type in the following command on terminal.
$ python -c 'import sys, pprint; pprint.pprint(sys.path)'
generally it will print a path similar to '/Library/Python/2.7/site-packages' , you can notice this either as the last line or the second last line of the output that above commands generate.
Copy the gasp folder to the site-packages folder,
$ sudo cp -R ~/Desktop/python-gasp-0.1.1/gasp/ /Library/Python/2.7/site-packages/gasp
This should copy all the required files to the location specified as the second argument.
Now go to terminal and type
$ python
>>> import gasp
If everything goes fine, you will not get any error or any messages.
PS: Ensure that in your site-packages directory there are no duplicate copies of pygame/gasp. In case of duplicates, it may throw lots of tantrums. Also, if any of the steps go wrong you may get error "import cairo" some 25th or 26th line on base.py in gasp package.In that event, please clean up your site packages directory by removing pygame and gasp and re-install them, that should solve it.
Also while installing mercurial you may get some warning related to Certificates. You can solve them by typing in following commands,
$ openssl req -new -x509 -extensions v3_ca -keyout /dev/null -out dummycert.pem -days 3650
$ sudo cp dummycert.pem /etc/hg-dummy-cert.pem
$ cd ~
$ nano .hgrc
{ This will open up an empty .hgrc file }
Type in the following
[web]
cacerts = /etc/hg-dummy-cert.pem
save the above 2 lines by pressing ctrl+ o
exit nano by pressing ctrl + x
Thats it. This should solve your problems with GASP installation and enjoy the book " How to Think Like a Computer Scientist" it's a wonderful introduction to the world of computing.

Categories