We Keep Coding

Python-How to parse online pdf file while the url doesn't have .pdf extension [duplicate]

I am getting the following error:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/Matthew/Desktop/Skypebot 2.0/bot.py", line 271, in process
info = urllib2.urlopen(req).read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
This is the code that is causing this error:
if input.startswith("!web"):
input = input.replace("!web ", "")
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXX' })
info = urllib2.urlopen(req).read()
Message.Chat.SendMessage ("" + info)
The API I'm using requires me to use HTTPS. How can I make it bypass the verification?

This isn't a solution to your specific problem, but I'm putting it here because this thread is the top Google result for "SSL: CERTIFICATE_VERIFY_FAILED", and it lead me on a wild goose chase.
If you have installed Python 3.6 on OSX and are getting the "SSL: CERTIFICATE_VERIFY_FAILED" error when trying to connect to an https:// site, it's probably because Python 3.6 on OSX has no certificates at all, and can't validate any SSL connections. This is a change for 3.6 on OSX, and requires a post-install step, which installs the certifi package of certificates. This is documented in the file ReadMe.rtf, which you can find at /Applications/Python\ 3.6/ReadMe.rtf (see also the file Conclusion.rtf, and the script build-installer.py that generates the macOS installer).
The ReadMe will have you run the post-install script at
/Applications/Python\ 3.10/Install\ Certificates.command (Terminal App, this command alone should, fix the issue. Be sure to update the file path using your current subversion.)
(its source is install_certificates.command), which:
first installs the Python package certifi, and
then creates a symbolic link from the OpenSSL certificates file to the certificates file installed by the package certifi.
Release notes have some more info: https://www.python.org/downloads/release/python-360/
On newer versions of Python, there is more documentation about this:
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/ReadMe.rtf#L22-L34
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Conclusion.rtf#L15-L19
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Welcome.rtf#L23-L25
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/install_certificates.command
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/README.rst
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/build-installer.py#L239-L246

If you just want to bypass verification, you can create a new SSLContext. By default newly created contexts use CERT_NONE.
Be careful with this as stated in section 17.3.7.2.1
When calling the SSLContext constructor directly, CERT_NONE is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you’re talking to. Therefore, when in client mode, it is highly recommended to use CERT_REQUIRED.
But if you just want it to work now for some other reason you can do the following, you'll have to import ssl as well:
input = input.replace("!web ", "")
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' })
gcontext = ssl.SSLContext() # Only for gangstars
info = urllib2.urlopen(req, context=gcontext).read()
Message.Chat.SendMessage ("" + info)
This should get round your problem but you're not really solving any of the issues, but you won't see the [SSL: CERTIFICATE_VERIFY_FAILED] because you now aren't verifying the cert!
To add to the above, if you want to know more about why you are seeing these issues you will want to have a look at PEP 476.
This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python's HTTP clients by default, subject to opt-out on a per-call basis. This change would be applied to Python 2.7, Python 3.4, and Python 3.5.
There is an advised opt out which isn't dissimilar to my advice above:
import ssl
# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)
It also features a highly discouraged option via monkeypatching which you don't often see in python:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Which overrides the default function for context creation with the function to create an unverified context.
Please note with this as stated in the PEP:
This guidance is aimed primarily at system administrators that wish to adopt newer versions of Python that implement this PEP in legacy environments that do not yet support certificate verification on HTTPS connections. For example, an administrator may opt out by adding the monkeypatch above to sitecustomize.py in their Standard Operating Environment for Python. Applications and libraries SHOULD NOT be making this change process wide (except perhaps in response to a system administrator controlled configuration setting).
If you want to read a paper on why not validating certs is bad in software you can find it here!

To expand on Craig Glennie's answer:
in Python 3.6.1 on MacOs Sierra
Entering this in the bash terminal solved the problem:
pip install certifi
/Applications/Python\ 3.6/Install\ Certificates.command

On Windows, Python does not look at the system certificate, it uses its own located at ?\lib\site-packages\certifi\cacert.pem.
The solution to your problem:
download the domain validation certificate as *.crt or *pem file
open the file in editor and copy it's content to clipboard
find your cacert.pem location: from requests.utils import DEFAULT_CA_BUNDLE_PATH; print(DEFAULT_CA_BUNDLE_PATH)
edit the cacert.pem file and paste your domain validation certificate at the end of the file.
Save the file and enjoy requests!

I was having a similar problem, though I was using urllib.request.urlopen in Python 3.4, 3.5, and 3.6. (This is a portion of the Python 3 equivalent of urllib2, per the note at the head of Python 2's urllib2 documentation page.)
My solution was to pip install certifi to install certifi, which has:
... a carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.
Then, in my code where I previously just had:
import urllib.request as urlrq
resp = urlrq.urlopen('https://example.com/bar/baz.html')
I revised it to:
import urllib.request as urlrq
import certifi
resp = urlrq.urlopen('https://example.com/bar/baz.html', cafile=certifi.where())
If I read the urllib2.urlopen documentation correctly, it also has a cafile argument. So, urllib2.urlopen([...], certifi.where()) might work for Python 2.7 as well.
UPDATE (2020-01-01): As of Python 3.6, the cafile argument to urlopen has been deprecated, with the context argument supposed to be specified instead. I found the following to work equally well on 3.5 through 3.8:
import urllib.request as urlrq
import certifi
import ssl
resp = urlrq.urlopen('https://example.com/bar/baz.html', context=ssl.create_default_context(cafile=certifi.where()))

My solution for Mac OS X:
Upgrade to Python 3.6.5 using the native app Python installer downloaded from the official Python language website https://www.python.org/downloads/
I've found that this installer is taking care of updating the links and symlinks for the new Python a lot better than homebrew.
Install a new certificate using "./Install Certificates.command" which is in the refreshed Python 3.6 directory
cd "/Applications/Python 3.6/"
sudo "./Install Certificates.command"

You could try adding this to your environment variables:
PYTHONHTTPSVERIFY=0
Note that this will disable all HTTPS verification so is a bit of a sledgehammer approach, however if verification isn't required it may be an effective solution.

I had a similar problem on one of my Linux machines. Generating fresh certificates and exporting an environment variable pointing to the certificates directory fixed it for me:
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs

import requests
requests.packages.urllib3.disable_warnings()
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
Taken from here https://gist.github.com/michaelrice/a6794a017e349fc65d01

Like I've written in a comment, this problem is probably related to this SO answer.
In short: there are multiple ways to verify the certificate. The verification used by OpenSSL is incompatible with the trusted root certificates you have on your system. OpenSSL is used by Python.
You could try to get the missing certificate for Verisign Class 3 Public Primary Certification Authority and then use the cafile option according to the Python documentation:
urllib2.urlopen(req, cafile="verisign.pem")

$ cd $HOME
$ wget --quiet https://curl.haxx.se/ca/cacert.pem
$ export SSL_CERT_FILE=$HOME/cacert.pem
Source: https://access.redhat.com/articles/2039753

I think you have several ways for fix this issue. I mentioned 5 ways below:
You can define context for each request and pass the context on each request for use it like below:
import certifi
import ssl
import urllib
context = ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com', context=context)
OR Set certificate file in environment.
import os
import certifi
import urllib
os.environ["REQUESTS_CA_BUNDLE"] = certifi.where()
os.environ["SSL_CERT_FILE"] = certifi.where()
result = urllib.request.urlopen('https://www.example.com')
OR Replace create default https context method:
import certifi
import ssl
ssl._create_default_https_context = lambda: ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com')
OR If you use Linux machine, generating fresh certificates and exporting an environment variable pointing to the certificates directory fixed it.
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs
OR If you use Mac machine, generating fresh certificates
$ cd "/Applications/$(python3 --version | awk '{print $2}'| awk -F. '{print "Python " $1"."$2}')"
$ sudo "./Install Certificates.command"

Solution for Anaconda
My setup is Anaconda Python 3.7 on MacOS with a proxy. The paths are different.
This is how you get the correct certificates path:
import ssl
ssl.get_default_verify_paths()
which on my system produced
Out[35]: DefaultVerifyPaths(cafile='/miniconda3/ssl/cert.pem', capath=None,
openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/miniconda3/ssl/cert.pem',
openssl_capath_env='SSL_CERT_DIR', openssl_capath='/miniconda3/ssl/certs')
Once you know where the certificate goes, then you concatenate the certificate used by the proxy to the end of that file.
I had already set up conda to work with my proxy, by running:
conda config --set ssl_verify <pathToYourFile>.crt
If you don't remember where your cert is, you can find it in ~/.condarc:
ssl_verify: <pathToYourFile>.crt
Now concatenate that file to the end of /miniconda3/ssl/cert.pem
and requests should work, and in particular sklearn.datasets and similar tools
should work.
Further Caveats
The other solutions did not work because the Anaconda setup is slightly different:
The path Applications/Python\ 3.X simply doesn't exist.
The path provided by the commands below is the WRONG path
from requests.utils import DEFAULT_CA_BUNDLE_PATH
DEFAULT_CA_BUNDLE_PATH

I need to add another answer because just like Craig Glennie, I went on a wild goose chase due to the many posts referring to this problem across the Web.
I am using MacPorts, and what I originally thought was a Python problem was in fact a MacPorts problem: it does not install a root certificate with its installation of openssl. The solution is to port install curl-ca-bundle, as mentioned in this blog post.

Python 2.7.12 (default, Jul 29 2016, 15:26:22) fixed the mentioned issue. This information might help someone else.

I am surprised all these instruction didn't solved my problem. Nonetheless, the diagnostic is correct (BTW, I am using Mac and Python3.6.1). So, to summarize the correct part :
On Mac, Apple is dropping OpenSSL
Python now uses it own set of CA Root Certificate
Binary Python installation provided a script to install the CA Root certificate Python needs ("/Applications/Python 3.6/Install Certificates.command")
Read "/Applications/Python 3.6/ReadMe.rtf" for details
For me, the script doesn't work, and all those certifi and openssl installation failed to fix too. Maybe because I have multiple python 2 and 3 installations, as well as many virtualenv. At the end, I need to fix it by hand.
pip install certifi # for your virtualenv
mkdir -p /Library/Frameworks/Python.framework/Versions/3.6/etc/openssl
cp -a <your virtualenv>/site-package/certifi/cacert.pem \
/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem
If that still fails you. Then re/install OpenSSL as well.
port install openssl

I have found this over here
I found this solution, insert this code at the beginning of your source file:
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
This code makes the verification undone so that the ssl certification is not verified.

The SSL: CERTIFICATE_VERIFY_FAILED error could also occur because an Intermediate Certificate is missing in the ca-certificates package on Linux. For example, in my case the intermediate certificate "DigiCert SHA2 Secure Server CA" was missing in the ca-certificates package even though the Firefox browser includes it. You can find out which certificate is missing by directly running the wget command on the URL causing this error. Then you can search for the corresponding link to the CRT file for this certificate from the official website (e.g. https://www.digicert.com/digicert-root-certificates.htm in my case) of the Certificate Authority. Now, to include the certificate that is missing in your case, you may run the below commands using your CRT file download link instead:
wget https://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt
mv DigiCertSHA2SecureServerCA.crt DigiCertSHA2SecureServerCA.der
openssl x509 -inform DER -outform PEM -in DigiCertSHA2SecureServerCA.der -out DigicertSHA2SecureServerCA.pem.crt
sudo mkdir /usr/share/ca-certificates/extra
sudo cp DigicertSHA2SecureServerCA.pem.crt /usr/share/ca-certificates/extra/
sudo dpkg-reconfigure ca-certificates
After this you may test again with wget for your URL as well as by using the python urllib package. For more details, refer to: https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1795242

Take a look at
/Applications/Python 3.6/Install Certificates.command
You can also go to the Applications folder in Finder and click on Certificates.command

For Python 3.4+ on Centos 6/7,Fedora, just install the trusted CA this way :
Copy the CA.crt to /etc/pki/ca-trust/source/anchors/
update-ca-trust force-enable
update-ca-trust extract

Like you, I am using python 2.7 on my old iMac (OS X 10.6.8), I met the problem too, using urllib2.urlopen :
urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
My programs were running fine without SSL certificate problems and suddently (after dowloading programs), they crashed with this SSL error.
The problem was the version of python used :
No problem with https://www.python.org/downloads and python-2.7.9-macosx10.6.pkg
problem with the one instaled by Homebrew tool : "brew install python", version located in /usr/local/bin.
A chapter, called Certificate verification and OpenSSL [CHANGED for Python 2.7.9], in /Applications/Python 2.7/ReadMe.rtf explains the problem with many details.
So, check, download and put in your PATH the right version of python.

I hang my head in semi-shame, as I had the same issue, except that in my case, the URL I was hitting was valid, the certificate was valid. What wasn't valid was my connection out to the web. I had failed to add proxy details into the browser (IE in this case). This stopped the verification process from happening correctly.
Added in the proxy details and my python was then very happy .

Python 2.7 on Amazon EC2 with centOS 7
I had to set the env variable SSL_CERT_DIR to point to my ca-bundle which was located at /etc/ssl/certs/ca-bundle.crt

There are cases when you can not use insecure connections or pass ssl context into urllib request. Here my solution based on
https://stackoverflow.com/a/28052583/6709778
In a case if you want use your own cert file
import ssl
def new_ssl_context_decorator(*args, **kwargs):
kwargs['cafile'] = '/etc/ssl/certs/ca-certificates.crt'
return ssl.create_default_context(*args, **kwargs)
ssl._create_default_https_context = ssl._create_unverified_context
or you can use shared file from certifi
def new_ssl_context_decorator(*args, **kwargs):
import certifi
kwargs['cafile'] = certifi.where()
return ssl.create_default_context(*args, **kwargs)

Installing PyOpenSSL using pip worked for me (without converting to PEM):
pip install PyOpenSSL

I had this problem solved by closing Fiddler (an HTTP debugging proxy) check if you have a proxy enabled and try again.

In python 2.7 adding Trusted root CA details at the end in file C:\Python27\lib\site-packages\certifi\cacert.pem helped
after that i did run (using admin rights)
pip install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org packageName

Installing certifi on Mac solved my issue:
pip install certifi

In my case I was getting this error because requests and urllib3 versions were incompatible, giving the following error during installation:
ERROR: requests 2.21.0 has requirement urllib3<1.25,>=1.21.1, but you'll have urllib3 1.25 which is incompatible.
pip install 'urllib3<1.25' --force-reinstall
did the trick.

Another Anaconda solution. I was getting CERTIFICATE_VERIFY_FAILED in my Python 2.7 environment on macOS. It turns out the conda paths were bad:
base (3.7) environment:
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/usr/local/anaconda3/ssl/cert.pem', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/anaconda3/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/anaconda3/ssl/certs')
2.7 environment (paths did not exist!):
DefaultVerifyPaths(cafile='', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/anaconda3/envs/py27/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/anaconda3/envs/py27/ssl/certs')
The fix:
cd /usr/local/anaconda3/envs/py27/
mkdir ssl
cd ssl
ln -s ../../../ssl/cert.pem

urllib.requests.urlopen() causes CERTIFICATE_VERIFY_FAILED on MacOS [duplicate]

I am getting the following error:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/Matthew/Desktop/Skypebot 2.0/bot.py", line 271, in process
info = urllib2.urlopen(req).read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
This is the code that is causing this error:
if input.startswith("!web"):
input = input.replace("!web ", "")
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXX' })
info = urllib2.urlopen(req).read()
Message.Chat.SendMessage ("" + info)
The API I'm using requires me to use HTTPS. How can I make it bypass the verification?

This isn't a solution to your specific problem, but I'm putting it here because this thread is the top Google result for "SSL: CERTIFICATE_VERIFY_FAILED", and it lead me on a wild goose chase.
If you have installed Python 3.6 on OSX and are getting the "SSL: CERTIFICATE_VERIFY_FAILED" error when trying to connect to an https:// site, it's probably because Python 3.6 on OSX has no certificates at all, and can't validate any SSL connections. This is a change for 3.6 on OSX, and requires a post-install step, which installs the certifi package of certificates. This is documented in the file ReadMe.rtf, which you can find at /Applications/Python\ 3.6/ReadMe.rtf (see also the file Conclusion.rtf, and the script build-installer.py that generates the macOS installer).
The ReadMe will have you run the post-install script at
/Applications/Python\ 3.10/Install\ Certificates.command (Terminal App, this command alone should, fix the issue. Be sure to update the file path using your current subversion.)
(its source is install_certificates.command), which:
first installs the Python package certifi, and
then creates a symbolic link from the OpenSSL certificates file to the certificates file installed by the package certifi.
Release notes have some more info: https://www.python.org/downloads/release/python-360/
On newer versions of Python, there is more documentation about this:
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/ReadMe.rtf#L22-L34
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Conclusion.rtf#L15-L19
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Welcome.rtf#L23-L25
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/install_certificates.command
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/README.rst
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/build-installer.py#L239-L246

If you just want to bypass verification, you can create a new SSLContext. By default newly created contexts use CERT_NONE.
Be careful with this as stated in section 17.3.7.2.1
When calling the SSLContext constructor directly, CERT_NONE is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you’re talking to. Therefore, when in client mode, it is highly recommended to use CERT_REQUIRED.
But if you just want it to work now for some other reason you can do the following, you'll have to import ssl as well:
input = input.replace("!web ", "")
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' })
gcontext = ssl.SSLContext() # Only for gangstars
info = urllib2.urlopen(req, context=gcontext).read()
Message.Chat.SendMessage ("" + info)
This should get round your problem but you're not really solving any of the issues, but you won't see the [SSL: CERTIFICATE_VERIFY_FAILED] because you now aren't verifying the cert!
To add to the above, if you want to know more about why you are seeing these issues you will want to have a look at PEP 476.
This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python's HTTP clients by default, subject to opt-out on a per-call basis. This change would be applied to Python 2.7, Python 3.4, and Python 3.5.
There is an advised opt out which isn't dissimilar to my advice above:
import ssl
# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)
It also features a highly discouraged option via monkeypatching which you don't often see in python:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Which overrides the default function for context creation with the function to create an unverified context.
Please note with this as stated in the PEP:
This guidance is aimed primarily at system administrators that wish to adopt newer versions of Python that implement this PEP in legacy environments that do not yet support certificate verification on HTTPS connections. For example, an administrator may opt out by adding the monkeypatch above to sitecustomize.py in their Standard Operating Environment for Python. Applications and libraries SHOULD NOT be making this change process wide (except perhaps in response to a system administrator controlled configuration setting).
If you want to read a paper on why not validating certs is bad in software you can find it here!

To expand on Craig Glennie's answer:
in Python 3.6.1 on MacOs Sierra
Entering this in the bash terminal solved the problem:
pip install certifi
/Applications/Python\ 3.6/Install\ Certificates.command

On Windows, Python does not look at the system certificate, it uses its own located at ?\lib\site-packages\certifi\cacert.pem.
The solution to your problem:
download the domain validation certificate as *.crt or *pem file
open the file in editor and copy it's content to clipboard
find your cacert.pem location: from requests.utils import DEFAULT_CA_BUNDLE_PATH; print(DEFAULT_CA_BUNDLE_PATH)
edit the cacert.pem file and paste your domain validation certificate at the end of the file.
Save the file and enjoy requests!

I was having a similar problem, though I was using urllib.request.urlopen in Python 3.4, 3.5, and 3.6. (This is a portion of the Python 3 equivalent of urllib2, per the note at the head of Python 2's urllib2 documentation page.)
My solution was to pip install certifi to install certifi, which has:
... a carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.
Then, in my code where I previously just had:
import urllib.request as urlrq
resp = urlrq.urlopen('https://example.com/bar/baz.html')
I revised it to:
import urllib.request as urlrq
import certifi
resp = urlrq.urlopen('https://example.com/bar/baz.html', cafile=certifi.where())
If I read the urllib2.urlopen documentation correctly, it also has a cafile argument. So, urllib2.urlopen([...], certifi.where()) might work for Python 2.7 as well.
UPDATE (2020-01-01): As of Python 3.6, the cafile argument to urlopen has been deprecated, with the context argument supposed to be specified instead. I found the following to work equally well on 3.5 through 3.8:
import urllib.request as urlrq
import certifi
import ssl
resp = urlrq.urlopen('https://example.com/bar/baz.html', context=ssl.create_default_context(cafile=certifi.where()))

My solution for Mac OS X:
Upgrade to Python 3.6.5 using the native app Python installer downloaded from the official Python language website https://www.python.org/downloads/
I've found that this installer is taking care of updating the links and symlinks for the new Python a lot better than homebrew.
Install a new certificate using "./Install Certificates.command" which is in the refreshed Python 3.6 directory
cd "/Applications/Python 3.6/"
sudo "./Install Certificates.command"

You could try adding this to your environment variables:
PYTHONHTTPSVERIFY=0
Note that this will disable all HTTPS verification so is a bit of a sledgehammer approach, however if verification isn't required it may be an effective solution.

I had a similar problem on one of my Linux machines. Generating fresh certificates and exporting an environment variable pointing to the certificates directory fixed it for me:
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs

import requests
requests.packages.urllib3.disable_warnings()
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
Taken from here https://gist.github.com/michaelrice/a6794a017e349fc65d01

Like I've written in a comment, this problem is probably related to this SO answer.
In short: there are multiple ways to verify the certificate. The verification used by OpenSSL is incompatible with the trusted root certificates you have on your system. OpenSSL is used by Python.
You could try to get the missing certificate for Verisign Class 3 Public Primary Certification Authority and then use the cafile option according to the Python documentation:
urllib2.urlopen(req, cafile="verisign.pem")

$ cd $HOME
$ wget --quiet https://curl.haxx.se/ca/cacert.pem
$ export SSL_CERT_FILE=$HOME/cacert.pem
Source: https://access.redhat.com/articles/2039753

I think you have several ways for fix this issue. I mentioned 5 ways below:
You can define context for each request and pass the context on each request for use it like below:
import certifi
import ssl
import urllib
context = ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com', context=context)
OR Set certificate file in environment.
import os
import certifi
import urllib
os.environ["REQUESTS_CA_BUNDLE"] = certifi.where()
os.environ["SSL_CERT_FILE"] = certifi.where()
result = urllib.request.urlopen('https://www.example.com')
OR Replace create default https context method:
import certifi
import ssl
ssl._create_default_https_context = lambda: ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com')
OR If you use Linux machine, generating fresh certificates and exporting an environment variable pointing to the certificates directory fixed it.
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs
OR If you use Mac machine, generating fresh certificates
$ cd "/Applications/$(python3 --version | awk '{print $2}'| awk -F. '{print "Python " $1"."$2}')"
$ sudo "./Install Certificates.command"

Solution for Anaconda
My setup is Anaconda Python 3.7 on MacOS with a proxy. The paths are different.
This is how you get the correct certificates path:
import ssl
ssl.get_default_verify_paths()
which on my system produced
Out[35]: DefaultVerifyPaths(cafile='/miniconda3/ssl/cert.pem', capath=None,
openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/miniconda3/ssl/cert.pem',
openssl_capath_env='SSL_CERT_DIR', openssl_capath='/miniconda3/ssl/certs')
Once you know where the certificate goes, then you concatenate the certificate used by the proxy to the end of that file.
I had already set up conda to work with my proxy, by running:
conda config --set ssl_verify <pathToYourFile>.crt
If you don't remember where your cert is, you can find it in ~/.condarc:
ssl_verify: <pathToYourFile>.crt
Now concatenate that file to the end of /miniconda3/ssl/cert.pem
and requests should work, and in particular sklearn.datasets and similar tools
should work.
Further Caveats
The other solutions did not work because the Anaconda setup is slightly different:
The path Applications/Python\ 3.X simply doesn't exist.
The path provided by the commands below is the WRONG path
from requests.utils import DEFAULT_CA_BUNDLE_PATH
DEFAULT_CA_BUNDLE_PATH

I need to add another answer because just like Craig Glennie, I went on a wild goose chase due to the many posts referring to this problem across the Web.
I am using MacPorts, and what I originally thought was a Python problem was in fact a MacPorts problem: it does not install a root certificate with its installation of openssl. The solution is to port install curl-ca-bundle, as mentioned in this blog post.

Python 2.7.12 (default, Jul 29 2016, 15:26:22) fixed the mentioned issue. This information might help someone else.

I am surprised all these instruction didn't solved my problem. Nonetheless, the diagnostic is correct (BTW, I am using Mac and Python3.6.1). So, to summarize the correct part :
On Mac, Apple is dropping OpenSSL
Python now uses it own set of CA Root Certificate
Binary Python installation provided a script to install the CA Root certificate Python needs ("/Applications/Python 3.6/Install Certificates.command")
Read "/Applications/Python 3.6/ReadMe.rtf" for details
For me, the script doesn't work, and all those certifi and openssl installation failed to fix too. Maybe because I have multiple python 2 and 3 installations, as well as many virtualenv. At the end, I need to fix it by hand.
pip install certifi # for your virtualenv
mkdir -p /Library/Frameworks/Python.framework/Versions/3.6/etc/openssl
cp -a <your virtualenv>/site-package/certifi/cacert.pem \
/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem
If that still fails you. Then re/install OpenSSL as well.
port install openssl

I have found this over here
I found this solution, insert this code at the beginning of your source file:
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
This code makes the verification undone so that the ssl certification is not verified.

The SSL: CERTIFICATE_VERIFY_FAILED error could also occur because an Intermediate Certificate is missing in the ca-certificates package on Linux. For example, in my case the intermediate certificate "DigiCert SHA2 Secure Server CA" was missing in the ca-certificates package even though the Firefox browser includes it. You can find out which certificate is missing by directly running the wget command on the URL causing this error. Then you can search for the corresponding link to the CRT file for this certificate from the official website (e.g. https://www.digicert.com/digicert-root-certificates.htm in my case) of the Certificate Authority. Now, to include the certificate that is missing in your case, you may run the below commands using your CRT file download link instead:
wget https://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt
mv DigiCertSHA2SecureServerCA.crt DigiCertSHA2SecureServerCA.der
openssl x509 -inform DER -outform PEM -in DigiCertSHA2SecureServerCA.der -out DigicertSHA2SecureServerCA.pem.crt
sudo mkdir /usr/share/ca-certificates/extra
sudo cp DigicertSHA2SecureServerCA.pem.crt /usr/share/ca-certificates/extra/
sudo dpkg-reconfigure ca-certificates
After this you may test again with wget for your URL as well as by using the python urllib package. For more details, refer to: https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1795242

Take a look at
/Applications/Python 3.6/Install Certificates.command
You can also go to the Applications folder in Finder and click on Certificates.command

For Python 3.4+ on Centos 6/7,Fedora, just install the trusted CA this way :
Copy the CA.crt to /etc/pki/ca-trust/source/anchors/
update-ca-trust force-enable
update-ca-trust extract

Like you, I am using python 2.7 on my old iMac (OS X 10.6.8), I met the problem too, using urllib2.urlopen :
urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
My programs were running fine without SSL certificate problems and suddently (after dowloading programs), they crashed with this SSL error.
The problem was the version of python used :
No problem with https://www.python.org/downloads and python-2.7.9-macosx10.6.pkg
problem with the one instaled by Homebrew tool : "brew install python", version located in /usr/local/bin.
A chapter, called Certificate verification and OpenSSL [CHANGED for Python 2.7.9], in /Applications/Python 2.7/ReadMe.rtf explains the problem with many details.
So, check, download and put in your PATH the right version of python.

I hang my head in semi-shame, as I had the same issue, except that in my case, the URL I was hitting was valid, the certificate was valid. What wasn't valid was my connection out to the web. I had failed to add proxy details into the browser (IE in this case). This stopped the verification process from happening correctly.
Added in the proxy details and my python was then very happy .

Python 2.7 on Amazon EC2 with centOS 7
I had to set the env variable SSL_CERT_DIR to point to my ca-bundle which was located at /etc/ssl/certs/ca-bundle.crt

There are cases when you can not use insecure connections or pass ssl context into urllib request. Here my solution based on
https://stackoverflow.com/a/28052583/6709778
In a case if you want use your own cert file
import ssl
def new_ssl_context_decorator(*args, **kwargs):
kwargs['cafile'] = '/etc/ssl/certs/ca-certificates.crt'
return ssl.create_default_context(*args, **kwargs)
ssl._create_default_https_context = ssl._create_unverified_context
or you can use shared file from certifi
def new_ssl_context_decorator(*args, **kwargs):
import certifi
kwargs['cafile'] = certifi.where()
return ssl.create_default_context(*args, **kwargs)

Installing PyOpenSSL using pip worked for me (without converting to PEM):
pip install PyOpenSSL

I had this problem solved by closing Fiddler (an HTTP debugging proxy) check if you have a proxy enabled and try again.

In python 2.7 adding Trusted root CA details at the end in file C:\Python27\lib\site-packages\certifi\cacert.pem helped
after that i did run (using admin rights)
pip install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org packageName

Installing certifi on Mac solved my issue:
pip install certifi

In my case I was getting this error because requests and urllib3 versions were incompatible, giving the following error during installation:
ERROR: requests 2.21.0 has requirement urllib3<1.25,>=1.21.1, but you'll have urllib3 1.25 which is incompatible.
pip install 'urllib3<1.25' --force-reinstall
did the trick.

Another Anaconda solution. I was getting CERTIFICATE_VERIFY_FAILED in my Python 2.7 environment on macOS. It turns out the conda paths were bad:
base (3.7) environment:
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/usr/local/anaconda3/ssl/cert.pem', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/anaconda3/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/anaconda3/ssl/certs')
2.7 environment (paths did not exist!):
DefaultVerifyPaths(cafile='', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/anaconda3/envs/py27/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/anaconda3/envs/py27/ssl/certs')
The fix:
cd /usr/local/anaconda3/envs/py27/
mkdir ssl
cd ssl
ln -s ../../../ssl/cert.pem

Pip install fails: SSL required

Collecting rsa==3.1.1 (from -r /racetrack/.requirements.txt (line 41))
eval (python -m virtualfish)
Downloading rsa-3.1.1.tar.gz
Complete output from command python setup.py egg_info:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.10.tar.gz
Traceback (most recent call last):
File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: SSL is required

Solution: Upgrade to the latest version of your library.
It all starts here, suddenly(on Oct-2017) the Distutils team decided to revoke support for non-https requests without worrying about breaking the backward compatibility of earlier versions of numerous python libraries. Bit unprofessional, but hey it's Python world.
So here is the fix, Simply move on to the latest version of the library (rsa==3.4.2 in my case) on whatever library (nltk==3.2.5 etc)
Alternate Solution: Fork(or locally copy) the version of repo & modify the http url into https
Nonetheless, watch out if you are doing this while maintaining another project since the dependency you are upgrading might not be compatible with the original library the author had intended, for example in my context rsa was used under another library as a dependency. Thus the solution was to upgrade the parent library so that this issue is automatically taken care.

Unfortunately none of the previous answers work for me.
IMHO it was very stupid pip / distutils chose to break packages on http repos.
I think a better choice would have been:
pip/distutils use https by default
in case of error, like 403, pip has to suggest you "the package repo is on http, do you want to download it?"
Still in 2020 many Python 2 packages are on http repos; with their decision, the installation of these packages is broken.
The working solution for me is a very simple patch of one python core modules:
--- /usr/local/lib/python2.7/urllib2.py.original
+++ /usr/local/lib/python2.7/urllib2.py
## -427,6 +427,9 ##
req = meth(req)
response = self._open(req, data)
+ if protocol == "http" and response.code == 403 :
+ if isinstance(fullurl, basestring) and fullurl.startswith("http://pypi.python.org/packages/source/d/distribute/") :
+ return self.open(fullurl.replace("http://", "https://"), data = data, timeout = timeout)
# post-process response
meth_name = protocol+"_response"
Working: if the failed url is on http, retry on https.
I know it is a little ugly, but it is very clear and also you can revert to the original module in a snap (make a copy of /usr/local/lib/python2.7/urllib2.py before to apply this patch).

Just an easy_install rsa==3.1.1 will do the job.

The accepted answer did not work in my case (on an elder Raspbian), but providing the download URL with the command helped me as described in this post:
sudo pip install paho-mqtt -i https://pypi.python.org/simple

Using easy_install rather than pip worked for me:
easy_install funkload
I was attempting to pip install funkload and was getting:
Collecting funkload
Using cached funkload-1.17.1.tar.gz
Complete output from command python setup.py egg_info:
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.14.tar.gz
Traceback (most recent call last):
...<snip>
urllib2.HTTPError: HTTP Error 403: SSL is required
As funkload dates from 2011, the older easy_install works.

urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error

I am getting the following error:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/Matthew/Desktop/Skypebot 2.0/bot.py", line 271, in process
info = urllib2.urlopen(req).read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
This is the code that is causing this error:
if input.startswith("!web"):
input = input.replace("!web ", "")
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXX' })
info = urllib2.urlopen(req).read()
Message.Chat.SendMessage ("" + info)
The API I'm using requires me to use HTTPS. How can I make it bypass the verification?

This isn't a solution to your specific problem, but I'm putting it here because this thread is the top Google result for "SSL: CERTIFICATE_VERIFY_FAILED", and it lead me on a wild goose chase.
If you have installed Python 3.6 on OSX and are getting the "SSL: CERTIFICATE_VERIFY_FAILED" error when trying to connect to an https:// site, it's probably because Python 3.6 on OSX has no certificates at all, and can't validate any SSL connections. This is a change for 3.6 on OSX, and requires a post-install step, which installs the certifi package of certificates. This is documented in the file ReadMe.rtf, which you can find at /Applications/Python\ 3.6/ReadMe.rtf (see also the file Conclusion.rtf, and the script build-installer.py that generates the macOS installer).
The ReadMe will have you run the post-install script at
/Applications/Python\ 3.10/Install\ Certificates.command (Terminal App, this command alone should, fix the issue. Be sure to update the file path using your current subversion.)
(its source is install_certificates.command), which:
first installs the Python package certifi, and
then creates a symbolic link from the OpenSSL certificates file to the certificates file installed by the package certifi.
Release notes have some more info: https://www.python.org/downloads/release/python-360/
On newer versions of Python, there is more documentation about this:
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/ReadMe.rtf#L22-L34
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Conclusion.rtf#L15-L19
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/Welcome.rtf#L23-L25
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/resources/install_certificates.command
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/README.rst
https://github.com/python/cpython/blob/e05a703848473b0365886dcc593cbddc46609f29/Mac/BuildScript/build-installer.py#L239-L246

If you just want to bypass verification, you can create a new SSLContext. By default newly created contexts use CERT_NONE.
Be careful with this as stated in section 17.3.7.2.1
When calling the SSLContext constructor directly, CERT_NONE is the default. Since it does not authenticate the other peer, it can be insecure, especially in client mode where most of time you would like to ensure the authenticity of the server you’re talking to. Therefore, when in client mode, it is highly recommended to use CERT_REQUIRED.
But if you just want it to work now for some other reason you can do the following, you'll have to import ssl as well:
input = input.replace("!web ", "")
url = "https://domainsearch.p.mashape.com/index.php?name=" + input
req = urllib2.Request(url, headers={ 'X-Mashape-Key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' })
gcontext = ssl.SSLContext() # Only for gangstars
info = urllib2.urlopen(req, context=gcontext).read()
Message.Chat.SendMessage ("" + info)
This should get round your problem but you're not really solving any of the issues, but you won't see the [SSL: CERTIFICATE_VERIFY_FAILED] because you now aren't verifying the cert!
To add to the above, if you want to know more about why you are seeing these issues you will want to have a look at PEP 476.
This PEP proposes to enable verification of X509 certificate signatures, as well as hostname verification for Python's HTTP clients by default, subject to opt-out on a per-call basis. This change would be applied to Python 2.7, Python 3.4, and Python 3.5.
There is an advised opt out which isn't dissimilar to my advice above:
import ssl
# This restores the same behavior as before.
context = ssl._create_unverified_context()
urllib.urlopen("https://no-valid-cert", context=context)
It also features a highly discouraged option via monkeypatching which you don't often see in python:
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
Which overrides the default function for context creation with the function to create an unverified context.
Please note with this as stated in the PEP:
This guidance is aimed primarily at system administrators that wish to adopt newer versions of Python that implement this PEP in legacy environments that do not yet support certificate verification on HTTPS connections. For example, an administrator may opt out by adding the monkeypatch above to sitecustomize.py in their Standard Operating Environment for Python. Applications and libraries SHOULD NOT be making this change process wide (except perhaps in response to a system administrator controlled configuration setting).
If you want to read a paper on why not validating certs is bad in software you can find it here!

To expand on Craig Glennie's answer:
in Python 3.6.1 on MacOs Sierra
Entering this in the bash terminal solved the problem:
pip install certifi
/Applications/Python\ 3.6/Install\ Certificates.command

On Windows, Python does not look at the system certificate, it uses its own located at ?\lib\site-packages\certifi\cacert.pem.
The solution to your problem:
download the domain validation certificate as *.crt or *pem file
open the file in editor and copy it's content to clipboard
find your cacert.pem location: from requests.utils import DEFAULT_CA_BUNDLE_PATH; print(DEFAULT_CA_BUNDLE_PATH)
edit the cacert.pem file and paste your domain validation certificate at the end of the file.
Save the file and enjoy requests!

I was having a similar problem, though I was using urllib.request.urlopen in Python 3.4, 3.5, and 3.6. (This is a portion of the Python 3 equivalent of urllib2, per the note at the head of Python 2's urllib2 documentation page.)
My solution was to pip install certifi to install certifi, which has:
... a carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts.
Then, in my code where I previously just had:
import urllib.request as urlrq
resp = urlrq.urlopen('https://example.com/bar/baz.html')
I revised it to:
import urllib.request as urlrq
import certifi
resp = urlrq.urlopen('https://example.com/bar/baz.html', cafile=certifi.where())
If I read the urllib2.urlopen documentation correctly, it also has a cafile argument. So, urllib2.urlopen([...], certifi.where()) might work for Python 2.7 as well.
UPDATE (2020-01-01): As of Python 3.6, the cafile argument to urlopen has been deprecated, with the context argument supposed to be specified instead. I found the following to work equally well on 3.5 through 3.8:
import urllib.request as urlrq
import certifi
import ssl
resp = urlrq.urlopen('https://example.com/bar/baz.html', context=ssl.create_default_context(cafile=certifi.where()))

My solution for Mac OS X:
Upgrade to Python 3.6.5 using the native app Python installer downloaded from the official Python language website https://www.python.org/downloads/
I've found that this installer is taking care of updating the links and symlinks for the new Python a lot better than homebrew.
Install a new certificate using "./Install Certificates.command" which is in the refreshed Python 3.6 directory
cd "/Applications/Python 3.6/"
sudo "./Install Certificates.command"

You could try adding this to your environment variables:
PYTHONHTTPSVERIFY=0
Note that this will disable all HTTPS verification so is a bit of a sledgehammer approach, however if verification isn't required it may be an effective solution.

I had a similar problem on one of my Linux machines. Generating fresh certificates and exporting an environment variable pointing to the certificates directory fixed it for me:
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs

import requests
requests.packages.urllib3.disable_warnings()
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
Taken from here https://gist.github.com/michaelrice/a6794a017e349fc65d01

Like I've written in a comment, this problem is probably related to this SO answer.
In short: there are multiple ways to verify the certificate. The verification used by OpenSSL is incompatible with the trusted root certificates you have on your system. OpenSSL is used by Python.
You could try to get the missing certificate for Verisign Class 3 Public Primary Certification Authority and then use the cafile option according to the Python documentation:
urllib2.urlopen(req, cafile="verisign.pem")

$ cd $HOME
$ wget --quiet https://curl.haxx.se/ca/cacert.pem
$ export SSL_CERT_FILE=$HOME/cacert.pem
Source: https://access.redhat.com/articles/2039753

I think you have several ways for fix this issue. I mentioned 5 ways below:
You can define context for each request and pass the context on each request for use it like below:
import certifi
import ssl
import urllib
context = ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com', context=context)
OR Set certificate file in environment.
import os
import certifi
import urllib
os.environ["REQUESTS_CA_BUNDLE"] = certifi.where()
os.environ["SSL_CERT_FILE"] = certifi.where()
result = urllib.request.urlopen('https://www.example.com')
OR Replace create default https context method:
import certifi
import ssl
ssl._create_default_https_context = lambda: ssl.create_default_context(cafile=certifi.where())
result = urllib.request.urlopen('https://www.example.com')
OR If you use Linux machine, generating fresh certificates and exporting an environment variable pointing to the certificates directory fixed it.
$ sudo update-ca-certificates --fresh
$ export SSL_CERT_DIR=/etc/ssl/certs
OR If you use Mac machine, generating fresh certificates
$ cd "/Applications/$(python3 --version | awk '{print $2}'| awk -F. '{print "Python " $1"."$2}')"
$ sudo "./Install Certificates.command"

Solution for Anaconda
My setup is Anaconda Python 3.7 on MacOS with a proxy. The paths are different.
This is how you get the correct certificates path:
import ssl
ssl.get_default_verify_paths()
which on my system produced
Out[35]: DefaultVerifyPaths(cafile='/miniconda3/ssl/cert.pem', capath=None,
openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/miniconda3/ssl/cert.pem',
openssl_capath_env='SSL_CERT_DIR', openssl_capath='/miniconda3/ssl/certs')
Once you know where the certificate goes, then you concatenate the certificate used by the proxy to the end of that file.
I had already set up conda to work with my proxy, by running:
conda config --set ssl_verify <pathToYourFile>.crt
If you don't remember where your cert is, you can find it in ~/.condarc:
ssl_verify: <pathToYourFile>.crt
Now concatenate that file to the end of /miniconda3/ssl/cert.pem
and requests should work, and in particular sklearn.datasets and similar tools
should work.
Further Caveats
The other solutions did not work because the Anaconda setup is slightly different:
The path Applications/Python\ 3.X simply doesn't exist.
The path provided by the commands below is the WRONG path
from requests.utils import DEFAULT_CA_BUNDLE_PATH
DEFAULT_CA_BUNDLE_PATH

I need to add another answer because just like Craig Glennie, I went on a wild goose chase due to the many posts referring to this problem across the Web.
I am using MacPorts, and what I originally thought was a Python problem was in fact a MacPorts problem: it does not install a root certificate with its installation of openssl. The solution is to port install curl-ca-bundle, as mentioned in this blog post.

Python 2.7.12 (default, Jul 29 2016, 15:26:22) fixed the mentioned issue. This information might help someone else.

I am surprised all these instruction didn't solved my problem. Nonetheless, the diagnostic is correct (BTW, I am using Mac and Python3.6.1). So, to summarize the correct part :
On Mac, Apple is dropping OpenSSL
Python now uses it own set of CA Root Certificate
Binary Python installation provided a script to install the CA Root certificate Python needs ("/Applications/Python 3.6/Install Certificates.command")
Read "/Applications/Python 3.6/ReadMe.rtf" for details
For me, the script doesn't work, and all those certifi and openssl installation failed to fix too. Maybe because I have multiple python 2 and 3 installations, as well as many virtualenv. At the end, I need to fix it by hand.
pip install certifi # for your virtualenv
mkdir -p /Library/Frameworks/Python.framework/Versions/3.6/etc/openssl
cp -a <your virtualenv>/site-package/certifi/cacert.pem \
/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem
If that still fails you. Then re/install OpenSSL as well.
port install openssl

I have found this over here
I found this solution, insert this code at the beginning of your source file:
import ssl
try:
_create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
# Legacy Python that doesn't verify HTTPS certificates by default
pass
else:
# Handle target environment that doesn't support HTTPS verification
ssl._create_default_https_context = _create_unverified_https_context
This code makes the verification undone so that the ssl certification is not verified.

The SSL: CERTIFICATE_VERIFY_FAILED error could also occur because an Intermediate Certificate is missing in the ca-certificates package on Linux. For example, in my case the intermediate certificate "DigiCert SHA2 Secure Server CA" was missing in the ca-certificates package even though the Firefox browser includes it. You can find out which certificate is missing by directly running the wget command on the URL causing this error. Then you can search for the corresponding link to the CRT file for this certificate from the official website (e.g. https://www.digicert.com/digicert-root-certificates.htm in my case) of the Certificate Authority. Now, to include the certificate that is missing in your case, you may run the below commands using your CRT file download link instead:
wget https://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt
mv DigiCertSHA2SecureServerCA.crt DigiCertSHA2SecureServerCA.der
openssl x509 -inform DER -outform PEM -in DigiCertSHA2SecureServerCA.der -out DigicertSHA2SecureServerCA.pem.crt
sudo mkdir /usr/share/ca-certificates/extra
sudo cp DigicertSHA2SecureServerCA.pem.crt /usr/share/ca-certificates/extra/
sudo dpkg-reconfigure ca-certificates
After this you may test again with wget for your URL as well as by using the python urllib package. For more details, refer to: https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1795242

Take a look at
/Applications/Python 3.6/Install Certificates.command
You can also go to the Applications folder in Finder and click on Certificates.command

For Python 3.4+ on Centos 6/7,Fedora, just install the trusted CA this way :
Copy the CA.crt to /etc/pki/ca-trust/source/anchors/
update-ca-trust force-enable
update-ca-trust extract

Like you, I am using python 2.7 on my old iMac (OS X 10.6.8), I met the problem too, using urllib2.urlopen :
urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
My programs were running fine without SSL certificate problems and suddently (after dowloading programs), they crashed with this SSL error.
The problem was the version of python used :
No problem with https://www.python.org/downloads and python-2.7.9-macosx10.6.pkg
problem with the one instaled by Homebrew tool : "brew install python", version located in /usr/local/bin.
A chapter, called Certificate verification and OpenSSL [CHANGED for Python 2.7.9], in /Applications/Python 2.7/ReadMe.rtf explains the problem with many details.
So, check, download and put in your PATH the right version of python.

I hang my head in semi-shame, as I had the same issue, except that in my case, the URL I was hitting was valid, the certificate was valid. What wasn't valid was my connection out to the web. I had failed to add proxy details into the browser (IE in this case). This stopped the verification process from happening correctly.
Added in the proxy details and my python was then very happy .

Python 2.7 on Amazon EC2 with centOS 7
I had to set the env variable SSL_CERT_DIR to point to my ca-bundle which was located at /etc/ssl/certs/ca-bundle.crt

There are cases when you can not use insecure connections or pass ssl context into urllib request. Here my solution based on
https://stackoverflow.com/a/28052583/6709778
In a case if you want use your own cert file
import ssl
def new_ssl_context_decorator(*args, **kwargs):
kwargs['cafile'] = '/etc/ssl/certs/ca-certificates.crt'
return ssl.create_default_context(*args, **kwargs)
ssl._create_default_https_context = ssl._create_unverified_context
or you can use shared file from certifi
def new_ssl_context_decorator(*args, **kwargs):
import certifi
kwargs['cafile'] = certifi.where()
return ssl.create_default_context(*args, **kwargs)

Installing PyOpenSSL using pip worked for me (without converting to PEM):
pip install PyOpenSSL

I had this problem solved by closing Fiddler (an HTTP debugging proxy) check if you have a proxy enabled and try again.

In python 2.7 adding Trusted root CA details at the end in file C:\Python27\lib\site-packages\certifi\cacert.pem helped
after that i did run (using admin rights)
pip install --trusted-host pypi.python.org --trusted-host pypi.org --trusted-host files.pythonhosted.org packageName

Installing certifi on Mac solved my issue:
pip install certifi

In my case I was getting this error because requests and urllib3 versions were incompatible, giving the following error during installation:
ERROR: requests 2.21.0 has requirement urllib3<1.25,>=1.21.1, but you'll have urllib3 1.25 which is incompatible.
pip install 'urllib3<1.25' --force-reinstall
did the trick.

Another Anaconda solution. I was getting CERTIFICATE_VERIFY_FAILED in my Python 2.7 environment on macOS. It turns out the conda paths were bad:
base (3.7) environment:
>>> import ssl
>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/usr/local/anaconda3/ssl/cert.pem', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/anaconda3/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/anaconda3/ssl/certs')
2.7 environment (paths did not exist!):
DefaultVerifyPaths(cafile='', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/local/anaconda3/envs/py27/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/local/anaconda3/envs/py27/ssl/certs')
The fix:
cd /usr/local/anaconda3/envs/py27/
mkdir ssl
cd ssl
ln -s ../../../ssl/cert.pem

How would you install a python module with chef?

We're using EngineYard which has Python installed by default. But when we enabled SSL we received the following error message from our logentries chef recipe.
"WARNING: The "ssl" module is not present. Using unreliable workaround, host identity cannot be verified. Please install "ssl" module or newer version of Python (2.6) if possible."
I'm looking for a way to install the SSL module with chef recipe but I simply don't have enough experience. Could someone point me in the right direction?
Resources:
Logentries chef recipe: https://github.com/logentries/le_chef
Logentries EY docs: https://logentries.com/doc/engineyard/
SSL Module: http://pypi.python.org/pypi/ssl/

There now appears to be a solution with better community support (based on the fact that it is documented on the opscode website).
You might try:
include_recipe 'python'
python_pip 'ssl'
As documented: here or here

I just wrote a recipe for this, and now am able to run the latest Logentries client on EngineYard. Here you go:
file_dir = "/mnt/src/python-ssl"
file_name = "ssl-1.15.tar.gz"
file_path = File.join(file_dir,file_name)
uncompressed_file_dir = File.join(file_dir, file_name.split(".tar.gz").first)
directory file_dir do
owner "deploy"
group "deploy"
mode "0755"
recursive true
action :create
end
remote_file file_path do
source "http://pypi.python.org/packages/source/s/ssl/ssl-1.15.tar.gz"
mode "0644"
not_if { File.exists?(file_path) }
end
execute "gunzip ssl" do
command "gunzip -c #{file_name} | tar xf -"
cwd file_dir
not_if { File.exists?(uncompressed_file_dir) }
end
installed_file_path = File.join(uncompressed_file_dir, "installed")
execute "install python ssl module" do
command "python setup.py install"
cwd uncompressed_file_dir
not_if { File.exists?(installed_file_path) }
end
execute "touch #{installed_file_path}" do
action :run
end

You could install a new Python using PythonBrew: https://github.com/utahta/pythonbrew. Just make you install libssl before you build, or it still won't be able to use SSL. However, based on the warning, it seems that SSL might work, but it won't be able to verify host. Of course, that is one major purposes of SSL, so that is likely a non-starter.
HTH