Python requests "certificate verify failed" - python

I'm maintaining a Python mini-app that uses requests + HTTPS.
The app worked until the IP address of the hostname in the HTTPS URL changed (legitimately). If I point my browser to the URL I can retrieve it fine.
Where does Python/requests keep the analog of ssh's known_hosts and how do I clear it for this host?
$ python --version
Python 2.7.3
$ cat foo.py
import requests
url = "https://somehost/resource.json"
requests.get(url, timeout=5, config={'danger_mode': True})
$ source venv/bin/activate
$ python foo.py
Traceback (most recent call last):
File "foo.py", line 3, in <module>
requests.get(url, timeout=5, config={'danger_mode': True})
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/safe_mode.py", line 39, in wrapped
return function(method, url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/api.py", line 51, in request
return session.request(method=method, url=url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 241, in request
r.send(prefetch=prefetch)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/models.py", line 641, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

You're using an ancient version of requests. You'll get a more helpful message if you upgrade to 2.0 and if your site has a certificate mismatch you may be able to fix it by specifying the system certificates which will be able to verify an intermediate certificate. You can also just have requests not verify your certificate as Andre suggested.

It turned out that during the server upgrade mentioned in the question an incorrectly-signed certificate was installed. HTTPS in the browser worked because of the root certificate differences between the Windows browser machine and Ubuntu Python client. HTTPS via a browser from the same Ubuntu machine on which Python was run revealed the certificate problem details.
The IP change had little to do with the problem except to confuse things.
Promoting my comment to an answer as:
this answered my question
this question is getting enough traffic I'd like to share the knowledge.

If the above doesn't work, and you find out it is local, then this solution worked for me...
Essentially run the Install Certificates.command file within Python folder.
https://www.dev2qa.com/how-to-fix-python-error-certificate-verify-failed-unable-to-get-local-issuer-certificate-in-mac-os/

Related

Wordpress xmlrpc SSL Certificate Error (Only On 1 Machine)

I'm using the Wordpress xmlrpc Python module on Python 3.6 to automatically write and publish blog posts to my Wordpress site (hosted directly by Wordpress).
The program runs great on one of my Windows machines, but when I try to run it using my second Windows machine, with the exact same code, on the same network, I receive an SSL error. Details below:
import ssl
import wordpress_xmlrpc
from wordpress_xmlrpc import Client
from wordpress_xmlrpc import WordPressPost
from wordpress_xmlrpc.methods.posts import GetPosts
from wordpress_xmlrpc.methods.posts import NewPost
from wordpress_xmlrpc.methods.users import GetUserInfo
from wordpress_xmlrpc.methods import posts
from wordpress_xmlrpc.compat import xmlrpc_client
wp = Client("https://website.io/xmlrpc.php", "wordpressusername", "wordpresspassword")
post = WordPressPost()
post.title = "title"
post.content = content
post.post_status = 'publish'
status_draft = 0
status_published = 1
wp.call(NewPost(post))
Here's the error:
File "C:\Python36\Lib\http\client.py", line 964, in send
self.connect()
File "C:\Python36\Lib\http\client.py", line 1400, in connect
server_hostname=server_hostname)
File "C:\Python36\Lib\ssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "C:\Python36\Lib\ssl.py", line 808, in __init__
self.do_handshake()
File "C:\Python36\Lib\ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "C:\Python36\Lib\ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)
I've used pip list to view all installed modules on both machines and everything matches exactly. The code is stored on a synced Google Drive folder, so it's literally the exact same .py file both times. I can't understand why it works on one machine but not the other.
I've read the thread here but I don't believe it applies to the wordpress xmlrpc tool. I've read through the documentation here but I can't see anything helpful.
Is this something I have to tweak/delete/refresh the ssl certificates in Chrome or something? Any answers or insight much appreciated. Thanks in advance
So, 3 weeks later, I finally found a way to fix this.
I ended up completely uninstalling/deleting Python on my secondary machine and reinstalling everything (along with reinstalling all modules, and confirming via pip list) and now it works (no more SSL error).
For what it's worth, and I can't be sure this is what was causing the issue in the first place, but previously, I was running Python 3.6.1 on the working machine and python 3.6.2 on the other, non-working machine.
When I reinstalled everything, I reinstalled Python 3.6.1 (to match the working machine) and it worked on both.

Softlayer Python API TransportError SSL CERTIFICATE_VERIFY_FAILED (_SSL.C:590)

What used to be working python code to make API calls from SoftLayer, now gives errors.
import SoftLayer
conn = SoftLayer.create_client_from_env(username='',api_key='')
allParents = conn.call('Account','getAllTopLevelBillingItems')
allParents[0] # returns the first billing_Item as a dict
It used to work, but now the following error messages appear:
result = conn['SoftLayer_Account'].getAllTopLevelBillingItems(mask=objectMask)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 363, in call_handler
return self(name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 331, in call
return self.client.call(self.name, name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 227, in call
return self.transport(request)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/transports.py", line 164, in __call__
raise exceptions.TransportError(0, str(ex))
SoftLayer.exceptions.TransportError: TransportError(0): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
This may be related with this one SSL InsecurePlatform error when using Requests package
Softlayer has documented this about SSL errors:
On Python versions below Python 2.7.9, requests has started emitting a security warning (InsecurePlatformWarning) due to insecurities with creating SSL connections. To resolve this, upgrade to Python 2.7.9+ or follow the instructions here: https://stackoverflow.com/a/29099439.
please make sure you are using Python 2.7.9 or supirior and try again
I found a couple of months ago that if the python certifi package was installed, this would cause SoftLayer API traffic to fail certificate validation.
I never found out why, I just avoided the packages that created that dependency. In my case, I was trying to install flower.

Python requests.exceptions.SSLError: EOF occurred in violation of protocol

I would retrieve some information from an ABB G13 gateway that offer a RESTful JSON API. API is hosted by the gateway via https endpoint.
Basic authentication mechanism is used for authentication. However all traffic
goes through SSL layers.
On linux with command:
curl -s -k -X GET -u user:password https://host/meters/a_serial/power
All goes well!
I'm trying to write a script for windows in Python 2.7.10 with Requests 2.8.1 and with this code:
import requests
requests.get('https://host/meters/a_serial/power', auth=('user', 'password'))
I have this error:
Traceback (most recent call last):
File "C:/Users/mzilio/PycharmProjects/pwrgtw/test.py", line 20, in <module>
requests.get('https://host/meters/a_serial/power', auth=('user', 'password'))
File "C:\Python27\lib\site-packages\requests\api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 433, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:590)
I've searched for a solution and I've tried to fix with this code:
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
import ssl
class MyAdapter(HTTPAdapter):
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize,
block=block,
ssl_version=ssl.PROTOCOL_TLSv1)
s = requests.Session()
s.mount('https://', MyAdapter())
s.get('https://host/meters/a_serial/power')
But it doesn't work for me cause I get this error:
Traceback (most recent call last):
File "C:/Users/mzilio/PycharmProjects/pwrgtw/test.py", line 16, in <module>
s.get('https://host/meters/a_serial/power')
File "C:\Python27\lib\site-packages\requests\sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 433, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:590)
I'm stuck on this problem. Could someone help me? Thanks!
This thing worked for me, just make sure whether these modules are installed or not, if not then install them, following are:
pip install ndg-httpsclient
pip install pyopenssl
pip install pyasn1
It removed my SSLError: EOF occurred in violation of protocol (_ssl.c:590) error.
Hope it helps.
Step 1: Check that Python supports TLS 1.1
You may have a Python setup that only supports TLS 1.0 – not TLS 1.1 or above.
You can check it like this:
Python 3
from urllib.request import urlopen
urlopen('https://www.howsmyssl.com/a/check').read()
Python 2
from urllib2 import urlopen
urlopen('https://www.howsmyssl.com/a/check').read()
(If you get urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)> you may have to disable certificate verification. NOTE: doing this will disable SSL protections against evildoers who would impersonate or intercept traffic to that website - see https://en.wikipedia.org/wiki/Man-in-the-middle_attack
)
import ssl
urlopen('https://www.howsmyssl.com/a/check', context=ssl._create_unverified_context()).read()
Check the output for the key tls_version. If it says TLS 1.0 and not TLS 1.1 or TLS 1.2 that could be the problem.
If you're using a virtualenv, be sure to run the command inside.
Step 2: Install Python with a newer version of OpenSSL
In order support TLS 1.1 or above, you may need to install a newer version of OpenSSL, and the install Python again afterwards. This should give you a Python that supports TLS 1.1.
The process depends on your operating system – here's a guide for OS X.
virtualenv users
For me, the Python outside of my virtualenv had TLS 1.2 support, so just I removed my old virtualenv, and created a new one with the same packages and then it worked. Easy peasy!
I found it was going through a proxy when it should have connected to the server directly.
I fixed this by doing
unset https_proxy
I had exactly the same error, turns out that I didn't have ndg-httpsclient installed, see my original issue raised in github.
If you are getting this error for intermediate requests, you can refer to the solution mentioned in https://github.com/requests/requests/issues/3391.
Basically, if you are making a lot of requests to a server and facing this issue with some of those requests you can use Session to just retry the requests.
Let me know if this works.
Close you proxy and try it again.
The only time I have seen errors of this nature have been times that I was using requests to screen scrape data and I was using a multiprocessing library. I would either try your code without the pool manager or split your app into two apps, one that doles out urls and another that consumes them.
The client-server pair here should give you some ideas. I was able to scale my client out horizontally when I used the hacky server code to load all urls into a Queue in memory just before app.run. Hope that helps.
ENV: Python 3.10, www.howsmyssl.com returns tls_version: TLS 1.3:
For poor guys like me who MUST make query thorough proxy, this cloud be blame on your incorrect HTTPS proxy setting (perhaps you aren't set it, but python somehow believes you've set it, don't know why exactly, maybe because you've set the http proxy), you need to set it "properly".
I'm using windows10, haven't set it before, after set it, python works normally, give it a try.
try to run:
pip install urllib3==1.25.11
It works for me anyway.

Azure ML Execute Python Module: Network I/O Disabled?

Is there no way to connect to a URL from azure ml and get it's content
my code:
import requests
def azureml_main(dataframe1 = None, dataframe2 = None):
b= requests.get("http://www.google.com",timeout=30)
dataframe1 = b.content
return dataframe1
Is there any change need to be made to connect to URL
ERROR:
Error 0085: The following error occurred during script evaluation, please view the output log for more information:
---------- Start of error message from Python interpreter ----------
data:text/plain,Caught exception while executing function: Traceback (most recent call last):
File "C:\server\invokepy.py", line 167, in batch
odfs = mod.azureml_main(*idfs)
File "C:\temp\azuremod.py", line 24, in azureml_main
b= requests.get("http://www.google.com",timeout=30)
File "C:\pyhome\lib\site-packages\requests\api.py", line 55, in get
return request('get', url, **kwargs)
File "C:\pyhome\lib\site-packages\requests\api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "C:\pyhome\lib\site-packages\requests\sessions.py", line 456, in request
resp = self.send(prep, **send_kwargs)
File "C:\pyhome\lib\site-packages\requests\sessions.py", line 559, in send
r = adapter.send(request, **kwargs)
File "C:\pyhome\lib\site-packages\requests\adapters.py", line 375, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='www.google.com', port=80): Max retries exceeded with url: / (Caused by <class 'socket.gaierror'>: [Errno 11001] getaddrinfo failed)
---------- End of error message from Python interpreter ---------
Or is there any change needed to be made on the azure ml settings
UPDATE 1/28/2016
Network I/O for Execute Python Script is now supported.
Out of date
Network I/O is not support from Execute Python Modules. In order to execute such a program, you should instead launch a virtual machine(Windows or Linux your choice).
Windows:
RDP into Virtual Machine
Install your choice of Python
You can drag and drop your Python program from your Local Windows machine onto your RDP screen to transfer your code
Then run your program
Ubuntu:
SSH into your virtual machine using Cygwin or Putty(Windows) or Terminal SSH (mac) ssh yourUserName#yourAzureVM.cloudapps.net
install Python sudo apt-get install python
open your preferred Linux text editor vi myProgram.py
Copy and Paste your code into the editor (leave vi with esc :wq )
Run your code python myProgram.py
To move data from your VM to AzureML please check out the Azure-MachineLearning-ClientLibrary-Python on Github

Python SSL connection "EOF occurred in violation of protocol"

I'm using Django Celery task to connect to Facebook Graph API with requests lib using Gevent. Issue I'm constantly running at is that every now and then I get EOF occurred in violation of protocol exception. I've searched around and various sources offer different fixes but none seems to work.
I've tried monkey patching the ssl module(gevent.monkey.patch_all()) and some others too but no luck.
I'm not even sure if this is openssl issue as some sources might suggest as I haven't encountered it before applying Gevent optimisation
Connection error: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol
Traceback (most recent call last):
File "/home/user/workspace/startup/project/events/tasks.py", line 52, in _process_page
data = requests.get(current_url)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/sessions.py", line 354, in request
resp = self.send(prep, **send_kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
r = adapter.send(request, **kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/adapters.py", line 250, in send
raise SSLError(e)
SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol
I'm using latest 1.0rc Gevent version.
Another issue that keeps poping up time to time although URL is correct is:
Retrying (5 attempts remain) after connection broken by 'error(2, 'No such file or directory')': /ID/events?limit=5000&fields=description,name,location,start_time,end_time&access_token=TOKEN
Using the forced TLSv1 fix as suggested by J.F Sebastian fixed all the issues I was facing.
Hints for future questions regarding:
DNSError exception - upgrading Gevent from 0.13.X to 1.0rc fixes this issue
SSL issues - look at fix in link provided by J.F Sebastian
I installed the latest Python 2.7 (2.7.11) and the problem went away. I believe the problem might even be solved back in 2.7.6 (I was using 2.7.5 on Mac OSX).
I was having the same error during fetching tweets for my machine learning . Doing the pip install of the following helped me. This works:
pip install ndg-httpsclient
pip install pyopenssl
pip install pyasn1
It removed my SSLError: EOF occurred in violation of protocol (_ssl.c:590) error.
Hope it helps.

Categories