I would retrieve some information from an ABB G13 gateway that offer a RESTful JSON API. API is hosted by the gateway via https endpoint.
Basic authentication mechanism is used for authentication. However all traffic
goes through SSL layers.
On linux with command:
curl -s -k -X GET -u user:password https://host/meters/a_serial/power
All goes well!
I'm trying to write a script for windows in Python 2.7.10 with Requests 2.8.1 and with this code:
import requests
requests.get('https://host/meters/a_serial/power', auth=('user', 'password'))
I have this error:
Traceback (most recent call last):
File "C:/Users/mzilio/PycharmProjects/pwrgtw/test.py", line 20, in <module>
requests.get('https://host/meters/a_serial/power', auth=('user', 'password'))
File "C:\Python27\lib\site-packages\requests\api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "C:\Python27\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 433, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:590)
I've searched for a solution and I've tried to fix with this code:
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
import ssl
class MyAdapter(HTTPAdapter):
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(num_pools=connections,
maxsize=maxsize,
block=block,
ssl_version=ssl.PROTOCOL_TLSv1)
s = requests.Session()
s.mount('https://', MyAdapter())
s.get('https://host/meters/a_serial/power')
But it doesn't work for me cause I get this error:
Traceback (most recent call last):
File "C:/Users/mzilio/PycharmProjects/pwrgtw/test.py", line 16, in <module>
s.get('https://host/meters/a_serial/power')
File "C:\Python27\lib\site-packages\requests\sessions.py", line 480, in get
return self.request('GET', url, **kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 433, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:590)
I'm stuck on this problem. Could someone help me? Thanks!
This thing worked for me, just make sure whether these modules are installed or not, if not then install them, following are:
pip install ndg-httpsclient
pip install pyopenssl
pip install pyasn1
It removed my SSLError: EOF occurred in violation of protocol (_ssl.c:590) error.
Hope it helps.
Step 1: Check that Python supports TLS 1.1
You may have a Python setup that only supports TLS 1.0 – not TLS 1.1 or above.
You can check it like this:
Python 3
from urllib.request import urlopen
urlopen('https://www.howsmyssl.com/a/check').read()
Python 2
from urllib2 import urlopen
urlopen('https://www.howsmyssl.com/a/check').read()
(If you get urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)> you may have to disable certificate verification. NOTE: doing this will disable SSL protections against evildoers who would impersonate or intercept traffic to that website - see https://en.wikipedia.org/wiki/Man-in-the-middle_attack
)
import ssl
urlopen('https://www.howsmyssl.com/a/check', context=ssl._create_unverified_context()).read()
Check the output for the key tls_version. If it says TLS 1.0 and not TLS 1.1 or TLS 1.2 that could be the problem.
If you're using a virtualenv, be sure to run the command inside.
Step 2: Install Python with a newer version of OpenSSL
In order support TLS 1.1 or above, you may need to install a newer version of OpenSSL, and the install Python again afterwards. This should give you a Python that supports TLS 1.1.
The process depends on your operating system – here's a guide for OS X.
virtualenv users
For me, the Python outside of my virtualenv had TLS 1.2 support, so just I removed my old virtualenv, and created a new one with the same packages and then it worked. Easy peasy!
I found it was going through a proxy when it should have connected to the server directly.
I fixed this by doing
unset https_proxy
I had exactly the same error, turns out that I didn't have ndg-httpsclient installed, see my original issue raised in github.
If you are getting this error for intermediate requests, you can refer to the solution mentioned in https://github.com/requests/requests/issues/3391.
Basically, if you are making a lot of requests to a server and facing this issue with some of those requests you can use Session to just retry the requests.
Let me know if this works.
Close you proxy and try it again.
The only time I have seen errors of this nature have been times that I was using requests to screen scrape data and I was using a multiprocessing library. I would either try your code without the pool manager or split your app into two apps, one that doles out urls and another that consumes them.
The client-server pair here should give you some ideas. I was able to scale my client out horizontally when I used the hacky server code to load all urls into a Queue in memory just before app.run. Hope that helps.
ENV: Python 3.10, www.howsmyssl.com returns tls_version: TLS 1.3:
For poor guys like me who MUST make query thorough proxy, this cloud be blame on your incorrect HTTPS proxy setting (perhaps you aren't set it, but python somehow believes you've set it, don't know why exactly, maybe because you've set the http proxy), you need to set it "properly".
I'm using windows10, haven't set it before, after set it, python works normally, give it a try.
try to run:
pip install urllib3==1.25.11
It works for me anyway.
Related
I'm using the Wordpress xmlrpc Python module on Python 3.6 to automatically write and publish blog posts to my Wordpress site (hosted directly by Wordpress).
The program runs great on one of my Windows machines, but when I try to run it using my second Windows machine, with the exact same code, on the same network, I receive an SSL error. Details below:
import ssl
import wordpress_xmlrpc
from wordpress_xmlrpc import Client
from wordpress_xmlrpc import WordPressPost
from wordpress_xmlrpc.methods.posts import GetPosts
from wordpress_xmlrpc.methods.posts import NewPost
from wordpress_xmlrpc.methods.users import GetUserInfo
from wordpress_xmlrpc.methods import posts
from wordpress_xmlrpc.compat import xmlrpc_client
wp = Client("https://website.io/xmlrpc.php", "wordpressusername", "wordpresspassword")
post = WordPressPost()
post.title = "title"
post.content = content
post.post_status = 'publish'
status_draft = 0
status_published = 1
wp.call(NewPost(post))
Here's the error:
File "C:\Python36\Lib\http\client.py", line 964, in send
self.connect()
File "C:\Python36\Lib\http\client.py", line 1400, in connect
server_hostname=server_hostname)
File "C:\Python36\Lib\ssl.py", line 401, in wrap_socket
_context=self, _session=session)
File "C:\Python36\Lib\ssl.py", line 808, in __init__
self.do_handshake()
File "C:\Python36\Lib\ssl.py", line 1061, in do_handshake
self._sslobj.do_handshake()
File "C:\Python36\Lib\ssl.py", line 683, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:748)
I've used pip list to view all installed modules on both machines and everything matches exactly. The code is stored on a synced Google Drive folder, so it's literally the exact same .py file both times. I can't understand why it works on one machine but not the other.
I've read the thread here but I don't believe it applies to the wordpress xmlrpc tool. I've read through the documentation here but I can't see anything helpful.
Is this something I have to tweak/delete/refresh the ssl certificates in Chrome or something? Any answers or insight much appreciated. Thanks in advance
So, 3 weeks later, I finally found a way to fix this.
I ended up completely uninstalling/deleting Python on my secondary machine and reinstalling everything (along with reinstalling all modules, and confirming via pip list) and now it works (no more SSL error).
For what it's worth, and I can't be sure this is what was causing the issue in the first place, but previously, I was running Python 3.6.1 on the working machine and python 3.6.2 on the other, non-working machine.
When I reinstalled everything, I reinstalled Python 3.6.1 (to match the working machine) and it worked on both.
What used to be working python code to make API calls from SoftLayer, now gives errors.
import SoftLayer
conn = SoftLayer.create_client_from_env(username='',api_key='')
allParents = conn.call('Account','getAllTopLevelBillingItems')
allParents[0] # returns the first billing_Item as a dict
It used to work, but now the following error messages appear:
result = conn['SoftLayer_Account'].getAllTopLevelBillingItems(mask=objectMask)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 363, in call_handler
return self(name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 331, in call
return self.client.call(self.name, name, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/API.py", line 227, in call
return self.transport(request)
File "/usr/local/lib/python2.7/dist-packages/SoftLayer/transports.py", line 164, in __call__
raise exceptions.TransportError(0, str(ex))
SoftLayer.exceptions.TransportError: TransportError(0): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
This may be related with this one SSL InsecurePlatform error when using Requests package
Softlayer has documented this about SSL errors:
On Python versions below Python 2.7.9, requests has started emitting a security warning (InsecurePlatformWarning) due to insecurities with creating SSL connections. To resolve this, upgrade to Python 2.7.9+ or follow the instructions here: https://stackoverflow.com/a/29099439.
please make sure you are using Python 2.7.9 or supirior and try again
I found a couple of months ago that if the python certifi package was installed, this would cause SoftLayer API traffic to fail certificate validation.
I never found out why, I just avoided the packages that created that dependency. In my case, I was trying to install flower.
I have a script on python3.4 and it has been fine until the website I download the file from decides to use https and now I am getting error but can't figure out how I can retrive the file.
My script import the following library and uses the urlretrive to get the file previously. Since it is now forwarded to https with 302 redirection. I am getting some error.
import urllib
import urllib.request
urllib.request.urlretrieve("http://wordpress.org/latest.tar.gz", "/thefile.gz")
My error:-
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.4/urllib/request.py", line 178, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
File "/usr/local/lib/python3.4/urllib/request.py", line 153, in urlopen
return opener.open(url, data, timeout)
File "/usr/local/lib/python3.4/urllib/request.py", line 461, in open
response = meth(req, response)
File "/usr/local/lib/python3.4/urllib/request.py", line 571, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/local/lib/python3.4/urllib/request.py", line 493, in error
result = self._call_chain(*args)
File "/usr/local/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.4/urllib/request.py", line 676, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/local/lib/python3.4/urllib/request.py", line 455, in open
response = self._open(req, data)
File "/usr/local/lib/python3.4/urllib/request.py", line 478, in _open
'unknown_open', req)
File "/usr/local/lib/python3.4/urllib/request.py", line 433, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.4/urllib/request.py", line 1257, in unknown_open
raise URLError('unknown url type: %s' % type)
urllib.error.URLError: <urlopen error unknown url type: https>
Most likely your Python installation or operating system is broken.
Python has only support for HTTPS if it was compiled with HTTPS support. However, this should be the default for all sane installations.
HTTPS support is only available if the socket module was compiled with SSL support.
https://docs.python.org/3/library/http.client.html
Please clarify how you installed Python. Official Python distributions are available at python.org
I had the same problem with Anaconda but after installing the OpenSSL package, it works well.
conda install -c anaconda openssl
I ran into the similar problem.
I used anaconda, and I simply moved these two file to anaconda3\DLLs and it worked.
libcrypto-1_1-x64.dll
libssl-1_1-x64.dll
I don't know why.
The reason I know this is that I tried to use ssl module to ignore certificate errors.
import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
After these lines, the https error went away and gave me another error:
DLL load failed while importing _ssl: The specified module could not be found.
So I went to this post and found these two important files.
ps: the ssl module is not necessary.
Had this issue, and it was solved by upgrading Python with
brew upgrade python
For me the error was resolved by downloading the correct version of openssl.
I was using python 3.7.5 32 bit version on windows 10 machine.
goto
https://slproweb.com/products/Win32OpenSSL.html and download Win32 OpenSSL v1.1.1h
EXE | MSI 54MB Installer version . I used 32 bit as my python interpreter was 32 bit.
Install and run.
Issue fixed :)
for HTTP
people who encountered the error
ValueError: unknown url type: 'http
or
ValueError: unknown url type: b'http
while opening some url like below with urllib.request.Request
'http://localhost/simple_form/insert.php'
just change localhost to 127.0.0.1
looks like Request method is looking for a domain.something in url
If your url has single or double quotes, i.e. '' or "", then it also gives this error.
Replace your single or double quotes with %20
url='http://stackoverflow.com/wizard.php?id="hah"'
should be changed to
url='http://stackoverflow.com/wizard.php?id=%20hah%20'
I'm maintaining a Python mini-app that uses requests + HTTPS.
The app worked until the IP address of the hostname in the HTTPS URL changed (legitimately). If I point my browser to the URL I can retrieve it fine.
Where does Python/requests keep the analog of ssh's known_hosts and how do I clear it for this host?
$ python --version
Python 2.7.3
$ cat foo.py
import requests
url = "https://somehost/resource.json"
requests.get(url, timeout=5, config={'danger_mode': True})
$ source venv/bin/activate
$ python foo.py
Traceback (most recent call last):
File "foo.py", line 3, in <module>
requests.get(url, timeout=5, config={'danger_mode': True})
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/safe_mode.py", line 39, in wrapped
return function(method, url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/api.py", line 51, in request
return session.request(method=method, url=url, **kwargs)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/sessions.py", line 241, in request
r.send(prefetch=prefetch)
File "/home/dfukdev/corsair-scripts/alfred/venv/local/lib/python2.7/site-packages/requests/models.py", line 641, in send
raise SSLError(e)
requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
You're using an ancient version of requests. You'll get a more helpful message if you upgrade to 2.0 and if your site has a certificate mismatch you may be able to fix it by specifying the system certificates which will be able to verify an intermediate certificate. You can also just have requests not verify your certificate as Andre suggested.
It turned out that during the server upgrade mentioned in the question an incorrectly-signed certificate was installed. HTTPS in the browser worked because of the root certificate differences between the Windows browser machine and Ubuntu Python client. HTTPS via a browser from the same Ubuntu machine on which Python was run revealed the certificate problem details.
The IP change had little to do with the problem except to confuse things.
Promoting my comment to an answer as:
this answered my question
this question is getting enough traffic I'd like to share the knowledge.
If the above doesn't work, and you find out it is local, then this solution worked for me...
Essentially run the Install Certificates.command file within Python folder.
https://www.dev2qa.com/how-to-fix-python-error-certificate-verify-failed-unable-to-get-local-issuer-certificate-in-mac-os/
I'm using Django Celery task to connect to Facebook Graph API with requests lib using Gevent. Issue I'm constantly running at is that every now and then I get EOF occurred in violation of protocol exception. I've searched around and various sources offer different fixes but none seems to work.
I've tried monkey patching the ssl module(gevent.monkey.patch_all()) and some others too but no luck.
I'm not even sure if this is openssl issue as some sources might suggest as I haven't encountered it before applying Gevent optimisation
Connection error: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol
Traceback (most recent call last):
File "/home/user/workspace/startup/project/events/tasks.py", line 52, in _process_page
data = requests.get(current_url)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/sessions.py", line 354, in request
resp = self.send(prep, **send_kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
r = adapter.send(request, **kwargs)
File "/home/user/workspace/startup/env/local/lib/python2.7/site-packages/requests/adapters.py", line 250, in send
raise SSLError(e)
SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol
I'm using latest 1.0rc Gevent version.
Another issue that keeps poping up time to time although URL is correct is:
Retrying (5 attempts remain) after connection broken by 'error(2, 'No such file or directory')': /ID/events?limit=5000&fields=description,name,location,start_time,end_time&access_token=TOKEN
Using the forced TLSv1 fix as suggested by J.F Sebastian fixed all the issues I was facing.
Hints for future questions regarding:
DNSError exception - upgrading Gevent from 0.13.X to 1.0rc fixes this issue
SSL issues - look at fix in link provided by J.F Sebastian
I installed the latest Python 2.7 (2.7.11) and the problem went away. I believe the problem might even be solved back in 2.7.6 (I was using 2.7.5 on Mac OSX).
I was having the same error during fetching tweets for my machine learning . Doing the pip install of the following helped me. This works:
pip install ndg-httpsclient
pip install pyopenssl
pip install pyasn1
It removed my SSLError: EOF occurred in violation of protocol (_ssl.c:590) error.
Hope it helps.