I've used two different python oauth libraries with Django to authenticate with twitter. The setup is on apache with WSGI. When I restart the server everything works great for about 10 minutes and then the httplib seems to lock up (see the following error).
I'm running only 1 process and 1 thread of WSGI but that seems to make no difference.
I cannot figure out why it's locking up and giving this CannotSendRequest error. I've spent a lot of hours on this frustrating problem. Any hints/suggestions of what it could be would be greatly appreciated.
File "/usr/lib/python2.5/site-packages/django/core/handlers/base.py", line 92, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "mypath/auth/decorators.py", line 9, in decorated
return f(*args, **kwargs)
File "mypath/auth/views.py", line 30, in login
token = get_unauthorized_token()
File "/root/storm/eye/auth/utils.py", line 49, in get_unauthorized_token
return oauth.OAuthToken.from_string(oauth_response(req))
File "mypath/auth/utils.py", line 41, in oauth_response
connection().request(req.http_method, req.to_url())
File "/usr/lib/python2.5/httplib.py", line 866, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.5/httplib.py", line 883, in _send_request
self.putrequest(method, url, **skips)
File "/usr/lib/python2.5/httplib.py", line 770, in putrequest
raise CannotSendRequest()
CannotSendRequest
This exception is raised when you reuse httplib.HTTP object for new request while you havn't called its getresponse() method for previous request. Probably there was some other error before this one that left connection in broken state. The simplest reliable way to fix the problem is creating new connection for each request, not reusing it. Sure, it will be a bit slower, but I think it's not an issue having you are running application in single process and single thread.
Also check your Python version. I had a similar situation after updating to Py-2.7 from Py-2.6. In Py-2.6 everything worked without any problems. Py-2.7 httplib uses HTTP/1.1 by default which made the server did not send back the Connection: close option in the respond, therefore the connection handling was broken. In my case this worked with HTTP/1.0 though.
http.client.CannotSendRequest: Request-sent
while using http.client module HTTPConnection class ran into the error caus my host name was incorrect
Related
I ask this question having run out of ideas - I assumed I was doing something stupid but even if I am I can't figure out what... I'm new to python having drifted away from coding 10 years ago but I was reasonably proficient once :-D I've written a python script to query a google CSE and search for relevant items, and the idea is that it will check a Podio materials list to see if they are already listed there. So at the moment all I want to do is authenticate with Podio and grab items so I can query them in my app.
I'm trying to connect to the Podio API using the prescribed method in the docs (and in questions here, and everywhere else I have looked)
At this point literally all this script is doing is this except for the credentials replaced with the actual strings, just in case I was missing some python variable madness.
import pypodio2
from pypodio2.api import OAuthClient
c = OAuthClient(
"<myappID>",
'<myAPIkey>',
'<myusername>',
'<mypassword>'
)
I'm using pythonanywhere as a sandbox and I am running the script from a bash console, I get this
16:26 ~/mysite $ python apitest.py
Traceback (most recent call last):
File "apitest.py", line 15, in <module>
'Splat_100'
File "/home/Trebuchet/.local/lib/python2.7/site-packages/pypodio2/api.py", line 13, in OAuthClient
api_key, api_secret, domain)
File "/home/Trebuchet/.local/lib/python2.7/site-packages/pypodio2/transport.py", line 41, in __init__
urlencode(body), headers=headers)
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 2135, in request
cachekey,
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1796, in _request
conn, request_uri, method, body, headers
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1701, in _conn_request
conn.connect()
File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py", line 1358, in connect
sock.connect((self.host, self.port) + sockaddr[:2])
File "/usr/local/lib/python2.7/dist-packages/httplib2/socks.py", line 496, in connect
self.__negotiatehttp(destpair[0], destpair[1])
File "/usr/local/lib/python2.7/dist-packages/httplib2/socks.py", line 457, in __negotiatehttp
raise HTTPError((statuscode, statusline[2]))
httplib2.socks.HTTPError: (403, 'Forbidden'
The password, key etc are correct. The username is correct. The clientID is correct. So I must be missing something that is obvious to people used to working with APIs. Mustn't I? Thanks :)
Free accounts on PythonAnywhere can only access http(s) sites that are on our whitelist. If the site you're using has a publicly documented API, send a link to it to PythonAnywhere support (support#pythonanywhere.com) and we'll consider add it to the whitelist.
Since we started running selenium UI tests in jenkins, we noticed a small but annoying frequency of errors during tests. We get BadStatusLine and CannotSendRequest errors on seemingly random selenium actions (click, quit, visit, etc.).
They would usually look something like:
File "/usr/lib/python2.7/unittest/case.py", line 327, in run
testMethod()
File "/home/jenkins/workspace/Create and Upload Functional Testing/shapeways/test_suite/Portal/CreateAndUpload/TestUploadWhenNotLoggedIn_ExpectLoginModal.py", line 22, in runTest
self.dw.visit(ShapewaysUrlBuilder.build_model_upload_url())
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 212, in visit
return self.execute_and_handle_webdriver_exceptions(lambda: _visit(url))
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 887, in execute_and_handle_webdriver_exceptions
return function_to_execute()
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 212, in <lambda>
return self.execute_and_handle_webdriver_exceptions(lambda: _visit(url))
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 205, in _visit
return self.driver.get(url)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 185, in get
self.execute(Command.GET, {'url': url})
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 380, in _request
resp = self._conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
raise BadStatusLine(line)
This particular case came from the following stack:
selenium==2.44.0
python==2.7.3
firefox==34.0
jenkins
xvfb (using the jenkins plugin for headless displays)
though we've seen these errors popping up all the time across many different version permutations of firefox/selenium.
I ran a tcpdump to capture the actual request sent right before the BadStatusLine error came up and got the following.
POST /hub/session/ab64574a-4a17-447a-b2e8-5b0f5ed5e923/url HTTP/1.1
Host: 127.0.0.1:41246
Accept-Encoding: identity Content-Length: 102
Connection: keep-alive
Content-type: application/json;charset="UTF-8"
POST: /hub/session/ab64574a-4a17-447a-b2e8-5b0f5ed5e923/url
Accept: application/json
User-Agent: Python http auth
{"url": "http://example.com/login", "sessionId": "ab64574a-4a17-447a-b2e8-5b0f5ed5e923"}
Response comes back with 0 bytes. So the BadStatusLine was caused by an empty response, which makes sense.
The question is, why would selenium's server return an empty response. If the server died, wouldn't we get a ConnectionError or something along those lines?
For a while, I had no repro and no idea what the cause was. I was finally able to repro by running:
import requests
import json
while True:
requests.post('http://127.0.0.1/hub/session/', data=json.dumps({"url": "http://example.com/login", "sessionId": "ab64574a-4a17-447a-b2e8-5b0f5ed5e923"}))
While this was running, I quit the browser and got a BadStatusLine error! When I tried making that request again, that's when I got the expected "ConnectionError" that you would see from any dead server.
SO, what I suspect happens is that when the browser is sent the kill signal, there is a short window during its shutdown where any response will still be returned but with 0 bytes. That's why you get different types of exceptions for essentially the same problem (browser dies). Turns out we had a cron which was killing our browsers in the background.
I'm getting some weird behavior -- when the application starts up a new instance for the first time, I get a DeadlineExceededError. When I hit refresh in the browser it works just fine And it doesn't matter which page I try. The strange thing is I can see all my debugging code just fine. In fact, I write to the log just prior to calling self.response and it shows up in the console's log. This is pretty hard to troubleshoot, since I'm not having any page load problems in the development environment, and the traceback is a bit opaque to me:
E 2013-09-29 00:10:03.975
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 267, in Handle
for chunk in result:
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1286, in appstats_wsgi_wrapper
end_recording(status)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 1410, in end_recording
rec.save()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 654, in save
key, len_part, len_full = self._save()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/appstats/recording.py", line 678, in _save
namespace=config.KEY_NAMESPACE)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 1008, in set_multi
namespace=namespace)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 907, in _set_multi_with_policy
status_dict = rpc.get_result()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result
return self.__get_result_hook(self)
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/memcache/__init__.py", line 974, in __set_with_policy_hook
rpc.check_success()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 578, in check_success
self.__rpc.CheckSuccess()
File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
raise self.exception
DeadlineExceededError
I 2013-09-29 00:10:03.988
This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.
I'm not sure how to even go about debugging this, since the error seems to be after all my code has already run.
Edit: I should add this:
I 2013-09-29 00:09:06.919
DEBUG: Writing output!
E 2013-09-29 00:10:03.975
You can see there's nearly a full minute between logging "Writing output!" just before self.response is called, and when the error occurs.
Deadlineexceedederror happens in app engine, if any request to a frontend instance does not get a response within 60 seconds. So what is happening in your case must be that when there are no running instance and your app receives a new user request, a new instance is started for processing. This will lead to an overall response time = instance startup time like library loading and initial data access + the time for processing the user request and this causes a deadlineexceeded error. Then when you are accessing your app immediately , there is an already running instance and so response time = the time for processing the user request and you do not get any error.
Please check the suggested approaches for handling deadlineexceedederror including warmup requests, which is like keeping an instance ready before arrival of a live user request.
I want to use mwclient to edit an internal wiki behind HTTPS. This page suggests that you can pass in a host tuple like this:
site = mwclient.Site(('https','wiki.whatever.com'))
However I don't see how to authenticate with Apache. Do you need to set up a urllib2 opener handler with the authentication?
A solution that works with python-wikitools would alternatively be welcome, but also has a lack of https examples.
Update:
Perhaps it uses the same user ID and password as you're using to login to the wiki? I still get an error in that case though:
Traceback (most recent call last):
File "C:\ddc\DDC_Toolbox\python_root\apps\create_new_project\trunk\create_new_project.py", line 36, in <module>
site = mwclient.Site(('https',url))
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 92, in __init__
self.site_init()
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 100, in site_init
siprop = 'general|namespaces', uiprop = 'groups|rights')
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 165, in api
info = self.raw_api(action, **kwargs)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 248, in raw_api
json_data = self.raw_call('api', data).read()
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\client.py", line 223, in raw_call
url, data = data, headers = headers)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\http.py", line 226, in post
path, headers, data)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\http.py", line 161, in post
return self.request('POST', host, path, headers, data)
File "C:\bin\Python27\lib\site-packages\mwclient-0.6.5-py2.7.egg\mwclient\http.py", line 152, in request
raise errors.HTTPStatusError, (res.status, res)
mwclient.errors.HTTPStatusError: (401, <httplib.HTTPResponse instance at 0x0241C0D0>)
Well, mwclient doesn't use urllib2, so installing an urllib2 AuthHandler won't do anything.
It's using httplib, and from a quick look at the source, there seems to be no way to add your own headers to the requests sent by the library, so adding a custom Authorization header won't work without modifying the library itself.
python-wikitools on the other hand uses urllib2, but instantiates it's own opener, and to be able to add an auth handler to that, you'd also need to modify the library-
So, unfortunately both apis don't allow you to use authentication out of the box.
I've been able to add HTTPS authentication to python-wikitools. I'll see about getting a patch uploaded to the maintainers of the package.
I have a data-intensive Python script that uses HTTP connections to download data. I usually run it overnight. Sometimes the connection will fail, or a website will be unavailable momentarily. I have basic error-handling that catches these exceptions and tries again periodically, exiting gracefully (and logging errors) after 5 minutes of retrying.
However, I've noticed that sometimes the job just freezes. No error is thrown, and the job is still running, sometimes hours after the last print message.
What is the best way to:
monitor a Python script,
detect if it is unresponsive after a given interval,
exit it if it is unresponsive,
and start another one?
UPDATE
Thank you all for your help. As a few of you have pointed out, the urllib and socket modules don't have timeouts set properly. I am using Python 2.5 with the Freebase and urllib2 modules, and catching and handling MetawebErrors and urllib2.URLErrors. Here is a sample of err output after the last script hung for 12 hours:
File "/home/matthew/dev/projects/myapp_module/project/app/myapp/contrib/freebase/api/session.py", line 369, in _httpreq_json
resp, body = self._httpreq(*args, **kws)
File "/home/matthew/dev/projects/myapp_module/project/app/myapp/contrib/freebase/api/session.py", line 355, in _httpreq
return self._http_request(url, method, body, headers)
File "/home/matthew/dev/projects/myapp_module/project/app/myapp/contrib/freebase/api/httpclients.py", line 33, in __call__
resp = self.opener.open(req)
File "/usr/lib/python2.5/urllib2.py", line 381, in open
response = self._open(req, data)
File "/usr/lib/python2.5/urllib2.py", line 399, in _open
'_open', req)
File "/usr/lib/python2.5/urllib2.py", line 360, in _call_chain
result = func(*args)
File "/usr/lib/python2.5/urllib2.py", line 1107, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.5/urllib2.py", line 1080, in do_open
r = h.getresponse()
File "/usr/lib/python2.5/httplib.py", line 928, in getresponse
response.begin()
File "/usr/lib/python2.5/httplib.py", line 385, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.5/httplib.py", line 343, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.5/socket.py", line 372, in readline
data = recv(1)
KeyboardInterrupt
You'll notice the socket error at the bottom. Since I'm using Python 2.5 and don't have access to the third urllib2.urlopen option, is there another way to watch for and catch this error? For example, I'm catching URLErrrors - is there another type of error in urllib2 or socket that I can catch which will help me?
It sounds like there is a bug in your script. The answer is not to monitor the bug, but to hunt down the bug and fix it.
We can't help you find the bug without seeing some code. But as a general idea, you might want to use logging to pinpoint where the problem is occurring, and write unit tests to help you build confidence about which parts of your code do not have the bug.
Another idea is to break your "stuck" program with Ctrl-C and to study the traceback message. It will show you what line your program was last executing.
That may give you a clue where the script is going wrong.
Since the program is doing web communication, I'd fire up a debugging proxy like Charles http://www.charlesproxy.com/ and see if there's anything kooky happening in the back-and-forth between your script and the server.
Also consider that the socket module has no timeout set by default and therefore can hang. As of python 2.6, however, you can pass a third argument to urllib2.urlopen (if you are using urllib2, that is), specifying a request timeout period in seconds. That way the script will error out rather than go catatonic waiting from a response from a perhaps uncooperative server. If you haven't already, I'd check these things out before trying anything more elaborate.
Update for python 2.5:
To do this in python < 2.6, you would have to set the timeout value directly in the socket module, which urllib2 uses. I haven't tried this, but it presumably works. Found this info at http://www.voidspace.org.uk/python/articles/urllib2.shtml:
import socket
import urllib2
# timeout in seconds
timeout = 10
socket.setdefaulttimeout(timeout)
# this call to urllib2.urlopen now uses the default timeout
# we have set in the socket module
req = urllib2.Request('http://www.voidspace.org.uk')
response = urllib2.urlopen(req)
a simple way to do what you ask is to make use of UDP packets sent by your current program to another harvesting program that monitors the output. If it doesn't receive a packet in a certain amount of time, it kills the other python process then restarts another one
You could run your script in pdb and break in when you suspect it's frozen. It won't work on its own, but might help you figure out why it's freezing.