I build a short url translator engine in Python, and I'm seeing a TON of "broken pipe" errors, and I'm curious how to trap it best when using the BaseHTTPServer classes. This isn't the entire code, but gives you an idea of what I'm doing so far:
from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
import memcache
class clientThread(BaseHTTPRequestHandler):
def do_GET(self):
content = None
http_code,response_txt,long_url = \
self.ag_trans_url(self.path,content,'GET')
self.http_output( http_code, response_txt, long_url )
return
def http_output(self,http_code,response_txt,long_url):
self.send_response(http_code)
self.send_header('Content-type','text/plain')
if long_url:
self.send_header('Location', long_url)
self.end_headers()
if response_txt:
self.wfile.write(response_txt)
return
def ag_trans_url(self, orig_short_url, post_action, getpost):
short_url = 'http://foo.co' + orig_short_url
# fetch it from memcache
long_url = mc.get(short_url)
# other magic happens to look it up from db if there was nothing
# in memcache, etc
return (302, None, log_url)
def populate_memcache()
# connect to db, do lots of mc.set() calls
def main():
populate_memcache()
try:
port = 8001
if len(sys.argv) > 1:
port = int(sys.argv[1])
server = HTTPServer(('',port), clientThread)
#server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
print '[',str(datetime.datetime.now()),'] short url processing has begun'
server.serve_forever()
except KeyboardInterrupt,SystemExit:
print '^C received, shutting down server'
server.socket.close()
The code itself works great, but started throwing errors almost immediately when in production:
Traceback (most recent call last):
File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
self.handle()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 310, in handle_one_request
method()
File "/opt/short_url_redirector/shorturl.py", line 38, in do_GET
self.http_output( http_code, response_txt, long_url )
File "/opt/short_url_redirector/shorturl.py", line 52, in http_output
self.send_response(http_code)
File "/usr/lib/python2.5/BaseHTTPServer.py", line 370, in send_response
self.send_header('Server', self.version_string())
File "/usr/lib/python2.5/BaseHTTPServer.py", line 376, in send_header
self.wfile.write("%s: %s\r\n" % (keyword, value))
File "/usr/lib/python2.5/socket.py", line 274, in write
self.flush()
File "/usr/lib/python2.5/socket.py", line 261, in flush
self._sock.sendall(buffer)
error: (32, 'Broken pipe')
The bulk of these errors seem to stem from having a problem calling the send_header() method where all I'm writing out is this:
self.send_header('Location', long_url)
So I'm curious where in my code to try to trap for this IO exception... do I write try/except calls around each of the self.send_header/self.end_headers/self.wfile.write calls? The other error I see from time to time is this one, but not sure which exception to watch to even catch this:
Traceback (most recent call last):
File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
self.process_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
self.handle()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
self.handle_one_request()
File "/usr/lib/python2.5/BaseHTTPServer.py", line 299, in handle_one_request
self.raw_requestline = self.rfile.readline()
File "/usr/lib/python2.5/socket.py", line 381, in readline
data = self._sock.recv(self._rbufsize)
error: (104, 'Connection reset by peer')
This appears to be a bug in SocketServer, see this link Python Bug: 14574
A fix (works for me in Python 2.7) is to override the SocketServer.StreamRequestHandler finish() method, something like this:
...
def finish(self,*args,**kw):
try:
if not self.wfile.closed:
self.wfile.flush()
self.wfile.close()
except socket.error:
pass
self.rfile.close()
#Don't call the base class finish() method as it does the above
#return SocketServer.StreamRequestHandler.finish(self)
The "broken pipe" exception means that your code tried to write to a socket/pipe which the other end has closed. If the other end is a web browser, the user could have stopped the request. You can ignore the traceback; it does not indicate a serious problem. If you want to suppress the message, you can put a try ... except block around all of the code in your http_output function, and log the exception if you like.
Additionally, if you want your HTTP server to process more than one request at a time, you need your server class to use one of the SocketServer.ForkingMixIn and SocketServer.ThreadingMixIn classes. Check the documentation of the SocketServer module for details.
Add: The "connection reset by peer" exception means that your code tried to read from a dead socket. If you want to suppress the traceback, you will need to extend the BaseHTTPServer class and override the handle_one_request method to add a try ... except block. You will need a new server class anyway, to implement the earlier suggestion about processing more than one request at a time.
In my application, the error didn't occur in finish(), it occurred in handle(). This fix catches the broken pipe errors:
class MyHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
...
def handle(self):
try:
BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
except socket.error:
pass
Related
So in my flask app right now I am using Celery to deploy servers on remote machines. Right now, I have an enum, status, which indicates the lifecycle of my deployment process:
#celery.task(bind=True)
def deploy_server(self, server_id):
server = Server.query.get(server_id)
if not server.can_launch():
return
try:
server.status = RemoteStatus.LAUNCHING
db.session.commit()
verify_DNS(server)
host = server.server.ssh_user + '#' + server.server.ip
execute(fabric_deploy_server, self, server, hosts=host)
server.status = RemoteStatus.LAUNCHED
db.session.commit()
except Exception as e:
server.status = RemoteStatus.ERROR
db.session.commit()
traceback.print_exc()
raise e
As you can see, when a server is being deployed, its status is changed to "Launching". If there is an exception, it will be changed to ERROR.
I found one exception which completely bypasses this bloc of code: when I overloaded my celery server with too many requests, I get this exception:
[2017-07-09 18:00:03,127: WARNING/PoolWorker-3] /app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py:542: RuntimeWarning: Exception raised outside body: ConnectionError('max number of clients reached',):
Traceback (most recent call last):
File "/app/.heroku/python/lib/python2.7/site-packages/celery/app/trace.py", line 427, in trace_task
uuid, retval, task_request, publish_result,
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 152, in mark_as_done
self.store_result(task_id, result, state, request=request)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 309, in store_result
request=request, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/base.py", line 652, in _store_result
self.set(self.get_key_for_task(task_id), self.encode(meta))
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 204, in set
return self.ensure(self._set, (key, value), **retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 194, in ensure
**retry_policy)
File "/app/.heroku/python/lib/python2.7/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/app/.heroku/python/lib/python2.7/site-packages/celery/backends/redis.py", line 213, in _set
pipe.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2641, in execute
return execute(conn, stack, raise_on_error)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/client.py", line 2495, in _execute_transaction
connection.send_packed_command(all_cmds)
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 538, in send_packed_command
self.connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 446, in connect
self.on_connect()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 514, in on_connect
if nativestr(self.read_response()) != 'OK':
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 577, in read_response
response = self._parser.read_response()
File "/app/.heroku/python/lib/python2.7/site-packages/redis/connection.py", line 255, in read_response
raise error
ConnectionError: max number of clients reached
exc, exc_info.traceback)))
My biggest problem with this is that this error is raised somewhere outside of my Try/Catch bloc. Hence, when this exception occurs, all my servers remain in the "Launching" mode rather than "Error".
How can I catch this exception and modify Server.status?
In redis 2.4 there is a hard coded limit of max number of connections which is 10,000. In redis 2.6+ you can specify the max number of clients in redis.conf. also this is not a problem of celery your broker redis refused to accept connections that's the problem.
Set the max number of clients that can be handled by redis simultaneously using redis CLI. Check out redis clients
I have a program that streams prices and is getting a badstatusline error during slow hours. This causes issues with other files that need to interact with the stream. I am having much trouble simply catching the exceptions, leading to other exceptions that I cannot catch for some reason BadStatusLine leads to CannotSendRequest leads to ResponseNotReady. How can I simply restart (in this case) trading.py when execution.py raises the exception BadStatusLine?
Here is how I'm handling it now..
while True:
try:
response = self.conn.getresponse().read()
print response
except Exception:
pass
else:
break
Its a stream using Httplib if thats of importance
Thanks for the help
Here is the error as well:
Exception in thread Thread-1:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/Users/mattduhon/trading4.py", line 30, in trade
execution.execute_order(event)
File "/Users/mattduhon/execution.py", line 34, in execute_order
response = self.conn.getresponse().read()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1073, in getresponse
response.begin()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 415, in begin
version, status, reason = self._read_status()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 379, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
If you are file continuously then you can put it in supervisor and add
auto_start = True
Or In your code you can do something like that
import os
while True:
try:
response = self.conn.getresponse().read()
print response
except:
os.system("python trading.py")
I added broad exception because you don't know which exception is occuring
Create another script to run your main script, and try and except the whole thing:
try:
execfile('main.py')
except:
pass
So my twisted mail receiver is working nicely. Right up until we try to handle a case where the config is fubarred, and a mismatched cert/key is passed to the certificate options object for the factory.
I have a module, custom_esmtp.py, which includes an overload of ext_STARTLS(self,rest) which I have modified as follows, to include a try/except:
elif self.ctx and self.canStartTLS:
try:
self.sendCode(220, 'Begin TLS negotiation now')
self.transport.startTLS(self.ctx)
self.startedTLS = True
except:
log.err()
self.sendCode(550, "Internal server error")
return
When I run the code, having passed a cert and key that do not match, I get the following call stack:
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
self.lineReceived(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 568, in lineReceived
return getattr(self, 'state_' + self.mode)(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 582, in state_COMMAND
method('')
--- <exception caught here> ---
File "custom_esmtp.py", line 286, in ext_STARTTLS
self.transport.startTLS(self.ctx)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 179, in startTLS
startTLS(self, ctx, normal, FileDescriptor)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 139, in startTLS
tlsFactory = TLSMemoryBIOFactory(contextFactory, client, None)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 769, in __init__
contextFactory = _ContextFactoryToConnectionFactory(contextFactory)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 648, in __init__
oldStyleContextFactory.getContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1429, in getContext
self._context = self._makeContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1439, in _makeContext
ctx.use_privatekey(self.privateKey)
OpenSSL.SSL.Error: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
Line 286 of custom_esmtp.py is the self.transport.startTLS(self.ctx). I've looked through all the twisted modules listed in the stack, at the quoted lines, and there are no other try/except blocks.... So my understanding is that the error should be passed back up the stack, unhandled, until it reaches my handler in custom_esmtp.py? So why is it not getting handled - especially since the only except I have is a "catch all"?
Thanks in advance!
If you want this error to be caught, you can do:
from OpenSSL import SSL
# ...
try:
# ...
except SSL.Error:
# ...
Perhaps the syntax changes a bit. I can't check because I don't use this precise package, but the idea is that you have to declare the import path of the exceptions you want to catch.
I am trying to make an application that serves a simple HTML form to the user and then calls a function when the user submits the form. It uses wsgiref.simple_server to serve the HTML. The server is encountering an error and I can't understand why. The code is as follows:
#!/usr/bin/python3
from wsgiref.simple_server import make_server
from wsgiref.util import setup_testing_defaults
import webbrowser # open user's web browser to url when server is run
from sys import exc_info
from traceback import format_tb
# Easily serves an html form at path_to_index with style at path_to_style
# Calls on_submit when the form is submitted, passing a dictionary with key
# value pairs { "input name" : submitted_value }
class SimpleServer:
def __init__(self, port=8000, on_submit=None, index_path="./index.html", css_path="./style.css"):
self.port = port
self.on_submit = on_submit
self.index_path = index_path
self.css_path = css_path
# Forwards request to proper method, or returns 404 page
def wsgi_app(self, environ, start_response):
urls = [
(r"^$", self.index),
(r"404$", self.error_404),
(r"style.css$", self.css)
]
path = environ.get("PATH_INFO", "").lstrip("/")
# Call another application if they called a path defined in urls
for regex, application in urls:
match = re.search(regex, path)
# if the match was found, return that page
if match:
environ["myapp.url_args"] = match.groups()
return application(environ, start_response)
return error_404(environ, start_response)
# Gives the user a form to submit all their input. If the form has been
# submitted, it sends the ouput of self.on_submit(user_input)
def index(self, environ, start_response):
# user_input is a dictionary, with keys from the names of the fields
user_input = parse_qs(environ['QUERY_STRING'])
# return either the form or the calculations
index_html = open(self.index_path).read()
body = index_html if user_input == {} else calculate(user_input)
mime_type = "text/html" if user_input == {} else "text/plain"
# return the body of the message
status = "200 OK"
headers = [ ("Content-Type", mime_type),
("Content-Length", str(len(body))) ]
start_response(status, headers)
return [body.encode("utf-8")]
def start_form(self):
httpd = make_server('', self.port, ExceptionMiddleware(self.wsgi_app))
url = "http://localhost:" + str(self.port)
print("Visit " + url)
# webbrowser.open(url)
httpd.serve_forever()
if __name__ == "__main__":
server = SimpleServer()
server.start_form()
When I run it, I get the error
127.0.0.1 - - [16/Dec/2014 21:15:57] "GET / HTTP/1.1" 500 0
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 138, in run
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 266, in write
"write() argument must be a bytes instance"
AssertionError: write() argument must be a bytes instance
127.0.0.1 - - [16/Dec/2014 21:15:57] "GET / HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 49354)
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 138, in run
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 266, in write
"write() argument must be a bytes instance"
AssertionError: write() argument must be a bytes instance
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/wsgiref/handlers.py", line 141, in run
self.handle_error()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 368, in handle_error
self.finish_response()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 180, in finish_response
self.write(data)
File "/usr/lib/python3.4/wsgiref/handlers.py", line 274, in write
self.send_headers()
File "/usr/lib/python3.4/wsgiref/handlers.py", line 331, in send_headers
if not self.origin_server or self.client_is_modern():
File "/usr/lib/python3.4/wsgiref/handlers.py", line 344, in client_is_modern
return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.4/socketserver.py", line 305, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 331, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python3.4/socketserver.py", line 344, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python3.4/socketserver.py", line 669, in __init__
self.handle()
File "/usr/lib/python3.4/wsgiref/simple_server.py", line 133, in handle
handler.run(self.server.get_app())
File "/usr/lib/python3.4/wsgiref/handlers.py", line 144, in run
self.close()
File "/usr/lib/python3.4/wsgiref/simple_server.py", line 35, in close
self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'
This output doesn't actually include the script I am running, which I am confused about. Any thoughts?
Just to register the solution for this issue, the problem is with len() function.
str(len(body))
It calculate the wrong size and when return the server Content-Length, then it wait more bytes that needed.
Thus, always send bytes using a buffer with UTF-8, follow example:
from io import StringIO
stdout = StringIO()
print("Hello world!", file=stdout)
start_response("200 OK", [('Content-Type', 'text/plain; charset=utf-8')])
return [stdout.getvalue().encode("utf-8")]
Looking at your code I don't see a direct reason for this error. However, I would strongly advise that unless you're trying to learn how wsgi works (or implement your own framework), you should use an existing micro-framework. WSGI is NOT meant to be used directly by applications. It provides a very thin interface between Python and a web server.
A nice and light framework is bottle.py -- I use it for all Python webapps. But there are many many others, look for "Non Full-Stack Frameworks" in https://wiki.python.org/moin/WebFrameworks.
A nice advantage of bottle is that it's a single file, which makes it easy to distribute with your server.
I am working on a Django project. All went well till I created an Ajax request to send values from the html page to the backend (views.py).
When I send the data using Ajax, I am able to view the values being passed to views.py, and it even reaches the render_to_response method and displays my page, but throws the broken pipe error in the terminal. I don't see any kind of disruption to the program, but I wanted to know if there is a way to prevent this error from occurring. I checked the other responses. But no luck so far.
When I try to hit submit again on the refreshed page, I get this message:
The page that you're looking for used information that you entered. Returning to that page might cause any action you took to be repeated. Do you want to continue? [Submit] [Cancel]`
Here is the dump:
Traceback (most recent call last):
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 34812)
----------------------------------------
File "/usr/lib/python2.7/dist-packages/django/core/servers/basehttp.py", line 284, in run
self.finish_response()
File "/usr/lib/python2.7/dist-packages/django/core/servers/basehttp.py", line 324, in finish_response
self.write(data)
File "/usr/lib/python2.7/dist-packages/django/core/servers/basehttp.py", line 403, in write
self.send_headers()
File "/usr/lib/python2.7/dist-packages/django/core/servers/basehttp.py", line 467, in send_headers
self.send_preamble()
File "/usr/lib/python2.7/dist-packages/django/core/servers/basehttp.py", line 385, in send_preamble
'Date: %s\r\n' % http_date()
File "/usr/lib/python2.7/socket.py", line 324, in write
self.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
self.process_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
self.finish_request(request, client_address)
File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
self.RequestHandlerClass(request, client_address, self)
File "/usr/lib/python2.7/dist-packages/django/core/servers/basehttp.py", line 570, in __init__
BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
self.finish()
File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
self.wfile.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
Update:
Here is the code that I am sending:
$( document ).ready(function() {
$.csrftoken();
$("#submitdata").click(function(){
//values = [tmode, fmode, t_cool, t_heat, hold];
values = {
"tmode": tmode,
"fmode": fmode,
"t_cool": t_cool,
"t_heat": t_heat,
"hold": hold
};
var jsonText = JSON.stringify(values);
$.ajax({
url: "/submitdata/",
type: 'POST',
data: jsonText,
dataType: 'json',
success:function(data){
console.log(data.success);
},
complete:function(){
console.log('complete');
},
error:function (xhr, textStatus, thrownError){
console.log(thrownError);
console.log(obj);
}
});
});
});
And here is my views.py:
#login_required
def submitvalues(request):
#context = RequestContext(request)
if request.POST:
jsonvalues = json.loads(request.raw_post_data)
print jsonvalues
return HttpResponse(json.dumps(dict(status='updated')), mimetype="application/json")
I am still facing the same issue. Can someone help me with this?
Edit on 5/28/2014:
I just figured out the reason for a Broken Pipe. It was because I was not sending back the response from Python and was just expecting the page to refresh automatically. I am a newbie to all of this, and took me a while to figure out why this happened.
You haven't posted any code, but this is probably because you have triggered the Ajax request on a button submit but haven't prevented the default action. So the Ajax request is made, but by the time it comes to return the data, the browser has already requested the next page anyway, so there is nothing to receive it.
I have solved this problem by adding this:
self.send_header("Access-Control-Allow-Origin", "*")
Because I found some error on sending post request page:
No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is there
Then I got this solution and solved above problem.