I'm trying to write a tornado web application that uses sqlalchemy in some request handlers. These handlers have two parts: one that takes a long time to complete, and another that uses sqlalchemy and is relatively fast.
I would like to make the slow part of the request asynchronous, but not the sqlalchemy part. Can I do something like the following code and be safe?
class ExampleHandler(BaseHandler):
async def post(self):
loop = asyncio.get_event_loop()
await loop.run_in_executor(...) # very slow (no sqlalchemy here)
with self.db_session() as s: # sqlalchemy session
s.add(...)
s.commit()
self.render(...)
The idea is to have sqlalchemy still blocking, but have the computational heavy part not blocking the application.
The tornado web server uses asynchronous code to get around the limit of the python Global Interpreter Lock. The GIL, as it is colloquially known, allows only one thread of execution to take place in the python interpreter process. Tornado is able to answer many requests simultaneously because of its use of an event loop. The event loop can perform one small task at a time. Let's take your own post handler to understand this better.
In this handler, when the python interpreter gets to the await keyword, it pauses the execution of the function and queues it for later on its event loop. It then checks the event loop to respond to other events that may have queued up there, like responding to a new connection or servicing another handler.
When you block in an asynchronous function, you freeze the entire event loop as it is unable to pause your function and service anything else. What this actually means for you is that your web server will not accept or service any requests while your async function blocks. It will appear as if your web server is hanging and indeed it is stuck.
To keep the server responsive, you have to find a way to execute your sqlalchemy query in an asynchronous non-blocking manner.
Related
I have a minimal async python server based on aiohttp.
It is very straightforward, just a websocket endpoint exposed as in
#routes.get('/my_endpoint')
async def my_func(request):
ws = web.WebSocketResponse()
await ws.prepare(request)
return ws
I want to expose as prometheus metrics the request rate (and potentially the error rate).
After performing a brief investigation on the topic, I realised that it seems like there is a distinction between approaching prometheus metrics exposure when it comes to sync vs async apps.
For my case, where I want a simple request count/rate, is there a reason not to just use the plain' old prometheus python client (e.g by simply decorating my_func?)
Would the request count actually fail in such a case?
The following is based on my understanding on asyncio and the way the official prometheus client describes how it exposes metrics.
aiohttp is to be used on top of asyncio. Now, asyncio is running something called an "event loop" which runs inside a single thread (usually the main thread)
You can look at it as an entity that decides to suspend or execute functions that were assigned to run in the loop. In your case my_func.
For prometheus_client to expose your metrics you will probably need to run it in a different thread
Metrics are usually exposed over HTTP, to be read by the Prometheus server. The easiest way to do this is via start_http_server, which will start a HTTP server in a daemon thread on the given port
This is outside "the control of the event loop" which might lead to performance issues and to unexpected behavior as a result. So the request count might not fail, but if for some reason its doing some blocking task (I/O) it will block the main thread as well. If you'd use the async approach and run it as part of the event loop your blocking task can be awaited and give back the control to the main thread.
There are open source projects that support prometheus in async functions such as aioprometheus and prometheus-async.
When using time.sleep(1) before sendMessage, the hole process stops (even the others connections).
def handleConnected(self):
print self.address, 'connected'
for client in clients:
time.sleep(1)
client.sendMessage(self.address[0] + u' - connected')
Server: https://github.com/dpallot/simple-websocket-server
How to solve it?
The server that you are using is a synchronous, "select" type server. These servers use a single process and a single thread, they achieve concurrency through the use of the select() function to efficiently wait for I/O on multiple socket connections.
The advantage of select servers is that they can easily scale to very large number of clients. The disadvantage is that when the server invokes an application handler (the handleConnected(), handleMessage() and handleClose() methods for this server), the server blocks on them, meaning that while the handlers are running the server is suspended, because both the handlers and the server run on the same thread. The only way for the server to be responsive in this type of architecture is to code the handlers in such a way that they do what they need to do quickly and return control back to the server.
Your handleConnected handler function is not a good match for this type of server, because it is a long running function. This function will run for several seconds (as many seconds as there are clients), so during all that time the server is going to be blocked.
You can maybe work around the limitations in this server by creating a background thread for your long running task. That way your handler can return back to the server after launching the thread. The server will then regain control and go back to work, while the background thread does that loop with the one second sleeps inside. The only problem you have to consider is that now you have sort of a home-grown multithreaded server, so you will not be able to scale as easily.
Another option for you to consider is to use a different server architecture. A coroutine based server will support your handler function as you coded it, for example. The two servers that I recommend in this category are eventlet and gevent. The eventlet server comes with native WebSocket support. For gevent you have to install an extension called gevent-websocket.
Good luck!
You are suspending the thread with sleep and the server which you are using seems to be using select to handle the requests not threads. So no other request will be able to be handled.
So you can't use time.sleep.
Why do you need to sleep? Can you solve it some other way?
Maybe you can use something like threading.Timer()
def sendHello(client):
client.sendMessage("hello, world")
for client in clients:
t = Timer(1.0, lambda: sendHello(client))
t.start() # after 30 seconds, "hello, world" will be printed
This is off the top of my head. You would also need a way to cancel each timer so I guess you would need to save each t in a list and call it when done.
Are there any pure wsgi implementation of background task?
I want to use local variables under the same context directly, not serialize/deserialize to another daemon process via a broker.
Is it possible to make this happen under the current wsgi infrastructure? E.g. after return response yield, run some callback functions?
This is a duplicate of question asked on the Python WEB-SIG. I reference the same page as provided in response to the question on the Python WEB-SIG so others can see it:
http://code.google.com/p/modwsgi/wiki/RegisteringCleanupCode
In doing this though, it ties up the request thread and so it would not be able to handle other requests until your task has finished.
Creating background threads at the end of a request is not a good idea unless you do it using a pooling mechanism such that you limit the number of worker threads for your tasks. Because the process can crash or be shutdown, you loose the job as only in memory and thus not persistent.
Better to use Celery, or if you think that is too heavy weight, have a look at Redis Queue (RQ) instead.
You could look at Django async. It uses an in-database queue and so handles transactions much better. All arguments need to be JSONable as does the return type. In some cases this means you may need to schedule a wrapper function, but that oughtn't to cause you any headaches.
http://pypi.python.org/pypi/django-async
You don't want to be doing this sort of thing inside the web server -- it's absolutely not the right place to do it. Django async provides a manage.py command for flushing the queue which you can run in a loop, possible on another machine from the web server.
We're trying to use the new python 2.7 threading ability in Google App Engine and it seems like the created thread is getting killed before it finishes running. Our scenario:
User sends a message to the server
We update the user's data
We spawn a thread to do some more heavy duty processing
We return a response to the user before waiting for the heavy duty processing to finish
My assumption was that the thread would continue to run after the request had returned, as long as it did not exceed the total request time limit. What we're seeing though is that the thread is randomly killed partway through it's execution. No exceptions, no errors, nothing. It just stops running.
Are threads allowed to exist after the response has been returned? This does not repro on the dev server, only on live servers.
We could of course use a task queue instead, but that's a real pain since we'd have to set up a url for the action and serialize/deserialize the data.
The 'Sandboxing' section of this page:
http://code.google.com/appengine/docs/python/python27/using27.html#Sandboxing
indicates that threads cannot run past the end of the request.
Deferred tasks are the way to do this. You don't need a URL or serialization to use them:
from google.appengine.ext import deferred
deferred.defer(myfunction, arg1, arg2)
I'm porting one of my projects from C# and am having trouble solving a multithreading issue in Python. The problem relates to a long-lived HTTP request, which is expected (the request will respond when a certain event occurs on the server). Here's the summary:
I send the request using urllib2 on a separate thread. When the request returns or times out, the main thread is notified. This works fine. However, there are cases where I need to abort this outstanding request and switch to a different URL. There are four solutions that I can consider:
Abort the outstanding request. C# has WebRequest.Abort(), which I can call cross-thread to abort the request. Python urllib2.Request appears to be a pure data class, in that instances only store request information; responses are not connected to Request objects. So I can't do this.
Interrupt the thread. C# has Thread.Interrupt(), which will raise a ThreadInterruptedException in the thread if it is in a wait state, or the next time it enters such a state. (Waiting on a monitor and file/socket I/O are both waiting states.) Python doesn't seem to have anything comparable; there does not appear to be a way to wake up a thread that is blocked on I/O.
Set a low timeout on the request. On a timeout, check an "aborted" flag. If it's false, restart the request.
Similar to option 3, add an "aborted" flag to the state object so that when the request does finally end in one way or another, the thread knows that the response is no longer needed and just shuts itself down.
Options 3 and 4 seem to be the only ones supported by Python, but option 3 is a horrible solution and 4 will keep open a connection I don't need. I am hoping to be a good netizen and close this connection when I no longer need it. Is there any way to actually abort the outstanding request, one way or another?
Consider using gevent. Gevent uses non-thread cooperating units of execution called greenlets. Greenlets can "block" on IO, which really means "go to sleep until the IO is ready". You could have a requester greenlet that owns the socket and a main greenlet that decides when to abort. When you want to abort and switch URLs the main greenlet kills the requester greenlet. The requester catches the resulting exception, closes its socket/urllib2 request, and starts over.
Edited to add: Gevent is not compatible with threads, so be careful with that. You'll have to either use gevent all the way or threads all the way. Threads in python are kinda lame anyway because of the GIL.
Similar to Spike Gronim's answer, but even more heavy handed.
Consider rewriting this in twisted. You probably would want to subclass twisted.web.http.HTTPClient, in particular implementing handleResponsePart to do your client interaction (or handleResponseEnd if you don't need to see it before the response ends). To close the connection early, you just call the loseConnection method on the client protocol.
Maybe this snippet of "killable thread" could be useful to you if you have no other choice. But i would have the same opinion as Spike Gronim and recommend using gevent.
I found this question using google and used Spike Gronim's answer to come up with:
from gevent import monkey
monkey.patch_all()
import gevent
import requests
def post(*args, **kwargs):
if 'stop_event' in kwargs:
stop_event = kwargs['stop_event']
del kwargs['stop_event']
else:
stop_event = None
req = gevent.spawn(requests.post, *args, **kwargs)
while req.value is None:
req.join(timeout=0.1)
if stop_event and stop_event.is_set():
req.kill()
break
return req.value
I thought it might be useful for other people as well.
It works just like a regular request.post, but takes an extra keyword argument 'stop_event'. This is a threading.Event. The request will abort if the stop_event gets set.
Use with caution, because if it's not waiting for either the connection or the communitation, it can block GIL (as mentioned). It (gevent) does seem compatible with threading these days (through monkey patch).