We Keep Coding

Python Threading vs Multiprocessing to improve REST API responsiveness "fire and forget" tasks

I am somewhat new to both threading and multiprocessing in Python, as well as dealing with the concept of the GIL. I have a situation where I have time consuming fire and forget tasks that I need the server to run, but the server should immediately reply to the client and basically be like "okay, your thing was submitted" so that the client does not hang waiting for the thing to complete. An example of what one of the "things" might do is pull down some data from a database or two, compare that data, and then write the result to another database. The databases are remote, not locally on the same host as the server itself. Another example, is crunching some data and then sending a text as a result of that. The client does not care about the data, but someone will receive a text later with some information that is the result of the data crunching from the various dictionaries and database entries. However, there could be many such requests pouring in from many clients. The goal here is to spawn a thread, or process that essentially kills itself because we don't care at all about returning any data from it.
At a glance, my understanding is that both multiprocessing and threading can achieve similar results for this use case. My main concerns are that I can immediately launch the function to go do its own thing and return to the client quickly so it does not hang. There are many, many requests coming in simultaneously from many, many clients in this scenario. As a result, my understanding is that multiprocessing may be better, so that these tasks would not need to be executed as sequential threads because of the GIL. However, I am unsure of how to make the processes end themselves when they are done with their task rather than needing to wait for them.
An example of the problem
#route('/api/example', methods=["POST"])
def example_request(self, request):
request_data = request.get_json()
crunch_data_and_send_text(request_data) # Takes maybe 5-10 seconds, doesn't return data
return # Return to client. Would like to return to client immediately rather than waiting
Would threading or multiprocessing be better here? And how can I make the process (or thread) .join() itself effectively when it is done rather than needing to join it before I can return to the client.
I have also considered asyncio which I think would allow something that would also improve this, but the existing codebase I have inherited is so large that it is infeasible to rewrite in async for the time being, and library replacements may need to be found in that case, so it is not an option.
#Threading
from threading import Thread
#route('/api/example', methods=["POST"])
def example_request(self, request):
request_data = request.get_json()
fire_and_forget = Thread(target = crunch_data_and_send_text, args=(request_data,))
fire_and_forget.start()
return # Return to client. Would like to return to client immediately rather than waiting
# Multiprocessing
from multiprocessing import Process
#route('/api/example', methods=["POST"])
def example_request(self, request):
request_data = request.get_json()
fire_and_forget = Process(target = crunch_data_and_send_text, args=(request_data,))
fire_and_forget.start()
return # Return to client. Would like to return to client immediately rather than waiting
Which of these is better for this use case? Is there a way I can have them .join() themselves automatically when they finish rather than needing to actually sit here in the function and wait for them to complete before returning to the client?
To be clear, asyncio is unfortunately NOT an option for me.

I suggest using Advance Python Scheduler.
Instead of running your function in a thread, schedule it to run and immediately return to client.
After setting up your flask app, setup Flask-APScheduler and then schedule your function to run in the background.
from apscheduler.schedulers.background import BackgroundScheduler
scheduler = BackgroundScheduler({
--- setup the scheduler ---
})
#route('/api/example', methods=["POST"])
def example_request(self, request):
request_data = request.get_json()
job = scheduler.add_job(crunch_data_and_send_text, 'date', run_date=datetime.utcnow())
return "The request is being processed ..."
to pass arguments to crunch_data_and_send_text you can do:
lambda: crunch_data_and_send_text(request_data)
here is the User Guide

Returning a status code early for a time consuming function in Flask

I have an API route that calls a function (ie: doSomethingForALongTime) that takes some time to finish. If we assume that the function doesn't return anything to the client, is there a work around to just call the function and send the status code 200 while the function is doing it's job?
#application.route('/api', methods=['GET'])
def api_route():
doSomethingForALongTime()
return 200

Yes, with some caveats. The usual way to handle this is to use a 'task queue', such as
celery or
rq
(there's a walk-through of how to use rq in chapter 22 of the
flask mega tutorial). The approaches require that, at least, you have redis running, and are running separate worker processes.
The idea is hand a task off to the task queue in your handler (route), then return a response to the browser while a worker in a separate process picks the task up from the queue and runs it.
It's also possible to run a 'worker thread' in your app, and have the handler queue up work for it. I have a proof-of-concept for that here, with the caveat that I've only used it for personal apps. The caveat is that this is only really suitable for a personal webapp.

time.sleep, Flask and I/O wait

When using time.sleep(), will a Flask request be blocked?
One of my Flask endpoint launches a long processing subtask, and in some cases, instead of doing the work asynchronously, it is possible to wait for the completion of the task and return the result in the same request.
In this case, my Flask app starts the process, then waits for it to complete before returning the result. My issue here, is that while calling something like (simplified):
while True:
if process_is_done():
break
time.sleep(1)
Will Flask will block that request until it is done, or will it allow for other requests to come in the meantime?

Yes, that request is entirely blocked. time.sleep() does not inform anything of the sleep, it just 'idles' the CPU for the duration.
Flask is itself not asynchronous, it has no concept of putting a request handler on hold and giving other requests more time. A good WSGI server will use threads and or multiple worker processes to achieve concurrency, but this one request is blocked and taking up CPU time all the same.

Design of asynchronous request and blocking processing using Tornado

I'm trying to implement a Python app that uses async functions to receive and emit messages using NATS, using a client based on Tornado. Once a message is received, a blocking function must be called, that I'm trying to implement on a separate thread, to allow the reception and publication of messages to put messages in a Tornado queue for later processing of the blocking function.
I'm very new to Tornado (and to python multithreading), but after reading several times the Tornado documentation and other sources, I've been able to put up a working version of the code, that looks like this:
import tornado.gen
import tornado.ioloop
from tornado.queues import Queue
from concurrent.futures import ThreadPoolExecutor
from nats.io.client import Client as NATS
messageQueue = Queue()
nc = NATS()
#tornado.gen.coroutine
def consumer():
def processMessage(currentMessage):
# process the message ...
while True:
currentMessage = yield messageQueue.get()
try:
# execute the call in a separate thread to prevent blocking the queue
EXECUTOR.submit(processMessage, currentMessage)
finally:
messageQueue.task_done()
#tornado.gen.coroutine
def producer():
#tornado.gen.coroutine
def enqueueMessage(currentMessage):
yield messageQueue.put(currentMessage)
yield nc.subscribe("new_event", "", enqueueMessage)
#tornado.gen.coroutine
def main():
tornado.ioloop.IOLoop.current().spawn_callback(consumer)
yield producer()
if __name__ == '__main__':
main()
tornado.ioloop.IOLoop.current().start()
My questions are:
1) Is this the correct way of using Tornado to call a blocking function?
2) What's the best practice for implementing a consumer/producer scheme that is always listening? I'm afraid my while True: statement is actually blocking the processor...
3) How can I inspect the Queue to make sure a burst of calls is being enqueued? I've tried using Queue().qSize(), but it always returns zero, which makes me wonder if the enqueuing is done correctly or not.

General rule (credits to NYKevin) is:
multiprocessing for CPU- and GPU-bound computations.
Event-driven stuff for non-blocking I/O (which should be preferred over blocking I/O where possible, since it scales much more effectively).
Threads for blocking I/O (you can also use multiprocessing, but the per-process overhead probably isn't worth it).
ThreadPoolExecutor for IO, ProcessPoolExecutor for CPU. Both have internal queue, both scale to at most specified max_workers. More info about concurrent executors in docs.
So answer are:
Reimplementing pool is an overhead. Thread or Process depends on what you plan to do.
while True is not blocking if you have e.g. some yielded async calls (even yield gen.sleep(0.01)), it gives back control to ioloop
qsize() is the right to call, but since I have not run/debug this and I would take a different approach (existing pool), it is hard to find a problem here.

Implementing a custom queue in ZeroRPC

Currently, I am using ZeroRPC, I have "workers" connect to the "server" and do the work that the server sends them.
Currently calls are made over ZeroRPC as soon as there is a call to make, as far as I can tell it uses a FIFO queue.
I would like to use my own queue so that I throttle/prioritize the calls.
I'm hoping that ZeroRPC exposes a gevent Event that triggers when its internal queue runs empty.

What you want to do, is create your own work queue in your server. And dispatch yourself the calls in the priorities you wish.
Since few lines of code express more than any vampire story in 3 volumes, lets see in pseudo code what the server could look like:
myqueue = MySuperBadAssQueue()
def myqueueprocessor():
for request in myqueue: # blocks until next request
gevent.spawn(request.processme) # do the job asynchronously
gevent.spawn(myqueueprocessor) # do that at startup
class Server:
def dosomething(args...blabla...): # what users are calling
request = Request(args...blabla...)
myqueue.put(request) # something to do buddy!
return request.future.get() # return when request is completed
# (can also raise an exception)
# An example of what a request could look like:
class Request:
def __init__(self, ....blablabla...):
self.future = gevent.AsyncResult()
def process():
try:
result = someworker(self.args*) # call some worker
self.future.set(result) # complete the initial request
except Exception as e:
self.future.set_exception(e)
Its up to MySuperBadAssQueue to do all the smart work, throttle if you want, cancel out a request with an exception if necessary, etc...
ZeroRPC does not expose any event to let you know if its 'internal' queue runs
empty:
In fact, there is no explicit queue in ZeroRPC. What happens, is
simply first come first serve, and the exact order depend both of
ZeroMQ and the Gevent IOLoop (libevent or libev depending of the
version). It happens that in practice, this conveniently plays
like a FIFO queue.

I haven't tried this myself, but I have read through the source. I am motivated because I want to do this myself.
Seems like what you would do is inherit zerorpc.Server and override the _acceptor method. According to the source, _acceptor is what receives messages and then spawns threads to run them. So if you change up the logic/loop to incorporate your queue, you can use that to throttle.