Let's say I have a (websocket) API, api.py, as such:
from flask import Flask, request
from flask_socketio import SocketIO, emit
from worker import Worker
app = Flask()
socketio = SocketIO(app)
worker = Worker()
worker.start()
#socketio.on('connect')
def connect():
print("Client", request.sid, "connected")
#socketio.on('get_results')
def get_results(query):
"""
The only endpoing of the API.
"""
print("Client", request.sid, "requested results for query", query)
# Set the worker to work, wait for results to be ready, and
# send the results back to the client.
worker.task_queue.put(query)
results = worker.result_queue.get()
emit("results", results)
#socketio.on('disconnect')
def disconnect():
print("Client", request.sid, "disconnected, perhaps before results where ready")
# What to do here?
socketio.run(app, host='')
The a API will serve many clients but only has a single worker to produce the results that should be served. worker.py:
from multiprocessing import Process, Queue
class Worker(Process):
def __init__(self):
super().__init__()
self.task_queue = Queue()
self.result_queue = Queue()
self.some_stateful_variable = 0
# Do other computationally expensive work
def reset_state(self):
# Computationally inexpensive.
pass
def do_work(self, task):
# Computationally expensive. Takes long time.
# Modifies internal state.
pass
def run(self):
while True:
task = self.task_queue.get()
results = self.do_work(task)
self.result_queue.put(results)
The worker gets a request, i.e. a task to do, and sets forth producing a result. When the result is ready, the client will be served it.
But not all clients are patient. They may leave, i.e. disconnect from the API, before the results are ready. They don't want them, and the worker therefore ends up working on a task that does not need to finish. That makes other client in queue wait unnecessarily. How to avoid this situation, and get the worker to abort executing do_work for a task that does not need to finish?
In client side: when user closes browser tab or leave the page send request to your Flask server, the request should contain id of the task you would like to cancel.
In server side put cancel status for the task in database or any shared variable between Flask Server and your Worker Process
Divide Task processing in several stages and check status of task in database before each stage, if status is canceled - stop the task processing.
Another choice for point 1 is to do some monitoring in Server side in separate Process - count interval between status requests from client side.
I've handled similar problems by launching an entirely separate process via:
sp.call('start python path\\worker.py', shell=True)
worker.py would then report its PID back to the api.py via redis, then its straightforward to kill the process at any point from api.py
Of course, how viable that is for you will depend on how much data resides within api.py and is shared to worker.py - whether its feasible for that to also pass via redis or not is for you to decide.
The added benefit is you decouple socket from heavy compute - and you can go quasi-multi-core (single thread per worker.py). You could go full multi core by incorporating multiprocessing into each worker.py if you wished.
Related
I have a long-running background task that spins the flask app again and to do some auditing in the background. The front end is a web application and uses socketio to communicate with the backend main flask app to handle multiple async behaviors.
I make sure to only fire the background task when the main thread is created and I do eventlet.monkey_patch() only at the very beginning of the script.
if the background thread has a lot of stuff to audit, it blocks the main thread, the more stuff in memory, the longer it blocks the main thread. The audit is not CPU intensive at all, it's just some db inserts and logging.
The items that need to be audited are added to an object in memory from the main thread and are passed by reference to the child thread. (Like a in memory queue)
If I don't monkey patch eventlet, then everything works fine but then flask's auto reload won't work, and I need it for development.
I run the app like socketio.run(app) in dev
Behavior persists when using gunicorn/eventlet
When the background task is sleeping sleep(2), there's no block happening.
import eventlet
eventlet.monkey_patch()
# ... rest of code is a basic flask app create_app factory that at some # point starts the new thread if it's the main thread
# the async code that runs is the following
class AsyncAuditor(threading.Thread):
def __init__(self, tasks: list, stop: threading.Event):
super().__init__()
self.tasks = tasks
self.stop_event = stop
def run(self):
from app import init_app
from dal import db
app = init_app(mode='sys')
app.logger.info('starting async audit thread')
with app.app_context():
try:
while not self.stop_event.is_set():
if len(self.tasks) > 0:
task: dict
for task in self.tasks:
app.logger.debug(threading.current_thread().name + ' new audit record')
task.payload = encryptor.encrypt(task.payload)
task.ip = struct.unpack("!I", socket.inet_aton(task.ip))[0]
db.session.add(task)
self.tasks.clear()
db.session.commit()
sleep(2)
app.logger.info('exiting async audit thread')
except BaseException as e:
app.logger.exception('Exception')
# there's some code that tries to gracefully exit if app needs to exit
stop_event = threading.Event()
async_task = AsyncAuditor(API.audit_tasks, stop_event)
async_task.start()
def exit_async_thread():
stop_event.set()
async_task.join()
atexit.register(exit_async_thread)
I expect that while the child thread is working, the main thread would not be blocked by any db operations, in fact, like I mentioned before, if I don't monkey patch eventlet, then everything works fine in the main thread and the child one as well. Instead, I'm getting 9 and even 30 seconds delays when hitting an endpoint in the flask application while the background task is working.
Problem Outline
I have a python flask server where one of the endpoints has a moderate amount of work to do (the real code reads, resizes and returns an image). I want to optimise the endpoint so that it can be called multiple times in parallel.
The code I currently have (shown below) does not work because it relies on passing a multiprocessing.Event object through a multiprocessing.JoinableQueue which is not allowed and results in the following error:
RuntimeError: Condition objects should only be shared between processes
through inheritance
How can I use a separate process to compute some jobs and notify the main thread when a specific job is complete?
Proof of Concept
Flask can be multithreaded so if one request is waiting on a result other threads can continue to process other requests. I have a basic proof of concept here that shows that parallel requests can be optimised using multiprocessing: https://github.com/alanbacon/flask_multiprocessing
The example code on github spawns a new process for every request which I understand has considerable overheads, plus I've noticed that my proof-of-concept server crashes if there are more than 10 or 20 concurrent requests, I suspect this is because there are too many processes being spawned.
Current Attempt
I have tried to create a set of workers that pick jobs off a queue. When a job is complete the result is written to a shared memory area. Each job contains the work to be done and an Event object that can be set when the job is complete to signal the main thread.
Each request thread passes in a job with a newly created Event object, it then immediately waits on that event before returning the result. While one server request thread is waiting the server is able to use other threads to continue to serve other requests.
The problem as mentioned above is that Event objects can not be passed around in this way.
What approach should I take to circumvent this problem?
from flask import Flask, request, Response,
import multiprocessing
import uuid
app = Flask(__name__)
# flask config
app.config['PROPAGATE_EXCEPTIONS'] = True
app.config['DEBUG'] = False
def simpleWorker(complexity):
temp = 0
for i in range(0, complexity):
temp += 1
mgr = multiprocessing.Manager()
results = mgr.dict()
joinableQueue = multiprocessing.JoinableQueue()
lock = multiprocessing.Lock()
def mpWorker(joinableQueue, lock, results):
while True:
next_task = joinableQueue.get() # blocking call
if next_task is None: # poison pill to kill worker
break
simpleWorker(next_task['complexity']) # pretend to do heavy work
result = next_task['val'] * 2 # compute result
ID = next_task['ID']
with lock:
results[ID] = result # output result to shared memory
next_task['event'].set() # tell main process result is calculated
joinableQueue.task_done() # remove task from queue
#app.route("/work/<ID>", methods=['GET'])
def work(ID=None):
if request.method == 'GET':
# send a task to the consumer and wait for it to finish
uid = str(uuid.uuid4())
event = multiprocessing.Event()
# pass event to job so that job can tell this thread when processing is
# complete
joinableQueue.put({
'val': ID,
'ID': uid,
'event': event,
'complexity': 100000000
})
event.wait() # wait for result to be calculated
# get result from shared memory area, and clean up
with lock:
result = results[ID]
del results[ID]
return Response(str(result), 200)
if __name__ == "__main__":
num_consumers = multiprocessing.cpu_count() * 2
consumers = [
multiprocessing.Process(
target=mpWorker,
args=(joinableQueue, lock, results))
for i in range(num_consumers)
]
for c in consumers:
c.start()
host = '127.0.0.1'
port = 8080
app.run(host=host, port=port, threaded=True)
I have designed a REST API which receives inputs through POST requests and then applies some logic to the inputs and returns to the callback uri which is part of the inputs.
This design was working fine for single input, but then i want to implement multithreading so that i can handle multiple POST requests. I have tried using 'app.run(threaded=True)' but was not successful.
I am running this code on linux platform. Not sure what is wrong in the following code, and am not so good at using threads in python, would appreciate if someone can let me know where the issue is:
I am able to get the '200' response once there is a POST request and the inputs are appended to 'inp_params', after which there is no processing in the thread.
from flask import Flask, jsonify, request
import time
import json
import os
import threading
import Queue
import test_func_module as tf
app = Flask(__name__)
inp_params = []
# Create the queue and threader
q = Queue.Queue()
#app.route('/', methods = ['GET', 'POST'] )
def get_data():
if request.method == 'GET':
return 'RESTful API'
elif request.method == 'POST':
global inp_params
inputs = {"fileName": request.json["fileName"], "fileId": request.json["fileId"], "ModuleId": request.json["ModuleId"], "WorkflowId": request.json["WorkflowId"],"Language": request.json["Language"], "callbackuri": request.json["callbackuri"]}
inp_params.append(inputs)
return '200'
def test_integrate(worker):
TF_output = tf.test_func(worker)
return TF_output
def threader():
while True:
# gets an worker from the queue
worker = q.get()
# Run the example job with the avail worker in queue (thread)
test_integrate(worker)
# completed with the job
q.task_done()
if __name__ == '__main__':.
for worker in inp_params:
q.put(worker)
for x in range(4): #4 cores
t = threading.Thread(target=threader)
# classifying as a daemon, so they will die when the main dies
t.daemon = True
# begins, must come after daemon definition
t.start()
# wait until the thread terminates.
q.join()
app.run(threaded=True)
#Shilparani Since you mentioned
I have tried using 'app.run(threaded=True)' but was not successful.
May not be exact answer for your question but I would like to share my experience for achieving concurrency through uwsgi/gunicorn :
Keep it simple by coding Flask for REST endpoints and move Multithreading , MultiProcessing logic to gunicorn or uwsgi where you can mention threads and workers which help for achieving concurrency , parallelism if that's what you are trying to achieve.
gunicorn -b localhost:8080 -w 4 -t 4 app:app
Based on your need and operations:
If tasks are CPU intensive try to keep #workers as #CPU-cores
If tasks are I/O intensive may be safe to try with more threads
I have a Tornado web application, this app can receive GET and POST request from the client.
The POSTs request put an information received in a Tornado Queue, then I pop this information from the queue and with it I do an operation on the database, this operation can be very slow, it can take several seconds to complete!
In the meantime that this database operation goes on I want to be able to receive other POSTs (that put other information in the queue) and GET. The GET are instead very fast and must return to the client their result immediatly.
The problem is that when I pop from the queue and the slow operation begin the server doesn't accept other requests from the client. How can I resolve this?
This is the semplified code I have written so far (import are omitted for avoid wall of text):
# URLs are defined in a config file
application = tornado.web.Application([
(BASE_URL, Variazioni),
(ARTICLE_URL, Variazioni),
(PROMO_URL, Variazioni),
(GET_FEEDBACK_URL, Feedback)
])
class Server:
def __init__(self):
http_server = tornado.httpserver.HTTPServer(application, decompress_request=True)
http_server.bind(8889)
http_server.start(0)
transactions = TransactionsQueue() #contains the queue and the function with interact with it
IOLoop.instance().add_callback(transactions.process)
def start(self):
try:
IOLoop.instance().start()
except KeyboardInterrupt:
IOLoop.instance().stop()
if __name__ == "__main__":
server = Server()
server.start()
class Variazioni(tornado.web.RequestHandler):
''' Handle the POST request. Put an the data received in the queue '''
#gen.coroutine
def post(self):
TransactionsQueue.put(self.request.body)
self.set_header("Location", FEEDBACK_URL)
class TransactionsQueue:
''' Handle the queue that contains the data
When a new request arrive, the generated uuid is putted in the queue
When the data is popped out, it begin the operation on the database
'''
queue = Queue(maxsize=3)
#staticmethod
def put(request_uuid):
''' Insert in the queue the uuid in postgres format '''
TransactionsQueue.queue.put(request_uuid)
#gen.coroutine
def process(self):
''' Loop over the queue and load the data in the database '''
while True:
# request_uuid is in postgres format
transaction = yield TransactionsQueue.queue.get()
try:
# this is the slow operation on the database
yield self._load_json_in_db(transaction )
finally:
TransactionsQueue.queue.task_done()
Moreover I don't understand why if I do 5 POST in a row, it put all five data in the queue though the maximun size is 3.
I'm going to guess that you use a synchronous database driver, so _load_json_in_db, although it is a coroutine, is not actually async. Therefore it blocks the entire event loop until the long operation completes. That's why the server doesn't accept more requests until the operation is finished.
Since _load_json_in_db blocks the event loop, Tornado can't accept more requests while it's running, so your queue never grows to its max size.
You need two fixes.
First, use an async database driver written specifically for Tornado, or run database operations on threads using Tornado's ThreadPoolExecutor.
Once that's done your application will be able to fill the queue, so second, TransactionsQueue.put must do:
TransactionsQueue.queue.put_nowait(request_uuid)
This throws an exception if there are already 3 items in the queue, which I think is what you intend.
This is a probably basic question, but I have not been able to find the answer.
I have a long-running process that produces data every few minutes that I would like the client to receive as soon as it is ready. Currently I have the long-running process in a Task Queue, and it adds channel messages to another Task Queue from within a for loop. The client successfully receives the channel messages and downloads the data using a get request; however, the messages are being sent from the task queue after the long-running process finishes (after about 10 minutes) instead of when the messages are added to the task queue.
How can I have the messages in the task queue sent immediately? Do I need to have the for loop broken into several tasks? The for loop creates a number of dictionaries I think I would need to post to the data store and then retrieve for the next iteration (does not seem like an ideal solution), unless there is an easier way to return data from a task.
When I do not add the messages to a Task Queue and send the messages directly in the for loop, the server does not seem to respond to the client's get request for the data (possibly due to the for loop of the long-running process blocking the response?)
Here is a simplified version of my server code:
from google.appengine.ext import db
from google.appengine.api import channel
from google.appengine.api import taskqueue
from google.appengine.api import rdbms
class MainPage(webapp2.RequestHandler):
def get(self):
## This opens the GWT app
class Service_handler(webapp2.RequestHandler):
def get(self, parameters):
## This is called by the GWT app and generates the data to be
## sent to the client.
#This adds the long-process to a task queue
taskqueue.Task(url='/longprocess/', params = {'json_request': json_request}).add(queue_name='longprocess-queue')
class longprocess_handler(webapp2.RequestHandler):
def post(self):
#This has a for loop that recursively uses data in dictionaries to
#produce kml files every few minutes
for j in range(0, Time):
# Process data
# Send message to client using a task queue to send the message.
taskqueue.Task(url='/send/', params).add(queue_name=send_queue_name)
class send_handler(webapp2.RequestHandler):
def post(self):
# This sends the message to the client
# This is currently not happening until the long-process finishes,
# but I would like it to occur immediately.
class kml_handler(webapp2.RequestHandler):
def get(self, client_id):
## When the client receives the message, it picks up the data here.
app = webapp2.WSGIApplication([
webapp2.Route(r'/', handler=MainPage),
webapp2.Route(r'/Service/', handler=Service_handler),
webapp2.Route(r'/_ah/channel/<connected>/', handler = connection_handler),
webapp2.Route(r'/longprocess/', handler = longprocess_handler),
webapp2.Route(r'/kml/<client_id>', handler = kml_handler),
webapp2.Route(r'/send/', handler = send_handler)
],
debug=True)
Do I need to break up the long-process into tasks that send and retrieve results from the data store in order to have the send_handler execute immediately, or am I missing something? Thanks
The App Engine development server only processes one request at a time. In production, these things will occur simultaneously. Try in production, and check that things behave as expected there.
There's also not much reason to use a separate task to send the channel messages in production - just send them directly from the main task.