I have designed a REST API which receives inputs through POST requests and then applies some logic to the inputs and returns to the callback uri which is part of the inputs.
This design was working fine for single input, but then i want to implement multithreading so that i can handle multiple POST requests. I have tried using 'app.run(threaded=True)' but was not successful.
I am running this code on linux platform. Not sure what is wrong in the following code, and am not so good at using threads in python, would appreciate if someone can let me know where the issue is:
I am able to get the '200' response once there is a POST request and the inputs are appended to 'inp_params', after which there is no processing in the thread.
from flask import Flask, jsonify, request
import time
import json
import os
import threading
import Queue
import test_func_module as tf
app = Flask(__name__)
inp_params = []
# Create the queue and threader
q = Queue.Queue()
#app.route('/', methods = ['GET', 'POST'] )
def get_data():
if request.method == 'GET':
return 'RESTful API'
elif request.method == 'POST':
global inp_params
inputs = {"fileName": request.json["fileName"], "fileId": request.json["fileId"], "ModuleId": request.json["ModuleId"], "WorkflowId": request.json["WorkflowId"],"Language": request.json["Language"], "callbackuri": request.json["callbackuri"]}
inp_params.append(inputs)
return '200'
def test_integrate(worker):
TF_output = tf.test_func(worker)
return TF_output
def threader():
while True:
# gets an worker from the queue
worker = q.get()
# Run the example job with the avail worker in queue (thread)
test_integrate(worker)
# completed with the job
q.task_done()
if __name__ == '__main__':.
for worker in inp_params:
q.put(worker)
for x in range(4): #4 cores
t = threading.Thread(target=threader)
# classifying as a daemon, so they will die when the main dies
t.daemon = True
# begins, must come after daemon definition
t.start()
# wait until the thread terminates.
q.join()
app.run(threaded=True)
#Shilparani Since you mentioned
I have tried using 'app.run(threaded=True)' but was not successful.
May not be exact answer for your question but I would like to share my experience for achieving concurrency through uwsgi/gunicorn :
Keep it simple by coding Flask for REST endpoints and move Multithreading , MultiProcessing logic to gunicorn or uwsgi where you can mention threads and workers which help for achieving concurrency , parallelism if that's what you are trying to achieve.
gunicorn -b localhost:8080 -w 4 -t 4 app:app
Based on your need and operations:
If tasks are CPU intensive try to keep #workers as #CPU-cores
If tasks are I/O intensive may be safe to try with more threads
Related
Let's say I have a (websocket) API, api.py, as such:
from flask import Flask, request
from flask_socketio import SocketIO, emit
from worker import Worker
app = Flask()
socketio = SocketIO(app)
worker = Worker()
worker.start()
#socketio.on('connect')
def connect():
print("Client", request.sid, "connected")
#socketio.on('get_results')
def get_results(query):
"""
The only endpoing of the API.
"""
print("Client", request.sid, "requested results for query", query)
# Set the worker to work, wait for results to be ready, and
# send the results back to the client.
worker.task_queue.put(query)
results = worker.result_queue.get()
emit("results", results)
#socketio.on('disconnect')
def disconnect():
print("Client", request.sid, "disconnected, perhaps before results where ready")
# What to do here?
socketio.run(app, host='')
The a API will serve many clients but only has a single worker to produce the results that should be served. worker.py:
from multiprocessing import Process, Queue
class Worker(Process):
def __init__(self):
super().__init__()
self.task_queue = Queue()
self.result_queue = Queue()
self.some_stateful_variable = 0
# Do other computationally expensive work
def reset_state(self):
# Computationally inexpensive.
pass
def do_work(self, task):
# Computationally expensive. Takes long time.
# Modifies internal state.
pass
def run(self):
while True:
task = self.task_queue.get()
results = self.do_work(task)
self.result_queue.put(results)
The worker gets a request, i.e. a task to do, and sets forth producing a result. When the result is ready, the client will be served it.
But not all clients are patient. They may leave, i.e. disconnect from the API, before the results are ready. They don't want them, and the worker therefore ends up working on a task that does not need to finish. That makes other client in queue wait unnecessarily. How to avoid this situation, and get the worker to abort executing do_work for a task that does not need to finish?
In client side: when user closes browser tab or leave the page send request to your Flask server, the request should contain id of the task you would like to cancel.
In server side put cancel status for the task in database or any shared variable between Flask Server and your Worker Process
Divide Task processing in several stages and check status of task in database before each stage, if status is canceled - stop the task processing.
Another choice for point 1 is to do some monitoring in Server side in separate Process - count interval between status requests from client side.
I've handled similar problems by launching an entirely separate process via:
sp.call('start python path\\worker.py', shell=True)
worker.py would then report its PID back to the api.py via redis, then its straightforward to kill the process at any point from api.py
Of course, how viable that is for you will depend on how much data resides within api.py and is shared to worker.py - whether its feasible for that to also pass via redis or not is for you to decide.
The added benefit is you decouple socket from heavy compute - and you can go quasi-multi-core (single thread per worker.py). You could go full multi core by incorporating multiprocessing into each worker.py if you wished.
Problem Outline
I have a python flask server where one of the endpoints has a moderate amount of work to do (the real code reads, resizes and returns an image). I want to optimise the endpoint so that it can be called multiple times in parallel.
The code I currently have (shown below) does not work because it relies on passing a multiprocessing.Event object through a multiprocessing.JoinableQueue which is not allowed and results in the following error:
RuntimeError: Condition objects should only be shared between processes
through inheritance
How can I use a separate process to compute some jobs and notify the main thread when a specific job is complete?
Proof of Concept
Flask can be multithreaded so if one request is waiting on a result other threads can continue to process other requests. I have a basic proof of concept here that shows that parallel requests can be optimised using multiprocessing: https://github.com/alanbacon/flask_multiprocessing
The example code on github spawns a new process for every request which I understand has considerable overheads, plus I've noticed that my proof-of-concept server crashes if there are more than 10 or 20 concurrent requests, I suspect this is because there are too many processes being spawned.
Current Attempt
I have tried to create a set of workers that pick jobs off a queue. When a job is complete the result is written to a shared memory area. Each job contains the work to be done and an Event object that can be set when the job is complete to signal the main thread.
Each request thread passes in a job with a newly created Event object, it then immediately waits on that event before returning the result. While one server request thread is waiting the server is able to use other threads to continue to serve other requests.
The problem as mentioned above is that Event objects can not be passed around in this way.
What approach should I take to circumvent this problem?
from flask import Flask, request, Response,
import multiprocessing
import uuid
app = Flask(__name__)
# flask config
app.config['PROPAGATE_EXCEPTIONS'] = True
app.config['DEBUG'] = False
def simpleWorker(complexity):
temp = 0
for i in range(0, complexity):
temp += 1
mgr = multiprocessing.Manager()
results = mgr.dict()
joinableQueue = multiprocessing.JoinableQueue()
lock = multiprocessing.Lock()
def mpWorker(joinableQueue, lock, results):
while True:
next_task = joinableQueue.get() # blocking call
if next_task is None: # poison pill to kill worker
break
simpleWorker(next_task['complexity']) # pretend to do heavy work
result = next_task['val'] * 2 # compute result
ID = next_task['ID']
with lock:
results[ID] = result # output result to shared memory
next_task['event'].set() # tell main process result is calculated
joinableQueue.task_done() # remove task from queue
#app.route("/work/<ID>", methods=['GET'])
def work(ID=None):
if request.method == 'GET':
# send a task to the consumer and wait for it to finish
uid = str(uuid.uuid4())
event = multiprocessing.Event()
# pass event to job so that job can tell this thread when processing is
# complete
joinableQueue.put({
'val': ID,
'ID': uid,
'event': event,
'complexity': 100000000
})
event.wait() # wait for result to be calculated
# get result from shared memory area, and clean up
with lock:
result = results[ID]
del results[ID]
return Response(str(result), 200)
if __name__ == "__main__":
num_consumers = multiprocessing.cpu_count() * 2
consumers = [
multiprocessing.Process(
target=mpWorker,
args=(joinableQueue, lock, results))
for i in range(num_consumers)
]
for c in consumers:
c.start()
host = '127.0.0.1'
port = 8080
app.run(host=host, port=port, threaded=True)
I've been pulling my hair out trying to figure this one out, hoping someone else has already encountered this and knows how to solve it :)
I'm trying to build a very simple Flask endpoint that just needs to call a long running, blocking php script (think while true {...}). I've tried a few different methods to async launch the script, but the problem is my browser never actually receives the response back, even though the code for generating the response after running the script is executed.
I've tried using both multiprocessing and threading, neither seem to work:
# multiprocessing attempt
#app.route('/endpoint')
def endpoint():
def worker():
subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)
p = multiprocessing.Process(target=worker)
print '111111'
p.start()
print '222222'
return json.dumps({
'success': True
})
# threading attempt
#app.route('/endpoint')
def endpoint():
def thread_func():
subprocess.Popen('nohup php script.php &', shell=True, preexec_fn=os.setpgrp)
t = threading.Thread(target=thread_func)
print '111111'
t.start()
print '222222'
return json.dumps({
'success': True
})
In both scenarios I see the 111111 and 222222, yet my browser still hangs on the response from the endpoint. I've tried p.daemon = True for the process, as well as p.terminate() but no luck. I had hoped launching a script with nohup in a different shell and separate processs/thread would just work, but somehow Flask or uWSGI is impacted by it.
Update
Since this does work locally on my Mac when I start my Flask app directly with python app.py and hit it directly without going through my Nginx proxy and uWSGI, I'm starting to believe it may not be the code itself that is having issues. And because my Nginx just forwards the request to uWSGI, I believe it may possibly be something there that's causing it.
Here is my ini configuration for the domain for uWSGI, which I'm running in emperor mode:
[uwsgi]
protocol = uwsgi
max-requests = 5000
chmod-socket = 660
master = True
vacuum = True
enable-threads = True
auto-procname = True
procname-prefix = michael-
chdir = /srv/www/mysite.com
module = app
callable = app
socket = /tmp/mysite.com.sock
This kind of stuff is the actual and probably main use case for Python Celery (https://docs.celeryproject.org/). As a general rule, do not run long-running jobs that are CPU-bound in the wsgi process. It's tricky, it's inefficient, and most important thing, it's more complicated than setting up an async task in a celery worker. If you want to just prototype you can set the broker to memory and not using an external server, or run a single-threaded redis on the very same machine.
This way you can launch the task, call task.result() which is blocking, but it blocks in an IO-bound fashion, or even better you can just return immediately by retrieving the task_id and build a second endpoint /result?task_id=<task_id> that checks if result is available:
result = AsyncResult(task_id, app=app)
if result.state == "SUCCESS":
return result.get()
else:
return result.state # or do something else depending on the state
This way you have a non-blocking wsgi app that does what is best suited for: short time CPU-unbound calls that have IO calls at most with OS-level scheduling, then you can rely directly to the wsgi server workers|processes|threads or whatever you need to scale the API in whatever wsgi-server like uwsgi, gunicorn, etc. for the 99% of workloads as celery scales horizontally by increasing the number of worker processes.
This approach works for me, it calls the timeout command (sleep 10s) in the command line and lets it work in the background. It returns the response immediately.
#app.route('/endpoint1')
def endpoint1():
subprocess.Popen('timeout 10', shell=True)
return 'success1'
However, not testing on WSGI server, but just locally.
Would it be enough to use a background task? Then you only need to import threading e.g.
import threading
import ....
def endpoint():
"""My endpoint."""
try:
t = BackgroundTasks()
t.start()
except RuntimeError as exception:
return f"An error occurred during endpoint: {exception}", 400
return "successful.", 200
return "successfully started.", 200
class BackgroundTasks(threading.Thread):
def run(self,*args,**kwargs):
...do long running stuff
I need a webserver which routes the incoming requests to back-end workers by batching them every 0.5 second or when it has 50 http requests whichever happens earlier. What will be a good way to implement it in python/tornado or any other language?
What I am thinking is to publish the incoming requests to a rabbitMQ queue and then somehow batch them together before sending to the back-end servers. What I can't figure out is how to pick multiple requests from the rabbitMq queue. Could someone point me to right direction or suggest some alternate apporach?
I would suggest using a simple python micro web framework such as bottle. Then you would send the requests to a background process via a queue (thus allowing the connection to end).
The background process would then have a continuous loop that would check your conditions (time and number), and do the job once the condition is met.
Edit:
Here is an example webserver that batches the items before sending them to any queuing system you want to use (RabbitMQ always seemed overcomplicated to me with Python. I have used Celery and other simpler queuing systems before). That way the backend simply grabs a single 'item' from the queue, that will contain all required 50 requests.
import bottle
import threading
import Queue
app = bottle.Bottle()
app.queue = Queue.Queue()
def send_to_rabbitMQ(items):
"""Custom code to send to rabbitMQ system"""
print("50 items gathered, sending to rabbitMQ")
def batcher(queue):
"""Background thread that gathers incoming requests"""
while True:
batcher_loop(queue)
def batcher_loop(queue):
"""Loop that will run until it gathers 50 items,
then will call then function 'send_to_rabbitMQ'"""
count = 0
items = []
while count < 50:
try:
next_item = queue.get(timeout=.5)
except Queue.Empty:
pass
else:
items.append(next_item)
count += 1
send_to_rabbitMQ(items)
#app.route("/add_request", method=["PUT", "POST"])
def add_request():
"""Simple bottle request that grabs JSON and puts it in the queue"""
request = bottle.request.json['request']
app.queue.put(request)
if __name__ == '__main__':
t = threading.Thread(target=batcher, args=(app.queue, ))
t.daemon = True # Make sure the background thread quits when the program ends
t.start()
bottle.run(app)
Code used to test it:
import requests
import json
for i in range(101):
req = requests.post("http://localhost:8080/add_request",
data=json.dumps({"request": 1}),
headers={"Content-type": "application/json"})
How to run a given module given I want to run some functions concurrently that are not necessarily using routing (could be daemon services) while at the same time running the app server?
For example:
#some other route functions app.post(...)
#some other concurrent functions
def alarm():
'''
Run this service every X duration
'''
ALARM = 21
try:
while 1:
#checking time and doing something. Then finding INTERVAL
gevent.sleep(INTERVAL)
except KeyboardInterrupt,e:
print 'exiting'
Do I have to use the above like this after main ?
gevent.joinall(gevent.spawn(alarm))
app.run(....)
or
gevent.joinall((gevent.spawn(alarm),gevent.spawn(app.run)))
The objective is run these alarm like daemon services, do their work and snooze while rest of service operations work as usual.
The server should start concurrently as well. correct me if im not on the right track.
Gevent comes with it's own WSGI servers, so it is really not necessary to use app.run. The servers are:
gevent.pywsgi.WSGIServer
gevent.wsgi.WSGIServer
Both provide the same interface.
You can use these to achieve what you want:
import gevent
import gevent.monkey
gevent.monkey.patch_all()
import requests
from gevent.pywsgi import WSGIServer
# app = YourBottleApp
def alarm():
'''
Run this service every X duration
'''
ALARM = 21
while 1:
#checking time and doing something. Then finding INTERVAL
gevent.sleep(INTERVAL)
if __name__ == '__main__':
http_server = WSGIServer(('', 8080), app)
srv_greenlet = gevent.spawn(http_server.serve_forever)
alarm_greenlet = gevent.spawn(alarm)
try:
gevent.joinall([srv_greenlet, alarm_greenlet])
except KeyboardInterrupt:
http_server.stop()
print 'Quitting'