Problem Outline
I have a python flask server where one of the endpoints has a moderate amount of work to do (the real code reads, resizes and returns an image). I want to optimise the endpoint so that it can be called multiple times in parallel.
The code I currently have (shown below) does not work because it relies on passing a multiprocessing.Event object through a multiprocessing.JoinableQueue which is not allowed and results in the following error:
RuntimeError: Condition objects should only be shared between processes
through inheritance
How can I use a separate process to compute some jobs and notify the main thread when a specific job is complete?
Proof of Concept
Flask can be multithreaded so if one request is waiting on a result other threads can continue to process other requests. I have a basic proof of concept here that shows that parallel requests can be optimised using multiprocessing: https://github.com/alanbacon/flask_multiprocessing
The example code on github spawns a new process for every request which I understand has considerable overheads, plus I've noticed that my proof-of-concept server crashes if there are more than 10 or 20 concurrent requests, I suspect this is because there are too many processes being spawned.
Current Attempt
I have tried to create a set of workers that pick jobs off a queue. When a job is complete the result is written to a shared memory area. Each job contains the work to be done and an Event object that can be set when the job is complete to signal the main thread.
Each request thread passes in a job with a newly created Event object, it then immediately waits on that event before returning the result. While one server request thread is waiting the server is able to use other threads to continue to serve other requests.
The problem as mentioned above is that Event objects can not be passed around in this way.
What approach should I take to circumvent this problem?
from flask import Flask, request, Response,
import multiprocessing
import uuid
app = Flask(__name__)
# flask config
app.config['PROPAGATE_EXCEPTIONS'] = True
app.config['DEBUG'] = False
def simpleWorker(complexity):
temp = 0
for i in range(0, complexity):
temp += 1
mgr = multiprocessing.Manager()
results = mgr.dict()
joinableQueue = multiprocessing.JoinableQueue()
lock = multiprocessing.Lock()
def mpWorker(joinableQueue, lock, results):
while True:
next_task = joinableQueue.get() # blocking call
if next_task is None: # poison pill to kill worker
break
simpleWorker(next_task['complexity']) # pretend to do heavy work
result = next_task['val'] * 2 # compute result
ID = next_task['ID']
with lock:
results[ID] = result # output result to shared memory
next_task['event'].set() # tell main process result is calculated
joinableQueue.task_done() # remove task from queue
#app.route("/work/<ID>", methods=['GET'])
def work(ID=None):
if request.method == 'GET':
# send a task to the consumer and wait for it to finish
uid = str(uuid.uuid4())
event = multiprocessing.Event()
# pass event to job so that job can tell this thread when processing is
# complete
joinableQueue.put({
'val': ID,
'ID': uid,
'event': event,
'complexity': 100000000
})
event.wait() # wait for result to be calculated
# get result from shared memory area, and clean up
with lock:
result = results[ID]
del results[ID]
return Response(str(result), 200)
if __name__ == "__main__":
num_consumers = multiprocessing.cpu_count() * 2
consumers = [
multiprocessing.Process(
target=mpWorker,
args=(joinableQueue, lock, results))
for i in range(num_consumers)
]
for c in consumers:
c.start()
host = '127.0.0.1'
port = 8080
app.run(host=host, port=port, threaded=True)
Related
Let's say I have a (websocket) API, api.py, as such:
from flask import Flask, request
from flask_socketio import SocketIO, emit
from worker import Worker
app = Flask()
socketio = SocketIO(app)
worker = Worker()
worker.start()
#socketio.on('connect')
def connect():
print("Client", request.sid, "connected")
#socketio.on('get_results')
def get_results(query):
"""
The only endpoing of the API.
"""
print("Client", request.sid, "requested results for query", query)
# Set the worker to work, wait for results to be ready, and
# send the results back to the client.
worker.task_queue.put(query)
results = worker.result_queue.get()
emit("results", results)
#socketio.on('disconnect')
def disconnect():
print("Client", request.sid, "disconnected, perhaps before results where ready")
# What to do here?
socketio.run(app, host='')
The a API will serve many clients but only has a single worker to produce the results that should be served. worker.py:
from multiprocessing import Process, Queue
class Worker(Process):
def __init__(self):
super().__init__()
self.task_queue = Queue()
self.result_queue = Queue()
self.some_stateful_variable = 0
# Do other computationally expensive work
def reset_state(self):
# Computationally inexpensive.
pass
def do_work(self, task):
# Computationally expensive. Takes long time.
# Modifies internal state.
pass
def run(self):
while True:
task = self.task_queue.get()
results = self.do_work(task)
self.result_queue.put(results)
The worker gets a request, i.e. a task to do, and sets forth producing a result. When the result is ready, the client will be served it.
But not all clients are patient. They may leave, i.e. disconnect from the API, before the results are ready. They don't want them, and the worker therefore ends up working on a task that does not need to finish. That makes other client in queue wait unnecessarily. How to avoid this situation, and get the worker to abort executing do_work for a task that does not need to finish?
In client side: when user closes browser tab or leave the page send request to your Flask server, the request should contain id of the task you would like to cancel.
In server side put cancel status for the task in database or any shared variable between Flask Server and your Worker Process
Divide Task processing in several stages and check status of task in database before each stage, if status is canceled - stop the task processing.
Another choice for point 1 is to do some monitoring in Server side in separate Process - count interval between status requests from client side.
I've handled similar problems by launching an entirely separate process via:
sp.call('start python path\\worker.py', shell=True)
worker.py would then report its PID back to the api.py via redis, then its straightforward to kill the process at any point from api.py
Of course, how viable that is for you will depend on how much data resides within api.py and is shared to worker.py - whether its feasible for that to also pass via redis or not is for you to decide.
The added benefit is you decouple socket from heavy compute - and you can go quasi-multi-core (single thread per worker.py). You could go full multi core by incorporating multiprocessing into each worker.py if you wished.
I want to make repeated requests to a server that will return with some tasks. The response from the server will be a dictionary with a list of functions that need to be called. For example:
{
tasks: [
{
function: "HelloWorld",
id: 1212
},
{
function: "GoodbyeWorld"
id: 1222
}
]
}
NOTE: I'm dummying it down.
For each of these tasks, I will run the specified function using multiprocessing. Here is an example of my code:
r = requests.get('https://localhost:5000', auth=('user', 'pass'))
data = r.json()
if len(data["tasks"]) > 0:
manager = multiprocessing.Manager()
for task in data["tasks"]:
if task["function"] == "HelloWorld":
helloObj = HelloWorldClass()
hello = multiprocessing.Process(target=helloObj.helloWorld)
hello.start()
hello.join()
elif task["function"] == "GoodbyeWorld":
byeObj = GoodbyeWorldClass()
bye = multiprocessing.Process(target=byeObj.byeWorld)
bye.start()
bye.join()
The problem is, I want to make repeated requests and fill the data["tasks"] array as the other processes are running. If I throw everything into some while loop, it'll only make a request after all the processes from the initial response is done (when join() has been reached for all processes).
Can anyone help me to make repeated requests and fill the array continuously? Please let me know if I need to make any clarifications.
If I understood you correctly, you need something like this:
import time
from multiprocessing import Process
import requests
from task import FunctionFactory
def get_tasks():
resp = requests.get('https://localhost:5000', auth=('user', 'pass'))
data = resp.json()
return data['tasks']
if __name__ == '__main__':
procs = {}
for _ in range(10):
tasks = get_tasks()
if not tasks:
time.sleep(5)
continue
for task in tasks:
if task['id'] in procs:
# This task has been already submitted for execution.
continue
func = FunctionFactory.build(task['function'])
proc = Process(target=func)
proc.start()
procs[task['id']] = proc
# Waiting for all the submitted tasks to finish.
for proc in procs.values():
proc.join()
Here, the function get_tasks is used to request a list of dictionaries with id and function keys from the server. In the main section, there is a procs dictionary that maps id to running process instances which execute functions built by a FunctionFactory using received tasks' function names. In the case there is already a running task with the same id, it gets ignored.
With this approach, you can request tasks as often as needed (here, 10 requests are used in a for loop) and start processes to execute them in parallel. In the end, you just wait for all the submitted tasks to finish.
You have a bug in your program, you should call the joins after you've created all the tasks. Join blocks until the process has finished -- in your case before you start the next one. Which practically makes you whole program run sequentially.
I am creating a simple TCP server-client script in Python. The server is threaded and forks a new worker/thread for every client connection. So far I have pretty much coded the entire server module. But my function called the handle_clients() which is forked for every incoming client connection is getting very long. In order to improve the readability of the code I want to split my handle_clients() into multiple small functions. I do understand that when I split handle_client() into smaller functions, the split functions should be wrapped around mutex locks to synchronize shared usage between multiple handle_clients() threads. Doing this will actually reduce the efficiency of the program because handle_clients() will have to wait for other threads to unlock the shared functions before actually using it. My other thought was to create these smaller functions as threads within the handle_clients() thread. And wait for these threads to finish using Thread.join() before continuing. Is there a better way to do this?
My code:
#!/usr/bin/python
import socket
import threading
import pandas as pd
class TCPServer(object):
NUMBER_OF_THREADS = 0
BUFFER = 4096
threads_list = []
def __init__(self, port, hostname):
self.socket = socket.socket(
family=socket.AF_INET, type=socket.SOCK_STREAM)
self.socket.bind((hostname, port))
def listen_for_clients(self):
self.socket.listen(5)
while True:
client, address = self.socket.accept()
client_ID = client.recv(TCPServer.BUFFER)
print(f'Connected to client: {client_ID}')
if client_ID:
TCPServer.NUMBER_OF_THREADS = TCPServer.NUMBER_OF_THREADS + 1
thread = threading.Thread(
target=TCPServer.create_worker, args=(self, client, address, client_ID))
TCPServer.threads_list.append(thread)
thread.start()
if TCPServer.NUMBER_OF_THREADS > 2:
break
TCPServer.wait_for_workers()
def wait_for_workers():
for thread in TCPServer.threads_list:
thread.join()
def create_worker(self, client, address, client_ID):
print(f'Spawned a new worker for {client_ID}. Worker #: {TCPServer.NUMBER_OF_THREADS}')
data_list = []
data_frame = pd.DataFrame()
client.send("SEND_REQUEST_TYPE".encode())
request_type = client.recv(TCPServer.BUFFER).decode('utf-8')
if request_type == 'KMEANS':
print(f'Client: REQUEST_TYPE {request_type}')
client.send("SEND_DATA".encode())
while True:
data = client.recv(TCPServer.BUFFER).decode('utf-8')
if data == 'ROW':
client.send("OK".encode())
while True:
data = client.recv(TCPServer.BUFFER).decode('utf-8')
print(f'Client: {data}')
if data == 'ROW_END':
print('Data received: ', data_list)
series = pd.Series(data_list)
data_frame.append(series, ignore_index=True)
data_list = []
client.send("OK".encode())
break
else:
data_list.append(int(data))
client.send("OK".encode())
elif data == 'DATA_END':
client.send("WAIT".encode())
# (Vino) pass data to algorithm
print('Data received from client {client_ID}: ', data_frame)
elif request_type == 'NEURALNET':
pass
elif request_type == 'LINRIGRESSION':
pass
elif request_type == 'LOGRIGRESSION':
pass
def main():
port = input("Port: ")
server = TCPServer(port=int(port), hostname='localhost')
server.listen_for_clients()
if __name__ == '__main__':
main()
Note: This following block of code is repetative and will e used multiple times within the handle_client() function.
while True:
data = client.recv(TCPServer.BUFFER).decode('utf-8')
if data == 'ROW':
client.send("OK".encode())
while True:
data = client.recv(TCPServer.BUFFER).decode('utf-8')
print(f'Client: {data}')
if data == 'ROW_END':
print('Data received: ', data_list)
series = pd.Series(data_list)
data_frame.append(series, ignore_index=True)
data_list = []
client.send("OK".encode())
break
else:
data_list.append(int(data))
client.send("OK".encode())
elif data == 'DATA_END':
client.send("WAIT".encode())
# (Vino) pass data to algorithm
print('Data received from client {client_ID}: ', data_frame)
This is the block I want a place in a separate function and calls it within the handle_client() thread.
Your code is already long, I'll not dive into it but try to keep things general.
I do understand that when I split handle_client() into smaller functions, the split functions should be wrapped around mutex locks.
That's not directly true, between threads you already have to use locks to guard against memory overwriting, regarless your function calls.
The server is threaded
Looks like you're doing CPU-intensive work (I see LINALG, NEURALNET, ...), it is not logical to use threads, in Python, to dispatch CPU-intensive loads as the GIL will linearize CPU usage between your threads.
The way to parallelize CPU intensive work in Python is to use processes.
Processes do not share memory so you'll be able to manipulate variables freely without mutexes, but they won't be shared at all, I hope your jobs are independent, as they can't share any state.
If you need to share state, avoid locks, it's complicated to handle, it's the way to dead locks, and it's not readable, try to implement your "state sharing" with queues, as a pipeline of jobs, each worker pulling from a queue, doing work, and pushing to another queue, this way keep things clear and easy to understand. Plus there's implementation of queues for threads and processes so you'll be able to switch from both almost seamlessly.
if TCPServer.NUMBER_OF_THREADS > 2:
break
Hey, you're breaking out of your main loop when you have more than two threads, existing your main process, killing your server, I bet that now what you want. Oh and if you use processes instead of threads, you should prefork a pool of them, as their creation costs more than a thread. And reuse them, a process can do a job after finishing one, it does not have to die (typically use queues to send job to your processes).
Side note: I'd implement this using HTTP instead of raw TCP to benefit from the notions of request, response, error reporting, existing frameworks, and the ability to use existing clients (curl/wget in command line, your browser, requests in Python). I'd implement this fully asynchronously (no blocking HTTP request), like one request to create a job, and following requests to get the status and the result, like:
$ curl -X POST http://localhost/linalg/jobs/ -d '{your data}'
201 Created
Location: http://localhost/linalg/jobs/1
$ curl -XGET http://localhost/linalg/jobs/1
200 OK
{"status": "queued"}
Some time later…
$ curl -XGET http://localhost/linalg/jobs/1
200 OK
{"status": "in progress"}
Some time later…
$ curl -XGET http://localhost/linalg/jobs/1
200 OK
{"status": "done", "result": "..."}
To implement this there's a lot of nice work already done, typically aiohttp, apistar, and so on.
I have designed a REST API which receives inputs through POST requests and then applies some logic to the inputs and returns to the callback uri which is part of the inputs.
This design was working fine for single input, but then i want to implement multithreading so that i can handle multiple POST requests. I have tried using 'app.run(threaded=True)' but was not successful.
I am running this code on linux platform. Not sure what is wrong in the following code, and am not so good at using threads in python, would appreciate if someone can let me know where the issue is:
I am able to get the '200' response once there is a POST request and the inputs are appended to 'inp_params', after which there is no processing in the thread.
from flask import Flask, jsonify, request
import time
import json
import os
import threading
import Queue
import test_func_module as tf
app = Flask(__name__)
inp_params = []
# Create the queue and threader
q = Queue.Queue()
#app.route('/', methods = ['GET', 'POST'] )
def get_data():
if request.method == 'GET':
return 'RESTful API'
elif request.method == 'POST':
global inp_params
inputs = {"fileName": request.json["fileName"], "fileId": request.json["fileId"], "ModuleId": request.json["ModuleId"], "WorkflowId": request.json["WorkflowId"],"Language": request.json["Language"], "callbackuri": request.json["callbackuri"]}
inp_params.append(inputs)
return '200'
def test_integrate(worker):
TF_output = tf.test_func(worker)
return TF_output
def threader():
while True:
# gets an worker from the queue
worker = q.get()
# Run the example job with the avail worker in queue (thread)
test_integrate(worker)
# completed with the job
q.task_done()
if __name__ == '__main__':.
for worker in inp_params:
q.put(worker)
for x in range(4): #4 cores
t = threading.Thread(target=threader)
# classifying as a daemon, so they will die when the main dies
t.daemon = True
# begins, must come after daemon definition
t.start()
# wait until the thread terminates.
q.join()
app.run(threaded=True)
#Shilparani Since you mentioned
I have tried using 'app.run(threaded=True)' but was not successful.
May not be exact answer for your question but I would like to share my experience for achieving concurrency through uwsgi/gunicorn :
Keep it simple by coding Flask for REST endpoints and move Multithreading , MultiProcessing logic to gunicorn or uwsgi where you can mention threads and workers which help for achieving concurrency , parallelism if that's what you are trying to achieve.
gunicorn -b localhost:8080 -w 4 -t 4 app:app
Based on your need and operations:
If tasks are CPU intensive try to keep #workers as #CPU-cores
If tasks are I/O intensive may be safe to try with more threads
I need a webserver which routes the incoming requests to back-end workers by batching them every 0.5 second or when it has 50 http requests whichever happens earlier. What will be a good way to implement it in python/tornado or any other language?
What I am thinking is to publish the incoming requests to a rabbitMQ queue and then somehow batch them together before sending to the back-end servers. What I can't figure out is how to pick multiple requests from the rabbitMq queue. Could someone point me to right direction or suggest some alternate apporach?
I would suggest using a simple python micro web framework such as bottle. Then you would send the requests to a background process via a queue (thus allowing the connection to end).
The background process would then have a continuous loop that would check your conditions (time and number), and do the job once the condition is met.
Edit:
Here is an example webserver that batches the items before sending them to any queuing system you want to use (RabbitMQ always seemed overcomplicated to me with Python. I have used Celery and other simpler queuing systems before). That way the backend simply grabs a single 'item' from the queue, that will contain all required 50 requests.
import bottle
import threading
import Queue
app = bottle.Bottle()
app.queue = Queue.Queue()
def send_to_rabbitMQ(items):
"""Custom code to send to rabbitMQ system"""
print("50 items gathered, sending to rabbitMQ")
def batcher(queue):
"""Background thread that gathers incoming requests"""
while True:
batcher_loop(queue)
def batcher_loop(queue):
"""Loop that will run until it gathers 50 items,
then will call then function 'send_to_rabbitMQ'"""
count = 0
items = []
while count < 50:
try:
next_item = queue.get(timeout=.5)
except Queue.Empty:
pass
else:
items.append(next_item)
count += 1
send_to_rabbitMQ(items)
#app.route("/add_request", method=["PUT", "POST"])
def add_request():
"""Simple bottle request that grabs JSON and puts it in the queue"""
request = bottle.request.json['request']
app.queue.put(request)
if __name__ == '__main__':
t = threading.Thread(target=batcher, args=(app.queue, ))
t.daemon = True # Make sure the background thread quits when the program ends
t.start()
bottle.run(app)
Code used to test it:
import requests
import json
for i in range(101):
req = requests.post("http://localhost:8080/add_request",
data=json.dumps({"request": 1}),
headers={"Content-type": "application/json"})