I would like to implement the following use case with the ViewFlow library:
Problem
Processes of a particular Flow, started by a user, must wait in a queue before executing a celery job. Each user has a queue of these processes. Based on a schedule, or triggered manually, the next process in the queue is allowed to proceed.
Example
A node within my flow enters a named queue. Other logic within the application determines, for each queue, when to allow the next task to proceed. The next task in the queue is selected and its activation's done() method called.
An example flow might look like this:
class MyFlow(Flow):
start = flow.Start(...).Next(queue_wait)
queue_wait = QueueWait("myQueue").Next(job)
job = celery.Job(...).Next(end)
end = flow.End()
Question
What would be the best approach to implement queueing? In the above example, I don't know what "QueueWait" should be.
I've read through the docs and viewflow code, but it's not yet clear to me if this can be done using built-in Node and Activation classes, such as func.Function, or if I need to extend with custom classes.
After much experimentation, I arrived at a workable and simple solution:
from viewflow.flow import base
from viewflow.flow.func import FuncActivation
from viewflow.activation import STATUS
class Queue(base.NextNodeMixin,
base.UndoViewMixin,
base.CancelViewMixin,
base.DetailsViewMixin,
base.Event):
"""
Node that halts the flow and waits in a queue. To process the next waiting task
call the dequeue method, optionally specifying the task owner.
Example placing a job in a queue::
class MyFlow(Flow):
wait = Queue().Next(this.job)
job = celery.Job(send_stuff).Next(this.end)
end = flow.End()
somewhere in the application code:
MyFlow.wait.dequeue()
or:
MyFlow.wait.dequeue(process__myprocess__owner=user)
Queues are logically separated by the task_type, so new queues defined in a
subclass by overriding task_type attribute.
"""
task_type = 'QUEUE'
activation_cls = FuncActivation
def __init__(self, **kwargs):
super(Queue, self).__init__(**kwargs)
def dequeue(self, **kwargs):
"""
Process the next task in the queue by created date/time. kwargs is
used to add task filter arguments, thereby effectively splitting the queue
into subqueues. This could be used to implement per-user queues.
Returns True if task was found and dequeued, False otherwise
"""
filter_kwargs = {'flow_task_type': self.task_type, 'status': STATUS.NEW}
if kwargs is not None:
filter_kwargs.update(kwargs)
task = self.flow_cls.task_cls.objects.filter(**filter_kwargs).order_by('created').first()
if task is not None:
lock = self.flow_cls.lock_impl(self.flow_cls.instance)
with lock(self.flow_cls, task.process_id):
task = self.flow_cls.task_cls._default_manager.get(pk=task.pk)
activation = self.activation_cls()
activation.initialize(self, task)
activation.prepare()
activation.done()
return True
return False
I tried to make it as generic as possible and support the definition of multiple named queues as well as sub-queues, such as per-user queues.
Related
I have a high QPS of requests.
I can only handle ONE request at a time. So the pending request needs to be stored in a local Array/Queue.
When there is contention, I do not want FCFS (first-come-first-serve). Instead I want to process the request based own some custom logic.
Pseudocode like this
def webApiHandler(request):
future = submit(request)
response = wait(future) # Wait time is depending on its priority
return response
What primitives I can use to implement this? Event loop? Asyncio? Threads?
------------edit------------
This is synchronous API call, everything should be handled locally and response ASAP once the computation is done. I do not plan to use job queue like celery.
With your requirements (but why do you not want concurrency ?) what you might want to use is literraly a priority queue, which is a queue with ... priority : info of implementation here , and you can use it in python with the queue module (doc here)
it is sorted by priority, so higher priority are at the end of the queue.
Your implementation will then decide how to value the request and set the priority on this particular request. But two identic priority will be treated as in a queue.
Then, you write a consumer in another thread that will pop the first (or last depending on what you consider top priority) item in the queue.
What you might want to look at, and enable concurrency and extra features is celery, which is a distributed task queue framework. (it allows for queue, priority, and also can be run with a any number of worker (any=1 in your case, but are you really into non-concurrency for high number of request ?).
Example:
import asyncio
import threading
from typing import Dict
# Local queue for pending compute (watchout, may need to replace dict for thread-safe ⚠️)
pending_computes: Dict[int, list[threading.Event]] = {}
# The queue manager to pick a pending compute
async def poll_next():
# 1. A computed example after updating model priority
priorities = [2, 1, 3] # TODO: handle the empty array.
# 2. Find the next compute event reference
next = pending_computes.get(priorities[0]) # TODO: handle empty dict.
# 3. Kick off the next compute
next.set()
# The FastAPI async handler
async def cloud_compute_api(model_id: int, intput: bytes):
# 1. Enqueue current compute as an event.
compute_event = threading.Event()
pending_computes.get(model_id, []).append(compute_event)
# 2. Poll the next pending compute based on.
asyncio.create_task(poll_next)
# 3. Wait until its own compute is set to GO. Wait up to 10 seconds.
compute_event.wait(10)
# 4. compute starts 🚀🚀🚀
res = compute(model_id, intput)
asyncio.create_task(poll_next) # Poll next pending compute, if any
return res
I have a task consisting of subtasks in a chain. How can I ensure a second call of this task does not start before the first one has finished?
#shared_task
def task(user):
res = chain(subtask_1.s(), # each subtask takes ~1 hour
subtask_2.s(),
subtask_3.s())
return res.apply_async()
A django view might now trigger to call this task:
# user A visits page that triggers task
task.delay(userA)
# 10 seconds later, while task() is still executing, user B visits page
task.delay(userB)
This leads to the tasks racing each other instead of being executed in sequential order. E.g. once a worker has finished with subtask_1() of the first task, it begins working on subtask_1() of the second task, instead of subtask_2() and subtask_3() of the first one.
Is there a way to elegently avoid this? I guess the problem is the order the subtasks get added to the queue.
I have already set worker --concurreny=1, however that still doesn't change the order he consumes from the queue.
Official docs (task cookbook) seem to offer a solution which I don't understand and doesn't work for me unfortunately.
Perhaps include a blocking mechanism within the task, after the chain, with a while not res.ready(): sleep(1) kind of hack?
You can wait for first task to finish and then execute second one like this.
res = task.delay(userA)
res.get() # will block until finished
task.delay(userB)
But it will block the calling thread until first one finished. You can chain tasks to avoid blocking, but for that you have to modify task signature a little to accept task result as argument.
#shared_task
def task(_, user): signature takes one extra argument
# skipped
and
from celery.canvas import chain
chain(task.s(None, userA), task.s(userB))()
I'm working with a multi-threaded script, where a Controller thread puts a varying number of items in a class queue shared by multiple other Worker threads. I'm looking for a way to have the Controller class wait until all tasks are completed by the other threads. I have something similar to the below:
Worker class
class Worker(threading.Thread):
q = queue.Queue()
evt_stop = threading.Event()
def task(self, *data):
result = data[1] + data[1]
data[0].q.put(result)
def __init__(self):
...
def run(self):
while not Worker.evt_stop.is_set():
if not Worker.q.empty():
data = Worker.q.get()
task(data[0], data[1])
Worker.q.task_done()
Controller class
class Controller(threading.Thread):
evt_stop = threading.Event()
def qsize(self, n):
if self.q.qsize() != n:
return False
else:
return True
def __init__(self):
self.q = queue.Queue()
self.await = threading.Condition()
def run(self):
r = [<list of unknown length>]
for i in r:
Worker.q.put(self, i)
with self.await:
if self.await.wait_for(lambda: self.qsize(len(r)), timeout=5.0):
while not self.q.empty():
x = self.q.get()
print(x)
self.q.task_done()
But from my understanding of the answer and clarification provided here, the lambda will always return True, as it is the function object that is returned, not necessarily the value ("self.test(1) is the object that is the result of calling the method, which is a bool, not a callable"). Am I understanding this correctly? Additionally, am I greatly over-complicating this, and a simpler solution exists?
Clarification of intent:
This is essentially a script that performs multiple, network oriented functions. The main thread runs a menu system from which the user can select different network I/O tasks. Each of these tasks runs in a separate thread, instantiated by instances of the controller class. When the script first runs, it instantiates a set of worker threads, typically 4 or 6. The idea is that the user can select one of the instantiated controller class objects, and it will set about fulfilling its logic, which involves sending/receiving data through the worker threads, then modifying that data, and repeating the process. The issue I'm trying to tackle is having the controller object wait until the worker threads complete all the tasks that specific controller instance put into the shared worker class queue. It is unknown in advance how many tasks that will be, as it is dependent upon the results of the very first network I/O task.
I am building a Python application that works with taskqueue in the following flow
Add a pull task to taskqueue by calling taskqueue.add()
Lease the same task by calling taskqueue.lease_tasks()
After some time, we may want to shorten the lease time by calling taskqueue.modify_task_lease()
The problem is, those 3 steps happen in different web sessions. At step 3, the modify_task_lease() function need a task instance as argument, while I only have task_name in hand, which is passed from step 2 with web hooks.
So is there any way to retrieve a task with its name?
In the document, I found delete_tasks_by_name(), but there is no modify_task_lease_by_name(), which is exactly what I wanted to do.
The delete_tasks_by_name() is just a wrapper around delete_tasks_by_name_async(), which is implemented as
if isinstance(task_name, str):
return self.delete_tasks_async(Task(name=task_name), rpc)
else:
tasks = [Task(name=name) for name in task_name]
return self.delete_tasks_async(tasks, rpc)
So I guess you could similarly use the Task() constructor to obtain the task instance needed by modify_task_lease():
modify_task_lease(Task(name=your_task_name), lease_seconds)
I am trying to create a class than can run a separate process to go do some work that takes a long time, launch a bunch of these from a main module and then wait for them all to finish. I want to launch the processes once and then keep feeding them things to do rather than creating and destroying processes. For example, maybe I have 10 servers running the dd command, then I want them all to scp a file, etc.
My ultimate goal is to create a class for each system that keeps track of the information for the system in which it is tied to like IP address, logs, runtime, etc. But that class must be able to launch a system command and then return execution back to the caller while that system command runs, to followup with the result of the system command later.
My attempt is failing because I cannot send an instance method of a class over the pipe to the subprocess via pickle. Those are not pickleable. I therefore tried to fix it various ways but I can't figure it out. How can my code be patched to do this? What good is multiprocessing if you can't send over anything useful?
Is there any good documentation of multiprocessing being used with class instances? The only way I can get the multiprocessing module to work is on simple functions. Every attempt to use it within a class instance has failed. Maybe I should pass events instead? I don't understand how to do that yet.
import multiprocessing
import sys
import re
class ProcessWorker(multiprocessing.Process):
"""
This class runs as a separate process to execute worker's commands in parallel
Once launched, it remains running, monitoring the task queue, until "None" is sent
"""
def __init__(self, task_q, result_q):
multiprocessing.Process.__init__(self)
self.task_q = task_q
self.result_q = result_q
return
def run(self):
"""
Overloaded function provided by multiprocessing.Process. Called upon start() signal
"""
proc_name = self.name
print '%s: Launched' % (proc_name)
while True:
next_task_list = self.task_q.get()
if next_task is None:
# Poison pill means shutdown
print '%s: Exiting' % (proc_name)
self.task_q.task_done()
break
next_task = next_task_list[0]
print '%s: %s' % (proc_name, next_task)
args = next_task_list[1]
kwargs = next_task_list[2]
answer = next_task(*args, **kwargs)
self.task_q.task_done()
self.result_q.put(answer)
return
# End of ProcessWorker class
class Worker(object):
"""
Launches a child process to run commands from derived classes in separate processes,
which sit and listen for something to do
This base class is called by each derived worker
"""
def __init__(self, config, index=None):
self.config = config
self.index = index
# Launce the ProcessWorker for anything that has an index value
if self.index is not None:
self.task_q = multiprocessing.JoinableQueue()
self.result_q = multiprocessing.Queue()
self.process_worker = ProcessWorker(self.task_q, self.result_q)
self.process_worker.start()
print "Got here"
# Process should be running and listening for functions to execute
return
def enqueue_process(target): # No self, since it is a decorator
"""
Used to place an command target from this class object into the task_q
NOTE: Any function decorated with this must use fetch_results() to get the
target task's result value
"""
def wrapper(self, *args, **kwargs):
self.task_q.put([target, args, kwargs]) # FAIL: target is a class instance method and can't be pickled!
return wrapper
def fetch_results(self):
"""
After all processes have been spawned by multiple modules, this command
is called on each one to retreive the results of the call.
This blocks until the execution of the item in the queue is complete
"""
self.task_q.join() # Wait for it to to finish
return self.result_q.get() # Return the result
#enqueue_process
def run_long_command(self, command):
print "I am running number % as process "%number, self.name
# In here, I will launch a subprocess to run a long-running system command
# p = Popen(command), etc
# p.wait(), etc
return
def close(self):
self.task_q.put(None)
self.task_q.join()
if __name__ == '__main__':
config = ["some value", "something else"]
index = 7
workers = []
for i in range(5):
worker = Worker(config, index)
worker.run_long_command("ls /")
workers.append(worker)
for worker in workers:
worker.fetch_results()
# Do more work... (this would actually be done in a distributor in another class)
for worker in workers:
worker.close()
Edit: I tried to move the ProcessWorker class and the creation of the multiprocessing queues outside of the Worker class and then tried to manually pickle the worker instance. Even that doesn't work and I get an error
RuntimeError: Queue objects should only be shared between processes
through inheritance
. But I am only passing references of those queues into the worker instance?? I am missing something fundamental. Here is the modified code from the main section:
if __name__ == '__main__':
config = ["some value", "something else"]
index = 7
workers = []
for i in range(1):
task_q = multiprocessing.JoinableQueue()
result_q = multiprocessing.Queue()
process_worker = ProcessWorker(task_q, result_q)
worker = Worker(config, index, process_worker, task_q, result_q)
something_to_look_at = pickle.dumps(worker) # FAIL: Doesn't like queues??
process_worker.start()
worker.run_long_command("ls /")
So, the problem was that I was assuming that Python was doing some sort of magic that is somehow different from the way that C++/fork() works. I somehow thought that Python only copied the class, not the whole program into a separate process. I seriously wasted days trying to get this to work because all of the talk about pickle serialization made me think that it actually sent everything over the pipe. I knew that certain things could not be sent over the pipe, but I thought my problem was that I was not packaging things up properly.
This all could have been avoided if the Python docs gave me a 10,000 ft view of what happens when this module is used. Sure, it tells me what the methods of multiprocess module does and gives me some basic examples, but what I want to know is what is the "Theory of Operation" behind the scenes! Here is the kind of information I could have used. Please chime in if my answer is off. It will help me learn.
When you run start a process using this module, the whole program is copied into another process. But since it is not the "__main__" process and my code was checking for that, it doesn't fire off yet another process infinitely. It just stops and sits out there waiting for something to do, like a zombie. Everything that was initialized in the parent at the time of calling multiprocess.Process() is all set up and ready to go. Once you put something in the multiprocess.Queue or shared memory, or pipe, etc. (however you are communicating), then the separate process receives it and gets to work. It can draw upon all imported modules and setup just as if it was the parent. However, once some internal state variables change in the parent or separate process, those changes are isolated. Once the process is spawned, it now becomes your job to keep them in sync if necessary, either through a queue, pipe, shared memory, etc.
I threw out the code and started over, but now I am only putting one extra function out in the ProcessWorker, an "execute" method that runs a command line. Pretty simple. I don't have to worry about launching and then closing a bunch of processes this way, which has caused me all kinds of instability and performance issues in the past in C++. When I switched to launching processes at the beginning and then passing messages to those waiting processes, my performance improved and it was very stable.
BTW, I looked at this link to get help, which threw me off because the example made me think that methods were being transported across the queues: http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html
The second example of the first section used "next_task()" that appeared (to me) to be executing a task received via the queue.
Instead of attempting to send a method itself (which is impractical), try sending a name of a method to execute.
Provided that each worker runs the same code, it's a matter of a simple getattr(self, task_name).
I'd pass tuples (task_name, task_args), where task_args were a dict to be directly fed to the task method:
next_task_name, next_task_args = self.task_q.get()
if next_task_name:
task = getattr(self, next_task_name)
answer = task(**next_task_args)
...
else:
# poison pill, shut down
break
REF: https://stackoverflow.com/a/14179779
Answer on Jan 6 at 6:03 by David Lynch is not factually correct when he says that he was misled by
http://www.doughellmann.com/PyMOTW/multiprocessing/communication.html.
The code and examples provided are correct and work as advertised. next_task() is executing a task received via the queue -- try and understand what the Task.__call__() method is doing.
In my case what, tripped me up was syntax errors in my implementation of run(). It seems that the sub-process will not report this and just fails silently -- leaving things stuck in weird loops! Make sure you have some kind of syntax checker running e.g. Flymake/Pyflakes in Emacs.
Debugging via multiprocessing.log_to_stderr()F helped me narrow down the problem.