I have a high QPS of requests.
I can only handle ONE request at a time. So the pending request needs to be stored in a local Array/Queue.
When there is contention, I do not want FCFS (first-come-first-serve). Instead I want to process the request based own some custom logic.
Pseudocode like this
def webApiHandler(request):
future = submit(request)
response = wait(future) # Wait time is depending on its priority
return response
What primitives I can use to implement this? Event loop? Asyncio? Threads?
------------edit------------
This is synchronous API call, everything should be handled locally and response ASAP once the computation is done. I do not plan to use job queue like celery.
With your requirements (but why do you not want concurrency ?) what you might want to use is literraly a priority queue, which is a queue with ... priority : info of implementation here , and you can use it in python with the queue module (doc here)
it is sorted by priority, so higher priority are at the end of the queue.
Your implementation will then decide how to value the request and set the priority on this particular request. But two identic priority will be treated as in a queue.
Then, you write a consumer in another thread that will pop the first (or last depending on what you consider top priority) item in the queue.
What you might want to look at, and enable concurrency and extra features is celery, which is a distributed task queue framework. (it allows for queue, priority, and also can be run with a any number of worker (any=1 in your case, but are you really into non-concurrency for high number of request ?).
Example:
import asyncio
import threading
from typing import Dict
# Local queue for pending compute (watchout, may need to replace dict for thread-safe ⚠️)
pending_computes: Dict[int, list[threading.Event]] = {}
# The queue manager to pick a pending compute
async def poll_next():
# 1. A computed example after updating model priority
priorities = [2, 1, 3] # TODO: handle the empty array.
# 2. Find the next compute event reference
next = pending_computes.get(priorities[0]) # TODO: handle empty dict.
# 3. Kick off the next compute
next.set()
# The FastAPI async handler
async def cloud_compute_api(model_id: int, intput: bytes):
# 1. Enqueue current compute as an event.
compute_event = threading.Event()
pending_computes.get(model_id, []).append(compute_event)
# 2. Poll the next pending compute based on.
asyncio.create_task(poll_next)
# 3. Wait until its own compute is set to GO. Wait up to 10 seconds.
compute_event.wait(10)
# 4. compute starts 🚀🚀🚀
res = compute(model_id, intput)
asyncio.create_task(poll_next) # Poll next pending compute, if any
return res
Related
I have a list of objects to process. Some can be processed immediately, but others need to be processed by first fetching a URL. The organization looks something like:
processed_items = []
for item in list:
if url := item.get('location'):
fetched_item = fetch_item_from(url)
processed_item = process(fetched_item)
else:
processed_item = process(item)
if processed_item:
processed_items.append(processed_item)
The problem is that there are so many items that the only way to handle this in a memory efficient way is to process these files as they come in. On the other hand, doing them sequentially like this takes forever -- it's much more efficient to make the network requests asynchronously.
In theory, you could save all the items with URLs, then fetch them all at once using tasks and asyncio.gather. I have actually done this and it works. But this list of unfetched items can quickly eat up your memory, since the items are being streamed in, and making a ton of network requests all at once can make the server mad.
I think I'm looking for a result that leaves me with an array like
processed_items = [1, 2, <awaitable>, 3, <awaitable>, ...]
which I can then await the result of.
Is this the right approach? And if so, what's this design pattern called? Any first steps?
Just execute your code above in an asynchronous function - in a way that each item is processed in a separate task, and wrap your "fetch_item_from" function in an async function that uses an asyncio.Semaphore to limit the number of parallel requests to one you find optimal - be it 7, 10, 50 or 100.
If the rest of your processing is just CPU intensive you won't need any other async features there.
Actually, if your `fetch_item_from" is not async itself, you can simply do "run_in_executor" - and the nature of the process.future.Executor itself will limit the amount of concurrent requests, without the need to use a Semaphore are all.
import asyncio
MAXREQUESTS = 20
# Use this part if your original `fetch_item_from` is synchronous:
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(MAXREQUESTS)
async def fetch_item_from_with_executor(url):
asyncio.get_running_loop()
# This is automatically limited to the number of workers in the executor
return await asyncio.run_in_executor(executor, fetch_item_from, url)
# Use this part if fetch_item_from is asynchronous itself
semaphore = asyncio.Semaphore(MAXREQUESTS)
async def fetch_item_from_async(url):
with semaphore:
return await fetch_item_from(url)
# common code:
async def process_item(item):
if url := item.get('location'):
item = await fetch_item_from_executor(url) # / fetch_item_from_async
return process(item)
async def main(list_):
pending_list = [asyncio.create_task(item) for item in list_]
processed_items = []
while pending_list:
# The timeout=10 bellow is optional, and will return the control
# here with the already completed tasks each 10 seconds:
# this way you can print some progress indicator to see how
# things are going - or even improve the code so that
# finished tasks are yielded earlier to be consumed by the callers of "main"
# in parallel
# if the timeout argument is omitted, all items are processed in a single batch.
done, pending_list = await asyncio.wait(pending_list, timeout=10)
processed_items.extend(done) # the filter builtin will add just
# retrieve the results from each task and filter out the falsy (None?) ones:
return [result for item in processed_items if (result:=item.result())]
list_= ...
processed_items = asyncio.run(main(list_))
(missing above is any error handling - if either fetch_item_from or process can raise any exception, you have to unfold the list-comprehension which calls .result() in each task blindly to
separate the ones that raised from the ones that completed sucessfully)
In one of my asyncio projects I use one synchronisation method quite a lot and was wondering, if it is some kind of standard tool with a name I could give to google to learn more. I used the term "1-item queue" only because I don't have a better name. It is a degraded queue and it is NOT related to Queue(maxsize=1).
# [controller] ---- commands ---> [worker]
The controller sends commands to a worker (queue.put, actually put_nowait) and the worker waits for them (queue.get) and executes them, but the special rule is that the only the last command is important and immediately replaces all prior unfinished commands. For this reason, there is never more than 1 command waiting for the execution in the queue.
To implement this, the controller clears the queue before the put. There is no queue.clear, so it must discard (with get_nowait) the waiting item, if any. (The absence of queue.clear started my doubts resulting in this question.)
On the worker's side, if a command execution requires a sleep, it is replaced by a newcmd=queue.get with a timeout. When the timeout occurs, it was a sleep; when the get succeeds, the current work is aborted and the execution of newcmd starts.
The type of queue you are using is not standard - there is such a thing as a one-shot queue, but it's a different thing altogether.
The queue doesn't really fit your use case, though you made it work with some effort. You don't really need queuing of any kind, you need a slot that holds a single object (which can be replaced) and a wakeup mechanism. asyncio.Event can be used for the wakeup and you can attach the payload object (the command) to an attribute of the event. For example:
async def worker(evt):
while True:
await evt.wait()
evt.clear()
if evt.last_command is None:
continue
last_command = evt.last_command
evt.last_command = None
# execute last_command, possibly with timeout
print(last_command)
async def main():
evt = asyncio.Event()
workers = [asyncio.create_task(worker(evt)) for _ in range(5)]
for i in itertools.count():
await asyncio.sleep(1)
evt.last_command = f"foo {i}"
evt.set()
asyncio.run(main())
One difference between this and the queue-based approach is that setting the event will wake up all workers (if there is more than one), even if the first worker immediately calls evt.clear(). A queue item, on the other hand, will be guaranteed to be handed off to a single awaiter of queue.get().
I have a method that fetches some data from an API using IDs that are passed to it. Before this happens, the IDs are placed in a queue so that multiple (up to 100) can be used per API call. API calls are carried out by a method called flush_queue, which must be called at the end of my program to guarantee that all of the IDs that were added to the queue will have been used. I wrote an async function that takes in an ID, creates a future that flush_queue will eventually set the result of, and then returns the data obtained for that ID when it's available:
async def get_data(self, id):
self.queued_ids.append(id)
self.results[id] = asyncio.get_running_loop().create_future()
if len(self.queued_ids) == 100:
await self.flush_queue()
return await self.results[id]
(The future created and placed in the results dict will have the data obtained from the API that corresponds to that ID set as its result by flush_queue so that the data can be returned by the above method.) This works well for the case where the queue is flushed because there are enough IDs to make an API request; the problem is, I need to ensure that each user id is added to the queue before I call flush_queue at the completion of my program. Therefore, I need each call to get_data to have started before I do that; however, I can't wait for each coroutine object returned by each call to finish, because they won't finish until after flush_queue completes. Is there any way to ensure that a series of coroutines has started (but not necessarily finished) before doing something like calling flush_queue?
As discussed in the comments, just use a countdown latch/counting semaphore. Increment the counter when you enter get_data, and wait for it become zero.
I have a situation where a periodic monthly big_task reads a file and enqueue one chained-task per row in this file, where the chained tasks are small_task_1 and small_task_2:
class BigTask(PeriodicTask):
run_every = crontab(hour=00, minute=00, day_of_month=1)
def run(self):
task_list = []
with open("the_file.csv" as f:
for row in f:
t = chain(
small_task_1.s(row),
small_task_2.s(),
)
task_list.append(t)
gr = group(*task_list)
r = gr.apply_async()
I would like to get statistics about the number of enqueued, failed tasks (and detail about the exception) for each small_task, as soon as all of them are finished (whatever the status is) to send a summary email to the project admins.
I first thought of using chord, but callback is not executed if any of the headers task fails, which will surely happen in my case.
I could also use r.get() in the BigTask, very convenient, but not recommended to wait for a task result into another task (even if here, I guess the risk of worker deadlock is poor since task will be executed only once a month).
Important note: input file contains ~700k rows.
How would you recommend to proceed?
I'm not sure if it can help you to monitor, but about the chord and the callback issue you could use link_error callback (for catching exceptions). In your case for example you can use it like:
small_task_1.s(row).set(link_error=error_task))
and implement celery error_task that send you notification or whatever.
In celery 4, you can set it once for the all canvas (but it didn't work for me in 3.1):
r = gr.apply_async(link_error=error_task)
For the monitoring part, you can use flower of course.
Hope that help
EDIT: An alternative (without using additional persistency) would be to catch the exception and add some logic to the result and the callback. For example:
def small_task_1():
try:
// do stuff
return 'success', result
except:
return 'fail', result
and then in your callback task iterate over the results tuples and check for fails because doing the actual logic.
I found the best solution to be iterate over the group results, after the group is ready.
When you issue a Group, you have a ResultSet object. You can .save() this object, to get it later and check if .is_ready, or you can call .join() and wait for the results.
When it ends, you can access .results and you have a list of AsyncResult objects. These objects all have a .state property that you can access and check if the task was successul or not.
However, you can only check the results after the group ends. During the process, you can get the value of .completed_count() and have an idea of group progress.
https://docs.celeryproject.org/en/latest/reference/celery.result.html#celery.result.ResultSet
The solution we use for a partly similar problem where celery builtin stuff (tasks states etc) doesn't really cut it is to manually store desired informations in Redis and retrieve them when needed.
I would like to implement the following use case with the ViewFlow library:
Problem
Processes of a particular Flow, started by a user, must wait in a queue before executing a celery job. Each user has a queue of these processes. Based on a schedule, or triggered manually, the next process in the queue is allowed to proceed.
Example
A node within my flow enters a named queue. Other logic within the application determines, for each queue, when to allow the next task to proceed. The next task in the queue is selected and its activation's done() method called.
An example flow might look like this:
class MyFlow(Flow):
start = flow.Start(...).Next(queue_wait)
queue_wait = QueueWait("myQueue").Next(job)
job = celery.Job(...).Next(end)
end = flow.End()
Question
What would be the best approach to implement queueing? In the above example, I don't know what "QueueWait" should be.
I've read through the docs and viewflow code, but it's not yet clear to me if this can be done using built-in Node and Activation classes, such as func.Function, or if I need to extend with custom classes.
After much experimentation, I arrived at a workable and simple solution:
from viewflow.flow import base
from viewflow.flow.func import FuncActivation
from viewflow.activation import STATUS
class Queue(base.NextNodeMixin,
base.UndoViewMixin,
base.CancelViewMixin,
base.DetailsViewMixin,
base.Event):
"""
Node that halts the flow and waits in a queue. To process the next waiting task
call the dequeue method, optionally specifying the task owner.
Example placing a job in a queue::
class MyFlow(Flow):
wait = Queue().Next(this.job)
job = celery.Job(send_stuff).Next(this.end)
end = flow.End()
somewhere in the application code:
MyFlow.wait.dequeue()
or:
MyFlow.wait.dequeue(process__myprocess__owner=user)
Queues are logically separated by the task_type, so new queues defined in a
subclass by overriding task_type attribute.
"""
task_type = 'QUEUE'
activation_cls = FuncActivation
def __init__(self, **kwargs):
super(Queue, self).__init__(**kwargs)
def dequeue(self, **kwargs):
"""
Process the next task in the queue by created date/time. kwargs is
used to add task filter arguments, thereby effectively splitting the queue
into subqueues. This could be used to implement per-user queues.
Returns True if task was found and dequeued, False otherwise
"""
filter_kwargs = {'flow_task_type': self.task_type, 'status': STATUS.NEW}
if kwargs is not None:
filter_kwargs.update(kwargs)
task = self.flow_cls.task_cls.objects.filter(**filter_kwargs).order_by('created').first()
if task is not None:
lock = self.flow_cls.lock_impl(self.flow_cls.instance)
with lock(self.flow_cls, task.process_id):
task = self.flow_cls.task_cls._default_manager.get(pk=task.pk)
activation = self.activation_cls()
activation.initialize(self, task)
activation.prepare()
activation.done()
return True
return False
I tried to make it as generic as possible and support the definition of multiple named queues as well as sub-queues, such as per-user queues.