I am trying to implement the prototype for my application in Python and stuck on choosing libraries, frameworks...
The application is a master-workers application (event loop?), where workers requests a master about a work they should do and respond to master with the result of their work.
All tasks (works) are stored in PostgreSQL table, and only master can access its data. The table looks like:
task(task_id int, status varchar, length int, error_msg varchar)
Master process should have the following API methods to outer world (REST/HTTP):
get_workers_count: retutns number of workers. When it starts first time, the initial number of workers is 0
set_workers(workers_count): sets new count of workers. If new count is greater than current one, master should spawn new workers. If new count is less then current one, some workers should die after they complete current work
add_task(time): Adds a tsak in task table with status 'READY' and length equals to time
Master process should also have the following API methods to workers (should not be acceptable to outer world):
get_task Returns task_id and length of the first record in task table in status 'READY'. After returning to worker it changes the status to 'EXECUTING'. Returns -1 if there are no tasks to execute. Returns -2 if worker should die.
set_task_status (task_id, status) - sets task status
Worker process should be run by master process and works as follows:
calls get_task method of master. If it gets -2 it terminates. If it gets -1, it sleeps and calls get_task_again
if it gets positive task numbert, ot sleeps for length of seconds (simulate work) and responds with a status (SUCCESS for prototype).
I am new in Python and ask somebody to help me in choosing frameworks/libraries for my application. My current state is:
I want to use Flask/gunicorn for REST Api in master process
I have no idea what to use for communication between master/worker processes. Is SocketServer is a good choice for me?
almost all work by worker process will be performed by C extension module
- workers and master will work on a single machine
I have no idea how to start workers: should I spawn thread/greenlet or should I fork a new process?
Please advise.
ASync is probably your best bet, I personally LOVE gevent. You could look at GIPC which multi processes gevent and gives you a read write channel back and forth. Or you can just have them communicate over restAPI's.
Personally I would fire up two distinct processes, a master channel that manages the pool and handles the queues. Then I would have worker processes poke at the API for new work, and when they retrieve the work they go do their business in a separate thread.
The advantage of this would be when you want to split the workers to other machines (micro computers) the only change required is an ip address.
Don't know much about master/worker architecture, but you can use pika/RabbitMQ + Celery for event handling and task queues.
Consider RabbitMQ instead of Postgres for events - see some discussion here.
Hope it helps.
Related
I am currently building a python app which should trigger functions at given timestamps entered by the user (not entered in chronological order).
I ran into a problem because I don't want my program to be busy-waiting checking if a new timestamp has been entered that must be added to the timer queue but also not creating a whole bunch of threads for every time a new timestamp is creating with its single purpose waiting for that timestamp.
What I thought of is putting it all together in one thread and doing something like an interruptable sleep, but I can't think of another way besides this:
while timer_not_depleted:
sleep(1)
if something_happened:
break
which is essentially busy-waiting.
So any suggestions on realizing an interruptable sleep?
Your intuition of using threads is correct. The following master-worker construction can work:
The master thread spawns a worker thread that waits for "jobs";
The two threads share a Queue - whenever a new job needs to be scheduled, the master thread puts a job specification into the queue;
Meanwhile, the worker thread does the following:
Maintain a separate list of future jobs to run and keep track of how long to keep sleeping until the next job runs;
Continue listening to new jobs by calling Queue.get(block=True, timeout=<time-to-next-job>);
In this case, if no new jobs are scheduled until the timeout, Queue.get will raise Empty exception - and at this point the worker thread should run the scheduled function and get back to polling. If a new job is scheduled in the meantime, Queue.get returns the new job, such that you can update the timeout value and then get back to waiting.
I'd like to suggest select.
Call it with a timeout equal to the delay to the nearest event (heap queue is a good data structure to maintain a queue of future timestamps) and provide a socket (as an item in the rlist arg), where your program listens on for updates from the user.
The select call returns when the socket has incoming data or when the timeout has occurred.
I'm working on a new monitoring system that can measure Celery queue throughput and help alert the team when the queue is getting backed up. Over the course of my work, I've come across some peculiar behaviors that I don't understand (and are not well documented in the Celery specs).
For testing purposes, I've set up an endpoint that will populate the queue with 16 several long-running tasks that can be used to simulate a backed-up queue. The framework is Flask and the Queue broker is Redis. Celery is configured for each worker to work on up to 4 tasks in parallel, and I have 2 workers running.
api/health.py
def health():
health = Blueprint("health", __name__)
#health.route("/api/debug/create-long-queue", methods=["GET"])
def long_queue():
for i in range(16):
sleepy_job.delay()
return make_response({}, 200)
return health
jobs.py
#celery.task(priority=HIGH_PRIORITY)
def sleepy_job(*args, **kwargs):
time.sleep(30)
Here's what I do to simulate a backed-up production queue:
I call /api/debug/create-long-queue to simulate a back-up in my queue. Based on the above math, the workers should be busy sleeping for 1 minute each (Together, they can concurrently handle 8 tasks at a time. Each task just sleeps for 30 seconds, and there are 16 tasks total.)
I make another API call shortly after (< 5 s), which kicks of a different job with real business logic (processing of an inbound webhook API call). We'll call this job handle_incoming_message.
Here's what I see Using flower to inspect the queue:
While all workers are blocked by the first 8 sleepy_job tasks, I see no sign of the new handle_incoming_message on the queue, even though I am certain handle_incoming_message.delay() has been called as a result of the 2nd API call.
After the first 8 sleepy_job tasks have been completed (~30s), I see the new handle_incoming_message on the queue with state RECIEVED.
After the second (and final) 8 sleepy_job tasks have been completed, I now see handle_incoming_message has state STARTED (and I can confirm this as the UI updates with the new data that was received and processed in that task.)
Questions
So it seems clear that when the workers are momentarily unblocked after handling the first 8 sleepy_job tasks, they are doing something to mark/acknowledge the new handle_incoming_message task in a way that is visible to flower. But this leaves several unanswered questions:
What is the state of the new handle_incoming_message task when the workers are blocked?
What changes after workers are unblocked that makes it so flower now has visibility into the new handle_incoming_message task?
What does the "RECEIVED" state actually mean?
(Bonus: How can I get visibility into tasks that are queued while workers are blocked?)
When all workers are blocked SOME tasks could be in the received state because of prefetching (look in the documentation for that). So chances are very high that your tasks are simply in the queue, waiting to be received by Celery workers (coordinating processes - these are not actual worker processes).
Flower is a simple service that is built upon a Celery feature called "task events". In simple terms it (Flower) subscribes itself as receiver of all events (received, succeeded, started, failed, etc) and then visually represents those to the web clients. More about it here. So when task gets received by a Celery worker, a "task-received" event is sent. Flower fetches this event, and changes the state of that task in the dashboard.
When a task is "received" it means that particular Celery worker took that task off the queue and it may be executed immediately (if there is a free worker-process to execute it), or Celery worker will wait for a worker process to become ready to run the task. I have already mentioned prefetching - Celery workers will often take more tasks then available worker-processes.
Celery does not give users a way to list what is in particular queue. That is why you will see many similar questions - including this one which offers answers. You will see my short answer there among others. In short, it depends on your broker of choice. If it is Redis, then you simply go through the list of objects. If it is RabbitMQ then you can use their tool to inspect queues. I think the decision not to provide this is good one as this information is never reliable. By the time you list all the tasks in particular queue, there may be thousands new ones...
We have a large application, which uses django as an ORM, and celery as a task running infrastructure.
We run complex pipelines triggered by events (user driven or automatic), which look something like this:
def pipeline_a:
# all lines are synchronous, so second line must happen after first is finished successfully
first_res = a1()
all_results = in_parallel.do(a2, a3, a4)
a5(first_res, all_results)
We wish to run a1, a2, ... on different machines (each task may need different resources), and the number of parallel running pipelines is always changing.
Today we use celery which is super convenient for implementing the above - but isn't suitable for auto-scaling (we hacked it to work with kubernetes, but it doesn't have native support with it).
Mainly the issues I want to solve are:
How to "run the next pipeline step" only after all previous ones are done (I may not know in advance which steps will be run - it depends on the results of previous steps, so the steps are dynamic in nature)
Today we try and use kubernetes (EKS) to autoscale some of the tasks (SQS queue size is the hpa metric). How to make kubernetes not try and terminate currently running tasks, but still "start pods" if a new task arrives at the queue (many tasks take ~half an hour to complete)
My experience so far says that to solve 1, celery is the most convenient way, but then it clashes with 2. So How would you solve 1 without celery, and then how could I harness kubernetes for long running tasks?
If I understand your question correctly,
you have async job which can run upto 30 mins.
Job are running on K8s.
Output of current job may decide the next Job.
You have ability to use SQS.
You can maintain queue for each of task. for each queue implement an consumer. Using Django first add the task to 'a1'. Update the job status in db.
When consumer of a1 finished execution it update the status in db and push to right queue. Let's say 'a3'.
Consumer of 'a3' will read the task. Update the db. Execute. Push the task in right queue. Update the db.
If you use SQS, then you store infinite task in queue. You will have to increase the number of consumer based on size of SQS queue. For this you can use https://github.com/Wattpad/kube-sqs-autoscaler
from celery import Celery
app = Celery('tasks', backend='amqp://guest#localhost//', broker='amqp://guest#localhost//')
a_num = 0
#app.task
def addone():
global a_num
a_num = a_num + 1
return a_num
this is the code I used to test celery.
I hope every time I use addone() the return value should increase.
But it's always 1
why???
Results
python
>> from tasks import addone
>> r = addone.delay()
>> r.get()
1
>> r = addone.delay()
>> r.get()
1
>> r = addone.delay()
>> r.get()
1
By default when a worker is started Celery starts it with a concurrency of 4, which means it has 4 processes started to handle task requests. (Plus a process that controls the other processes.) I don't know what algorithm is used to assign task requests to the processes started for a worker but eventually, if you execute addone.delay().get() enough, you'll see the number get greater than 1. What happens is that each process (not each task) gets its own copy of a_num. When I try it here, my fifth execution of addone.delay().get() returns 2.
You could force the number to increment each time by starting your worker with a single process to handle requests. (e.g. celery -A tasks worker -c1) However, if you ever restart your worker, the numbering will be reset to 0. Moreover, I would not design code that works only if the number of processes handling requests is forced to be 1. One day down the road a colleague decides that multiple processes should handle the requests for the tasks and then things break. (Big fat warnings in comments in the code can only do so much.)
At the end of the day, such state should be shared in a cache, like Redis, or a database used as a cache, which would work for the code in your question.
However, in a comment you wrote:
Let's see I want use a task to send something. Instead of connecting every time in task, I want to share a global connection.
Storing the connection in a cache won't work. I would strongly advocate having each process that Celery starts use its own connection rather than try to share it among processes. The connection does not need to be reopened with each new task request. It is opened once per process, and then each task request served by this process reuses the connection.
In many cases, trying to share the same connection among processes (through sharing virtual memory through a fork, for instance) would flat out not work anyway. Connections often carry state with them (e.g. whether a database connection is in autocommit mode). If two parts of the code expect the connection to be in different states, the code will operate inconsistently.
The tasks will run asynchronously so every time it starts a new task a_num will be set to 0. They are run as separate instances.
If you want to work with values I suggest a value store or database of some sort.
I'm working on a Python based system, to enqueue long running tasks to workers.
The tasks originate from an outside service that generate a "token", but once they're created based on that token, they should run continuously, and stopped only when explicitly removed by code.
The task starts a WebSocket and loops on it. If the socket is closed, it reopens it. Basically, the task shouldn't reach conclusion.
My goals in architecting this solutions are:
When gracefully restarting a worker (for example to load new code), the task should be re-added to the queue, and picked up by some worker.
Same thing should happen when ungraceful shutdown happens.
2 workers shouldn't work on the same token.
Other processes may create more tasks that should be directed to the same worker that's handling a specific token. This will be resolved by sending those tasks to a queue named after the token, which the worker should start listening to after starting the token's task. I am listing this requirement as an explanation to why a task engine is even required here.
Independent servers, fast code reload, etc. - Minimal downtime per task.
All our server side is Python, and looks like Celery is the best platform for it.
Are we using the right technology here? Any other architectural choices we should consider?
Thanks for your help!
According to the docs
When shutdown is initiated the worker will finish all currently executing tasks before it actually terminates, so if these tasks are important you should wait for it to finish before doing anything drastic (like sending the KILL signal).
If the worker won’t shutdown after considerate time, for example because of tasks stuck in an infinite-loop, you can use the KILL signal to force terminate the worker, but be aware that currently executing tasks will be lost (unless the tasks have the acks_late option set).
You may get something like what you want by using retry or acks_late
Overall I reckon you'll need to implement some extra application-side job control, plus, maybe, a lock service.
But, yes, overall you can do this with celery. Whether there are better technologies... that's out of the scope of this site.