Setup: My application uses multiple workers to process elements in parallel. Processing those elements is CPU-intensive, so I need worker processes. The application will be used via a Flask API and GUnicorn. GUnicorn itself has multiple worker processes to process requests in parallel. In the Flask API, the request data is put onto a queue and the worker processes of my background application take this data from that queue.
Problem: Forking worker processes is quite time intensive and the application has to meet a certain speed requirement. Therefore, I would like to spawn the background worker processes when the app starts. To avoid mixing results, I need n background worker processes for every GUnicorn worker.
Question: How can I determine during construction time how many background workers I have to spawn and how is it possible to link those workers to GUnicorn workers?
Approach: I can read the number of GUnicorn workers from gunicorn_config.py by importing the workers variable. However, at this point, I do not know the GUnicorn worker process IDs. Do they have internal IDs that I could use at this point (e.g., GUnicorn worker #1, ...)?
You need to be aware (and account for) that a gunicorn worker can be stopped at any time (for example due to crash of the request hander or timeout in processing). So this means that hardwiring of your particular worker to particular gunicorn process cannot be permanent.
If you want to link your worker to particular gunicorn worker then relinking should happen every time gunicorn worker is restarted.
One approach would be to define your own post_fork handler and/or port_fork_init that would do that wiring.
Initially you can start the required number of workers and then post_fork handler may "borrow" (or as you call it "link") them to a worker that just have been created.
Related
I have a Flask app that uses a Gunicorn server for hosting. We are running into an issue where our workers keep getting locked up by long-running requests to different microservices. We currently only have Gunicorn set to give us 3 workers, so if there are 3 requests that are waiting on calls to those microservices, the server is completely locked up.
I started searching around and ran into this post:
gunicorn async worker class
This made sense to me, and it seemed like I could make the endpoint whose only job is to call these microservices asynchronous, then install gunicorn[gevent] and add --worker-class gevent to my start script. I implemented this and tested by using only 1 worker and adding a long time.sleep to the microservice being called. Everything worked perfectly and my server could process other requests while waiting for the async process to complete.
Then I opened pandora's box and added another long time.sleep to a synchronous endpoint within my server, expecting that because this endpoint is synchronous, everything would be locked up during the time it took for the one worker to finish that process. I was surprised that my worker was responding to pings, even while it was processing the synchronous task.
To me, this suggests that the Gunicorn server is adding threads for the worker to use even when the worker is in the middle of processing a synchronous task, instead of only freeing up the worker to operate on a new thread while waiting for an asynchronous IO process like the above post suggested.
I'm relatively new to thread safety so I want to confirm if I am going about solving my issue the right way with this implementation, and if so, how can I expect the new worker class to generate new threads for synchronous vs asynchronous processes?
in my setup I am using Gunicorn for my deployment on a single CPU machine, with three worker process. I have came to ask this question from this answer: https://stackoverflow.com/a/53327191/10268003 . I have experienced that it is taking upto one and a half second to send mail, so I was trying to send email asynchronously. I am trying to understand what will happen to the worker process started by Gunicorn, which will be starting a new thread to send the mail, will the Process gets blocked until the mail sending thread finishes. In that case I beleive my application's throughput will decrease. I did not want to use celery because it seems to be overkill for setting up celery for just sending emails. I am currently running two containers on the same machine with three gunicorn workers each in development machine.
Below is the approach in question, the only difference is i will be using threading for sending mails.
import threading
from .models import Crawl
def startCrawl(request):
task = Crawl()
task.save()
t = threading.Thread(target=doCrawl,args=[task.id])
t.setDaemon(True)
t.start()
return JsonResponse({'id':task.id})
def checkCrawl(request,id):
task = Crawl.objects.get(pk=id)
return JsonResponse({'is_done':task.is_done, result:task.result})
def doCrawl(id):
task = Crawl.objects.get(pk=id)
# Do crawling, etc.
task.result = result
task.is_done = True
task.save()
Assuming that you are using gunicorn Sync (default), Gthread or Async workers, you can indeed spawn threads and gunicorn will take no notice/interfere. The threads are reused to answer following requests immediately after returning a result, not only after all Threads are joined again.
I have used this code to fire an independent event a minute or so after a request:
Timer(timeout, function_that_does_something, [arguments_to_function]).start()
You will find some more technical details in this other answer:
In normal operations, these Workers run in a loop until the Master either tells them to graceful shutdown or kills them. Workers will periodically issue a heartbeat to the Master to indicate that they are still alive and working. If a heartbeat timeout occurs, then the Master will kill the Worker and restart it.
Therefore, daemon and non-daemon threads that do not interfere with the Worker's main loop should have no impact. If the thread does interfere with the Worker's main loop, such as a scenario where the thread is performing work and will provide results to the HTTP Response, then consider using an Async Worker. Async Workers allow for the TCP connection to remain alive for a long time while still allowing the Worker to issue heartbeats to the Master.
I have recently gone on to use asynchronous event loop based solutions like the uvicorn worker for gunicorn with the fastapi framework that provide alternatives to waiting in threads for IO.
Each task execution in a unique process space?
Do Celery pool (not Master) processes spawn off a process for each task execution?
In other words, is each task execution through a new process spawned by worker pool process?
Or is it the other way?
task is executed as part of worker pool process?
One implication of that: If celery task relies on data stored in the process memory space, that data is part of the worker pool process which is executing it. And, all tasks executed by the worker pool process have access to that copy of the data.
These details depend on the concurrency model you pick for your workers.
In the default, prefork model (based on processes), every task is executed inside one of the pre-forked processes (worker processes). So yes - it is a process pool. You can configure Celery to create a new worker-process for each task, but that is not the default behaviour. By default Celery does not replace old worker processes with new ones, but you can control that with the worker_max_tasks_per_child setting.
The question context is from the stand point of disposable infrastructure using docker for celery workers.
I am looking out for a option or ways to achieve the following:
I start a docker container, which starts the celery worker [along with concurrency]
It waits and picks ups one task per worker
the workers shutdown after task is completed [success or failure]
The main process dies after all workers are down.
I dispose off the container and start a new one.
I'm just starting out with celery in a Django project, and am kinda stuck at this particular problem: Basically, I need to distribute a long-running task to different workers. The task is actually broken into several steps, each of which takes considerable time to complete. Therefore, if some step fails, I'd like celery to retry this task using the same worker to reuse the results from the completed steps. I understand that celery uses routing to distribute tasks to certain server, but I can't find anything about this particular problem. I use RabbitMQ as my broker.
You could have every celeryd instance consume from a queue named after the hostname of the worker:
celeryd -l info -n worker1.example.com -Q celery,worker1.example.com
sets the hostname to worker1.example.com and will consume from a queue named the same, as well as the default queue (named celery).
Then to direct a task to a specific worker you can use:
task.apply_async(args, kwargs, queue="worker1.example.com")
similary to direct a retry:
task.retry(queue="worker1.example.com")
or to direct the retry to the same worker:
task.retry(queue=task.request.hostname)