I am a little frustrated with celery documentation. I understand that the command
celery -A my-module worker -l INFO -n w1
starts a worker instance named w1. This means that the worker instance starts some default number of "processes" in the OS.
But can celery also start threads instead of processes? For example what does the following command do?
celery -A my-module worker --pool threads -l INFO -n w1
I tried reading through the documentation but could not find anything that would give an answer to the question "What does multi threading mean when it comes to celery? Can celery support multi threading in place of multi processing?"
Related
I use celery4.x with Djangoand have more than two tasks in my celery queue. Due to the limit of GPU, I can only run at most two at the same time. Is there a way to let the third task wait and run until one of the previous two task? I have set CELERYD_CONCURRENCY paremeter in Django's settings.py which seems not work.
Anyone knows? Thanks
Run your worker using concurrency argument:
celery -A proj worker -l info --concurrency 2 -Q queue_name
I am running a celery worker like this:
celery worker --app=portalmq --logfile=/tmp/portalmq.log --loglevel=INFO -E --pidfile=/tmp/portalmq.pid
Now I want to run this worker in the background. I have tried several things, including:
nohup celery worker --app=portalmq --logfile=/tmp/portal_mq.log --loglevel=INFO -E --pidfile=/tmp/portal_mq.pid >> /tmp/portal_mq.log 2>&1 </dev/null &
But it is not working. I have checked the celery documentation, and I found this:
Running the worker as a daemon
Running the celery worker server
Specially this comment is relevant:
In production you will want to run the worker in the background as a daemon.
To do this you need to use the tools provided by your platform, or something
like supervisord (see Running the worker as a daemon for more information).
This is too much overhead just to run a process in the background. I would need to install supervisord in my servers, and get familiar with it. No go at the moment. Is there a simple way of running a celery worker in the backrground?
supervisor is really simple and requires really little work to get it setup up, same applies for to celery in combination with supervisor.
It should not take more than 10 minutes to setup it up :)
install supervisor with apt-get
create /etc/supervisor/conf.d/celery.conf config file
paste somethis in the celery.conf file
[program:celery]
directory = /my_project/
command = /usr/bin/python manage.py celery worker
plus (if you need) some optional and useful stuff (with dummy
values)
user = celery_user
group = celery_group
stdout_logfile = /var/log/celeryd.log
stderr_logfile = /var/log/celeryd.err
autostart = true
environment=PATH="/some/path/",FOO="bar"
restart supervisor (or do supervisorctl reread; supervisorctl add
celery)
after that you get the nice ctl commands to manage the celery process:
supervisorctl start/restart/stop celery
supervisorctl tail [-f] celery [stderr]
celery worker -A app.celery --loglevel=info --detach
For me this one worked, I was using celery with django
celery -A proj_name worker -l INFO --detach
I have faced the same problem as a lazy solution is to use & at the end of the command.
For example
celery worker -A <app>.celery --loglevel=info &
Below command when executed in terminal will start celery as a background process.
celery -A app.celery worker --loglevel=info --detach
Incase you want stop it then ps aux | grep celery as mentioned #Kaiss B. in another answer's comment & kill -9 <process id> to kill the process.
But first of all you need to install the celery for
apt install python-celery-common.
Some of the guys might be wondering why the other answers which are upvoted but not working in there system is because celery changed the command syntax from
celery worker -A app.celery --loglevel=info --detach
to
celery -A app.celery worker --loglevel=info --detach
Hope that helps.
My colleague has written celery tasks, necessary configuration in settings file, also supervisors config file. Everything is working perfectly fine. The projects is handed over to me and I seeing some issues that I have to fix.
There are two projects running on a single machine, both projects are almost same, lets call them projA and projB.
supervisord.conf file is as:
;for projA
[program:celeryd]
directory=/path_to_projA/
command=celery -A project worker -l info
...
[program:celerybeat]
directory=/path_to_projA/
command=celery -A project beat -l info
...
; For projB
[program:celerydB]
directory=/path_to_projB/
command=celery -A project worker -l info
...
[program:celerybeatB]
directory=/path_to_projB/
command=celery -A project beat -l info
...
The issue is, I am creating tasks through a loop and only one task is received from celeryd of projA, and remaining task are not in received (or could be received by celeryd of projB).
But when I stop celery programs for projB everything works well. Please note, the actual name of django-app is project hence celery -A project worker/beat -l info.
Please bare, I am new to celery, any help is appreciated. TIA.
As the Celery docs says,
Celery is an asynchronous task queue/job queue based on distributed message passing.
When multiple tasks are created through a loop, tasks are evenly distributed to two different workers ie worker of projA and worker of projB since your workers are same.
If projects are similar or as you mentioned almost same, you can use Celery Queue but of course your queues across projects should be different.
Celery Docs for the same is provided here.
You need to set CELERY_DEFAULT_QUEUE, CELERY_DEFAULT_ROUTING_KEY and CELERY_QUEUES
in your settings.py file.
And your supervisor.conf file needs queue name in the commands line for all the programs.
For Ex: command=celery -A project beat -l info -Q <queue_name>
And that should work, based on my experience.
While running the celery working using the following command creates two files w1.log and w1.pid which I do not want.
celery multi start w1 -A destiPak.celery -l info
Output
celery multi v3.1.20 (Cipater)
> Starting nodes...
> w1#foo-bar: OK
Show worker
celery multi show w1
Output
/Users/foo/bar/bin/python -m celery worker --detach -n w1#foo-bar --pidfile=w1.pid --logfile=w1.log --executable=/Users/foo/bar/bin/python
Suggestions, how to avoid creating those log file while Running the worker as a daemon
It's hard to understand why you would not want these files - the pid file is only a few bytes and the log files will contain useful information, and you can use logrotate or whatever to ensure that they do not take up too much space.
That said, if you use supervisord to manage the workers instead of celery multi you can configure it to not generate log files, plus it does not use .pid files.
here's a supervisord config file that should do what you want
[program:celery]
command=celery worker -n w1#foo-bar
autostart=true
stdout_logfile=/dev/null
redirect_stderr=true
I am building a Python 3 application that will consume messages from RabbitMQ. Is there some Python background job library that can make this easy? I am looking for something similar to Sneakers in Ruby. I would like library to have:
easy way to define tasks that process RabbitMQ messages (I have a separate non-Python producer application that will create messages and put them into RabbitMQ)
configure number of worker processes that run
tasks
run workers as daemonized processes
I believe you're looking for Celery
You'll define task as follows
#task
def mytask(param):
return 1 + 1
It will be put in message broker (for example mentioned RabbitMQ), and then consumed and executed from celery
You can configure number of workers
celery worker --concurrency=10
And yes, it can be demonized
To consume task of RabbitMq you have to define worker, but to run worker in a daemonized mode you have to create a supervisor for that worker
command to start worker
celery worker --concurrency=10 -Ofair --loglevel=DEBUG -A file_name_without_extension -Q queue_name
steps to create supervisor
https://thomassileo.name/blog/2012/08/20/how-to-keep-celery-running-with-supervisor/
http://python-rq.org/patterns/supervisor/