I have celery running in a docker container processing tasks from rabbitmq. I am trying to stop and remove the celery container, while allowing the current running tasks to complete. The docs suggest that sending the TERM or INT signals to the main process should warm shutdown celery, although I am finding that the child processes are just being killed.
When I send TERM the running processes it throws:
WorkerLostError('Worker exited prematurely: signal 15 (SIGTERM).',)
When I send INT the running process just exits with no error, although it too doesn't allow the tasks to finish as the docs suggest.
I am starting the docker container with the command:
su -m celery_user -c "python manage.py celery worker -Q queue-name"
Any thoughts on why this might be happening? Could it be that the signal is terminating the container as well as the celery process?
I am sending the signal with:
docker kill --signal="TERM" containerid
or docker exec containerid kill -15 1
docker kill will kill the container. What you need to do is to send the signal only to the main celery process.
Personally I use supservisord inside the docker container to manage the celery worker. By default supervisord will send SIGTERM to stop the process.
Here's a sample supervisor config for celery
[program:celery]
command=celery worker -A my.proj.tasks --loglevel=DEBUG -Ofair --hostname celery.host.domain.com --queues=celery
environment=PYTHONPATH=/etc/foo/celeryconfig:/bar/Source,PATH=/foo/custom/bin:/usr/kerberos/bin
user=celery-user
autostart=true
stdout_logfile=/var/log/supervisor/celery.log
redirect_stderr=true
Related
I have some celery workers in a Heroku app. My app is using python3.6and django, these are the relevant dependencies and their versions:
celery==3.1.26.post2
redis==2.10.3
django-celery==3.2.2
I do not know if the are useful to this question, but just in case. On Heroku we are running the Heroku-18 stack.
As it's usual, we have our workers declared in a Procfile, with the following content:
web: ... our django app ....
celeryd: python manage.py celery worker -Q celery --loglevel=INFO -O fair
one_type_of_worker: python manage.py celery worker -Q ... --maxtasksperchild=3 --loglevel=INFO -O fair
another_type: python manage.py celery worker -Q ... --maxtasksperchild=3 --loglevel=INFO -O fair
So, my current understanding of this process is the following:
Our celery queues run on multiple workers, each worker runs as a dyno on Heroku (not a server, but a “worker process” kind of thing, since servers aren’t a concept on Heroku). We also have multiple dynos running the same celery worker with the same queue, which results in multiple parallel “threads” for that queue to run more tasks simultaneously (scalability).
The web workers, celery workers, and celery queues can talk to each other because celery manages the orchestration between them. I think it's specifically the broker that handles this responsibility. But for example, this lets our web workers schedule a celery task on a specific queue and it is routed to the correct queue/worker, or a task running in one queue/worker can schedule a task on a different queue/worker.
Now here is when comes my question, so does the worker communicate? Do they use an API endpoint in localhost with a port? RCP? Do they use the broker url? Magic?
I'm asking this because I'm trying to replicate this setup in ECS and I need to know how to set it up for celery.
Here you go to know how celery works at heroku: https://devcenter.heroku.com/articles/celery-heroku
You can't run celery on Heroku without getting a Heroku dyno for celery. Also, make sure you have Redis configured on your Django celery settings.
to run the celery on Heroku, you just add this line to your Procfile
worker: celery -A YOUR-PROJECT_NAME worker -l info -B
Note: above celery commands will run both celery worker and celery beat
If you want to run it separately, you can use separate commands but one command is recommended
I am running a celery worker like this:
celery worker --app=portalmq --logfile=/tmp/portalmq.log --loglevel=INFO -E --pidfile=/tmp/portalmq.pid
Now I want to run this worker in the background. I have tried several things, including:
nohup celery worker --app=portalmq --logfile=/tmp/portal_mq.log --loglevel=INFO -E --pidfile=/tmp/portal_mq.pid >> /tmp/portal_mq.log 2>&1 </dev/null &
But it is not working. I have checked the celery documentation, and I found this:
Running the worker as a daemon
Running the celery worker server
Specially this comment is relevant:
In production you will want to run the worker in the background as a daemon.
To do this you need to use the tools provided by your platform, or something
like supervisord (see Running the worker as a daemon for more information).
This is too much overhead just to run a process in the background. I would need to install supervisord in my servers, and get familiar with it. No go at the moment. Is there a simple way of running a celery worker in the backrground?
supervisor is really simple and requires really little work to get it setup up, same applies for to celery in combination with supervisor.
It should not take more than 10 minutes to setup it up :)
install supervisor with apt-get
create /etc/supervisor/conf.d/celery.conf config file
paste somethis in the celery.conf file
[program:celery]
directory = /my_project/
command = /usr/bin/python manage.py celery worker
plus (if you need) some optional and useful stuff (with dummy
values)
user = celery_user
group = celery_group
stdout_logfile = /var/log/celeryd.log
stderr_logfile = /var/log/celeryd.err
autostart = true
environment=PATH="/some/path/",FOO="bar"
restart supervisor (or do supervisorctl reread; supervisorctl add
celery)
after that you get the nice ctl commands to manage the celery process:
supervisorctl start/restart/stop celery
supervisorctl tail [-f] celery [stderr]
celery worker -A app.celery --loglevel=info --detach
For me this one worked, I was using celery with django
celery -A proj_name worker -l INFO --detach
I have faced the same problem as a lazy solution is to use & at the end of the command.
For example
celery worker -A <app>.celery --loglevel=info &
Below command when executed in terminal will start celery as a background process.
celery -A app.celery worker --loglevel=info --detach
Incase you want stop it then ps aux | grep celery as mentioned #Kaiss B. in another answer's comment & kill -9 <process id> to kill the process.
But first of all you need to install the celery for
apt install python-celery-common.
Some of the guys might be wondering why the other answers which are upvoted but not working in there system is because celery changed the command syntax from
celery worker -A app.celery --loglevel=info --detach
to
celery -A app.celery worker --loglevel=info --detach
Hope that helps.
I am running celery on production using supervisord. My supervisor configuration is below.
[program:celeryd]
command=%(ENV_PROJECT_PATH)s/scripts/celery_worker.sh
stdout_logfile=%(ENV_PROJECT_PATH)s/celeryd.log
stderr_logfile=%(ENV_PROJECT_PATH)s/celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=1000
priority=1000
My command to run celery worker is
celery_path=$(which celery)
$celery_path -A Project_Name worker --loglevel=info
I want to ask, how to restart celery worker when my codebase changes in production?
The main issue I run into is that long running tasks may get killed if you tell supervisor to killasgroup which would result in lost data.
The solution I've moved to using is to tell the mainprocess to TERM which will kill off the workers as they finish their tasks. supervisor will then restart the main process after all the workers finish.
ps aux | grep celery.*MainProcess | awk '{print $2}' | xargs kill -TERM
This is also related.
Celery Production Graceful Restart
Add following in supervisor file and restart supervisor.
killasgroup=true
I am building a Python 3 application that will consume messages from RabbitMQ. Is there some Python background job library that can make this easy? I am looking for something similar to Sneakers in Ruby. I would like library to have:
easy way to define tasks that process RabbitMQ messages (I have a separate non-Python producer application that will create messages and put them into RabbitMQ)
configure number of worker processes that run
tasks
run workers as daemonized processes
I believe you're looking for Celery
You'll define task as follows
#task
def mytask(param):
return 1 + 1
It will be put in message broker (for example mentioned RabbitMQ), and then consumed and executed from celery
You can configure number of workers
celery worker --concurrency=10
And yes, it can be demonized
To consume task of RabbitMq you have to define worker, but to run worker in a daemonized mode you have to create a supervisor for that worker
command to start worker
celery worker --concurrency=10 -Ofair --loglevel=DEBUG -A file_name_without_extension -Q queue_name
steps to create supervisor
https://thomassileo.name/blog/2012/08/20/how-to-keep-celery-running-with-supervisor/
http://python-rq.org/patterns/supervisor/
I have celeryd daemons, working on small tasks. This daemon was configured with Upstart script
start on starting cessna
stop on stopping cessna
respawn
script
chdir /home/ubuntu/projects/cessna
exec su -c 'cd /home/ubuntu/projects/cessna; export MAX_POOL_SIZE="50";export newrelic-admin run-program celeryd -A cessna.celeryconfig --loglevel=info --concurrency=50 --pool=eventlet --queue=cessna_celery -E --pidfile=/tmp/cessna-3.pid >> /home/ubuntu/logs/cessna-w\
orker-3.log 2>> /home/ubuntu/errs/cessna-worker-3.log';
end script
Not so long I saw a lot of unack tasks in rabbitmq, no crashes in log files etc. We moved to native /etc/init.d/celeryd daemon, it solved the problem.
So, how it could be - Is there any relation between starting Celery with Upstart, and unacknowled tasks in Celery?