We Keep Coding

Celery worker stops when console is closed [duplicate]

I am running a celery worker like this:
celery worker --app=portalmq --logfile=/tmp/portalmq.log --loglevel=INFO -E --pidfile=/tmp/portalmq.pid
Now I want to run this worker in the background. I have tried several things, including:
nohup celery worker --app=portalmq --logfile=/tmp/portal_mq.log --loglevel=INFO -E --pidfile=/tmp/portal_mq.pid >> /tmp/portal_mq.log 2>&1 </dev/null &
But it is not working. I have checked the celery documentation, and I found this:
Running the worker as a daemon
Running the celery worker server
Specially this comment is relevant:
In production you will want to run the worker in the background as a daemon.
To do this you need to use the tools provided by your platform, or something
like supervisord (see Running the worker as a daemon for more information).
This is too much overhead just to run a process in the background. I would need to install supervisord in my servers, and get familiar with it. No go at the moment. Is there a simple way of running a celery worker in the backrground?

supervisor is really simple and requires really little work to get it setup up, same applies for to celery in combination with supervisor.
It should not take more than 10 minutes to setup it up :)
install supervisor with apt-get
create /etc/supervisor/conf.d/celery.conf config file
paste somethis in the celery.conf file
[program:celery]
directory = /my_project/
command = /usr/bin/python manage.py celery worker
plus (if you need) some optional and useful stuff (with dummy
values)
user = celery_user
group = celery_group
stdout_logfile = /var/log/celeryd.log
stderr_logfile = /var/log/celeryd.err
autostart = true
environment=PATH="/some/path/",FOO="bar"
restart supervisor (or do supervisorctl reread; supervisorctl add
celery)
after that you get the nice ctl commands to manage the celery process:
supervisorctl start/restart/stop celery
supervisorctl tail [-f] celery [stderr]

celery worker -A app.celery --loglevel=info --detach

For me this one worked, I was using celery with django
celery -A proj_name worker -l INFO --detach

I have faced the same problem as a lazy solution is to use & at the end of the command.
For example
celery worker -A <app>.celery --loglevel=info &

Below command when executed in terminal will start celery as a background process.
celery -A app.celery worker --loglevel=info --detach
Incase you want stop it then ps aux | grep celery as mentioned #Kaiss B. in another answer's comment & kill -9 <process id> to kill the process.
But first of all you need to install the celery for
apt install python-celery-common.
Some of the guys might be wondering why the other answers which are upvoted but not working in there system is because celery changed the command syntax from
celery worker -A app.celery --loglevel=info --detach
to
celery -A app.celery worker --loglevel=info --detach
Hope that helps.

Running celery worker + beat in the same container

My flask app is comprised of four containers: web app, postgres, rabbitMQ and Celery. Since I have celery tasks that run periodically, I am using celery beat. I've configured my docker-compose file like this:
version: '2'
services:
rabbit:
# ...
web:
# ...
rabbit:
# ...
celery:
build:
context: .
dockerfile: Dockerfile.celery
And my Dockerfile.celery looks like this:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-B", "-l", "INFO"]
While I read in the docs that I shouldn't go to production with the -B option, I hastily added it anyway (and forgot about changing it) and quickly learned that my scheduled tasks were running multiple times. For those interested, if you do a ps aux | grep celery from within your celery container, you'll see multiple celery + beat processes running (but there should only be one beat process and however many worker processes). I wasn't sure from the docs why you shouldn't run -B in production but now I know.
So then I changed my Dockerfile.celery to:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-l", "INFO"]
CMD ["celery", "-A", "app.tasks.celery", "beat", "-l", "INFO"]
No when I start my app, the worker processes start but beat does not. When I flip those commands around so that beat is called first, then beat starts but the worker processes do not. So my question is: how do I run celery worker + beat together in my container? I have combed through many articles/docs but I'm still unable to figure this out.
EDITED
I changed my Dockerfile.celery to the following:
ENTRYPOINT [ "/bin/sh" ]
CMD [ "./docker.celery.sh" ]
And my docker.celery.sh file looks like this:
#!/bin/sh -ex
celery -A app.tasks.celery beat -l debug &
celery -A app.tasks.celery worker -l info &
However, I'm receiving the error celery_1 exited with code 0
Edit #2
I added the following blocking command to the end of my docker.celery.sh file and all was fixed:
tail -f /dev/null

docker run only one CMD, so only the first CMD get executed, the work around is to create a bash script that execute both worker and beat and use the docker CMD to execute this script

I got by putting in the entrypoint as explained above, plus I added the &> to have the output in a log file.
my entrypoint.sh
#!/bin/bash
python3 manage.py migrate
python3 manage.py migrate catalog --database=catalog
python manage.py collectstatic --clear --noinput --verbosity 0
# Start Celery Workers
celery worker --workdir /app --app dri -l info &> /log/celery.log &
# Start Celery Beat
celery worker --workdir /app --app dri -l info --beat &> /log/celery_beat.log &
python3 manage.py runserver 0.0.0.0:8000

Starting from the same concept #shahaf has highlighted I solved starting from this other solution using bash -c in this way:
command: bash -c "celery -A app.tasks.celery beat & celery -A app.tasks.celery worker --loglevel=debug"

You can use celery beatX for beat. It is allowed (and recommended) to have multiple beatX instances. They use locks to synchronize.
Cannot say if it is production-ready, but it works for me like a charm (with -B key)

Restart celery beat and worker during Django deployment

I am using celery==4.1.0 and django-celery-beat==1.1.0.
I am running gunicorn + celery + rabbitmq with Django.
This is my config for creating beat and worker
celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach
During Django deployment I am doing following:
rm -f celerybeat.pid
rm -f celeryd.pid
celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach
service nginx restart
service gunicorn stop
sleep 1
service gunicorn start
I want to restart both celery beat and worker and it seems that this logic works. But I noticed that celery starts to use more and more memory during deployment and after several deployments I hit 100% memory use. I tried different server setups and it seems that it is not related.

rabbitmq may be to blame for high memory usage. Can you safely restart rabbit?
Also can you confirm that after a restart there is the expected amount of workers?

You are starting 2 new workers for every deployment without stopping/killing the previous workers.
During deployment, stop the existing workers with
kill -9 $PID
kill -9 `cat /var/run/myProcess.pid`
Alternatively, you can just kill all the workers with
pkill -9 celery
Now you can start workers as usual.
celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach

How to Restart Celery Wroker ran by Supervisord

I am running celery on production using supervisord. My supervisor configuration is below.
[program:celeryd]
command=%(ENV_PROJECT_PATH)s/scripts/celery_worker.sh
stdout_logfile=%(ENV_PROJECT_PATH)s/celeryd.log
stderr_logfile=%(ENV_PROJECT_PATH)s/celeryd.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=1000
priority=1000
My command to run celery worker is
celery_path=$(which celery)
$celery_path -A Project_Name worker --loglevel=info
I want to ask, how to restart celery worker when my codebase changes in production?

The main issue I run into is that long running tasks may get killed if you tell supervisor to killasgroup which would result in lost data.
The solution I've moved to using is to tell the mainprocess to TERM which will kill off the workers as they finish their tasks. supervisor will then restart the main process after all the workers finish.
ps aux | grep celery.*MainProcess | awk '{print $2}' | xargs kill -TERM
This is also related.
Celery Production Graceful Restart

Add following in supervisor file and restart supervisor.
killasgroup=true

Celery-Supervisor: How to restart a supervisor job to make newly updated celery-tasks working?

I have a running supervisor job for my celery server. Now I need to add a new task to it, but unfortunately my celery server command is not configured to track those dynamic changes automatically.
Here is my celery command:
python manage.py celery worker --broker=amqp://username:password#localhost/our_app_vhost
To restart my celery process, I have tried,
sudo supervisorctl -c /etc/supervisor/supervisord.conf restart <process_name>
supervisorctl stop all
supervisorctl start all
service supervisor restart
But nothing found working. How to restart it?

If you want to manage process with supervisorctl, you should configure supervisorctl, rpcinterface in your configuration file.
Here is a sample configuration file.
sample.conf
[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
[program:my_worker]
command = python manage.py celery worker --broker=amqp://username:password#localhost/our_app_vhost
[unix_http_server]
file=/tmp/supervisor.sock ; (the path to the socket file)
[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL for a unix socket
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
Now start supervisor with
supervisord -c sample.conf
Now if you want to restart your worker you can do it with
supervisorctl -c sample.conf restart my_worker
This restarts your worker. Alternatively you can also drop to supervisor shell and you can restart it
sudo supervisorctl -c sample.conf
supervisor> restart my_worker
my_worker: stopped
my_worker: started
Note:
There is an option to autoreload workers in Celery
python manage.py celery worker --autoreload --broker=amqp://username:password#localhost/our_app_vhost
This should be used in development mode only. Using this in production is not recommended.
More about this on celery docs.

you can write your celery task in /etc/supervisor/conf.d/. create a new config file for celery like celery.conf.
Assuming your virtualenv is venv, your django project is sample and your celery script is in _celery.py
The file should look like
[program:celery]
command=/home/ubuntu/.virtualenvs/venv/bin/celery --app=sample._celery:app worker --loglevel=INFO
directory=/home/ubuntu/sample/
user=ubuntu
numprocs=1
stdout_logfile=/home/ubuntu/logs/celery-worker.log
stderr_logfile=/home/ubuntu/logs/celery-error.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
after writing this supervisor program you need to run
If you add the supervisor program run this
$ sudo supervisorctl reread
celery: available
If you add/update the supervisor program run this
$ sudo supervisorctl update
celery: added process group
To check the status of your celery task
$ sudo supervisorctl status celery
celery RUNNING pid 18020, uptime 0:00:50
To stop the celery task
$ sudo supervisorctl stop celery
celery: stopped
To start the celery task
$ sudo supervisorctl start celery
celery: started
To restart the celery task (this would stop and again start the specified task)
$ sudo supervisorctl restart celery
celery: stopped
celery: started

If some task running then restart celery waiting for complete them. So need to kill all running process.
run following command for kill all celery process:
kill -9 $(ps aux | grep celery | grep -v grep | awk '{print $2}' | tr '\n' ' ') > /dev/null 2>&1
Restart celery:
sudo supervisorctl stop all
sudo supervisorctl start all