We Keep Coding

Celery worker stops when console is closed [duplicate]

I am running a celery worker like this:
celery worker --app=portalmq --logfile=/tmp/portalmq.log --loglevel=INFO -E --pidfile=/tmp/portalmq.pid
Now I want to run this worker in the background. I have tried several things, including:
nohup celery worker --app=portalmq --logfile=/tmp/portal_mq.log --loglevel=INFO -E --pidfile=/tmp/portal_mq.pid >> /tmp/portal_mq.log 2>&1 </dev/null &
But it is not working. I have checked the celery documentation, and I found this:
Running the worker as a daemon
Running the celery worker server
Specially this comment is relevant:
In production you will want to run the worker in the background as a daemon.
To do this you need to use the tools provided by your platform, or something
like supervisord (see Running the worker as a daemon for more information).
This is too much overhead just to run a process in the background. I would need to install supervisord in my servers, and get familiar with it. No go at the moment. Is there a simple way of running a celery worker in the backrground?

supervisor is really simple and requires really little work to get it setup up, same applies for to celery in combination with supervisor.
It should not take more than 10 minutes to setup it up :)
install supervisor with apt-get
create /etc/supervisor/conf.d/celery.conf config file
paste somethis in the celery.conf file
[program:celery]
directory = /my_project/
command = /usr/bin/python manage.py celery worker
plus (if you need) some optional and useful stuff (with dummy
values)
user = celery_user
group = celery_group
stdout_logfile = /var/log/celeryd.log
stderr_logfile = /var/log/celeryd.err
autostart = true
environment=PATH="/some/path/",FOO="bar"
restart supervisor (or do supervisorctl reread; supervisorctl add
celery)
after that you get the nice ctl commands to manage the celery process:
supervisorctl start/restart/stop celery
supervisorctl tail [-f] celery [stderr]

celery worker -A app.celery --loglevel=info --detach

For me this one worked, I was using celery with django
celery -A proj_name worker -l INFO --detach

I have faced the same problem as a lazy solution is to use & at the end of the command.
For example
celery worker -A <app>.celery --loglevel=info &

Below command when executed in terminal will start celery as a background process.
celery -A app.celery worker --loglevel=info --detach
Incase you want stop it then ps aux | grep celery as mentioned #Kaiss B. in another answer's comment & kill -9 <process id> to kill the process.
But first of all you need to install the celery for
apt install python-celery-common.
Some of the guys might be wondering why the other answers which are upvoted but not working in there system is because celery changed the command syntax from
celery worker -A app.celery --loglevel=info --detach
to
celery -A app.celery worker --loglevel=info --detach
Hope that helps.

How to run django's "python manage.py runserver" , celery's "celery -A app_name worker -l info" and redis-server in one command

I have recently started with django. And I started doing a small project. I've been using celery with redis worker. And every to use celery and redis I have to run the celery and redis server and then django server. Which is a bit lengthy process.
I have two questions.
1. Am I doing the right thing by running the servers everytime or are there any other right method to this process?
2. If I'm in the right direction, is there any method to do this?
I tried circus.ini , but it did not work.

If you use UNIX system:
For this purpose you can get along just with bash. Just run celery and redis in background - use & command.
redis-server & celery -A app_name worker -l info & python manage.py runserver
Disadvantage of this approach - redis and celery will work in the background even after a shutdown of django dev server. So you need to terminate these processes. See this unix se answer for examples how to do that.
So you can create 2 bash scripts start.sh (contains commands with &) and cleanup.sh (terminate processes) and run them respectively.
For production see purpose #2
Use systemd or supervisor. You need to create conf files for your daemons and then run them.

Building upon Yevhenii M.'s answer, you can start a subshell command with a trap to kill all running processes in that subshell when you hit Ctrl+C:
(trap "kill 0" SIGINT; redis-server & celery -A app_name worker -l info & python manage.py runserver)
or as a more readable multiline command:
(
trap "kill 0" SIGINT
redis-server &
celery -A app_name worker -l info &
python manage.py runserver
)
Another option is to use a Procfile manager, but that requires installing additional dependencies/programs. Something like foreman or one of it's ports in other languages:
forego - Go
node-foreman - Node.js
gaffer - Java/JVM
goreman - Go
honcho - python
proclet - Perl
shoreman - shell
crank - Crystal
houseman - Haskell
(Source: foreman's README)
For this you create a Procfile (file in your project root) where you specify which commands to run:
redis: redis-server
worker: celery -A app_name worker
web: python manage.py runserver
Then run foreman start

Running celery worker + beat in the same container

My flask app is comprised of four containers: web app, postgres, rabbitMQ and Celery. Since I have celery tasks that run periodically, I am using celery beat. I've configured my docker-compose file like this:
version: '2'
services:
rabbit:
# ...
web:
# ...
rabbit:
# ...
celery:
build:
context: .
dockerfile: Dockerfile.celery
And my Dockerfile.celery looks like this:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-B", "-l", "INFO"]
While I read in the docs that I shouldn't go to production with the -B option, I hastily added it anyway (and forgot about changing it) and quickly learned that my scheduled tasks were running multiple times. For those interested, if you do a ps aux | grep celery from within your celery container, you'll see multiple celery + beat processes running (but there should only be one beat process and however many worker processes). I wasn't sure from the docs why you shouldn't run -B in production but now I know.
So then I changed my Dockerfile.celery to:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-l", "INFO"]
CMD ["celery", "-A", "app.tasks.celery", "beat", "-l", "INFO"]
No when I start my app, the worker processes start but beat does not. When I flip those commands around so that beat is called first, then beat starts but the worker processes do not. So my question is: how do I run celery worker + beat together in my container? I have combed through many articles/docs but I'm still unable to figure this out.
EDITED
I changed my Dockerfile.celery to the following:
ENTRYPOINT [ "/bin/sh" ]
CMD [ "./docker.celery.sh" ]
And my docker.celery.sh file looks like this:
#!/bin/sh -ex
celery -A app.tasks.celery beat -l debug &
celery -A app.tasks.celery worker -l info &
However, I'm receiving the error celery_1 exited with code 0
Edit #2
I added the following blocking command to the end of my docker.celery.sh file and all was fixed:
tail -f /dev/null

docker run only one CMD, so only the first CMD get executed, the work around is to create a bash script that execute both worker and beat and use the docker CMD to execute this script

I got by putting in the entrypoint as explained above, plus I added the &> to have the output in a log file.
my entrypoint.sh
#!/bin/bash
python3 manage.py migrate
python3 manage.py migrate catalog --database=catalog
python manage.py collectstatic --clear --noinput --verbosity 0
# Start Celery Workers
celery worker --workdir /app --app dri -l info &> /log/celery.log &
# Start Celery Beat
celery worker --workdir /app --app dri -l info --beat &> /log/celery_beat.log &
python3 manage.py runserver 0.0.0.0:8000

Starting from the same concept #shahaf has highlighted I solved starting from this other solution using bash -c in this way:
command: bash -c "celery -A app.tasks.celery beat & celery -A app.tasks.celery worker --loglevel=debug"

You can use celery beatX for beat. It is allowed (and recommended) to have multiple beatX instances. They use locks to synchronize.
Cannot say if it is production-ready, but it works for me like a charm (with -B key)

How can I automatically reload tasks modules with Celery daemon?

I am using Fabric to deploy a Celery broker (running RabbitMQ) and multiple Celery workers with celeryd daemonized through supervisor. I cannot for the life of me figure out how to reload the tasks.py module short of rebooting the servers.
/etc/supervisor/conf.d/celeryd.conf
[program:celeryd]
directory=/fab-mrv/celeryd
environment=[RABBITMQ crendentials here]
command=xvfb-run celeryd --loglevel=INFO --autoreload
autostart=true
autorestart=true
celeryconfig.py
import os
## Broker settings
BROKER_URL = "amqp://%s:%s#hostname" % (os.environ["RMQU"], os.environ["RMQP"])
# List of modules to import when celery starts.
CELERY_IMPORTS = ("tasks", )
## Using the database to store task state and results.
CELERY_RESULT_BACKEND = "amqp"
CELERYD_POOL_RESTARTS = True
Additional information
celery --version 3.0.19 (Chiastic Slide)
python --version 2.7.3
lsb_release -a Ubuntu 12.04.2 LTS
rabbitmqctl status ... 2.7.1 ...
Here are some things I have tried:
The celeryd --autoreload flag
sudo supervisorctl restart celeryd
celery.control.broadcast('pool_restart', arguments={'reload': True})
ps auxww | grep celeryd | grep -v grep | awk '{print $2}' | xargs kill -HUP
And unfortunately, nothing causes the workers to reload the tasks.py module (e.g. after running git pull to update the file). The gist of the relevant fab functions is available here.
The brokers/workers run fine after a reboot.

Just a shot in the dark, with the celeryd --autoreload option did you make sure you have one of the file system notification backends? It recommends PyNotify for linux, so I'd start by making sure you have that installed.

I faced a similar problem and was able to use Watchdog to reload the tasks.py tasks modules when there are changes detected. To install:
pip install watchdog
You can programmatically use the Watchdog API, for example, to monitor for directory changes in the file system. Additionally Watchdog provides an optional shell utility called watchmedo that can be used to execute commands on event. Here is an example that starts the Celery worker via Watchdog and reloads on any changes to .py files including changes via git pull:
watchmedo auto-restart --directory=./ --pattern="*.py" --recursive -- celery worker --app=worker.app --concurrency=1 --loglevel=INFO
Using Watchdog's watchmedo I was able to git pull changes and the respective tasks.py modules were auto reloaded without any reboot of the container or server.

Upstart script for Celery

I have celeryd daemons, working on small tasks. This daemon was configured with Upstart script
start on starting cessna
stop on stopping cessna
respawn
script
chdir /home/ubuntu/projects/cessna
exec su -c 'cd /home/ubuntu/projects/cessna; export MAX_POOL_SIZE="50";export newrelic-admin run-program celeryd -A cessna.celeryconfig --loglevel=info --concurrency=50 --pool=eventlet --queue=cessna_celery -E --pidfile=/tmp/cessna-3.pid >> /home/ubuntu/logs/cessna-w\
orker-3.log 2>> /home/ubuntu/errs/cessna-worker-3.log';
end script
Not so long I saw a lot of unack tasks in rabbitmq, no crashes in log files etc. We moved to native /etc/init.d/celeryd daemon, it solved the problem.
So, how it could be - Is there any relation between starting Celery with Upstart, and unacknowled tasks in Celery?