We Keep Coding

How to run django's "python manage.py runserver" , celery's "celery -A app_name worker -l info" and redis-server in one command

I have recently started with django. And I started doing a small project. I've been using celery with redis worker. And every to use celery and redis I have to run the celery and redis server and then django server. Which is a bit lengthy process.
I have two questions.
1. Am I doing the right thing by running the servers everytime or are there any other right method to this process?
2. If I'm in the right direction, is there any method to do this?
I tried circus.ini , but it did not work.

If you use UNIX system:
For this purpose you can get along just with bash. Just run celery and redis in background - use & command.
redis-server & celery -A app_name worker -l info & python manage.py runserver
Disadvantage of this approach - redis and celery will work in the background even after a shutdown of django dev server. So you need to terminate these processes. See this unix se answer for examples how to do that.
So you can create 2 bash scripts start.sh (contains commands with &) and cleanup.sh (terminate processes) and run them respectively.
For production see purpose #2
Use systemd or supervisor. You need to create conf files for your daemons and then run them.

Building upon Yevhenii M.'s answer, you can start a subshell command with a trap to kill all running processes in that subshell when you hit Ctrl+C:
(trap "kill 0" SIGINT; redis-server & celery -A app_name worker -l info & python manage.py runserver)
or as a more readable multiline command:
(
trap "kill 0" SIGINT
redis-server &
celery -A app_name worker -l info &
python manage.py runserver
)
Another option is to use a Procfile manager, but that requires installing additional dependencies/programs. Something like foreman or one of it's ports in other languages:
forego - Go
node-foreman - Node.js
gaffer - Java/JVM
goreman - Go
honcho - python
proclet - Perl
shoreman - shell
crank - Crystal
houseman - Haskell
(Source: foreman's README)
For this you create a Procfile (file in your project root) where you specify which commands to run:
redis: redis-server
worker: celery -A app_name worker
web: python manage.py runserver
Then run foreman start

Running celery worker + beat in the same container

My flask app is comprised of four containers: web app, postgres, rabbitMQ and Celery. Since I have celery tasks that run periodically, I am using celery beat. I've configured my docker-compose file like this:
version: '2'
services:
rabbit:
# ...
web:
# ...
rabbit:
# ...
celery:
build:
context: .
dockerfile: Dockerfile.celery
And my Dockerfile.celery looks like this:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-B", "-l", "INFO"]
While I read in the docs that I shouldn't go to production with the -B option, I hastily added it anyway (and forgot about changing it) and quickly learned that my scheduled tasks were running multiple times. For those interested, if you do a ps aux | grep celery from within your celery container, you'll see multiple celery + beat processes running (but there should only be one beat process and however many worker processes). I wasn't sure from the docs why you shouldn't run -B in production but now I know.
So then I changed my Dockerfile.celery to:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-l", "INFO"]
CMD ["celery", "-A", "app.tasks.celery", "beat", "-l", "INFO"]
No when I start my app, the worker processes start but beat does not. When I flip those commands around so that beat is called first, then beat starts but the worker processes do not. So my question is: how do I run celery worker + beat together in my container? I have combed through many articles/docs but I'm still unable to figure this out.
EDITED
I changed my Dockerfile.celery to the following:
ENTRYPOINT [ "/bin/sh" ]
CMD [ "./docker.celery.sh" ]
And my docker.celery.sh file looks like this:
#!/bin/sh -ex
celery -A app.tasks.celery beat -l debug &
celery -A app.tasks.celery worker -l info &
However, I'm receiving the error celery_1 exited with code 0
Edit #2
I added the following blocking command to the end of my docker.celery.sh file and all was fixed:
tail -f /dev/null

docker run only one CMD, so only the first CMD get executed, the work around is to create a bash script that execute both worker and beat and use the docker CMD to execute this script

I got by putting in the entrypoint as explained above, plus I added the &> to have the output in a log file.
my entrypoint.sh
#!/bin/bash
python3 manage.py migrate
python3 manage.py migrate catalog --database=catalog
python manage.py collectstatic --clear --noinput --verbosity 0
# Start Celery Workers
celery worker --workdir /app --app dri -l info &> /log/celery.log &
# Start Celery Beat
celery worker --workdir /app --app dri -l info --beat &> /log/celery_beat.log &
python3 manage.py runserver 0.0.0.0:8000

Starting from the same concept #shahaf has highlighted I solved starting from this other solution using bash -c in this way:
command: bash -c "celery -A app.tasks.celery beat & celery -A app.tasks.celery worker --loglevel=debug"

You can use celery beatX for beat. It is allowed (and recommended) to have multiple beatX instances. They use locks to synchronize.
Cannot say if it is production-ready, but it works for me like a charm (with -B key)

Restart celery beat and worker during Django deployment

I am using celery==4.1.0 and django-celery-beat==1.1.0.
I am running gunicorn + celery + rabbitmq with Django.
This is my config for creating beat and worker
celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach
During Django deployment I am doing following:
rm -f celerybeat.pid
rm -f celeryd.pid
celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach
service nginx restart
service gunicorn stop
sleep 1
service gunicorn start
I want to restart both celery beat and worker and it seems that this logic works. But I noticed that celery starts to use more and more memory during deployment and after several deployments I hit 100% memory use. I tried different server setups and it seems that it is not related.

rabbitmq may be to blame for high memory usage. Can you safely restart rabbit?
Also can you confirm that after a restart there is the expected amount of workers?

You are starting 2 new workers for every deployment without stopping/killing the previous workers.
During deployment, stop the existing workers with
kill -9 $PID
kill -9 `cat /var/run/myProcess.pid`
Alternatively, you can just kill all the workers with
pkill -9 celery
Now you can start workers as usual.
celery -A myproject beat -l info -f /var/log/celery/celery.log --detach
celery -A myproject worker -l info -f /var/log/celery/celery.log --detach

Django with Unicorn and Nginx, creating gunicorn logs

I Launch my Django app in a docker container, using a startup script as per the below:-
#!/bin/bash
python manage.py collectstatic --noinput
python manage.py makemigrations
python manage.py migrate
/etc/init.d/celeryd start
/etc/init.d/celerybeat start
exec gunicorn itapp.wsgi -b 0.0.0.0:8000
Unicorn and ngnix every now and then give me a 502 error, so I went to look for logs and it looks by default gunicorn does not log any? im on version 19.7.1
so I added the to the exec guncicorn command:
exec gunicorn itapp.wsgi -b 0.0.0.0:8000 --error-logfile /var/log/gunicorn/errors.log , --log-file /var/log/gunicorn/access.log
Which I goy form the gunicorn documentation. however now Gunicorn fails to launch with the following error:-
usage: gunicorn [OPTIONS] [APP_MODULE]
gunicorn: error: unrecognized arguments: ,
How can I create the gunicorn logs to debug these errors?
Thanks

Remove comma and you should be fine:
exec gunicorn itapp.wsgi -b 0.0.0.0:8000 --error-logfile /var/log/gunicorn/errors.log --log-file /var/log/gunicorn/access.log

Run Django Celery Nohup in Crontab

I want to run some tasks in django. I'm using Celery to do this. Usually, I run this command to execute the tasks:
source myvirtualenvpath/bin/activate
nohup python manage.py celeryd -E -B --loglevel=DEBUG < /dev/null &>/dev/null &
I want to do this each time the machine reboot with a crontab. How can I
do this?
Thanks

I solved making this:
nohup path_to_virtual_env/bin/python path_to_project/manage.py celeryd -E -B --loglevel=DEBUG < /dev/null &>/dev/null &