Running Celery as a Flask app with Gunicorn - python

I'm running Celery as a Flask microservice where it has tasks.py with tasks and manage.py contains the call to run the flask server.
This is part of the manage.py
class CeleryWorker(Command):
"""Starts the celery worker."""
name = 'celery'
capture_all_args = True
def run(self, argv):
if "down" in argv:
ret = subprocess.call(
['pkill', '-9', '-f', "my_app.celery"])
sys.exit(ret)
else:
ret = subprocess.call(
['celery', 'worker', '-A', 'my_app.celery'] + argv)
sys.exit(ret)
manager.add_command("celery", CeleryWorker())
I can start the service with either python manage.py runserver or `celery worker -A my_app.celery and it runs perfectly and registers all tasks in tasks.py.
But in production, i want to handle multiple requests to this microservice and want to add gunicorn to serve those requests. How do i do it?
I'm not able to figure out how i can run both my gunicorn command and celery command together.
Also, i'm running other api services using gunicorn in production from its create_app, since i dont need them to run the celery command.


Recommend to use Supervisor, which allow you to control a number of processes.
step1: pip install supervisor
step2: vi supervisord.conf
[program:flask_wsgi]
command=gunicorn -w 3 --worker-class gevent wsgi:app
directory=$SRC_PATH
autostart=true
[program:celery]
command=celery worker -A app.celery --loglevel=info
directory=$SRC_PATH
autostart=true
step3: run supervisord -c supervisord.conf

Related

How to configure celery on amazon linux 2?

I tried several ways to configure a celery worker service to run on beanstalk amazon linux 2, I followed this tutorial, however it doesn't work for me, I got the following error:
Failed at step EXEC spawning /var/app/current:$PYTHONPATH/celery: No such file or directory
This my 01_create_service_celery.sh:
#!/usr/bin/env bash
# Create the celery systemd service file
echo "[Unit]
Description=Celery service for __
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=30
User=root
WorkingDirectory=/var/app/current
ExecStart=$PYTHONPATH/celery -A notifications worker --loglevel=INFO
ExecReload=$PYTHONPATH/celery -A notifications worker --loglevel=INFO
EnvironmentFile=/opt/elasticbeanstalk/deployment/env
[Install]
WantedBy=multi-user.target
" | tee /etc/systemd/system/celery.service
# Start celery service
systemctl start celery.service
# Enable celery service to load on system start
systemctl enable celery.service

Celery not running as daemon

I launch Celery workers through the following python script
subprocess.Popen("celery -A test_celery worker -Q {} --loglevel=info -n {} -Ofair ".format(str(ContactID)+'_rec',worker1),shell=True,stdin=None, stdout=open(os.devnull, 'wb'), stderr=open(os.devnull, 'wb'))
When I run this Python script as sudo user or as service, the Celery does not work, but executing it as normal Ubuntu user works fine.

It can be done by specifying the user in the Service section of the .service file

Running celery worker + beat in the same container

My flask app is comprised of four containers: web app, postgres, rabbitMQ and Celery. Since I have celery tasks that run periodically, I am using celery beat. I've configured my docker-compose file like this:
version: '2'
services:
rabbit:
# ...
web:
# ...
rabbit:
# ...
celery:
build:
context: .
dockerfile: Dockerfile.celery
And my Dockerfile.celery looks like this:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-B", "-l", "INFO"]
While I read in the docs that I shouldn't go to production with the -B option, I hastily added it anyway (and forgot about changing it) and quickly learned that my scheduled tasks were running multiple times. For those interested, if you do a ps aux | grep celery from within your celery container, you'll see multiple celery + beat processes running (but there should only be one beat process and however many worker processes). I wasn't sure from the docs why you shouldn't run -B in production but now I know.
So then I changed my Dockerfile.celery to:
# ...code up here...
CMD ["celery", "-A", "app.tasks.celery", "worker", "-l", "INFO"]
CMD ["celery", "-A", "app.tasks.celery", "beat", "-l", "INFO"]
No when I start my app, the worker processes start but beat does not. When I flip those commands around so that beat is called first, then beat starts but the worker processes do not. So my question is: how do I run celery worker + beat together in my container? I have combed through many articles/docs but I'm still unable to figure this out.
EDITED
I changed my Dockerfile.celery to the following:
ENTRYPOINT [ "/bin/sh" ]
CMD [ "./docker.celery.sh" ]
And my docker.celery.sh file looks like this:
#!/bin/sh -ex
celery -A app.tasks.celery beat -l debug &
celery -A app.tasks.celery worker -l info &
However, I'm receiving the error celery_1 exited with code 0
Edit #2
I added the following blocking command to the end of my docker.celery.sh file and all was fixed:
tail -f /dev/null

docker run only one CMD, so only the first CMD get executed, the work around is to create a bash script that execute both worker and beat and use the docker CMD to execute this script

I got by putting in the entrypoint as explained above, plus I added the &> to have the output in a log file.
my entrypoint.sh
#!/bin/bash
python3 manage.py migrate
python3 manage.py migrate catalog --database=catalog
python manage.py collectstatic --clear --noinput --verbosity 0
# Start Celery Workers
celery worker --workdir /app --app dri -l info &> /log/celery.log &
# Start Celery Beat
celery worker --workdir /app --app dri -l info --beat &> /log/celery_beat.log &
python3 manage.py runserver 0.0.0.0:8000

Starting from the same concept #shahaf has highlighted I solved starting from this other solution using bash -c in this way:
command: bash -c "celery -A app.tasks.celery beat & celery -A app.tasks.celery worker --loglevel=debug"

You can use celery beatX for beat. It is allowed (and recommended) to have multiple beatX instances. They use locks to synchronize.
Cannot say if it is production-ready, but it works for me like a charm (with -B key)

Celery beat doesn't run periodic task

I have task
class BasecrmSync(PeriodicTask):
run_every = schedules.crontab(minute='*/1')
def run(self, **kwargs):
bc = basecrm.Client(access_token=settings.BASECRM_AUTH_TOKEN)
sync = basecrm.Sync(client=bc, device_uuid=settings.BASECRM_DEVICE_UUID)
sync.fetch(synchronize)
And celery config with db broker
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend'
BROKER_URL = 'django://'
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
I run
celery -A renuval_api worker -B --loglevel=debug
But it doesn't run task...
Also I've tried run by
python3 manage.py celery worker --loglevel=DEBUG -E -B -c 1 --settings=renuval_api.settings.local
But It uses amqp transport and I can't understand why.

I run a separate process for the beat function itself. I could never get periodic tasks to fire otherwise. Of course, I may have this completely wrong, but it works for me and has for some time.
For example, I have the celery worker with its app running in one process:
celery worker --app=celeryapp:app -l info --logfile="/var/log/celery/worker.log"
And I have the beat pointed to the same app in its own process:
celery --app=celeryapp:app beat
They are pointed at the same app and settings, and beat fires off the task which the worker picks up and does. This app is in the same code tree as my Django apps, but the processes are not running in Django. Perhaps you could run something like:
python3 manage.py celery beat --loglevel=DEBUG -E -B -c 1 --settings=renuval_api.settings.local
I hope that helps.

Celery Worker is offline

I am running a celery worker
foreman run python manage.py celery worker -E --maxtasksperchild=1000
And a celerymon
foreman run python manage.py celerymon
as well as celerycam
foreman run python manage.py celerycam
The django admin shows that my worker is offline and all tasks remain in the delayed state. I have tried killing and restarting it several times but it does not seem to be online.
Here is my configuration
BROKER_TRANSPORT = 'amqplib'
BROKER_POOL_LIMIT=0
BROKER_CONNECTION_MAX_RETRIES = 0
BROKER_URL = os.environ.get('AMQP_URL')
CELERY_RESULT_BACKEND = 'database'
CELERY_TASK_RESULT_EXPIRES = 14400

Categories