In creating scheduled tasks I've used both Cron and a specially set up daemon for django.
Cron is silly-simple, and the daemon (in my opinion) might be excessive. The daemon set up an independent Django instance.
Django itself (If I'm not mistaken) runs as a daemon anyway, correct?
I'm wondering - how do you schedule tasks within the Django environment without leaving off from standard use?
You can use Celery to run periodic tasks but depending on what are you trying to do it could be overkill.
If your use case it's simple, cron+management command it's way easier. You can use Kronos, django-cron or any of this libraries for this
Related
I'm having different python programs doing long polling at different machines, and am thinking of a queuing based mechanism to manage the load and provide an async job functionality.
These programs are standalone, and aren't part of any framework.
I'm primarily thinking about Celery due to its ability for multi-processing and sharing tasks across multiple celery workers. Is celery a good choice here, or am I better off simply using an event based system with RabbitMQ directly?
I would say yes - Celery is definitely a good choice! We do have tasks that run sometimes for over 20 hours, and Celery works just fine. Furthermore it is extremely simple to setup and use (Celery + Redis is supersimple).
As far as I've seen, the native implementation of celery beat scheduler does not provide a way of adding and syncing scheduled tasks at runtime. There is django-celery-beat, but I do not want to add django dependency (which I don't really need).
I have found a couple of third-party packages backed by Redis (celery-redbeat, redisbeat);
do you know alternatives to achieve this goal? I was also thinking of subclassing Scheduler interface by myself, but it seems not so easy to gather all the necessary methods overriding.
I recommend celery-redbeat as it is a scheduler used in production by many companies.
I'm in need of a way to execute external long running processes from a web app written in Django and Python.
Right now I'm using Supervisord and the API. My problem with this solution is that it's very static. I need to build the commands from my app instead of having to pre configure Supervisord with all possible commands. The argument and the command is dynamic.
I need to execute the external process, save a pid/identifier and later be able to check if it's still alive and running and stop the process.
I've found https://github.com/mnaberez/supervisor_twiddler to add processes on the fly to supervisord. Maybe that's the best way to go?
Any other ideas how to best solve this problem?
I suggest you have a look at this post:
Processing long-running Django tasks using Celery + RabbitMQ + Supervisord + Monit
As the title says, there are a few additional components involved (mainly celery and rabbitMQ), but these are good and proven technologies for this kind of requirement.
I've been able to deploy a test application by using pyramid with pserve and running pceleryd (I just send an email without blocking while it is sent).
But there's one point that I don't understand: I want to run my application with mod_wsgi, and I don't understand if I can can do it without having to run pceleryd from a shell, but if I can do something in the virtualhost configuration.
Is it possible? How?
There are technically ways you could use Apache/mod_wsgi to manage a process distinct from that handling web requests, but the pain point is that Celery will want to fork off further worker processes. Forking further processes from a process managed by Apache can cause problems at times and so is not recommended.
You are thus better of starting up Celery process separately. One option is to use supervisord to start it up and manage it.
I want to give celery a try. I'm interested in a simple way to schedule crontab-like tasks, similar to Spring's quartz.
I see from celery's documentation that it requires running celeryd as a daemon process. Is there a way to refrain from running another external process and simply running this embedded in my django instance? Since I'm not interested in distributing the work at the moment, I'd rather keep it simple.
Add CELERY_ALWAYS_EAGER=True option in your django settings file and all your tasks will be executed locally. Seems like for the periodic tasks you have to execute celery beat as well.