I need a queue to send data from ruby to python
The system is an application with a Ruby frontend and python backend and I'd rather not add another complicated piece. If it was ruby only I'd just go with delayed_job, but ruby->python is harder.
So
I'm looking for a simple database based queue (similar to delayed_job) for python for which I'm planning to hack a ruby 'producer' part.
Or just surprise me with a solution I haven't think of yet.
Maybe you could have a look at Celery.
Pretty old question, but just for anyone stumbling across this question now and looking for a simple answer that isn't Celery:
django-background-tasks is based Ruby's DelayedJob.
Django Background Task is a databased-backed work queue for Django,
loosely based around Ruby's DelayedJob library. This project was
adopted and adapted from this repo.
To avoid conflicts on PyPI we renamed it to django-background-tasks
(plural). For an easy upgrade from django-background-task to
django-background-tasks, the internal module structure were left
untouched.
In Django Background Task, all tasks are implemented as functions (or
any other callable).
There are two parts to using background tasks:
creating the task functions and registering them with the scheduler
setup a cron task (or long running process) to execute the tasks
Related
I'm working on a Django project, in which I will need the use of multithreading and multiprocessing concepts (Send and receive data from other servers such as PACS server/ I/O Threads... ).
The question I have is Django capable of applying multithreading /multiprocessing?
Thank you
By far the most popular tool in the Django world for doing this is Celery Here is a good intro tutorial
There are some more lightweight packages like Dramatiq and django-db-queue, which are intended for use cases where the configuration associated with Celery could be considered overkill.
You could, of course, 'roll-your-own' with the threading module, as sketched out in this answer
I'm building a web app using a Django framework. There isn't much user interaction with only a few static links, navbar, and a few plots which come from my app itself. The main part of the app comes from a python script which reads data from an external source, does some data processing on it, and then writes to my django database. Then after writing to the database a new page is created with information about the database entry. Note that there is no user interaction so no starting or stopping the task. I want the task to run in the background 24/7.
Thus far I've looked into celery and django-background-tasks. Celery seems like a bit much for my use case. I don't think I need a broker service as I just want to run 1 task which the user will never interact with. Additionally, I don't need multiple workers. Django-background-tasks seems like a good lightweight alternative but it seems it does not support indefinite tasks without having to refresh the task every once in a while (ideally I don't want this). Is there a tool that is better suited for this use case? Or am I just completely misunderstanding celery and django-background-tasks.
Update
Thanks for the comments, everyone! So I looked up what was mentioned by #D Malan and I think a tool like Supervisord might fit my use case. That is I can run a python script separately from the django application in the background and then have the python script interact with the django application. The only problem I have now is getting the process to interact with the django app. #deceze mentioned invoking a manage command from the python script. So this would involve creating a subprocess from the python script calling a custom django management command which updates my database? Or can I use the django.core.management.call_command but from a python file separate from the django application. If this is the case how would it know where to get the command from?
As far as I've seen, the native implementation of celery beat scheduler does not provide a way of adding and syncing scheduled tasks at runtime. There is django-celery-beat, but I do not want to add django dependency (which I don't really need).
I have found a couple of third-party packages backed by Redis (celery-redbeat, redisbeat);
do you know alternatives to achieve this goal? I was also thinking of subclassing Scheduler interface by myself, but it seems not so easy to gather all the necessary methods overriding.
I recommend celery-redbeat as it is a scheduler used in production by many companies.
I have a running CLI application in Python that uses threads to execute some workers. Now I am writing a GUI using electron for this application. For simple requests/responses I am using gRPC to communicate between the Python application and the GUI.
I am, however, struggling to find a proper publishing mechanism to push data to the GUI: gRPCs integrated streaming won't work since it uses generators; as already mentioned my longer, blocking tasks are executed using threads (subclasses of threading.Thread). Also I'd like to emit certain events (e.g., the progress) from within those threads.
Then I've found the Flasks SocketIO implementation, which is, however, a blocking execution and thus not really suited for what I have in mind - I'd have to again execute two processes (Flask and my CLI application)...
Another package I've found is websockets but I can't get my head around how I could implement this producer() function that they mention in the patterns.
My last idea would be to deploy a broker-based message system like Redis or simply fall back to the brokerless zmq, which is a bit of a hassle to setup for the GUI application.
So the simple question:
Is there any easy framework that allows to create a server-"task" in a Python that I can pass messages to publish to?
For anyone struggling with concurrency in python:
No, there isn't any simple framework. IMHO pythons' concurrency handling is a bit of a mess (compared to other languages like golang, where concurrency is built in). There's multiple major packages implementing this, one of them asyncio, but most of them are incompatible. I've ended up using a similar solution like proposed in this question.
We have a collection of Unix scripts (and/or Python modules) that each perform a long running task. I would like to provide a web interface for them that does the following:
Asks for relevant data to pass into scripts.
Allows for starting/stopping/killing them.
Allows for monitoring the progress and/or other information provided by the scripts.
Possibly some kind of logging (although the scripts already do logging).
I do know how to write a server that does this (e.g. by using Python's built-in HTTP server/JSON), but doing this properly is non-trivial and I do not want to reinvent the wheel.
Are there any existing solutions that allow for maintaining asynchronous server-side tasks?
Django is great for writing web applications, and the subprocess module (subprocess.Popen en .communicate()) is great for executing shell scripts. You can give it a stdin,stdout and stderr stream for communication if you want.
Answering my own question, I recently saw the announcement of Celery 1.0, which seems to do much of what I am looking for.
I would use SGE, but I think it could be overkill for your need...