So I have a scheduled celery beat task (celery.py):
#app.on_after_configure.connect
def setup_periodic_tasks(sender,
**kwargs):
sender.add_periodic_task(10.0, test_event, name='test')
And the task (events/tasks.py):
#shared_task
def test_event():
from .models import Event
Event.objects.create()
When the event is created, a receiver is fired, that should send a message to a channels group (events/receivers.py):
#receiver(post_save, sender=Event)
def event_post_add(sender, instance, created, *args, **kwargs):
if created:
print("receiver fired")
Group("test").send({
"text": json.dumps({
'type': 'test',
})
})
The main problem is that the receiver is being fired in the celery beat process, and nothing is getting sent via django channels. No error messages, nothing, it's simply not being sent.
How can I integrate these two so that I will be able to send messages to channels from celery background processes?
hi i dont know if you found a solution or not. but as i was stuck on this problem myself so i tried a work around.
i created a view for the message that needs to be send by websocket and make a request to it from celery beat
the view:
def send_message(request,uuid,name):
print('lamo')
ty = f"uuid_{uuid}"
data={
'message':f'{name} Driver is now Avaliable',
'sender':'HQ',
'id':str(uuid),
'to':str(uuid),
'type':'DriverAvailable',
}
channel_layer = get_channel_layer()
async_to_sync(channel_layer.group_send)(
ty,
{'type':'chat_message',
'message':data,
}
)
and the task:
def my_task():
list=[]
for d in Driver_api.objects.all():
if d.available_on !=None:
if d.available_on <= timezone.now():
d.available_on = None
d.save()
uuid = str(d.user.uuid)
requests.get(f'{DOMAIN}message/sendMessage/{uuid}/{d.name}')
logger.info('Adding {0}'.format(d.user.uuid))
return list
sorry for any neglects or overlooks in my approach to the problem.
Signals are actually not being asynchronous in Django.
So in:
#shared_task
def test_event():
from .models import Event
Event.objects.create() # This will fire a signal and the signal will
# still be interpreted by celery
This issue is described at length in the following link:
https://githubmemory.com/repo/CJWorkbench/channels_rabbitmq/issues/37
I've checked the claim of reconnections and suboptimal perfomance of redis_channels (as described in link) and I couldn't find it happening.
here is my code that's working and Celery is sending a message to django channels.
class DjangoView(APIView):
def get(request):
send_message_to_channels.delay()
#shared_task
def send_message_to_channels():
send_test_message("Hello from celery", 200)
def send_test_message(message: str, code: int):
channel_layer = get_channel_layer()
channel_name = "celery-channels-test"
async_to_sync(channel_layer.group_send)(channel_name, {
'type': 'consumer.test.message',
'message': message,
'code': code
})
Pre Requisites to make my code work
install channels_redis (other packages just fail)
define channel_layer in django settings.py pointing to redis database
If this doesn't work I guess simple but very nasty approach would be to fire a request to a view in django like in #cosmo_boi answer
Related
I have a view that sends a message to a RabbitMQ queue.
message = {'origin': 'Bytes CSV',
'data': {'csv_key': str(csv_entry.key),
'csv_fields': csv_fields
'order_by': order_by,
'filters': filters}}
...
queue_service.send(message=message, headers={}, exchange_name=EXCHANGE_IN_NAME,
routing_key=MESSAGES_ROUTING_KEY.replace('#', 'bytes_counting.create'))
On my consumer, I have a long process to generate a CSV.
def create(self, data):
csv_obj = self._get_object(key=data['csv_key'])
if csv_obj.status == CSVRequestStatus.CANCELED:
self.logger.info(f'CSV {csv_obj.key} was canceled by the user')
return
result = self.generate_result_data(filters=data['filters'], order_by=data['order_by'], csv_obj=csv_obj)
csv_data = self._generate_csv(result=result, csv_fields=data['csv_fields'], csv_obj=csv_obj)
file_key = self._post_csv(csv_data=csv_data, csv_obj=csv_obj)
csv_obj.status = CSVRequestStatus.READY
csv_obj.status_additional = CSVRequestStatusAdditional.SUCCESS
csv_obj.file_key = file_key
csv_obj.ready_at = timezone.now()
csv_obj.save(update_fields=['status', 'status_additional', 'ready_at', 'file_key'])
self.logger.info(f'CSV {csv_obj.name} created')
The long proccess happens inside self._generate_csv, because self.generate_result_data returns a queryset, which is lazy.
As you can see, if a user changes the status of the csv_request through an endpoint BEFORE the message starts to be consumed the proccess will not be evaluated. My goal is to let this happen during the execution of self._generate_csv.
So far I tried to use Threading, but unsuccessfully.
How can I achive my goal?
Thanks a lot!
Why don't you checkout Celery library ? Using celery with django with RabbitMQ backend is much easier than directly leveraging rabbitmq queues.
Celery has an inbuilt function revoke to terminate an ongoing task:
>>> from celery.task.control import revoke
>>> revoke(task_id, terminate=True)
related SO answer
celery docs
For your use case, you probably want something like (code snippets):
## celery/tasks.py
from celery import app
#app.task(queue="my_queue")
def create_csv(message):
# ...snip...
pass
## main.py
from celery import uuid, current_app
def start_task(task_id, message):
current_app.send_task(
"create_csv",
args=[message],
task_id=task_id,
)
def kill_task(task_id):
current_app.control.revoke(task_id, terminate=True)
## signals.py
from django.dispatch import receiver
from .models import MyModel
from .main import kill_task
# choose appropriate signal to listen for DB change
#receiver(models.signals.post_save, sender=MyModel)
def handler(sender, instance, **kwargs):
kill_task(instance.task_id)
Use celery.uuid to generate task IDs which can be stored in DB or cache and use the same task ID to control the task i.e. request termination.
Since self._generate_csv is the slowest, the obvious solution would be to work with this function.
To do this, you can divide the creation of the csv file into several pieces. After creating each piece, check the status and see if you can continue to create the file. At the very end, glue all the pieces into a finished file.
Here is a method for combining multiple files into one.
I want to send a telegram notification when the user performs a specific task in my flask application. I'm using python-telegram-bot to handle telegram. Here's the simplified code:
#app.route('/route')
def foo():
# do some stuff...
# if stuff is completed successfully - send the notification
app.telegram_notifier.send_notification(some_data)
return ''
I'm using messagequeue from python-telegram-bot to avoid flood limits. As you might have expected, that's not working and I'm getting the following error:
telegram.ext.messagequeue.DelayQueueError: Could not process callback in stopped thread
I tried to launch it in a separate daemon thread but I also ended up with that error.
This functionality is used only once in the entire application so I want things to be simple and don't want to install more dependencies like Celery.
Is there a way to achieve this using threads or some other simple way?
EDIT (more code)
Here's simplified implementation of the telegram bot:
from telegram import Bot, ParseMode
from telegram.ext import messagequeue as mq
class Notifier(Bot):
def __init__(self):
super(Notifier, self).__init__('my token')
# Message queue setup
self._is_messages_queued_default = True
self._msg_queue = mq.MessageQueue(all_burst_limit=3, all_time_limit_ms=3500)
self.chat_id = 'my chat ID'
#mq.queuedmessage
def _send_message(self, *args, **kwargs):
return super(Notifier, self).send_message(*args, **kwargs)
def send_notification(self, data: str):
msg = f'Notification content: <b>{data}</b>'
self._send_message(self.chat_id, msg, ParseMode.HTML)
In the app factory method:
from notifier import Notifier
def init_app():
app = Flask(__name__)
app.telegram_notifier = Notifier()
# some other init stuff...
return app
The thing with threads I tried:
#app.route('/route')
def foo():
# do some stuff...
# First method
t = Thread(target=app.telegram_notifier.send_notification, args=('some notification data',), daemon=True)
t.start()
# Second method
t = Thread(target=app.telegram_notifier.send_notification, args=('some notification data',))
t.start()
t.join()
return ''
I am currently trying to setup celery to handle responses from a chatbot and forward those responses to a user.
The chatbot hits the /response endpoint of my server, that triggers the following function in my server.py module:
def handle_response(user_id, message):
"""Endpoint to handle the response from the chatbot."""
tasks.send_message_to_user.apply_async(args=[user_id, message])
return ('OK', 200,
{'Content-Type': 'application/json; charset=utf-8'})
In my tasks.py file, I import celery and create the send_message_to_user function:
from celery import Celery
celery_app = Celery('tasks', broker='redis://')
#celery_app.task(name='send_message_to_user')
def send_message_to_user(user_id, message):
"""Send the message to a user."""
# Here is the logic to send the message to a specific user
My problem is, my chatbot may answer multiple messages to a user, so the send_message_to_user task is properly put in the queue but then a race condition arises and sometimes the messages arrive to the user in the wrong order.
How could I make each send_message_to_user task wait for the previous task with the same name and with the same argument "user_id" before executing it ?
I have looked at this thread Running "unique" tasks with celery but a lock isn't my solution, as I don't want to implement ugly retries when the lock is released.
Does anyone have any idea how to solve that issue in a clean(-ish) way ?
Also, it's my first post here so I'm open to any suggestions to improve my request.
Thanks!
I'm trying to send emails and below works perfectly fine if executed through the web server. However, when I try to send the task to Celery, I always get an Assertion Error returned telling me that "to" needs to be a list or tuple.
I don't want the emails to be sent via the web server as it will slow things down so if anyone can help me fix this, that would be greatly appreciated.
from celery import Celery
from django.core.mail import send_mail, EmailMessage
app = Celery('tasks', backend='amqp', broker='amqp://')
#app.task
def send_mail_link():
subject = 'Thanks'
message = 'body'
recipients = ['someemail#gmail.com']
email = EmailMessage(subject=subject, body=message, from_email='info#example.com', to=recipients)
email.send()
I'm not 100% sure why, but I made some changes and it now works with no errors.
I removed the import for send_mail and changed the name of the method from send_mail_link() to send_link(). I also restarted the Celery worker and now everything works as it should.
New code is:
from celery import Celery
from django.core.mail import EmailMessage
app = Celery('tasks', backend='amqp', broker='amqp://')
#app.task
def send_link():
subject = 'Thanks'
message = 'body'
recipients = ['someemail#gmail.com']
email = EmailMessage(subject=subject, body=message, from_email='info#example.com', to=recipients)
email.send()
Hopefully somebody in the future may find this helpful.
I am trying to create a Tornado application with several chats. The chats should be based on HTML5 websocket. The Websockets communicate nicely, but I always run into the problem that each message is posted twice.
The application uses four classes to handle the chat:
Chat contains all written messages so far and a list with all waiters which should be notified
ChatPool serves as a lookup for new Websockets - it creates a new chat when there is no one with the required scratch_id or returns an existing chat instance.
ScratchHandler is the entry point for all HTTP requests - it parses the base template and returns all details of client side.
ScratchWebSocket queries the database for user information, sets up the connection and notifies the chat instance if a new message has to be spread.
How can I prevent that the messages are posted several times?
How can I build a multi chat application with tornado?
import uuid
import tornado.websocket
import tornado.web
import tornado.template
from site import models
from site.handler import auth_handler
class ChatPool(object):
# contains all chats
chats = {}
#classmethod
def get_or_create(cls, scratch_id):
if scratch_id in cls.chats:
return cls.chats[scratch_id]
else:
chat = Chat(scratch_id)
cls.chats[scratch_id] = chat
return chat
#classmethod
def remove_chat(cls, chat_id):
if chat_id not in cls.chats: return
del(cls.chats[chat_id])
class Chat(object):
def __init__(self, scratch_id):
self.scratch_id = scratch_id
self.messages = []
self.waiters = []
def add_websocket(self, websocket):
self.waiters.append(websocket)
def send_updates(self, messages, sending_websocket):
print "WAITERS", self.waiters
for waiter in self.waiters:
waiter.write_message(messages)
self.messages.append(messages)
class ScratchHandler(auth_handler.BaseHandler):
#tornado.web.authenticated
def get(self, scratch_id):
chat = ChatPool.get_or_create(scratch_id)
return self.render('scratch.html', messages=chat.messages,
scratch_id=scratch_id)
class ScratchWebSocket(tornado.websocket.WebSocketHandler):
def allow_draft76(self):
# for iOS 5.0 Safari
return True
def open(self, scratch_id):
self.scratch_id = scratch_id
scratch = models.Scratch.objects.get(scratch_id=scratch_id)
if not scratch:
self.set_status(404)
return
self.scratch_id = scratch.scratch_id
self.title = scratch.title
self.description = scratch.description
self.user = scratch.user
self.chat = ChatPool.get_or_create(scratch_id)
self.chat.add_websocket(self)
def on_close(self):
# this is buggy - only remove the websocket from the chat.
ChatPool.remove_chat(self.scratch_id)
def on_message(self, message):
print 'I got a message'
parsed = tornado.escape.json_decode(message)
chat = {
"id": str(uuid.uuid4()),
"body": parsed["body"],
"from": self.user,
}
chat["html"] = tornado.escape.to_basestring(self.render_string("chat-message.html", message=chat))
self.chat.send_updates(chat, self)
NOTE: After the feedback from #A. Jesse I changed the send_updates method from Chat. Unfortunately, it still returns double values.
class Chat(object):
def __init__(self, scratch_id):
self.scratch_id = scratch_id
self.messages = []
self.waiters = []
def add_websocket(self, websocket):
self.waiters.append(websocket)
def send_updates(self, messages, sending_websocket):
for waiter in self.waiters:
if waiter == sending_websocket:
continue
waiter.write_message(messages)
self.messages.append(messages)
2.EDIT: I compared my code with the example provided demos. In the websocket example a new message is spread to the waiters through the WebSocketHandler subclass and a class method. In my code, it is done with a separated object:
From the demos:
class ChatSocketHandler(tornado.websocket.WebSocketHandler):
#classmethod
def send_updates(cls, chat):
logging.info("sending message to %d waiters", len(cls.waiters))
for waiter in cls.waiters:
try:
waiter.write_message(chat)
except:
logging.error("Error sending message", exc_info=True)
My application using an object and no subclass of WebSocketHandler
class Chat(object):
def send_updates(self, messages, sending_websocket):
for waiter in self.waiters:
if waiter == sending_websocket:
continue
waiter.write_message(messages)
self.messages.append(messages)
If you want to create a multi-chat application based on Tornado I recommend you use some kind of message queue to distribute new message. This way you will be able to launch multiple application process behind a load balancer like nginx. Otherwise you will be stuck to one process only and thus be severely limited in scaling.
I updated my old Tornado Chat Example to support multi-room chats as you asked for. Have a look at the repository:
Tornado-Redis-Chat
Live Demo
This simple Tornado application uses Redis Pub/Sub feature and websockets to distribute chat messages to clients. It was very easy to extend the multi-room functionality by simply using the chat room ID as the Pub/Sub channel.
on_message sends the message to all connected websockets, including the websocket that sent the message. Is that the problem: that messages are echoed back to the sender?