We Keep Coding

Sentry logging integration prevents sentry events being sent (Python)

I have a python 3 flask server running through WSGI. I have a config file that is imported by my api code that handles environment variables, and I set up sentry in this file. This is my code which is setup exactly as described in the sentry docs https://docs.sentry.io/platforms/python/logging/
if sentry_dsn is not None:
sentry_logging = LoggingIntegration(
level=logging.INFO,
event_level=logging.CRITICAL,
)
LOG.debug(f"Initialising sentry for environment {sentry_environment}")
sentry_sdk.init(
sentry_dsn,
environment=sentry_environment,
integrations=[sentry_logging],
)
else:
LOG.warn("Sentry key not set up")
The problem is this does not send any events to sentry, in exception logs or even uncaught exceptions. I know that the DSN is correct because if I set up sentry like this, all uncaught exceptions as well as error and exception logs are sent to my sentry project:
if sentry_dsn is not None:
LOG.debug("Initialising sentry")
sentry_sdk.init(sentry_dsn, environment=sentry_environment)
else:
LOG.warn("Sentry key not set up")
I've tried the setup with the debug=True setting in the sentry init and logs confirm that sentry intialises and sets up integrations. But when an event occurs that it should report, there is no log or anything recorded by sentry.
Any help would be appreciated

Turns out that flask was intercepting the exceptions because the default sentry integrations exclude Flask. This meant that nothing was reaching sentry. I added the flask integration according to sentry docs (https://docs.sentry.io/platforms/python/flask/) and sentry messages were sent as I expected.

Django steps or process messages via REST

For learning purpose I want to implement the next thing:
I have a script that runs selenium for example in the background and I have some log messages that help me to see what is going on in the terminal.
But I want to get the same messages in my REST request to the Angular app.
print('Started')
print('Logged in')
...
print('Processing')
...
print('Success')
In my view.py file
class RunTask(viewsets.ViewSet):
queryset = Task.objects.all()
#action(detail=False, methods=['GET'], name='Run Test Script')
def run(self, request, *args, **kwargs):
task = task()
if valid['success']:
return Response(data=task)
else:
return Response(data=task['message'])
def task()
print('Staring')
print('Logged in')
...
print('Processing')
...
print('Success')
return {
'success': True/False,
'message': 'my status message'
}
Now it shows me only the result of the task. But I want to get the same messages to indicate process status in frontend.
And I can't understand how to organize it.
Or how I can tell angular about my process status?

Unfortunately, it's not that simple. Indeed, the REST API lets you start the task, but since it runs in the same thread, the HTTP request will block until the task is finished before sending the response. Your print statements won't appear in the HTTP response but on your server output (if you look at the shell where you ran python manage.py runserver, you'll see those print statements).
Now, if you wish to have those output in real-time, you'll have to look for WebSockets. They allow you to open a "tunnel" between the browser and the server, and send/receive messages in real-time. The django-channels library allow you to implement them.
However, for long-running background tasks (like a Selenium scraper), I would advise to look into the Celery task queue. Basically, your Django process will schedule task into the queue. The tasks into the queue will then be executed by one (or more !) "worker" processes. The advantage of this is that your Django process won't be blocked by the long task: it justs add some work into the queue and then respond.
When you add tasks in the queue, Celery will give you a unique identifier for this task, that you can return in the HTTP response. You can then very well implement another endpoint which takes a task id in parameter and return the state of the task (is it pending ? done ? failed ?).
For this to work, you'll have to setup a "broker", a kind of database that will store the tasks to do and their results (typically RabbitMQ or Redis). Celery documentation explains this well: https://docs.celeryproject.org/en/latest/getting-started/brokers/index.html
Either way you choose, it's not a trivial thing and will need quite some work before having some results ; but it's interesting to see how it expands the possibilities of a classical HTTP server.

Python signal from celery task

I am running a celery task which on success runs subtask to send signal.
#celery.task(name='sendmail')
def send_async_email(msg):
return mail.send(msg)
def send_mail(msg):
// do some processing
send_async_email.apply_async((msg,), link=send_email_signal.s(msg))
#celery.task
def send_email_signal(result, email_type, msg):
email_sent_signal.send(msg, email_type=email_type)
signals.email_sent_signal.connect(track.track_emails_sent)
def track_emails_sent(msg):
// adds logs to logging system
Problem is when i send email everything works expected but I see duplicate entry in my logging system.
I receive 1 email as expected, as per celery flower send_email_signal ran once. But in logs we have 2 log entries.
I have multiple celery workers running in the celery box. Want to understand how the python signal sent from celery callback task is handled.

Celery have task wait for completion of same task called previously with shared argument

I am currently trying to setup celery to handle responses from a chatbot and forward those responses to a user.
The chatbot hits the /response endpoint of my server, that triggers the following function in my server.py module:
def handle_response(user_id, message):
"""Endpoint to handle the response from the chatbot."""
tasks.send_message_to_user.apply_async(args=[user_id, message])
return ('OK', 200,
{'Content-Type': 'application/json; charset=utf-8'})
In my tasks.py file, I import celery and create the send_message_to_user function:
from celery import Celery
celery_app = Celery('tasks', broker='redis://')
#celery_app.task(name='send_message_to_user')
def send_message_to_user(user_id, message):
"""Send the message to a user."""
# Here is the logic to send the message to a specific user
My problem is, my chatbot may answer multiple messages to a user, so the send_message_to_user task is properly put in the queue but then a race condition arises and sometimes the messages arrive to the user in the wrong order.
How could I make each send_message_to_user task wait for the previous task with the same name and with the same argument "user_id" before executing it ?
I have looked at this thread Running "unique" tasks with celery but a lock isn't my solution, as I don't want to implement ugly retries when the lock is released.
Does anyone have any idea how to solve that issue in a clean(-ish) way ?
Also, it's my first post here so I'm open to any suggestions to improve my request.
Thanks!

Celery Why Does Task Stay In Queue

So I am using Celery with RabbitMQ. I have a RESTful API that registers a user. I am using remote Celery worker to send a registration email asynchronously so my API can return fast response.
from .tasks import send_registration_email
def register_user(user_data):
# save user to the database etc
send_registration_email.delay(user.id)
return {'status': 'success'}
This works fine. Email is being sent in a non blocking asynchronous way (and can be retried if fails which is cool). The problem is when I look at RabbitMQ management console. I can see that the send_registration_email has created a random queue. Something like:
I can see that the task has been successfully executed. So why does the random queue stays in RabbitMQ forever? This is the task payload:
{"status": "SUCCESS", "traceback": null, "result": true, "task_id": "aad10877-3508-4179-a5fb-99f1bd0b8b2f", "children": []}

This normal behaviour, if you have configured CELERY_RESULT_BACKEND in your settings. Please check here: Celery result backend description
You could disable result backend, or decrease each message life time.