We Keep Coding

How to debug a stuck asyncio coroutine in Python?

There are lots of coroutines in my production code, which are stuck at unknown position while processing request. I attached gdb with Python support extension to the process, but it doesn't show the exact line in the coroutine where the process is stuck, only primary stack trace. Here is a minimal example:
import asyncio
async def hello():
await asyncio.sleep(30)
print('hello world')
asyncio.run(hello())
(gdb) py-bt
Traceback (most recent call first):
File "/usr/lib/python3.8/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
File "/usr/lib/python3.8/asyncio/base_events.py", line 2335, in _run_once
File "/usr/lib/python3.8/asyncio/base_events.py", line 826, in run_forever
None, getaddr_func, host, port, family, type, proto, flags)
File "/usr/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/usr/lib/python3.8/asyncio/runners.py", line 299, in run
File "main.py", line 7, in <module>
GDB shows a trace that ends on line 7, but the code is obviously stuck on line 4. How to make it show a more complete trace with nested coroutines?

You can use the aiodebug.log_slow_callbacks.enable(0.05)
Follow for more : https://pypi.org/project/aiodebug/

handle postgress exceptions when inserting from pipeline

I am trying to to handle errors which may be thrown by psycopg2, so I imported the errors the pipeline was facing and got a weird error when I ran the spider.
pipelines.py
from psycopg2.errors import InvalidTextRepresentation
from psycopg2.errors import InFailedSqlTransaction
...
try:
self.cursor.execute(
"""INSERT INTO links (url,price) VALUES (%s, %s)""",
(
item["productURL"],
item["price"]
))
except InvalidTextRepresentation, InFailedSqlTransaction:
pass
I got this error directly and the spider didn't run
Unhandled error in Deferred:
Temporarily disabling observer LegacyLogObserverWrapper(<bound method PythonLoggingObserver.emit of <twisted.python.log.PythonLoggingObserver object at 0x7ff963789e80>>) due to exception: [Failure instance: Traceback: <class 'TypeError'>: _findCaller() takes from 1 to 2 positional arguments but 3 were given
/usr/lib/python3/dist-packages/twisted/internet/defer.py:953:__del__
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:270:critical
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:144:emit
--- <exception caught here> ---
/usr/lib/python3/dist-packages/twisted/logger/_observer.py:131:__call__
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:93:__call__
/usr/lib/python3/dist-packages/twisted/python/log.py:595:emit
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:154:publishToNewObserver
/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py:115:__call__
/usr/lib/python3.8/logging/__init__.py:1500:log
/usr/lib/python3.8/logging/__init__.py:1565:_log
]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 953, in __del__
log.critical("Unhandled error in Deferred:",
File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 270, in critical
self.emit(LogLevel.critical, format, **kwargs)
File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 144, in emit
self.observer(event)
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/logger/_observer.py", line 131, in __call__
observer(event)
File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 93, in __call__
self.legacyObserver(event)
File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 595, in emit
_publishNew(self._newObserver, eventDict, textFromEventDict)
File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver
observer(eventDict)
File "/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py", line 115, in __call__
self.logger.log(
File "/usr/lib/python3.8/logging/__init__.py", line 1500, in log
self._log(level, msg, args, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1565, in _log
fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
builtins.TypeError: _findCaller() takes from 1 to 2 positional arguments but 3 were given
I changed back to except: instead of except InvalidTextRepresentation, InFailedSqlTransaction: it worked fine, Is there something I am doing wrong or is there a better to handle the errors of psycopg2

Mongodb and Celery "pymongo.errors.OperationFailure: command createIndexes requires authentication"

I am working Celery with Mongodb( as backend and as a broker).
I follow this tutorial on how to set them up: https://skillachie.com/2013/06/15/intro-celery-and-mongodb/
When I have the security enable on the /etc/mongod.conf file like this:
security:
authorization: enabled
and I call the .get() to get the results of the tasks that I have set up I get this error:
Traceback (most recent call last):
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/kombu/utils/__init__.py", line 323, in __get__
return obj.__dict__[self.__name__]
KeyError: 'collection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/result.py", line 169, in get
no_ack=no_ack,
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/base.py", line 229, in wait_for
meta = self.get_task_meta(task_id)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/base.py", line 307, in get_task_meta
meta = self._get_task_meta_for(task_id)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/mongodb.py", line 158, in _get_task_meta_for
obj = self.collection.find_one({'_id': task_id})
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/kombu/utils/__init__.py", line 325, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/celery/backends/mongodb.py", line 246, in collection
collection.ensure_index('date_done', background='true')
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 2028, in ensure_index
self.__create_index(keys, kwargs, session=None)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 1894, in __create_index
session=session)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/collection.py", line 250, in _command
user_fields=user_fields)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/pool.py", line 613, in command
user_fields=user_fields)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/network.py", line 167, in command
parse_write_concern_error=parse_write_concern_error)
File "/home/celeryProject/celeryProject/lib/python3.6/site-packages/pymongo/helpers.py", line 159, in _check_command_response
raise OperationFailure(msg % errmsg, code, response)
pymongo.errors.OperationFailure: command createIndexes requires authentication
But when I disable the security it is working!
I also tried to change the roles on the user on mongodb and nothing happened.
Also, I have tried to log in with that way:
BROKER_URL = 'mongodb://tester:123456#178.128.250.181:27017/test?authSource=admin'
celery = Celery('EOD_TASKS',broker=BROKER_URL)
Is this a security problem or I can ignore it and move on?

if i'm not mistaken, using authorization: enabled means you need to setup a user/pass to login in order to query the mongodb. by default it is left wide open, which is a security consideration

Try/except not working with twisted starttls given cert/key mismatch

So my twisted mail receiver is working nicely. Right up until we try to handle a case where the config is fubarred, and a mismatched cert/key is passed to the certificate options object for the factory.
I have a module, custom_esmtp.py, which includes an overload of ext_STARTLS(self,rest) which I have modified as follows, to include a try/except:
elif self.ctx and self.canStartTLS:
try:
self.sendCode(220, 'Begin TLS negotiation now')
self.transport.startTLS(self.ctx)
self.startedTLS = True
except:
log.err()
self.sendCode(550, "Internal server error")
return
When I run the code, having passed a cert and key that do not match, I get the following call stack:
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
self.lineReceived(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 568, in lineReceived
return getattr(self, 'state_' + self.mode)(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 582, in state_COMMAND
method('')
--- <exception caught here> ---
File "custom_esmtp.py", line 286, in ext_STARTTLS
self.transport.startTLS(self.ctx)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 179, in startTLS
startTLS(self, ctx, normal, FileDescriptor)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 139, in startTLS
tlsFactory = TLSMemoryBIOFactory(contextFactory, client, None)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 769, in __init__
contextFactory = _ContextFactoryToConnectionFactory(contextFactory)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 648, in __init__
oldStyleContextFactory.getContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1429, in getContext
self._context = self._makeContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1439, in _makeContext
ctx.use_privatekey(self.privateKey)
OpenSSL.SSL.Error: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
Line 286 of custom_esmtp.py is the self.transport.startTLS(self.ctx). I've looked through all the twisted modules listed in the stack, at the quoted lines, and there are no other try/except blocks.... So my understanding is that the error should be passed back up the stack, unhandled, until it reaches my handler in custom_esmtp.py? So why is it not getting handled - especially since the only except I have is a "catch all"?
Thanks in advance!

If you want this error to be caught, you can do:
from OpenSSL import SSL
# ...
try:
# ...
except SSL.Error:
# ...
Perhaps the syntax changes a bit. I can't check because I don't use this precise package, but the idea is that you have to declare the import path of the exceptions you want to catch.

Asterisk + Starpy : Failed to originate call

I'm using Starpy to automate Asterisk. Everything works except sometimes I get one call out of 150 where Asterisk fails to originate call.
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starpy/manager.py", line 154, in lineReceived
self.dispatchIncoming() # does dispatch and clears cache
File "/usr/local/lib/python2.7/dist-packages/starpy/manager.py", line 242, in dispatchIncoming
callback(message)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 362, in callback
self._startRunCallbacks(result)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/starpy/manager.py", line 348, in errorUnlessResponse
raise error.AMICommandFailure(message)
starpy.error.AMICommandFailure: {'message': 'Originate failed', 'response': 'Error', 'actionid': '53345672-2'}
Based on my research, this error occurs when the callee does not answer the call. However, I'm not convinced and looking for more details about why "origiate failed". I thought about monitoring the channel before we start dialing, but it doesn't work. Channel monitor needs to be called after the callee answers the call.
Please let me know if you have any suggestions.
Thanks.

Use tcpdump or other similar tool to see what exactly asterisk sends
After that fix that library, very likly it just have bug.