I am trying to to handle errors which may be thrown by psycopg2, so I imported the errors the pipeline was facing and got a weird error when I ran the spider.
pipelines.py
from psycopg2.errors import InvalidTextRepresentation
from psycopg2.errors import InFailedSqlTransaction
...
try:
self.cursor.execute(
"""INSERT INTO links (url,price) VALUES (%s, %s)""",
(
item["productURL"],
item["price"]
))
except InvalidTextRepresentation, InFailedSqlTransaction:
pass
I got this error directly and the spider didn't run
Unhandled error in Deferred:
Temporarily disabling observer LegacyLogObserverWrapper(<bound method PythonLoggingObserver.emit of <twisted.python.log.PythonLoggingObserver object at 0x7ff963789e80>>) due to exception: [Failure instance: Traceback: <class 'TypeError'>: _findCaller() takes from 1 to 2 positional arguments but 3 were given
/usr/lib/python3/dist-packages/twisted/internet/defer.py:953:__del__
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:270:critical
/usr/lib/python3/dist-packages/twisted/logger/_logger.py:144:emit
--- <exception caught here> ---
/usr/lib/python3/dist-packages/twisted/logger/_observer.py:131:__call__
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:93:__call__
/usr/lib/python3/dist-packages/twisted/python/log.py:595:emit
/usr/lib/python3/dist-packages/twisted/logger/_legacy.py:154:publishToNewObserver
/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py:115:__call__
/usr/lib/python3.8/logging/__init__.py:1500:log
/usr/lib/python3.8/logging/__init__.py:1565:_log
]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 953, in __del__
log.critical("Unhandled error in Deferred:",
File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 270, in critical
self.emit(LogLevel.critical, format, **kwargs)
File "/usr/lib/python3/dist-packages/twisted/logger/_logger.py", line 144, in emit
self.observer(event)
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/logger/_observer.py", line 131, in __call__
observer(event)
File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 93, in __call__
self.legacyObserver(event)
File "/usr/lib/python3/dist-packages/twisted/python/log.py", line 595, in emit
_publishNew(self._newObserver, eventDict, textFromEventDict)
File "/usr/lib/python3/dist-packages/twisted/logger/_legacy.py", line 154, in publishToNewObserver
observer(eventDict)
File "/usr/lib/python3/dist-packages/twisted/logger/_stdlib.py", line 115, in __call__
self.logger.log(
File "/usr/lib/python3.8/logging/__init__.py", line 1500, in log
self._log(level, msg, args, **kwargs)
File "/usr/lib/python3.8/logging/__init__.py", line 1565, in _log
fn, lno, func, sinfo = self.findCaller(stack_info, stacklevel)
builtins.TypeError: _findCaller() takes from 1 to 2 positional arguments but 3 were given
I changed back to except: instead of except InvalidTextRepresentation, InFailedSqlTransaction: it worked fine, Is there something I am doing wrong or is there a better to handle the errors of psycopg2
Related
So my twisted mail receiver is working nicely. Right up until we try to handle a case where the config is fubarred, and a mismatched cert/key is passed to the certificate options object for the factory.
I have a module, custom_esmtp.py, which includes an overload of ext_STARTLS(self,rest) which I have modified as follows, to include a try/except:
elif self.ctx and self.canStartTLS:
try:
self.sendCode(220, 'Begin TLS negotiation now')
self.transport.startTLS(self.ctx)
self.startedTLS = True
except:
log.err()
self.sendCode(550, "Internal server error")
return
When I run the code, having passed a cert and key that do not match, I get the following call stack:
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/twisted/internet/tcp.py", line 220, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 454, in dataReceived
self.lineReceived(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 568, in lineReceived
return getattr(self, 'state_' + self.mode)(line)
File "/usr/local/lib/python2.7/site-packages/twisted/mail/smtp.py", line 582, in state_COMMAND
method('')
--- <exception caught here> ---
File "custom_esmtp.py", line 286, in ext_STARTTLS
self.transport.startTLS(self.ctx)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 179, in startTLS
startTLS(self, ctx, normal, FileDescriptor)
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_newtls.py", line 139, in startTLS
tlsFactory = TLSMemoryBIOFactory(contextFactory, client, None)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 769, in __init__
contextFactory = _ContextFactoryToConnectionFactory(contextFactory)
File "/usr/local/lib/python2.7/site-packages/twisted/protocols/tls.py", line 648, in __init__
oldStyleContextFactory.getContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1429, in getContext
self._context = self._makeContext()
File "/usr/local/lib/python2.7/site-packages/twisted/internet/_sslverify.py", line 1439, in _makeContext
ctx.use_privatekey(self.privateKey)
OpenSSL.SSL.Error: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
Line 286 of custom_esmtp.py is the self.transport.startTLS(self.ctx). I've looked through all the twisted modules listed in the stack, at the quoted lines, and there are no other try/except blocks.... So my understanding is that the error should be passed back up the stack, unhandled, until it reaches my handler in custom_esmtp.py? So why is it not getting handled - especially since the only except I have is a "catch all"?
Thanks in advance!
If you want this error to be caught, you can do:
from OpenSSL import SSL
# ...
try:
# ...
except SSL.Error:
# ...
Perhaps the syntax changes a bit. I can't check because I don't use this precise package, but the idea is that you have to declare the import path of the exceptions you want to catch.
I have a Twisted ServerFactory, which I started with listenTCP. How do I start a task that is a function of that factory?
I tried:
if __name__ == "__main__":
factory = MyFactory()
reactor.listenTCP(555558, factory)
reactor.connectTCP("127.0.0.1", 55555, MyConnector(factory))
sanitizing = task.LoopingCall(factory.sanitize())
sanitizing.start(3, False)
reactor.run()
But that throws an error:
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run self.mainLoop()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
self.runUntilCurrent()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/usr/lib/python2.7/dist-packages/twisted/internet/task.py", line 218, in __call__
d = defer.maybeDeferred(self.f, *self.a, **self.kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 139, in maybeDeferred
result = f(*args, **kw)
exceptions.TypeError: 'NoneType' object is not callable
The factory is actually listening, so I don't understand why it is NoneType.
In Python, name() is the syntax to call an object (usually a function or method). By extension, name(another_name()) is how you call an object (referenced by the name another_name) and then pass the return value of that call to another object (referenced by the name name).
To apply this to your example, LoopingCall(factory.sanitize()) is how you call factory.sanitize and pass the return value to LoopingCall.
This is most likely not what you meant. Instead, you probably meant to pass factory.sanitize as an argument to LoopingCall. This is what you would need to do if you wanted LoopingCall to call factory.sanitize periodically, anyway. I'm just guessing this is what you want - you didn't actually explicitly say what you want in your question. :)
I'm using Starpy to automate Asterisk. Everything works except sometimes I get one call out of 150 where Asterisk fails to originate call.
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/starpy/manager.py", line 154, in lineReceived
self.dispatchIncoming() # does dispatch and clears cache
File "/usr/local/lib/python2.7/dist-packages/starpy/manager.py", line 242, in dispatchIncoming
callback(message)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 362, in callback
self._startRunCallbacks(result)
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 458, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 545, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/starpy/manager.py", line 348, in errorUnlessResponse
raise error.AMICommandFailure(message)
starpy.error.AMICommandFailure: {'message': 'Originate failed', 'response': 'Error', 'actionid': '53345672-2'}
Based on my research, this error occurs when the callee does not answer the call. However, I'm not convinced and looking for more details about why "origiate failed". I thought about monitoring the channel before we start dialing, but it doesn't work. Channel monitor needs to be called after the callee answers the call.
Please let me know if you have any suggestions.
Thanks.
Use tcpdump or other similar tool to see what exactly asterisk sends
After that fix that library, very likly it just have bug.
I get an exception on both client and server side when I run the first example at http://buildbot.twistedmatrix.com/builds/sphinx-html/291-15849/projects/web/howto/xmlrpc.html. The server code I used is below:
from twisted.web import xmlrpc, server
class Example(xmlrpc.XMLRPC):
"""An example object to be published."""
def xmlrpc_echo(self, x):
"""
Return all passed args.
"""
return x
def xmlrpc_add(self, a, b):
"""
Return sum of arguments.
"""
return a + b
def xmlrpc_fault(self):
"""
Raise a Fault indicating that the procedure should not be used.
"""
raise xmlrpc.Fault(123, "The fault procedure is faulty.")
if __name__ == '__main__':
from twisted.internet import reactor
r = Example()
reactor.listenTCP(7080, server.Site(r))
reactor.run()
Client side is below:
import xmlrpclib
s = xmlrpclib.Server('http://localhost:7080/')
print s.echo('Hello world')
The server side exception is:
Traceback (most recent call last):
File "/usr/lib/python2.6/dist-packages/twisted/web/xmlrpc.py", line 150, in render_POST
d.addCallback(self._cbRender, request, responseFailed)
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 260, in addCallback
callbackKeywords=kw)
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 249, in addCallbacks
self._runCallbacks()
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, *args, **kw)
--- <exception caught here> ---
File "/usr/lib/python2.6/dist-packages/twisted/web/xmlrpc.py", line 170, in _cbRender
allow_none=self.allowNone)
exceptions.TypeError: dumps() got an unexpected keyword argument 'allow_none'
Client side exception is:
Traceback (most recent call last):
File "./client.py", line 6, in <module>
print s.echo('Hello world')
File "/usr/local/lib/python2.6/dist-packages/xmlrpclib-1.0.1-py2.6.egg/xmlrpclib.py", line 986, in __call__
return self.__send(self.__name, args)
File "/usr/local/lib/python2.6/dist-packages/xmlrpclib-1.0.1-py2.6.egg/xmlrpclib.py", line 1239, in __request
verbose=self.__verbose
File "/usr/local/lib/python2.6/dist-packages/xmlrpclib-1.0.1-py2.6.egg/xmlrpclib.py", line 1037, in request
return self._parse_response(h.getfile(), sock)
File "/usr/local/lib/python2.6/dist-packages/xmlrpclib-1.0.1-py2.6.egg/xmlrpclib.py", line 1136, in _parse_response
p.close()
File "/usr/local/lib/python2.6/dist-packages/xmlrpclib-1.0.1-py2.6.egg/xmlrpclib.py", line 508, in close
self._parser.Parse("", 1) # end of data
xml.parsers.expat.ExpatError: no element found: line 1, column 0
Looks like you have an old version of xmlrpclib?
What version of python are you using?
Where is the xmlrpclib coming from that your xmlrpc server is using, and what version is it?
$ python -v
>>> import xmlrpclib
# /usr/lib/python2.6/xmlrpclib.pyc matches /usr/lib/python2.6/xmlrpclib.py
>>> xmlrpclib.__version__
'1.0.1'
>>> xmlrpclib.dumps((None,), allow_none=True)
'<params>\n<param>\n<value><nil/></value></param>\n</params>\n
i.e. this works for me. Perhaps you are somehow using an old version of xmlrpclib?
I'm using this:
from twisted.web.client import getPage
df = getPage(url) # there is some url
I'm getting the following error. Please can anyone guide me on this
ERROR:twsited:Unhandled error in Deferred:
ERROR:twsited:Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.6/dist-packages/starpy/manager.py", line 123, in lineReceived
self.dispatchIncoming() # does dispatch and clears cache
File "/usr/local/lib/python2.6/dist-packages/starpy/manager.py", line 200, in dispatchIncoming
callback( message )
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 243, in callback
self._startRunCallbacks(result)
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 312, in _startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 328, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/usr/local/lib/python2.6/dist-packages/starpy/manager.py", line 298, in errorUnlessResponse
raise error.AMICommandFailure( message )
starpy.error.AMICommandFailure: {'message': 'Channel not specified', 'response': 'Error', 'actionid': 'askme-158811948-5'}
I'm not sure this error is due to getPage() method because even when i've commented this it still give me the same error. Can anyone help. I can't figure out the reason for the error and where it is generated
The code posted is not complete. The error is not due to getPage.
From the stack trace clues, this uses AMIProtocol (line receiver) .
I guess some where you have to specify your protocol channel in AMIProtocol
setVar(self, channel, variable, value) in star.py.
This is not a twisted issue.