I am trying to make some API calls to firebase using the google.cloud.firestore.AsyncClient module. Fire2 is just a wrapper for this object.
The traceback suggests its a problem with asyncio, but if I make a dummy await it actually runs just fine. What is the problem and why does it occur?
Sample code:
import Fire2
class Test:
def __init__(self):
doc = asyncio.run(self.wait())
async def wait(self):
doc = await Fire2("test1").get(g1) # gives the error
# doc = await asyncio.sleep(1) # runs without error
return doc
def test(self):
x = Test2(p1)
class Test2:
def __init__(self, p):
doc = asyncio.run(self.run(p))
print(doc.to_dict())
async def run(self, p):
doc = await Fire2('test2').get(p)
return doc
p1 = 'foo'
g1 = 'bar'
h = Test()
h.test()
Traceback:
Traceback (most recent call last):
File "<project path>\scratch.py", line 137, in <module>
h.test()
File "<project path>\scratch.py", line 123, in test
x = Test2(p1)
File "<project path>\scratch.py", line 127, in __init__
doc = asyncio.run(self.run(p))
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\runners.py", line 44, in run
return loop.run_until_complete(main)
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
return future.result()
File "<project path>\scratch.py", line 131, in run
doc = await Fire2('test2').get(p)
File "<project path>\<Fire2 file>", line 422, in get
res = await self._collection.document(doc_id).get()
File "<project path>\venv\lib\site-packages\google\cloud\firestore_v1\async_document.py", line 364, in get
response_iter = await self._client._firestore_api.batch_get_documents(
File "<project path>\venv\lib\site-packages\google\api_core\grpc_helpers_async.py", line 171, in error_remapped_callable
call = callable_(*args, **kwargs)
File "<project path>\venv\lib\site-packages\grpc\aio\_channel.py", line 165, in __call__
call = UnaryStreamCall(request, deadline, metadata, credentials,
File "<project path>\venv\lib\site-packages\grpc\aio\_call.py", line 553, in __init__
self._send_unary_request_task = loop.create_task(
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 431, in create_task
self._check_closed()
File "<user AppData>\Local\Programs\Python\Python39\lib\asyncio\base_events.py", line 510, in _check_closed
raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed
sys:1: RuntimeWarning: coroutine 'UnaryStreamCall._send_unary_request' was never awaited
Process finished with exit code 1
You are making two separate calls to asyncio.run. We can see from the stack trace that the error is coming from inside the grpc library. I suspect that the same grpc connection is being used both times, and that during its initialization it integrates itself with the async event loop in order to maintain the connection in the background between calls.
The problem is that every time you call asyncio.run, you will create a new event loop, and every time it returns that event loop will be closed. So by the time you call this function again, and the grpc layer tries to do its work, it realizes that the original event loop has been closed and it returns this error.
To fix this, You can have a single call to asyncio.run, typically calling a "main" function, or similar. This would then make calls to async functions which perform the actual work.
If the problem persists then there might be some issues in the “Fire2”class.
For more reference you can check this link which outlines high-level asyncio APIs to work with coroutines and Tasks with the help of various examples.
Use pytest with using fixture with scope="session" in conftest.py
#pytest.fixture(scope="session")
def event_loop():
loop = get_event_loop()
yield loop
loop.close()
I want to run a method I know this method doesn't work and I want to get the error returned by the method.
This is my code :
def is_connect(s):
print("ok connection")
print(s)
ioloop.stop()
try:
current_job_ready = 0
print("ok1")
beanstalk = beanstalkt.Client(host='host', port=port)
print("ok1")
beanstalk.connect(callback=is_connect)
ioloop = tornado.ioloop.IOLoop.instance()
ioloop.start()
print("ok2")
except IOError as e:
print(e)
And this is the error I have when I run my program with wring port :
WARNING:tornado.general:Connect error on fd 7: ECONNREFUSED
ERROR:tornado.application:Exception in callback <functools.partial object at 0x7f5a0eac6f18>
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 604, in _run_callback
ret = callback()
File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
return fn(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 619, in <lambda>
self.add_future(ret, lambda f: f.result())
File "/usr/local/lib/python2.7/dist-packages/tornado/concurrent.py", line 237, in result
raise_exc_info(self._exc_info)
File "/usr/local/lib/python2.7/dist-packages/tornado/gen.py", line 270, in wrapper
result = func(*args, **kwargs)
TypeError: connect() takes exactly 1 argument (2 given)
I want to have e when I enter a false port or host.
How can I do this?
I tired to add raise IOError("connection error") after beanstalk = beanstalkt.Client(host='host', port=port) But this force the error, and I just want to have error when it exist.
Here's where reading the code helps. In beanstalkt 0.6's connect, it creates an IOStream to connect to the server:
https://github.com/nephics/beanstalkt/blob/v0.6.0/beanstalkt/beanstalkt.py#L108
It registers your callback to be executed on success, but if the connection fails it'll just call Client._reconnect once per second forever. I think you should open a feature request in their GitHub project asking for an error-notification system for connect. With the current beanstalkt implementation, you just have to decide how long you're willing to wait for success:
import sys
from datetime import timedelta
from tornado.ioloop import IOLoop
def is_connect(s):
print("ok connection")
print(s)
loop.remove_timeout(timeout)
# Do something with Beanstalkd....
def connection_failed():
print(sys.stderr, "Connection failed!")
# Could call IOLoop.stop() or just quit.
sys.exit(1)
loop = IOLoop.current()
timeout = loop.add_timeout(timedelta(seconds=1), connection_failed)
beanstalk.connect(callback=is_connect)
loop.start()
I have a Twisted ServerFactory, which I started with listenTCP. How do I start a task that is a function of that factory?
I tried:
if __name__ == "__main__":
factory = MyFactory()
reactor.listenTCP(555558, factory)
reactor.connectTCP("127.0.0.1", 55555, MyConnector(factory))
sanitizing = task.LoopingCall(factory.sanitize())
sanitizing.start(3, False)
reactor.run()
But that throws an error:
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run self.mainLoop()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
self.runUntilCurrent()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/usr/lib/python2.7/dist-packages/twisted/internet/task.py", line 218, in __call__
d = defer.maybeDeferred(self.f, *self.a, **self.kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 139, in maybeDeferred
result = f(*args, **kw)
exceptions.TypeError: 'NoneType' object is not callable
The factory is actually listening, so I don't understand why it is NoneType.
In Python, name() is the syntax to call an object (usually a function or method). By extension, name(another_name()) is how you call an object (referenced by the name another_name) and then pass the return value of that call to another object (referenced by the name name).
To apply this to your example, LoopingCall(factory.sanitize()) is how you call factory.sanitize and pass the return value to LoopingCall.
This is most likely not what you meant. Instead, you probably meant to pass factory.sanitize as an argument to LoopingCall. This is what you would need to do if you wanted LoopingCall to call factory.sanitize periodically, anyway. I'm just guessing this is what you want - you didn't actually explicitly say what you want in your question. :)
Is there a way to stop Twisted reactor from automatically swallowing exceptions (eg. NameError)? I just want it to stop execution, and give me a stack trace in console?
There's even a FAQ question about it, but to say the least, it's not very helpful.
Currently, in every errback I do this:
def errback(value):
import traceback
trace = traceback.format_exc()
# rest of the errback...
but that feels clunky, and there has to be a better way?
Update
In response to Jean-Paul's answer, I've tried running the following code (with Twisted 11.1 and 12.0):
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.internet import protocol, reactor
class Broken(protocol.Protocol):
def connectionMade(self):
buggy_user_code()
e = TCP4ClientEndpoint(reactor, "127.0.0.1", 22)
f = protocol.Factory()
f.protocol = Broken
e.connect(f)
reactor.run()
After running it, it just hangs there, so I have to Ctrl-C it:
> python2.7 tx-example.py
^CUnhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
Let's explore "swallow" a little bit. What does it mean to "swallow" an exception?
Here's the most direct and, I think, faithful interpretation:
try:
user_code()
except:
pass
Here any exceptions from the call to user code are caught and then discarded with no action taken. If you look through Twisted, I don't think you'll find this pattern anywhere. If you do, it's a terrible mistake and a bug, and you would be helping out the project by filing a bug pointing it out.
What else might lead to "swallowing exceptions"? One possibility is that the exception is coming from application code that isn't supposed to be raising exceptions at all. This is typically dealt with in Twisted by logging the exception and then moving on, perhaps after disconnecting the application code from whatever event source it was connected to. Consider this buggy application:
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.internet import protocol, reactor
class Broken(protocol.Protocol):
def connectionMade(self):
buggy_user_code()
e = TCP4ClientEndpoint(reactor, "127.0.0.1", 22)
f = protocol.Factory()
f.protocol = Broken
e.connect(f)
reactor.run()
When run (if you have a server running on localhost:22, so the connection succeeds and connectionMade actually gets called), the output produced is:
Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
why = getattr(selectable, method)()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 674, in doConnect
self._connectDone()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 681, in _connectDone
self.protocol.makeConnection(self)
File "/usr/lib/python2.7/dist-packages/twisted/internet/protocol.py", line 461, in makeConnection
self.connectionMade()
File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 64, in connectionMade
self._wrappedProtocol.makeConnection(self.transport)
File "/usr/lib/python2.7/dist-packages/twisted/internet/protocol.py", line 461, in makeConnection
self.connectionMade()
File "proderr.py", line 6, in connectionMade
buggy_user_code()
exceptions.NameError: global name 'buggy_user_code' is not defined
This error clearly isn't swallowed. Even though the logging system hasn't been initialized in any particular way by this application, the logged error still shows up. If the logging system had been initialized in a way that caused errors to go elsewhere - say some log file, or /dev/null - then the error might not be as apparent. You would have to go out of your way to cause this to happen though, and presumably if you direct your logging system at /dev/null then you won't be surprised if you don't see any errors logged.
In general there is no way to change this behavior in Twisted. Each exception handler is implemented separately, at the call site where application code is invoked, and each one is implemented separately to do the same thing - log the error.
One more case worth inspecting is how exceptions interact with the Deferred class. Since you mentioned errbacks I'm guessing this is the case that's biting you.
A Deferred can have a success result or a failure result. When it has any result at all and more callbacks or errbacks, it will try to pass the result to either the next callback or errback. The result of the Deferred then becomes the result of the call to one of those functions. As soon as the Deferred has gone though all of its callbacks and errbacks, it holds on to its result in case more callbacks or errbacks are added to it.
If the Deferred ends up with a failure result and no more errbacks, then it just sits on that failure. If it gets garbage collected before an errback which handles that failure is added to it, then it will log the exception. This is why you should always have errbacks on your Deferreds, at least so that you can log unexpected exceptions in a timely manner (rather than being subject to the whims of the garbage collector).
If we revisit the previous example and consider the behavior when there is no server listening on localhost:22 (or change the example to connect to a different address, where no server is listening), then what we get is exactly a Deferred with a failure result and no errback to handle it.
e.connect(f)
This call returns a Deferred, but the calling code just discards it. Hence, it has no callbacks or errbacks. When it gets its failure result, there's no code to handle it. The error is only logged when the Deferred is garbage collected, which happens at an unpredictable time. Often, particularly for very simple examples, the garbage collection won't happen until you try to shut down the program (eg via Control-C). The result is something like this:
$ python someprog.py
... wait ...
... wait ...
... wait ...
<Control C>
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
If you've accidentally written a large program and fallen into this trap somewhere, but you're not exactly sure where, then twisted.internet.defer.setDebugging might be helpful. If the example is changed to use it to enable Deferred debugging:
from twisted.internet.defer import setDebugging
setDebugging(True)
Then the output is somewhat more informative:
exarkun#top:/tmp$ python proderr.py
... wait ...
... wait ...
... wait ...
<Control C>
Unhandled error in Deferred:
(debug: C: Deferred was created:
C: File "proderr.py", line 15, in <module>
C: e.connect(f)
C: File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 240, in connect
C: wf = _WrappingFactory(protocolFactory, _canceller)
C: File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 121, in __init__
C: self._onConnection = defer.Deferred(canceller=canceller)
I: First Invoker was:
I: File "proderr.py", line 16, in <module>
I: reactor.run()
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1162, in run
I: self.mainLoop()
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1174, in mainLoop
I: self.doIteration(t)
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 140, in doSelect
I: _logrun(selectable, _drdw, selectable, method, dict)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
I: return callWithContext({"system": lp}, func, *args, **kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
I: return context.call({ILogContext: newCtx}, func, *args, **kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
I: return self.currentContext().callWithContext(ctx, func, *args, **kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
I: return func(*args,**kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
I: why = getattr(selectable, method)()
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 638, in doConnect
I: self.failIfNotConnected(error.getConnectError((err, strerror(err))))
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 592, in failIfNotConnected
I: self.connector.connectionFailed(failure.Failure(err))
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1048, in connectionFailed
I: self.factory.clientConnectionFailed(self, reason)
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 144, in clientConnectionFailed
I: self._onConnection.errback(reason)
)
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
Notice near the top, where the e.connect(f) line is given as the origin of this Deferred - telling you a likely place where you should be adding an errback.
However, the code should have been written to add an errback to this Deferred in the first place, at least to log the error.
There are shorter (and more correct) ways to display exceptions than the one you've given, though. For example, consider:
d = e.connect(f)
def errback(reason):
reason.printTraceback()
d.addErrback(errback)
Or, even more succinctly:
from twisted.python.log import err
d = e.connect(f)
d.addErrback(err, "Problem fetching the foo from the bar")
This error handling behavior is somewhat fundamental to the idea of Deferred and so also isn't very likely to change.
If you have a Deferred, errors from which really are fatal and must stop your application, then you can define a suitable errback and attach it to that Deferred:
d = e.connect(f)
def fatalError(reason):
err(reason, "Absolutely needed the foo, could not get it")
reactor.stop()
d.addErrback(fatalError)
What you could do as a workaround is register a log listener and stop the reactor whenever you see a critical error! This is a twisted(verb) approach but luckily all "Unhandled errors" are raised with LogLevel.critical.
from twisted.logger._levels import LogLevel
def analyze(event):
if event.get("log_level") == LogLevel.critical:
print "Stopping for: ", event
reactor.stop()
globalLogPublisher.addObserver(analyze)
I am running an app using twisted and tkinter that sends the result to the server, waits for the server to send back a confirmation, and then exits. So, the function I use to exit is this:
def term():
'''To end the program'''
reactor.stop()
root.quit()
root.destroy()
This is then set in the factory and called in the dataReceived function of the protocol. I run it, and the program runs fine and even sends the necessary data and closes, but it also gives me the following error report:
Unhandled error in Deferred:
Traceback (most recent call last):
File "D:\Python25\Lib\site-packages\twisted\internet\base.py", line 1128, in run
self.mainLoop()
File "D:\Python25\Lib\site-packages\twisted\internet\base.py", line 1137, in mainLoop
self.runUntilCurrent()
File "D:\Python25\Lib\site-packages\twisted\internet\base.py", line 757, in runUntilCurrent
call.func(*call.args, **call.kw)
File "D:\Python25\Lib\site-packages\twisted\internet\task.py", line 114, in __call__
d = defer.maybeDeferred(self.f, *self.a, **self.kw)
--- <exception caught here> ---
File "D:\Python25\Lib\site-packages\twisted\internet\defer.py", line 106, in maybeDeferred
result = f(*args, **kw)
File "D:\Python25\lib\lib-tk\Tkinter.py", line 917, in update
self.tk.call('update')
_tkinter.TclError: can't invoke "update" command: application has been destroyed
Does anyone know why?
You only need to call reactor.stop to exit: the root.quit() and root.destroy() calls are superfluous. Consider this short example which runs Twisted and Tk for three seconds and then exits:
import Tkinter
from twisted.internet import tksupport
root = Tkinter.Tk()
tksupport.install(root)
from twisted.internet import reactor
reactor.callLater(3, reactor.stop)
reactor.run()