We Keep Coding

pymongo: What to do if monitor thread is dead?

For some reason ReplicaSet's Monitor appeared dead when refresh was scheduled.
I've got following Traceback in a call to find_one():
File "pymongo/collection.py", line 604, in find_one
for result in self.find(spec_or_id, *args, **kwargs).limit(-1):
File "pymongo/cursor.py", line 904, in next
if len(self.__data) or self._refresh():
File "pymongo/cursor.py", line 848, in _refresh
self.__uuid_subtype))
File "pymongo/cursor.py", line 805, in __send_message
client.disconnect()
File "pymongo/mongo_replica_set_client.py", line 1255, in disconnect
self.__schedule_refresh()
File "pymongo/mongo_replica_set_client.py", line 1067, in __schedule_refresh
self.__monitor.schedule_refresh()
File "pymongo/mongo_replica_set_client.py", line 295, in schedule_refresh
"Monitor thread is dead: Perhaps started before a fork?")
I studied code a bit and found out that Monitor.monitor() contains:
# RSC has been collected or there
# was an unexpected error.
except:
break
Which means whatever bad happened, I will never find out what was it.
So, what should I do, if I catch InvalidOperation("Monitor thread is dead: Perhaps started before a fork?")
Is there some nice way to restart Monitor instance?
(I use flask-pymongo with pymongo.version 2.6.2)

I'm not sure the exact cause of your problem, but it sounds like a bug I fixed in PyMongo 2.7, issue PYTHON-549, "recreate monitors". Please upgrade.

Getting "pika.exceptions.ConnectionClosed" error while using rabbitmq in python

I am using "hello world" tutorial in :http://www.rabbitmq.com/tutorials/tutorial-two-python.html .
worker.py looks like this
import pika
import time
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='task_queue', durable=True)
print ' [*] Waiting for messages. To exit press CTRL+C'
def callback(ch, method, properties, body):
print " [x] Received %r" % (body,)
time.sleep( body.count('.') )
print " [x] Done"
ch.basic_ack(delivery_tag = method.delivery_tag)
channel.basic_qos(prefetch_count=1)
channel.basic_consume(callback,
queue='task_queue')
channel.start_consuming()
I have used this code to implement in my work. Everything works smoothly untill there comes a point in a queue for which it raises an exception after printing [x] Done
Traceback (most recent call last):
File "hullworker2.py", line 242, in <module>
channel.basic_consume(callback,queue='test_queue2')
File "/usr/local/lib/python2.7/dist-packages/pika/channel.py", line 211, in basic_consume
{'consumer_tag': consumer_tag})])
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 904, in _rpc
self.connection.process_data_events()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 88, in process_data_events
if self._handle_read():
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 184, in _handle_read
super(BlockingConnection, self)._handle_read()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 300, in _handle_read
return self._handle_error(error)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/base_connection.py", line 264, in _handle_error
self._handle_disconnect()
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 181, in _handle_disconnect
self._on_connection_closed(None, True)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 232, in _on_connection_closed
self._channels[channel]._on_close(method_frame)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 817, in _on_close
self._send_method(spec.Channel.CloseOk(), None, False)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 920, in _send_method
self.connection.send_method(self.channel_number, method_frame, content)
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 120, in send_method
self._send_method(channel_number, method_frame, content)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1331, in _send_method
self._send_frame(frame.Method(channel_number, method_frame))
File "/usr/local/lib/python2.7/dist-packages/pika/adapters/blocking_connection.py", line 245, in _send_frame
super(BlockingConnection, self)._send_frame(frame_value)
File "/usr/local/lib/python2.7/dist-packages/pika/connection.py", line 1312, in _send_frame
raise exceptions.ConnectionClosed
pika.exceptions.ConnectionClosed
I don't understand how the connection is closing automatically in between the process. Process runs fine for 100's of messages in the queue then suddenly this error comes up.
Any help appreciated.

There is a concept of heartbeats. It's basically a way how the server can make sure that the client is still connected.
when you do
time.sleep( body.count('.') )
You blocking the code by N number of seconds. It means that if server would like to send a heartbeat frame to check if your client is still alive, then it will not get a response back, because your code is blocked and doesn't know if heartbeat arrived.
Instead of using time.sleep() you should use connection.sleep() this will also make the code "sleep" for N number of seconds, but it will also communicate with the server and will respond back.
sleep(duration)[source]
A safer way to sleep than calling time.sleep() directly which will keep the adapter from ignoring frames sent from RabbitMQ. The connection will “sleep” or block the number of seconds specified in duration in small intervals.

Bandwidth throttling using Twisted

I'm trying to set speed limits on downloading/uploading files and found that twisted provides twisted.protocols.policies.ThrottlingFactory to handle this job, but I can't get it right. I set readLimit and writeLimit, but file is still downloading on a maximum speed. What am I doing wrong?
from twisted.protocols.basic import FileSender
from twisted.protocols.policies import ThrottlingFactory
from twisted.web import server, resource
from twisted.internet import reactor
import os
class DownloadPage(resource.Resource):
isLeaf = True
def __init__(self, producer):
self.producer = producer
def render(self, request):
size = os.stat(somefile).st_size
request.setHeader('Content-Type', 'application/octet-stream')
request.setHeader('Content-Length', size)
request.setHeader('Content-Disposition', 'attachment; filename="' + somefile + '"')
request.setHeader('Accept-Ranges', 'bytes')
fp = open(somefile, 'rb')
d = self.producer.beginFileTransfer(fp, request)
def err(error):
print "error %s", error
def cbFinished(ignored):
fp.close()
request.finish()
d.addErrback(err).addCallback(cbFinished)
return server.NOT_DONE_YET
producer = FileSender()
root_resource = resource.Resource()
root_resource.putChild('download', DownloadPage(producer))
site = server.Site(root_resource)
tsite = ThrottlingFactory(site, readLimit=10000, writeLimit=10000)
tsite.protocol.producer = producer
reactor.listenTCP(8080, tsite)
reactor.run()
UPDATE
So sometime after I run it:
2012-10-25 09:17:03+0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 402, in startReactor
self.config, oldstdout, oldstderr, self.profiler, reactor)
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 323, in runReactorWithLogging
reactor.run()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 334, in unthrottleWrites
p.unthrottleWrites()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 225, in unthrottleWrites
self.producer.resumeProducing()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 919, in resumeProducing
self.consumer.unregisterProducer()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/web/http.py", line 811, in unregisterProducer
self.transport.unregisterProducer()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 209, in unregisterProducer
del self.producer
exceptions.AttributeError: ThrottlingProtocol instance has no attribute 'producer'
I see that I'm not supposed to assign producer like I do know tsite.protocol.producer = producer, I'm new to Twisted and I don't know how to do that another way.

Every producer needs (eventually) to be registered with whatever you want to consume the data. I don't see registration happening anywhere here. Maybe that is the issue you are having?
Twisted has been used on some big-time projects like Friendster, but all the callbacks do not sit well with the usual way I write in python (and I have some experience with functional programming). I switched to gevent.
If you are working with gevent libraries, many of the details (callbacks/generators that provide the asynchronous functionality) are abstracted out, so that you can typically get away with just monkey patching your code and writing it in the usual object-oriented style you are used to. If you are working on a project with anyone unfamiliar with a callback-heavy language like js/lisp, I bet they will appreciate gevent over twisted.

As egbutter said, you have to register a producer. So instead of this:
tsite.protocol.producer = producer
you have to call a registerProducer method explicitly:
tsite.protocol.registerProducer( ... )
or, if you're using FileSender as a producer, call its beginFileTransfer method, in our case:
file_to_send = open( ... )
producer.beginFileTransfer(file_to_send, tsite.protocol)

Stopping Twisted from swallowing exceptions

Is there a way to stop Twisted reactor from automatically swallowing exceptions (eg. NameError)? I just want it to stop execution, and give me a stack trace in console?
There's even a FAQ question about it, but to say the least, it's not very helpful.
Currently, in every errback I do this:
def errback(value):
import traceback
trace = traceback.format_exc()
# rest of the errback...
but that feels clunky, and there has to be a better way?
Update
In response to Jean-Paul's answer, I've tried running the following code (with Twisted 11.1 and 12.0):
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.internet import protocol, reactor
class Broken(protocol.Protocol):
def connectionMade(self):
buggy_user_code()
e = TCP4ClientEndpoint(reactor, "127.0.0.1", 22)
f = protocol.Factory()
f.protocol = Broken
e.connect(f)
reactor.run()
After running it, it just hangs there, so I have to Ctrl-C it:
> python2.7 tx-example.py
^CUnhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.

Let's explore "swallow" a little bit. What does it mean to "swallow" an exception?
Here's the most direct and, I think, faithful interpretation:
try:
user_code()
except:
pass
Here any exceptions from the call to user code are caught and then discarded with no action taken. If you look through Twisted, I don't think you'll find this pattern anywhere. If you do, it's a terrible mistake and a bug, and you would be helping out the project by filing a bug pointing it out.
What else might lead to "swallowing exceptions"? One possibility is that the exception is coming from application code that isn't supposed to be raising exceptions at all. This is typically dealt with in Twisted by logging the exception and then moving on, perhaps after disconnecting the application code from whatever event source it was connected to. Consider this buggy application:
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.internet import protocol, reactor
class Broken(protocol.Protocol):
def connectionMade(self):
buggy_user_code()
e = TCP4ClientEndpoint(reactor, "127.0.0.1", 22)
f = protocol.Factory()
f.protocol = Broken
e.connect(f)
reactor.run()
When run (if you have a server running on localhost:22, so the connection succeeds and connectionMade actually gets called), the output produced is:
Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
why = getattr(selectable, method)()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 674, in doConnect
self._connectDone()
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 681, in _connectDone
self.protocol.makeConnection(self)
File "/usr/lib/python2.7/dist-packages/twisted/internet/protocol.py", line 461, in makeConnection
self.connectionMade()
File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 64, in connectionMade
self._wrappedProtocol.makeConnection(self.transport)
File "/usr/lib/python2.7/dist-packages/twisted/internet/protocol.py", line 461, in makeConnection
self.connectionMade()
File "proderr.py", line 6, in connectionMade
buggy_user_code()
exceptions.NameError: global name 'buggy_user_code' is not defined
This error clearly isn't swallowed. Even though the logging system hasn't been initialized in any particular way by this application, the logged error still shows up. If the logging system had been initialized in a way that caused errors to go elsewhere - say some log file, or /dev/null - then the error might not be as apparent. You would have to go out of your way to cause this to happen though, and presumably if you direct your logging system at /dev/null then you won't be surprised if you don't see any errors logged.
In general there is no way to change this behavior in Twisted. Each exception handler is implemented separately, at the call site where application code is invoked, and each one is implemented separately to do the same thing - log the error.
One more case worth inspecting is how exceptions interact with the Deferred class. Since you mentioned errbacks I'm guessing this is the case that's biting you.
A Deferred can have a success result or a failure result. When it has any result at all and more callbacks or errbacks, it will try to pass the result to either the next callback or errback. The result of the Deferred then becomes the result of the call to one of those functions. As soon as the Deferred has gone though all of its callbacks and errbacks, it holds on to its result in case more callbacks or errbacks are added to it.
If the Deferred ends up with a failure result and no more errbacks, then it just sits on that failure. If it gets garbage collected before an errback which handles that failure is added to it, then it will log the exception. This is why you should always have errbacks on your Deferreds, at least so that you can log unexpected exceptions in a timely manner (rather than being subject to the whims of the garbage collector).
If we revisit the previous example and consider the behavior when there is no server listening on localhost:22 (or change the example to connect to a different address, where no server is listening), then what we get is exactly a Deferred with a failure result and no errback to handle it.
e.connect(f)
This call returns a Deferred, but the calling code just discards it. Hence, it has no callbacks or errbacks. When it gets its failure result, there's no code to handle it. The error is only logged when the Deferred is garbage collected, which happens at an unpredictable time. Often, particularly for very simple examples, the garbage collection won't happen until you try to shut down the program (eg via Control-C). The result is something like this:
$ python someprog.py
... wait ...
... wait ...
... wait ...
<Control C>
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
If you've accidentally written a large program and fallen into this trap somewhere, but you're not exactly sure where, then twisted.internet.defer.setDebugging might be helpful. If the example is changed to use it to enable Deferred debugging:
from twisted.internet.defer import setDebugging
setDebugging(True)
Then the output is somewhat more informative:
exarkun#top:/tmp$ python proderr.py
... wait ...
... wait ...
... wait ...
<Control C>
Unhandled error in Deferred:
(debug: C: Deferred was created:
C: File "proderr.py", line 15, in <module>
C: e.connect(f)
C: File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 240, in connect
C: wf = _WrappingFactory(protocolFactory, _canceller)
C: File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 121, in __init__
C: self._onConnection = defer.Deferred(canceller=canceller)
I: First Invoker was:
I: File "proderr.py", line 16, in <module>
I: reactor.run()
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1162, in run
I: self.mainLoop()
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1174, in mainLoop
I: self.doIteration(t)
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 140, in doSelect
I: _logrun(selectable, _drdw, selectable, method, dict)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
I: return callWithContext({"system": lp}, func, *args, **kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
I: return context.call({ILogContext: newCtx}, func, *args, **kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
I: return self.currentContext().callWithContext(ctx, func, *args, **kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
I: return func(*args,**kw)
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
I: why = getattr(selectable, method)()
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 638, in doConnect
I: self.failIfNotConnected(error.getConnectError((err, strerror(err))))
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 592, in failIfNotConnected
I: self.connector.connectionFailed(failure.Failure(err))
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1048, in connectionFailed
I: self.factory.clientConnectionFailed(self, reason)
I: File "/usr/lib/python2.7/dist-packages/twisted/internet/endpoints.py", line 144, in clientConnectionFailed
I: self._onConnection.errback(reason)
)
Unhandled Error
Traceback (most recent call last):
Failure: twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 111: Connection refused.
Notice near the top, where the e.connect(f) line is given as the origin of this Deferred - telling you a likely place where you should be adding an errback.
However, the code should have been written to add an errback to this Deferred in the first place, at least to log the error.
There are shorter (and more correct) ways to display exceptions than the one you've given, though. For example, consider:
d = e.connect(f)
def errback(reason):
reason.printTraceback()
d.addErrback(errback)
Or, even more succinctly:
from twisted.python.log import err
d = e.connect(f)
d.addErrback(err, "Problem fetching the foo from the bar")
This error handling behavior is somewhat fundamental to the idea of Deferred and so also isn't very likely to change.
If you have a Deferred, errors from which really are fatal and must stop your application, then you can define a suitable errback and attach it to that Deferred:
d = e.connect(f)
def fatalError(reason):
err(reason, "Absolutely needed the foo, could not get it")
reactor.stop()
d.addErrback(fatalError)

What you could do as a workaround is register a log listener and stop the reactor whenever you see a critical error! This is a twisted(verb) approach but luckily all "Unhandled errors" are raised with LogLevel.critical.
from twisted.logger._levels import LogLevel
def analyze(event):
if event.get("log_level") == LogLevel.critical:
print "Stopping for: ", event
reactor.stop()
globalLogPublisher.addObserver(analyze)

Twisted/tkinter program crashes on exit

I am running an app using twisted and tkinter that sends the result to the server, waits for the server to send back a confirmation, and then exits. So, the function I use to exit is this:
def term():
'''To end the program'''
reactor.stop()
root.quit()
root.destroy()
This is then set in the factory and called in the dataReceived function of the protocol. I run it, and the program runs fine and even sends the necessary data and closes, but it also gives me the following error report:
Unhandled error in Deferred:
Traceback (most recent call last):
File "D:\Python25\Lib\site-packages\twisted\internet\base.py", line 1128, in run
self.mainLoop()
File "D:\Python25\Lib\site-packages\twisted\internet\base.py", line 1137, in mainLoop
self.runUntilCurrent()
File "D:\Python25\Lib\site-packages\twisted\internet\base.py", line 757, in runUntilCurrent
call.func(*call.args, **call.kw)
File "D:\Python25\Lib\site-packages\twisted\internet\task.py", line 114, in __call__
d = defer.maybeDeferred(self.f, *self.a, **self.kw)
--- <exception caught here> ---
File "D:\Python25\Lib\site-packages\twisted\internet\defer.py", line 106, in maybeDeferred
result = f(*args, **kw)
File "D:\Python25\lib\lib-tk\Tkinter.py", line 917, in update
self.tk.call('update')
_tkinter.TclError: can't invoke "update" command: application has been destroyed
Does anyone know why?

You only need to call reactor.stop to exit: the root.quit() and root.destroy() calls are superfluous. Consider this short example which runs Twisted and Tk for three seconds and then exits:
import Tkinter
from twisted.internet import tksupport
root = Tkinter.Tk()
tksupport.install(root)
from twisted.internet import reactor
reactor.callLater(3, reactor.stop)
reactor.run()