We Keep Coding

Twisted task.LoopingCall : Unhandled error in Deferred:

My Code
from twisted.internet import task, reactor
def stuff():
print('Hello, world!')
scheduler = task.LoopingCall(stuff())
scheduler.start(10)
reactor.run()
This is the Error I am getting
Hello, world! Unhandled error in Deferred:
Traceback (most recent call last): File "C:\Users\Usama
fiaz\AppData\Local\Programs\Python\Python39\lib\site-packages\twisted\internet\base.py",
line 1315, in run
self.mainLoop() File "C:\Users\Usama fiaz\AppData\Local\Programs\Python\Python39\lib\site-packages\twisted\internet\base.py",
line 1325, in mainLoop
reactorBaseSelf.runUntilCurrent() File "C:\Users\Usama fiaz\AppData\Local\Programs\Python\Python39\lib\site-packages\twisted\internet\base.py",
line 991, in runUntilCurrent
call.func(*call.args, **call.kw) File "C:\Users\Usama fiaz\AppData\Local\Programs\Python\Python39\lib\site-packages\twisted\internet\task.py",
line 251, in call
d = maybeDeferred(self.f, *self.a, **self.kw)
--- --- File "C:\Users\Usama fiaz\AppData\Local\Programs\Python\Python39\lib\site-packages\twisted\internet\defer.py",
line 190, in maybeDeferred
result = f(*args, **kwargs) builtins.TypeError: 'NoneType' object is not callable

Per the API documentation, LoopingCall.__init__ accepts a callable parameter f as its first argument.
In your example:
from twisted.internet import task, reactor
def stuff():
print('Hello, world!')
scheduler = task.LoopingCall(stuff())
scheduler.start(10)
reactor.run()
the value being passed for f is the result of evaluating stuff() - in other words, it is the return value of the stuff function. stuff implicitly returns None so your LoopingCall construction is equivalent to:
scheduler = task.LoopingCall(stuff())
Then, when you start the loop with LoopingCall.start you don't attach any error handlers. When LoopingCall tries to call None an exception is raised. Since there are no error handlers attached, the exception is reported as an "Unhandled error in Deferred" and logged for you.
If you want the LoopingCall to call stuff at the defined interval, pass stuff to it (instead of None). If you want to deal with errors in your loop, attach an error handler to the Deferred returned by LoopingCall.start.

Django Channels Redis: Exception inside application: Lock is not acquired

Fully loaded multi-tenant Django application with 1000's of WebSockets using Daphne/Channels, running fine for a few months and suddenly tenants all calling it the support line the application running slow or outright hanging. Narrowed it down to WebSockets as HTTP REST API hits came through fast and error free.
None of the application logs or OS logs indicate some issue, so only thing to go on is the exception noted below. It happened over and over again here and there throughout 2 days.
I don't expect any deep debugging help, just some off-the-cuff advice on possibilities.
AWS Linux 1
Python 3.6.4
Elasticache Redis 5.0
channels==2.4.0
channels-redis==2.4.2
daphne==2.5.0
Django==2.2.13
Split configuration HTTP served by uwsgi, daphne serves asgi, Nginx
May 10 08:08:16 prod-b-web1: [pid 15053] [version 119.5.10.5086] [tenant_id -] [domain_name -] [pathname /opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/daphne/server.py] [lineno 288] [priority ERROR] [funcname application_checker] [request_path -] [request_method -] [request_data -] [request_user -] [request_stack -] Exception inside application: Lock is not acquired.
Traceback (most recent call last):
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 435, in receive
real_channel
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 484, in receive_single
await self.receive_clean_locks.acquire(channel_key)
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 152, in acquire
return await self.locks[channel].acquire()
File "/opt/python3.6/lib/python3.6/asyncio/locks.py", line 176, in acquire
yield from fut
concurrent.futures._base.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/sessions.py", line 183, in __call__
return await self.inner(receive, self.send)
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/middleware.py", line 41, in coroutine_call
await inner_instance(receive, send)
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/consumer.py", line 59, in __call__
[receive, self.channel_receive], self.dispatch
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels/utils.py", line 58, in await_many_dispatch
await task
File "/opt/releases/r119.5.10.5086/env/lib/python3.6/site-packages/channels_redis/core.py", line 447, in receive
self.receive_lock.release()
File "/opt/python3.6/lib/python3.6/asyncio/locks.py", line 201, in release
raise RuntimeError('Lock is not acquired.')
RuntimeError: Lock is not acquired.

First, lets have a look at the source of the RuntimeError: Lock is not acquired. error. As given by the traceback, the release() method in the file /opt/python3.6/lib/python3.6/asyncio/locks.py is defined like so:
def release(self):
"""Release a lock.
When the lock is locked, reset it to unlocked, and return.
If any other coroutines are blocked waiting for the lock to become
unlocked, allow exactly one of them to proceed.
When invoked on an unlocked lock, a RuntimeError is raised.
There is no return value.
"""
if self._locked:
self._locked = False
self._wake_up_first()
else:
raise RuntimeError('Lock is not acquired.')
A primitive lock is a synchronization primitive that is not owned by a particular thread when locked.
When attempting to release an unlocked lock by calling the release() method, the RuntimeError will be raised, as the method should only be called in the locked state. The state changes to unlocked when called in the locked state.
Now for the previous error raised in the acquire() method in the same file, the acquire() method is defined like so:
async def acquire(self):
"""Acquire a lock.
This method blocks until the lock is unlocked, then sets it to
locked and returns True.
"""
if (not self._locked and (self._waiters is None or
all(w.cancelled() for w in self._waiters))):
self._locked = True
return True
if self._waiters is None:
self._waiters = collections.deque()
fut = self._loop.create_future()
self._waiters.append(fut)
# Finally block should be called before the CancelledError
# handling as we don't want CancelledError to call
# _wake_up_first() and attempt to wake up itself.
try:
try:
await fut
finally:
self._waiters.remove(fut)
except exceptions.CancelledError:
if not self._locked:
self._wake_up_first()
raise
self._locked = True
return True
So in order for the concurrent.futures._base.CancelledError error you're getting to be raised, the await fut must've caused the issue.
To fix it, you can have a look at Awaiting an asyncio.Future raises concurrent.futures._base.CancelledError instead of waiting for a value/exception to be set
Basically, you might have an awaitable in your code that you didn't await, and by not awaiting it, you never handed control back to the event loop or store the awaitable, causing it to be immediately cleaned up, completely cancelling it (and all of the awaitables it controlled).
Simply make sure you await the results of the awaitables in your code, finding any you missed.

Google PubSub python client returning StatusCode.UNAVAILABLE

I am trying to establish a long running Pull subscription to a Google Cloud PubSub topic.
I am using a code very similar to the example given in the documentation here, i.e.:
def receive_messages(project, subscription_name):
"""Receives messages from a pull subscription."""
subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(
project, subscription_name)
def callback(message):
print('Received message: {}'.format(message))
message.ack()
subscriber.subscribe(subscription_path, callback=callback)
# The subscriber is non-blocking, so we must keep the main thread from
# exiting to allow it to process messages in the background.
print('Listening for messages on {}'.format(subscription_path))
while True:
time.sleep(60)
The problem is that I'm receiving the following traceback sometimes:
Exception in thread Consumer helper: consume bidirectional stream:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/path/to/google/cloud/pubsub_v1/subscriber/_consumer.py", line 248, in _blocking_consume
self._policy.on_exception(exc)
File "/path/to/google/cloud/pubsub_v1/subscriber/policy/thread.py", line 135, in on_exception
raise exception
File "/path/to/google/cloud/pubsub_v1/subscriber/_consumer.py", line 234, in _blocking_consume
for response in response_generator:
File "/path/to/grpc/_channel.py", line 348, in __next__
return self._next()
File "/path/to/grpc/_channel.py", line 342, in _next
raise self
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, The service was unable to fulfill your request. Please try again. [code=8a75])>
I saw that this was referenced in another question but here I am asking to how to handle it properly in Python. I have tried to wrap the request in an exception but it seems to run in the background and I am not able to retry in case of that error.

A somewhat hacky approach that is working for me is a custom policy_class. The default one has an on_exception function that ignores DEADLINE_EXCEEDED. You can make a class that inherits the default and also ignores UNAVAILABLE. Mine looks like this:
from google.cloud import pubsub
from google.cloud.pubsub_v1.subscriber.policy import thread
import grpc
class AvailablePolicy(thread.Policy):
def on_exception(self, exception):
"""The parent ignores DEADLINE_EXCEEDED. Let's also ignore UNAVAILABLE.
I'm not sure what triggers that error, but if you ignore it, your
subscriber seems to work just fine. It's probably an intermittent
thing and it reconnects later if you just give it a chance.
"""
# If this is UNAVAILABLE, then we want to retry.
# That entails just returning None.
unavailable = grpc.StatusCode.UNAVAILABLE
if getattr(exception, 'code', lambda: None)() == unavailable:
return
# For anything else, fallback on super.
super(AvailablePolicy, self).on_exception(exception)
subscriber = pubsub.SubscriberClient(policy_class=AvailablePolicy)
# Continue to set up as normal.
It looks a lot like the original on_exception just ignores a different error. If you want, you can add some logging whenever the exception is thrown and verify that everything still works. Future messages will still come through.

Method to make a new twisted reactor?

I am making a IRC Log Bot which saves thel logs datewise. I want the program to close the present reactor and make a new one ( this is because, it will save the logs in a new file). I wrote a sample program but it is unable to work-
def event():
if no date_change:
do normal work that has to be done
else:
stop present reactor
make a new reactor
Here is the actual code that I am using:-
def irc_NICK(self, prefix, params):
"""Called when an IRC user changes their nickname."""
old_nick = prefix.split('!')[0]
new_nick = params[0]
if self.factory.filename.find(file_name_gen())!=-1:
self.logger.log("<em>%s is now known as %s</em>" % (old_nick, new_nick),1)
else:
print "new itng"
reactor.stop()
irc.IRCClient.connectionLost(self, "Day Change")
#earlier the LogBotFactory object is f
f1 = LogBotFactory("meeting-test", file_name_gen())
reactor.connectTCP("irc.freenode.net", 6667, f1)
reactor.run()
The second LogBotFactory object gets created but, the program stops due to some Unhandled error.
This is the traceback that I am getting...
1971-01-02 23:59:41+0530 [-] Log opened.
1971-01-02 23:59:41+0530 [-] Starting factory <__main__.LogBotFactory instance at 0x27318c0>
1971-01-03 00:00:10+0530 [LogBot,client] new itng
1971-01-03 00:00:10+0530 [LogBot,client] Starting factory <__main__.LogBotFactory instance at 0x2989cb0>
1971-01-03 00:00:10+0530 [LogBot,client] Unhandled Error
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 221, in _dataReceived
rval = self.protocol.dataReceived(data)
File "/usr/lib/python2.7/dist-packages/twisted/words/protocols/irc.py", line 2412, in dataReceived
basic.LineReceiver.dataReceived(self, data.replace('\r', ''))
File "/usr/lib/python2.7/dist-packages/twisted/protocols/basic.py", line 581, in dataReceived
why = self.lineReceived(line)
File "/usr/lib/python2.7/dist-packages/twisted/words/protocols/irc.py", line 2420, in lineReceived
self.handleCommand(command, prefix, params)
--- <exception caught here> ---
File "/usr/lib/python2.7/dist-packages/twisted/words/protocols/irc.py", line 2464, in handleCommand
method(prefix, params)
File "irc.py", line 141, in irc_NICK
reactor.run()
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1191, in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1171, in startRunning
ReactorBase.startRunning(self)
File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 681, in startRunning
raise error.ReactorAlreadyRunning()
twisted.internet.error.ReactorAlreadyRunning:
1971-01-03 00:00:10+0530 [-] Main loop terminated.
I am new to python twisted.
Please help, Thanks.

print "new itng"
reactor.stop()
irc.IRCClient.connectionLost(self, "Day Change")
#earlier the LogBotFactory object is f
f1 = LogBotFactory("meeting-test", file_name_gen())
reactor.connectTCP("irc.freenode.net", 6667, f1)
reactor.run()
This problem is even easier to solve than you think. Delete the lines reactor.stop() and reactor.run() and you'll be all set. In other words, just leave the reactor running.
Separately, you also need to replace the line irc.IRCClient.connectionLost(self, "Day Change") with self.loseConnection(). Calling connectionLost does not close the connection. It gets called when Twisted sees the connection has been closed. If you call it yourself, your program might think the connection has been closed but it won't really have been closed - and after this happens enough times you'll be out of resources and your program won't work anymore.
You should only stop the reactor when you're done using Twisted (usually right before your program exits).

Bandwidth throttling using Twisted

I'm trying to set speed limits on downloading/uploading files and found that twisted provides twisted.protocols.policies.ThrottlingFactory to handle this job, but I can't get it right. I set readLimit and writeLimit, but file is still downloading on a maximum speed. What am I doing wrong?
from twisted.protocols.basic import FileSender
from twisted.protocols.policies import ThrottlingFactory
from twisted.web import server, resource
from twisted.internet import reactor
import os
class DownloadPage(resource.Resource):
isLeaf = True
def __init__(self, producer):
self.producer = producer
def render(self, request):
size = os.stat(somefile).st_size
request.setHeader('Content-Type', 'application/octet-stream')
request.setHeader('Content-Length', size)
request.setHeader('Content-Disposition', 'attachment; filename="' + somefile + '"')
request.setHeader('Accept-Ranges', 'bytes')
fp = open(somefile, 'rb')
d = self.producer.beginFileTransfer(fp, request)
def err(error):
print "error %s", error
def cbFinished(ignored):
fp.close()
request.finish()
d.addErrback(err).addCallback(cbFinished)
return server.NOT_DONE_YET
producer = FileSender()
root_resource = resource.Resource()
root_resource.putChild('download', DownloadPage(producer))
site = server.Site(root_resource)
tsite = ThrottlingFactory(site, readLimit=10000, writeLimit=10000)
tsite.protocol.producer = producer
reactor.listenTCP(8080, tsite)
reactor.run()
UPDATE
So sometime after I run it:
2012-10-25 09:17:03+0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 402, in startReactor
self.config, oldstdout, oldstderr, self.profiler, reactor)
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/application/app.py", line 323, in runReactorWithLogging
reactor.run()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 334, in unthrottleWrites
p.unthrottleWrites()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 225, in unthrottleWrites
self.producer.resumeProducing()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/basic.py", line 919, in resumeProducing
self.consumer.unregisterProducer()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/web/http.py", line 811, in unregisterProducer
self.transport.unregisterProducer()
File "/home/chambylov/environments/transfer/local/lib/python2.7/site-packages/twisted/protocols/policies.py", line 209, in unregisterProducer
del self.producer
exceptions.AttributeError: ThrottlingProtocol instance has no attribute 'producer'
I see that I'm not supposed to assign producer like I do know tsite.protocol.producer = producer, I'm new to Twisted and I don't know how to do that another way.

Every producer needs (eventually) to be registered with whatever you want to consume the data. I don't see registration happening anywhere here. Maybe that is the issue you are having?
Twisted has been used on some big-time projects like Friendster, but all the callbacks do not sit well with the usual way I write in python (and I have some experience with functional programming). I switched to gevent.
If you are working with gevent libraries, many of the details (callbacks/generators that provide the asynchronous functionality) are abstracted out, so that you can typically get away with just monkey patching your code and writing it in the usual object-oriented style you are used to. If you are working on a project with anyone unfamiliar with a callback-heavy language like js/lisp, I bet they will appreciate gevent over twisted.

As egbutter said, you have to register a producer. So instead of this:
tsite.protocol.producer = producer
you have to call a registerProducer method explicitly:
tsite.protocol.registerProducer( ... )
or, if you're using FileSender as a producer, call its beginFileTransfer method, in our case:
file_to_send = open( ... )
producer.beginFileTransfer(file_to_send, tsite.protocol)