We Keep Coding

redis Python catching exception that doesn't inherit from BaseException

We're seeing exceptions in our log like the following:
ERROR Exception ignored in: <function Connection.__del__ at 0x7f9b70a5cc20>
Traceback (most recent call last):
File "/app/.heroku/python/lib/python3.7/site-packages/redis/connection.py", line 537, in __del__
File "/app/.heroku/python/lib/python3.7/site-packages/redis/connection.py", line 667, in disconnect
TypeError: catching classes that do not inherit from BaseException is not allowed
According to the Redis source code, the offending line is the except in the following snippet:
try:
if os.getpid() == self.pid:
shutdown(self._sock, socket.SHUT_RDWR)
self._sock.close()
except socket.error:
pass
Which would indicate that socket.exception doesn't inherit from BaseException. However, as far as I can tell (based on the docs and the mro class method), socket.exception does inherit from BaseException.
Why is this happening? What can I do to prevent it?
By the way, our code doesn't call Redis directly. We are using Redis Queue (rq), which is implemented using Redis.

This would happen if you didn't close the redis client explicitly. That's why you saw __del__ in the traceback.
I'm not using rq but I'll take celery as an example, yet the idea could also be applied to rq.
# tasks/__init__.py
from celeryapp import redis_client
#worker_shutdown.connect # this signal means it's about to shut down the worker
def cleanup(**kwargs):
redis_client.close() # without this you may see error
# celeryapp.py
from celery import Celery
import redis
app = Celery(config_source="celeryconfig")
redis_client = redis.StrictRedis()

Parallel: Import a python file from sibling folder

I have a directory tree
working_dir\
main.py
my_agent\
my_worker.py
my_utility\
my_utils.py
Code in each file is as follows
""" main.py """
import os, sys
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from my_agent.my_worker import MyWorker
import ray
ray.init()
workers = [MyWorker.remote(i) for i in range(10)]
ids = [worker.get_id.remote() for worker in workers]
# print(*ids, sep='\n')
print(*ray.get(ids), sep='\n')
""" worker.py """
from my_utility import my_utils
import ray
#ray.remote
class MyWorker():
def __init__(self, id):
self.id = id
def get_id(self):
return my_utils.f(self.id)
""" my_utils.py """
def f(id):
return '{}: Everything is fine...'.format(id)
Here's a part of the error message I received
Traceback (most recent call last):
File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/function_manager.py", line 616, in fetch_and_register_actor
unpickled_class = pickle.loads(pickled_class)
File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/cloudpickle/cloudpickle.py", line 894, in subimport
import(name)
ImportError: No module named 'my_utility'
Traceback (most recent call last):
File "main.py", line 12, in
print(*ray.get(ids), sep='\n')
File "/Users/aptx4869/anaconda3/envs/p35/lib/python3.5/site-packages/ray/worker.py", line 2377, in get
raise value
ray.worker.RayTaskError: ray_worker (pid=30025, host=AiMacbook)
Exception: The actor with name MyWorker failed to be imported, and so cannot execute this method
If I remove all statements related to ray, the above code works fine. Therefore, I boldly guess the reason is that ray runs each actor in a new process and sys.path.append only works in the main process. So I add the following code to worker.py
import os, sys
sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
But it still does not work: the same error message shows up. Now I run out of ideas, what should I do?

You are correct about what the issue is.
In your example, you modify sys.path in main.py in order to be able to import my_agent.my_worker and my_utility.my_utils.
However, this path change is not propagated to the worker processes, so if you were to run a remote function like
#ray.remote
def f():
# Print the PYTHONPATH on the worker process.
import sys
print(sys.path)
f.remote()
You would see that sys.path on the worker does not include the parent directory that you added.
The reason that modifying sys.path on the worker (e.g., in the MyWorker constructor) doesn't work is that the MyWorker class definition is pickled and shipped to the workers. Then the worker unpickles it, and the process of unpickling the class definition requires my_utils to be imported, and this fails because the actor constructor hasn't had a chance to run yet.
There are a couple possible solutions here.
Run the script with something like
PYTHONPATH=$(dirname $(pwd)):$PYTHONPATH python main.py
(from within working_dir/). That should solve the issue because in this case the worker processes are forked from the scheduler process (which is forked from the main Python interpreter when you call ray.init() and so the environment variable will be inherited by the workers (this doesn't happen for sys.path presumably because it is not an environment variable).
It looks like adding the line
parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
os.environ["PYTHONPATH"] = parent_dir + ":" + os.environ.get("PYTHONPATH", "")
in main.py (before the ray.init() call) also works for the same reason as above.
Consider adding a setup.py and installing your project as a Python package so that it's automatically on the relevant path.

The new "Runtime Environments" feature, which didn't exist at the time of this post, should help with this issue: https://docs.ray.io/en/latest/handling-dependencies.html#runtime-environments. (See the working_dir and py_modules entries.)

ReactorNotRestartable when launching two equivalent unittest with twisted and trial

I've two test classes (TrialTest1 and TrialTest2) written in two files (test_trial1.py and test_trial2.py) mostly identical (the only difference is the class name):
from twisted.internet import reactor
from twisted.trial import unittest
class TrialTest1(unittest.TestCase):
def setUp(self):
print("setUp()")
def test_main(self):
print("test_main")
reactor.callLater(1, self._called_by_deffered1)
reactor.run()
def _called_by_deffered1(self):
print("_called_by_deffered1")
reactor.callLater(1, self._called_by_deffered2)
def _called_by_deffered2(self):
print("_called_by_deffered2")
reactor.stop()
def tearDown(self):
print("tearDown()")
When I run each test idepently, everything is fine. But when I launch both I've the following output:
setUp()
test_main
_called_by_deffered1
_called_by_deffered2
tearDown()
setUp()
test_main
tearDown()
Error
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 137, in maybeDeferred
result = f(*args, **kw)
File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 203, in runWithWarningsSuppressed
reraise(exc_info[1], exc_info[2])
File "/usr/lib/python2.7/site-packages/twisted/internet/utils.py", line 199, in runWithWarningsSuppressed
result = f(*a, **kw)
File "/home/kartoch/works/python/netkython/tests/test_twisted_trial2.py", line 13, in test_main
reactor.run()
File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 1191, in run
self.startRunning(installSignalHandlers=installSignalHandlers)
File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 1171, in startRunning
ReactorBase.startRunning(self)
File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 683, in startRunning
raise error.ReactorNotRestartable()
ReactorNotRestartable
Error
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x8d6482c [0.98535490036s] called=0 cancelled=0 TrialTest2._called_by_deffered1()>
Process finished with exit code 0
It seems the reactor is not switched off correctly after the first test. Does anyone know where is the problem ? It seems tearDown() is called to early (before _called_by_deffered1 in the second test), maybe a fix would be to use deferTearDown (not documented method of trial unittest).
EDIT
One of the solution proposed was to remove reactor.run() and reactor.stop() because a reactor is not restartable and you have only one reactor instance for all test by default:
class TrialTest1(unittest.TestCase):
def setUp(self):
print("setUp()")
def test_main(self):
print("test_main")
reactor.callLater(1, self._called_by_deffered1)
def _called_by_deffered1(self):
print("_called_by_deffered1")
reactor.callLater(1, self._called_by_deffered2)
def _called_by_deffered2(self):
print("_called_by_deffered2")
def tearDown(self):
print("tearDown()")
But when removing calls to such methods, my tests fail without executing _called_by_deffered methods:
setUp()
test_main
tearDown()
Error
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x94967ec [0.99936413765s] called=0 cancelled=0 TrialTest1._called_by_deffered1()>
setUp()
test_main
tearDown()
Error
DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x94968cc [0.99958896637s] called=0 cancelled=0 TrialTest2._called_by_deffered1()>
If I want to use only one instance of reactor shared between tests, how _called_by_deffered methods could be part of the test (i.e. executed before tearDown) ?

The reactor is not restartable. There are two obvious options for your to pursue for writing your tests.
One is to use the global reactor. trial starts and stops it - your tests never have to call reactor.run or reactor.stop (and they never should). It is accessible in the usual way, from twisted.internet import reactor.
The other is to use a new reactor instance per test. There are some test-oriented reactor instances in twisted.test.proto_helpers (that is the only part of twisted.test that is a public, supported interface by the way). MemoryReactor and StringTransport get you most of the way towards being able to test network interactions. twisted.internet.task.Clock helps you out with testing time-based events.

With the help of Jean-Paul, this page and this question, I've been able to fix the problem using twisted.internet.task.deferLater(). To summarize the point i was looking for: if a test method returning a deferred, the 'tearDown()' method will be called only when all deferreds are fired.
This is the code:
from twisted.trial import unittest
from twisted.internet import reactor, task
class TrialTest1(unittest.TestCase):
def setUp(self):
print("setUp()")
def test_main(self):
print("test_main()")
return task.deferLater(reactor, 1, self._called_by_deffered1)
def _called_by_deffered1(self):
print("_called_by_deffered1()")
return task.deferLater(reactor, 1, self._called_by_deffered2)
def _called_by_deffered2(self):
print("_called_by_deffered2()")
def tearDown(self):
print("tearDown()")
Output:
setUp()
test_main()
// (1s wait)
_called_by_deffered1()
// (1s wait)
_called_by_deffered2()
tearDown()

How do you call class methods in a signal handler in python daemon?

I'm trying to write a signal handler that will call methods from a class variable.
I have code that looks like this:
import daemon
class bar():
def func():
print "Hello World!\n"
def sigusr1_handler(signum,frame):
foo.func()
def main():
foo = bar()
context = daemon.DaemonContext(stdout=sys.stdout)
context.signal_map = {
signal.SIGUSR1: sigusr1_handler
}
with context:
if (__name__="__main__"):
main()
This doesn't work. Python throws a NameError exception when I do a kill -USR1 on the daemon.
I also tried defining functions inside main that would handle the exception and call those functions from the signal handlers, but that didn't work either.
Anybody have ideas on how to implement this?

One option would be to import class bar inside your sigusr1_handler function. It's probably a good idea to have it in a different file anyway

Do you import signal? Because if I run you code I get:
Traceback (most recent call last):
File "pydaemon.py", line 16, in <module>
signal.SIGUSR1: sigusr1_handler
NameError: name 'signal' is not defined
You might fix this with:
import signal
And have a look at your string comparison oparator
with context:
if (__name__="__main__"):
main()
I generally use the '==' operator instead of '='

Is it possible to subclass Lock() objects in Python? If not, other ways to debug deadlock?

So, I've got a multithreaded python program, which is currently suffering from deadlock. I was going to log lock acquiring by subclassing threading.Lock objects:
import traceback
class DebugLock(threading.Lock):
def acquire(self):
print >>sys.stderr, "acquired", self
#traceback.print_tb
threading.Lock.acquire(self)
def release(self):
print >>sys.stderr, "released", self
#traceback.print_tb
threading.Lock.release(self)
When I try to run the program, I get the following error:
class DebugLock(threading.Lock):
TypeError: Error when calling the metaclass bases
cannot create 'builtin_function_or_method' instances
So, my question is twofold:
Is it possible to subclass Lock objects to do what I'm doing?
If not, what is the best way to debug deadlock in python?
Note: I'm not writing any Python extension. There's a similar question: How to debug deadlock with python?
However, it deals with compiling C++ code and using GDB, which I can't do since my code is pure python.

You could just use the "has a lock" versus "is a lock" approach, like so:
import threading, traceback, sys
class DebugLock(object):
def __init__(self):
self._lock = threading.Lock()
def acquire(self):
print("acquired", self)
#traceback.print_tb
self._lock.acquire()
def release(self):
print("released", self)
#traceback.print_tb
self._lock.release()
def __enter__(self):
self.acquire()
def __exit__(self, type, value, traceback):
self.release()
where I've thrown in the appropriate context guards since you likely want to use the with syntax with your locks (who wouldn't?).
Usage shown below:
>>> lock = DebugLock()
>>> with lock:
... print("I'm atomic!")
...
acquired <__main__.DebugLock object at 0x7f8590e50190>
I'm atomic!
released <__main__.DebugLock object at 0x7f8590e50190>
>>>

Russ answered the important question (#2), I'll answer question #1.
Doesn't appear to be possible. threading.Lock() is a factory function (documentation). It calls thread.allocate_lock() - there's no control over Lock object creation. You also cannot monkeypatch the thread.LockType class definition (the class skeleton exposed in thread.pi).
>>> thread.LockType.foo = "blah"
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type 'thread.lock'

If you want to do something like inheritance without running into this error, I suggest you try
import traceback
from threading import Lock
class DebugLock():
def __init__(self,lock = None):
self.lock = lock or Lock()
# normally done with __dict__
for command in dir(self.lock):
self.__dict__[command] = getattr(self.lock,command)
My normal method of using self.__dict__.update(lock.__dict__) doesn't seem to be working. I tested this out with the locking code
X = DebugLock()
y = X.lock
Y = DebugLock(y)
X.acquire()
Y.acquire()
X.release()
Y.release()
and that hangs, so I think it is working.