Background
I have a class in python that takes in a list of mutexes. It then sorts that list, and uses __enter__() and __exit__() to lock/unlock all of the mutexes in a specific order to prevent deadlocks.
The class currently saves us a lot of hassle with potential deadlocks, as we can just invoke it in an RAII style, i.e.:
self.lock = SuperLock(list_of_locks)
# Lock all mutexes.
with self.lock:
# Issue calls to all hardware protected by these locks.
Problem
We'd like to expose ways for this class to provide an RAII-style API so we can lock only half of the mutexes at once, when called in a certain way, i.e.:
self.lock = SuperLock(list_of_locks)
# Lock all mutexes.
with self.lock:
# Issue calls to all hardware protected by these locks.
# Lock the first half of the mutexes in SuperLock.list_of_locks
with self.lock.first_half_only:
# Issue calls to all hardware protected by these locks.
# Lock the second half of the mutexes in SuperLock.list_of_locks
with self.lock.second_half_only:
# Issue calls to all hardware protected by these locks.
Question
Is there a way to provide this type of functionality so I could invoke with self.lock.first_half_only or with self.lock_first_half_only() to provide a simple API to users? We'd like to keep all this functionality in a single class.
Thank you.
Yes, you can get this interface. The object that will be entered/exited in context of a with statement is the resolved attribute. So you can go ahead and define context managers as attributes of your context manager:
from contextlib import ExitStack # pip install contextlib2
from contextlib import contextmanager
#contextmanager
def lock(name):
print("entering lock {}".format(name))
yield
print("exiting lock {}".format(name))
#contextmanager
def many(contexts):
with ExitStack() as stack:
for cm in contexts:
stack.enter_context(cm)
yield
class SuperLock(object):
def __init__(self, list_of_locks):
self.list_of_locks = list_of_locks
def __enter__(self):
# implement for entering the `with self.lock:` use case
return self
def __exit__(self, exce_type, exc_value, traceback):
pass
#property
def first_half_only(self):
return many(self.list_of_locks[:4])
#property
def second_half_only(self):
# yo dawg, we herd you like with-statements
return many(self.list_of_locks[4:])
When you create and return a new context manager, you may use state from the instance (i.e. self).
Example usage:
>>> list_of_locks = [lock(i) for i in range(8)]
>>> super_lock = SuperLock(list_of_locks)
>>> with super_lock.first_half_only:
... print('indented')
...
entering lock 0
entering lock 1
entering lock 2
entering lock 3
indented
exiting lock 3
exiting lock 2
exiting lock 1
exiting lock 0
Edit: class based equivalent of the lock generator context manager shown above
class lock(object):
def __init__(self, name):
self.name = name
def __enter__(self):
print("entering lock {}".format(self.name))
return self
def __exit__(self, exce_type, exc_value, traceback):
print("exiting lock {}".format(self.name))
# If you want to handle the exception (if any), you may use the
# return value of this method to suppress re-raising error on exit
from contextlib import contextmanager
class A:
#contextmanager
def i_am_lock(self):
print("entering")
yield
print("leaving")
a = A()
with a.i_am_lock():
print("inside")
Output:
entering
inside
leaving
Futher you can use contextlib.ExitStack to manage your locks better.
I'd use a SimpleNamespace to allow attribute access to different SuperLock objects, e.g.:
from types import SimpleNamespace
self.lock = SimpleNamespace(
all=SuperLock(list_of_locks),
first_two_locks=SuperLock(list_of_locks[:2]),
other_locks=SuperLock(list_of_locks[2:])
)
with self.lock.all:
# Issue calls to all hardware protected by these locks.
with self.lock.first_two_locks:
# Issue calls to all hardware protected by these locks.
with self.lock.other_locks:
# Issue calls to all hardware protected by these locks.
Edit:
For python 2, you can use this class to achieve a similar behavior:
class SimpleNamespace:
def __init__(self, **kwargs):
self.__dict__.update(kwargs)
Related
My resource can by of type R1 which requires locking or of type R2
which does not require it:
class MyClass(object): # broken
def __init__ (self, ...):
if ...:
self.resource = R1(...)
self.lock = threading.Lock()
else:
self.resource = R2(...)
self.lock = None
def foo(self): # there are many locking methods
with self.lock:
operate(self.resource)
The above obviously fails if self.lock is None.
My options are:
if:
def foo(self):
if self.lock:
with self.lock:
operate(self.resource)
else:
operate(self.resource)
cons: too verbose
pro: does not create an unnecessary threading.Lock
always set self.lock to threading.Lock
pro: code is simplified
cons: with self.lock appears to be relatively expensive
(comparable to disk i/o!)
define a trivial lock class:
class TrivialLock(object):
def __enter__(self): pass
def __exit__(self, _a, _b, _c): pass
def acquire(self): pass
def release(self): pass
and use it instead of None for R2.
pro: simple code
cons: I have to define TrivialLock
Questions
What method is preferred by the community?
Regardless of (1), does anyone actually define something like
TrivialLock? (I actually expected that something like that would be
in the standard library...)
Is my observation that locking cost is comparable to that of a
write conforms to expectations?
I would define TrivialLock. It can be even more trivial, though, since you just need a context manager, not a lock.
class TrivialLock(object):
def __enter__(self):
pass
def __exit__(*args):
pass
You can make this even more trivial using contextlib:
import contextlib
#contextlib.contextmanager
def TrivialLock():
yield
self.lock = TrivialLock()
And since yield can be an expression, you can define TrivalLock inline instead:
self.lock = contextlib.contextmanager(lambda: (yield))()
Note the parentheses; lambda: yield is invalid. However, the generator expression (yield) makes this a single-use context manager; if you try to use the same value in a second with statement, you get a Runtime error because the generator is exhausted.
I want to force object instantiation via class context manager. So make it impossible to instantiate directly.
I implemented this solution, but technically user can still instantiate object.
class HessioFile:
"""
Represents a pyhessio file instance
"""
def __init__(self, filename=None, from_context_manager=False):
if not from_context_manager:
raise HessioError('HessioFile can be only use with context manager')
And context manager:
#contextmanager
def open(filename):
"""
...
"""
hessfile = HessioFile(filename, from_context_manager=True)
Any better solution ?
If you consider that your clients will follow basic python coding principles then you can guarantee that no method from your class will be called if you are not within the context.
Your client is not supposed to call __enter__ explicitly, therefore if __enter__ has been called you know your client used a with statement and is therefore inside context (__exit__ will be called).
You just need to have a boolean variable that helps you remember if you are inside or outside context.
class Obj:
def __init__(self):
self._inside_context = False
def __enter__(self):
self._inside_context = True
print("Entering context.")
return self
def __exit__(self, *exc):
print("Exiting context.")
self._inside_context = False
def some_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some stuff with", name)
def some_other_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some other stuff with", name)
with Obj() as inst_a:
inst_a.some_stuff("A")
inst_a.some_other_stuff("A")
inst_b = Obj()
with inst_b:
inst_b.some_stuff("B")
inst_b.some_other_stuff("B")
inst_c = Obj()
try:
inst_c.some_stuff("c")
except Exception:
print("Instance C couldn't do stuff.")
try:
inst_c.some_other_stuff("c")
except Exception:
print("Instance C couldn't do some other stuff.")
This will print:
Entering context.
Doing some stuff with A
Doing some other stuff with A
Exiting context.
Entering context.
Doing some stuff with B
Doing some other stuff with B
Exiting context.
Instance C couldn't do stuff.
Instance C couldn't do some other stuff.
Since you'll probably have many methods that you want to "protect" from being called from outside context, then you can write a decorator to avoid repeating the same code to test for your boolean:
def raise_if_outside_context(method):
def decorator(self, *args, **kwargs):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
return method(self, *args, **kwargs)
return decorator
Then change your methods to:
#raise_if_outside_context
def some_other_stuff(self, name):
print("Doing some other stuff with", name)
I suggest the following approach:
class MainClass:
def __init__(self, *args, **kwargs):
self._class = _MainClass(*args, **kwargs)
def __enter__(self):
print('entering...')
return self._class
def __exit__(self, exc_type, exc_val, exc_tb):
# Teardown code
print('running exit code...')
pass
# This class should not be instantiated directly!!
class _MainClass:
def __init__(self, attribute1, attribute2):
self.attribute1 = attribute1
self.attribute2 = attribute2
...
def method(self):
# execute code
if self.attribute1 == "error":
raise Exception
print(self.attribute1)
print(self.attribute2)
with MainClass('attribute1', 'attribute2') as main_class:
main_class.method()
print('---')
with MainClass('error', 'attribute2') as main_class:
main_class.method()
This will outptut:
entering...
attribute1
attribute2
running exit code...
---
entering...
running exit code...
Traceback (most recent call last):
File "scratch_6.py", line 34, in <module>
main_class.method()
File "scratch_6.py", line 25, in method
raise Exception
Exception
None that I am aware of. Generally, if it exists in python, you can find a way to call it. A context manager is, in essence, a resource management scheme... if there is no use-case for your class outside of the manager, perhaps the context management could be integrated into the methods of the class? I would suggest checking out the atexit module from the standard library. It allows you to register cleanup functions much in the same way that a context manager handles cleanup, but you can bundle it into your class, such that each instantiation has a registered cleanup function. Might help.
It is worth noting that no amount of effort will prevent people from doing stupid things with your code. Your best bet is generally to make it as easy as possible for people to do smart things with your code.
You can think of hacky ways to try and enforce this (like inspecting the call stack to forbid direct calls to your object, boolean attribute that is set upon __enter__ that you check before allowing other actions on the instance) but that will eventually become a mess to understand and explain to others.
Irregardless, you should also be certain that people will always find ways to bypass it if wanted. Python doesn't really tie your hands down, if you want to do something silly it lets you do it; responsible adults, right?
If you need an enforcement, you'd be better off supplying it as a documentation notice. That way if users opt to instantiate directly and trigger unwanted behavior, it's their fault for not following guidelines for your code.
I have a UUT class which instantiates Worker objects, and calls their do_stuff() method.
The Worker objects uses a Provider object for two things:
Calls methods on the provider object to do some stuff
Gets notifications from the provider by subscribing a method with the provider's events
When a worker gets a notification, it processes it, an notifies the UUT object, which in reponse can create more Worker objects.
I've already tested each class on its own, and I want to test UUT+Worker together. For that, I intend to mock-out Provider.
import mock
import unittest
import provider
class Worker():
def __init__(self, *args):
resource.default_resource.subscribe('on_spam', self._on_spam) # I'm going to patch 'resource.default_resource'
def do_stuff(self):
self.resource.do_stuff()
def _on_spam(self, message):
self._tell_uut_to_create_more_workers(message['num_of_new_workers_to_create'])
class UUT():
def __init__(self, *args):
self._workers = []
def gen_worker_and_do_stuff(self, *args)
worker = Worker(*args)
self._workers.append(resource)
worker.do_stuff()
class TestCase1(unittest.TestCase):
#mock.patch('resource.default_resource', spec_set=resource.Resource)
def test_1(self, mock_resource):
uut = UUT()
uut.gen_worker_and_do_stuff('Egg') # <-- say I automagically grabbed the resulting Worker into self.workers
self.workers[0]._on_spam({'num_of_new_workers_to_create':5}) # <-- I also want to get hold of the newly-created workers
Is there a way to grab the worker objects generated by uut, without directly accessing the _workers list in uut (which is an implementation detail)?
I guess I can do it in Worker.__init__, where the worker subscribes to provider events, so I guess the question reduces to:
How to I extract the self in the callee, when calling resource.default_resource.subscribe('on_spam', self._on_spam)?
As an application of the Dependency Inversion principle, I'd pass the Worker class as a dependency to UUT:
class UUT():
def __init__(self, make_worker=Worker):
self._workers = []
self._make_worker = make_worker
def gen_worker_and_connect(self, *args)
worker = self._make_worker(*args)
self._workers.append(resource)
worker.connect()
Then provide anything you want from the test instead of Worker. This own function could share the created object with the test scope. Besides solving this particular problem, that would also make the dependency explicit and independent of the UUT implementation. And you would not need to mock the resource thing as well, which makes the test dependent on things unrelated to the class under test.
I am trying to debug a multi-threaded Python application that uses various locks.
Rather than place log.debug(...) statements all over the shot to track where and when the locks are acquired and released, my idea is to decorate the methods threading.Lock.acquire() and threading.Lock.release(), and prefix their invocation with something like the following:
log.debug("lock::acquire() [%s.%s.%s]" %
(currentThread().getName(),
self.__class__.__name__,
sys._getframe().f_code.co_name))
Where log is some global logging object - for the sake of discussion.
Now ideally the name "lock" in the log entry should be derived at runtime, so that irrespective of which lock object these methods are invoked upon the log will output its name, the operation decorated, the current thread, class, and function in which the operation (acquire | release) is called.
Disclaimer: I acknowledge that the code given above would not be sufficient for any such decorator implementation. It is only provided just to give a flavour of what I think could be achieved.
Does anyone know if I can decorate standard library methods, without doctoring the original source code of the threading library, i.e., from within my calling application code?
Perhaps I am barking up the wrong tree and there is a better way of achieving the same ends, without using decorators? Many thanks in advance for any guidance if this is indeed the case.
Solution: (inspired by lazyr)
The following code logs the lock operations and gives me the name of the method/function calling the lock operation (I am also adapting the code to work with Conditions and their additional wait() and notify() methods):
# Class to wrap Lock and simplify logging of lock usage
class LogLock(object):
"""
Wraps a standard Lock, so that attempts to use the
lock according to its API are logged for debugging purposes
"""
def __init__(self, name, log):
self.name = str(name)
self.log = log
self.lock = threading.Lock()
self.log.debug("{0} created {1}".format(
inspect.stack()[1][3], self.name))
def acquire(self, blocking=True):
self.log.debug("{0} trying to acquire {1}".format(
inspect.stack()[1][3], self.name))
ret = self.lock.acquire(blocking)
if ret == True:
self.log.debug("{0} acquired {1}".format(
inspect.stack()[1][3], self.name))
else:
self.log.debug("{0} non-blocking acquire of {1} lock failed".format(
inspect.stack()[1][3], self.name))
return ret
def release(self):
self.log.debug("{0} releasing {1}".format(inspect.stack()[1][3], self.name))
self.lock.release()
def __enter__(self):
self.acquire()
def __exit__(self, exc_type, exc_val, exc_tb):
self.release()
return False # Do not swallow exceptions
Where the log instance passed to LogLock.init was defined with a logging.Formatter as follows to given me the calling thread's identity:
# With the following format
log_format = \
logging.Formatter('%(asctime)s %(levelname)s %(threadName)s %(message)s')
I recently had just your problem. I set up my logger to automatically log thread name, like in this answer. I found out it was not possible to subclass Lock, so I had to wrap it, like this:
class LogLock(object):
def __init__(self, name):
self.name = str(name)
self.lock = Lock()
def acquire(self, blocking=True):
log.debug("{0:x} Trying to acquire {1} lock".format(
id(self), self.name))
ret = self.lock.acquire(blocking)
if ret == True:
log.debug("{0:x} Acquired {1} lock".format(
id(self), self.name))
else:
log.debug("{0:x} Non-blocking aquire of {1} lock failed".format(
id(self), self.name))
return ret
def release(self):
log.debug("{0:x} Releasing {1} lock".format(id(self), self.name))
self.lock.release()
def __enter__(self):
self.acquire()
def __exit__(self, exc_type, exc_val, exc_tb):
self.release()
return False # Do not swallow exceptions
I logged the id of the object so I could distinguish between multiple locks with the same name, you might not need it.
I need to register an atexit function for use with a class (see Foo below for an example) that, unfortunately, I have no direct way of cleaning up via a method call: other code, that I don't have control over, calls Foo.start() and Foo.end() but sometimes doesn't call Foo.end() if it encounters an error, so I need to clean up myself.
I could use some advice on closures in this context:
class Foo:
def cleanup(self):
# do something here
def start(self):
def do_cleanup():
self.cleanup()
atexit.register(do_cleanup)
def end(self):
# cleanup is no longer necessary... how do we unregister?
Will the closure work properly, e.g. in do_cleanup, is the value of self bound correctly?
How can I unregister an atexit() routine?
Is there a better way to do this?
edit: this is Python 2.6.5
Make a registry a global registry and a function that calls a function in it, and remove them from there when necessary.
cleaners = set()
def _call_cleaners():
for cleaner in list(cleaners):
cleaner()
atexit.register(_call_cleaners)
class Foo(object):
def cleanup(self):
if self.cleaned:
raise RuntimeError("ALREADY CLEANED")
self.cleaned = True
def start(self):
self.cleaned = False
cleaners.add(self.cleanup)
def end(self):
self.cleanup()
cleaners.remove(self.cleanup)
I think the code is fine. There's no way to unregister, but you can set a boolean flag that would disable cleanup:
class Foo:
def __init__(self):
self.need_cleanup = True
def cleanup(self):
# do something here
print 'clean up'
def start(self):
def do_cleanup():
if self.need_cleanup:
self.cleanup()
atexit.register(do_cleanup)
def end(self):
# cleanup is no longer necessary... how do we unregister?
self.need_cleanup = False
Lastly, bear in mind that atexit handlers don't get called if "the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called."
self is bound correctly inside the callback to do_cleanup, but in fact if all you are doing is calling the method you might as well use the bound method directly.
You use atexit.unregister() to remove the callback, but there is a catch here as you must unregister the same function that you registered and since you used a nested function that means you have to store a reference to that function. If you follow my suggestion of using a bound method then you still have to save a reference to it:
class Foo:
def cleanup(self):
# do something here
def start(self):
self._cleanup = self.cleanup # Need to save the bound method for unregister
atexit.register(self._cleanup)
def end(self):
atexit.unregister(self._cleanup)
Note that it is still possible for your code to exit without calling ther atexit registered functions, for example if the process is aborted with ctrl+break on windows or killed with SIGABRT on linux.
Also as another answer suggests you could just use __del__ but that can be problematic for cleanup while a program is exiting as it may not be called until after other globals it needs to access have been deleted.
Edited to note that when I wrote this answer the question didn't specify Python 2.x. Oh well, I'll leave the answer here anyway in case it helps anyone else.
Since shanked deleted his posting, I'll speak in favor of __del__ again:
import atexit, weakref
class Handler:
def __init__(self, obj):
self.obj = weakref.ref(obj)
def cleanup(self):
if self.obj is not None:
obj = self.obj()
if obj is not None:
obj.cleanup()
class Foo:
def __init__(self):
self.start()
def cleanup(self):
print "cleanup"
self.cleanup_handler = None
def start(self):
self.cleanup_handler = Handler(self)
atexit.register(self.cleanup_handler.cleanup)
def end(self):
if self.cleanup_handler is None:
return
self.cleanup_handler.obj = None
self.cleanup()
def __del__(self):
self.end()
a1=Foo()
a1.end()
a1=Foo()
a2=Foo()
del a2
a3=Foo()
a3.m=a3
This supports the following cases:
objects where .end is called regularly; cleanup right away
objects that are released without .end being called; cleanup when the last
reference goes away
objects living in cycles; cleanup atexit
objects that are kept alive; cleanup atexit
Notice that it is important that the cleanup handler holds a weak reference
to the object, as it would otherwise keep the object alive.
Edit: Cycles involving Foo will not be garbage-collected, since Foo implements __del__. To allow for the cycle being deleted at garbage collection time, the cleanup must be taken out of the cycle.
class Cleanup:
cleaned = False
def cleanup(self):
if self.cleaned:
return
print "cleanup"
self.cleaned = True
def __del__(self):
self.cleanup()
class Foo:
def __init__(self):...
def start(self):
self.cleaner = Cleanup()
atexit.register(Handler(self).cleanup)
def cleanup(self):
self.cleaner.cleanup()
def end(self):
self.cleanup()
It's important that the Cleanup object has no references back to Foo.
Why don't you try it? It only took me a minute to check.
(Answer: Yes)
However, you can simplify it. The closure isn't needed.
class Foo:
def cleanup(self):
pass
def start(self):
atexit.register(self.cleanup)
And to not cleanup twice, just check in the cleanup method if a cleanup is needed or not before you clean up.