We Keep Coding

Should the "opening work" of a context manager happen in __init__ or __enter__?

I found the following example of a Context Manager for a File object:
class File(object):
def __init__(self, file_name, method):
self.file_obj = open(file_name, method)
def __enter__(self):
return self.file_obj
def __exit__(self, type, value, traceback):
self.file_obj.close()
Here, the work done by the manager, that is actually opening the file, happens in the __init__ method. However, in the accompanying text, they suggest that the file opening should happen in the __enter__ call:
Let’s talk about what happens under-the-hood.
The with statement stores the exit method of the File class.
It calls the enter method of the File class.
The enter method opens the file and returns it.
The opened file handle is passed to opened_file.
We write to the file using .write().
The with statement calls the stored exit method.
The exit method closes the file.
Which is the correct approach in general? It seems to be that the work undone by __exit__ should happen in __enter__, not __init__ since those are paired 1:1 by the context manager mechanism, but this example leaves me doubtful.

There is no general answer. It depends on what the work is. For example, for a file, opening happens in __init__, but for a lock, locking happens in __enter__.
One important thing to think about is, what should happen if the object is not used as a context manager, or not immediately used as a context manager? What should the object's state be after construction? Should relevant resources already be acquired at that point?
For a file, the answer is yes, the file should already be open, so opening happens in __init__. For a lock, the answer is no, the lock should not be locked, so locking goes in __enter__.
Another thing to consider is, should this object be usable as a context manager more than once? If entering a context manager twice should do a thing twice, that thing needs to happen in __enter__.

Hereis a better example
class TraceBlock:
def message(self, arg):
print('running ' + arg)
def __enter__(self):
print('starting with block')
return self
def __exit__(self, exc_type, exc_value, exc_tb):
if exc_type is None:
print('exited normally\n')
else:
print('raise an exception! ' + str(exc_type))
return False # Propagate
#--------------------------
if __name__ == '__main__':
with TraceBlock() as action:
action.message('test 1')
print('reached')
with TraceBlock() as action:
action.message('test 2')
raise TypeError
print('not reached')
If The Exit returns False the exception will pass on to other handler, if it return True the exception would not go to others.

Multiple ways to invoke context manager in python

Background
I have a class in python that takes in a list of mutexes. It then sorts that list, and uses __enter__() and __exit__() to lock/unlock all of the mutexes in a specific order to prevent deadlocks.
The class currently saves us a lot of hassle with potential deadlocks, as we can just invoke it in an RAII style, i.e.:
self.lock = SuperLock(list_of_locks)
# Lock all mutexes.
with self.lock:
# Issue calls to all hardware protected by these locks.
Problem
We'd like to expose ways for this class to provide an RAII-style API so we can lock only half of the mutexes at once, when called in a certain way, i.e.:
self.lock = SuperLock(list_of_locks)
# Lock all mutexes.
with self.lock:
# Issue calls to all hardware protected by these locks.
# Lock the first half of the mutexes in SuperLock.list_of_locks
with self.lock.first_half_only:
# Issue calls to all hardware protected by these locks.
# Lock the second half of the mutexes in SuperLock.list_of_locks
with self.lock.second_half_only:
# Issue calls to all hardware protected by these locks.
Question
Is there a way to provide this type of functionality so I could invoke with self.lock.first_half_only or with self.lock_first_half_only() to provide a simple API to users? We'd like to keep all this functionality in a single class.
Thank you.

Yes, you can get this interface. The object that will be entered/exited in context of a with statement is the resolved attribute. So you can go ahead and define context managers as attributes of your context manager:
from contextlib import ExitStack # pip install contextlib2
from contextlib import contextmanager
#contextmanager
def lock(name):
print("entering lock {}".format(name))
yield
print("exiting lock {}".format(name))
#contextmanager
def many(contexts):
with ExitStack() as stack:
for cm in contexts:
stack.enter_context(cm)
yield
class SuperLock(object):
def __init__(self, list_of_locks):
self.list_of_locks = list_of_locks
def __enter__(self):
# implement for entering the `with self.lock:` use case
return self
def __exit__(self, exce_type, exc_value, traceback):
pass
#property
def first_half_only(self):
return many(self.list_of_locks[:4])
#property
def second_half_only(self):
# yo dawg, we herd you like with-statements
return many(self.list_of_locks[4:])
When you create and return a new context manager, you may use state from the instance (i.e. self).
Example usage:
>>> list_of_locks = [lock(i) for i in range(8)]
>>> super_lock = SuperLock(list_of_locks)
>>> with super_lock.first_half_only:
... print('indented')
...
entering lock 0
entering lock 1
entering lock 2
entering lock 3
indented
exiting lock 3
exiting lock 2
exiting lock 1
exiting lock 0
Edit: class based equivalent of the lock generator context manager shown above
class lock(object):
def __init__(self, name):
self.name = name
def __enter__(self):
print("entering lock {}".format(self.name))
return self
def __exit__(self, exce_type, exc_value, traceback):
print("exiting lock {}".format(self.name))
# If you want to handle the exception (if any), you may use the
# return value of this method to suppress re-raising error on exit

from contextlib import contextmanager
class A:
#contextmanager
def i_am_lock(self):
print("entering")
yield
print("leaving")
a = A()
with a.i_am_lock():
print("inside")
Output:
entering
inside
leaving
Futher you can use contextlib.ExitStack to manage your locks better.

I'd use a SimpleNamespace to allow attribute access to different SuperLock objects, e.g.:
from types import SimpleNamespace
self.lock = SimpleNamespace(
all=SuperLock(list_of_locks),
first_two_locks=SuperLock(list_of_locks[:2]),
other_locks=SuperLock(list_of_locks[2:])
)
with self.lock.all:
# Issue calls to all hardware protected by these locks.
with self.lock.first_two_locks:
# Issue calls to all hardware protected by these locks.
with self.lock.other_locks:
# Issue calls to all hardware protected by these locks.
Edit:
For python 2, you can use this class to achieve a similar behavior:
class SimpleNamespace:
def __init__(self, **kwargs):
self.__dict__.update(kwargs)

Python How to force object instantiation via Context Manager?

I want to force object instantiation via class context manager. So make it impossible to instantiate directly.
I implemented this solution, but technically user can still instantiate object.
class HessioFile:
"""
Represents a pyhessio file instance
"""
def __init__(self, filename=None, from_context_manager=False):
if not from_context_manager:
raise HessioError('HessioFile can be only use with context manager')
And context manager:
#contextmanager
def open(filename):
"""
...
"""
hessfile = HessioFile(filename, from_context_manager=True)
Any better solution ?

If you consider that your clients will follow basic python coding principles then you can guarantee that no method from your class will be called if you are not within the context.
Your client is not supposed to call __enter__ explicitly, therefore if __enter__ has been called you know your client used a with statement and is therefore inside context (__exit__ will be called).
You just need to have a boolean variable that helps you remember if you are inside or outside context.
class Obj:
def __init__(self):
self._inside_context = False
def __enter__(self):
self._inside_context = True
print("Entering context.")
return self
def __exit__(self, *exc):
print("Exiting context.")
self._inside_context = False
def some_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some stuff with", name)
def some_other_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some other stuff with", name)
with Obj() as inst_a:
inst_a.some_stuff("A")
inst_a.some_other_stuff("A")
inst_b = Obj()
with inst_b:
inst_b.some_stuff("B")
inst_b.some_other_stuff("B")
inst_c = Obj()
try:
inst_c.some_stuff("c")
except Exception:
print("Instance C couldn't do stuff.")
try:
inst_c.some_other_stuff("c")
except Exception:
print("Instance C couldn't do some other stuff.")
This will print:
Entering context.
Doing some stuff with A
Doing some other stuff with A
Exiting context.
Entering context.
Doing some stuff with B
Doing some other stuff with B
Exiting context.
Instance C couldn't do stuff.
Instance C couldn't do some other stuff.
Since you'll probably have many methods that you want to "protect" from being called from outside context, then you can write a decorator to avoid repeating the same code to test for your boolean:
def raise_if_outside_context(method):
def decorator(self, *args, **kwargs):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
return method(self, *args, **kwargs)
return decorator
Then change your methods to:
#raise_if_outside_context
def some_other_stuff(self, name):
print("Doing some other stuff with", name)

I suggest the following approach:
class MainClass:
def __init__(self, *args, **kwargs):
self._class = _MainClass(*args, **kwargs)
def __enter__(self):
print('entering...')
return self._class
def __exit__(self, exc_type, exc_val, exc_tb):
# Teardown code
print('running exit code...')
pass
# This class should not be instantiated directly!!
class _MainClass:
def __init__(self, attribute1, attribute2):
self.attribute1 = attribute1
self.attribute2 = attribute2
...
def method(self):
# execute code
if self.attribute1 == "error":
raise Exception
print(self.attribute1)
print(self.attribute2)
with MainClass('attribute1', 'attribute2') as main_class:
main_class.method()
print('---')
with MainClass('error', 'attribute2') as main_class:
main_class.method()
This will outptut:
entering...
attribute1
attribute2
running exit code...
---
entering...
running exit code...
Traceback (most recent call last):
File "scratch_6.py", line 34, in <module>
main_class.method()
File "scratch_6.py", line 25, in method
raise Exception
Exception

None that I am aware of. Generally, if it exists in python, you can find a way to call it. A context manager is, in essence, a resource management scheme... if there is no use-case for your class outside of the manager, perhaps the context management could be integrated into the methods of the class? I would suggest checking out the atexit module from the standard library. It allows you to register cleanup functions much in the same way that a context manager handles cleanup, but you can bundle it into your class, such that each instantiation has a registered cleanup function. Might help.
It is worth noting that no amount of effort will prevent people from doing stupid things with your code. Your best bet is generally to make it as easy as possible for people to do smart things with your code.

You can think of hacky ways to try and enforce this (like inspecting the call stack to forbid direct calls to your object, boolean attribute that is set upon __enter__ that you check before allowing other actions on the instance) but that will eventually become a mess to understand and explain to others.
Irregardless, you should also be certain that people will always find ways to bypass it if wanted. Python doesn't really tie your hands down, if you want to do something silly it lets you do it; responsible adults, right?
If you need an enforcement, you'd be better off supplying it as a documentation notice. That way if users opt to instantiate directly and trigger unwanted behavior, it's their fault for not following guidelines for your code.

Composing functionality into a method

I have something written in Perl/Moose that uses method modifiers (before/after/around) and I'm wondering what the "Pythonic" way of composing functionality into a method would be?
To expand, I have a data structure that needs to be processed, data is "tagged" to mark it in terms of how it should be processed. I check those tags and then dynamically build my processor accordingly. Using Perl's Moose I can pull in the "traits" I need and then run a single method to process the data. Each "Role" I pull in assumes there's a "process" method and it just bolts it's functionality on with an after 'process' => sub { }. This way if I have a new type of data to deal with I can simply add another Role without having to touch any other code. All this specifically avoids any inheritance and is very simple to manage.
I thought about trying to recreate Moose's "method modifiers" using decorators and/or metaclasses, but I'm wondering what the idiomatic way would be in Python...
Thanks

I hardly know any Perl, but from looking at the first example in the MethodModifier docs, it seems to me context managers in Python would be the thing that most resembles Moose's method modifiers:
class MyContextManager(object):
def __init__(self, data):
self.data = data
def __call__(self):
print 'foo'
def __enter__(self):
print "entering... (data=%s)" % self.data
# this will be bound to the name after the 'as' clause
return self
def __exit__(self, exc_type, exc_value, traceback):
print "exiting... (data=%s)" % self.data
with MyContextManager('bar') as manager:
print "before"
manager()
print "after"
Output:
entering... (data=bar)
before
foo
after
exiting... (data=bar)
Unfortunately, documentation for context managers is a bit spread out:
contextlib docs
Context Manager Types
With Statement Context Managers

Just to expand on #LukasGraf answer pulling in contexts dynamically... (in Python 3)
from contextlib import ExitStack
class Manager1():
def __init__(self, data):
self.data = data
def __enter__(self):
print("entering 1... (data=%s)" % self.data)
return self
def __exit__(self, exc_type, exc_value, traceback):
print("exiting 1... (data=%s)" % self.data)
class Manager2():
def __init__(self, data):
self.data = data
def __enter__(self):
print("entering 2... (data=%s)" % self.data)
return self
def __exit__(self, exc_type, exc_value, traceback):
print("exiting 2... (data=%s)" % self.data
contexts = [Manager2('test'), Manager1('test')]
with ExitStack() as stack:
[ stack.enter_context(context) for context in contexts ]
print('here')
generates...
entering 2... (data=test)
entering 1... (data=test)
here
exiting 1... (data=test)
exiting 2... (data=test)

Encapsulating retries into `with` block

I'm looking to encapsulate logic for database transactions into a with block; wrapping the code in a transaction and handling various exceptions (locking issues). This is simple enough, however I'd like to also have the block encapsulate the retrying of the code block following certain exceptions. I can't see a way to package this up neatly into the context manager.
Is it possible to repeat the code within a with statement?
I'd like to use it as simply as this, which is really neat.
def do_work():
...
# This is ideal!
with transaction(retries=3):
# Atomic DB statements
...
...
I'm currently handling this with a decorator, but I'd prefer to offer the context manager (or in fact both), so I can choose to wrap a few lines of code in the with block instead of an inline function wrapped in a decorator, which is what I do at the moment:
def do_work():
...
# This is not ideal!
#transaction(retries=3)
def _perform_in_transaction():
# Atomic DB statements
...
_perform_in_transaction()
...

Is it possible to repeat the code within a with statement?
No.
As pointed out earlier in that mailing list thread, you can reduce a bit of duplication by making the decorator call the passed function:
def do_work():
...
# This is not ideal!
#transaction(retries=3)
def _perform_in_transaction():
# Atomic DB statements
...
# called implicitly
...

The way that occurs to me to do this is just to implement a standard database transaction context manager, but allow it to take a retries argument in the constructor. Then I'd just wrap that up in your method implementations. Something like this:
class transaction(object):
def __init__(self, retries=0):
self.retries = retries
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, traceback):
pass
# Implementation...
def execute(self, query):
err = None
for _ in range(self.retries):
try:
return self._cursor.execute(query)
except Exception as e:
err = e # probably ought to save all errors, but hey
raise err
with transaction(retries=3) as cursor:
cursor.execute('BLAH')

As decorators are just functions themselves, you could do the following:
with transaction(_perform_in_transaction, retries=3) as _perf:
_perf()
For the details, you'd need to implement transaction() as a factory method that returns an object with __callable__() set to call the original method and repeat it up to retries number of times on failure; __enter__() and __exit__() would be defined as normal for database transaction context managers.
You could alternatively set up transaction() such that it itself executes the passed method up to retries number of times, which would probably require about the same amount of work as implementing the context manager but would mean actual usage would be reduced to just transaction(_perform_in_transaction, retries=3) (which is, in fact, equivalent to the decorator example delnan provided).

While I agree it can't be done with a context manager... it can be done with two context managers!
The result is a little awkward, and I am not sure whether I approve of my own code yet, but this is what it looks like as the client:
with RetryManager(retries=3) as rm:
while rm:
with rm.protect:
print("Attempt #%d of %d" % (rm.attempt_count, rm.max_retries))
# Atomic DB statements
There is an explicit while loop still, and not one, but two, with statements, which leaves a little too much opportunity for mistakes for my liking.
Here's the code:
class RetryManager(object):
""" Context manager that counts attempts to run statements without
exceptions being raised.
- returns True when there should be more attempts
"""
class _RetryProtector(object):
""" Context manager that only raises exceptions if its parent
RetryManager has given up."""
def __init__(self, retry_manager):
self._retry_manager = retry_manager
def __enter__(self):
self._retry_manager._note_try()
return self
def __exit__(self, exc_type, exc_val, traceback):
if exc_type is None:
self._retry_manager._note_success()
else:
# This would be a good place to implement sleep between
# retries.
pass
# Suppress exception if the retry manager is still alive.
return self._retry_manager.is_still_trying()
def __init__(self, retries=1):
self.max_retries = retries
self.attempt_count = 0 # Note: 1-based.
self._success = False
self.protect = RetryManager._RetryProtector(self)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, traceback):
pass
def _note_try(self):
self.attempt_count += 1
def _note_success(self):
self._success = True
def is_still_trying(self):
return not self._success and self.attempt_count < self.max_retries
def __bool__(self):
return self.is_still_trying()
Bonus: I know you don't want to separate your work off into separate functions wrapped with decorators... but if you were happy with that, the redo package from Mozilla offers the decorators to do that, so you don't have to roll your own. There is even a Context Manager that effective acts as temporary decorator for your function, but it still relies on your retrievable code to be factored out into a single function.

This question is a few years old but after reading the answers I decided to give this a shot.
This solution requires the use of a "helper" class, but I I think it does provide an interface with retries configured through a context manager.
class Client:
def _request(self):
# do request stuff
print("tried")
raise Exception()
def request(self):
retry = getattr(self, "_retry", None)
if not retry:
return self._request()
else:
for n in range(retry.tries):
try:
return self._request()
except Exception:
retry.attempts += 1
class Retry:
def __init__(self, client, tries=1):
self.client = client
self.tries = tries
self.attempts = 0
def __enter__(self):
self.client._retry = self
def __exit__(self, *exc):
print(f"Tried {self.attempts} times")
del self.client._retry
>>> client = Client()
>>> with Retry(client, tries=3):
... # will try 3 times
... response = client.request()
tried once
tried once
tried once
Tried 3 times