Is it possible to ensure the __exit__() method is called even if there is an exception in __enter__()?
>>> class TstContx(object):
... def __enter__(self):
... raise Exception('Oops in __enter__')
...
... def __exit__(self, e_typ, e_val, trcbak):
... print "This isn't running"
...
>>> with TstContx():
... pass
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in __enter__
Exception: Oops in __enter__
>>>
Edit
This is as close as I could get...
class TstContx(object):
def __enter__(self):
try:
# __enter__ code
except Exception as e
self.init_exc = e
return self
def __exit__(self, e_typ, e_val, trcbak):
if all((e_typ, e_val, trcbak)):
raise e_typ, e_val, trcbak
# __exit__ code
with TstContx() as tc:
if hasattr(tc, 'init_exc'): raise tc.init_exc
# code in context
In hind sight, a context manager might have not been the best design decision
Like this:
import sys
class Context(object):
def __enter__(self):
try:
raise Exception("Oops in __enter__")
except:
# Swallow exception if __exit__ returns a True value
if self.__exit__(*sys.exc_info()):
pass
else:
raise
def __exit__(self, e_typ, e_val, trcbak):
print "Now it's running"
with Context():
pass
To let the program continue on its merry way without executing the context block you need to inspect the context object inside the context block and only do the important stuff if __enter__ succeeded.
class Context(object):
def __init__(self):
self.enter_ok = True
def __enter__(self):
try:
raise Exception("Oops in __enter__")
except:
if self.__exit__(*sys.exc_info()):
self.enter_ok = False
else:
raise
return self
def __exit__(self, e_typ, e_val, trcbak):
print "Now this runs twice"
return True
with Context() as c:
if c.enter_ok:
print "Only runs if enter succeeded"
print "Execution continues"
As far as I can determine, you can't skip the with-block entirely. And note that this context now swallows all exceptions in it. If you wish not to swallow exceptions if __enter__ succeeds, check self.enter_ok in __exit__ and return False if it's True.
No. If there is the chance that an exception could occur in __enter__() then you will need to catch it yourself and call a helper function that contains the cleanup code.
I suggest you follow RAII (resource acquisition is initialization) and use the constructor of your context to do the potentially failing allocation. Then your __enter__ can simply return self which should never ever raise an exception. If your constructor fails, the exception may be thrown before even entering the with context.
class Foo:
def __init__(self):
print("init")
raise Exception("booh")
def __enter__(self):
print("enter")
return self
def __exit__(self, exc_type, exc_val, exc_tb):
print("exit")
return False
with Foo() as f:
print("within with")
Output:
init
Traceback (most recent call last):
File "<input>", line 1, in <module>
...
raise Exception("booh")
Exception: booh
Edit:
Unfortunately this approach still allows the user to create "dangling" resources that wont be cleaned up if he does something like:
foo = Foo() # this allocates resource without a with context.
raise ValueError("bla") # foo.__exit__() will never be called.
I am quite curious if this could be worked around by modifying the new implementation of the class or some other python magic that forbids object instantiation without a with context.
You could use contextlib.ExitStack (not tested):
with ExitStack() as stack:
cm = TstContx()
stack.push(cm) # ensure __exit__ is called
with ctx:
stack.pop_all() # __enter__ succeeded, don't call __exit__ callback
Or an example from the docs:
stack = ExitStack()
try:
x = stack.enter_context(cm)
except Exception:
# handle __enter__ exception
else:
with stack:
# Handle normal case
See contextlib2 on Python <3.3.
if inheritance or complex subroutines are not required, you can use a shorter way:
from contextlib import contextmanager
#contextmanager
def test_cm():
try:
# dangerous code
yield
except Exception, err
pass # do something
class MyContext:
def __enter__(self):
try:
pass
# exception-raising code
except Exception as e:
self.__exit__(e)
def __exit__(self, *args):
# clean up code ...
if args[0]:
raise
I've done it like this. It calls __exit__() with the error as the argument. If args[0] contains an error it reraises the exception after executing the clean up code.
The docs contain an example that uses contextlib.ExitStack for ensuring the cleanup:
As noted in the documentation of ExitStack.push(), this method can be useful in cleaning up an already allocated resource if later steps in the __enter__() implementation fail.
So you would use ExitStack() as a wrapping context manager around the TstContx() context manager:
from contextlib import ExitStack
with ExitStack() as stack:
ctx = TstContx()
stack.push(ctx) # Leaving `stack` now ensures that `ctx.__exit__` gets called.
with ctx:
stack.pop_all() # Since `ctx.__enter__` didn't raise it can handle the cleanup itself.
... # Here goes the body of the actual context manager.
Related
I'm making a logging module in python which reports every exception that happens in run-time to a server, so in every function I have to write:
def a_func():
try:
#stuff here
pass
except:
Logger.writeError(self.__class__.__name__, inspect.stack()[1][3],\
tracer(self, vars()))
As you can see I'm using vars() function to get the variables which caused the exception. I read about decorators and I decided to use them:
def flog(func):
def flog_wrapper(*args, **kwargs):
try:
func(*args, **kwargs)
except Exception as e:
print "At flog:", e
#self.myLogger.writeError(self.__class__.__name__, inspect.stack()[1][3], tracer(self, vars()))
return flog_wrapper
The problem is I don't have access to the original function's (func) variables (vars()) here. Is there a way to access them in the decorator function?
You don't need to use vars(). The traceback of an exception has everything you need:
import sys
def flog(func):
def flog_wrapper(self, *args, **kwargs):
try:
return func(self, *args, **kwargs)
except Exception:
exc_type, exc_value, tb = sys.exc_info()
print "At flog:", exc_value
locals = tb.tb_frame.f_locals
self.myLogger.writeError(type(self).__name__, inspect.stack()[1][3], tracer(self, locals))
del tb
return flog_wrapper
The traceback contains a chained series of execution frames; each frame has a reference to the locals used in that frame.
You do very much want to clean up the reference to the traceback; because the traceback includes the wrapper function frame, you have a circular reference and that is best broken early.
Trying to understanding how context managers work to catch errors, but more specifically the role of the __enter__() method in a class created to be used as a context manager, how it works in the 'error catching' process here, and why it is that self is all that's returned in the __enter__() method.
Given the following use of a context manager to catch an error:
import unittest
class InvoiceCalculatorTests(unittest.TestCase):
def test_no_pay(self):
with self.assertRaises(ValueError):
pay = divide_pay(0, {"Alice": 3.0, "Bob": 3.0, "Carol": 6.0})
Here is what I believe is the source code for assertRaises:
class _AssertRaisesContext(_AssertRaisesBaseContext):
"""A context manager used to implement TestCase.assertRaises* methods."""
_base_type = BaseException
_base_type_str = 'an exception type or tuple of exception types'
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, tb):
if exc_type is None:
try:
exc_name = self.expected.__name__
except AttributeError:
exc_name = str(self.expected)
if self.obj_name:
self._raiseFailure("{} not raised by {}".format(exc_name,
self.obj_name))
else:
self._raiseFailure("{} not raised".format(exc_name))
else:
traceback.clear_frames(tb)
if not issubclass(exc_type, self.expected):
# let unexpected exceptions pass through
return False
# store exception, without traceback, for later retrieval
self.exception = exc_value.with_traceback(None)
if self.expected_regex is None:
return True
expected_regex = self.expected_regex
if not expected_regex.search(str(exc_value)):
self._raiseFailure('"{}" does not match "{}"'.format(
expected_regex.pattern, str(exc_value)))
return True
I've tried going through PEP-0343 to gain some insight, but it's a bit beyond my current knowledge/understanding to make sense of what's contained therein. Could someone explain, in relative layman's terms, the role of __enter__() and __exit__() in the process of 'catching' the ValueError here and why it is that __enter__() is just returning self?
__enter__() is for setup, This particular context manager doesn't require any setup. So all it has to do is return object to be specified in the as clause, which is just itself.
I think I've read that exceptions inside a with do not allow __exit__ to be call correctly. If I am wrong on this note, pardon my ignorance.
So I have some pseudo code here, my goal is to use a lock context that upon __enter__ logs a start datetime and returns a lock id, and upon __exit__ records an end datetime and releases the lock:
def main():
raise Exception
with cron.lock() as lockid:
print('Got lock: %i' % lockid)
main()
How can I still raise errors in addition to existing the context safely?
Note: I intentionally raise the base exception in this pseudo-code as I want to exit safely upon any exception, not just expected exceptions.
Note: Alternative/standard concurrency prevention methods are irrelevant, I want to apply this knowledge to any general context management. I do not know if different contexts have different quirks.
PS. Is the finally block relevant?
The __exit__ method is called as normal if the context manager is broken by an exception. In fact, the parameters passed to __exit__ all have to do with handling this case! From the docs:
object.__exit__(self, exc_type, exc_value, traceback)
Exit the runtime context related to this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be None.
If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.
Note that __exit__() methods should not reraise the passed-in exception; this is the caller’s responsibility.
So you can see that the __exit__ method will be executed and then, by default, any exception will be re-raised after exiting the context manager. You can test this yourself by creating a simple context manager and breaking it with an exception:
DummyContextManager(object):
def __enter__(self):
print('Entering...')
def __exit__(self, exc_type, exc_value, traceback):
print('Exiting...')
# If we returned True here, any exception would be suppressed!
with DummyContextManager() as foo:
raise Exception()
When you run this code, you should see everything you want (might be out of order since print tends to end up in the middle of tracebacks):
Entering...
Exiting...
Traceback (most recent call last):
File "C:\foo.py", line 8, in <module>
raise Exception()
Exception
The best practice when using #contextlib.contextmanager was not quite clear to me from the above answer. I followed the link in the comment from #BenUsman.
If you are writing a context manager you must wrap the yield in try-finally block:
from contextlib import contextmanager
#contextmanager
def managed_resource(*args, **kwds):
# Code to acquire resource, e.g.:
resource = acquire_resource(*args, **kwds)
try:
yield resource
finally:
# Code to release resource, e.g.:
release_resource(resource)
>>> with managed_resource(timeout=3600) as resource:
... # Resource is released at the end of this block,
... # even if code in the block raises an exception
I encountered a strange behaviour in Python's with-statement recently. I have a code which uses Python's context managers to rollback configuration changes in __exit__ method. The manager had a return False value in a finally block in __exit__. I've isolated the case in following code - the only difference is with the indent of return statement:
class Manager1(object):
def release(self):
pass # Implementation not important
def rollback(self):
# Rollback fails throwing an exception:
raise Exception("A failure")
def __enter__(self):
print "ENTER1"
def __exit__(self, exc_type, exc_val, exc_tb):
print "EXIT1"
try:
self.rollback()
finally:
self.release()
return False # The only difference here!
class Manager2(object):
def release(self):
pass # Implementation not important
def rollback(self):
# Rollback fails throwing an exception:
raise Exception("A failure")
def __enter__(self):
print "ENTER2"
def __exit__(self, exc_type, exc_val, exc_tb):
print "EXIT2"
try:
self.rollback()
finally:
self.release()
return False # The only difference here!
In the code above the rollback fails of with an Exception. My question is, why Manager1 is behaving differently than Manager2. The exception is not thrown outside of with-statement in Manager1 and why it IS thrown on exit in Manager2.
with Manager1() as m:
pass # The Exception is NOT thrown on exit here
with Manager2() as m:
pass # The Exception IS thrown on exit here
According to documentation of __exit__:
If an exception is supplied, and the method wishes to suppress the
exception (i.e., prevent it from being propagated), it should return a
true value. Otherwise, the exception will be processed normally upon
exit from this method.
In my opinion in both cases the exit is not returning True, thus the exception should not be supressed in both cases. However in Manager1 it is. Can anyone explain that?
I use Python 2.7.6.
If the finally clause is activated that means that either the try block has successfully completed, or it raised an error that has been processed, or that the try block executed a return.
In Manager1 the execution of the return statement as part of the finally clause makes it terminate normally, returning False. In your Manager2 class the finally clause still executes, but if it was executed as a result of an exception being raised it does nothing to stop that exception propagating back up the call chain until caught (or until it terminates you program with a traceback).
Manager2.__exit__() will only return False if no exception is raised.
I think a good way to understand this is by looking at a separate example that is independent of all the context manager stuff:
>>> def test ():
try:
print('Before raise')
raise Exception()
print('After raise')
finally:
print('In finally')
print('Outside of try/finally')
>>> test()
Before raise
In finally
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
test()
File "<pyshell#6>", line 4, in test
raise Exception()
Exception
So you can see that when an exception is thrown within the try block, any code before the exception is executed and any code inside the finally block is executed. Apart from that, everything else is skipped. That is because the exception that is being thrown ends the function invocation. But because the exception is thrown within a try block, the respective finally block has a final chance to run.
Now, if you comment out the raise line in the function, you will see that all code is executed, since the function does not end prematurely.
I use a with statement with the following class.
def __init__(self):
...
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
print "EXIT Shutting the SDK down"
ret = self.sdkobject.ShutDown()
self.error_check(ret)
This catches any error that occur when I am using the object of the class and safely shuts down the SDK that I am using. However, it catch problems when the class is still initializing. I have recently found the "del" function which neatly solves this problem. However, it can't be used in conjunction with the exit function (as the with statement evokes the exit and the del gets an exception). How can I set up a destructor using a with statemtent, which will catch failures even during initialization?
Exceptions in the __init__ need to be dealt with directly in that method:
class YourContextManager(object):
sdkobject = None
def __init__(self):
try:
self._create_sdk_object()
except Exception:
if self.sdkobject is not None:
self.sdkobject.ShutDown()
raise
def _create_sdk_object(self):
self.sdkobject = SomeSDKObject()
self.sdkobject.do_something_that_could_raise_an_exception()
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
print "EXIT Shutting the SDK down"
ret = self.sdkobject.ShutDown()
self.error_check(ret)
Note that the exception is re-raised; you want to give the consumer of the context manager an opportunity to handle the failure to create a context manager.
Create a separate shutdown function that gets called in the try/except block of the __init__ and wherever else you need a proper shutdown.
Catch the exception in __init__ and handle it. __del__ is unnecessary.