We Keep Coding

Coroutine that is guaranteed to exit its context managers

I'd like to use a context manager within a coroutine. This coroutine should handle unknown number of steps. However, due to unknown number of steps, it's unclear when should the context manager exit. I'd like it to exit when the co-routine goes out of scope / is garbage collected; however this seems not to happen in the example below:
import contextlib
#contextlib.contextmanager
def cm():
print("STARTED")
yield
print("ENDED")
def coro(a: str):
with cm():
print(a)
while True:
val1, val2 = yield
print(val1, val2)
c = coro("HI")
c.send(None)
print("---")
c.send((1, 2))
print("---!")
Output of this program:
STARTED
HI
---
1 2
---!
The context manager never printed "ENDED".
How can I make a coroutine that will support any number of steps, and be guaranteed to exit gracefully? I don't want to make this a responsibility of the caller.

TLDR: So the issue is that when an exception is raised (and not handled) inside a with block. The __exit__ method of the context manager is called with that exception. For contextmanager-decorated generators, this causes the exception to be thrown to the generator. cm does not handle this exception and thus the cleanup code is not run. When coro is garbage collected, its close method is called which throws a GeneratorExit to coro (which then gets thrown to cm). What follows is a detailed description of the above steps.
The close method throws a GeneratorExit to coro which means a GeneratorExit is raised at the point of yield. coro doesn't handle the GeneratorExit so it exits the context via an error. This causes the __exit__ method of the context to be called with an error and error information. What does the __exit__ method from a contextmanager-decorated generator do? If it is called with an exception, it throws that exception to the underlying generator.
At this point the a GeneratorExit is raised from the yield statement in the body of our context manager. That unhandled exception causes the cleanup code to not be run. That unhandled exception is raised by context manager and is passed back to the __exit__ of the contextmanager decorator. Being the same error that was thrown, __exit__ returns False to indicate the original error sent to __exit__ was unhandled.
Finally, this continues the GeneratorExit's propagation outside of the with block inside coro where it continues to be unhandled. However, not handling GeneratorExits is regular for generators, so the original close method suppresses the GeneratorExit.
See this part of the yield documentation:
If the generator is not resumed before it is finalized (by reaching a zero reference count or by being garbage collected), the generator-iterator’s close() method will be called, allowing any pending finally clauses to execute.
Looking at the close documentation we see:
Raises a GeneratorExit at the point where the generator function was paused. If the generator function then exits gracefully, is already closed, or raises GeneratorExit (by not catching the exception), close returns to its caller.
This part of the with statement documentation:
The suite is executed.
The context manager’s exit() method is invoked. If an exception caused the suite to be exited, its type, value, and traceback are passed as arguments to exit(). Otherwise, three None arguments are supplied.
And the code of the __exit__ method for the contextmanager decorator.
So with all this context (rim-shot), the easiest way we can get the desired behavior is with a try-except-finally in the definition of our context manager. This is the suggested method from the contextlib docs. And all their examples follow this form.
Thus, you can use a try…except…finally statement to trap the error (if any), or ensure that some cleanup takes place.
import contextlib
#contextlib.contextmanager
def cm():
try:
print("STARTED")
yield
except Exception:
raise
finally:
print("ENDED")
def coro(a: str):
with cm():
print(a)
while True:
val1, val2 = yield
print(val1, val2)
c = coro("HI")
c.send(None)
print("---")
c.send((1, 2))
print("---!")
The output is now:
STARTED
HI
---
1 2
---!
ENDED
as desired.
We could also define our context manager in the traditional manner: as a class with an __enter__ and __exit__ method and still gotten the correct behavior:
class CM:
def __enter__(self):
print('STARTED')
def __exit__(self, exc_type, exc_value, traceback):
print('ENDED')
return False
The situation is somewhat simpler, because we can see exactly what the __exit__ method is without having to go to the source code. The GeneratorExit gets sent (as a parameter) to __exit__ where __exit__ happily runs its cleanup code and then returns False. This is not strictly necessary as otherwise None (another Falsey value) would have been returned, but it indicates that any exception that was sent to __exit__ was not handled. (The return value of __exit__ doesn't matter if there was no exception).

You can do it by telling the coroutine to shutdown by sending it something the will cause it to break out of the loop and return as illustrated below. Doing so will cause a StopIteration exception to be raised where this is done, so I added another context manager to allow it to be suppressed. Note I have also added a coroutine decorator to make them start-up automatically when first called, but that part is strictly optional.
import contextlib
from typing import Callable
QUIT = 'quit'
def coroutine(func: Callable):
""" Decorator to make coroutines automatically start when called. """
def start(*args, **kwargs):
cr = func(*args, **kwargs)
next(cr)
return cr
return start
#contextlib.contextmanager
def ignored(*exceptions):
try:
yield
except exceptions:
pass
#contextlib.contextmanager
def cm():
print("STARTED")
yield
print("ENDED")
#coroutine
def coro(a: str):
with cm():
print(a)
while True:
value = (yield)
if value == QUIT:
break
val1, val2 = value
print(val1, val2)
print("---")
with ignored(StopIteration):
c = coro("HI")
#c.send(None) # No longer needed.
c.send((1, 2))
c.send((3, 5))
c.send(QUIT) # Tell coroutine to clean itself up and exit.
print("---!")
Output:
STARTED
HI
---
1 2
3 5
ENDED
---!

How to write a unit test for uncaught exception handler

I have a function to catch uncaught exceptions, below. Is there any way to write a unit test that will execute the uncaught_exception_handler() function, but exit the test normally?
import logging
def config_logger():
# logger setup here
def init_uncaught_exception_logger(logger):
'''Setup an exception handler to log uncaught exceptions.
This is typically called once per main executable.
This function only exists to provide a logger context to the nested function.
Args:
logger (Logger): The logger object to log uncaught exceptions with.
'''
def uncaught_exception_handler(*exc_args):
'''Log uncaught exceptions with logger.
Args:
exc_args: exception type, value, and traceback
'''
print("Triggered uncaught_exception_handler")
logger.error("uncaught: {}: {}\n{}".format(*exc_args))
sys.excepthook = uncaught_exception_handler
if __name__ == '__main__':
LOGGER = config_logger()
init_uncaught_exception_logger(LOGGER)
raise Exception("This is an intentional uncaught exception")

Instead of testing that your function is called for uncaught exceptions, it's probably best to instead test that the excepthook is installed, and that the function does the right thing when you call it manually. That gives you pretty good evidence that the excepthook will behave properly in real usage. You'll want to move your uncaught_exception_handler outside of init_uncaught_exception_logger so your tests can access it more easily.
assert sys.excepthook is uncaught_exception_handler
with your_preferred_output_capture_mechanism:
try:
1/0
except ZeroDivisionError:
uncaught_exception_handler(*sys.exc_info())
assert_something_about_captured_output()
If you want to actually invoke excepthook through an uncaught exception, then you'll need to launch a subprocess and examine its output. The subprocess module is the way to go for that.

In order to write assertions about raised exceptions, you can use pytest.raises as a context manager like this:
with raises(expected_exception: Exception[, match][, message])
import pytest
def test_which_will_raise_exception():
with pytest.raises(Exception):
# Your function to test.
Now, this unit test will pass only if any code under pytest.raises context manager will raise an exception provided as a parameter. In this case, it's Exception.

re-raise exception from method as argument

I have a method that needs some wrapping called joule, so I wrap that joule method inside a wrapper called respond (which you will see shortly):
someclass.respond(somemodule.joule(someArgument, someStrategy), 202)
I have a wrapper called respond:
#classmethod
def respond(cls, method, successStatus):
try:
method, successStatus
except Exception as e:
return {
'status': 500,
'message': str(e)
}
The actual method that gets called and raises an Exception:
def joule(params, strategy):
try:
return strategy(params)
finally:
session.rollback()
conn.execute('UNLOCK TABLES')
For some reason, the re-raised exception does not seem to get caught in the respond wrapper! Can you folks help me understand what am I doing incorrectly here?
If this helps, the exception being thrown by sqlalchemy is (please note that this is a scenario being forcibly created to handle the exception correctly):
ProgrammingError: (ProgrammingError) (1146, u"Table 'matrix.vmop_queue' doesn't exist") 'LOCK TABLES vmop_queue WRITE' ()

You are misunderstanding how exception handling works. Exception handling operates on the stack frame of called functions.
In the example you give, someclass.respond does not actually invoke somemodule.joule, instead wherever the line that you have written in your example, which is some outer context is the place that receives the uncaught exception. Thus someclass.respond can't possibly handle the exception thrown by somemodule.joule.
There are other ways to achieve what you are trying to accomplish, but I would need more context in order to give you a better suggestion.
To make this a bit more concrete, let's say that foo contains the example line you gave:
def foo():
someclass.respond(somemodule.joule(someArgument, someStrategy), 202)
You could add the try block to foo to handle the exception thrown by somemodule.joule. This would look like this:
def foo():
try:
someclass.respond(somemodule.joule(someArgument, someStrategy), 202)
except Exception as e:
pass # do something with the exception here
Alternatively, if the whole purpose for someclass.respond is to handle this exception, then you should move the invocation of somemodule.joule inside of someclass.respond. You could even do this more than one way. You could generically take a function and its arguments, and apply that function to the arguments inside of someclass.respond or you could just directly do the invocation inside of someclass.respond.
Let's take the first approach, since you've said that you don't want to repeat the exception handling. I'll call this new method exception_catcher:
def exception_catcher(func, *args):
try:
return func(*args)
except Exception as e:
pass # do whatever you want to the exception
Now the foo context will look like this:
def foo():
exception_catcher(somemodule.joule, someArgument, someStrategy)
Note that exception_catcher is taking somemodule.joule as an argument, and the remaining arguments will be passed to somemodule.joule from within exception_catcher.

How to safely handle an exception inside a context manager

I think I've read that exceptions inside a with do not allow __exit__ to be call correctly. If I am wrong on this note, pardon my ignorance.
So I have some pseudo code here, my goal is to use a lock context that upon __enter__ logs a start datetime and returns a lock id, and upon __exit__ records an end datetime and releases the lock:
def main():
raise Exception
with cron.lock() as lockid:
print('Got lock: %i' % lockid)
main()
How can I still raise errors in addition to existing the context safely?
Note: I intentionally raise the base exception in this pseudo-code as I want to exit safely upon any exception, not just expected exceptions.
Note: Alternative/standard concurrency prevention methods are irrelevant, I want to apply this knowledge to any general context management. I do not know if different contexts have different quirks.
PS. Is the finally block relevant?

The __exit__ method is called as normal if the context manager is broken by an exception. In fact, the parameters passed to __exit__ all have to do with handling this case! From the docs:
object.__exit__(self, exc_type, exc_value, traceback)
Exit the runtime context related to this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be None.
If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.
Note that __exit__() methods should not reraise the passed-in exception; this is the caller’s responsibility.
So you can see that the __exit__ method will be executed and then, by default, any exception will be re-raised after exiting the context manager. You can test this yourself by creating a simple context manager and breaking it with an exception:
DummyContextManager(object):
def __enter__(self):
print('Entering...')
def __exit__(self, exc_type, exc_value, traceback):
print('Exiting...')
# If we returned True here, any exception would be suppressed!
with DummyContextManager() as foo:
raise Exception()
When you run this code, you should see everything you want (might be out of order since print tends to end up in the middle of tracebacks):
Entering...
Exiting...
Traceback (most recent call last):
File "C:\foo.py", line 8, in <module>
raise Exception()
Exception

The best practice when using #contextlib.contextmanager was not quite clear to me from the above answer. I followed the link in the comment from #BenUsman.
If you are writing a context manager you must wrap the yield in try-finally block:
from contextlib import contextmanager
#contextmanager
def managed_resource(*args, **kwds):
# Code to acquire resource, e.g.:
resource = acquire_resource(*args, **kwds)
try:
yield resource
finally:
# Code to release resource, e.g.:
release_resource(resource)
>>> with managed_resource(timeout=3600) as resource:
... # Resource is released at the end of this block,
... # even if code in the block raises an exception

With statement destructor to catch init exceptions

I use a with statement with the following class.
def __init__(self):
...
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
print "EXIT Shutting the SDK down"
ret = self.sdkobject.ShutDown()
self.error_check(ret)
This catches any error that occur when I am using the object of the class and safely shuts down the SDK that I am using. However, it catch problems when the class is still initializing. I have recently found the "del" function which neatly solves this problem. However, it can't be used in conjunction with the exit function (as the with statement evokes the exit and the del gets an exception). How can I set up a destructor using a with statemtent, which will catch failures even during initialization?

Exceptions in the __init__ need to be dealt with directly in that method:
class YourContextManager(object):
sdkobject = None
def __init__(self):
try:
self._create_sdk_object()
except Exception:
if self.sdkobject is not None:
self.sdkobject.ShutDown()
raise
def _create_sdk_object(self):
self.sdkobject = SomeSDKObject()
self.sdkobject.do_something_that_could_raise_an_exception()
def __enter__(self):
return self
def __exit__(self, type, value, traceback):
print "EXIT Shutting the SDK down"
ret = self.sdkobject.ShutDown()
self.error_check(ret)
Note that the exception is re-raised; you want to give the consumer of the context manager an opportunity to handle the failure to create a context manager.

Create a separate shutdown function that gets called in the try/except block of the __init__ and wherever else you need a proper shutdown.

Catch the exception in __init__ and handle it. __del__ is unnecessary.