We Keep Coding

How do I use GeneratorExit?

I have the following mcve:
import logging
class MyGenIt(object):
def __init__(self, name, content):
self.name = name
self.content = content
def __iter__(self):
with self:
for o in self.content:
yield o
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
if exc_type:
logging.error("Aborted %s", self,
exc_info=(exc_type, exc_value, traceback))
And here is sample use:
for x in MyGenIt("foo",range(10)):
if x == 5:
raise ValueError("got 5")
I would like logging.error to report the ValueError, but instead it reports GeneratorExit:
ERROR:root:Aborted <__main__.MyGenIt object at 0x10ca8e350>
Traceback (most recent call last):
File "<stdin>", line 8, in __iter__
GeneratorExit
When I catch GeneratorExit in __iter__:
def __iter__(self):
with self:
try:
for o in self.content:
yield o
except GeneratorExit:
return
nothing is logged (of course) because __exit__ is called with exc_type=None.
Why do I see GeneratorExit instead of ValueError in __exit__?
What do I do to get the desired behavior, i.e., ValueError in __exit__?

Just a quick note that you could "bring the context manager out" of the generator, and by only changing 3 lines get:
import logging
class MyGenIt(object):
def __init__(self, name, content):
self.name = name
self.content = content
def __iter__(self):
for o in self.content:
yield o
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
if exc_type:
logging.error("Aborted %s", self,
exc_info=(exc_type, exc_value, traceback))
with MyGenIt("foo", range(10)) as gen:
for x in gen:
if x == 5:
raise ValueError("got 5")
A context manager that could also act as an iterator -- and would catch caller code exceptions like your ValueError.

The basic problem is that you are trying to use a with statement inside the generator to catch an exception that is raised outside the generator. You cannot get __iter__ to see the ValueError, because __iter__ is not executing at the time the ValueError is raised.
The GeneratorExit exception is raised when the generator itself is deleted, which happens when it is garbage collected. As soon as the exception occurs, the for loop terminates; since the only reference to the generator (the object obtained by calling __iter__) is in the loop expression, terminating the loop removes the only reference to the iterator and makes it available for garbage collection. It appears that here it is being garbage collected immediately, meaning that the GeneratorExit exception happens between the raising of the ValueError and the propagation of that ValueError to the enclosing code. The GeneratorExit is normally handled totally internally; you are only seeing it because your with statement is inside the generator itself.
In other words, the flow goes something like this:
Exception is raised outside the generator
for loop exits
Generator is now available for garbage collection
Generator is garbage collected
Generator's .close() is called
GeneratorExit is raised inside the generator
ValueError propagates to calling code
The last step does not occur until after your context manager has seen the GeneratorExit. When I run your code, I see the ValueError raised after the log message is printed.
You can see that the garbage collection is at work, because if you create another reference to the iterator itself, it will keep the iterator alive, so it won't be garbage collected, and so the GeneratorExit won't occur. That is, this "works":
it = iter(MyGenIt("foo",range(10)))
for x in it:
if x == 5:
raise ValueError("got 5")
The result is that the ValueError propagates and is visible; no GeneratorExit occurs and nothing is logged. You seem to think that the GeneratorExit is somehow "masking" your ValueError, but it isn't really; it's just an artifact introduced by not keeping any other references to the iterator. The fact that GeneratorExit occurs immediately in your example isn't even guaranteed behavior; it's possible that the iterator might not be garbage-collected until some unknown time in the future, and the GeneratorExit would then be logged at that time.
Turning to your larger question of "why do I see GeneratorExit", the answer is that that is the only exception that actually occurs within the generator function. The ValueError occurs outside the generator, so the generator can't catch it. This means your code can't really work in the way you seem to intend it to. Your with statement is inside the generator function. Thus it can only catch exceptions that happen in the process of yielding items from the generator; there generator has no knowledge of what happens between the times when it advances. But your ValueError is raised in the body of the loop over the generator contents. The generator is not executing at this time; it's just sitting there suspended.
You can't use a with statement in a generator to magically trap exceptions that occur in the code that iterates over the generator. The generator does not "know" about the code that iterates over it and can't handle exceptions that occur there. If you want to catch exceptions within the loop body, you need a separate with statement enclosing the loop itself.

The GeneratorExit is raised whenever a generator or coroutine is closed. Even without the context manager, we can replicate the exact condition with a simple generator function that prints out the exception information when it errors (further reducing the provided code to show exactly how and where that exception is generated).
import sys
def dummy_gen():
for idx in range(5):
try:
yield idx
except:
print(sys.exc_info())
raise
for i in dummy_gen():
raise ValueError('foo')
Usage:
(<class 'GeneratorExit'>, GeneratorExit(), <traceback object at 0x7f96b26b4cc8>)
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: foo
Note there was also an exception that was raised inside the generator itself, as noted that the except block was executed. Note that the exception was also further raise'd after the print statement but note how that isn't actually shown anywhere, because it is handled internally.
We can also abuse this fact to see if we can manipulate the flow by swallowing the GeneratorExit exception and see what happens. This can be done by removing the raise statement inside the dummy_gen function to get the following output:
(<class 'GeneratorExit'>, GeneratorExit(), <traceback object at 0x7fd1f0438dc8>)
Exception ignored in: <generator object dummy_gen at 0x7fd1f0436518>
RuntimeError: generator ignored GeneratorExit
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: foo
Note how there is an internal RuntimeError that was raised that complained about the generator ignoring the GeneratorExit function. So we from this we can clearly see that this exception is produced by the generator itself inside the generator function, and the ValueError that is raised outside that scope is never present inside the generator function.
Since a context manager will trap all exceptions as is, and the context manager is inside the generator function, whatever exception raised inside it will simply be passed to __exit__ as is. Consider the following:
class Context(object):
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
if exc_type:
logging.error("Aborted %s", self,
exc_info=(exc_type, exc_value, traceback))
Modify the dummy_gen to the following:
def dummy_gen():
with Context():
for idx in range(5):
try:
yield idx
except:
print(sys.exc_info())
raise
Running the resulting code:
(<class 'GeneratorExit'>, GeneratorExit(), <traceback object at 0x7f44b8fb8908>)
ERROR:root:Aborted <__main__.Context object at 0x7f44b9032d30>
Traceback (most recent call last):
File "foo.py", line 26, in dummy_gen
yield idx
GeneratorExit
Traceback (most recent call last):
File "foo.py", line 41, in <module>
raise ValueError('foo')
ValueError: foo
The same GeneratorExit that is raised is now presented to the context manager, because this is the behavior that was defined.

Unable to pass assertRaises test in Python

So, I have the most trivial in the world example. This is my class to be tested:
# My_Class.py
class My_Class(object):
#staticmethod
def doit(name, params):
try:
raise Exception("This is my error message")
except Exception:
print("Exception: I raised Exception")
And this is the tester itself:
# test.py
import unittest
from My_Class import My_Class
class Test_MyClass(unittest.TestCase):
def setUp(self):
self.my_class = My_Class()
def test_my_class(self):
name = "Abrakadabra"
params = {}
self.assertRaises(Exception, self.my_class.doit, name, params)
And this is what I see in the console, when I'm running my test.py:
$ nosetests test.py
F
======================================================================
FAIL: test_my_class (test.Test_MyClass)
----------------------------------------------------------------------
Traceback (most recent call last):
File ....
nose.proxy.AssertionError: Exception not raised by doit
-------------------- >> begin captured stdout << ---------------------
Exception: I raised Exception
--------------------- >> end captured stdout << ----------------------
----------------------------------------------------------------------
Ran 1 test in 0.001s
FAILED (failures=1)
It is reaaly iteresting, because it is controversial. On the one hand the test says that "Exception not raised by doit", but one line below it clearly prints out a message from the Exception block. So, what I'm doing wrong here??? Thanks!

To directly answer your question, the reason why you are getting that message is because with this assertion:
self.assertRaises(Exception, self.my_class.doit, name, params)
You are testing to make sure an exception was raised. But your try/except suppresses this. If you actually remove your try/except your test will in fact pass, because now your method will raise.
Since you do not want to do this, what you should be doing instead is testing the behaviour of your method when an exception is raised. Ultimately, you want to make sure that your print method is called in your except. I have put together an example below to help understand this.
Keeping in mind what #user2357112 mentioned, which is very important to keep in mind when unittesting, here is an example to help expand on that to provide a practical use for what you are trying to do:
Let us just put together some method:
def some_method():
pass
We will now put this in to your staticmethod you defined as such:
# My_Class.py
class My_Class(object):
#staticmethod
def doit(name, params):
try:
some_method()
except Exception:
print("Exception: I raised Exception")
So now, when it comes to your unittesting, you want to test the behaviour of your method doit. With that in mind, what you will do in this case, is test that some_method will raise an exception and you will validate how your doit method behaves to that exception being raised.
At this point, I suggest taking a look at the documentation behind unittest and mock to get more familiar with what you can do with your testing, but here is an example using mock patching to test the behaviour of your code if an exception is being raised:
#patch('builtins.print')
#patch('__main__.some_method')
def test_my_class(self, m_some_method, m_print):
name = "Abrakadabra"
params = {}
# have the side_effect raise the exception when some_method is called in doit
m_some_method.side_effect = Exception()
self.my_class.doit(name, params)
# check to make sure you caught the exception by checking print was called
self.assertEqual(m_print.call_count, 1)
When you put it all together, the following is functional code that I ran on my end that you can play around with to understand what is happening:
def some_method():
pass
# My_Class.py
class My_Class(object):
#staticmethod
def doit(name, params):
try:
some_method()
except Exception:
print("Exception: I raised Exception")
# test.py
import unittest
from mock import patch
class Test_MyClass(unittest.TestCase):
def setUp(self):
self.my_class = My_Class()
#patch('builtins.print')
#patch('__main__.some_method')
def test_my_class(self, m_some_method, m_print):
name = "Abrakadabra"
params = {}
m_some_method.side_effect = Exception()
self.my_class.doit(name, params)
self.assertEqual(m_print.call_count, 1)
if __name__ == '__main__':
unittest.main()

assertRaises is an assertion about the function's visible behavior, not its internals. It asserts that the stated exception passes out of the function. Any exceptions that are handled inside the function are not assertRaises's concern.

assertRaises failed since there was actually no exception raised. Well, it was raised but handled with except inside the doit() method. The problem is here:
try:
raise Exception("This is my error message")
except Exception:
print("Exception: I raised Exception")
You are raising an exception and then catching it without re-raising. From a caller (assertRaises is the caller in your case) perspective, no errors were thrown during the function call. Re-raising an exception allows a caller to handle an exception as well. Put a raise after the print:
try:
raise Exception("This is my error message")
except Exception:
print("Exception: I raised Exception")
raise # re-raising
Also see Handling Exceptions.

Return in finally block in python context manager

I encountered a strange behaviour in Python's with-statement recently. I have a code which uses Python's context managers to rollback configuration changes in __exit__ method. The manager had a return False value in a finally block in __exit__. I've isolated the case in following code - the only difference is with the indent of return statement:
class Manager1(object):
def release(self):
pass # Implementation not important
def rollback(self):
# Rollback fails throwing an exception:
raise Exception("A failure")
def __enter__(self):
print "ENTER1"
def __exit__(self, exc_type, exc_val, exc_tb):
print "EXIT1"
try:
self.rollback()
finally:
self.release()
return False # The only difference here!
class Manager2(object):
def release(self):
pass # Implementation not important
def rollback(self):
# Rollback fails throwing an exception:
raise Exception("A failure")
def __enter__(self):
print "ENTER2"
def __exit__(self, exc_type, exc_val, exc_tb):
print "EXIT2"
try:
self.rollback()
finally:
self.release()
return False # The only difference here!
In the code above the rollback fails of with an Exception. My question is, why Manager1 is behaving differently than Manager2. The exception is not thrown outside of with-statement in Manager1 and why it IS thrown on exit in Manager2.
with Manager1() as m:
pass # The Exception is NOT thrown on exit here
with Manager2() as m:
pass # The Exception IS thrown on exit here
According to documentation of __exit__:
If an exception is supplied, and the method wishes to suppress the
exception (i.e., prevent it from being propagated), it should return a
true value. Otherwise, the exception will be processed normally upon
exit from this method.
In my opinion in both cases the exit is not returning True, thus the exception should not be supressed in both cases. However in Manager1 it is. Can anyone explain that?
I use Python 2.7.6.

If the finally clause is activated that means that either the try block has successfully completed, or it raised an error that has been processed, or that the try block executed a return.
In Manager1 the execution of the return statement as part of the finally clause makes it terminate normally, returning False. In your Manager2 class the finally clause still executes, but if it was executed as a result of an exception being raised it does nothing to stop that exception propagating back up the call chain until caught (or until it terminates you program with a traceback).
Manager2.__exit__() will only return False if no exception is raised.

I think a good way to understand this is by looking at a separate example that is independent of all the context manager stuff:
>>> def test ():
try:
print('Before raise')
raise Exception()
print('After raise')
finally:
print('In finally')
print('Outside of try/finally')
>>> test()
Before raise
In finally
Traceback (most recent call last):
File "<pyshell#7>", line 1, in <module>
test()
File "<pyshell#6>", line 4, in test
raise Exception()
Exception
So you can see that when an exception is thrown within the try block, any code before the exception is executed and any code inside the finally block is executed. Apart from that, everything else is skipped. That is because the exception that is being thrown ends the function invocation. But because the exception is thrown within a try block, the respective finally block has a final chance to run.
Now, if you comment out the raise line in the function, you will see that all code is executed, since the function does not end prematurely.

Invalid argument raise exception

How do I test my parameter if it will raise an exception without actually raising it, using try and except?
class MyClass:
def function(parameter):
pass
parameter is an ambiguous function that may raise 1 or more of any exception, for example:
parameter = pow("5", 5)
A TypeError is raised as soon as the function is called and before the function can execute its statements.

In a comment to another answer you said: "parameter is another function; take for example: parameter = pow("5", 5) which raises a TypeError, but it could be any type of function and any type of exception."
If you want to catch the exeption inside your function you have to call the paramenter (which I'm assuming is callable) inside that function:
def function(callable, args=()):
try:
callable(*args)
except:
print('Ops!')
Example:
>>> function(pow, args=("5", 5))
Ops!
This is if you really need to call your "paramenter" inside the function. Otherwise your should manage its behaviour outside, maybe with something like:
>>> try:
... param = pow('5', 5)
... except:
... param = 10
...
>>> param
10
>>> function(param)
In this example, to raise an exception is pow not function, so it's a good practice to separate the the two different call, and wrap with a try-except statement the code that might fail.

From what I can understand, you want to handle the exceptions raised and also inspect what sort of errors were raised for further inspection? Here is one way of doing it.
class Foo(object):
def find_errors(arg):
errors = []
try:
# do something
except TypeError as e:
errors.append(e)
# handle exception somehow
except ValueError as e:
errors.append(e)
# handle exception somehow
# and so on ...
finally:
pass #something here
return errors, ans
Now you can inspect errors and find out what exceptions have been raised.

If you expect the parameter to be a certain type, you can use type(paramter) is parametertype.
For example, if you wanted to verify that 'i' is an int, run instructions if(type(i) is int):
By edit:
try:
pow("5",5)
return 0
except Exception, err:
sys.stderr.write('ERROR: %s\n' % str(err))
return 1

Perhaps what you mean is how to catch the TypeError exceptions caused by invalid function calls?
Like this:
def foo(bar):
pass
foo(1, 2)
You don't catch them in the function and certainly not in the def foo(bar): line.
It's the caller of the function that made an error so that's where you catch the exception:
try:
foo(1, 2)
except TypeError:
print('call failed')

Python: Can I overload the raise statement with def __raise__(self):?

Here's my exception class that is using raise:
class SCE(Exception):
"""
An error while performing SCE functions.
"""
def __init__(self, value=None):
"""
Message: A string message or an iterable of strings.
"""
if value is None:
self._values = []
elif isinstance(value, str):
self._values = [value]
else:
self._values = list(value)
def __raise__(self):
print('raising')
if not len(self._values):
return
def __str__(self):
return self.__repr__()
def __iter__(self):
return iter(self._values)
def __repr__(self):
return repr(self._values)
Currently if I raise this exception with no value I get traceback followed by:
__main__.SCE: []
Instead of what I expected which was:
raising
>>>
How do you overload raise?

As the other answer says, there is no __raise__ special method. There was a thread in 2004 on comp.lang.python where someone suggested adding such a method, but I don't think there was any followup to that. The only way I can think of to hook exception raising is either by patching the interpreter, or some kind of source or bytecode rewriting that inserts a function call next to the raise operation.

There is no such special method __raise__ (at least none that I have ever heard of or that I can find in the Python documentation).
Why do you want to do this? I can't think of any reason why you want custom code be be executed when the exception is raised (as opposed to either when the exception is constructed, which you can do with the __init__ method, or when the exception is caught, which you can do with an except block). What is your use case for this behavior, and why do you expect that Python supports it?

As others have stated, there is no such private method __raise__. Nothing prevents defining one. For example:
#!/usr/bin/env python3
class MyClass(object):
def __init__(self, raise_exceptions=False):
self.raise_exceptions = raise_exceptions
def __raise__(self, err=None):
print(err, flush=True)
if self.raise_exceptions:
raise err
def run(self):
try:
assert False, 'assertion False'
except Exception as err:
self.__raise__(err)
if __name__ == '__main__':
MyClass(raise_exceptions=False).run()
MyClass(raise_exceptions=True).run()
Here is the output:
$ python3 my_class.py
assertion False
assertion False
Traceback (most recent call last):
File "my_class.py", line 22, in <module>
MyClass(raise_exceptions=True).run()
File "my_class.py", line 17, in run
self.__raise__(err)
File "my_class.py", line 11, in __raise__
raise err
File "my_class.py", line 15, in run
assert False, 'assertion False'
AssertionError: assertion False
Process finished with exit code 1