Problem
Cannot catch a StopIteration raised from within a iterator.
Example
from typing import (
Generator,
Iterable,
List
)
import sys
import pathlib
from itertools import islice
import numpy as np
def take(n: int, iterable: Iterable) -> List:
taken = list(islice(iterable, 0, n))
if len(taken) > 0:
return taken
else:
print("Raising StopIteration")
raise StopIteration("Nothing to take") # <----- Raise StopIteration
def file_line_stream(path: str) -> Generator[str, None, None]:
if not pathlib.Path(path).is_file():
raise FileNotFoundError(f"file {path} does not exist or non file")
try:
_file = pathlib.Path(path)
with _file.open() as f:
for line in f:
yield line.rstrip()
except (IOError, FileNotFoundError) as e:
raise RuntimeError("Read file failure") from e
def get_sentences(path_to_file, num_sentences):
counter = 0
stream = file_line_stream(path_to_file)
try:
while True:
_lines = take(num_sentences, stream)
print(f"Count {counter}: yielding [{_lines[0]}]\n")
yield np.array(_lines)
counter += 1
finally:
stream.close()
def test():
path_to_input = "test_has_two_lines.txt"
generator = get_sentences(path_to_file=path_to_input, num_sentences=100)
for i in range(10):
# ----------------------------------------------------------------------
# Want to catch the StopIteration raised from within the generator.
# ----------------------------------------------------------------------
try:
next(generator)
except StopIteration as e:
print("Caught at StopIteration except")
generator.close()
except Exception as e: # <--- Raised StopIteration will get caught here
print("Caught at catch-all except: %s", sys.exc_info()[0])
generator.close()
raise e
if __name__ == '__main__':
test()
Result
Count 0: yielding [previous monthly <unk> of the major market index futures and standard & poor 's <unk> index options have produced spectacular volatility]
Raising StopIteration
Caught at catch-all except: %s <class 'RuntimeError'>
Traceback (most recent call last):
File "test_050_word2vec.py", line 39, in get_sentences
_lines = take(num_sentences, stream)
File "test_050_word2vec.py", line 19, in take
raise StopIteration("Nothing to take") # <----- Raise StopIteration
StopIteration: Nothing to take
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test_050_word2vec.py", line 67, in <module>
test()
File "test_050_word2vec.py", line 63, in test
raise e
File "test_050_word2vec.py", line 55, in test
next(generator)
RuntimeError: generator raised StopIteration <----- Converted into RuntimeError
Cause
exception StopIteration
Changed in version 3.7: Enable PEP 479 for all code by default: a StopIteration error raised in a generator is transformed into a RuntimeError.
Reason
PEP 479 -- Change StopIteration handling inside generators
Currently, StopIteration raised accidentally inside a generator function will be interpreted as the end of the iteration by the loop construct driving the generator.
Don't raise StopIteration, raise something sane like ValueError
StopIteration serves a very specific purpose (to allow a next method to indicate iteration is complete), and reusing it for other purposes will cause problems, as you've seen.
The conversion to RuntimeError here is saving you; if Python hadn't done that, the generator would have silently stopped iterating (StopIteration is swallowed silently, causing iteration to end without propagating the exception; you'd never catch it anyway).
Related
I want to save exception in one function, and raise it on upper levels of call stack, but it's not possible. Saving exception with try/except clause doesn't save traceback
Next code:
def saving_exception() -> SyntaxError:
# try:
raise SyntaxError('Exception')
# except SyntaxError as e:
# return e
def middle_function() -> SyntaxError:
exception = saving_exception()
return exception
if __name__ == '__main__':
raise middle_function()
Returns 3 frames:
<module>
middle_function
saving_exception
Uncommenting 3 lines in saving_exception() will return 2 frames:
<module>
saving_exception
Also return e.with_traceback(sys.exc_info()[2]) in saving_exception() do the same as return e
How to make saving_exception() return exception with 3 frames without big changes outside?
def simple_generator():
print("-> start ..")
try:
x = yield
print("-> receive {} ..".format(x))
except StopIteration:
print("simple_generator exit..")
I know that each call of next on the generator object runs the code until the next yield statement, and returns the yielded value. If there is no more to get, StopIteration is raised.
So I wanted to catch the StopIteration in function simple_generator as the code above. Then I tried:
>>>
>>> sg3 = simple_generator()
>>> sg3.send(None)
-> start ..
>>> sg3.send("hello generator!")
-> receive hello generator! ..
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
It did throw StopIteration while the try ...excep dit not catch it at all, I could not understand what's underlying reason, could anyone explain that? Thanks in advance.
Of course, I also knew that if I dealed with the StopIteration exception outside function simple_generator, it did work as what I expected, for example.
>>> try:
... sg4 = simple_generator()
... while True:
... next(sg4)
... except StopIteration:
... print("sg4 exit ..")
...
-> start ..
-> receive None ..
sg4 exit ..
>>>
So my question is Why we could not catch Stopiteration exception in a generator definite function?
Is there a cleaner or more pythonic way to do the following?
try:
error_prone_function(arg1)
except MyError:
try:
error_prone_function(arg2)
except MyError:
try:
another_error_prone_function(arg3)
except MyError:
try:
last_error_prone_function(arg4)
except MyError:
raise MyError("All backup parameters failed.")
Basically it's: If attempt #1 fails, try #2. If #2 fails, try #3. If #3 fails, try #4. If #4 fails, ... if #n fails, then finally raise some exception.
Note that I'm not necessarily calling the same function every time, nor am I using the same function arguments every time. I am, however, expecting the same exception MyError on each function.
Thanks to John Kugelman's post here, I decided to go with this which utilizes the lesser-known else clause of a for loop to execute code when an entire list has been exhausted without a break happening.
funcs_and_args = [(func1, "150mm"),
(func1, "100mm",
(func2, "50mm"),
(func3, "50mm"),
]
for func, arg in funcs_and_args :
try:
func(arg)
# exit the loop on success
break
except MyError:
# repeat the loop on failure
continue
else:
# List exhausted without break, so there must have always been an Error
raise MyError("Error text")
As Daniel Roseman commented below, be careful with indentation since the try statement also has an else clause.
A generator based approach might give you a little more flexibility than the data-driven approach:
def attempts_generator():
# try:
# <the code you're attempting to run>
# except Exception as e:
# # failure
# yield e.message
# else:
# # success
# return
try:
print 'Attempt 1'
raise Exception('Failed attempt 1')
except Exception as e:
yield e.message
else:
return
try:
print 'Attempt 2'
# raise Exception('Failed attempt 2')
except Exception as e:
yield e.message
else:
return
try:
print 'Attempt 3'
raise Exception('Failed attempt 3')
except Exception as e:
yield e.message
else:
return
try:
print 'Attempt 4'
raise Exception('Failed attempt 4')
except Exception as e:
yield e.message
else:
return
raise Exception('All attempts failed!')
attempts = attempts_generator()
for attempt in attempts:
print attempt + ', retrying...'
print 'All good!'
The idea is to build a generator that steps through attempt blocks via a retry loop.
Once the generator hits a successful attempt it stops its own iteration with a hard return. Unsuccessful attempts yield to the retry loop for the next fallback. Otherwise if it runs out of attempts it eventually throws an error that it couldn't recover.
The advantage here is that the contents of the try..excepts can be whatever you want, not just individual function calls if that's especially awkward for whatever reason. The generator function can also be defined within a closure.
As I did here, the yield can pass back information for logging as well.
Output of above, btw, noting that I let attempt 2 succeed as written:
mbp:scratch geo$ python ./fallback.py
Attempt 1
Failed attempt 1, retrying...
Attempt 2
All good!
If you uncomment the raise in attempt 2 so they all fail you get:
mbp:scratch geo$ python ./fallback.py
Attempt 1
Failed attempt 1, retrying...
Attempt 2
Failed attempt 2, retrying...
Attempt 3
Failed attempt 3, retrying...
Attempt 4
Failed attempt 4, retrying...
Traceback (most recent call last):
File "./fallback.py", line 47, in <module>
for attempt in attempts:
File "./fallback.py", line 44, in attempts_generator
raise Exception('All attempts failed!')
Exception: All attempts failed!
Edit:
In terms of your pseudocode, this looks like:
def attempts_generator():
try:
error_prone_function(arg1)
except MyError
yield
else:
return
try:
error_prone_function(arg2)
except MyError
yield
else:
return
try:
another_error_prone_function(arg3)
except MyError:
yield
else:
return
try:
last_error_prone_function(arg4)
except MyError:
yield
else:
return
raise MyError("All backup parameters failed.")
attempts = attempts_generator()
for attempt in attempts:
pass
It'll let any exception but MyError bubble out and stop the whole thing. You also could choose to catch different errors for each block.
I know that if I want to re-raise an exception, I simple use raise without arguments in the respective except block. But given a nested expression like
try:
something()
except SomeError as e:
try:
plan_B()
except AlsoFailsError:
raise e # I'd like to raise the SomeError as if plan_B()
# didn't raise the AlsoFailsError
how can I re-raise the SomeError without breaking the stack trace? raise alone would in this case re-raise the more recent AlsoFailsError. Or how could I refactor my code to avoid this issue?
As of Python 3, the traceback is stored in the exception, so a simple raise e will do the (mostly) right thing:
try:
something()
except SomeError as e:
try:
plan_B()
except AlsoFailsError:
raise e # or raise e from None - see below
The traceback produced will include an additional notice that SomeError occurred while handling AlsoFailsError (because of raise e being inside except AlsoFailsError). This is misleading because what actually happened is the other way around - we encountered AlsoFailsError, and handled it, while trying to recover from SomeError. To obtain a traceback that doesn't include AlsoFailsError, replace raise e with raise e from None.
In Python 2 you'd store the exception type, value, and traceback in local variables and use the three-argument form of raise:
try:
something()
except SomeError:
t, v, tb = sys.exc_info()
try:
plan_B()
except AlsoFailsError:
raise t, v, tb
Even if the accepted solution is right, it's good to point to the Six library which has a Python 2+3 solution, using six.reraise.
six.reraise(exc_type, exc_value, exc_traceback=None)
Reraise an exception, possibly with a different traceback.
[...]
So, you can write:
import six
try:
something()
except SomeError:
t, v, tb = sys.exc_info()
try:
plan_B()
except AlsoFailsError:
six.reraise(t, v, tb)
As per Drew McGowen's suggestion, but taking care of a general case (where a return value s is present), here's an alternative to user4815162342's answer:
try:
s = something()
except SomeError as e:
def wrapped_plan_B():
try:
return False, plan_B()
except:
return True, None
failed, s = wrapped_plan_B()
if failed:
raise
Python 3.5+ attaches the traceback information to the error anyway, so it's no longer necessary to save it separately.
>>> def f():
... try:
... raise SyntaxError
... except Exception as e:
... err = e
... try:
... raise AttributeError
... except Exception as e1:
... raise err from None
>>> f()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 9, in f
File "<stdin>", line 3, in f
SyntaxError: None
>>>
I am reading some python code written a while ago, and found this:
try:
# do some stuff
except 0:
# exception handling stuff
And I'm just not sure what except 0 means? I do have my guesses: Supposed to catch nothing i.e. let the exception propagate or it could be some sort of switch to turn debugging mode on and off by removing the 0 which will then catch everything.
Can anyone lend some insight? a google search yielded nothing...
Thanks!
Some sample code (by request):
try:
if logErrors:
dbStuffer.setStatusToError(prop_id, obj)
db.commit()
except 0:
traceback.print_exc()
From what I understand, This is very useful for debugging purposes (Catching the type of exception)
In your example 0 acts as a placeholder to determine the type of exception.
>>> try:
... x = 5/1 + 4*a/3
... except 0:
... print 'error'
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
NameError: name 'a' is not defined
>>> try:
... x = 5/0 + 4*a/3
... except 0:
... print 'error'
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ZeroDivisionError: integer division or modulo by zero
In the first case, the exception is NameError and ZeroDivisionError in the second.
the 0 acts as a placeholder for any type of exception being caught.
>>> try:
... print 'error'
... except:
...
KeyboardInterrupt
>>> try:
... x = 5/0 + 4*a/3
... except:
... print 'error'
...
error
From the Python docs:
"[...] the [except] clause matches the exception if the resulting object is “compatible” with the exception. An object is compatible with an exception if it is the class or a base class of the exception object, or a tuple containing an item compatible with the exception."
In effect the type of the expression is used to determine whether the except clauses matches the exception. As 0 is of an integer type, and exception of that type would match.
Since integers cannot be raised as exception, this is a disabled exceptclass that will not catch anything.