We Keep Coding

How can I prevent or trap StopIteration exception in the yield-calling function?

A generator-returning function (i.e. one with a yield statement in it) in one of our libraries fails some tests due to an unhandled StopIteration exception. For convenience, in this post I'll refer to this function as buggy.
I have not been able to find a way for buggy to prevent the exception (without affecting the function's normal operation). Similarly, I have not found a way to trap the exception (with a try/except) within buggy.
(Client code using buggy can trap this exception, but this happens too late, because the code that has the information necessary to properly handle the condition leading to this exception is the buggy function.)
The actual code and test case I am working with are far too complicated to post here, so I have created a very simple, but also extremely artificial toy example that illustrates the problem.
First, the module with the buggy function:
# mymod.py
import csv # essential!
def buggy(csvfile):
with open(csvfile) as stream:
reader = csv.reader(stream)
# how to test *here* if either stream is at its end?
for row in reader:
yield row
As indicated by the comment, the use of the csv module (from the Python 3.x standard library) is an essential feature of this problem1.
The next file for the example is a script that is meant to stand in for "client code". In other word, this script's "real purpose" beyond this example is largely irrelevant. Its role in the example is to provide a simple, reliable way to elicit the problem with the buggy function. (Some of its code could be repurposed for a test case in a test suite, for example.)
#!/usr/bin/env python3
# myscript.py
import sys
import mymod
def print_row(row):
print(*row, sep='\t')
def main(csvfile, mode=None):
if mode == 'first':
print_row(next(mymod.buggy(csvfile)))
else:
for row in mymod.buggy(csvfile):
print_row(row)
if __name__ == '__main__':
main(*sys.argv[1:])
The script takes the path to a CSV file as a mandatory argument, and an optional second argument. If the second argument is ommitted, or it is anything other than the string "first", the script will print to stdout the information in the CSV file, but in TSV format. If the second argument is the string "first", only the information in the first row will be so printed.
The StopIteration exception I am trying to trap arises when myscript.py script is invoked with an empty file and the string "first" as arguments2.
Here is an example of this code in action:
% cat ok_input.csv
1,2,3
4,5,6
7,8,9
% ./myscript.py ok_input.csv
1 2 3
4 5 6
7 8 9
% ./myscript.py ok_input.csv first
1 2 3
% cat empty_input.csv
# no output (of course)
% ./myscript.py empty_input.csv
# no output (as desired)
% ./myscript.py empty_input.csv first
Traceback (most recent call last):
File "./myscript.py", line 19, in <module>
main(*sys.argv[1:])
File "./myscript.py", line 13, in main
print_row(next(mymod.buggy(csvfile)))
StopIteration
Q: How can I prevent or trap this StopIteration exception in the lexical scope of the buggy function?
IMPORTANT: Please keep in mind that, in the example given above, the myscript.py script is stand-in for "client code", and is therefore outside of our control. This means that any approach that would require changing the myscript.py script would not solve the actual real-world problem, and therefore it would not be an acceptable answer to this question.
One important difference between the simple example shown above and our actual situation is that in our case, the problematic input stream does not come from an empty file. The problem arises in cases where buggy (or, rather, its real-world counterpart) reaches the end of this stream "too early", so to speak.
I think it may be enough if I could test whether either stream is at its end, before the for row in reader: line, but I have not figured a way to do this either. Testing whether the value returned by stream.read(1) is 0 or 1 will tell me if stream is at its end, but in the latter case stream's internal pointer will be left pointing one byte too far into csvfile's content. (Neither stream.seek(-1, 1) nor stream.tell() work at this point.)
Lastly, to anyone who would like post an answer to this question: it would be most efficient if you were to take advantage of the example code I have provided above to test your proposal before posting it.
EDIT: One variation of mymod.py that I tried was this:
import csv # essential!
def buggy(csvfile):
with open(csvfile) as stream:
reader = csv.reader(stream)
try:
firstrow = next(reader)
except StopIteration:
firstrow = None
if firstrow != None:
yield firstrow
for row in reader:
yield row
This variation fails with pretty much the same error message as does the original version.
When I first read #mcernak's proposal, I thought that it was pretty similar to the variation above, and therefore expected it to fail too. Then I was pleasantly surprised to discover that this is not the case! Therefore, as of now, there is one definite candidate to get bounty. That said, I would love to understand why the variation above fails to trap the exception, while #mcernak's succeeds.
1 The actual case I'm dealing with is legacy code; switching from the csv module to some alternative is not an option for us in the short term.
2 Please, disregard entirely the question of what this demonstration script's "right response should be" when it gets invoked with an empty file and the string "first" as arguments. The particular combination of inputs that elicits the StopIteration exception in this post's demonstration does not represent the real-world condition that causes our code to emit the problematic StopIteration exception. Therefore, the "correct response", whatever that may be, of the demonstration script to the empty file plus "first" string combination would be irrelevant to the real-world problem I am dealing with.

You can trap the StopIteration exception in the lexical scope of the buggy function this way:
import csv # essential!
def buggy(csvfile):
with open(csvfile) as stream:
reader = csv.reader(stream)
try:
yield next(reader)
except StopIteration:
yield 'dummy value'
for row in reader:
yield row
You basically manually request the first value from the reader iterator and
if this succeeds, the first line is read from the csv file and is yielded to the caller of buggy function
if this fails, as is the case for empty csv files, some string e.g. dummy value is yielded to prevent the caller of the buggy function from crashing
Afterwards, if the csv file was not empty, the remaining rows will be read (and yielded) in the for cycle.
EDIT: to illustrate why the other variation of mymod.py mentioned in the question does not work, I've added some print statements to it:
import csv # essential!
def buggy(csvfile):
with open(csvfile) as stream:
reader = csv.reader(stream)
try:
print('reading first row')
firstrow = next(reader)
except StopIteration:
print('no first row exists')
firstrow = None
if firstrow != None:
print('yielding first row: ' + firstrow)
yield firstrow
for row in reader:
print('yielding next row: ' + row)
yield row
print('exiting function open')
Running it gives the following output:
% ./myscript.py empty_input.csv first
reading first row
no first row exists
exiting function open
Traceback (most recent call last):
File "myscript.py", line 15, in <module>
main(*sys.argv[1:])
File "myscript.py", line 9, in main
print_row(next(mymod.buggy(csvfile)))
That shows, that in case that the input file is empty, the first try..except block correctly handles the StopIteration exception and that the buggy function continues on normally.
The exception that the caller of the buggy gets in this case is due to the fact that the buggy function does not yield any value before completing.

mcernak solves and describes the problem you have well
However, there is a design problem underlying the issue: the caller sometimes isn't expecting a generator, but a non-empty iterator
Looking at this from another aspect, what should happen if the file is missing? Does it make more sense for the function handle IOError from open and return some sentinel or raise it to the caller?
Instead of trying to coerce your generator to work with callers mistreating it, consider
providing two functions (one could call the other)
providing an argument for a maximal count of rows from the generator (probably best)
# mymod.py
import csv
import itertools
def notbuggy(csvfile, max_rows=None):
with open(csvfile) as stream:
yield from itertools.islice(csv.reader(stream), max_rows)
#!/usr/bin/env python3
# myscript.py
import sys
import mymod
def print_row(row):
print(*row, sep='\t')
def main(csvfile, mode=None):
max_rows = 1 if mode == "first" else None
for row in mymod.notbuggy(csvfile, max_rows):
print_row(row)
if __name__ == '__main__':
main(*sys.argv[1:])
When using next(), the calling logic must agree to one of
never call it on an empty iterable (check file first?)
handle an Exception from the generator (StopIteration, some custom Exception)
handle some empty sentinel (perhaps "", some string, None, or object..)
However, the caller does none of these, so the guarantees are not well set!
What if the caller wanted more than a single row or interprets the empty sentinel as a value? Unless these are somehow communicated in documentation, the caller can always misuse a function and not know why it has unexpected behavior.
>>> next(iter(()))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>> g = iter((1,))
>>> next(g)
1
>>> next(g)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
>>> print_row("STOP SENTINEL")
S T O P S E N T I N E L

How can I read lines from a txt file (which contains a code) and decide whether it is correct or not?

The text file contains Python codes.
My task is to write a program which checks if it could run in python. I cannot find out how to check if the lines are correct in python.
I can read lines from the text file, but still don't know how to decide if the current line would run without any error messages. If it would run the program prints True, else False.
lst=[]
with open("be.txt","r") as be:
for line in be:
lst.append(line.strip("\n"))
For example: A line is: "for i in range(8:"
The result should be False, because this ")" is missing.

You can use exec:
with open("be.txt","r") as be:
source = be.read()
try:
exec(source)
print("True")
except Exception as e:
print(e) #this add some info on where the error occurred
print("False")
First create a single string source holding all the code in your file.
Then in a try except block use exec to se if it runs. If it fails anywhere, an exception will be raised, and the except block is executed.
Generally functions like exec or eval are considered dangerous and nasty (see for example here or here) but I think in your case is the simplest thing you can do.

The only way to know whether the code would run in python without any error messages is to execute it. The following would do that. Of course, you would need to go to extra lengths to suppress any output if the program is correct and produces output.
def runner():
with open("be.txt","r") as f:
code = f.read()
try:
exec(code)
except Exception as e:
print("False")
else:
print("True")
runner()

Use exec and catch exceptions. exec will attempt to execute the line as Python code. If it succeeds then the line is executed and any side-effects (eg. the creation of new variables) take place. If not, an exception is thrown and you can skip it and move on to the next line. Something like this:
lst=[]
with open("be.txt","r") as be:
for line in be:
try:
exec(line)
lst.append(line.strip("\n"))
except:
pass
Note this will not work if a single line is not a complete Python statement. Eg. if you have a function def or a for loop over several lines, you'll have to exec them all at once to succeed.

Python: catch any exception and put it in a variable

To figure out what it would take to avoid some recursion, I need to catch any exception (edit: Not just ones derived from Exception, but all exceptions, including KeyboardInterrupt and user exceptions), put it in a variable, and later re-raise it outside the catch block. Essentially, I'm trying to roll my own finally block. Is this possible?
The actual problem is to call a number of cleanup functions, and if any of them fail, all the others should also be called, then the exception for the one that failed should still propagate. Here's my current solution, it takes a list of Popen objects:
def cleanupProcs(procs):
if not procs:
return
proc = procs.pop(0)
try:
proc.terminate()
proc.wait()
finally:
cleanupProcs(procs)
Is there an iterative way to do this? A more elegant way? A more Pythonic way?

If you want the stacktrace included:
try:
# something
except:
the_type, the_value, the_traceback = sys.exc_info()
later
raise the_type, the_value, the_traceback
(Related to this answer)
See also here for Python 2.7.

In this case, I think I would argue that you may not be doing it right.
To me, the point of an exception is to signal a bum bum baaaaa exceptional circumstance. And when you are writing code that may fire an exception you have two responsible options - catch it and do something with it (like recover), or completely ignore it.
From what you say in your post, you don't really care that an exception happened. It shouldn't halt your program, and program flow should continue as normal. You do however want to know that an exception happened. And this is where the logging module comes in:
import logging
log = logging
def get_some_cheese():
raise ValueError("Sorry, we're right out.")
try:
get_some_cheese()
except:
log.exception("What a waste of life")
When you log an exception it automagically adds stack trace information for you. After you config your logging a bit, you can get it setup to do all sorts of whatever things you want to do - send an email, write to a file or stdout/err, whatever. But then you'll get informed that an exception occured, but you can also simply recover from the error and continue on your merry way cleaning up whatever it is you need to clean up.

I'd probably use BaseException to catch anything that gets thrown, and iterate through all your cleanup functions (instead of using recursion). Then append any exceptions to a list, to deal with (re-raise, log, etc) as appropriate when you finish the cleanup.
def runCleanup(procs):
exceptions = []
for proc in procs:
try:
proc.terminate()
proc.wait()
except BaseException as e:
exceptions.append(e) # Use sys.exc_info() for more detail
return exceptions # To be handled or re-raised as needed

you can use:
procexceptions = []
except Exception, e:
procexceptions.append(e)
and then later (after the loop for terminating processes) you can
raise procexceptions[0]
etc.

Openstack does something very similar for a different reason. Have a look at https://github.com/openstack/nova/blob/master/nova/openstack/common/excutils.py#L30 (function save_and_reraise_exception). In their case it works like a resource manager.

Like just about everything else in Python, exceptions are objects and therefore can be bound to names and operated upon. Here's a short example showing how to grab an exception and use it later:
>>> def to_int(x):
... try:
... return int(x)
... except Exception, e:
... print 'in exception block:', e
... print 'after exception block:', e
>>> to_int('12')
12
>>> to_int('abc')
in exception block: invalid literal for int() with base 10: 'abc'
after exception block: invalid literal for int() with base 10: 'abc'

Thats easy:
>>> try:
... #something
... except BaseException, e: # OK. Using BaseException instead of Exception
... pass
...
>>>
>>> raise e
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined
>>>

Restarting a program after exception

I have a program that queries an API every few seconds. Each response triggers a few functions which themselves make some calls to websites and such -- calls that I don't want to blindly trust to succeed. If I catch an exception in foo(), for example, or even in a function that foo() calls, is it possible to restart the program entirely in the except block? Essentially, I want to call queryRepeatedly() upon an exception in one of its sub-functions, without keeping the previous call on the stack.
Of course, I could return marker values and solve this another way, but the program is structured in a way such that the above approach seems much simpler and cleaner.
# Sample "main" function that I want to call
def queryRepeatedly():
while True:
foo()
bar()
baz()
time.sleep(15)
def foo():
# do something
try:
foo2() # makes a urllib2 call that I don't trust
except:
#restart queryRepeatedly
queryRepeatedly()

To restart anything, just use a while loop outside the try. For example:
def foo():
while True:
try:
foo2()
except:
pass
else:
break
And if you want to pass the exception up the chain, just do this in the outer function instead of the inner function:
def queryRepeatedly():
while True:
while True:
try:
foo()
bar()
baz()
except:
pass
else:
break
time.sleep(15)
def foo():
foo2()
All that indentation is a little hard to read, but it's easy to refactor this:
def queryAttempt()
foo()
bar()
baz()
def queryOnce():
while True:
try:
queryAttempt()
except:
pass
else:
break
def queryRepeatedly():
while True:
queryOnce()
time.sleep(15)
But if you think about it, you can also merge the two while loops into one. The use of continue may be a bit confusing, but see if you like it better:
def queryRepeatedly():
while True:
try:
foo()
bar()
baz()
except:
continue
time.sleep(15)

Refactor this - you'll get a stackoverflow error sooner or later if you have enough failures.
queryRepeatedly should just be query. It should return void and throw exceptions on failures.
Wrap in something that looks like this, your true queryRepeatedly function?
while True:
try:
query()
except:
#handle
time.sleep(15)
All looping, no recursion needed.
Note that you must think carefully about how much of your program you need to restart. From your question it sounded like your actual problem was ensuring the query could try again if it sporadically fails, which is what my solution ensures. But if you want to clean up program resources - say, bounce SQL connections, which may have broken - then you need to think more carefully about how much of your program you need to "restart." In general you need to understand why your query failed to know what to fix, and in the extreme case, the right thing to do is an email or SMS to someone on call who can inspect the situation and write an appropriate patch or fix.

First make two files.
One file called run.py and one called forever.py and put them in the same folder.
Go to your terminal within that folder and type chmod +x forever.py
run.py
whatever code you want to run
forever.py
#!/usr/local/lib/python3.7
from subprocess import Popen
import sys
filename = sys.argv[1]
while True:
print("\nStarting " + filename)
p = Popen("python3 " + filename, shell=True)
p.wait()
Open a terminal window from the folder and type this:
python3 ./forever.py run.py
to start run.py and if it fails or has an exception, it'll just start over again.
You now have a template to make sure if a file crashes or has an exception, you can restart it without being around. If this helps you, please give me a vote!

In your exception make a recursive call
except:
queryRepeatedly()

Guidance on exceptions given by the Python docs

According to the Python documentation on idioms and anti-idioms in relation to exceptions: "You should try to use as few except clauses in your code as you can — the ones you do use will usually be inside calls which should always succeed, or a catch-all in a main function." Taking this sentence in sections...
"You should try to use as few except clauses in your code as you can"
A bit confusing for a newbie like myself, I thought it was good practise in Python to use the EAFP style -i.e. many try and except statements. Or am I missing the point?
"the ones you do use will usually be inside calls which should always succeed"
I don't understand the point that's being made here.
"or a catch-all in a main function."
So it's good style to any let code that throws an exception to simply pass it up the call stack until it reaches the top level where you have really generic exception handling?

"You should try to use as few except clauses in your code as you can"
It's easy to litter your code with exceptions:
def solve_linear(mat1, mat2):
det1 = determinant(mat1)
det2 = determinant(mat2)
try:
return det1 / det2
except ZeroDivisionError:
raise NoSolution
Here, it's probably fine to let the ZeroDivisionError propagate. You don't need to catch it.
"the ones you do use will usually be inside calls which should always succeed"
For example, look at this code which reads a file, or returns a cached value. It normally succeeds, in spite of the KeyError exception:
def read_file(path):
try:
return cache[path]
except KeyError:
fp = open(path, 'rb')
data = fp.read()
fp.close()
cache[path] = data
return data
"or a catch-all in a main function."
If your program is interactive, you'll probably want to catch almost everything at the top level. Here's the top loop of an interactive command-line program:
def main():
while True:
try:
text = raw_input('enter command: ')
cmd = parse_command(text)
except EOFError:
return
except KeyboardInterrupt:
return
except ValueError:
print 'Syntax error'
continue
try:
run_cmd(cmd)
except SystemExit:
raise
except Exception:
traceback.print_exc()

Concerning the first point, the whole point of using exceptions is that you don't have to wrap every line in one! E.g. in C, errors are usually determined by the return value of a function call. So you have to check those after every call if you want to catch all errors. With Python you can group a (possibly large) block of statements that go together in a try/except block and only deal with all errors once.
The second point is that (if possible) you want to solve failures close to the point where they occur. E.g. you are reading data from a network and get zero bytes. In that case it is usually perfectly allright to wait and try again.
The last point is that sometimes an error is so big that it cannot be handled at a low level. E.g. if you are trying to open a file that not exist, it wil fail. And your program cannot do whatever it was going to do with the contents of the file. It is best to handle that at the top level of the program and ask the user for another file name or maybe quit the program.

I think the point is that exceptions should be used only for 'exceptional' circumstances. That is the meaning behind the use "in calls which will usually succeed". An example of this might be some computation that under some really weird circumstances ends having to do a division by zero. Then you can enclose that in a try / except statement to deal with that possibility.
The catch-all in a main function would be applicable to the same scenario. Say your calculations end up with a division by zero somewhere deep in the call stack. From this point you can't continue, so there's no point in having a try/except statement right at the point of failure. It would make more sense to just have one at a higher level where you can reasonably recover from the error.
The example they give in the documentation is an example of this. When calling 'get_status' you would expect the file to be there. If it doesn't then you have the except statement to deal with that (although, as mentioned, in that particular case the 'with' statement is much better).

The Python philosophy is typically to "ask forgiveness, not permission." But the idea isn't to use a try-except clause to catch ALL possible errors. Ideally, each try-except will only catch the error(s) that are relevant.
BAD (doesn't specify a specific exception type):
a = [1, 2, 3]
i = 3
try:
print a[i]
except:
print "Not found!"
GOOD (only handles the exception we expect to get):
a = [1, 2, 3]
i = 3
try:
print a[i]
except IndexError:
print "Not found!"
The reason this is important is so we don't obscure other possible code errors. What if, in this example, i was 1.8? The first example would print Not Found!, obscuring the real issue, but the second example would return TypeError: list indices must be integers, not float letting us know that there's a logical flaw in our code.