Sometimes when using ipython you might hit an exception in a function which has opened a file in write mode. This means that the next time you run the function you get a value error,
ValueError: The file 'filename' is already opened. Please close it before reopening in write mode.
However since the function bugged out, the file handle (which was created inside the function) is lost, so it can't be closed. The only way round it seems to be to close the ipython session, at which point you get the message:
Closing remaining open files: filename... done
Is there a way to instruct ipython to close the files without quitting the session?
You should try to always use the with statement when working with files. For example, use something like
with open("x.txt") as fh:
...do something with the file handle fh
This ensures that if something goes wrong during the execution of the with block, and an exception is raised, the file is guaranteed to be closed. See the with documentation for more information on this.
Edit: Following a discussion in the comments, it seems that the OP needs to have a number of files open at the same time and needs to use data from multiple files at once. Clearly having lots of nested with statements, one for each file opened, is not an option and goes against the ideal that "flat is better than nested".
One option would be to wrap the calculation in a try/finally block. For example
file_handles = []
try:
for file in file_list:
file_handles.append(open(file))
# Do some calculations with open files
finally:
for fh in file_handles:
fh.close()
The finally block contains code which should be run after any try, except or else block, even if an exception occured. From the documentation:
If finally is present, it specifies a "cleanup" handler. The try clause is executed, including any except and else clauses. If an exception occurs in any of the clauses and is not handled, the exception is temporarily saved. The finally clause is executed. If there is a saved exception, it is re-raised at the end of the finally clause. If the finally clause raises another exception or executes a return or break statement, the saved exception is lost. The exception information is not available to the program during execution of the finally clause.
A few ideas:
use always finally (or a with block) when working with files, so they are properly closed.
you can blindly close the non standard file descriptors using os.close(n) where n is a number greater than 2 (this is unix specific, so you might want to peek /proc/ipython_pid/fd/ to see what descriptors the process have opened so far).
you can inspect the captured stack frames locals to see if you can find the reference to the wayward file and close it... take a look to sys.last_traceback
Related
I have a piece of code that opens a workbook by means of the xlwings package and reads some stuff from it. When done, the workbook is closed. Sometimes, errors occur between the opening and the closing statements. This leaves the workbook open. When I attempt to run the code again, the file cannot be opened as it is already 'open'.
How can I deal with this?
Also, I cannot even open manually the workbook when this happens.
w1=app.books.open("WorkbookName.xlsx")
# Do stuff that can potentially give errors.
w1.close()
I expect to create some error handling so that if an exception is found along the way, the workbook is closed after it.
Use a try-finally construct:
w1=app.books.open("WorkbookName.xlsx")
try:
# Do stuff that can potentially give errors.
finally:
w1.close()
A finally clause is always executed. If the code in the try clause raises an unhandled exception, then the python interpreter will jump to the finally clause and execute it (and terminate itself only afterwards).
In a script data is written like so:
result = open("c:/filename.csv", "w")
result.write("\nTC-"+str(TC_index))
The .csv-file is filled with data in a while(1) loop.
I run the script in Eclipse and exit by hitting the stop button.
Unfortunately most of the time when I open the file it is completely empty.
Is there a way to fix that?
To ensure a content is flushed and written to file without having to close the file handle:
import os
# ...
result.write("\nTC-"+str(TC_index))
result.flush()
os.fsync(result)
But of course, if you break the loop manually there's no guarantee you won't break it between the write and the flush, thereby failing to get the last line. I'm unfamiliar with the Eclipse stop button but perhaps it stops execution by causing a KeyboardInterrupt exception to be raised. If so you could always catch that and explicitly close the file. Better still, use a with statement which will cause that to happen automatically:
with open("c:/filename.csv", "w") as result:
for TC_index in range(100): # or whatever loop
result.write("\nTC-"+str(TC_index))
# flush & fsync here if still necessary (but might not be)
I have a file that I open before a loop starts, and I'm writing to that file almost at each iteration of the loop. Then I close the file once the loop has finished. So e.g. something like:
testfile = open('datagathered','w')
for i in range(n):
...
testfile.write(line)
testfile.close()
The issue I'm having is that, in case the program crashes or I want to crash it, what has already been written to testfile will be deleted, and the text file datagathered will be empty. I understand that this happens because I'm closing the file only after the loop, but if I close and open the file after each write (i.e. in the loop) doesn't that lead to an incredible slow-down?
If yes, what alternatives do I have for doing the writing, and making sure that in case of a crash the already-written-lines won't get lost, in an efficient way?
The linked posts do bring up good suggestions that arguably answer this question, but they don't cover risks and efficiency differences involved. More precisely: Are there any risks involved with playing with the buffersize? e.g. testfile = open('datagathered','w',0) Finally is using with open... still a viable alternative if there are multiple files to write to?
Small note: This is asked in the context of a very long run, where the file is being written to for 2-3 days. Thus having a speedy and safe way of doing the writing is definitely valuable here.
From the question I understood that you are talking about exceptions may occur at runtime and SIGINT.
You may use 'try-except-finally' block to achieve your goal. It enables you to catch both exceptions and SIGINT signal. Since the finally block will be executed either exception is caught or everything goes well, closing file there is the best choice. Following sample code would solve your problem I guess.
testfile = open('datagathered','w')
try:
for i in range(n):
...
testfile.write(line)
except KeyboardInterrupt:
print "Interrupt from keyboard"
except:
print "Other exception"
finally:
testfile.close()
Use a context:
with open('datagathered','w') as f:
f.write(data)
I have read that when file is opened using the below format
with open(filename) as f:
#My Code
f.close()
explicit closing of file is not required . Can someone explain why is it so ? Also if someone does explicitly close the file, will it have any undesirable effect ?
The mile-high overview is this: When you leave the nested block, Python automatically calls f.close() for you.
It doesn't matter whether you leave by just falling off the bottom, or calling break/continue/return to jump out of it, or raise an exception; no matter how you leave that block. It always knows you're leaving, so it always closes the file.*
One level down, you can think of it as mapping to the try:/finally: statement:
f = open(filename)
try:
# My Code
finally:
f.close()
One level down: How does it know to call close instead of something different?
Well, it doesn't really. It actually calls special methods __enter__ and __exit__:
f = open()
f.__enter__()
try:
# My Code
finally:
f.__exit__()
And the object returned by open (a file in Python 2, one of the wrappers in io in Python 3) has something like this in it:
def __exit__(self):
self.close()
It's actually a bit more complicated than that last version, which makes it easier to generate better error messages, and lets Python avoid "entering" a block that it doesn't know how to "exit".
To understand all the details, read PEP 343.
Also if someone does explicitly close the file, will it have any undesirable effect ?
In general, this is a bad thing to do.
However, file objects go out of their way to make it safe. It's an error to do anything to a closed file—except to close it again.
* Unless you leave by, say, pulling the power cord on the server in the middle of it executing your script. In that case, obviously, it never gets to run any code, much less the close. But an explicit close would hardly help you there.
Closing is not required because the with statement automatically takes care of that.
Within the with statement the __enter__ method on open(...) is called and as soon as you go out of that block the __exit__ method is called.
So closing it manually is just futile since the __exit__ method will take care of that automatically.
As for the f.close() after, it's not wrong but useless. It's already closed so it won't do anything.
Also see this blogpost for more info about the with statement: http://effbot.org/zone/python-with-statement.htm
The Python Tutorial states:
The try ... except statement has an optional else clause, which,
when present, must follow all except clauses. It is useful for code
that must be executed if the try clause does not raise an exception.
For example:
for arg in sys.argv[1:]:
try:
f = open(arg, 'r')
except IOError:
print 'cannot open', arg
else:
print arg, 'has', len(f.readlines()), 'lines'
f.close()
The use of the else clause is better than adding additional code
to the try clause because it avoids accidentally catching an exception
that wasn’t raised by the code being protected by the try ... except
statement.
Question 1> Ａｆｔｅｒ reading the above document, I still don't get the idea why we cannot simply move the code from else clause into try clause.
Question 2> How does try clause can accidentally catch an exception since all those catches are done in the except clause, right?
You could put the else code in the try suite, but then you'd catch any exceptions that might be raised there. If you didn't intend that to happen, it would be "accidental," hence the wording of the document you linked to.
Best practice is to put as little code as possible in a try block, so that when an error occurs, you know what operation caused it and can handle it appropriately. If you have five lines of code in a try block and only expect one of them to ever raise an exception, your exception-handling code will be ill-prepared when an exception occurs in a line you didn't expect it to. Better in that case to let the exception be raised than handle it the wrong way.
If you move the code from the else into the try then that becomes part of the "critical path" which can raise an exception. If f.readlines() raises some sort of exception (perhaps an I/O error while reading the file because of a bad sector on the disk) then that error will be conflated with the one error that you currently catch. (Technically the "cannot open" error message would be wrong at that point ... because opening a file can succeed when reading it later fails; in fact opening it must succeed before you can even get an I/O error while processing it).
Normally you'd use a pattern more like:
foo = None
try:
# some code to access/load/initialize foo from an external source
except ...:
# handle various types of file open/read, database access, etc errors
else:
foo = something
... so that your subsequently run code and simply check if foo is None and use it or
work around it's unavailability in whatever way you see fit.
Answer for both questions is similar,
If you move code to the try clause, then you can not be sure from where the exception is coming from. Thus if you have another line of code that produces an unexpected IOError you could end searching for a problem where there is not.
So to better disect your code you want to simplify as much as possible the lines in the try making the catch as especific as possible.
1) Of course you could just move code from the else clause into the try clause. You could move it outside of the try block entirely, but this allows extra flexibility, and further modulation of the code. Also, perhaps specificity with the errors being caught. You could list a whole load of different exceptions that might be likely to happen, each with different statements. The stuff in the else clause would still only happen if no exception was raised, after the execution of the final line in the try block. e.g. printing a successful return message.
Also, try clauses add some extra CPU overhead with regards to garbage collection management and error tracing, so anything outside of the try clause is not protected in the same manner and may run more efficiently.
2) Error catching is quite specific. e.g. The except clause in your above example will only be run if an IOError is raised when running the f = open(arg,'r') line. If you want to catch any form of exception, use except Exception:.