Let us say, we have the following code:
from sys import exit
def parseLine(l):
if '#' not in l:
print 'Invalid expresseion'
exit(1)
return l
with open('somefile.txt') as f:
for l in f:
print parseLine(l)
(Note that this is a demo code. The actual program is much more complex.)
Now, how do I know if I have safely closed all the open files when I exit from the program? At this point I am just assuming that the files have been closed. Currently my programs are working OK, but I want them to be robust and free of problems related to files not closed properly.
One of the chief benefits of the with block with files is that it will automatically close the file, even if there's an exception.
https://docs.python.org/2/tutorial/inputoutput.html#methods-of-file-objects
It's already closing properly, since you're using a with statement when you open the file. That'll automatically close the file when control leaves the with statement, even if there's an exception. This is usually considered the best way to ensure files are closed when they should be.
If you don't use a with statement or close the file yourself, there are a few built-in safeties and a few pitfalls.
First, in CPython, the file object's destructor will close the file when it gets garbage-collected. However, that isn't guaranteed to happen in other Python implementations, and even in CPython, it isn't guaranteed to happen promptly.
Second, when your program exits, the operating system will close any files the program left open. This means if you accidentally do something that makes the program never close its files (perhaps you had to issue a kill -9 or something else that prevents cleanup code from running), you don't have to reboot the machine or perform filesystem repair to make the file usable again. Relying on this as your usual means of closing files would be inadvisable, though.
If you're using a with block, you essentially have your open call inside of a try block and the close in a finally block. See https://docs.python.org/2/tutorial/inputoutput.html for more information from the official docs.
Since calling exit() actually raises the SystemExit exception, all code within finally blocks will be run before the program completely exits. Since this is the case, and since you're using with open(...) blocks, the file will be closed with any uncaught exception.
Below is your code (runnable/debuggable/steppable at http://python.dbgr.cc/s)
from sys import exit
def parseLine(l):
if '#' not in l:
print 'Invalid expresseion'
exit(1)
return l
with open('somefile.txt') as f:
for l in f:
print parseLine(l)
print("file is closed? %r" % f.closed)
Equivalent code without using the with open(...) block is shown below (runnable/debuggable at http://python.dbgr.cc/g):
from sys import exit
def parseLine(l):
if '#' not in l:
print 'Invalid expresseion'
exit(1)
return l
try:
f = open('somefile.txt')
for l in f:
print parseLine(l)
finally:
print("Closing open file!")
f.close()
print("file is closed? %r" % f.closed)
Related
I have a python script which dump yaml file every second while running. However, I found sometimes the yaml file is not completed. My guess is it is coincidence that at the same time I am closing the progress (the script is running in Windows commend line), while the file is saving. Sample code as following:
class State(object):
def __init__(self):
...
self.__t = threading.Thread(name='StateAutoSave', target=self.__auto_save)
self.__t.start()
def __auto_save(self):
while 1:
try:
...
self.__save()
except Exception as err:
logging.exception(err)
time.sleep(1)
def __save(self):
...
with open(self.__yaml_file, 'w') as outfile:
yaml.dump(data, outfile, default_flow_style=False)
How to avoid this problem? or is there a method like destructor function in python, so that we can do something when the program is being closed? (It seems that 'with' does not fully work here)
The atexit module is made just for this purpose.
https://docs.python.org/3/library/atexit.html
Just beware that it only works for normal termination of the script (ctrl + c is considered normal), and it won't work if you app suddenly crashes or you have to force close it.
In a script data is written like so:
result = open("c:/filename.csv", "w")
result.write("\nTC-"+str(TC_index))
The .csv-file is filled with data in a while(1) loop.
I run the script in Eclipse and exit by hitting the stop button.
Unfortunately most of the time when I open the file it is completely empty.
Is there a way to fix that?
To ensure a content is flushed and written to file without having to close the file handle:
import os
# ...
result.write("\nTC-"+str(TC_index))
result.flush()
os.fsync(result)
But of course, if you break the loop manually there's no guarantee you won't break it between the write and the flush, thereby failing to get the last line. I'm unfamiliar with the Eclipse stop button but perhaps it stops execution by causing a KeyboardInterrupt exception to be raised. If so you could always catch that and explicitly close the file. Better still, use a with statement which will cause that to happen automatically:
with open("c:/filename.csv", "w") as result:
for TC_index in range(100): # or whatever loop
result.write("\nTC-"+str(TC_index))
# flush & fsync here if still necessary (but might not be)
So I want to write some files that might be locked/blocked for write/delete by other processes and like to test that upfront.
As I understand: os.access(path, os.W_OK) only looks for the permissions and will return true although a file cannot currently be written. So I have this little function:
def write_test(path):
try:
fobj = open(path, 'a')
fobj.close()
return True
except IOError:
return False
It actually works pretty well, when I try it with a file that I manually open with a Program. But as a wannabe-good-developer I want to put it in a test to automatically see if it works as expected.
Thing is: If I just open(path, 'a') the file I can still open() it again no problem! Even from another Python instance. Although Explorer will actually tell me that the file is currently open in Python!
I looked up other posts here & there about locking. Most are suggesting to install a package. You migth understand that I don't wanna do that to test a handful lines of code. So I dug up the packages to see the actual spot where the locking is eventually done...
fcntl? I don't have that. win32con? Don't have it either... Now in filelock there is this:
self.fd = os.open(self.lockfile, os.O_CREAT|os.O_EXCL|os.O_RDWR)
When I do that on a file it moans that the file exists!! Ehhm ... yea! That's the idea! But even when I do it on a non-existing path. I can still open(path, 'a') it! Even from another python instance...
I'm beginning to think that I fail to understand something very basic here. Am I looking for the wrong thing? Can someone point me into the right direction?
Thanks!
You are trying to implement the file locking problem using just the system call open(). The Unix-like systems uses by default advisory file locking. This means that cooperating processes may use locks to coordinate access to a file among themselves, but uncooperative processes are also free to ignore locks and access the file in any way they choose. In other words, file locks lock out other file lockers only, not I/O. See Wikipedia.
As stated in system call open() reference the solution for performing atomic file locking using a lockfile is to create a unique file on the same file system (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful.
That is why in filelock they also use the function fcntl.flock() and puts all that stuff in a module as it should be.
Alright! Thanks to those guys I actually have something now! So this is my function:
def lock_test(path):
"""
Checks if a file can, aside from it's permissions, be changed right now (True)
or is already locked by another process (False).
:param str path: file to be checked
:rtype: bool
"""
import msvcrt
try:
fd = os.open(path, os.O_APPEND | os.O_EXCL | os.O_RDWR)
except OSError:
return False
try:
msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
os.close(fd)
return True
except (OSError, IOError):
os.close(fd)
return False
And the unittest could look something like this:
class Test(unittest.TestCase):
def test_lock_test(self):
testfile = 'some_test_name4142351345.xyz'
testcontent = 'some random blaaa'
with open(testfile, 'w') as fob:
fob.write(testcontent)
# test successful locking and unlocking
self.assertTrue(lock_test(testfile))
os.remove(testfile)
self.assertFalse(os.path.exists(testfile))
# make file again, lock and test False locking
with open(testfile, 'w') as fob:
fob.write(testcontent)
fd = os.open(testfile, os.O_APPEND | os.O_RDWR)
msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
self.assertFalse(lock_test(testfile))
msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
self.assertTrue(lock_test(testfile))
os.close(fd)
with open(testfile) as fob:
content = fob.read()
self.assertTrue(content == testcontent)
os.remove(testfile)
Works. Downsides are:
It's kind of testing itself with itself
so the initial OSError catch is not even tested, only locking again with msvcrt
But I dunno how to make it better now.
I have read that when file is opened using the below format
with open(filename) as f:
#My Code
f.close()
explicit closing of file is not required . Can someone explain why is it so ? Also if someone does explicitly close the file, will it have any undesirable effect ?
The mile-high overview is this: When you leave the nested block, Python automatically calls f.close() for you.
It doesn't matter whether you leave by just falling off the bottom, or calling break/continue/return to jump out of it, or raise an exception; no matter how you leave that block. It always knows you're leaving, so it always closes the file.*
One level down, you can think of it as mapping to the try:/finally: statement:
f = open(filename)
try:
# My Code
finally:
f.close()
One level down: How does it know to call close instead of something different?
Well, it doesn't really. It actually calls special methods __enter__ and __exit__:
f = open()
f.__enter__()
try:
# My Code
finally:
f.__exit__()
And the object returned by open (a file in Python 2, one of the wrappers in io in Python 3) has something like this in it:
def __exit__(self):
self.close()
It's actually a bit more complicated than that last version, which makes it easier to generate better error messages, and lets Python avoid "entering" a block that it doesn't know how to "exit".
To understand all the details, read PEP 343.
Also if someone does explicitly close the file, will it have any undesirable effect ?
In general, this is a bad thing to do.
However, file objects go out of their way to make it safe. It's an error to do anything to a closed file—except to close it again.
* Unless you leave by, say, pulling the power cord on the server in the middle of it executing your script. In that case, obviously, it never gets to run any code, much less the close. But an explicit close would hardly help you there.
Closing is not required because the with statement automatically takes care of that.
Within the with statement the __enter__ method on open(...) is called and as soon as you go out of that block the __exit__ method is called.
So closing it manually is just futile since the __exit__ method will take care of that automatically.
As for the f.close() after, it's not wrong but useless. It's already closed so it won't do anything.
Also see this blogpost for more info about the with statement: http://effbot.org/zone/python-with-statement.htm
Sometimes when using ipython you might hit an exception in a function which has opened a file in write mode. This means that the next time you run the function you get a value error,
ValueError: The file 'filename' is already opened. Please close it before reopening in write mode.
However since the function bugged out, the file handle (which was created inside the function) is lost, so it can't be closed. The only way round it seems to be to close the ipython session, at which point you get the message:
Closing remaining open files: filename... done
Is there a way to instruct ipython to close the files without quitting the session?
You should try to always use the with statement when working with files. For example, use something like
with open("x.txt") as fh:
...do something with the file handle fh
This ensures that if something goes wrong during the execution of the with block, and an exception is raised, the file is guaranteed to be closed. See the with documentation for more information on this.
Edit: Following a discussion in the comments, it seems that the OP needs to have a number of files open at the same time and needs to use data from multiple files at once. Clearly having lots of nested with statements, one for each file opened, is not an option and goes against the ideal that "flat is better than nested".
One option would be to wrap the calculation in a try/finally block. For example
file_handles = []
try:
for file in file_list:
file_handles.append(open(file))
# Do some calculations with open files
finally:
for fh in file_handles:
fh.close()
The finally block contains code which should be run after any try, except or else block, even if an exception occured. From the documentation:
If finally is present, it specifies a "cleanup" handler. The try clause is executed, including any except and else clauses. If an exception occurs in any of the clauses and is not handled, the exception is temporarily saved. The finally clause is executed. If there is a saved exception, it is re-raised at the end of the finally clause. If the finally clause raises another exception or executes a return or break statement, the saved exception is lost. The exception information is not available to the program during execution of the finally clause.
A few ideas:
use always finally (or a with block) when working with files, so they are properly closed.
you can blindly close the non standard file descriptors using os.close(n) where n is a number greater than 2 (this is unix specific, so you might want to peek /proc/ipython_pid/fd/ to see what descriptors the process have opened so far).
you can inspect the captured stack frames locals to see if you can find the reference to the wayward file and close it... take a look to sys.last_traceback