I have code like this:
with open('foo.txt') as file:
...do something with file...
...move foo.txt to another place while it's still open...
Are there any problems with that?
This depends on the operating system. In Unix-based systems this is generally safe (you can move or even delete file while still having it open). Windows will produce Access Denied error if you try to move/delete an opened file.
So yes, the safest and portable way is to close the file first. Note that with clause closes the file automatically, so all you need is to perform the move outside of with block.
On Windows:
>>> import os
>>> with open('old.txt') as file:
os.rename('old.txt', 'new.txt')
Traceback (most recent call last):
File "<pyshell#4>", line 2, in <module>
os.rename('test.txt', 'newtest.txt')
WindowsError: [Error 32] The process cannot access the file because it is being used by another process
You can't move the file, because someone(you) is already holding it. You need to close the file before you can move it.
Best practice with file is close and move, because if you will not close then it might be create problem in some OS, like Windows.
To make your code more portable, close file before move.
yes , I was getting same issue,
I was doing some operation in for loop(if rows/content is valid then move destination folder)
outside for loop, if no header/no content/no rows then move to error folder
and I was getting error
I could resolve it using "read_obj.close" jus before filemove method
here is the snippet
with open(csv_file, 'r') as read_obj:
# pass the file object to reader() to get the reader object
csv_dic_reader = csv.DictReader(read_obj)
data = [row for row in csv_dic_reader]
for row in data:
read_obj.close()
filesMove(csv_file, destination_file_path, src_file_path)
if len(data) <= 1: # this will check no rows/content
read_obj.close()
filesMove(csv_file, error_file_path, src_file_path)
Related
I use pickle to save a Python dictionary to a file.
with open(FILENAME, 'wb') as f:
pickle.dump(DATA, f, protocol=pickle.HIGHEST_PROTOCOL)
It was all good until the disk run out of space on a shared server and my file became empty (0 byte).
Traceback (most recent call last):
File "****.py", line 81, in ****
with open(FILENAME, 'wb') as f:
OSError: [Errno 28] No space left on device
What is the best solution to prevent overwriting the previous data if the above error occurs?
Write to a temporary file (on the same filesystem!), and move it to the real destination when finished. Maybe with a fsync in between, to make sure the new data is really written.
If you go with #snild-dolkow's answer, keep in mind some interesting behaviour with shutil.move when working with files in different filesystems: How to move a file in Python?.
As has been noted in comments on other answers, shutil.move simply calls os.rename in most cases. However, if the destination is on a different disk than the source, it will instead copy and then delete the source file.
If you create the temporary file on the same filesystem like he suggests then you should be fine, but if you're worried, use os.rename instead, and it'll fail rather than copy cross filesystem (which could fail in the same way that you're trying to avoid).
I have code like this:
with open('foo.txt') as file:
...do something with file...
...move foo.txt to another place while it's still open...
Are there any problems with that?
This depends on the operating system. In Unix-based systems this is generally safe (you can move or even delete file while still having it open). Windows will produce Access Denied error if you try to move/delete an opened file.
So yes, the safest and portable way is to close the file first. Note that with clause closes the file automatically, so all you need is to perform the move outside of with block.
On Windows:
>>> import os
>>> with open('old.txt') as file:
os.rename('old.txt', 'new.txt')
Traceback (most recent call last):
File "<pyshell#4>", line 2, in <module>
os.rename('test.txt', 'newtest.txt')
WindowsError: [Error 32] The process cannot access the file because it is being used by another process
You can't move the file, because someone(you) is already holding it. You need to close the file before you can move it.
Best practice with file is close and move, because if you will not close then it might be create problem in some OS, like Windows.
To make your code more portable, close file before move.
yes , I was getting same issue,
I was doing some operation in for loop(if rows/content is valid then move destination folder)
outside for loop, if no header/no content/no rows then move to error folder
and I was getting error
I could resolve it using "read_obj.close" jus before filemove method
here is the snippet
with open(csv_file, 'r') as read_obj:
# pass the file object to reader() to get the reader object
csv_dic_reader = csv.DictReader(read_obj)
data = [row for row in csv_dic_reader]
for row in data:
read_obj.close()
filesMove(csv_file, destination_file_path, src_file_path)
if len(data) <= 1: # this will check no rows/content
read_obj.close()
filesMove(csv_file, error_file_path, src_file_path)
I cannot figure out what I am doing wrong. I am running some tests and writing the results to a file. The portion of the code which writes to a file is the following (in a class called Tester):
#staticmethod
def printHeader(resultsFileName):
if not os.path.isfile(resultsFileName):
# The file does not exist, thus
# we need to print the header
# Opens the results file
with open(resultsFileName,"a") as file:
# Prints the header
file.write("A long header")
#staticmethod
def printResults(resultsFileName,otherArguments):
# Prints the header if it does not exist
Tester.printHeader(resultsFileName)
# Prints the results
with open(resultsFileName,"a") as file:
file.write(otherArguments)
Sometimes I get this error:
Traceback (most recent call last):
File "main.py", line 74, in <module>
File "tester.py", line 88, in methodOne
File "tester.py", line 441, in printResults
File "tester.py", line 428, in printHeader
IOError: [Errno 13] Permission denied: 'results.txt'
while other times it runs smoothly. I cannot figure out where the problem is. Any ideas?
NOTE1: I have rwxpermission on the directory where the file is written.
NOTE2: The error happens after several lines of results have already been written. Thus, it happens when the code is checking whether the header should be printed or not (but it is not supposed to print it, since the file exists).
UPDATE 1:
As suggested, I have changed my code to avoid opening and closing the file multiple times. Now it writes everything in one shot. This is the updated code:
#staticmethod
def printResults(resultsFileName,otherArguments):
# Prints the header if it does not exist
if not os.path.exists(resultsFileName):
# The file does not exist, thus
# we need to print the header
# Opens the results file
# HERE IS WHERE ERRNO 13 HAPPENS
# STRANGELY, THE FILE DOES EXIST
# AS SEVERAL LINES OF RESULTS
# HAVE ALREADY BEEN WRITTEN
with open(resultsFileName,"w") as file:
# Prints the header
file.write("A suitable header")
# Prints the results
file.write(otherArguments)
else:
# Prints the results
with open(resultsFileName,"a") as file:
file.write(otherArguments)
It seems that os.path.exists() at some point returns FALSEeven if the file does exist. Probably, there is something revoking me permission to write (perhaps the file is not properly closed after writing?).
The explanation of os.path.exists() says that:
On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
UPDATE 2
I have changed my code to the following, to avoid os.path.isfile():
# Opens the results file
with open(resultsFileName,"a") as file:
if file.tell() == 0:
# Prints the header
file.write("Header")
# Prints the results
file.write(otherArguments)
file.close()
else:
# Prints the results
file.write(otherArguments)
file.close()
Nevertheless, ERRNO 13 happens at with open(resultsFileName,"a") as file:.
I have rwpermissions both on the folder and on the file, on which several lines are written before the error happens. The OS is Linux.
Try closing the file result.txt before running the program (I'd its open).
Check the writing permissions.
Another cause might be writing into a .txt file in a directory without those permissions. Try running python from the specified directory.
os.path.isfile(path) returns True if path exists AND is a file AND you have (at least) read permissions on it. IOW, it returns False if path exists but is a directory and/or you don't have permissions on it. So your test is wrong from the start - os.path.exists() would be a better choice. But anyway: those operations are not atomic so the file could well be created between the moment where you test for it's existence and the moment where you try to open it, so the whole design is actually unsafe. Also, stating, opening and closing a file are all costly operations, so I suggest - if possible - that you rethink your whole design to open a file only once and only close it once when you're done.
You can check if the file is open with another program. If so, you can try closing them all.
Cheers!
Surely it will work
When running the following code:
KEEP=[]
for file in glob.glob("./KEEP/thing*.[tT][iI][fF]"):
m = pattern.search(file)
filename=m.group(1)
keep=Image.open(file)
keep.load()
KEEP.append(keep)
KEEP_NAMES.append(filename)
keep.close()
over more than a thousand files, I get the error message:
Traceback (most recent call last):
File "/hom/yannis/texmf/python/remove-harakat.py", line 123, in <module>
File "/usr/local/lib/python2.7/dist-packages/PIL/Image.py", line 2237, in open
IOError: [Errno 24] Too many open files: './KEEP/thing1118_26.TIF'
I don't understand why this is happening, since I'm load()ing and then close()ing all files, why should they remain open?
Is there a solution to this problem, other than reducing the number of files (which is not an option for me)? Some way to close them after their contents have been read in memory?
This may be a bug with the Image.load method - see Pillow issue #1144. I ran into the same too many open files error - see #1237.
My work-around was to load the image into a temp object, make a copy, and then explicitly close the temp. For your code it would look something like this:
KEEP=[]
for file in glob.glob("./KEEP/thing*.[tT][iI][fF]"):
m = pattern.search(file)
filename = m.group(1)
temp = Image.open(file)
keep = temp.copy()
KEEP.append(keep)
KEEP_NAMES.append(filename)
temp.close()
I also ran into this issue and resolved the issue with a slightly different approach.
This workaround uses copy.deepcopy(), which is based on similar logic of #indirectlylit's solution but avoids creating of temp. See code snippet below
import copy
KEEP=[]
for file in glob.glob("./KEEP/thing*.[tT][iI][fF]"):
m = pattern.search(file)
filename = m.group(1)
keep = copy.deepcopy(Image.open(file))
KEEP.append(keep)
KEEP_NAMES.append(filename)
The intent is to look in a json file in the directory above the script and load up what it finds in that file. This is what I've got:
import os
import json
settings_file = '/home/me/foo/bar.txt'
root = os.path.dirname(os.path.dirname(os.path.abspath(settings_file))) # '/home/me'
target = os.path.join(root,'.extras.txt') # '/home/me/.extras.txt'
db_file= open(target)
databases = json.load(db_file) # works, returns object
databases2 = json.load(open(target)) # equivalent to above, also works
# try to condense code, lose pointless variables target and file
databases3 = json.load(open(os.path.join(root,'.extras.txt'))) # equivalent (I thought!) to above, doesn't work.
So... why doesn't the all-at-once, no holding variables version work? Oh, the error returned is (now in it's entirety):
$ ./json_test.py
Traceback (most recent call last):
File "./json_test.py", line 69, in <module>
databases = json.load(open(os.path.join(root,'/.extras.txt')))
IOError: [Errno 2] No such file or directory: '/.extras.txt'
And to satisfy S.Lott's well-intentioned advice... it doesn't matter what target is set to. The databases and databases2 populate correctly while databases3 does not. target exists, is readable and contains what json expects to see. I suspect there's something I don't understand about the nature of stringing commands together... I can make the code work, was just wondering why the concise (or complex?) version failed.
Code looks fine, make sure referenced files are in the appropriate places. Given your code that includes target/file variable assignment, full path to .extras.txt is
/home/me/.extras.txt
You need to do:
file = open(target, 'w')
because by default open will try to open the file in read mode (r) but you need to open it in w (write) mode if you want it to be created.
Also, I would not use the variable name file since it is also a type (<type 'file'>) in python.
You could add the write-mode flag to this line as well:
databases = json.load(open(os.path.join(root,'.extras.txt'), 'w'))
because from the limited information we have in the question it appears your /.extras file does not previously exist.
Final note, you are losing the handle to your open file in this line (since you are not storing it in your file variable):
databases = json.load(open(os.path.join(root,'.extras.txt')))
How do you intend to close the file when you're finished with it?
You could do this with a context manager (python >=2.6 or 2.5 if import with_statement used):
with open(os.path.join(root,'.extras.txt'), 'w') as f:
databases = json.load(f)
which will take care of closing the file for you.