I have an old script doing its duty for years now, which suddenly failed, and I don't see how it can even get to the point where it fails, since I thought to have that case eliminated:
It should go through a list of files (content of a directory), and if the current file ends with .sum, it should check if there is a file with the same name minus the .sum ending (like file1.txt.sum and file1.txt).
If both files exist, it should make a checksum of file.txt` and compare it to the content of the checksum file.
Only if this check is successful, there will be further operations ending with both files getting deleted from the directory.
for file in filelist:
checksum_downloaded_file, checksum_from_checksumfile = '0', '1'
if file.endswith('.sum'):
# read checksum from checksum file
f_sum = open(SAVETO + file, 'rb')
with f_sum:
checksum_from_checksumfile = f_sum.readline().split(' ')[0].rstrip().lower()
# checksum() the file
try:
f = open(SAVETO + file[:-4], 'rb')
except:
continue
with f:
checksum_downloaded_file = hashlib.sha256(f.read()).hexdigest()
# compare delivered checksum with selfmade
if checksum_downloaded_file == checksum_from_checksumfile:
[...]
sftp.remove(file)
sftp.remove(file[:-4])
I thought the check if a file exitsts with the same name as the checksum file except the .sum ending would be done in a reliable way by the
try:
f = open(SAVETO + file[:-4], 'rb')
except:
continue
block. But today the script failed while trying to delete both files at the end of the script, because the file without .sum ending didn't exist. How can this happen? Doesn't continue do its job in that context?
EDIT:
Since it seems there is nothing logically wrong with the code, the error might lay outside (although I don't know how it could). Therefore further informations: The actual source of the files is a directory reached via SFTP (pysftp used). The filelist is filled by filelist = sftp.listdir(), and all files are downloaded (to SAVETO) before the problematic block starts. They should get removed on the SFTP server with sftp.remove(file[:-4]) afterwards and this is where the script fails throwing
Traceback (most recent call last):
File "script.py", line 71, in <module>
sftp.remove(file[:-4])
File "/home/alba/.local/lib/python2.7/site-packages/pysftp/__init__.py", line 728, in remove
self._sftp.remove(remotefile)
File "/home/alba/.local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 365, in remove
self._request(CMD_REMOVE, path)
File "/home/alba/.local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 780, in _request
return self._read_response(num)
File "/home/alba/.local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 832, in _read_response
self._convert_status(msg)
File "/home/alba/.local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 861, in _convert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] No such file or directory
EDIT2: Still can't let this go. I guess if the file got deleted on the server after the script downloaded it, the script would fail like this because of course the file wouldn't be there then anymore. But the script failed every time from then on. Doesn't make sense..
EDIT3: Solved in answer. Thank you for your time and patience.
My first approach to search for the bug in the code gave the title to this thread, and it turned out to be wrong.
The bug happens when the input directory gets fed with a checksum file, which already has been fed in the past, and back then correctly, ergo with a correspondent data file, but now without that correspondent data file:
The (again) downloaded checksum file gets evaluated and the script looks for a correspondent data file, which is still present in the working directory from the past. It runs its operations and finally tries to remove the checksum and the data file from the input directory, which fails for the data file.
An unforeseen, unhandled case; out of the initial specifications, but still embarrassing. I learned from this to sort the list of debugging approaches more realistically.
Related
I am using a watchdog module to move video files to a different folder. But the problem is the move process starts as soon as the file is created and is still under copy process from its origin location. To overcome this, I did a file size check, but windows is not refreshing the filesize in real time and hence the file size check is not proving to be useful.
Is there another way to check if the file is fully copied (maybe if file can be read/opened?) prior to moving the file. I tried the code below but to no avail.
while True:
try:
with open(os.path.basename(event.src_path)) as f:
s = f.read()
print ("can read the file now")
except IOError:
print ("cannot read the file")
#shutil.move operation here
Any ideas will be appreciated.
I am trying to implement a quick code that checks to see if an Excel file is open. When the file is open, the IOError works and tells me the file is open. I can close and reopen the file at this point. If I do not have the file open, and either by calling a subprocess.call() or simply clicking on it, I an error.
Why does the try corrupt it if there is no error raises?
I have tried to close the file after the try but this does not work either.
file_path = (r'C:\users\Desktop\Build-Temp.xlsx')
if os.path.exists(file_path) is True:
report_closed = True
try:
report_opcl = open(file_path, 'w+')
# report_opcl.close()
except IOError:
print("file open already")
report_closed = False
# if report_closed is True:
# with report_opcl:
# subprocess.call(file_path,shell=True)
Errors
Excel cannot open the file 'Build-Temp.xlsx' because the file format or file extension is not valid. Verify that the file has not been corrupted and that the file extension matches the format of the file.
Opening a file with 'w' will delete the file contents. The docs call this "truncating". You might avoid the corruption using 'a' instead (not tested).
Anyway this is not going to work. By the time you check that report_closed is true another process might have opened the file.
I think you are trying to solve an unsolvable problem. There is no way to make sure that a file (identified by its path) will be available to another process in the future.
I cannot figure out what I am doing wrong. I am running some tests and writing the results to a file. The portion of the code which writes to a file is the following (in a class called Tester):
#staticmethod
def printHeader(resultsFileName):
if not os.path.isfile(resultsFileName):
# The file does not exist, thus
# we need to print the header
# Opens the results file
with open(resultsFileName,"a") as file:
# Prints the header
file.write("A long header")
#staticmethod
def printResults(resultsFileName,otherArguments):
# Prints the header if it does not exist
Tester.printHeader(resultsFileName)
# Prints the results
with open(resultsFileName,"a") as file:
file.write(otherArguments)
Sometimes I get this error:
Traceback (most recent call last):
File "main.py", line 74, in <module>
File "tester.py", line 88, in methodOne
File "tester.py", line 441, in printResults
File "tester.py", line 428, in printHeader
IOError: [Errno 13] Permission denied: 'results.txt'
while other times it runs smoothly. I cannot figure out where the problem is. Any ideas?
NOTE1: I have rwxpermission on the directory where the file is written.
NOTE2: The error happens after several lines of results have already been written. Thus, it happens when the code is checking whether the header should be printed or not (but it is not supposed to print it, since the file exists).
UPDATE 1:
As suggested, I have changed my code to avoid opening and closing the file multiple times. Now it writes everything in one shot. This is the updated code:
#staticmethod
def printResults(resultsFileName,otherArguments):
# Prints the header if it does not exist
if not os.path.exists(resultsFileName):
# The file does not exist, thus
# we need to print the header
# Opens the results file
# HERE IS WHERE ERRNO 13 HAPPENS
# STRANGELY, THE FILE DOES EXIST
# AS SEVERAL LINES OF RESULTS
# HAVE ALREADY BEEN WRITTEN
with open(resultsFileName,"w") as file:
# Prints the header
file.write("A suitable header")
# Prints the results
file.write(otherArguments)
else:
# Prints the results
with open(resultsFileName,"a") as file:
file.write(otherArguments)
It seems that os.path.exists() at some point returns FALSEeven if the file does exist. Probably, there is something revoking me permission to write (perhaps the file is not properly closed after writing?).
The explanation of os.path.exists() says that:
On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
UPDATE 2
I have changed my code to the following, to avoid os.path.isfile():
# Opens the results file
with open(resultsFileName,"a") as file:
if file.tell() == 0:
# Prints the header
file.write("Header")
# Prints the results
file.write(otherArguments)
file.close()
else:
# Prints the results
file.write(otherArguments)
file.close()
Nevertheless, ERRNO 13 happens at with open(resultsFileName,"a") as file:.
I have rwpermissions both on the folder and on the file, on which several lines are written before the error happens. The OS is Linux.
Try closing the file result.txt before running the program (I'd its open).
Check the writing permissions.
Another cause might be writing into a .txt file in a directory without those permissions. Try running python from the specified directory.
os.path.isfile(path) returns True if path exists AND is a file AND you have (at least) read permissions on it. IOW, it returns False if path exists but is a directory and/or you don't have permissions on it. So your test is wrong from the start - os.path.exists() would be a better choice. But anyway: those operations are not atomic so the file could well be created between the moment where you test for it's existence and the moment where you try to open it, so the whole design is actually unsafe. Also, stating, opening and closing a file are all costly operations, so I suggest - if possible - that you rethink your whole design to open a file only once and only close it once when you're done.
You can check if the file is open with another program. If so, you can try closing them all.
Cheers!
Surely it will work
I'm supposed to upload a database for an assignment, but I'm having a problem. These are the instructions:
This application will read the mailbox data (mbox.txt) count up the
number email messages per organization (i.e. domain name of the email
address) using a database with the following schema to maintain the
counts.
CREATE TABLE Counts (org TEXT, count INTEGER) When you have run the
program on mbox.txt upload the resulting database file above for
grading. If you run the program multiple times in testing or with
different files, make sure to empty out the data before each run.
The data file for this application is the same as in previous
assignments: http://www.pythonlearn.com/code/mbox.txt.
Because the sample code is using an UPDATE statement and committing
the results to the database as each record is read in the loop, it
might take as long as a few minutes to process all the data. The
commit insists on completely writing all the data to disk every time
it is called.
The error message that it keeps sending me is:
Traceback (most recent call last):
File "<pyshell#11>", line 1, in <module>
fh = open(fname)
IOError: [Errno 2] No such file or directory: 'mbox.txt'
I saved them both in the same folder.
Can anybody help with this?
The code that I entered is here
Your code is having problem to find the file mbox.txt. It has nothing to do
with anything in the database as you did not run so far yet.
Good practice (at least during development) is to make sure, the things you
hope are true are really true. For this purpose I would use following code
which makes sure, the file really exists.
import os.path
fname = "mbox.txt"
assert os.path.exists(fname), "The file shall exist"
If you happen to run the code in situation, the file does not exist, it will throw an
AssertionError telling you what went wrong.
This exception is very practical as it will tell you quickly what assumption does not holds true and
you know, what to fix.
Your code is looking for a file called mbox.txt and not finding it. My guess is that open(fname) is looking for mbox.txt in the current directory, but the code is being run from a different directory.
Something like this might help resolve your issue:
import os
# figure out directory of the Python file
mdir = os.path.dirname(os.path.abspath(__file__))
# assuming that mbox.txt is in the same folder as the Python file,
# get the path to that file
mpath = os.path.join(mdir, 'mbox.txt')
# open the file
with open(mpath, 'r') as fh:
# ...
Another approach is using command line arguments. Perhaps there are other files like mbox.txt that you want to work with. In these cases, you could accept the path to mbox.txt as a command line option:
import argparse
argp = argparse.ArgumentParser(description='foo the mbox')
argp.add_argument('mbox_path', help='Path to mbox file')
opts = argp.parse_args()
with open(opts.mbox_path, 'r') as fh:
# ...
Or get fancier and use argparse.FileType for the type argument to argparse.add_argument.
while True:
try:
enterName = input("Enter the name of the file:") + ".txt"
latinFile = open(enterName,"r")
read = latinFile.readlines()
for lines in read:
store.append(lines.strip())
checkSquare (store)
print ("File:")
for content in store:
print (content)
saveData(store)
I think the problem is with the code for saving the file contents.
The aim is for the code to open the file,read the contents and it checks to see if the format of the file is correct etc and then if that is all true and it works, it will save the file again (saveData) but it will rename the file so that it says in the directory that it has been validated.
However, the code isn't working ( the os.rename part) and also I keep getting a permissionerror and I don't know how to fix it.
Error message:
Traceback (most recent call last):
File "C:\Users\---\Desktop\python\idkll.py", line 44,in <module>
saveData(store)
File "C:\Users\---\Desktop\python\idkll.py", line 18, in saveData
os.rename (fileName,"VALIDATED" + fileName)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'OPENEDve.txt'
You should close your files as soon as you finish reading from them or writing to them. And you should close the file before attempting to rename it. You can either use the file's .close() method, or use a with statement so that files get closed automatically.
Incidentally, your saveData() function should probably not prompt the user for the filename: just pass it the name you already have in enterName. Also, prepending "VALIDATED" to the filename is not a good strategy. It's ok if you're just using relative filenames in the current directory, but it will make a mess of a proper file path.
In response to the update:
saveFile does not contain a file name string, it's a Python object representing your open file (also known as a file handle). The name of that file is the string in enterName. So you need to do something like
os.rename(enterName, enterName + "VALIDATED")
Just swap os.rename (fileName,"VALIDATED" + fileName) and file.close() lines inside saveData() function.
You should close the file prior to trying to rename it. The error is because your program is actually using the file it is trying to rename.