I'm building a basic file server and my program cannot find files.
def sendfile(sock, myfile):
print 'Serving file:', myfile
print 'File exists?:', os.path.exists(myfile)
path = os.path.normpath(os.path.join(os.getcwd(), myfile))
print 'Serving file:', path
print 'File exists?:', os.path.exists(path)
These always return False even though the 'myfile' and 'path' are correct [the file is in the same directory as the server program].
IDLE works fine, but without passing to functions.
>>> print os.path.exists("/user/server/foo.txt")
True
What have I missed?
[EDIT:] Output:
Serving file: foo.txt
File exists?: False
Serving file: /user/server/foo.txt
File exists?: False
I'm almost 100% sure you're not sanitizing your input before you check if the path exists. Here's something I ran in my interpreter:
>>> from os.path import exists
>>> exists('dog.png')
True
>>> exists('dog.png\n')
False
Try stripping whitespace on path before you check if it exists.
If you read the Python documentation of os.path.exists(), it says that there are specific cases in which a file or folder exists but os.path.exists() returns false:
Return True if path refers to an existing path or an open file
descriptor. Returns False for broken symbolic links. On some
platforms, this function may return False if permission is not granted
to execute os.stat() on the requested file, even if the path
physically exists.
Not directly answering the question stated here, but I found this topic when os.path.exists() was keep giving me "False", even after using strip() or os.path.join(). In my case, I was using ~ (tylda) to point to the home directory like this:
fileName = "~/path/to/file.txt"
The best way to fix this was to use os.path.expanduser(fileName), and then check if the file exists. Alternatively restore absolute path with os.path.abspath(), followed by removig of "~" from the path (but this solution will not work in all scenarios).
os.path.exists(os.path.abspath(fileName).replace("~",""))
Maybe this will be helpful to someone.
As said in this answer
this mainly occurs do to white spaces.
I also faced this issue. It took me a lot of time to figure this out.
python has a function called strip() which removes white spaces.
if variable path_to_file consists the path to the actual file then try using
if path.exists(path_to_file.strip())
print("file exists")
else:
print("file doesn't exist")
this worked for me.
This may not answer your question directly, but you could go with the "try/except" method:
Whatever function uses the file should return an exception if the file doesn't exist (especially if it's a built-in function), and you can act accordingly. Then you have no need to check whether or not the file exists yourself. Dangerous? Perhaps, but that depends on what you are actually trying to do.
This honestly should be considered a bug if it fails due to spaces.
I have found that in some cases, the result will be spurious, depending somehow on the state of the file server. This does not happen all the time, only once in a while.
I found that with at least a 10 second delay, I could avoid a failure.
In this case, I am opening zip archives repeatedly to access specific zipped files. Prior to the attempt to open it, it checks that the path exists (and try used below this because of this strange issue). If it fails, then it waits in a loop with increasing delay. I found that it usually winds up finding the file exists again after 4 loops (10 sec delay).
Here is the output of my loop print statements:
Archive r:\ballotimagearchive\ca_san_francisco_2020_pri\d03.zip does not exist according to os.path.exists().
Waiting 1 seconds
Waiting 2 seconds
Waiting 3 seconds
Waiting 4 seconds
After wait of 10 secs, r:\ballotimagearchive\ca_san_francisco_2020_pri\d03.zip now exists according to os.path.exists().
and the code segment that produces this.
if os.path.isfile(source_path):
print(f"Verified that {source_path} exists.")
else:
print(f"Archive {source_path} does not exist according to os.path.exists().")
# this may be a spurious problem related to using a file server.
tot_time = 0
for i in range(1,20):
print(f"Waiting {i} seconds")
time.sleep(i)
tot_time += i
if os.path.isfile(source_path):
print(f"After wait of {tot_time} secs, {source_path} now exists according to os.path.exists().")
break
else:
print(f"After wait of {tot_time} secs, {source_path} still not found according to os.path.exists().")
sys.exit(1)
os.path.exists returns false also if the given path is longer than 256 characters.
I was getting this problem fairly often until I tried adding getcwd(), now it never fails
os.path.join(os.getcwd(), source_file)
I had this same issue and found the following solution:
Using the OS's native directory separator '\' or '/' even on Windows, using the forward slash in all occasions works just fine for me. os.sep can help.
In addition to this, sanitizing the path string a bit similar to this:
import re
from os import path
strPath = "c:/dir1/dir2/dir3/dir4/important-file.ext"
strPath = re.escape(strPath)
bTest = os.access(strPath, os.F_OK)
# or the classic
bTest = path.exists(strPath)
print(bTest)
I want to use the file in the Downloads folder on ubuntu, but os.path.exists can't find using the absolute path ~/Downloads/filename.txt.
Then I use os.path.abspath('') to get the root path /home/yourPCname/ and replace ~, it works!
Related
I am getting a rather bizarre race condition in Mac OS X with Python (I've only tested Python 3.3). I am making several temporary directories, writing things to them, and then clearing them. Something along the lines of
while running:
(do something)
tempdir = mkdtemp('name')
try:
(write some stuff to tempdir)
finally:
shutil.rmtree(tempdir)
However, in some of the later loops of the (write some stuff to tempdir), I get errors like
with open(os.path.join("/var/folders/yc/8wpl9rlx47qgzxqpcf003k280000gn/T/tmp0fh2ztname", "file"), 'w', encoding='utf-8') as fn:
FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/yc/8wpl9rlx47qgzxqpcf003k280000gn/T/tmpups5dpname/file'
(I've inlined the temp dir path for clarity)
Notice how the path being opened is not the same as the path that it can't find. In each case, the path in the error message is the temporary directory from the previous iteration of the loop.
The error is reproducible most of the time in the same place (after about the fourth iteration), but not every time.
EDIT: I just realized this is probably relevant. The (write some stuff to tempdir) stuff actually happens in a subprocess. This is how I am sure about the tempdir path, I have to pass it on to the subprocess (I actually lied about the "clarity" bit, I am actually writing out a Python file with that exact with open line). This is how I know for sure that the tempdir path is indeed different from the one being used.
I figured it out. It turns out it has nothing to do with mkdtemp (a sigh of relief that Mac OS X and Python are doing the right things there).
The problem is that I was writing out the code to a file, including the with open(os.path.join("/var/folders/yc/8wpl9rlx47qgzxqpcf003k280000gn/T/tmp0fh2ztname", "file"), 'w', encoding='utf-8') as fn: bit, and running it in a subprocess. The issue was that I was using the same file each time, and the .pyc files were not being invalidated correctly.
The error message was confusing because when Python generates a traceback, it reads the .py file (where the actual code is), but what is actually run is the .pyc file.
If I understand http://nedbatchelder.com/blog/200804/the_structure_of_pyc_files.html correctly, the timestamps in .pyc files ony have one second granularity (this explains why this was reproducible in the same place each time: the same fourth item in the loop ran in under a second).
The solution was to explicitly delete the .pyc files when writing out the file (in other circumstances you could also write out to a temp file itself, but in my case I needed the file to be importable under the same name).
Something along the lines of
if sys.version_info >= (3,):
os.unlink(os.path.join(path_to_file, '__pycache__', 'file.cpython-%s%s.pyc' % sys.version_info[:2]))
os.unlink(os.path.join(path_to_file, '__pycache__', 'file.cpython-%s%s.pyo' % sys.version_info[:2]))
else:
os.unlink(os.path.join(path_to_file, 'file.pyc'))
os.unlink(os.path.join(path_to_file, 'file.pyo'))
import os
path=r'C:\Users\User\Documents\prog'
folderlist=os.listdir(path)
def is_file_contain_word(folderlist,query_word):
for file in folderlist:
if query_word in open(folderlist).read():
print (file)
print("finishing searching.")
query_word=input("please enter the keyword:")
is_file_contain_word(folderlist,query_word)
This is what I have so far. It has returned that I have type error:
invalid file: ['1.doc', '1.odt', 'new.txt']
I got this error when I swapped path with folderlist in the open
PermissionError: [Errno 13] Permission denied: 'C:\\Users\\God\\Documents\\prog'
import os
This is my new code:
import os
path=r'C:\Users\God\Documents\prog'
folderlist=os.listdir(path)
print(folderlist)
def is_file_contain_word(folderlist,query_word):
for i in os.listdir(path):
if query_word in i:
print("Found %s" %query_word)
else:
print("not found")
query_word=input("please enter the keyword:")
is_file_contain_word(path,query_word)
This goes through each of the 3 files to search for the filename. It only stops when it finds it or it goes through it.
You are making multiple mistakes here:
First, try to indent your code correctly:
def is_file_contain_word(folderlist,query_word):
for file in folderlist:
if query_word in open(folderlist).read():
print (file)
print("finishing searching.")
Then the real problem.
open(folderlist) should be open(file), as I believe that's your intention.
And that's one reason why the program won't run: open(folderlist) opens C:\Users\User\Documents\prog, which is a directory and will cause problem.(You probably thought this line opens all files contained in that folder)
BTW, don't use "file" as a var, it's a reserved word in python.
However, even though you fix this part, the original function still won't work.
def is_file_contain_word(folderlist,query_word):
for f in folderlist:
if query_word in open(f).read():
print (f)
print("finishing searching.")
Why? If you check this folderlist(folderlist=os.listdir(path)), it would be something like:
["Prog's Movies", "Homework3.pdf", "blahblah.txt"]
Which is problematic because first, you can't open them because they aren't absolute path(Like C:\Users\God\Documents\prog\blahblah.txt), second, listdir would also list all directories(In this case "Prog's Movies" directory), which you can't use open to open them.
At last, if you are running this on command line, don't make two folderlist vars. I know in this case it won't cause problem, but it dose reduce your code's readibility and seem to confuse you a little bit.
How to fix it?
How to list all files of a directory?
Here is a nice answer to how to open all files in a directory correctly.
enter code here
import os
os.chdir('I:\\Movies')
files = os.popen('dir').readlines()
disk = raw_input("Enter the disk: ")
while disk != "done":
os.chdir(disk + ':\\' + 'Movies')
files_in_disk = os.popen('dir').readlines()
for each_file in files_in_disk:
for item in files:
if ' '.join(each_file.split()[3:]) in item:
each_file = ' '.join(each_file.split()[3:])
os.system('rmdir /q /s ' + '"' + each_file + '"')
break
disk = raw_input("Enter the disk: ")
I had two copies of the same movies on two different drives, I wrote this script to delete one of the copy. But on E drive it erased nearly all of my files, why did this happen can someone please point out my mistake.
I think something here is not doing what you expect:
if ' '.join(each_file.split()[3:]) in item:
If a any file is has less than 4 space-delimited parts, the first bit of the if will be the empty string, and this will return true.
The problem is your loop. For each file in the E:\Movies, it checks if any file in I:\Movies matches (well, everything past the third word). If one of the files in I:\Movies happens to have less than 4 words (not entirely implausible), then the if will be true on every run.
I'm not sure what the intent is here, but this is my best guess as to what may be causing the problem.
Your mistake was not initially running this program with a print each_file statement rather than immediately jumping to the rmdir command.
Though this may read as a snarky answer, it is truly meant to be helpful. Whenever making irreversible changes (like deleting items from a file system or DB), one should always take some step to verify that the appropriate instructions are being generated/executed.
According to Microsft's TechNet article about rmdir:
/s : Removes the specified directory and all subdirectories including any files. Use /s to remove a tree.
So, if according to the other answers it is possible to supply non-matching file paths to rmdir it is not very difficult to delete whole subtrees on the disk. Especially if the list of files also contains subdirectories that point to parent subdirectories (for instance i:\movies..), you could be in a world of hurt in cases like that.
But I don't have access to a Windows machine with Python installed to prove it.
I am trying to write a detector that checks if a certain directory can be deleted using shutil.rmtree. I have a partial code finished as below that now works partial.
This code is now able to gives warning when any .exe files under the target folder is still running. But, this code is not yet able to flag warnings if any particular file under a folder is opened by an editor (which is another cause that makes a directory not deletable). Any guidance will be appreciated. Thanks in advance
Note: I've used open method to check for any locked file.
def list_locked_files(dir):
isLocked = False
for name in os.listdir(dir):
uni_name = unicode(name)
fullname = dir + u'/' + uni_name
if os.path.isdir(fullname):
list_locked_files(fullname)
else:
try:
f = open(fullname, 'r+')
f.close()
except IOError:
print fullname + u' is locked!'
isLocked = True
if isLocked is True:
print u'Please close the files/dir above !'
sys.exit(0)
It is not necessarily possible to determine whether a file deletion will succeed or fail on Windows. The file could be opened in a fully permissive share mode which means another attempt to open the file will succeed (no matter what kind of access you request).
The only way to tell whether a file can be deleted is to actually try it.
Even if there were an accurate way to tell beforehand, once you get the information it is instantly out of date. For example, after you call list_locked_files, a program could open another file in that directory which would cause rmtree() to fail.
I have some xml-configuration files that we create in a Windows environment but is deployed on Linux. These configuration files reference each other with filepaths. We've had problems with case-sensitivity and trailing spaces before, and I'd like to write a script that checks for these problems. We have Cygwin if that helps.
Example:
Let's say I have a reference to the file foo/bar/baz.xml, I'd do this
<someTag fileref="foo/bar/baz.xml" />
Now if we by mistake do this:
<someTag fileref="fOo/baR/baz.Xml " />
It will still work on Windows, but it will fail on Linux.
What I want to do is detect these cases where the file reference in these files don't match the real file with respect to case sensitivity.
os.listdir on a directory, in all case-preserving filesystems (including those on Windows), returns the actual case for the filenames in the directory you're listing.
So you need to do this check at each level of the path:
def onelevelok(parent, thislevel):
for fn in os.listdir(parent):
if fn.lower() == thislevel.lower():
return fn == thislevel
raise ValueError('No %r in dir %r!' % (
thislevel, parent))
where I'm assuming that the complete absence of any case variation of a name is a different kind of error, and using an exception for that; and, for the whole path (assuming no drive letters or UNC that wouldn't translate to Windows anyway):
def allpathok(path):
levels = os.path.split(path)
if os.path.isabs(path):
top = ['/']
else:
top = ['.']
return all(onelevelok(p, t)
for p, t in zip(top+levels, levels))
You may need to adapt this if , e.g., foo/bar is not to be taken to mean that foo is in the current directory, but somewhere else; or, of course, if UNC or drive letters are in fact needed (but as I mentioned translating them to Linux is not trivial anyway;-).
Implementation notes: I'm taking advantage of the fact that zip just drop "extra entries" beyond the length of the shortest of the sequences it's zipping; so I don't need to explicitly slice off the "leaf" (last entry) from levels in the first argument, zip does it for me. all will short circuit where it can, returning False as soon as it detects a false value, so it's just as good as an explicit loop but faster and more concise.
it's hard to judge what exactly your problem is, but if you apply os.path.normcase along with str.stript before saving your file name, it should solve all your problems.
as I said in comment, it's not clear how are you ending up with such a mistake. However, it would be trivial to check for existing file, as long as you have some sensible convention (all file names are lower case, for example):
try:
open(fname)
except IOError:
open(fname.lower())