I try to find a solution for the following problems but can't find any good solution.
I have a folder with subfolder and files in it.
Some of the files may be in use by another process (the other process is writing data to the data (a .mdf file)).
I simply want to check if the files are in use or not.
Structure:
A_Folder
Setup
Data1
.mdf-file1
.mdf-file2
Data2
Data3
Evaluation
something like:
def file_in_use():
*your solution*
for file in folder:
if file_in_use(file):
print("file in use")
break
I'm Using Win10, PyCharm and a venv.
I tried so far form other "solutions":
psutil (works, but is too slow)
open(), os.rename - won't work for me
subprocess wont work either -cant find my filename: using the method from Amit
Gupta from my link down below, file looks like this: "C:\Data\S_t_h\S-t-h\H001.mdf"
basically I tried everything from this question:
Check if a file is not open nor being used by another process
from subprocess import check_output, Popen, PIPE
src = r"C:\Data\S_t_h\S-t-h\H001.mdf"
files_in_use = False
def file_in_use(src):
try:
lsout = Popen(['lsof', src], stdout=PIPE, shell=False)
check_output(["grep", src], stdin=lsout.stdout, shell=False)
except:
return False
return True
if file_in_use(src):
files_in_use = True
and im getting:
FileNotFoundError: [WinError 2] The system cannot find the file specified
this link suggesting setting
winerror-2-the-system-cannot-find-the-file-specified-python
shell=True
Im getting "lsof" and "grep" cant be found or are wrong now.
Here the psutil method that works for me, but is too slow (~10 Seconds)
import psutil
src = r"C:\Data\S_t_h\S-t-h\H001.mdf"
def has_handle(src):
for proc in psutil.process_iter():
try:
for item in proc.open_files():
if src == item.path:
return True
except Exception:
pass
return False
print(has_handle(src))
My Solution:
Sorry for the delayed answer.
It simply worked with:
try:
os.rename(src, src)
return False
except OSError: # file is in use
return True
I made it more complicated than it actually was i guess.
But thank you guys anyway for your feedback and critizism.
I have an os.remove() in my code that sometimes, when ran locally, fails due to OSError 13 - Permission Denied - thus I've set up a try-except. My automated testing (Travis CI) is ran on Linux VM instances, so I don't know how to make os.remove fail there for sake of coverage.
What are my options - how do I force the except block to execute? Alternatively, how do I delete-protect a file with Python?
Note: Removing it in the test code before calling the test method isn't an option; the method itself fetches files to be removed:
from pathlib import Path
paths = [str(x) for x in Path("directory/").iterdir() if 'abc' in x.stem]
if len(paths) > 0: # if files are removed beforehand, len(paths) == 0
try:
[os.remove(p) for p in paths]
except:
pass # stuff here
You can use unittest.mock.patch to patch os.remove and specify OSError as a side_effect:
from unittest.mock import patch
...
with patch('os.remove') as mock_remove:
mock_remove.side_effect = OSError('Permission Denied')
try:
[os.remove(p) for p in paths]
except OSError as e:
pass # handle error here
I have a problem. I run tests with the help of the question.
In the beginning, the test calls a method that causes me to enter the address of the database (where I am testing). However, I am getting an error:
element = "http://" +sys.stdin.readline()../../python/lib/python3.6/site-packages/_pytest/capture.py:702: in read
raise IOError ("reading from stdin while output is captured")
E OSError: reading from stdin while output is captured
below my code.
#staticmethod
def setAddress():
print("Give database:")
element = "http://"+sys.stdin.readline()
return element
I need to addres add http. How I can change my code? Thanks for help!
Set an environment variable when running tests in your shell:
DB_URL=http://xxx pytest
and then retrieve it in your tests:
import os
…
db_url = os.getenv('DB_URL')
Lets say I have directories like:
foo/bar/
bar is chmod 777 and foo is 000.
When I call os.path.isdir('foo/bar') it returns just False, without any Permission Denied Exception or anything, why is it like that? Shouldn't it return True?
If you are not root then you cannot access foo. Therefore you can't check if foo/bar exists and it returns False because it cannot find a directory with that name (as it cannot access the parent directory).
os.path.isdir can return True or False, but cannot raise an exception.
So if the directory cannot be accessed (because parent directory doesn't have traversing rights), it returns False.
If you want an exception, try using os.chdir or os.listdir that are designed to raise exceptions.
You could implement a try/except block:
import os
path = '/foo/bar'
if os.path.exists(path):
try:
os.chdir(path)
except PermissionError:
print ("Access Denied To:", path)
If I call os.stat() on a broken symlink, python throws an OSError exception. This makes it useful for finding them. However, there are a few other reasons that os.stat() might throw a similar exception. Is there a more precise way of detecting broken symlinks with Python under Linux?
A common Python saying is that it's easier to ask forgiveness than permission. While I'm not a fan of this statement in real life, it does apply in a lot of cases. Usually you want to avoid code that chains two system calls on the same file, because you never know what will happen to the file in between your two calls in your code.
A typical mistake is to write something like:
if os.path.exists(path):
os.unlink(path)
The second call (os.unlink) may fail if something else deleted it after your if test, raise an Exception, and stop the rest of your function from executing. (You might think this doesn't happen in real life, but we just fished another bug like that out of our codebase last week - and it was the kind of bug that left a few programmers scratching their head and claiming 'Heisenbug' for the last few months)
So, in your particular case, I would probably do:
try:
os.stat(path)
except OSError, e:
if e.errno == errno.ENOENT:
print 'path %s does not exist or is a broken symlink' % path
else:
raise e
The annoyance here is that stat returns the same error code for a symlink that just isn't there and a broken symlink.
So, I guess you have no choice than to break the atomicity, and do something like
if not os.path.exists(os.readlink(path)):
print 'path %s is a broken symlink' % path
This is not atomic but it works.
os.path.islink(filename) and not os.path.exists(filename)
Indeed by RTFM
(reading the fantastic manual) we see
os.path.exists(path)
Return True if path refers to an existing path. Returns False for broken symbolic links.
It also says:
On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
So if you are worried about permissions, you should add other clauses.
os.lstat() may be helpful. If lstat() succeeds and stat() fails, then it's probably a broken link.
Can I mention testing for hardlinks without python? /bin/test has the FILE1 -ef FILE2 condition that is true when files share an inode.
Therefore, something like find . -type f -exec test \{} -ef /path/to/file \; -print works for hard link testing to a specific file.
Which brings me to reading man test and the mentions of -L and -h which both work on one file and return true if that file is a symbolic link, however that doesn't tell you if the target is missing.
I did find that head -0 FILE1 would return an exit code of 0 if the file can be opened and a 1 if it cannot, which in the case of a symbolic link to a regular file works as a test for whether it's target can be read.
os.path
You may try using realpath() to get what the symlink points to, then trying to determine if it's a valid file using is file.
(I'm not able to try that out at the moment, so you'll have to play around with it and see what you get)
I used this variant, When symlink is broken it will return false for the path.exists and true for path.islink, so combining this two facts we may use the following:
def kek(argum):
if path.exists("/root/" + argum) == False and path.islink("/root/" + argum) == True:
print("The path is a broken link, location: " + os.readlink("/root/" + argum))
else:
return "No broken links fond"
I'm not a python guy but it looks like os.readlink()? The logic I would use in perl is to use readlink() to find the target and the use stat() to test to see if the target exists.
Edit: I banged out some perl that demos readlink. I believe perl's stat and readlink and python's os.stat() and os.readlink()are both wrappers for the system calls, so this should translate reasonable well as proof of concept code:
wembley 0 /home/jj33/swap > cat p
my $f = shift;
while (my $l = readlink($f)) {
print "$f -> $l\n";
$f = $l;
}
if (!-e $f) {
print "$f doesn't exist\n";
}
wembley 0 /home/jj33/swap > ls -l | grep ^l
lrwxrwxrwx 1 jj33 users 17 Aug 21 14:30 link -> non-existant-file
lrwxrwxrwx 1 root users 31 Oct 10 2007 mm -> ../systems/mm/20071009-rewrite//
lrwxrwxrwx 1 jj33 users 2 Aug 21 14:34 mmm -> mm/
wembley 0 /home/jj33/swap > perl p mm
mm -> ../systems/mm/20071009-rewrite/
wembley 0 /home/jj33/swap > perl p mmm
mmm -> mm
mm -> ../systems/mm/20071009-rewrite/
wembley 0 /home/jj33/swap > perl p link
link -> non-existant-file
non-existant-file doesn't exist
wembley 0 /home/jj33/swap >
I had a similar problem: how to catch broken symlinks, even when they occur in some parent dir? I also wanted to log all of them (in an application dealing with a fairly large number of files), but without too many repeats.
Here is what I came up with, including unit tests.
fileutil.py:
import os
from functools import lru_cache
import logging
logger = logging.getLogger(__name__)
#lru_cache(maxsize=2000)
def check_broken_link(filename):
"""
Check for broken symlinks, either at the file level, or in the
hierarchy of parent dirs.
If it finds a broken link, an ERROR message is logged.
The function is cached, so that the same error messages are not repeated.
Args:
filename: file to check
Returns:
True if the file (or one of its parents) is a broken symlink.
False otherwise (i.e. either it exists or not, but no element
on its path is a broken link).
"""
if os.path.isfile(filename) or os.path.isdir(filename):
return False
if os.path.islink(filename):
# there is a symlink, but it is dead (pointing nowhere)
link = os.readlink(filename)
logger.error('broken symlink: {} -> {}'.format(filename, link))
return True
# ok, we have either:
# 1. a filename that simply doesn't exist (but the containing dir
does exist), or
# 2. a broken link in some parent dir
parent = os.path.dirname(filename)
if parent == filename:
# reached root
return False
return check_broken_link(parent)
Unit tests:
import logging
import shutil
import tempfile
import os
import unittest
from ..util import fileutil
class TestFile(unittest.TestCase):
def _mkdir(self, path, create=True):
d = os.path.join(self.test_dir, path)
if create:
os.makedirs(d, exist_ok=True)
return d
def _mkfile(self, path, create=True):
f = os.path.join(self.test_dir, path)
if create:
d = os.path.dirname(f)
os.makedirs(d, exist_ok=True)
with open(f, mode='w') as fp:
fp.write('hello')
return f
def _mklink(self, target, path):
f = os.path.join(self.test_dir, path)
d = os.path.dirname(f)
os.makedirs(d, exist_ok=True)
os.symlink(target, f)
return f
def setUp(self):
# reset the lru_cache of check_broken_link
fileutil.check_broken_link.cache_clear()
# create a temporary directory for our tests
self.test_dir = tempfile.mkdtemp()
# create a small tree of dirs, files, and symlinks
self._mkfile('a/b/c/foo.txt')
self._mklink('b', 'a/x')
self._mklink('b/c/foo.txt', 'a/f')
self._mklink('../..', 'a/b/c/y')
self._mklink('not_exist.txt', 'a/b/c/bad_link.txt')
bad_path = self._mkfile('a/XXX/c/foo.txt', create=False)
self._mklink(bad_path, 'a/b/c/bad_path.txt')
self._mklink('not_a_dir', 'a/bad_dir')
def tearDown(self):
# Remove the directory after the test
shutil.rmtree(self.test_dir)
def catch_check_broken_link(self, expected_errors, expected_result, path):
filename = self._mkfile(path, create=False)
with self.assertLogs(level='ERROR') as cm:
result = fileutil.check_broken_link(filename)
logging.critical('nothing') # trick: emit one extra message, so the with assertLogs block doesn't fail
error_logs = [r for r in cm.records if r.levelname is 'ERROR']
actual_errors = len(error_logs)
self.assertEqual(expected_result, result, msg=path)
self.assertEqual(expected_errors, actual_errors, msg=path)
def test_check_broken_link_exists(self):
self.catch_check_broken_link(0, False, 'a/b/c/foo.txt')
self.catch_check_broken_link(0, False, 'a/x/c/foo.txt')
self.catch_check_broken_link(0, False, 'a/f')
self.catch_check_broken_link(0, False, 'a/b/c/y/b/c/y/b/c/foo.txt')
def test_check_broken_link_notfound(self):
self.catch_check_broken_link(0, False, 'a/b/c/not_found.txt')
def test_check_broken_link_badlink(self):
self.catch_check_broken_link(1, True, 'a/b/c/bad_link.txt')
self.catch_check_broken_link(0, True, 'a/b/c/bad_link.txt')
def test_check_broken_link_badpath(self):
self.catch_check_broken_link(1, True, 'a/b/c/bad_path.txt')
self.catch_check_broken_link(0, True, 'a/b/c/bad_path.txt')
def test_check_broken_link_badparent(self):
self.catch_check_broken_link(1, True, 'a/bad_dir/c/foo.txt')
self.catch_check_broken_link(0, True, 'a/bad_dir/c/foo.txt')
# bad link, but shouldn't log a new error:
self.catch_check_broken_link(0, True, 'a/bad_dir/c')
# bad link, but shouldn't log a new error:
self.catch_check_broken_link(0, True, 'a/bad_dir')
if __name__ == '__main__':
unittest.main()
For Python 3, you can use the pathlib module. From its docs,
If the path points to a symlink, exists() returns whether the symlink points to an existing file or directory.
So this works too.
import pathlib
path = pathlib.Path("/path/to/somewhere")
if path.is_symlink() and not path.exists():
print(f"found dangling symlink at {path}")