I have an os.remove() in my code that sometimes, when ran locally, fails due to OSError 13 - Permission Denied - thus I've set up a try-except. My automated testing (Travis CI) is ran on Linux VM instances, so I don't know how to make os.remove fail there for sake of coverage.
What are my options - how do I force the except block to execute? Alternatively, how do I delete-protect a file with Python?
Note: Removing it in the test code before calling the test method isn't an option; the method itself fetches files to be removed:
from pathlib import Path
paths = [str(x) for x in Path("directory/").iterdir() if 'abc' in x.stem]
if len(paths) > 0: # if files are removed beforehand, len(paths) == 0
try:
[os.remove(p) for p in paths]
except:
pass # stuff here
You can use unittest.mock.patch to patch os.remove and specify OSError as a side_effect:
from unittest.mock import patch
...
with patch('os.remove') as mock_remove:
mock_remove.side_effect = OSError('Permission Denied')
try:
[os.remove(p) for p in paths]
except OSError as e:
pass # handle error here
Related
i'm writing a test of this function
def create_folder_if_not_exists(
sdk: looker_sdk,
folder_name: str,
parent_folder_name: str) -> dict:
folder = sdk.search_folders(name=folder_name)[0]
try:
parent_id = sdk.search_folders(name=parent_folder_name)[0].id
logger.info(f'Creating folder "{folder_name}"')
folder = sdk.create_folder(
body=models.CreateFolder(
name=folder_name,
parent_id=parent_id
)
)
return folder
except looker_sdk.error.SDKError as err:
logger.error(err.args[0])
return folder
This is my current test, using the python pytest library, but i keep getting this for my test Failed: DID NOT RAISE <class 'looker_sdk.error.SDKError'>
def test_create_folder_if_not_exists_parent1(mocker):
# Tests if a folder has parent id of 1 we raise an exception
sdk = fake_methods_data.MockSDK()
sf_data = fake_methods_data.MockSearchFolder(
name='goog', parent_id=1, id=3)
mocker.patch.object(sdk, "search_folders")
mocker.patch.object(sdk, "create_folder",
side_effect=[looker_sdk.error.SDKError])
sdk.search_folders.return_value = [sf_data]
with pytest.raises(looker_sdk.error.SDKError) as err:
test = fc.create_folder_if_not_exists(
sdk=sdk, folder_name='googn', parent_folder_name='1')
assert str(err.value) == 'test'
assert test.parent_id == 1
assert test.name == 'googn'
Does anyone know how to force a function to return a class error using pytest ? I've been looking at this [stackoverflow] (Mocking a function to raise an Exception to test an except block) but am struggling to get it to work. Hoping for some other thoughts.
This sounds like something I have done for work (open-source software dev stuff). In my case, I needed to test an except block raised when an executable file could not be run on a particular OS version. In our testing framework we use pytest and monkeypatch to test things. I've included the relevant bits of code below, along with some explanation about what is happening. I think this is probably what you mean by 'patch the sdk error', and I believe that is probably what you need to do. If anything is unclear, or you have more questions, let me know.
In conftest.py I define pytest fixtures that get used for tests in more than one test file. Here, I mock the scenario I want to test, using monkeypatch to fake the results I want from the parts of the get_version() function I'm not trying to test.
# conftest.py
import subprocess
import shutil
import os
import re
import platform
from pathlib import Path
import pytest
#pytest.fixture
def executable_incompatible_with_os(monkeypatch):
"""
Mocks an executable file that is incompatible with the OS.
(This situation likely only applies to blastall.)
"""
def mock_which(*args, **kwargs):
"""Mock an absolute file path."""
return args[0]
def mock_isfile(*args, **kwargs):
"""Mock a call to `os.path.isfile()`."""
return True
def mock_access(*args, **kwargs):
"""Mock a call to `os.access()`."""
return True
def mock_subprocess(*args, **kwargs):
"""Mock a call to `subprocess.run()` with an incompatible program."""
raise OSError
# Replace calls to existing methods with my mocked versions
monkeypatch.setattr(shutil, "which", mock_which)
monkeypatch.setattr(Path, "is_file", mock_isfile)
monkeypatch.setattr(os.path, "isfile", mock_isfile)
monkeypatch.setattr(os, "access", mock_access)
monkeypatch.setattr(subprocess, "run", mock_subprocess)
In test_aniblastall.py I test parts of aniblastall.py. In this case, I'm testing the behaviour when an OSError is raised; the code that raises the error in the test is in conftest.py. The entire pytest fixture I defined there is passed as a parameter to the test.
# test_aniblastall.py
from pathlib import Path
import unittest
# Test case 4: there is an executable file, but it will not run on the OS
def test_get_version_os_incompatible(executable_incompatible_with_os):
"""Test behaviour when the program can't run on the operating system.
This will happen with newer versions of MacOS."""
test_file_4 = Path("/os/incompatible/blastall")
assert (
aniblastall.get_version(test_file_4)
== f"blastall exists at {test_file_4} but could not be executed"
)
aniblastall.py contains the function the error should be raised from.
# aniblastall.py
import logging
import os
import platform
import re
import shutil
import subprocess
from pathlib import Path
def get_version(blast_exe: Path = pyani_config.BLASTALL_DEFAULT) -> str:
"""
The following circumstances are explicitly reported as strings
- no executable at passed path
- non-executable file at passed path (this includes cases where the user doesn't have execute permissions on the file)
- no version info returned
- executable cannot be run on this OS
"""
logger = logging.getLogger(__name__)
try:
blastall_path = Path(shutil.which(blast_exe)) # type:ignore
except TypeError:
return f"{blast_exe} is not found in $PATH"
if not blastall_path.is_file(): # no executable
return f"No blastall at {blastall_path}"
# This should catch cases when the file can't be executed by the user
if not os.access(blastall_path, os.X_OK): # file exists but not executable
return f"blastall exists at {blastall_path} but not executable"
if platform.system() == "Darwin":
cmdline = [blast_exe, "-version"]
else:
cmdline = [blast_exe]
try:
result = subprocess.run(
cmdline, # type: ignore
shell=False,
stdout=subprocess.PIPE, # type: ignore
stderr=subprocess.PIPE,
check=False, # blastall doesn't return 0
)
except OSError:
logger.warning("blastall executable will not run", exc_info=True)
return f"blastall exists at {blastall_path} but could not be executed"
version = re.search( # type: ignore
r"(?<=blastall\s)[0-9\.]*", str(result.stderr, "utf-8")
).group()
if 0 == len(version.strip()):
return f"blastall exists at {blastall_path} but could not retrieve version"
return f"{platform.system()}_{version} ({blastall_path})"
This is super valuable #baileythegreen, however my problem was far simpler. I had an if/else and the else had the try/catch error code piece. I was so focused on that I didn't check the simple part of if it was even getting to the else. :(
I am migrating an application to python3 but there are legacy systems that cannot be upgraded yet using python2.
I have the application that needs to update a cache file but if for some reason the user launching the application cannot read/update the cache. it is no big deal and he can just query the db again instead of using the cache.
I would like therefore pass the permission denied exception on python2 it is an OSError [Errno 13].
on python3 i use PermissionError so that is okay. I assume that this exception will only catch the errno 13.
this is what i have on python3
for filename in os.listdir(cache_dir):
try:
if filename.endswith('.cache'):
os.remove(os.path.join(cache_dir, filename))
Except PermissionError:
pass
Except OSError:
#handle all other errors
how can I mimic the same on python2 so that only Errno 13 is passed and not anything else? eg I can pass if its a permission denied but not if the fs is read-only or the disk is full.
There is only OSError to catch, but you can check the error number contained in the exception.
import errno
for filename in os.listdir(cache_dir):
# Keep the try block as focused as possible.
if not filename.endswith('.cache'):
continue
fname = os.path.join(cache_dir, filename)
try:
os.remove(fname)
except OSError as e:
if e.errno != errno.EACCES:
# handle other errors
Use the errno module, because error numbers vary from one operating system to another.
I try to find a solution for the following problems but can't find any good solution.
I have a folder with subfolder and files in it.
Some of the files may be in use by another process (the other process is writing data to the data (a .mdf file)).
I simply want to check if the files are in use or not.
Structure:
A_Folder
Setup
Data1
.mdf-file1
.mdf-file2
Data2
Data3
Evaluation
something like:
def file_in_use():
*your solution*
for file in folder:
if file_in_use(file):
print("file in use")
break
I'm Using Win10, PyCharm and a venv.
I tried so far form other "solutions":
psutil (works, but is too slow)
open(), os.rename - won't work for me
subprocess wont work either -cant find my filename: using the method from Amit
Gupta from my link down below, file looks like this: "C:\Data\S_t_h\S-t-h\H001.mdf"
basically I tried everything from this question:
Check if a file is not open nor being used by another process
from subprocess import check_output, Popen, PIPE
src = r"C:\Data\S_t_h\S-t-h\H001.mdf"
files_in_use = False
def file_in_use(src):
try:
lsout = Popen(['lsof', src], stdout=PIPE, shell=False)
check_output(["grep", src], stdin=lsout.stdout, shell=False)
except:
return False
return True
if file_in_use(src):
files_in_use = True
and im getting:
FileNotFoundError: [WinError 2] The system cannot find the file specified
this link suggesting setting
winerror-2-the-system-cannot-find-the-file-specified-python
shell=True
Im getting "lsof" and "grep" cant be found or are wrong now.
Here the psutil method that works for me, but is too slow (~10 Seconds)
import psutil
src = r"C:\Data\S_t_h\S-t-h\H001.mdf"
def has_handle(src):
for proc in psutil.process_iter():
try:
for item in proc.open_files():
if src == item.path:
return True
except Exception:
pass
return False
print(has_handle(src))
My Solution:
Sorry for the delayed answer.
It simply worked with:
try:
os.rename(src, src)
return False
except OSError: # file is in use
return True
I made it more complicated than it actually was i guess.
But thank you guys anyway for your feedback and critizism.
So I'm building a simple script that backs up certain documents to my second hard-drive (you never know what could happen!). So, I used the shutil.copytree function to replicate my data on the second drive. It works beautifully, and that is not the problem.
I use the shutil.rmtree function to remove the tree if the destination already exists. I'll show you my code:
import shutil
import os
def overwrite(src, dest):
if(not os.path.exists(src)):
print(src, "does not exist, so nothing may be copied.")
return
if(os.path.exists(dest)):
shutil.rmtree(dest)
shutil.copytree(src, dest)
print(dest, "overwritten with data from", src)
print("")
overwrite(r"C:\Users\Centurion\Dropbox\Documents", r"D:\Backup\Dropbox Documents")
overwrite(r"C:\Users\Centurion\Pictures", r"D:\Backup\All Pictures")
print("Press ENTER to continue...")
input()
As you can see, a simple script. Now, when I run the script for the first time, everything is fine. Pictures and Documents copy over to my D: drive just fine. However, when I run for the second time, this is my output:
C:\Users\Centurion\Programming\Python>python cpdocsnpics.py
D:\Backup\Dropbox Documents overwritten with data from C:\Users\Centurion\Dropbox\Documents
Traceback (most recent call last):
File "cpdocsnpics.py", line 17, in <module>
overwrite(r"C:\Users\Centurion\Pictures", r"D:\Backup\All Pictures")
File "cpdocsnpics.py", line 10, in overwrite
shutil.rmtree(dest)
File "C:\Python34\lib\shutil.py", line 477, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python34\lib\shutil.py", line 376, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "C:\Python34\lib\shutil.py", line 374, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 5] Access is denied: 'D:\\Backup\\All Pictures'
The error only happens when I copy Pictures after the first time; I'm assuming it has something to do with being a Library.
What should I do?
That's a cross-platform consistency issue.
You've copied files/dirs with readonly attribute. On the first time "dest" not exists, thus rmtree method is not performed. However, when you try run "overwrite" function we can notice that "dest" location exists (and its subtree) but it was copied with readonly access. So here we got a problem.
In order to "fix" issue, you must provide a handler for onerror parameter of shutil.rmtree. As long as your problem is regarding readonly issues the workaround is somewhat like this:
def readonly_handler(func, path, execinfo):
os.chmod(path, 128) #or os.chmod(path, stat.S_IWRITE) from "stat" module
func(path)
As you can see in the python doc onerror must be a callable that accepts three parameters: function, path, and excinfo. For further info, read the docs.
def overwrite(src, dest):
if(not os.path.exists(src)):
print(src, "does not exist, so nothing may be copied.")
return
if(os.path.exists(dest)):
shutil.rmtree(dest, onerror=readonly_handler)
shutil.copytree(src, dest)
print(dest, "overwritten with data from", src)
print("")
Of course, this handler is simple and specific but if other errors occur, new exceptions will be raised and this handler may not be able to fix them!
Note:
Tim Golden (Python for windows contributor) has been patching the shutil.rmtree issue and it seems it will be resolved in Python 3.5 (see issue 19643).
I found a problem other than the read-only file using shutil.rmtree on Windows (testing on Windows 7). I was using a combination of shutil.rmtree and shutil.copytree to create a test fixture in a test suite, so the sequence was being called repeatedly in a short period of tme (<1 sec intervals), and I was seeing unpredictable failures part way through the test suite, with both EACCES and ENOTEMPTY errors reported. The symptoms suggested to me that the shutil.rmtree function had not completed on return to the calling program, and that it was only after some time that the deleted filenames were available for re-use.
TL;DR: the solution isn't pretty - broadly, it renames the directory before deleting it, but there are a number of wrinkles that need to be handled because the Windows file system seems to take some time to catch up with the operations perfumed. The actual code catches a variety of failure conditions and retries a variant of the failed operation after a short delay.
A longer discussion follows, with my final code at the end.
My first thought was to try renaming the directory tree before removing it, so that the original directory name is immediately available for re-use. This does appear to help. To this end, I created a replacement for rmtree whose essence is this:
def removetree(tgt):
def error_handler(func, path, execinfo):
e = execinfo[1]
if e.errno == errno.ENOENT or not os.path.exists(path):
return # path does not exist - treat as success
if func in (os.rmdir, os.remove) and e.errno == errno.EACCES:
os.chmod(path, stat.S_IRWXU| stat.S_IRWXG| stat.S_IRWXO) # 0777
func(path) # read-only file; make writable and retry
raise e
tmp = os.path.join(os.path.dirname(tgt),"_removetree_tmp")
os.rename(tgt, tmp)
shutil.rmtree(tmp, onerror=error_handler)
return
I found this logic was an improvement, but it was subject to unpredictable failure of the os.rename operation, with one of several possible errors. So I also added some retry logic around os.rename, thus:
def removetree(tgt):
def error_handler(func, path, execinfo):
# figure out recovery based on error...
e = execinfo[1]
if e.errno == errno.ENOENT or not os.path.exists(path):
return # path does not exist
if func in (os.rmdir, os.remove) and e.errno == errno.EACCES:
os.chmod(path, stat.S_IRWXU| stat.S_IRWXG| stat.S_IRWXO) # 0777
func(path) # read-only file; make writable and retry
raise e
# Rename target directory to temporary value, then remove it
count = 0
while count < 10: # prevents indefinite loop
count += 1
tmp = os.path.join(os.path.dirname(tgt),"_removetree_tmp_%d"%(count))
try:
os.rename(tgt, tmp)
shutil.rmtree(tmp, onerror=error_handler)
break
except OSError as e:
time.sleep(1) # Give file system some time to catch up
if e.errno in [errno.EACCES, errno.ENOTEMPTY]:
continue # Try another temp name
if e.errno == errno.EEXIST:
shutil.rmtree(tmp, ignore_errors=True) # Try to clean up old files
continue # Try another temp name
if e.errno == errno.ENOENT:
break # 'src' does not exist(?)
raise # Other error - propagate
return
The above code is not tested, but the general idea here does seem to work. The full code I actually use is below, and uses two functions. It probably contains some unnecessary logic, but does seem to be working more reliably for me (in that my test suite now passes repeatedly on Windows where previously it failed unpredictably on a majority of runs):
def renametree_temp(src):
"""
Rename tree to temporary name, and return that name, or
None if the source directory does not exist.
"""
count = 0
while count < 10: # prevents indefinite loop
count += 1
tmp = os.path.join(os.path.dirname(src),"_removetree_tmp_%d"%(count))
try:
os.rename(src, tmp)
return tmp # Success!
except OSError as e:
time.sleep(1)
if e.errno == errno.EACCES:
log.warning("util.renametree_temp: %s EACCES, retrying"%tmp)
continue # Try another temp name
if e.errno == errno.ENOTEMPTY:
log.warning("util.renametree_temp: %s ENOTEMPTY, retrying"%tmp)
continue # Try another temp name
if e.errno == errno.EEXIST:
log.warning("util.renametree_temp: %s EEXIST, retrying"%tmp)
shutil.rmtree(tmp, ignore_errors=True) # Try to clean up old files
continue # Try another temp name
if e.errno == errno.ENOENT:
log.warning("util.renametree_temp: %s ENOENT, skipping"%tmp)
break # 'src' does not exist(?)
raise # Other error: propagaee
return None
def removetree(tgt):
"""
Work-around for python problem with shutils tree remove functions on Windows.
See:
https://stackoverflow.com/questions/23924223/
https://stackoverflow.com/questions/1213706/
https://stackoverflow.com/questions/1889597/
http://bugs.python.org/issue19643
"""
# shutil.rmtree error handler that attempts recovery from attempts
# on Windows to remove a read-only file or directory (see links above).
def error_handler(func, path, execinfo):
e = execinfo[1]
if e.errno == errno.ENOENT or not os.path.exists(path):
return # path does not exist: nothing to do
if func in (os.rmdir, os.remove) and e.errno == errno.EACCES:
try:
os.chmod(path, stat.S_IRWXU| stat.S_IRWXG| stat.S_IRWXO) # 0777
except Exception as che:
log.warning("util.removetree: chmod failed: %s"%che)
try:
func(path)
except Exception as rfe:
log.warning("util.removetree: 'func' retry failed: %s"%rfe)
if not os.path.exists(path):
return # Gone, assume all is well
raise
if e.errno == errno.ENOTEMPTY:
log.warning("util.removetree: Not empty: %s, %s"%(path, tgt))
time.sleep(1)
removetree(path) # Retry complete removal
return
log.warning("util.removetree: rmtree path: %s, error: %s"%(path, repr(execinfo)))
raise e
# Try renaming to a new directory first, so that the tgt is immediately
# available for re-use.
tmp = renametree_temp(tgt)
if tmp:
shutil.rmtree(tmp, onerror=error_handler)
return
(The above code incorporates a solution to the read-only file problem from What user do python scripts run as in windows?, which according to Deleting directory in Python is tested. I don't think I encounter the read-only file problem, so assume it is not tested in my test suite.)
I want to run mkdir command as:
mkdir -p directory_name
What's the method to do that in Python?
os.mkdir(directory_name [, -p]) didn't work for me.
You can try this:
# top of the file
import os
import errno
# the actual code
try:
os.makedirs(directory_name)
except OSError as exc:
if exc.errno == errno.EEXIST and os.path.isdir(directory_name):
pass
According to the documentation, you can now use this since python 3.2
os.makedirs("/directory/to/make", exist_ok=True)
and it will not throw an error when the directory exists.
Something like this:
if not os.path.exists(directory_name):
os.makedirs(directory_name)
UPD: as it is said in a comments you need to check for exception for thread safety
try:
os.makedirs(directory_name)
except OSError as err:
if err.errno!=17:
raise
If you're using pathlib, use Path.mkdir(parents=True, exist_ok=True)
from pathlib import Path
new_directory = Path('./some/nested/directory')
new_directory.mkdir(parents=True, exist_ok=True)
parents=True creates parent directories as needed
exist_ok=True tells mkdir() to not error if the directory already exists
See the pathlib.Path.mkdir() docs.
how about this
os.system('mkdir -p %s' % directory_name )