So I'm building a simple script that backs up certain documents to my second hard-drive (you never know what could happen!). So, I used the shutil.copytree function to replicate my data on the second drive. It works beautifully, and that is not the problem.
I use the shutil.rmtree function to remove the tree if the destination already exists. I'll show you my code:
import shutil
import os
def overwrite(src, dest):
if(not os.path.exists(src)):
print(src, "does not exist, so nothing may be copied.")
return
if(os.path.exists(dest)):
shutil.rmtree(dest)
shutil.copytree(src, dest)
print(dest, "overwritten with data from", src)
print("")
overwrite(r"C:\Users\Centurion\Dropbox\Documents", r"D:\Backup\Dropbox Documents")
overwrite(r"C:\Users\Centurion\Pictures", r"D:\Backup\All Pictures")
print("Press ENTER to continue...")
input()
As you can see, a simple script. Now, when I run the script for the first time, everything is fine. Pictures and Documents copy over to my D: drive just fine. However, when I run for the second time, this is my output:
C:\Users\Centurion\Programming\Python>python cpdocsnpics.py
D:\Backup\Dropbox Documents overwritten with data from C:\Users\Centurion\Dropbox\Documents
Traceback (most recent call last):
File "cpdocsnpics.py", line 17, in <module>
overwrite(r"C:\Users\Centurion\Pictures", r"D:\Backup\All Pictures")
File "cpdocsnpics.py", line 10, in overwrite
shutil.rmtree(dest)
File "C:\Python34\lib\shutil.py", line 477, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Python34\lib\shutil.py", line 376, in _rmtree_unsafe
onerror(os.rmdir, path, sys.exc_info())
File "C:\Python34\lib\shutil.py", line 374, in _rmtree_unsafe
os.rmdir(path)
PermissionError: [WinError 5] Access is denied: 'D:\\Backup\\All Pictures'
The error only happens when I copy Pictures after the first time; I'm assuming it has something to do with being a Library.
What should I do?
That's a cross-platform consistency issue.
You've copied files/dirs with readonly attribute. On the first time "dest" not exists, thus rmtree method is not performed. However, when you try run "overwrite" function we can notice that "dest" location exists (and its subtree) but it was copied with readonly access. So here we got a problem.
In order to "fix" issue, you must provide a handler for onerror parameter of shutil.rmtree. As long as your problem is regarding readonly issues the workaround is somewhat like this:
def readonly_handler(func, path, execinfo):
os.chmod(path, 128) #or os.chmod(path, stat.S_IWRITE) from "stat" module
func(path)
As you can see in the python doc onerror must be a callable that accepts three parameters: function, path, and excinfo. For further info, read the docs.
def overwrite(src, dest):
if(not os.path.exists(src)):
print(src, "does not exist, so nothing may be copied.")
return
if(os.path.exists(dest)):
shutil.rmtree(dest, onerror=readonly_handler)
shutil.copytree(src, dest)
print(dest, "overwritten with data from", src)
print("")
Of course, this handler is simple and specific but if other errors occur, new exceptions will be raised and this handler may not be able to fix them!
Note:
Tim Golden (Python for windows contributor) has been patching the shutil.rmtree issue and it seems it will be resolved in Python 3.5 (see issue 19643).
I found a problem other than the read-only file using shutil.rmtree on Windows (testing on Windows 7). I was using a combination of shutil.rmtree and shutil.copytree to create a test fixture in a test suite, so the sequence was being called repeatedly in a short period of tme (<1 sec intervals), and I was seeing unpredictable failures part way through the test suite, with both EACCES and ENOTEMPTY errors reported. The symptoms suggested to me that the shutil.rmtree function had not completed on return to the calling program, and that it was only after some time that the deleted filenames were available for re-use.
TL;DR: the solution isn't pretty - broadly, it renames the directory before deleting it, but there are a number of wrinkles that need to be handled because the Windows file system seems to take some time to catch up with the operations perfumed. The actual code catches a variety of failure conditions and retries a variant of the failed operation after a short delay.
A longer discussion follows, with my final code at the end.
My first thought was to try renaming the directory tree before removing it, so that the original directory name is immediately available for re-use. This does appear to help. To this end, I created a replacement for rmtree whose essence is this:
def removetree(tgt):
def error_handler(func, path, execinfo):
e = execinfo[1]
if e.errno == errno.ENOENT or not os.path.exists(path):
return # path does not exist - treat as success
if func in (os.rmdir, os.remove) and e.errno == errno.EACCES:
os.chmod(path, stat.S_IRWXU| stat.S_IRWXG| stat.S_IRWXO) # 0777
func(path) # read-only file; make writable and retry
raise e
tmp = os.path.join(os.path.dirname(tgt),"_removetree_tmp")
os.rename(tgt, tmp)
shutil.rmtree(tmp, onerror=error_handler)
return
I found this logic was an improvement, but it was subject to unpredictable failure of the os.rename operation, with one of several possible errors. So I also added some retry logic around os.rename, thus:
def removetree(tgt):
def error_handler(func, path, execinfo):
# figure out recovery based on error...
e = execinfo[1]
if e.errno == errno.ENOENT or not os.path.exists(path):
return # path does not exist
if func in (os.rmdir, os.remove) and e.errno == errno.EACCES:
os.chmod(path, stat.S_IRWXU| stat.S_IRWXG| stat.S_IRWXO) # 0777
func(path) # read-only file; make writable and retry
raise e
# Rename target directory to temporary value, then remove it
count = 0
while count < 10: # prevents indefinite loop
count += 1
tmp = os.path.join(os.path.dirname(tgt),"_removetree_tmp_%d"%(count))
try:
os.rename(tgt, tmp)
shutil.rmtree(tmp, onerror=error_handler)
break
except OSError as e:
time.sleep(1) # Give file system some time to catch up
if e.errno in [errno.EACCES, errno.ENOTEMPTY]:
continue # Try another temp name
if e.errno == errno.EEXIST:
shutil.rmtree(tmp, ignore_errors=True) # Try to clean up old files
continue # Try another temp name
if e.errno == errno.ENOENT:
break # 'src' does not exist(?)
raise # Other error - propagate
return
The above code is not tested, but the general idea here does seem to work. The full code I actually use is below, and uses two functions. It probably contains some unnecessary logic, but does seem to be working more reliably for me (in that my test suite now passes repeatedly on Windows where previously it failed unpredictably on a majority of runs):
def renametree_temp(src):
"""
Rename tree to temporary name, and return that name, or
None if the source directory does not exist.
"""
count = 0
while count < 10: # prevents indefinite loop
count += 1
tmp = os.path.join(os.path.dirname(src),"_removetree_tmp_%d"%(count))
try:
os.rename(src, tmp)
return tmp # Success!
except OSError as e:
time.sleep(1)
if e.errno == errno.EACCES:
log.warning("util.renametree_temp: %s EACCES, retrying"%tmp)
continue # Try another temp name
if e.errno == errno.ENOTEMPTY:
log.warning("util.renametree_temp: %s ENOTEMPTY, retrying"%tmp)
continue # Try another temp name
if e.errno == errno.EEXIST:
log.warning("util.renametree_temp: %s EEXIST, retrying"%tmp)
shutil.rmtree(tmp, ignore_errors=True) # Try to clean up old files
continue # Try another temp name
if e.errno == errno.ENOENT:
log.warning("util.renametree_temp: %s ENOENT, skipping"%tmp)
break # 'src' does not exist(?)
raise # Other error: propagaee
return None
def removetree(tgt):
"""
Work-around for python problem with shutils tree remove functions on Windows.
See:
https://stackoverflow.com/questions/23924223/
https://stackoverflow.com/questions/1213706/
https://stackoverflow.com/questions/1889597/
http://bugs.python.org/issue19643
"""
# shutil.rmtree error handler that attempts recovery from attempts
# on Windows to remove a read-only file or directory (see links above).
def error_handler(func, path, execinfo):
e = execinfo[1]
if e.errno == errno.ENOENT or not os.path.exists(path):
return # path does not exist: nothing to do
if func in (os.rmdir, os.remove) and e.errno == errno.EACCES:
try:
os.chmod(path, stat.S_IRWXU| stat.S_IRWXG| stat.S_IRWXO) # 0777
except Exception as che:
log.warning("util.removetree: chmod failed: %s"%che)
try:
func(path)
except Exception as rfe:
log.warning("util.removetree: 'func' retry failed: %s"%rfe)
if not os.path.exists(path):
return # Gone, assume all is well
raise
if e.errno == errno.ENOTEMPTY:
log.warning("util.removetree: Not empty: %s, %s"%(path, tgt))
time.sleep(1)
removetree(path) # Retry complete removal
return
log.warning("util.removetree: rmtree path: %s, error: %s"%(path, repr(execinfo)))
raise e
# Try renaming to a new directory first, so that the tgt is immediately
# available for re-use.
tmp = renametree_temp(tgt)
if tmp:
shutil.rmtree(tmp, onerror=error_handler)
return
(The above code incorporates a solution to the read-only file problem from What user do python scripts run as in windows?, which according to Deleting directory in Python is tested. I don't think I encounter the read-only file problem, so assume it is not tested in my test suite.)
Related
I have an os.remove() in my code that sometimes, when ran locally, fails due to OSError 13 - Permission Denied - thus I've set up a try-except. My automated testing (Travis CI) is ran on Linux VM instances, so I don't know how to make os.remove fail there for sake of coverage.
What are my options - how do I force the except block to execute? Alternatively, how do I delete-protect a file with Python?
Note: Removing it in the test code before calling the test method isn't an option; the method itself fetches files to be removed:
from pathlib import Path
paths = [str(x) for x in Path("directory/").iterdir() if 'abc' in x.stem]
if len(paths) > 0: # if files are removed beforehand, len(paths) == 0
try:
[os.remove(p) for p in paths]
except:
pass # stuff here
You can use unittest.mock.patch to patch os.remove and specify OSError as a side_effect:
from unittest.mock import patch
...
with patch('os.remove') as mock_remove:
mock_remove.side_effect = OSError('Permission Denied')
try:
[os.remove(p) for p in paths]
except OSError as e:
pass # handle error here
I'm trying to create a subdirectory if it doesn't already exist. Here is the relevant portion of my function (TMP is the constant "-InProg"):
def create_directories(parent, tmp_dir=""):
"""
Terminates program if tmp_dir already exists; otherwise
creates subdirectories that don't already exist
"""
if not tmp_dir:
tmp_dir = parent + TMP
try:
if os.path.exists(tmp_dir):
LOGGER.error('Automation already running')
return False
else:
os.makedirs(tmp_dir)
<...snip...>
except OSError:
LOGGER.error('Directory %s not valid', parent, exc_info=True)
sys.exit("ERROR: Directory " + parent + " not valid")
Here is the relevant portion of the log:
ERROR:Directory <parent> not valid
Traceback (most recent call last):
File "</path/to/my/script>.py", line 139, in create_directories
os.makedirs(tmp_dir)
File "/usr/local/Cellar/python#2/2.7.17/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 17] File exists: '<parent>-InProg'
Clearly tmp_dir already exists. But if tmp_dir exists then the code shouldn't reach os.makedirs(tmp_dir) at all, right? Yet that is definitely the specific line throwing the OSError.
The only clue I've been able to find is that the documentation for os.path.exists() says, "On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists." If this is the problem, what can I do about it?
NOTE: This explanation seems a bit unlikely, because the program itself is what creates tmp_dir, and I can't imagine it being able to create a subdirectory that it doesn't have permission to stat(). I can't think of an alternate explanation, though. Am I missing some other incredibly obvious bug?
In my python code am copying some installer to temp and installing that and then I want to delete the temp copy.
I am using "shutil.rmtree" function to remove the installer copy in the temp.
Sometime I am getting an error like "[Error 32] The process cannot access the file because it is being used by another process"
How can I avoid that ? Do we have a way to check a directory is not used by any other program ? If yes can we check for the availability of directory till the other process leaves control and delete it ?
Please help me with sample codes ! I am pretty new to python.
copy_dst = tempfile.mkdtemp()
if success:
success = robocopy_file_or_folder(installer_location, copy_dst)
local_path_to_setup = os.path.join(copy_dst, app_folder, 'setup.exe')
if success:
success, response_text = install_product(local_path_to_setup, success, response_text)
try:
if success:
shutil.rmtree(copy_dst, ignore_errors = False, onerror=handle_remove_readonly)
except OSError as e:
log(" %s\n Delete installer Local Copy at \"%s\" failed.\n" % (e, copy_dst))
success = False
This is the code !! and "install_product()" will install the product by calling the setup.exe in another command.robocopy_file_or_folder () -- will copy the files.
def handle_remove_readonly(func, path, exc):
"""
Changes the access mode of the specified folder to remove read_only
"""
exc_value = exc[1]
if func in (os.rmdir, os.remove) and exc_value.errno == errno.EACCES:
os.chmod(path, stat.S_IRWXU| stat.S_IRWXG| stat.S_IRWXO) # 0777
func(path)
else:
raise
"install_product()" will install the product by calling the setup.exe in another command.robocopy_file_or_folder() – will copy the files.
Wait for the installation to finish and setup.exe to exit before trying to remove the directory.
I'd like to know how to handle different exceptions that have the same "type". I'm trying to use some code to make a directory:
os.mkdir(target_dir_name)
And I know this can fail for a variety of reasons, for example if the directory already exists:
OSError: [Errno 17] File exists:
or if there's no permissions to create a new dir:
OSError: [Errno 13] Permission denied:
I'd like to tailor my error messages to the specific failure cause, so I came up with the following code:
try:
os.mkdir(target_dir_name)
except OSError as e:
if e.errno == 17:
print "Warning: Directory %s already exists, executing a rebuild" % (target_dir_name)
elif e.errno == 13:
sys.exit("Error: Directory "+target_dir_name+" cannot be created incorrect permissions")
but I'd like to do something a little less hardcoded. Is there a Pythonic way I could update my sub-failure checks?
The only information I've found... which I guess is my best answer:
With the errno module I can at least rewrite my current code as:
import errno
#...
except OSError as e:
if e.errno == errno.EEXIST:
print "Warning: Directory %s already exists, e..."
elif e.errno == errno.EACCESS:
sys.exit("Error: Directory "+target_dir_name+"..."
If I upgrade to python 3.2 there is the much nicer:
except FileExistsError:
print ("Warning: Directory %s already exists, e...")
except PermissionError:
sys.exit("Error: Directory "+target_dir_name+"...")
Seems I could do something fancy with using sys.version_info.major and minor to make sure it's at least 3.2 as well.
Hopefully something better will come up...
Here is the code:
def make_dir(dir_name):
if os.path.exists(dir_name):
shutil.rmtree(dir_name)
try:
os.makedirs(dir_name)
except OSError, e:
print "ErrorNo: %s (%s)" % (e.errno, errno.errorcode[e.errno])
raise
IFF the directory already exists, I get the following:
ErrorNo: 13 (EACCES)
Traceback (most recent call last):
File "run_pnoise.py", line 167, in <module>
make_dir("prep_dat")
File "run_pnoise.py", line 88, in make_dir
os.makedirs(dir_name)
File "c:\Program Files (x86)\Python27\lib\os.py", line 157, in makedirs
mkdir(name, mode)
WindowsError: [Error 5] Access is denied: 'prep_dat'
If I run the program again, it works, indicating that the program indeed does have access to the directory(s), since the shutil.rmtree call is obviously working. I have come up with a workaround which I will post. However, is there a better explanation and/or workaround?
My assumption is that the shutil.rmtree call is returning before the OS is totally done deleting all of the files and subdirectories. Also, since the shutil.rmtree call is not throwing an exception, any EACCESS (13) error on the makedirs call is likely bogus. My attempt (as modified after Apalala's comment):
def make_dir(dir_name):
retry = True
if os.path.exists(dir_name):
shutil.rmtree(dir_name)
while retry:
try:
# per Apalala, sleeping before the makedirs() eliminates the exception!
time.sleep(0.001)
os.makedirs(dir_name)
except OSError, e:
#time.sleep(0.001) # moved to before the makedirs() call
#print "ErrorNo: %s (%s)" % (e.errno, errno.errorcode[e.errno])
if e.errno != 13: # eaccess
raise
else:
retry = False
This seems to work reliably. There is the race condition problem mentioned in other posts, however that seems unlikely, and would probably result in a different exception.
I had the same problem, and this looks similar to my solution, except I was sleeping (0.1).
Can't you simply use an «except» statement ?
def make_dir(dir_name):
retry = True
if os.path.exists(dir_name):
shutil.rmtree(dir_name)
while retry:
try:
os.makedirs(dir_name)
except OSError, e:
time.sleep(0.001)
if e.errno != 13: # eaccess
raise
except WindowsError:
# (do some stuff)
else:
retry = False
It should work, no ?!
In your previous post, you said the program raised a «WindowsError» exception:
WindowsError: [Error 5] Access is denied: 'prep_dat'
Your code can handle «OSError» exceptions using the «except» statement, but it cannot handle «WindowsError» exceptions... if you want to handle «WindowsError» exceptions, you must use the «except» statement like this:
except WindowsError:
# (do some stuff)
Note that you can handle any exception like this:
except Exception, e:
# this code will catch all raised exceptions. The variable «e» contains an instance of the raised exception.