shutil.rmtree will not delete read-only files on Windows. Is there a python equivalent of "rm -rf" ? Why oh why is this such a pain?
shutil.rmtree can take an error-handling function that will be called when it has problem removing a file. You can use that to force the removal of the problematic file(s).
Inspired by http://mail.python.org/pipermail/tutor/2006-June/047551.html and http://techarttiki.blogspot.com/2008/08/read-only-windows-files-with-python.html:
import os
import stat
import shutil
def remove_readonly(func, path, excinfo):
os.chmod(path, stat.S_IWRITE)
func(path)
shutil.rmtree(top, onerror=remove_readonly)
(I haven't tested that snippet out, but it should be enough to get you started)
If you import win32api from PyWin32, you can use:
win32api.SetFileAttributes(path, win32con.FILE_ATTRIBUTE_NORMAL)
To make files cease to be read-only.
Another way is to define rmtree on Windows as
rmtree = lambda path: subprocess.check_call(['cmd', '/c', 'rd', '/s', '/q', path])
There's a comment at the ActiveState site that says:
shutil.rmtree has its shortcomings. Although it is true you can use shutil.rmtree() in many cases, there are some cases where it does not work. For example, files that are marked read-only under Windows cannot be deleted by shutil.rmtree().
By importing the win32api and win32con modules from PyWin32 and adding line like "win32api.SetFileAttributes(path, win32con.FILE_ATTRIBUTE_NORMAL" to the rmgeneric() function, this obstacle can be overcome. I used this approach to fix the hot-backup.py script of Subversion 1.4 so it will work under Windows. Thank you for the recipe.
I don't use Windows so can't verify whether this works or not.
This will presumably be fixed with the release of Python 3.5 (currently - June 2015 - still in development) in the sense of giving a hint about this in the documentation.
You can find the bugreport here. And this is the according changeset.
See the newly added example from the Python 3.5 docs:
import os, stat
import shutil
def remove_readonly(func, path, _):
"Clear the readonly bit and reattempt the removal"
os.chmod(path, stat.S_IWRITE)
func(path)
shutil.rmtree(directory, onerror=remove_readonly)
Here is a variant of what Steve posted, it uses the same basic mechanism, and this one is tested :-)
What user do python scripts run as in windows?
I had this issue on Python 3.7 when invoking rmtree() and worked around it by explicitly fixing the permissions before exiting the TemporaryDirectory() context manager. See akaihola/darker#453 for details. Here's a copy of the implementation:
import os
import sys
from pathlib import Path
from typing import Union
WINDOWS = sys.platform.startswith("win")
def fix_py37_win_tempdir_permissions(dirpath: Union[str, Path]) -> None:
"""Work around a `tempfile` clean-up issue on Windows with Python 3.7
Call this before exiting a ``with TemporaryDirectory():`` block or in teardown for
a Pytest fixture which creates a temporary directory.
See discussion in https://github.com/akaihola/darker/pull/393
Solution borrowed from https://github.com/python/cpython/pull/10320
:param dirpath: The root path of the temporary directory
"""
if not WINDOWS or sys.version_info >= (3, 8):
return
for root, dirs, files in os.walk(dirpath):
for name in dirs + files:
path = os.path.join(root, name)
try:
os.chflags(path, 0) # type: ignore[attr-defined]
except AttributeError:
pass
os.chmod(path, 0o700)
and here's how to use it in Pytest unit tests or when creating a temporary directory using tempfile:
import pytest
from my_utils import fix_py37_win_tempdir_permissions
#pytest.fixture
def myfixture(tmp_path):
# setup code here
yield tmp_path
fix_py37_win_tempdir_permissions(tmp_path)
def myfunc():
with TemporaryDirectory() as tmpdir:
# work on the temporary directory here
fix_py37_win_tempdir_permissions(tmp_path)
Related
I need to perform an action without changing the global working directory. My case is I have a few folders, and in each, there are a few files. I need to do some computations using those files. Initially, I tried the following:
with os.chdir('/directory'):
...some code needing execution inside
but got AttributeError: __enter__. After reading up online using with seems not to be an option. Therefore I'm looking to find another elegant way of doing so.
I also tried just using os statements like so:
cwd = os.getcwd()
os.chdir('/directory')
..run code inside directory
os.chdir(cwd)
but this is a pain during debugging and seems like a bad practice.
You can write your own context manager to temporarily change the working directory.
import contextlib
#contextlib.contextmanager
def new_cd(x):
d = os.getcwd()
# This could raise an exception, but it's probably
# best to let it propagate and let the caller
# deal with it, since they requested x
os.chdir(x)
try:
yield
finally:
# This could also raise an exception, but you *really*
# aren't equipped to figure out what went wrong if the
# old working directory can't be restored.
os.chdir(d)
with new_cd('/directory'):
...
Consider spawning a subprocess:
import subprocess
subprocess.run(cmd, cwd="/directory")
See https://docs.python.org/3/library/subprocess.html.
You can make your own context manager. This answer is similar #chepner 's, but will still change back the current working directory in case of error.
import contextlib
import os
#contextlib.contextmanager
#temporarily change to a different working directory
def temporaryWorkingDirectory(path):
_oldCWD = os.getcwd()
os.chdir(os.path.abspath(path))
try:
yield
finally:
os.chdir(_oldCWD)
I tested it using this
#test
print(os.getcwd())
with temporaryWorkingDirectory("../"):
print(os.getcwd())
print(os.getcwd())
I need to get the location of the home directory of the current logged-on user. Currently, I've been using the following on Linux:
os.getenv("HOME")
However, this does not work on Windows. What is the correct cross-platform way to do this ?
You want to use os.path.expanduser.
This will ensure it works on all platforms:
from os.path import expanduser
home = expanduser("~")
If you're on Python 3.5+ you can use pathlib.Path.home():
from pathlib import Path
home = str(Path.home())
I found that pathlib module also supports this.
from pathlib import Path
>>> Path.home()
WindowsPath('C:/Users/XXX')
I know this is an old thread, but I recently needed this for a large scale project (Python 3.8). It had to work on any mainstream OS, so therefore I went with the solution #Max wrote in the comments.
Code:
import os
print(os.path.expanduser("~"))
Output Windows:
PS C:\Python> & C:/Python38/python.exe c:/Python/test.py
C:\Users\mXXXXX
Output Linux (Ubuntu):
rxxx#xx:/mnt/c/Python$ python3 test.py
/home/rxxx
I also tested it on Python 2.7.17 and that works too.
This can be done using pathlib, which is part of the standard library, and treats paths as objects with methods, instead of strings.
Path.expanduser()
Path.home()
from pathlib import Path
home: str = str(Path('~').expanduser())
This doesn't really qualify for the question (it being tagged as cross-platform), but perhaps this could be useful for someone.
How to get the home directory for effective user (Linux specific).
Let's imagine that you are writing an installer script or some other solution that requires you to perform certain actions under certain local users. You would most likely accomplish this in your installer script by changing the effective user, but os.path.expanduser("~") will still return /root.
The argument needs to have the desired user name:
os.path.expanduser(f"~{USERNAME}/")
Note that the above works fine without changing EUID, but if the scenario previously described would apply, the example below shows how this could be used:
import os
import pwd
import grp
class Identity():
def __init__(self, user: str, group: str = None):
self.uid = pwd.getpwnam(user).pw_uid
if not group:
self.gid = pwd.getpwnam(user).pw_gid
else:
self.gid = grp.getgrnam(group).gr_gid
def __enter__(self):
self.original_uid = os.getuid()
self.original_gid = os.getgid()
os.setegid(self.uid)
os.seteuid(self.gid)
def __exit__(self, type, value, traceback):
os.seteuid(self.original_uid)
os.setegid(self.original_gid)
if __name__ == '__main__':
with Identity("hedy", "lamarr"):
homedir = os.path.expanduser(f"~{pwd.getpwuid(os.geteuid())[0]}/")
with open(os.path.join(homedir, "install.log"), "w") as file:
file.write("Your home directory contents have been altered")
I would like to see what is the best way to determine the current script directory in Python.
I discovered that, due to the many ways of calling Python code, it is hard to find a good solution.
Here are some problems:
__file__ is not defined if the script is executed with exec, execfile
__module__ is defined only in modules
Use cases:
./myfile.py
python myfile.py
./somedir/myfile.py
python somedir/myfile.py
execfile('myfile.py') (from another script, that can be located in another directory and that can have another current directory.
I know that there is no perfect solution, but I'm looking for the best approach that solves most of the cases.
The most used approach is os.path.dirname(os.path.abspath(__file__)) but this really doesn't work if you execute the script from another one with exec().
Warning
Any solution that uses current directory will fail, this can be different based on the way the script is called or it can be changed inside the running script.
os.path.dirname(os.path.abspath(__file__))
is indeed the best you're going to get.
It's unusual to be executing a script with exec/execfile; normally you should be using the module infrastructure to load scripts. If you must use these methods, I suggest setting __file__ in the globals you pass to the script so it can read that filename.
There's no other way to get the filename in execed code: as you note, the CWD may be in a completely different place.
If you really want to cover the case that a script is called via execfile(...), you can use the inspect module to deduce the filename (including the path). As far as I am aware, this will work for all cases you listed:
filename = inspect.getframeinfo(inspect.currentframe()).filename
path = os.path.dirname(os.path.abspath(filename))
In Python 3.4+ you can use the simpler pathlib module:
from inspect import currentframe, getframeinfo
from pathlib import Path
filename = getframeinfo(currentframe()).filename
parent = Path(filename).resolve().parent
You can also use __file__ (when it's available) to avoid the inspect module altogether:
from pathlib import Path
parent = Path(__file__).resolve().parent
#!/usr/bin/env python
import inspect
import os
import sys
def get_script_dir(follow_symlinks=True):
if getattr(sys, 'frozen', False): # py2exe, PyInstaller, cx_Freeze
path = os.path.abspath(sys.executable)
else:
path = inspect.getabsfile(get_script_dir)
if follow_symlinks:
path = os.path.realpath(path)
return os.path.dirname(path)
print(get_script_dir())
It works on CPython, Jython, Pypy. It works if the script is executed using execfile() (sys.argv[0] and __file__ -based solutions would fail here). It works if the script is inside an executable zip file (/an egg). It works if the script is "imported" (PYTHONPATH=/path/to/library.zip python -mscript_to_run) from a zip file; it returns the archive path in this case. It works if the script is compiled into a standalone executable (sys.frozen). It works for symlinks (realpath eliminates symbolic links). It works in an interactive interpreter; it returns the current working directory in this case.
The os.path... approach was the 'done thing' in Python 2.
In Python 3, you can find directory of script as follows:
from pathlib import Path
script_path = Path(__file__).parent
Note: this answer is now a package (also with safe relative importing capabilities)
https://github.com/heetbeet/locate
$ pip install locate
$ python
>>> from locate import this_dir
>>> print(this_dir())
C:/Users/simon
For .py scripts as well as interactive usage:
I frequently use the directory of my scripts (for accessing files stored alongside them), but I also frequently run these scripts in an interactive shell for debugging purposes. I define this_dir as:
When running or importing a .py file, the file's base directory. This is always the correct path.
When running an .ipyn notebook, the current working directory. This is always the correct path, since Jupyter sets the working directory as the .ipynb base directory.
When running in a REPL, the current working directory. Hmm, what is the actual "correct path" when the code is detached from a file? Rather, make it your responsibility to change into the "correct path" before invoking the REPL.
Python 3.4 (and above):
from pathlib import Path
this_dir = Path(globals().get("__file__", "./_")).absolute().parent
Python 2 (and above):
import os
this_dir = os.path.dirname(os.path.abspath(globals().get("__file__", "./_")))
Explanation:
globals() returns all the global variables as a dictionary.
.get("__file__", "./_") returns the value from the key "__file__" if it exists in globals(), otherwise it returns the provided default value "./_".
The rest of the code just expands __file__ (or "./_") into an absolute filepath, and then returns the filepath's base directory.
Alternative:
If you know for certain that __file__ is available to your surrounding code, you can simplify to this:
>= Python 3.4: this_dir = Path(__file__).absolute().parent
>= Python 2: this_dir = os.path.dirname(os.path.abspath(__file__))
Would
import os
cwd = os.getcwd()
do what you want? I'm not sure what exactly you mean by the "current script directory". What would the expected output be for the use cases you gave?
Just use os.path.dirname(os.path.abspath(__file__)) and examine very carefully whether there is a real need for the case where exec is used. It could be a sign of troubled design if you are not able to use your script as a module.
Keep in mind Zen of Python #8, and if you believe there is a good argument for a use-case where it must work for exec, then please let us know some more details about the background of the problem.
First.. a couple missing use-cases here if we're talking about ways to inject anonymous code..
code.compile_command()
code.interact()
imp.load_compiled()
imp.load_dynamic()
imp.load_module()
__builtin__.compile()
loading C compiled shared objects? example: _socket?)
But, the real question is, what is your goal - are you trying to enforce some sort of security? Or are you just interested in whats being loaded.
If you're interested in security, the filename that is being imported via exec/execfile is inconsequential - you should use rexec, which offers the following:
This module contains the RExec class,
which supports r_eval(), r_execfile(),
r_exec(), and r_import() methods, which
are restricted versions of the standard
Python functions eval(), execfile() and
the exec and import statements. Code
executed in this restricted environment
will only have access to modules and
functions that are deemed safe; you can
subclass RExec add or remove capabilities as
desired.
However, if this is more of an academic pursuit.. here are a couple goofy approaches that you
might be able to dig a little deeper into..
Example scripts:
./deep.py
print ' >> level 1'
execfile('deeper.py')
print ' << level 1'
./deeper.py
print '\t >> level 2'
exec("import sys; sys.path.append('/tmp'); import deepest")
print '\t << level 2'
/tmp/deepest.py
print '\t\t >> level 3'
print '\t\t\t I can see the earths core.'
print '\t\t << level 3'
./codespy.py
import sys, os
def overseer(frame, event, arg):
print "loaded(%s)" % os.path.abspath(frame.f_code.co_filename)
sys.settrace(overseer)
execfile("deep.py")
sys.exit(0)
Output
loaded(/Users/synthesizerpatel/deep.py)
>> level 1
loaded(/Users/synthesizerpatel/deeper.py)
>> level 2
loaded(/Users/synthesizerpatel/<string>)
loaded(/tmp/deepest.py)
>> level 3
I can see the earths core.
<< level 3
<< level 2
<< level 1
Of course, this is a resource-intensive way to do it, you'd be tracing
all your code.. Not very efficient. But, I think it's a novel approach
since it continues to work even as you get deeper into the nest.
You can't override 'eval'. Although you can override execfile().
Note, this approach only coveres exec/execfile, not 'import'.
For higher level 'module' load hooking you might be able to use use
sys.path_hooks (Write-up courtesy of PyMOTW).
Thats all I have off the top of my head.
Here is a partial solution, still better than all published ones so far.
import sys, os, os.path, inspect
#os.chdir("..")
if '__file__' not in locals():
__file__ = inspect.getframeinfo(inspect.currentframe())[0]
print os.path.dirname(os.path.abspath(__file__))
Now this works will all calls but if someone use chdir() to change the current directory, this will also fail.
Notes:
sys.argv[0] is not going to work, will return -c if you execute the script with python -c "execfile('path-tester.py')"
I published a complete test at https://gist.github.com/1385555 and you are welcome to improve it.
To get the absolute path to the directory containing the current script you can use:
from pathlib import Path
absDir = Path(__file__).parent.resolve()
Please note the .resolve() call is required, because that is the one making the path absolute. Without resolve(), you would obtain something like '.'.
This solution uses pathlib, which is part of Python's stdlib since v3.4 (2014). This is preferrable compared to other solutions using os.
The official pathlib documentation has a useful table mapping the old os functions to the new ones: https://docs.python.org/3/library/pathlib.html#correspondence-to-tools-in-the-os-module
This should work in most cases:
import os,sys
dirname=os.path.dirname(os.path.realpath(sys.argv[0]))
Hopefully this helps:-
If you run a script/module from anywhere you'll be able to access the __file__ variable which is a module variable representing the location of the script.
On the other hand, if you're using the interpreter you don't have access to that variable, where you'll get a name NameError and os.getcwd() will give you the incorrect directory if you're running the file from somewhere else.
This solution should give you what you're looking for in all cases:
from inspect import getsourcefile
from os.path import abspath
abspath(getsourcefile(lambda:0))
I haven't thoroughly tested it but it solved my problem.
If __file__ is available:
# -- script1.py --
import os
file_path = os.path.abspath(__file__)
print(os.path.dirname(file_path))
For those we want to be able to run the command from the interpreter or get the path of the place you're running the script from:
# -- script2.py --
import os
print(os.path.abspath(''))
This works from the interpreter.
But when run in a script (or imported) it gives the path of the place where
you ran the script from, not the path of directory containing
the script with the print.
Example:
If your directory structure is
test_dir (in the home dir)
├── main.py
└── test_subdir
├── script1.py
└── script2.py
with
# -- main.py --
import script1.py
import script2.py
The output is:
~/test_dir/test_subdir
~/test_dir
As previous answers require you to import some module, I thought that I would write one answer that doesn't. Use the code below if you don't want to import anything.
this_dir = '/'.join(__file__.split('/')[:-1])
print(this_dir)
If the script is on /path/to/script.py then this would print /path/to. Note that this will throw error on terminal as no file is executed. This basically parse the directory from __file__ removing the last part of it. In this case /script.py is removed to produce the output /path/to.
print(__import__("pathlib").Path(__file__).parent)
When changing directory with the OS module , the change is made globally. Is there a way to change directory locally?
EDIT:Yea, sorry . Locally in another thread. The normal chdir changes the working directory over all threads when you use the thread module.
It's not very difficult to write a decorator/context-manager to do this using contextlib2.contextmanager.
import contextlib2
import os
#contextlib2.contextmanager
def restore_chdir(dir_):
orig= os.getcwd()
os.chdir(dir_)
try:
yield
finally:
os.chdir(orig)
You can use this now as:
with restore_chdir('/foo'):
...
or as a function decorator.
You can simply define a function with os module, which can restore the directory.
import os
def func():
original = os.getcwd()
os.chdir("Your Directory Name")
## Do something here
os.chdir(original)
I have a python program that creates temporary directories under /temp by using tempfile.mkdtemp. Unfortunately, the Python program did not delete the directory after using it. So now the disk space is low.
Questions:
How do I delete the temporary directories left under /temp manually? I tried to delete them manually but got "permission denied" error.
In the Python program, how to delete temp directory after using them?
To manage resources (like files) in Python, best practice is to use the with keyword, which automatically releases the resources (i.e., cleans up, like closing files); this is available from Python 2.5.
From Python 3.2, you can use tempfile.TemporaryDirectory() instead of tempfile.mkdtmp() – this is usable in with and automatically cleans up the directory:
from tempfile import TemporaryDirectory
with TemporaryDirectory() as temp_dir:
# ... do something with temp_dir
# automatically cleaned up when context exited
If you are using an earlier version of Python (at least 2.5, so have with), you can use backports.tempfile; see Nicholas Bishop’s answer to tempfile.TemporaryDirectory context manager in Python 2.7.
It’s easy and instructive to roll your own class, called a
context manager. The return value of the __enter__() method is bound to the target of the as clause, while the __exit__() method is called when the context is exited – even by exception – and performs cleanup.
import shutil
import tempfile
class TemporaryDirectory(object):
"""Context manager for tempfile.mkdtemp() so it's usable with "with" statement."""
def __enter__(self):
self.name = tempfile.mkdtemp()
return self.name
def __exit__(self, exc_type, exc_value, traceback):
shutil.rmtree(self.name)
You can simplify this with the #contextlib.contextmanager decorator, so you don’t need to write a context manager manually. The code prior to the yield is executed when entering the context, the yielded value is bound to the target of the as, and the code after the yield is executed when exiting the context. This is fundamentally a coroutine that encapsulates the resource acquisition and release, with the yield yielding control to the suite (body) of the with clause. Note that here you do need to have a try...finally block, as #contextlib.contextmanager does not catch exceptions in the yield – this just factors the resource management into a coroutine.
from contextlib import contextmanager
import tempfile
import shutil
#contextmanager
def TemporaryDirectory():
name = tempfile.mkdtemp()
try:
yield name
finally:
shutil.rmtree(name)
As simplylizz notes, if you don’t mind the directory already being deleted (which the above code assumes does not happen), you can catch the “No such file or directory” exception as follows:
import errno
# ...
try:
shutil.rmtree(self.name)
except OSError as e:
# Reraise unless ENOENT: No such file or directory
# (ok if directory has already been deleted)
if e.errno != errno.ENOENT:
raise
You can compare with the standard implementation in tempfile.py; even this simple class has had bugs and evolved over the years.
For background on with, see:
The Python Tutorial: Methods of File Objects
With Statement Context Managers
PEP 343 -- The "with" Statement
Read the documentation, it's simple. ;) From the docs: the directory is readable, writable, and searchable only by the creating user ID.
To delete temp directory try something like this:
import errno
import shutil
import tempfile
try:
tmp_dir = tempfile.mkdtemp() # create dir
# ... do something
finally:
try:
shutil.rmtree(tmp_dir) # delete directory
except OSError as exc:
if exc.errno != errno.ENOENT: # ENOENT - no such file or directory
raise # re-raise exception
Also you can try tempdir package or see its sources.
I have had the same/similar problem using TemporaryDirectory() which is basically covering the functionality you defined above.
My problem was due to the usage of the temporary directory. I used to populate the content by cloning a git repository, it happened that there were read-only files created during the process and with a normal exit these read-only temp files were causing the complete temporary directory to stay there.
Did inherit the TemporaryDirectory to my own class and did override the class method _cleanup with the following code.
The code before super() might be optimized but for me, the performance was not the problem.
I did use the force and read the source of "tempfile"
import tempfile
import shutil
import stat
class myTempDir(tempfile.TemporaryDirectory):
#classmethod
def _cleanup(self,name, warn_message):
for root, dirs, files in os.walk(name):
for fname in files:
full_path = os.path.join(root, fname)
os.chmod(full_path ,stat.S_IWRITE)
super()
Solution worked for Windows 10 with Python 3
I think the user is responsible for deleting the temporary directory and its content created by using tempfile.mkdtemp().
it will not deleted automatically just like temporary file.
There are many ways you can delete the directory
If the directory is empty you can use
`os.removedirs or os.rmdir`
Note it can be used only if the directory is empty otherwise will raise
OSError
This will delete the entire directory path:
import shutil
shutil.rmtree('/path/to/your/dir/')
be careful while using this, it will delete the whole directory and files inside it.
An alternative to the with statement could be:
import tempfile
tmp_dir = tempfile.TemporaryDirectory()
print(tmp_dir.name)
# Your code Here
tmp_dir.cleanup()