I have a python program that creates temporary directories under /temp by using tempfile.mkdtemp. Unfortunately, the Python program did not delete the directory after using it. So now the disk space is low.
Questions:
How do I delete the temporary directories left under /temp manually? I tried to delete them manually but got "permission denied" error.
In the Python program, how to delete temp directory after using them?
To manage resources (like files) in Python, best practice is to use the with keyword, which automatically releases the resources (i.e., cleans up, like closing files); this is available from Python 2.5.
From Python 3.2, you can use tempfile.TemporaryDirectory() instead of tempfile.mkdtmp() – this is usable in with and automatically cleans up the directory:
from tempfile import TemporaryDirectory
with TemporaryDirectory() as temp_dir:
# ... do something with temp_dir
# automatically cleaned up when context exited
If you are using an earlier version of Python (at least 2.5, so have with), you can use backports.tempfile; see Nicholas Bishop’s answer to tempfile.TemporaryDirectory context manager in Python 2.7.
It’s easy and instructive to roll your own class, called a
context manager. The return value of the __enter__() method is bound to the target of the as clause, while the __exit__() method is called when the context is exited – even by exception – and performs cleanup.
import shutil
import tempfile
class TemporaryDirectory(object):
"""Context manager for tempfile.mkdtemp() so it's usable with "with" statement."""
def __enter__(self):
self.name = tempfile.mkdtemp()
return self.name
def __exit__(self, exc_type, exc_value, traceback):
shutil.rmtree(self.name)
You can simplify this with the #contextlib.contextmanager decorator, so you don’t need to write a context manager manually. The code prior to the yield is executed when entering the context, the yielded value is bound to the target of the as, and the code after the yield is executed when exiting the context. This is fundamentally a coroutine that encapsulates the resource acquisition and release, with the yield yielding control to the suite (body) of the with clause. Note that here you do need to have a try...finally block, as #contextlib.contextmanager does not catch exceptions in the yield – this just factors the resource management into a coroutine.
from contextlib import contextmanager
import tempfile
import shutil
#contextmanager
def TemporaryDirectory():
name = tempfile.mkdtemp()
try:
yield name
finally:
shutil.rmtree(name)
As simplylizz notes, if you don’t mind the directory already being deleted (which the above code assumes does not happen), you can catch the “No such file or directory” exception as follows:
import errno
# ...
try:
shutil.rmtree(self.name)
except OSError as e:
# Reraise unless ENOENT: No such file or directory
# (ok if directory has already been deleted)
if e.errno != errno.ENOENT:
raise
You can compare with the standard implementation in tempfile.py; even this simple class has had bugs and evolved over the years.
For background on with, see:
The Python Tutorial: Methods of File Objects
With Statement Context Managers
PEP 343 -- The "with" Statement
Read the documentation, it's simple. ;) From the docs: the directory is readable, writable, and searchable only by the creating user ID.
To delete temp directory try something like this:
import errno
import shutil
import tempfile
try:
tmp_dir = tempfile.mkdtemp() # create dir
# ... do something
finally:
try:
shutil.rmtree(tmp_dir) # delete directory
except OSError as exc:
if exc.errno != errno.ENOENT: # ENOENT - no such file or directory
raise # re-raise exception
Also you can try tempdir package or see its sources.
I have had the same/similar problem using TemporaryDirectory() which is basically covering the functionality you defined above.
My problem was due to the usage of the temporary directory. I used to populate the content by cloning a git repository, it happened that there were read-only files created during the process and with a normal exit these read-only temp files were causing the complete temporary directory to stay there.
Did inherit the TemporaryDirectory to my own class and did override the class method _cleanup with the following code.
The code before super() might be optimized but for me, the performance was not the problem.
I did use the force and read the source of "tempfile"
import tempfile
import shutil
import stat
class myTempDir(tempfile.TemporaryDirectory):
#classmethod
def _cleanup(self,name, warn_message):
for root, dirs, files in os.walk(name):
for fname in files:
full_path = os.path.join(root, fname)
os.chmod(full_path ,stat.S_IWRITE)
super()
Solution worked for Windows 10 with Python 3
I think the user is responsible for deleting the temporary directory and its content created by using tempfile.mkdtemp().
it will not deleted automatically just like temporary file.
There are many ways you can delete the directory
If the directory is empty you can use
`os.removedirs or os.rmdir`
Note it can be used only if the directory is empty otherwise will raise
OSError
This will delete the entire directory path:
import shutil
shutil.rmtree('/path/to/your/dir/')
be careful while using this, it will delete the whole directory and files inside it.
An alternative to the with statement could be:
import tempfile
tmp_dir = tempfile.TemporaryDirectory()
print(tmp_dir.name)
# Your code Here
tmp_dir.cleanup()
Related
I need to perform an action without changing the global working directory. My case is I have a few folders, and in each, there are a few files. I need to do some computations using those files. Initially, I tried the following:
with os.chdir('/directory'):
...some code needing execution inside
but got AttributeError: __enter__. After reading up online using with seems not to be an option. Therefore I'm looking to find another elegant way of doing so.
I also tried just using os statements like so:
cwd = os.getcwd()
os.chdir('/directory')
..run code inside directory
os.chdir(cwd)
but this is a pain during debugging and seems like a bad practice.
You can write your own context manager to temporarily change the working directory.
import contextlib
#contextlib.contextmanager
def new_cd(x):
d = os.getcwd()
# This could raise an exception, but it's probably
# best to let it propagate and let the caller
# deal with it, since they requested x
os.chdir(x)
try:
yield
finally:
# This could also raise an exception, but you *really*
# aren't equipped to figure out what went wrong if the
# old working directory can't be restored.
os.chdir(d)
with new_cd('/directory'):
...
Consider spawning a subprocess:
import subprocess
subprocess.run(cmd, cwd="/directory")
See https://docs.python.org/3/library/subprocess.html.
You can make your own context manager. This answer is similar #chepner 's, but will still change back the current working directory in case of error.
import contextlib
import os
#contextlib.contextmanager
#temporarily change to a different working directory
def temporaryWorkingDirectory(path):
_oldCWD = os.getcwd()
os.chdir(os.path.abspath(path))
try:
yield
finally:
os.chdir(_oldCWD)
I tested it using this
#test
print(os.getcwd())
with temporaryWorkingDirectory("../"):
print(os.getcwd())
print(os.getcwd())
I want to add libraries access control to my installation of python and I wanted to know if there is some way to hook into the import system, so that I can run a script to check if a python program is allowed to import libraries, to block untrusted modules from importing dangerous native modules that could leak information on my system, like os.
While researching by myself, I found out about PEP 302, which sounds like what I am looking for, but I couldn't find how to register those hooks installation-wide.
Would someone be able to tell me if there is a way in python to add such an import hook to all imports on the system rather than only on the currently executing program?
You can change the import of modules by implementing you own custom import loader object. A starting point in the documentation can be found here: https://docs.python.org/3/library/importlib.html
What you need to do is to create a loader that will act on the packages you want to check on, and then either load them, or raise a desired exception. In the case of modules what are not in your access control list, you should return None, this makes the import machinery load them normally. I have create a minimal example of this type of functionality that you can start from and extend to build your desired functionality.
import sys
import importlib
class ImportInterceptor(importlib.abc.Loader):
def __init__(self, package_permissions):
self.package_permissions = package_permissions
def find_module(self, fullname, path=None):
if fullname in self.package_permissions:
if self.package_permissions[fullname]:
return self
else:
raise ImportError("Package import was not allowed")
def load_module(self, fullname):
sys.meta_path = [x for x in sys.meta_path[1:] if x is not self]
module = importlib.import_module(fullname)
sys.meta_path = [self] + sys.meta_path
return module
if not hasattr(sys,'frozen'):
sys.meta_path = [ImportInterceptor({'textwrap': True, 'Pathlib': False})] + sys.meta_path
import textwrap
print(textwrap.dedent(' test'))
# Works fine
from pathlib import Path
# Raises exception
Note that the loader removes itself from sys.meta_path when loading the package. This is to avoid an infinite loop where it keeps calling itself every time it tries to load the module "for real".
I am writing a unit test for embedded software code in python.
One of the files require a specific file to exist. (e.g. "/dir_name/file_name.txt") otherwise it raises an error.
Normally, this file exists on hardware device and my python code reads this file. When writing unit tests for python code, how can I mock the existence of the file?
tempfile.mkstemp() does not seem to generate the exact path/file name I want, which is/dir_name/file_name.txt. It always adds some random letters.
This is with Python3.4. Is it possible to accomplish this with unittest.mock ?
You could create a context manager that creates and deletes the file.
from contextlib import contextmanager
#contextmanager
def mock_file(filepath, content=''):
with open(filepath, 'w') as f:
f.write(content)
yield filepath
try:
os.remove(filepath)
except Exception:
pass
def test_function():
with mock_file(r'/dirname/filename.txt'):
function()
This snippet is from an earlier answer here on SO. It is about a year old (and the answer was not accepted). I am new to Python and I am finding the system path a real pain. I have a few functions written in scripts in different directories, and I would like to be able to import them into new projects without having to jump through hoops.
This is the snippet:
def import_path(fullpath):
""" Import a file with full path specification. Allows one to
import from anywhere, something __import__ does not do.
"""
path, filename = os.path.split(fullpath)
filename, ext = os.path.splitext(filename)
sys.path.append(path)
module = __import__(filename)
reload(module) # Might be out of date
del sys.path[-1]
return module
Its from here:
How to do relative imports in Python?
I would like some feedback as to whether I can use it or not - and if there are any undesirable side effects that may not be obvious to a newbie.
I intend to use it something like this:
import_path(/home/pydev/path1/script1.py)
script1.func1()
etc
Is it 'safe' to use the function in the way I intend to?
The "official" and fully safe approach is the imp module of the standard Python library.
Use imp.find_module to find the module on your precisely-specified list of acceptable directories -- it returns a 3-tuple (file, pathname, description) -- if unsuccessful, file is actually None (but it can also raise ImportError so you should use a try/except for that as well as checking if file is None:).
If the search is successful, call imp.load_module (in a try/finally to make sure you close the file!) with the above three arguments after the first one which must be the same name you passed to find_module -- it returns the module object (phew;-).
As mentioned, please consider thread safety, if appropriate. I prefer something closer to a solution posted in a similar post. The main differences below: the use of insert to specify priority of the import, correct restoration of sys.path using try...finally, and setting the global namespace.
# inspired by Alex Martelli's solution to
# http://stackoverflow.com/questions/1096216/override-namespace-in-python/1096247#1096247
def import_from_absolute_path(fullpath, global_name=None):
"""Dynamic script import using full path."""
import os
import sys
script_dir, filename = os.path.split(fullpath)
script, ext = os.path.splitext(filename)
sys.path.insert(0, script_dir)
try:
module = __import__(script)
if global_name is None:
global_name = script
globals()[global_name] = module
sys.modules[global_name] = module
finally:
del sys.path[0]
It does feel like a bit of a hack, but at the moment, I can't think of any unintended side effects that are likely to occur, at least not as long as you're just using this for your own scripts. Basically what it does is temporarily add the parent directory of the specified file (in your example, /home/pydev/path1/) to the list of paths that Python checks when it's looking for a module to import.
The only risk I can think of right now would arise in a multithreaded environment, where two or more threads (or processes) are running this function simultaneously. If thread A wants to import module A from path dirA/A.py, and thread B wants to import module B from path dirB/B.py, you'd wind up with both dirA and dirB in sys.path for a short time. And if there is a file named B.py in dirA, it's possible that thread B will find that (dirA/B.py) instead of the file it's looking for (dirB/B.py), thus importing the wrong module. For this reason, I wouldn't use it in production code, or code that you're going to distribute to other people (at least not without warning them that this hack is in here!). In a situation like that, you could write a more complex function that allows you to specify the file to import without messing with the standard set of paths. (That's what mod_python does, for example)
I would be worried that your script name might correspond with a module that shows up earlier in the path. To dispel this fear, I would fully replace the path with a new list containing just the directory containing the module, then put it back once the import has completed. Also, you should wrap this in some sort of lock so that multiple threads trying to do the same thing don't interfere with each other.
shutil.rmtree will not delete read-only files on Windows. Is there a python equivalent of "rm -rf" ? Why oh why is this such a pain?
shutil.rmtree can take an error-handling function that will be called when it has problem removing a file. You can use that to force the removal of the problematic file(s).
Inspired by http://mail.python.org/pipermail/tutor/2006-June/047551.html and http://techarttiki.blogspot.com/2008/08/read-only-windows-files-with-python.html:
import os
import stat
import shutil
def remove_readonly(func, path, excinfo):
os.chmod(path, stat.S_IWRITE)
func(path)
shutil.rmtree(top, onerror=remove_readonly)
(I haven't tested that snippet out, but it should be enough to get you started)
If you import win32api from PyWin32, you can use:
win32api.SetFileAttributes(path, win32con.FILE_ATTRIBUTE_NORMAL)
To make files cease to be read-only.
Another way is to define rmtree on Windows as
rmtree = lambda path: subprocess.check_call(['cmd', '/c', 'rd', '/s', '/q', path])
There's a comment at the ActiveState site that says:
shutil.rmtree has its shortcomings. Although it is true you can use shutil.rmtree() in many cases, there are some cases where it does not work. For example, files that are marked read-only under Windows cannot be deleted by shutil.rmtree().
By importing the win32api and win32con modules from PyWin32 and adding line like "win32api.SetFileAttributes(path, win32con.FILE_ATTRIBUTE_NORMAL" to the rmgeneric() function, this obstacle can be overcome. I used this approach to fix the hot-backup.py script of Subversion 1.4 so it will work under Windows. Thank you for the recipe.
I don't use Windows so can't verify whether this works or not.
This will presumably be fixed with the release of Python 3.5 (currently - June 2015 - still in development) in the sense of giving a hint about this in the documentation.
You can find the bugreport here. And this is the according changeset.
See the newly added example from the Python 3.5 docs:
import os, stat
import shutil
def remove_readonly(func, path, _):
"Clear the readonly bit and reattempt the removal"
os.chmod(path, stat.S_IWRITE)
func(path)
shutil.rmtree(directory, onerror=remove_readonly)
Here is a variant of what Steve posted, it uses the same basic mechanism, and this one is tested :-)
What user do python scripts run as in windows?
I had this issue on Python 3.7 when invoking rmtree() and worked around it by explicitly fixing the permissions before exiting the TemporaryDirectory() context manager. See akaihola/darker#453 for details. Here's a copy of the implementation:
import os
import sys
from pathlib import Path
from typing import Union
WINDOWS = sys.platform.startswith("win")
def fix_py37_win_tempdir_permissions(dirpath: Union[str, Path]) -> None:
"""Work around a `tempfile` clean-up issue on Windows with Python 3.7
Call this before exiting a ``with TemporaryDirectory():`` block or in teardown for
a Pytest fixture which creates a temporary directory.
See discussion in https://github.com/akaihola/darker/pull/393
Solution borrowed from https://github.com/python/cpython/pull/10320
:param dirpath: The root path of the temporary directory
"""
if not WINDOWS or sys.version_info >= (3, 8):
return
for root, dirs, files in os.walk(dirpath):
for name in dirs + files:
path = os.path.join(root, name)
try:
os.chflags(path, 0) # type: ignore[attr-defined]
except AttributeError:
pass
os.chmod(path, 0o700)
and here's how to use it in Pytest unit tests or when creating a temporary directory using tempfile:
import pytest
from my_utils import fix_py37_win_tempdir_permissions
#pytest.fixture
def myfixture(tmp_path):
# setup code here
yield tmp_path
fix_py37_win_tempdir_permissions(tmp_path)
def myfunc():
with TemporaryDirectory() as tmpdir:
# work on the temporary directory here
fix_py37_win_tempdir_permissions(tmp_path)