Related
I have a path (including directory and file name).
I need to test if the file-name is a valid, e.g. if the file-system will allow me to create a file with such a name.
The file-name has some unicode characters in it.
It's safe to assume the directory segment of the path is valid and accessible (I was trying to make the question more gnerally applicable, and apparently I wen too far).
I very much do not want to have to escape anything unless I have to.
I'd post some of the example characters I am dealing with, but apparently they get automatically removed by the stack-exchange system. Anyways, I want to keep standard unicode entities like ö, and only escape things which are invalid in a filename.
Here is the catch. There may (or may not) already be a file at the target of the path. I need to keep that file if it does exist, and not create a file if it does not.
Basically I want to check if I could write to a path without actually opening the path for writing (and the automatic file creation/file clobbering that typically entails).
As such:
try:
open(filename, 'w')
except OSError:
# handle error here
from here
Is not acceptable, because it will overwrite the existent file, which I do not want to touch (if it's there), or create said file if it's not.
I know I can do:
if not os.access(filePath, os.W_OK):
try:
open(filePath, 'w').close()
os.unlink(filePath)
except OSError:
# handle error here
But that will create the file at the filePath, which I would then have to os.unlink.
In the end, it seems like it's spending 6 or 7 lines to do something that should be as simple as os.isvalidpath(filePath) or similar.
As an aside, I need this to run on (at least) Windows and MacOS, so I'd like to avoid platform-specific stuff.
``
tl;dr
Call the is_path_exists_or_creatable() function defined below.
Strictly Python 3. That's just how we roll.
A Tale of Two Questions
The question of "How do I test pathname validity and, for valid pathnames, the existence or writability of those paths?" is clearly two separate questions. Both are interesting, and neither have received a genuinely satisfactory answer here... or, well, anywhere that I could grep.
vikki's answer probably hews the closest, but has the remarkable disadvantages of:
Needlessly opening (...and then failing to reliably close) file handles.
Needlessly writing (...and then failing to reliable close or delete) 0-byte files.
Ignoring OS-specific errors differentiating between non-ignorable invalid pathnames and ignorable filesystem issues. Unsurprisingly, this is critical under Windows. (See below.)
Ignoring race conditions resulting from external processes concurrently (re)moving parent directories of the pathname to be tested. (See below.)
Ignoring connection timeouts resulting from this pathname residing on stale, slow, or otherwise temporarily inaccessible filesystems. This could expose public-facing services to potential DoS-driven attacks. (See below.)
We're gonna fix all that.
Question #0: What's Pathname Validity Again?
Before hurling our fragile meat suits into the python-riddled moshpits of pain, we should probably define what we mean by "pathname validity." What defines validity, exactly?
By "pathname validity," we mean the syntactic correctness of a pathname with respect to the root filesystem of the current system – regardless of whether that path or parent directories thereof physically exist. A pathname is syntactically correct under this definition if it complies with all syntactic requirements of the root filesystem.
By "root filesystem," we mean:
On POSIX-compatible systems, the filesystem mounted to the root directory (/).
On Windows, the filesystem mounted to %HOMEDRIVE%, the colon-suffixed drive letter containing the current Windows installation (typically but not necessarily C:).
The meaning of "syntactic correctness," in turn, depends on the type of root filesystem. For ext4 (and most but not all POSIX-compatible) filesystems, a pathname is syntactically correct if and only if that pathname:
Contains no null bytes (i.e., \x00 in Python). This is a hard requirement for all POSIX-compatible filesystems.
Contains no path components longer than 255 bytes (e.g., 'a'*256 in Python). A path component is a longest substring of a pathname containing no / character (e.g., bergtatt, ind, i, and fjeldkamrene in the pathname /bergtatt/ind/i/fjeldkamrene).
Syntactic correctness. Root filesystem. That's it.
Question #1: How Now Shall We Do Pathname Validity?
Validating pathnames in Python is surprisingly non-intuitive. I'm in firm agreement with Fake Name here: the official os.path package should provide an out-of-the-box solution for this. For unknown (and probably uncompelling) reasons, it doesn't. Fortunately, unrolling your own ad-hoc solution isn't that gut-wrenching...
O.K., it actually is. It's hairy; it's nasty; it probably chortles as it burbles and giggles as it glows. But what you gonna do? Nuthin'.
We'll soon descend into the radioactive abyss of low-level code. But first, let's talk high-level shop. The standard os.stat() and os.lstat() functions raise the following exceptions when passed invalid pathnames:
For pathnames residing in non-existing directories, instances of FileNotFoundError.
For pathnames residing in existing directories:
Under Windows, instances of WindowsError whose winerror attribute is 123 (i.e., ERROR_INVALID_NAME).
Under all other OSes:
For pathnames containing null bytes (i.e., '\x00'), instances of TypeError.
For pathnames containing path components longer than 255 bytes, instances of OSError whose errcode attribute is:
Under SunOS and the *BSD family of OSes, errno.ERANGE. (This appears to be an OS-level bug, otherwise referred to as "selective interpretation" of the POSIX standard.)
Under all other OSes, errno.ENAMETOOLONG.
Crucially, this implies that only pathnames residing in existing directories are validatable. The os.stat() and os.lstat() functions raise generic FileNotFoundError exceptions when passed pathnames residing in non-existing directories, regardless of whether those pathnames are invalid or not. Directory existence takes precedence over pathname invalidity.
Does this mean that pathnames residing in non-existing directories are not validatable? Yes – unless we modify those pathnames to reside in existing directories. Is that even safely feasible, however? Shouldn't modifying a pathname prevent us from validating the original pathname?
To answer this question, recall from above that syntactically correct pathnames on the ext4 filesystem contain no path components (A) containing null bytes or (B) over 255 bytes in length. Hence, an ext4 pathname is valid if and only if all path components in that pathname are valid. This is true of most real-world filesystems of interest.
Does that pedantic insight actually help us? Yes. It reduces the larger problem of validating the full pathname in one fell swoop to the smaller problem of only validating all path components in that pathname. Any arbitrary pathname is validatable (regardless of whether that pathname resides in an existing directory or not) in a cross-platform manner by following the following algorithm:
Split that pathname into path components (e.g., the pathname /troldskog/faren/vild into the list ['', 'troldskog', 'faren', 'vild']).
For each such component:
Join the pathname of a directory guaranteed to exist with that component into a new temporary pathname (e.g., /troldskog) .
Pass that pathname to os.stat() or os.lstat(). If that pathname and hence that component is invalid, this call is guaranteed to raise an exception exposing the type of invalidity rather than a generic FileNotFoundError exception. Why? Because that pathname resides in an existing directory. (Circular logic is circular.)
Is there a directory guaranteed to exist? Yes, but typically only one: the topmost directory of the root filesystem (as defined above).
Passing pathnames residing in any other directory (and hence not guaranteed to exist) to os.stat() or os.lstat() invites race conditions, even if that directory was previously tested to exist. Why? Because external processes cannot be prevented from concurrently removing that directory after that test has been performed but before that pathname is passed to os.stat() or os.lstat(). Unleash the dogs of mind-fellating insanity!
There exists a substantial side benefit to the above approach as well: security. (Isn't that nice?) Specifically:
Front-facing applications validating arbitrary pathnames from untrusted sources by simply passing such pathnames to os.stat() or os.lstat() are susceptible to Denial of Service (DoS) attacks and other black-hat shenanigans. Malicious users may attempt to repeatedly validate pathnames residing on filesystems known to be stale or otherwise slow (e.g., NFS Samba shares); in that case, blindly statting incoming pathnames is liable to either eventually fail with connection timeouts or consume more time and resources than your feeble capacity to withstand unemployment.
The above approach obviates this by only validating the path components of a pathname against the root directory of the root filesystem. (If even that's stale, slow, or inaccessible, you've got larger problems than pathname validation.)
Lost? Great. Let's begin. (Python 3 assumed. See "What Is Fragile Hope for 300, leycec?")
import errno, os
# Sadly, Python fails to provide the following magic number for us.
ERROR_INVALID_NAME = 123
'''
Windows-specific error code indicating an invalid pathname.
See Also
----------
https://learn.microsoft.com/en-us/windows/win32/debug/system-error-codes--0-499-
Official listing of all such codes.
'''
def is_pathname_valid(pathname: str) -> bool:
'''
`True` if the passed pathname is a valid pathname for the current OS;
`False` otherwise.
'''
# If this pathname is either not a string or is but is empty, this pathname
# is invalid.
try:
if not isinstance(pathname, str) or not pathname:
return False
# Strip this pathname's Windows-specific drive specifier (e.g., `C:\`)
# if any. Since Windows prohibits path components from containing `:`
# characters, failing to strip this `:`-suffixed prefix would
# erroneously invalidate all valid absolute Windows pathnames.
_, pathname = os.path.splitdrive(pathname)
# Directory guaranteed to exist. If the current OS is Windows, this is
# the drive to which Windows was installed (e.g., the "%HOMEDRIVE%"
# environment variable); else, the typical root directory.
root_dirname = os.environ.get('HOMEDRIVE', 'C:') \
if sys.platform == 'win32' else os.path.sep
assert os.path.isdir(root_dirname) # ...Murphy and her ironclad Law
# Append a path separator to this directory if needed.
root_dirname = root_dirname.rstrip(os.path.sep) + os.path.sep
# Test whether each path component split from this pathname is valid or
# not, ignoring non-existent and non-readable path components.
for pathname_part in pathname.split(os.path.sep):
try:
os.lstat(root_dirname + pathname_part)
# If an OS-specific exception is raised, its error code
# indicates whether this pathname is valid or not. Unless this
# is the case, this exception implies an ignorable kernel or
# filesystem complaint (e.g., path not found or inaccessible).
#
# Only the following exceptions indicate invalid pathnames:
#
# * Instances of the Windows-specific "WindowsError" class
# defining the "winerror" attribute whose value is
# "ERROR_INVALID_NAME". Under Windows, "winerror" is more
# fine-grained and hence useful than the generic "errno"
# attribute. When a too-long pathname is passed, for example,
# "errno" is "ENOENT" (i.e., no such file or directory) rather
# than "ENAMETOOLONG" (i.e., file name too long).
# * Instances of the cross-platform "OSError" class defining the
# generic "errno" attribute whose value is either:
# * Under most POSIX-compatible OSes, "ENAMETOOLONG".
# * Under some edge-case OSes (e.g., SunOS, *BSD), "ERANGE".
except OSError as exc:
if hasattr(exc, 'winerror'):
if exc.winerror == ERROR_INVALID_NAME:
return False
elif exc.errno in {errno.ENAMETOOLONG, errno.ERANGE}:
return False
# If a "TypeError" exception was raised, it almost certainly has the
# error message "embedded NUL character" indicating an invalid pathname.
except TypeError as exc:
return False
# If no exception was raised, all path components and hence this
# pathname itself are valid. (Praise be to the curmudgeonly python.)
else:
return True
# If any other exception was raised, this is an unrelated fatal issue
# (e.g., a bug). Permit this exception to unwind the call stack.
#
# Did we mention this should be shipped with Python already?
Done. Don't squint at that code. (It bites.)
Question #2: Possibly Invalid Pathname Existence or Creatability, Eh?
Testing the existence or creatability of possibly invalid pathnames is, given the above solution, mostly trivial. The little key here is to call the previously defined function before testing the passed path:
def is_path_creatable(pathname: str) -> bool:
'''
`True` if the current user has sufficient permissions to create the passed
pathname; `False` otherwise.
'''
# Parent directory of the passed path. If empty, we substitute the current
# working directory (CWD) instead.
dirname = os.path.dirname(pathname) or os.getcwd()
return os.access(dirname, os.W_OK)
def is_path_exists_or_creatable(pathname: str) -> bool:
'''
`True` if the passed pathname is a valid pathname for the current OS _and_
either currently exists or is hypothetically creatable; `False` otherwise.
This function is guaranteed to _never_ raise exceptions.
'''
try:
# To prevent "os" module calls from raising undesirable exceptions on
# invalid pathnames, is_pathname_valid() is explicitly called first.
return is_pathname_valid(pathname) and (
os.path.exists(pathname) or is_path_creatable(pathname))
# Report failure on non-fatal filesystem complaints (e.g., connection
# timeouts, permissions issues) implying this path to be inaccessible. All
# other exceptions are unrelated fatal issues and should not be caught here.
except OSError:
return False
Done and done. Except not quite.
Question #3: Possibly Invalid Pathname Existence or Writability on Windows
There exists a caveat. Of course there does.
As the official os.access() documentation admits:
Note: I/O operations may fail even when os.access() indicates that they would succeed, particularly for operations on network filesystems which may have permissions semantics beyond the usual POSIX permission-bit model.
To no one's surprise, Windows is the usual suspect here. Thanks to extensive use of Access Control Lists (ACL) on NTFS filesystems, the simplistic POSIX permission-bit model maps poorly to the underlying Windows reality. While this (arguably) isn't Python's fault, it might nonetheless be of concern for Windows-compatible applications.
If this is you, a more robust alternative is wanted. If the passed path does not exist, we instead attempt to create a temporary file guaranteed to be immediately deleted in the parent directory of that path – a more portable (if expensive) test of creatability:
import os, tempfile
def is_path_sibling_creatable(pathname: str) -> bool:
'''
`True` if the current user has sufficient permissions to create **siblings**
(i.e., arbitrary files in the parent directory) of the passed pathname;
`False` otherwise.
'''
# Parent directory of the passed path. If empty, we substitute the current
# working directory (CWD) instead.
dirname = os.path.dirname(pathname) or os.getcwd()
try:
# For safety, explicitly close and hence delete this temporary file
# immediately after creating it in the passed path's parent directory.
with tempfile.TemporaryFile(dir=dirname): pass
return True
# While the exact type of exception raised by the above function depends on
# the current version of the Python interpreter, all such types subclass the
# following exception superclass.
except EnvironmentError:
return False
def is_path_exists_or_creatable_portable(pathname: str) -> bool:
'''
`True` if the passed pathname is a valid pathname on the current OS _and_
either currently exists or is hypothetically creatable in a cross-platform
manner optimized for POSIX-unfriendly filesystems; `False` otherwise.
This function is guaranteed to _never_ raise exceptions.
'''
try:
# To prevent "os" module calls from raising undesirable exceptions on
# invalid pathnames, is_pathname_valid() is explicitly called first.
return is_pathname_valid(pathname) and (
os.path.exists(pathname) or is_path_sibling_creatable(pathname))
# Report failure on non-fatal filesystem complaints (e.g., connection
# timeouts, permissions issues) implying this path to be inaccessible. All
# other exceptions are unrelated fatal issues and should not be caught here.
except OSError:
return False
Note, however, that even this may not be enough.
Thanks to User Access Control (UAC), the ever-inimicable Windows Vista and all subsequent iterations thereof blatantly lie about permissions pertaining to system directories. When non-Administrator users attempt to create files in either the canonical C:\Windows or C:\Windows\system32 directories, UAC superficially permits the user to do so while actually isolating all created files into a "Virtual Store" in that user's profile. (Who could have possibly imagined that deceiving users would have harmful long-term consequences?)
This is crazy. This is Windows.
Prove It
Dare we? It's time to test-drive the above tests.
Since NULL is the only character prohibited in pathnames on UNIX-oriented filesystems, let's leverage that to demonstrate the cold, hard truth – ignoring non-ignorable Windows shenanigans, which frankly bore and anger me in equal measure:
>>> print('"foo.bar" valid? ' + str(is_pathname_valid('foo.bar')))
"foo.bar" valid? True
>>> print('Null byte valid? ' + str(is_pathname_valid('\x00')))
Null byte valid? False
>>> print('Long path valid? ' + str(is_pathname_valid('a' * 256)))
Long path valid? False
>>> print('"/dev" exists or creatable? ' + str(is_path_exists_or_creatable('/dev')))
"/dev" exists or creatable? True
>>> print('"/dev/foo.bar" exists or creatable? ' + str(is_path_exists_or_creatable('/dev/foo.bar')))
"/dev/foo.bar" exists or creatable? False
>>> print('Null byte exists or creatable? ' + str(is_path_exists_or_creatable('\x00')))
Null byte exists or creatable? False
Beyond sanity. Beyond pain. You will find Python portability concerns.
if os.path.exists(filePath):
#the file is there
elif os.access(os.path.dirname(filePath), os.W_OK):
#the file does not exists but write privileges are given
else:
#can not write there
Note that path.exists can fail for more reasons than just the file is not there so you might have to do finer tests like testing if the containing directory exists and so on.
After my discussion with the OP it turned out, that the main problem seems to be, that the file name might contain characters that are not allowed by the filesystem. Of course they need to be removed but the OP wants to maintain as much human readablitiy as the filesystem allows.
Sadly I do not know of any good solution for this.
However Cecil Curry's answer takes a closer look at detecting the problem.
I found a PyPI module called pathvalidate.
pip install pathvalidate
Inside it has a function called sanitize_filepath which will take a file path and convert it into a valid file path:
from pathvalidate import sanitize_filepath
file1 = "ap:lle/fi:le"
print(sanitize_filepath(file1))
# Output: "apple/file"
It also works with reserved names too. If you feed it the file path con, it will return con_.
With this knowledge, we can check if the entered file path is equal to the sanitized one and that will mean the file path is valid.
import os
from pathvalidate import sanitize_filepath
def check(filePath):
if os.path.exists(filePath):
return True
if filePath == sanitize_filepath(filePath):
return True
return False
With Python 3, how about:
try:
with open(filename, 'x') as tempfile: # OSError if file exists or is invalid
pass
except OSError:
# handle error here
With the 'x' option we also don't have to worry about race conditions. See documentation here.
Now, this WILL create a very shortlived temporary file if it does not exist already - unless the name is invalid. If you can live with that, it simplifies things a lot.
open(filename,'r') #2nd argument is r and not w
will open the file or give an error if it doesn't exist. If there's an error, then you can try to write to the path, if you can't then you get a second error
try:
open(filename,'r')
return True
except IOError:
try:
open(filename, 'w')
return True
except IOError:
return False
Also have a look here about permissions on windows
try os.path.exists this will check for the path and return True if exists and False if not.
I'm writting an python program and now I'm working at exceptions.
while True:
try:
os.makedirs("{}\\test".format(dest))
except PermissionError:
print("Make sure that you have access to specified path")
print("Try again specify your path: ", end='')
dest = input()
continue
break
It is working but later I need to delete that folder.
What is the better way to do it?
Don't.
It is almost never worth verifying that you have permissions to perform an operation that your program requires. For one thing, permissions are not the only possible reason for failure. A delete may also fail because of a file lock by another program, for instance. Unless you have a very good reason to do otherwise, it is both more efficient and more reliable to just write your code to try the operation and then abort on failure:
import shutil
try:
shutil.rmtree(path_to_remove) # Recursively deletes directory and files inside it
except Exception as ex:
print('Failed to delete directory, manual clean up may be required: {}'.format(path_to_remove))
sys.exit(1)
Other concerns about your code
Use os.path.join to concatenate file paths: os.makedirs(os.path.join(dest, test)). This will use the appropriate directory separator for the operating system.
Why are you looping on failure? In real world programs, simply aborting the entire operation is simpler and usually makes for a better user experience.
Are you sure you aren't looking for the tempfile library? It allows you to spit out a unique directory to the operating system's standard temporary location:
import tempfile
with tempfile.TemporaryDirectory() as tmpdir:
some_function_that_creates_several_files(tmpdir)
for f in os.walk(tmpdir):
# do something with each file
# tmpdir automatically deleted when context manager exits
# Or if you really only need the file
with tempfile.TemporaryFile() as tmpfile:
tmpfile.write('my data')
some_function_that_needs_a_file(tmpfile)
# tmpfile automatically deleted when context manager exits
I think what you want is os.access.
os.access(path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True)
Use the real uid/gid to test for access to path. Note that most operations will use the effective uid/gid, therefore this routine can be used in a suid/sgid environment to test if the invoking user has the specified access to path. mode should be F_OK to test the existence of path, or it can be the inclusive OR of one or more of R_OK, W_OK, and X_OK to test permissions. Return True if access is allowed, False if not.
For example:
os.access("/path", os.R_OK)
And the mode contains:
os.F_OK # existence
os.R_OK # readability
os.W_OK # writability
os.X_OK # executability
Refer: https://docs.python.org/3.7/library/os.html#os.access
I'm building a relatively simple application that asks for directories, checks if they're correct, and then removes one of them and recreates it with contents of the other one. I'm encountering this weird behaviour, I'll try to explain:
When I've got the destination folder window open, AND it's empty, there's an access denied exception, then I get kicked out of the folder and it gets removed. But then if it's not empty, it works just fine, no exceptions, the destination directory (from what it seems) gets emptied then filled with files from the source directory. Which is strange because it's supposed to straight out remove the destination folder no matter what, and then recreate it with the same name and contents from the source destination.
This doesn't make sense to me, shouldn't there be the exact same exception when I'm browsing the directory when it's not empty as when it's empty? What's the difference, it's still supposed to just delete the folder. Is there any logical explanation to that? Also, if there's an exception, why does the directory get removed anyway?
Code for this particular part is pretty straightforward (please keep in mind I'm a beginner :) )
def Delete(self, dest):
try:
shutil.rmtree(dest)
self.Paste(self.src, dest)
except (IOError, os.error) as e:
print e
def Paste(self, src, dest):
try:
shutil.copytree(src, dest)
except (IOError, os.error) as e:
print e
This is expected behaviour on Windows.
Internally shutil.rmtree calls the windows API function DeleteFile which is documented on MSDN (http://msdn.microsoft.com/en-us/library/windows/desktop/aa363915%28v=vs.85%29.aspx).
This function has the following property (highlight by me):
The DeleteFile function marks a file for deletion on close. Therefore, the file deletion does not occur until the last handle to the file is closed. Subsequent calls to CreateFile to open the file fail with ERROR_ACCESS_DENIED.
If any other process still has a handle open (e.g. virus scanners, windows explorer because you watch the directory or anything else that might still have a handle to that directory), it will not go away.
Usually you just catch the exception in your Paste operation and retry it a few times/for some dozend milliseconds, to handle all those weird virus scanner anomalies.
Little bonus: You can use windbg or ProcessExplorer to find out who still keeps an open handle to your file (just use Find Handle in Process explorer and search for the filename).
How do I change a symlink to point from one file to another in Python?
The os.symlink function only seems to work to create new symlinks.
If you need an atomic modification, unlinking won't work.
A better solution would be to create a new temporary symlink, and then rename it over the existing one:
os.symlink(target, tmpLink)
os.rename(tmpLink, linkName)
You can check to make sure it was updated correctly too:
if os.path.realpath(linkName) == target:
# Symlink was updated
According to the documentation for os.rename though, there may be no way to atomically change a symlink in Windows. In that case, you would just delete and re-create.
A little function for Python2 which tries to symlink and if it fails because of an existing file, it removes it and links again.
import os, errno
def symlink_force(target, link_name):
try:
os.symlink(target, link_name)
except OSError, e:
if e.errno == errno.EEXIST:
os.remove(link_name)
os.symlink(target, link_name)
else:
raise e
For python3 except condition should be except OSError as e:
You could os.unlink() it first, and then re-create using os.symlink() to point to the new target.
Given overwrite=True, this function will safely overwrite an existing file with a symlink.
It is cognisant of race conditions, which is why it is not short, but it is safe.
import os, tempfile
def symlink(target, link_name, overwrite=False):
'''
Create a symbolic link named link_name pointing to target.
If link_name exists then FileExistsError is raised, unless overwrite=True.
When trying to overwrite a directory, IsADirectoryError is raised.
'''
if not overwrite:
os.symlink(target, link_name)
return
# os.replace() may fail if files are on different filesystems
link_dir = os.path.dirname(link_name)
# Create link to target with temporary filename
while True:
temp_link_name = tempfile.mktemp(dir=link_dir)
# os.* functions mimic as closely as possible system functions
# The POSIX symlink() returns EEXIST if link_name already exists
# https://pubs.opengroup.org/onlinepubs/9699919799/functions/symlink.html
try:
os.symlink(target, temp_link_name)
break
except FileExistsError:
pass
# Replace link_name with temp_link_name
try:
# Pre-empt os.replace on a directory with a nicer message
if not os.path.islink(link_name) and os.path.isdir(link_name):
raise IsADirectoryError(f"Cannot symlink over existing directory: '{link_name}'")
os.replace(temp_link_name, link_name)
except:
if os.path.islink(temp_link_name):
os.remove(temp_link_name)
raise
Notes for pedants:
If the function fails (e.g. computer crashes), an additional random link to the target might exist.
An unlikely race condition still remains: the symlink created at the randomly-named temp_link_name could be modified by another process before replacing link_name.
I raised a python issue to highlight the issues of os.symlink() requiring the target to not exist, where I was advised to raise my suggestion on the python-ideas mailing list
Credit to Robert Siemer’s input.
I researched this question recently, and found out that the best way is indeed to unlink and then symlink. But if you need just to fix broken links, for example with auto-replace, then you can do os.readlink:
for f in os.listdir(dir):
path = os.path.join(dir, f)
old_link = os.readlink(path)
new_link = old_link.replace(before, after)
os.unlink(path)
os.symlink(new_link, path)
Don't forget to add a raise command in the case when e.errno != errno.EEXIST You don't want to hide an error then:
if e.errno == errno.EEXIST:
os.remove(link_name)
os.symlink(target, link_name)
else:
raise
A quick and easy solution:
while True:
try:
os.symlink(target, link_name)
break
except FileExistsError:
os.remove(link_name)
However this has a race condition when replacing a symlink which should
always exist, eg:
/lib/critical.so -> /lib/critical.so.1.2
When upgrading by:
my_symlink('/lib/critical.so.2.0', '/lib/critical.so')
There is a point in time when /lib/critical.so doesn't exist.
This answer avoids the race condition.
I am writing a file using Python, and I want it to be placed in a specific path. How can I safely make sure that the path exists?
That is: how can I check whether the folder exists, along with its parents? If there are missing folders along the path, how can I create them?
On Python ≥ 3.5, use pathlib.Path.mkdir:
from pathlib import Path
Path("/my/directory").mkdir(parents=True, exist_ok=True)
For older versions of Python, I see two answers with good qualities, each with a small flaw, so I will give my take on it:
Try os.path.exists, and consider os.makedirs for the creation.
import os
if not os.path.exists(directory):
os.makedirs(directory)
As noted in comments and elsewhere, there's a race condition – if the directory is created between the os.path.exists and the os.makedirs calls, the os.makedirs will fail with an OSError. Unfortunately, blanket-catching OSError and continuing is not foolproof, as it will ignore a failure to create the directory due to other factors, such as insufficient permissions, full disk, etc.
One option would be to trap the OSError and examine the embedded error code (see Is there a cross-platform way of getting information from Python’s OSError):
import os, errno
try:
os.makedirs(directory)
except OSError as e:
if e.errno != errno.EEXIST:
raise
Alternatively, there could be a second os.path.exists, but suppose another created the directory after the first check, then removed it before the second one – we could still be fooled.
Depending on the application, the danger of concurrent operations may be more or less than the danger posed by other factors such as file permissions. The developer would have to know more about the particular application being developed and its expected environment before choosing an implementation.
Modern versions of Python improve this code quite a bit, both by exposing FileExistsError (in 3.3+)...
try:
os.makedirs("path/to/directory")
except FileExistsError:
# directory already exists
pass
...and by allowing a keyword argument to os.makedirs called exist_ok (in 3.2+).
os.makedirs("path/to/directory", exist_ok=True) # succeeds even if directory exists.
Python 3.5+:
import pathlib
pathlib.Path('/my/directory').mkdir(parents=True, exist_ok=True)
pathlib.Path.mkdir as used above recursively creates the directory and does not raise an exception if the directory already exists. If you don't need or want the parents to be created, skip the parents argument.
Python 3.2+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
If using Python 3.4, even though it comes with pathlib, it is missing the useful exist_ok option. The backport is intended to offer a newer and superior implementation of mkdir which includes this missing option.
Using os:
import os
os.makedirs(path, exist_ok=True)
os.makedirs as used above recursively creates the directory and does not raise an exception if the directory already exists. It has the optional exist_ok argument only if using Python 3.2+, with a default value of False. This argument does not exist in Python 2.x up to 2.7. As such, there is no need for manual exception handling as with Python 2.7.
Python 2.7+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
Using os:
import os
try:
os.makedirs(path)
except OSError:
if not os.path.isdir(path):
raise
While a naive solution may first use os.path.isdir followed by os.makedirs, the solution above reverses the order of the two operations. In doing so, it prevents a common race condition having to do with a duplicated attempt at creating the directory, and also disambiguates files from directories.
Note that capturing the exception and using errno is of limited usefulness because OSError: [Errno 17] File exists, i.e. errno.EEXIST, is raised for both files and directories. It is more reliable simply to check if the directory exists.
Alternative:
mkpath creates the nested directory, and does nothing if the directory already exists. This works in both Python 2 and 3. Note however that distutils has been deprecated, and is scheduled for removal in Python 3.12.
import distutils.dir_util
distutils.dir_util.mkpath(path)
Per Bug 10948, a severe limitation of this alternative is that it works only once per python process for a given path. In other words, if you use it to create a directory, then delete the directory from inside or outside Python, then use mkpath again to recreate the same directory, mkpath will simply silently use its invalid cached info of having previously created the directory, and will not actually make the directory again. In contrast, os.makedirs doesn't rely on any such cache. This limitation may be okay for some applications.
With regard to the directory's mode, please refer to the documentation if you care about it.
Using try except and the right error code from errno module gets rid of the race condition and is cross-platform:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
In other words, we try to create the directories, but if they already exist we ignore the error. On the other hand, any other error gets reported. For example, if you create dir 'a' beforehand and remove all permissions from it, you will get an OSError raised with errno.EACCES (Permission denied, error 13).
Starting from Python 3.5, pathlib.Path.mkdir has an exist_ok flag:
from pathlib import Path
path = Path('/my/directory/filename.txt')
path.parent.mkdir(parents=True, exist_ok=True)
# path.parent ~ os.path.dirname(path)
This recursively creates the directory and does not raise an exception if the directory already exists.
(just as os.makedirs got an exist_ok flag starting from python 3.2 e.g os.makedirs(path, exist_ok=True))
Note: when i posted this answer none of the other answers mentioned exist_ok...
I would personally recommend that you use os.path.isdir() to test instead of os.path.exists().
>>> os.path.exists('/tmp/dirname')
True
>>> os.path.exists('/tmp/dirname/filename.etc')
True
>>> os.path.isdir('/tmp/dirname/filename.etc')
False
>>> os.path.isdir('/tmp/fakedirname')
False
If you have:
>>> directory = raw_input(":: ")
And a foolish user input:
:: /tmp/dirname/filename.etc
... You're going to end up with a directory named filename.etc when you pass that argument to os.makedirs() if you test with os.path.exists().
Check os.makedirs: (It makes sure the complete path exists.)
To handle the fact the directory might exist, catch OSError.
(If exist_ok is False (the default), an OSError is raised if the target directory already exists.)
import os
try:
os.makedirs('./path/to/somewhere')
except OSError:
pass
Try the os.path.exists function
if not os.path.exists(dir):
os.mkdir(dir)
Insights on the specifics of this situation
You give a particular file at a certain path and you pull the directory from the file path. Then after making sure you have the directory, you attempt to open a file for reading. To comment on this code:
filename = "/my/directory/filename.txt"
dir = os.path.dirname(filename)
We want to avoid overwriting the builtin function, dir. Also, filepath or perhaps fullfilepath is probably a better semantic name than filename so this would be better written:
import os
filepath = '/my/directory/filename.txt'
directory = os.path.dirname(filepath)
Your end goal is to open this file, you initially state, for writing, but you're essentially approaching this goal (based on your code) like this, which opens the file for reading:
if not os.path.exists(directory):
os.makedirs(directory)
f = file(filename)
Assuming opening for reading
Why would you make a directory for a file that you expect to be there and be able to read?
Just attempt to open the file.
with open(filepath) as my_file:
do_stuff(my_file)
If the directory or file isn't there, you'll get an IOError with an associated error number: errno.ENOENT will point to the correct error number regardless of your platform. You can catch it if you want, for example:
import errno
try:
with open(filepath) as my_file:
do_stuff(my_file)
except IOError as error:
if error.errno == errno.ENOENT:
print 'ignoring error because directory or file is not there'
else:
raise
Assuming we're opening for writing
This is probably what you're wanting.
In this case, we probably aren't facing any race conditions. So just do as you were, but note that for writing, you need to open with the w mode (or a to append). It's also a Python best practice to use the context manager for opening files.
import os
if not os.path.exists(directory):
os.makedirs(directory)
with open(filepath, 'w') as my_file:
do_stuff(my_file)
However, say we have several Python processes that attempt to put all their data into the same directory. Then we may have contention over creation of the directory. In that case it's best to wrap the makedirs call in a try-except block.
import os
import errno
if not os.path.exists(directory):
try:
os.makedirs(directory)
except OSError as error:
if error.errno != errno.EEXIST:
raise
with open(filepath, 'w') as my_file:
do_stuff(my_file)
I have put the following down. It's not totally foolproof though.
import os
dirname = 'create/me'
try:
os.makedirs(dirname)
except OSError:
if os.path.exists(dirname):
# We are nearly safe
pass
else:
# There was an error on creation, so make sure we know about it
raise
Now as I say, this is not really foolproof, because we have the possiblity of failing to create the directory, and another process creating it during that period.
Check if a directory exists and create it if necessary?
The direct answer to this is, assuming a simple situation where you don't expect other users or processes to be messing with your directory:
if not os.path.exists(d):
os.makedirs(d)
or if making the directory is subject to race conditions (i.e. if after checking the path exists, something else may have already made it) do this:
import errno
try:
os.makedirs(d)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
But perhaps an even better approach is to sidestep the resource contention issue, by using temporary directories via tempfile:
import tempfile
d = tempfile.mkdtemp()
Here's the essentials from the online doc:
mkdtemp(suffix='', prefix='tmp', dir=None)
User-callable function to create and return a unique temporary
directory. The return value is the pathname of the directory.
The directory is readable, writable, and searchable only by the
creating user.
Caller is responsible for deleting the directory when done with it.
New in Python 3.5: pathlib.Path with exist_ok
There's a new Path object (as of 3.4) with lots of methods one would want to use with paths - one of which is mkdir.
(For context, I'm tracking my weekly rep with a script. Here's the relevant parts of code from the script that allow me to avoid hitting Stack Overflow more than once a day for the same data.)
First the relevant imports:
from pathlib import Path
import tempfile
We don't have to deal with os.path.join now - just join path parts with a /:
directory = Path(tempfile.gettempdir()) / 'sodata'
Then I idempotently ensure the directory exists - the exist_ok argument shows up in Python 3.5:
directory.mkdir(exist_ok=True)
Here's the relevant part of the documentation:
If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file.
Here's a little more of the script - in my case, I'm not subject to a race condition, I only have one process that expects the directory (or contained files) to be there, and I don't have anything trying to remove the directory.
todays_file = directory / str(datetime.datetime.utcnow().date())
if todays_file.exists():
logger.info("todays_file exists: " + str(todays_file))
df = pd.read_json(str(todays_file))
Path objects have to be coerced to str before other APIs that expect str paths can use them.
Perhaps Pandas should be updated to accept instances of the abstract base class, os.PathLike.
fastest safest way to do it is:
it will create if not exists and skip if exists:
from pathlib import Path
Path("path/with/childs/.../").mkdir(parents=True, exist_ok=True)
Best way to do this in python
#Devil
import os
directory = "./out_dir/subdir1/subdir2"
if not os.path.exists(directory):
os.makedirs(directory)
In Python 3.4 you can also use the brand new pathlib module:
from pathlib import Path
path = Path("/my/directory/filename.txt")
try:
if not path.parent.exists():
path.parent.mkdir(parents=True)
except OSError:
# handle error; you can also catch specific errors like
# FileExistsError and so on.
For a one-liner solution, you can use IPython.utils.path.ensure_dir_exists():
from IPython.utils.path import ensure_dir_exists
ensure_dir_exists(dir)
From the documentation: Ensure that a directory exists. If it doesn’t exist, try to create it and protect against a race condition if another process is doing the same.
IPython is an extension package, not part of the standard library.
In Python3, os.makedirs supports setting exist_ok. The default setting is False, which means an OSError will be raised if the target directory already exists. By setting exist_ok to True, OSError (directory exists) will be ignored and the directory will not be created.
os.makedirs(path,exist_ok=True)
In Python2, os.makedirs doesn't support setting exist_ok. You can use the approach in heikki-toivonen's answer:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
The relevant Python documentation suggests the use of the EAFP coding style (Easier to Ask for Forgiveness than Permission). This means that the code
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
else:
print "\nBE CAREFUL! Directory %s already exists." % path
is better than the alternative
if not os.path.exists(path):
os.makedirs(path)
else:
print "\nBE CAREFUL! Directory %s already exists." % path
The documentation suggests this exactly because of the race condition discussed in this question. In addition, as others mention here, there is a performance advantage in querying once instead of twice the OS. Finally, the argument placed forward, potentially, in favour of the second code in some cases --when the developer knows the environment the application is running-- can only be advocated in the special case that the program has set up a private environment for itself (and other instances of the same program).
Even in that case, this is a bad practice and can lead to long useless debugging. For example, the fact we set the permissions for a directory should not leave us with the impression permissions are set appropriately for our purposes. A parent directory could be mounted with other permissions. In general, a program should always work correctly and the programmer should not expect one specific environment.
I found this Q/A after I was puzzled by some of the failures and errors I was getting while working with directories in Python. I am working in Python 3 (v.3.5 in an Anaconda virtual environment on an Arch Linux x86_64 system).
Consider this directory structure:
└── output/ ## dir
├── corpus ## file
├── corpus2/ ## dir
└── subdir/ ## dir
Here are my experiments/notes, which provides clarification:
# ----------------------------------------------------------------------------
# [1] https://stackoverflow.com/questions/273192/how-can-i-create-a-directory-if-it-does-not-exist
import pathlib
""" Notes:
1. Include a trailing slash at the end of the directory path
("Method 1," below).
2. If a subdirectory in your intended path matches an existing file
with same name, you will get the following error:
"NotADirectoryError: [Errno 20] Not a directory:" ...
"""
# Uncomment and try each of these "out_dir" paths, singly:
# ----------------------------------------------------------------------------
# METHOD 1:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but no file created (os.makedirs creates dir, not files! ;-)
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# [2] https://docs.python.org/3/library/os.html#os.makedirs
# Uncomment these to run "Method 1":
#directory = os.path.dirname(out_dir)
#os.makedirs(directory, mode=0o777, exist_ok=True)
# ----------------------------------------------------------------------------
# METHOD 2:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## works
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## works
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but creates a .../doc.txt./ dir
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# Uncomment these to run "Method 2":
#import os, errno
#try:
# os.makedirs(out_dir)
#except OSError as e:
# if e.errno != errno.EEXIST:
# raise
# ----------------------------------------------------------------------------
Conclusion: in my opinion, "Method 2" is more robust.
[1] How can I safely create a nested directory?
[2] https://docs.python.org/3/library/os.html#os.makedirs
You can use mkpath
# Create a directory and any missing ancestor directories.
# If the directory already exists, do nothing.
from distutils.dir_util import mkpath
mkpath("test")
Note that it will create the ancestor directories as well.
It works for Python 2 and 3.
In case you're writing a file to a variable path, you can use this on the file's path to make sure that the parent directories are created.
from pathlib import Path
path_to_file = Path("zero/or/more/directories/file.ext")
parent_directory_of_file = path_to_file.parent
parent_directory_of_file.mkdir(parents=True, exist_ok=True)
Works even if path_to_file is file.ext (zero directories deep).
See pathlib.PurePath.parent and pathlib.Path.mkdir.
Why not use subprocess module if running on a machine that supports command
mkdir with -p option ?
Works on python 2.7 and python 3.6
from subprocess import call
call(['mkdir', '-p', 'path1/path2/path3'])
Should do the trick on most systems.
In situations where portability doesn't matter (ex, using docker) the solution is a clean 2 lines. You also don't have to add logic to check if directories exist or not. Finally, it is safe to re-run without any side effects
If you need error handling:
from subprocess import check_call
try:
check_call(['mkdir', '-p', 'path1/path2/path3'])
except:
handle...
You have to set the full path before creating the directory:
import os,sys,inspect
import pathlib
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
your_folder = currentdir + "/" + "your_folder"
if not os.path.exists(your_folder):
pathlib.Path(your_folder).mkdir(parents=True, exist_ok=True)
This works for me and hopefully, it will works for you as well
I saw Heikki Toivonen and A-B-B's answers and thought of this variation.
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST or not os.path.isdir(path):
raise
I use os.path.exists(), here is a Python 3 script that can be used to check if a directory exists, create one if it does not exist, and delete it if it does exist (if desired).
It prompts users for input of the directory and can be easily modified.
Use this command check and create dir
if not os.path.isdir(test_img_dir):
os.mkdir(test_img_dir)
Call the function create_dir() at the entry point of your program/project.
import os
def create_dir(directory):
if not os.path.exists(directory):
print('Creating Directory '+directory)
os.makedirs(directory)
create_dir('Project directory')
If you consider the following:
os.path.isdir('/tmp/dirname')
means a directory (path) exists AND is a directory. So for me this way does what I need. So I can make sure it is folder (not a file) and exists.
You can use os.listdir for this:
import os
if 'dirName' in os.listdir('parentFolderPath')
print('Directory Exists')
This may not exactly answer the question. But I guess your real intention is to create a file and its parent directories, given its content all in 1 command.
You can do that with fastcore extension to pathlib: path.mk_write(data)
from fastcore.utils import Path
Path('/dir/to/file.txt').mk_write('Hello World')
See more in fastcore documentation