We Keep Coding

Python if, else and pass through list of files [duplicate]

How do I check whether a file exists or not, without using the try statement?

If the reason you're checking is so you can do something like if file_exists: open_it(), it's safer to use a try around the attempt to open it. Checking and then opening risks the file being deleted or moved or something between when you check and when you try to open it.
If you're not planning to open the file immediately, you can use os.path.isfile
Return True if path is an existing regular file. This follows symbolic links, so both islink() and isfile() can be true for the same path.
import os.path
os.path.isfile(fname)
if you need to be sure it's a file.
Starting with Python 3.4, the pathlib module offers an object-oriented approach (backported to pathlib2 in Python 2.7):
from pathlib import Path
my_file = Path("/path/to/file")
if my_file.is_file():
# file exists
To check a directory, do:
if my_file.is_dir():
# directory exists
To check whether a Path object exists independently of whether is it a file or directory, use exists():
if my_file.exists():
# path exists
You can also use resolve(strict=True) in a try block:
try:
my_abs_path = my_file.resolve(strict=True)
except FileNotFoundError:
# doesn't exist
else:
# exists

Use os.path.exists to check both files and directories:
import os.path
os.path.exists(file_path)
Use os.path.isfile to check only files (note: follows symbolic links):
os.path.isfile(file_path)

Unlike isfile(), exists() will return True for directories. So depending on if you want only plain files or also directories, you'll use isfile() or exists(). Here is some simple REPL output:
>>> os.path.isfile("/etc/password.txt")
True
>>> os.path.isfile("/etc")
False
>>> os.path.isfile("/does/not/exist")
False
>>> os.path.exists("/etc/password.txt")
True
>>> os.path.exists("/etc")
True
>>> os.path.exists("/does/not/exist")
False

import os
if os.path.isfile(filepath):
print("File exists")

Use os.path.isfile() with os.access():
import os
PATH = './file.txt'
if os.path.isfile(PATH) and os.access(PATH, os.R_OK):
print("File exists and is readable")
else:
print("Either the file is missing or not readable")

import os
os.path.exists(path) # Returns whether the path (directory or file) exists or not
os.path.isfile(path) # Returns whether the file exists or not

Although almost every possible way has been listed in (at least one of) the existing answers (e.g. Python 3.4 specific stuff was added), I'll try to group everything together.
Note: every piece of Python standard library code that I'm going to post, belongs to version 3.5.3.
Problem statement:
Check file (arguable: also folder ("special" file) ?) existence
Don't use try / except / else / finally blocks
Possible solutions:
1. [Python.Docs]: os.path.exists(path)
Also check other function family members like os.path.isfile, os.path.isdir, os.path.lexists for slightly different behaviors:
Return True if path refers to an existing path or an open file descriptor. Returns False for broken symbolic links. On some platforms, this function may return False if permission is not granted to execute os.stat() on the requested file, even if the path physically exists.
All good, but if following the import tree:
os.path - posixpath.py (ntpath.py)
genericpath.py - line ~20+
def exists(path):
"""Test whether a path exists. Returns False for broken symbolic links"""
try:
st = os.stat(path)
except os.error:
return False
return True
it's just a try / except block around [Python.Docs]: os.stat(path, *, dir_fd=None, follow_symlinks=True). So, your code is try / except free, but lower in the framestack there's (at least) one such block. This also applies to other functions (including os.path.isfile).
1.1. [Python.Docs]: pathlib - Path.is_file()
It's a fancier (and more [Wiktionary]: Pythonic) way of handling paths, but
Under the hood, it does exactly the same thing (pathlib.py - line ~1330):
def is_file(self):
"""
Whether this path is a regular file (also True for symlinks pointing
to regular files).
"""
try:
return S_ISREG(self.stat().st_mode)
except OSError as e:
if e.errno not in (ENOENT, ENOTDIR):
raise
# Path doesn't exist or is a broken symlink
# (see https://bitbucket.org/pitrou/pathlib/issue/12/)
return False
2. [Python.Docs]: With Statement Context Managers
Either:
Create one:
class Swallow: # Dummy example
swallowed_exceptions = (FileNotFoundError,)
def __enter__(self):
print("Entering...")
def __exit__(self, exc_type, exc_value, exc_traceback):
print("Exiting:", exc_type, exc_value, exc_traceback)
# Only swallow FileNotFoundError (not e.g. TypeError - if the user passes a wrong argument like None or float or ...)
return exc_type in Swallow.swallowed_exceptions
And its usage - I'll replicate the os.path.isfile behavior (note that this is just for demonstrating purposes, do not attempt to write such code for production):
import os
import stat
def isfile_seaman(path): # Dummy func
result = False
with Swallow():
result = stat.S_ISREG(os.stat(path).st_mode)
return result
Use [Python.Docs]: contextlib.suppress(*exceptions) - which was specifically designed for selectively suppressing exceptions
But, they seem to be wrappers over try / except / else / finally blocks, as [Python.Docs]: Compound statements - The with statement states:
This allows common try...except...finally usage patterns to be encapsulated for convenient reuse.
3. Filesystem traversal functions
Search the results for matching item(s):
[Python.Docs]: os.listdir(path='.') (or [Python.Docs]: os.scandir(path='.') on Python v3.5+, backport: [PyPI]: scandir)
Under the hood, both use:
Nix: [Man7]: OPENDIR(3) / [Man7]: READDIR(3) / [Man7]: CLOSEDIR(3)
Win: [MS.Learn]: FindFirstFileW function (fileapi.h) / [MS.Learn]: FindNextFileW function (fileapi.h) / [MS.Learn]: FindClose function (fileapi.h)
via [GitHub]: python/cpython - (main) cpython/Modules/posixmodule.c
Using scandir() instead of listdir() can significantly increase the performance of code that also needs file type or file attribute information, because os.DirEntry objects expose this information if the operating system provides it when scanning a directory. All os.DirEntry methods may perform a system call, but is_dir() and is_file() usually only require a system call for symbolic links; os.DirEntry.stat() always requires a system call on Unix, but only requires one for symbolic links on Windows.
[Python.Docs]: os.walk(top, topdown=True, onerror=None, followlinks=False)
Uses os.listdir (os.scandir when available)
[Python.Docs]: glob.iglob(pathname, *, root_dir=None, dir_fd=None, recursive=False, include_hidden=False) (or its predecessor: glob.glob)
Doesn't seem a traversing function per se (at least in some cases), but it still uses os.listdir
Since these iterate over folders, (in most of the cases) they are inefficient for our problem (there are exceptions, like non wildcarded globbing - as #ShadowRanger pointed out), so I'm not going to insist on them. Not to mention that in some cases, filename processing might be required.
4. [Python.Docs]: os.access(path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True)
Its behavior is close to os.path.exists (actually it's wider, mainly because of the 2nd argument).
User permissions might restrict the file "visibility" as the doc states:
... test if the invoking user has the specified access to path. mode should be F_OK to test the existence of path...
Security considerations:
Using access() to check if a user is authorized to e.g. open a file before actually doing so using open() creates a security hole, because the user might exploit the short time interval between checking and opening the file to manipulate it.
os.access("/tmp", os.F_OK)
Since I also work in C, I use this method as well because under the hood, it calls native APIs (again, via "${PYTHON_SRC_DIR}/Modules/posixmodule.c"), but it also opens a gate for possible user errors, and it's not as Pythonic as other variants. So, don't use it unless you know what you're doing:
Nix: [Man7]: ACCESS(2)
Warning: Using these calls to check if a user is authorized to, for example, open a file before actually doing so using open(2) creates a security hole, because the user might exploit the short time interval between checking and opening the file to manipulate it. For this reason, the use of this system call should be avoided.
Win: [MS.Learn]: GetFileAttributesW function (fileapi.h)
As seen, this approach is highly discouraged (especially on Nix).
Note: calling native APIs is also possible via [Python.Docs]: ctypes - A foreign function library for Python, but in most cases it's more complicated. Before working with CTypes, check [SO]: C function called from Python via ctypes returns incorrect value (#CristiFati's answer) out.
(Win specific): since vcruntime###.dll (msvcr###.dll for older VStudio versions - I'm going to refer to it as UCRT) exports a [MS.Learn]: _access, _waccess function family as well, here's an example (note that the recommended [Python.Docs]: msvcrt - Useful routines from the MS VC++ runtime doesn't export them):
Python 3.5.3 (v3.5.3:1880cb95a742, Jan 16 2017, 16:02:32) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes as cts, os
>>> cts.CDLL("msvcrt")._waccess(u"C:\\Windows\\Temp", os.F_OK)
0
>>> cts.CDLL("msvcrt")._waccess(u"C:\\Windows\\Temp.notexist", os.F_OK)
-1
Notes:
Although it's not a good practice, I'm using os.F_OK in the call, but that's just for clarity (its value is 0)
I'm using _waccess so that the same code works on Python 3 and Python 2 (in spite of [Wikipedia]: Unicode related differences between them - [SO]: Passing utf-16 string to a Windows function (#CristiFati's answer))
Although this targets a very specific area, it was not mentioned in any of the previous answers
The Linux (Ubuntu ([Wikipedia]: Ubuntu version history) 16 x86_64 (pc064)) counterpart as well:
Python 3.5.2 (default, Nov 17 2016, 17:05:23)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes as cts, os
>>> cts.CDLL("/lib/x86_64-linux-gnu/libc.so.6").access(b"/tmp", os.F_OK)
0
>>> cts.CDLL("/lib/x86_64-linux-gnu/libc.so.6").access(b"/tmp.notexist", os.F_OK)
-1
Notes:
Instead hardcoding libc.so (LibC)'s path ("/lib/x86_64-linux-gnu/libc.so.6") which may (and most likely, will) vary across systems, None (or the empty string) can be passed to CDLL constructor (ctypes.CDLL(None).access(b"/tmp", os.F_OK)). According to [Man7]: DLOPEN(3):
If filename is NULL, then the returned handle is for the main
program. When given to dlsym(3), this handle causes a search for a
symbol in the main program, followed by all shared objects loaded at
program startup, and then all shared objects loaded by dlopen() with
the flag RTLD_GLOBAL.
Main (current) program (python) is linked against LibC, so its symbols (including access) will be loaded
This has to be handled with care, since functions like main, Py_Main and (all the) others are available; calling them could have disastrous effects (on the current program)
This doesn't also apply to Windows (but that's not such a big deal, since UCRT is located in "%SystemRoot%\System32" which is in %PATH% by default). I wanted to take things further and replicate this behavior on Windows (and submit a patch), but as it turns out, [MS.Learn]: GetProcAddress function (libloaderapi.h) only "sees" exported symbols, so unless someone declares the functions in the main executable as __declspec(dllexport) (why on Earth the common person would do that?), the main program is loadable, but it is pretty much unusable
5. 3rd-party modules with filesystem capabilities
Most likely, will rely on one of the ways above (maybe with slight customizations). One example would be (again, Win specific) [GitHub]: mhammond/pywin32 - Python for Windows (pywin32) Extensions, which is a Python wrapper over WinAPIs.
But, since this is more like a workaround, I'm stopping here.
6. SysAdmin approach
I consider this a (lame) workaround (gainarie): use Python as a wrapper to execute shell commands:
Win:
(py35x64_test) [cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q000082831]> "e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" -c "import os; print(os.system('dir /b \"C:\\Windows\\Temp\" > nul 2>&1'))"
0
(py35x64_test) [cfati#CFATI-5510-0:e:\Work\Dev\StackOverflow\q000082831]> "e:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" -c "import os; print(os.system('dir /b \"C:\\Windows\\Temp.notexist\" > nul 2>&1'))"
1
Nix ([Wikipedia]: Unix-like) - Ubuntu:
[cfati#cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q000082831]> python3 -c "import os; print(os.system('ls \"/tmp\" > /dev/null 2>&1'))"
0
[cfati#cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q000082831]> python3 -c "import os; print(os.system('ls \"/tmp.notexist\" > /dev/null 2>&1'))"
512
Bottom line:
Do use try / except / else / finally blocks, because they can prevent you running into a series of nasty problems
A possible counterexample that I can think of, is performance: such blocks are costly, so try not to place them in code that it's supposed to run hundreds of thousands times per second (but since (in most cases) it involves disk access, it won't be the case)

Python 3.4+ has an object-oriented path module: pathlib. Using this new module, you can check whether a file exists like this:
import pathlib
p = pathlib.Path('path/to/file')
if p.is_file(): # or p.is_dir() to see if it is a directory
# do stuff
You can (and usually should) still use a try/except block when opening files:
try:
with p.open() as f:
# do awesome stuff
except OSError:
print('Well darn.')
The pathlib module has lots of cool stuff in it: convenient globbing, checking file's owner, easier path joining, etc. It's worth checking out. If you're on an older Python (version 2.6 or later), you can still install pathlib with pip:
# installs pathlib2 on older Python versions
# the original third-party module, pathlib, is no longer maintained.
pip install pathlib2
Then import it as follows:
# Older Python versions
import pathlib2 as pathlib

This is the simplest way to check if a file exists. Just because the file existed when you checked doesn't guarantee that it will be there when you need to open it.
import os
fname = "foo.txt"
if os.path.isfile(fname):
print("file does exist at this time")
else:
print("no such file exists at this time")

How do I check whether a file exists, using Python, without using a try statement?
Now available since Python 3.4, import and instantiate a Path object with the file name, and check the is_file method (note that this returns True for symlinks pointing to regular files as well):
>>> from pathlib import Path
>>> Path('/').is_file()
False
>>> Path('/initrd.img').is_file()
True
>>> Path('/doesnotexist').is_file()
False
If you're on Python 2, you can backport the pathlib module from pypi, pathlib2, or otherwise check isfile from the os.path module:
>>> import os
>>> os.path.isfile('/')
False
>>> os.path.isfile('/initrd.img')
True
>>> os.path.isfile('/doesnotexist')
False
Now the above is probably the best pragmatic direct answer here, but there's the possibility of a race condition (depending on what you're trying to accomplish), and the fact that the underlying implementation uses a try, but Python uses try everywhere in its implementation.
Because Python uses try everywhere, there's really no reason to avoid an implementation that uses it.
But the rest of this answer attempts to consider these caveats.
Longer, much more pedantic answer
Available since Python 3.4, use the new Path object in pathlib. Note that .exists is not quite right, because directories are not files (except in the unix sense that everything is a file).
>>> from pathlib import Path
>>> root = Path('/')
>>> root.exists()
True
So we need to use is_file:
>>> root.is_file()
False
Here's the help on is_file:
is_file(self)
Whether this path is a regular file (also True for symlinks pointing
to regular files).
So let's get a file that we know is a file:
>>> import tempfile
>>> file = tempfile.NamedTemporaryFile()
>>> filepathobj = Path(file.name)
>>> filepathobj.is_file()
True
>>> filepathobj.exists()
True
By default, NamedTemporaryFile deletes the file when closed (and will automatically close when no more references exist to it).
>>> del file
>>> filepathobj.exists()
False
>>> filepathobj.is_file()
False
If you dig into the implementation, though, you'll see that is_file uses try:
def is_file(self):
"""
Whether this path is a regular file (also True for symlinks pointing
to regular files).
"""
try:
return S_ISREG(self.stat().st_mode)
except OSError as e:
if e.errno not in (ENOENT, ENOTDIR):
raise
# Path doesn't exist or is a broken symlink
# (see https://bitbucket.org/pitrou/pathlib/issue/12/)
return False
Race Conditions: Why we like try
We like try because it avoids race conditions. With try, you simply attempt to read your file, expecting it to be there, and if not, you catch the exception and perform whatever fallback behavior makes sense.
If you want to check that a file exists before you attempt to read it, and you might be deleting it and then you might be using multiple threads or processes, or another program knows about that file and could delete it - you risk the chance of a race condition if you check it exists, because you are then racing to open it before its condition (its existence) changes.
Race conditions are very hard to debug because there's a very small window in which they can cause your program to fail.
But if this is your motivation, you can get the value of a try statement by using the suppress context manager.
Avoiding race conditions without a try statement: suppress
Python 3.4 gives us the suppress context manager (previously the ignore context manager), which does semantically exactly the same thing in fewer lines, while also (at least superficially) meeting the original ask to avoid a try statement:
from contextlib import suppress
from pathlib import Path
Usage:
>>> with suppress(OSError), Path('doesnotexist').open() as f:
... for line in f:
... print(line)
...
>>>
>>> with suppress(OSError):
... Path('doesnotexist').unlink()
...
>>>
For earlier Pythons, you could roll your own suppress, but without a try will be more verbose than with. I do believe this actually is the only answer that doesn't use try at any level in the Python that can be applied to prior to Python 3.4 because it uses a context manager instead:
class suppress(object):
def __init__(self, *exceptions):
self.exceptions = exceptions
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
if exc_type is not None:
return issubclass(exc_type, self.exceptions)
Perhaps easier with a try:
from contextlib import contextmanager
#contextmanager
def suppress(*exceptions):
try:
yield
except exceptions:
pass
Other options that don't meet the ask for "without try":
isfile
import os
os.path.isfile(path)
from the docs:
os.path.isfile(path)
Return True if path is an existing regular file. This follows symbolic
links, so both islink() and isfile() can be true for the same path.
But if you examine the source of this function, you'll see it actually does use a try statement:
# This follows symbolic links, so both islink() and isdir() can be true
# for the same path on systems that support symlinks
def isfile(path):
"""Test whether a path is a regular file"""
try:
st = os.stat(path)
except os.error:
return False
return stat.S_ISREG(st.st_mode)
>>> OSError is os.error
True
All it's doing is using the given path to see if it can get stats on it, catching OSError and then checking if it's a file if it didn't raise the exception.
If you intend to do something with the file, I would suggest directly attempting it with a try-except to avoid a race condition:
try:
with open(path) as f:
f.read()
except OSError:
pass
os.access
Available for Unix and Windows is os.access, but to use you must pass flags, and it does not differentiate between files and directories. This is more used to test if the real invoking user has access in an elevated privilege environment:
import os
os.access(path, os.F_OK)
It also suffers from the same race condition problems as isfile. From the docs:
Note:
Using access() to check if a user is authorized to e.g. open a file
before actually doing so using open() creates a security hole, because
the user might exploit the short time interval between checking and
opening the file to manipulate it. It’s preferable to use EAFP
techniques. For example:
if os.access("myfile", os.R_OK):
with open("myfile") as fp:
return fp.read()
return "some default data"
is better written as:
try:
fp = open("myfile")
except IOError as e:
if e.errno == errno.EACCES:
return "some default data"
# Not a permission error.
raise
else:
with fp:
return fp.read()
Avoid using os.access. It is a low level function that has more opportunities for user error than the higher level objects and functions discussed above.
Criticism of another answer:
Another answer says this about os.access:
Personally, I prefer this one because under the hood, it calls native APIs (via "${PYTHON_SRC_DIR}/Modules/posixmodule.c"), but it also opens a gate for possible user errors, and it's not as Pythonic as other variants:
This answer says it prefers a non-Pythonic, error-prone method, with no justification. It seems to encourage users to use low-level APIs without understanding them.
It also creates a context manager which, by unconditionally returning True, allows all Exceptions (including KeyboardInterrupt and SystemExit!) to pass silently, which is a good way to hide bugs.
This seems to encourage users to adopt poor practices.

Prefer the try statement. It's considered better style and avoids race conditions.
Don't take my word for it. There's plenty of support for this theory. Here's a couple:
Style: Section "Handling unusual conditions" of these course notes for Software Design (2007)
Avoiding Race Conditions

Use:
import os
#Your path here e.g. "C:\Program Files\text.txt"
#For access purposes: "C:\\Program Files\\text.txt"
if os.path.exists("C:\..."):
print "File found!"
else:
print "File not found!"
Importing os makes it easier to navigate and perform standard actions with your operating system.
For reference, also see How do I check whether a file exists without exceptions?.
If you need high-level operations, use shutil.

Testing for files and folders with os.path.isfile(), os.path.isdir() and os.path.exists()
Assuming that the "path" is a valid path, this table shows what is returned by each function for files and folders:
You can also test if a file is a certain type of file using os.path.splitext() to get the extension (if you don't already know it)
>>> import os
>>> path = "path to a word document"
>>> os.path.isfile(path)
True
>>> os.path.splitext(path)[1] == ".docx" # test if the extension is .docx
True

TL;DR
The answer is: use the pathlib module
Pathlib is probably the most modern and convenient way for almost all of the file operations. For the existence of a file or a folder a single line of code is enough. If file is not exists, it will not throw any exception.
from pathlib import Path
if Path("myfile.txt").exists(): # works for both file and folders
# do your cool stuff...
The pathlib module was introduced in Python 3.4, so you need to have Python 3.4+. This library makes your life much easier while working with files and folders, and it is pretty to use. Here is more documentation about it: pathlib — Object-oriented filesystem paths.
BTW, if you are going to reuse the path, then it is better to assign it to a variable.
So it will become:
from pathlib import Path
p = Path("loc/of/myfile.txt")
if p.exists(): # works for both file and folders
# do stuffs...
#reuse 'p' if needed.

In 2016 the best way is still using os.path.isfile:
>>> os.path.isfile('/path/to/some/file.txt')
Or in Python 3 you can use pathlib:
import pathlib
path = pathlib.Path('/path/to/some/file.txt')
if path.is_file():
...

It doesn't seem like there's a meaningful functional difference between try/except and isfile(), so you should use which one makes sense.
If you want to read a file, if it exists, do
try:
f = open(filepath)
except IOError:
print 'Oh dear.'
But if you just wanted to rename a file if it exists, and therefore don't need to open it, do
if os.path.isfile(filepath):
os.rename(filepath, filepath + '.old')
If you want to write to a file, if it doesn't exist, do
# Python 2
if not os.path.isfile(filepath):
f = open(filepath, 'w')
# Python 3: x opens for exclusive creation, failing if the file already exists
try:
f = open(filepath, 'wx')
except IOError:
print 'file already exists'
If you need file locking, that's a different matter.

You could try this (safer):
try:
# http://effbot.org/zone/python-with-statement.htm
# 'with' is safer to open a file
with open('whatever.txt') as fh:
# Do something with 'fh'
except IOError as e:
print("({})".format(e))
The ouput would be:
([Errno 2] No such file or directory:
'whatever.txt')
Then, depending on the result, your program can just keep running from there or you can code to stop it if you want.

Date: 2017-12-04
Every possible solution has been listed in other answers.
An intuitive and arguable way to check if a file exists is the following:
import os
os.path.isfile('~/file.md') # Returns True if exists, else False
# Additionally, check a directory
os.path.isdir('~/folder') # Returns True if the folder exists, else False
# Check either a directory or a file
os.path.exists('~/file')
I made an exhaustive cheat sheet for your reference:
# os.path methods in exhaustive cheat sheet
{'definition': ['dirname',
'basename',
'abspath',
'relpath',
'commonpath',
'normpath',
'realpath'],
'operation': ['split', 'splitdrive', 'splitext',
'join', 'normcase'],
'compare': ['samefile', 'sameopenfile', 'samestat'],
'condition': ['isdir',
'isfile',
'exists',
'lexists'
'islink',
'isabs',
'ismount',],
'expand': ['expanduser',
'expandvars'],
'stat': ['getatime', 'getctime', 'getmtime',
'getsize']}

Although I always recommend using try and except statements, here are a few possibilities for you (my personal favourite is using os.access):
Try opening the file:
Opening the file will always verify the existence of the file. You can make a function just like so:
def File_Existence(filepath):
f = open(filepath)
return True
If it's False, it will stop execution with an unhanded IOError
or OSError in later versions of Python. To catch the exception,
you have to use a try except clause. Of course, you can always
use a try except` statement like so (thanks to hsandt
for making me think):
def File_Existence(filepath):
try:
f = open(filepath)
except IOError, OSError: # Note OSError is for later versions of Python
return False
return True
Use os.path.exists(path):
This will check the existence of what you specify. However, it checks for files and directories so beware about how you use it.
import os.path
>>> os.path.exists("this/is/a/directory")
True
>>> os.path.exists("this/is/a/file.txt")
True
>>> os.path.exists("not/a/directory")
False
Use os.access(path, mode):
This will check whether you have access to the file. It will check for permissions. Based on the os.py documentation, typing in os.F_OK, it will check the existence of the path. However, using this will create a security hole, as someone can attack your file using the time between checking the permissions and opening the file. You should instead go directly to opening the file instead of checking its permissions. (EAFP vs LBYP). If you're not going to open the file afterwards, and only checking its existence, then you can use this.
Anyway, here:
>>> import os
>>> os.access("/is/a/file.txt", os.F_OK)
True
I should also mention that there are two ways that you will not be able to verify the existence of a file. Either the issue will be permission denied or no such file or directory. If you catch an IOError, set the IOError as e (like my first option), and then type in print(e.args) so that you can hopefully determine your issue. I hope it helps! :)

If the file is for opening you could use one of the following techniques:
with open('somefile', 'xt') as f: # Using the x-flag, Python 3.3 and above
f.write('Hello\n')
if not os.path.exists('somefile'):
with open('somefile', 'wt') as f:
f.write("Hello\n")
else:
print('File already exists!')
Note: This finds either a file or a directory with the given name.

Additionally, os.access():
if os.access("myfile", os.R_OK):
with open("myfile") as fp:
return fp.read()
Being R_OK, W_OK, and X_OK the flags to test for permissions (doc).

if os.path.isfile(path_to_file):
try:
open(path_to_file)
pass
except IOError as e:
print "Unable to open file"
Raising exceptions is considered to be an acceptable, and Pythonic,
approach for flow control in your program. Consider handling missing
files with IOErrors. In this situation, an IOError exception will be
raised if the file exists but the user does not have read permissions.
Source: Using Python: How To Check If A File Exists

If you imported NumPy already for other purposes then there is no need to import other libraries like pathlib, os, paths, etc.
import numpy as np
np.DataSource().exists("path/to/your/file")
This will return true or false based on its existence.

Check file or directory exists
You can follow these three ways:
1. Using isfile()
Note 1: The os.path.isfile used only for files
import os.path
os.path.isfile(filename) # True if file exists
os.path.isfile(dirname) # False if directory exists
2. Using exists
Note 2: The os.path.exists is used for both files and directories
import os.path
os.path.exists(filename) # True if file exists
os.path.exists(dirname) # True if directory exists
3. The pathlib.Path method (included in Python 3+, installable with pip for Python 2)
from pathlib import Path
Path(filename).exists()

You can write Brian's suggestion without the try:.
from contextlib import suppress
with suppress(IOError), open('filename'):
process()
suppress is part of Python 3.4. In older releases you can quickly write your own suppress:
from contextlib import contextmanager
#contextmanager
def suppress(*exceptions):
try:
yield
except exceptions:
pass

I'm the author of a package that's been around for about 10 years, and it has a function that addresses this question directly. Basically, if you are on a non-Windows system, it uses Popen to access find. However, if you are on Windows, it replicates find with an efficient filesystem walker.
The code itself does not use a try block… except in determining the operating system and thus steering you to the "Unix"-style find or the hand-buillt find. Timing tests showed that the try was faster in determining the OS, so I did use one there (but nowhere else).
>>> import pox
>>> pox.find('*python*', type='file', root=pox.homedir(), recurse=False)
['/Users/mmckerns/.python']
And the doc…
>>> print pox.find.__doc__
find(patterns[,root,recurse,type]); Get path to a file or directory
patterns: name or partial name string of items to search for
root: path string of top-level directory to search
recurse: if True, recurse down from root directory
type: item filter; one of {None, file, dir, link, socket, block, char}
verbose: if True, be a little verbose about the search
On some OS, recursion can be specified by recursion depth (an integer).
patterns can be specified with basic pattern matching. Additionally,
multiple patterns can be specified by splitting patterns with a ';'
For example:
>>> find('pox*', root='..')
['/Users/foo/pox/pox', '/Users/foo/pox/scripts/pox_launcher.py']
>>> find('*shutils*;*init*')
['/Users/foo/pox/pox/shutils.py', '/Users/foo/pox/pox/__init__.py']
>>>
The implementation, if you care to look, is here:
https://github.com/uqfoundation/pox/blob/89f90fb308f285ca7a62eabe2c38acb87e89dad9/pox/shutils.py#L190

Adding one more slight variation which isn't exactly reflected in the other answers.
This will handle the case of the file_path being None or empty string.
def file_exists(file_path):
if not file_path:
return False
elif not os.path.isfile(file_path):
return False
else:
return True
Adding a variant based on suggestion from Shahbaz
def file_exists(file_path):
if not file_path:
return False
else:
return os.path.isfile(file_path)
Adding a variant based on suggestion from Peter Wood
def file_exists(file_path):
return file_path and os.path.isfile(file_path):

Here's a one-line Python command for the Linux command line environment. I find this very handy since I'm not such a hot Bash guy.
python -c "import os.path; print os.path.isfile('/path_to/file.xxx')"

You can use the "OS" library of Python:
>>> import os
>>> os.path.exists("C:\\Users\\####\\Desktop\\test.txt")
True
>>> os.path.exists("C:\\Users\\####\\Desktop\\test.tx")
False

How do I check whether a file exists, without using the try statement?
In 2016, this is still arguably the easiest way to check if both a file exists and if it is a file:
import os
os.path.isfile('./file.txt') # Returns True if exists, else False
isfile is actually just a helper method that internally uses os.stat and stat.S_ISREG(mode) underneath. This os.stat is a lower-level method that will provide you with detailed information about files, directories, sockets, buffers, and more. More about os.stat here
Note: However, this approach will not lock the file in any way and therefore your code can become vulnerable to "time of check to time of use" (TOCTTOU) bugs.
So raising exceptions is considered to be an acceptable, and Pythonic, approach for flow control in your program. And one should consider handling missing files with IOErrors, rather than if statements (just an advice).

Thing to check if you have permissions to directory

I'm writting an python program and now I'm working at exceptions.
while True:
try:
os.makedirs("{}\\test".format(dest))
except PermissionError:
print("Make sure that you have access to specified path")
print("Try again specify your path: ", end='')
dest = input()
continue
break
It is working but later I need to delete that folder.
What is the better way to do it?

Don't.
It is almost never worth verifying that you have permissions to perform an operation that your program requires. For one thing, permissions are not the only possible reason for failure. A delete may also fail because of a file lock by another program, for instance. Unless you have a very good reason to do otherwise, it is both more efficient and more reliable to just write your code to try the operation and then abort on failure:
import shutil
try:
shutil.rmtree(path_to_remove) # Recursively deletes directory and files inside it
except Exception as ex:
print('Failed to delete directory, manual clean up may be required: {}'.format(path_to_remove))
sys.exit(1)
Other concerns about your code
Use os.path.join to concatenate file paths: os.makedirs(os.path.join(dest, test)). This will use the appropriate directory separator for the operating system.
Why are you looping on failure? In real world programs, simply aborting the entire operation is simpler and usually makes for a better user experience.
Are you sure you aren't looking for the tempfile library? It allows you to spit out a unique directory to the operating system's standard temporary location:
import tempfile
with tempfile.TemporaryDirectory() as tmpdir:
some_function_that_creates_several_files(tmpdir)
for f in os.walk(tmpdir):
# do something with each file
# tmpdir automatically deleted when context manager exits
# Or if you really only need the file
with tempfile.TemporaryFile() as tmpfile:
tmpfile.write('my data')
some_function_that_needs_a_file(tmpfile)
# tmpfile automatically deleted when context manager exits

I think what you want is os.access.
os.access(path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True)
Use the real uid/gid to test for access to path. Note that most operations will use the effective uid/gid, therefore this routine can be used in a suid/sgid environment to test if the invoking user has the specified access to path. mode should be F_OK to test the existence of path, or it can be the inclusive OR of one or more of R_OK, W_OK, and X_OK to test permissions. Return True if access is allowed, False if not.
For example:
os.access("/path", os.R_OK)
And the mode contains:
os.F_OK # existence
os.R_OK # readability
os.W_OK # writability
os.X_OK # executability
Refer: https://docs.python.org/3.7/library/os.html#os.access

Make file, 'Currently being used' in python [duplicate]

What is the most elegant way to solve this:
open a file for reading, but only if it is not already opened for writing
open a file for writing, but only if it is not already opened for reading or writing
The built-in functions work like this
>>> path = r"c:\scr.txt"
>>> file1 = open(path, "w")
>>> print file1
<open file 'c:\scr.txt', mode 'w' at 0x019F88D8>
>>> file2 = open(path, "w")
>>> print file2
<open file 'c:\scr.txt', mode 'w' at 0x02332188>
>>> file1.write("111")
>>> file2.write("222")
>>> file1.close()
scr.txt now contains '111'.
>>> file2.close()
scr.txt was overwritten and now contains '222' (on Windows, Python 2.4).
The solution should work inside the same process (like in the example above) as well as when another process has opened the file.
It is preferred, if a crashing program will not keep the lock open.

I don't think there is a fully crossplatform way. On unix, the fcntl module will do this for you. However on windows (which I assume you are by the paths), you'll need to use the win32file module.
Fortunately, there is a portable implementation (portalocker) using the platform appropriate method at the python cookbook.
To use it, open the file, and then call:
portalocker.lock(file, flags)
where flags are portalocker.LOCK_EX for exclusive write access, or LOCK_SH for shared, read access.

The solution should work inside the same process (like in the example above) as well as when another process has opened the file.
If by 'another process' you mean 'whatever process' (i.e. not your program), in Linux there's no way to accomplish this relying only on system calls (fcntl & friends). What you want is mandatory locking, and the Linux way to obtain it is a bit more involved:
Remount the partition that contains your file with the mand option:
# mount -o remount,mand /dev/hdXY
Set the sgid flag for your file:
# chmod g-x,g+s yourfile
In your Python code, obtain an exclusive lock on that file:
fcntl.flock(fd, fcntl.LOCK_EX)
Now even cat will not be able to read the file until you release the lock.

EDIT: I solved it myself! By using directory existence & age as a locking mechanism! Locking by file is safe only on Windows (because Linux silently overwrites), but locking by directory works perfectly both on Linux and Windows. See my GIT where I created an easy to use class 'lockbydir.DLock' for that:
https://github.com/drandreaskrueger/lockbydir
At the bottom of the readme, you find 3 GITplayers where you can see the code examples execute live in your browser! Quite cool, isn't it? :-)
Thanks for your attention
This was my original question:
I would like to answer to parity3 (https://meta.stackoverflow.com/users/1454536/parity3) but I can neither comment directly ('You must have 50 reputation to comment'), nor do I see any way to contact him/her directly. What do you suggest to me, to get through to him?
My question:
I have implemented something similiar to what parity3 suggested here as an answer: https://stackoverflow.com/a/21444311/3693375 ("Assuming your Python interpreter, and the ...")
And it works brilliantly - on Windows. (I am using it to implement a locking mechanism that works across independently started processes. https://github.com/drandreaskrueger/lockbyfile )
But other than parity3 says, it does NOT work the same on Linux:
os.rename(src, dst)
Rename the file or directory src to dst. ... On Unix, if dst exists
and is a file,
it will be replaced silently if the user has permission.
The operation may fail on some Unix flavors if src and dst
are on different filesystems. If successful, the renaming will
be an atomic operation (this is a POSIX requirement).
On Windows, if dst already exists, OSError will be raised
(https://docs.python.org/2/library/os.html#os.rename)
The silent replacing is the problem. On Linux.
The "if dst already exists, OSError will be raised" is great for my purposes. But only on Windows, sadly.
I guess parity3's example still works most of the time, because of his if condition
if not os.path.exists(lock_filename):
try:
os.rename(tmp_filename,lock_filename)
But then the whole thing is not atomic anymore.
Because the if condition might be true in two parallel processes, and then both will rename, but only one will win the renaming race. And no exception raised (in Linux).
Any suggestions? Thanks!
P.S.: I know this is not the proper way, but I am lacking an alternative. PLEASE don't punish me with lowering my reputation. I looked around a lot, to solve this myself. How to PM users in here? And meh why can't I?

Here's a start on the win32 half of a portable implementation, that does not need a seperate locking mechanism.
Requires the Python for Windows Extensions to get down to the win32 api, but that's pretty much mandatory for python on windows already, and can alternatively be done with ctypes. The code could be adapted to expose more functionality if it's needed (such as allowing FILE_SHARE_READ rather than no sharing at all). See also the MSDN documentation for the CreateFile and WriteFile system calls, and the article on Creating and Opening Files.
As has been mentioned, you can use the standard fcntl module to implement the unix half of this, if required.
import winerror, pywintypes, win32file
class LockError(StandardError):
pass
class WriteLockedFile(object):
"""
Using win32 api to achieve something similar to file(path, 'wb')
Could be adapted to handle other modes as well.
"""
def __init__(self, path):
try:
self._handle = win32file.CreateFile(
path,
win32file.GENERIC_WRITE,
0,
None,
win32file.OPEN_ALWAYS,
win32file.FILE_ATTRIBUTE_NORMAL,
None)
except pywintypes.error, e:
if e[0] == winerror.ERROR_SHARING_VIOLATION:
raise LockError(e[2])
raise
def close(self):
self._handle.close()
def write(self, str):
win32file.WriteFile(self._handle, str)
Here's how your example from above behaves:
>>> path = "C:\\scr.txt"
>>> file1 = WriteLockedFile(path)
>>> file2 = WriteLockedFile(path) #doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
LockError: ...
>>> file1.write("111")
>>> file1.close()
>>> print file(path).read()
111

I prefer to use filelock, a cross-platform Python library which barely requires any additional code. Here's an example of how to use it:
from filelock import FileLock
lockfile = r"c:\scr.txt"
lock = FileLock(lockfile + ".lock")
with lock:
file = open(path, "w")
file.write("111")
file.close()
Any code within the with lock: block is thread-safe, meaning that it will be finished before another process has access to the file.

Assuming your Python interpreter, and the underlying os and filesystem treat os.rename as an atomic operation and it will error when the destination exists, the following method is free of race conditions. I'm using this in production on a linux machine. Requires no third party libs and is not os dependent, and aside from an extra file create, the performance hit is acceptable for many use cases. You can easily apply python's function decorator pattern or a 'with_statement' contextmanager here to abstract out the mess.
You'll need to make sure that lock_filename does not exist before a new process/task begins.
import os,time
def get_tmp_file():
filename='tmp_%s_%s'%(os.getpid(),time.time())
open(filename).close()
return filename
def do_exclusive_work():
print 'exclusive work being done...'
num_tries=10
wait_time=10
lock_filename='filename.lock'
acquired=False
for try_num in xrange(num_tries):
tmp_filename=get_tmp_file()
if not os.path.exists(lock_filename):
try:
os.rename(tmp_filename,lock_filename)
acquired=True
except (OSError,ValueError,IOError), e:
pass
if acquired:
try:
do_exclusive_work()
finally:
os.remove(lock_filename)
break
os.remove(tmp_filename)
time.sleep(wait_time)
assert acquired, 'maximum tries reached, failed to acquire lock file'
EDIT
It has come to light that os.rename silently overwrites the destination on a non-windows OS. Thanks for pointing this out # akrueger!
Here is a workaround, gathered from here:
Instead of using os.rename you can use:
try:
if os.name != 'nt': # non-windows needs a create-exclusive operation
fd = os.open(lock_filename, os.O_WRONLY | os.O_CREAT | os.O_EXCL)
os.close(fd)
# non-windows os.rename will overwrite lock_filename silently.
# We leave this call in here just so the tmp file is deleted but it could be refactored so the tmp file is never even generated for a non-windows OS
os.rename(tmp_filename,lock_filename)
acquired=True
except (OSError,ValueError,IOError), e:
if os.name != 'nt' and not 'File exists' in str(e): raise
# akrueger You're probably just fine with your directory based solution, just giving you an alternate method.

To make you safe when opening files within one application, you could try something like this:
import time
class ExclusiveFile(file):
openFiles = {}
fileLocks = []
class FileNotExclusiveException(Exception):
pass
def __init__(self, *args):
sMode = 'r'
sFileName = args[0]
try:
sMode = args[1]
except:
pass
while sFileName in ExclusiveFile.fileLocks:
time.sleep(1)
ExclusiveFile.fileLocks.append(sFileName)
if not sFileName in ExclusiveFile.openFiles.keys() or (ExclusiveFile.openFiles[sFileName] == 'r' and sMode == 'r'):
ExclusiveFile.openFiles[sFileName] = sMode
try:
file.__init__(self, sFileName, sMode)
finally:
ExclusiveFile.fileLocks.remove(sFileName)
else:
ExclusiveFile.fileLocks.remove(sFileName)
raise self.FileNotExclusiveException(sFileName)
def close(self):
del ExclusiveFile.openFiles[self.name]
file.close(self)
That way you subclass the file class. Now just do:
>>> f = ExclusiveFile('/tmp/a.txt', 'r')
>>> f
<open file '/tmp/a.txt', mode 'r' at 0xb7d7cc8c>
>>> f1 = ExclusiveFile('/tmp/a.txt', 'r')
>>> f1
<open file '/tmp/a.txt', mode 'r' at 0xb7d7c814>
>>> f2 = ExclusiveFile('/tmp/a.txt', 'w') # can't open it for writing now
exclfile.FileNotExclusiveException: /tmp/a.txt
If you open it first with 'w' mode, it won't allow anymore opens, even in read mode, just as you wanted...

Check if file system is case-insensitive in Python

Is there a simple way to check in Python if a file system is case insensitive? I'm thinking in particular of file systems like HFS+ (OSX) and NTFS (Windows), where you can access the same file as foo, Foo or FOO, even though the file case is preserved.

import os
import tempfile
# By default mkstemp() creates a file with
# a name that begins with 'tmp' (lowercase)
tmphandle, tmppath = tempfile.mkstemp()
if os.path.exists(tmppath.upper()):
# Case insensitive.
else:
# Case sensitive.

The answer provided by Amber will leave temporary file debris unless closing and deleting are handled explicitly. To avoid this I use:
import os
import tempfile
def is_fs_case_sensitive():
#
# Force case with the prefix
#
with tempfile.NamedTemporaryFile(prefix='TmP') as tmp_file:
return(not os.path.exists(tmp_file.name.lower()))
Though my usage cases generally test this more than once, so I stash the result to avoid having to touch the filesystem more than once.
def is_fs_case_sensitive():
if not hasattr(is_fs_case_sensitive, 'case_sensitive'):
with tempfile.NamedTemporaryFile(prefix='TmP') as tmp_file:
setattr(is_fs_case_sensitive,
'case_sensitive',
not os.path.exists(tmp_file.name.lower()))
return(is_fs_case_sensitive.case_sensitive)
Which is marginally slower if only called once, and significantly faster in every other case.

Good point on the different file systems, etc., Eric Smith. But why not use tempfile.NamedTemporaryFile with the dir parameter and avoid doing all that context manager lifting yourself?
def is_fs_case_sensitive(path):
#
# Force case with the prefix
#
with tempfile.NamedTemporaryFile(prefix='TmP',dir=path, delete=True) as tmp_file:
return(not os.path.exists(tmp_file.name.lower()))
I should also mention that your solution does not guarantee that you are actually testing for case sensitivity. Unless you check the default prefix (using tempfile.gettempprefix()) to make sure it contains a lower-case character. So including the prefix here is not really optional.
Your solution cleans up the temp file. I agree that it seemed obvious, but one never knows, do one?

Variation on #Shrikant's answer, applicable within a module (i.e. not in the REPL), even if your user doesn't have a home:
import os.path
is_fs_case_insensitive = os.path.exists(__file__.upper()) and os.path.exists(__file__.lower())
print(f"{is_fs_case_insensitive=}")
output (macOS):
is_fs_case_insensitive=True 👈
And the Linux side of things:
(ssha)vagrant ~$python3.8 test.py
is_fs_case_insensitive=False 👈
(ssha)vagrant ~$lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
FWIW, I checked pathlib, os, os.path's contents via:
[k for k in vars(pathlib).keys() if "case" in k.lower()]
and nothing looks like it, though it does have a pathlib.supports_symlinks but nothing about case-sensitivity.
And the following will work in the REPL as well:
is_fs_case_insensitive = os.path.exists(os.path.__file__.upper()) and os.path.exists(os.path.__file__.lower())

Starting with Amber's answer, I came up with this code. I'm not sure it is totally robust, but it attempts to address some issues in the original (that I'll mention below).
import os
import sys
import tempfile
import contextlib
def is_case_sensitive(path):
with temp(path) as tmppath:
head, tail = os.path.split(tmppath)
testpath = os.path.join(head, tail.upper())
return not os.path.exists(testpath)
#contextlib.contextmanager
def temp(path):
tmphandle, tmppath = tempfile.mkstemp(dir=path)
os.close(tmphandle)
try:
yield tmppath
finally:
os.unlink(tmppath)
if __name__ == '__main__':
path = os.path.abspath(sys.argv[1])
print(path)
print('Case sensitive: ' + str(is_case_sensitive(path)))
Without specifying the dir parameter in mkstemp, the question of case sensitivity is vague. You're testing case sensitivity of wherever the temporary directory happens to be, but you may want to know about a specific path.
If you convert the full path returned from mkstemp to upper-case, you could potentially miss a transition somewhere in the path. For example, I have a USB flash drive on Linux mounted using vfat at /media/FLASH. Testing the existence of anything under /MEDIA/FLASH will always fail because /media is on a (case-sensitive) ext4 partition, but the flash drive itself is case-insensitive. Mounted network shares could be another situation like this.
Finally, and maybe it goes without saying in Amber's answer, you'll want to clean up the temp file created by mkstemp.

I think there's a much simpler (and probably faster) solution to this. The following seemed to be working for where I tested:
import os.path
home = os.path.expanduser('~')
is_fs_case_insensitive = os.path.exists(home.upper()) and os.path.exists(home.lower())

import os
if os.path.normcase('A') == os.path.normcase('a'):
# case insensitive
else:
# case sensitive

I think we can do this in one line with pathlib on Python 3.5+ without creating temporary files:
from pathlib import Path
def is_case_insensitive(path) -> bool:
return Path(str(Path.home()).upper()).exists()
Or for the inverse:
def is_case_sensitive(path) -> bool:
return not Path(str(Path.home()).upper()).exists()

I believe this to be the simplest solution to the question:
from fnmatch import fnmatch
os_is_case_insensitive = fnmatch('A','a')
From: https://docs.python.org/3.4/library/fnmatch.html
If the operating system is case-insensitive, then both parameters will
be normalized to all lower- or upper-case before the comparison is
performed.

How can I safely create a directory (possibly including intermediate directories)?

I am writing a file using Python, and I want it to be placed in a specific path. How can I safely make sure that the path exists?
That is: how can I check whether the folder exists, along with its parents? If there are missing folders along the path, how can I create them?

On Python ≥ 3.5, use pathlib.Path.mkdir:
from pathlib import Path
Path("/my/directory").mkdir(parents=True, exist_ok=True)
For older versions of Python, I see two answers with good qualities, each with a small flaw, so I will give my take on it:
Try os.path.exists, and consider os.makedirs for the creation.
import os
if not os.path.exists(directory):
os.makedirs(directory)
As noted in comments and elsewhere, there's a race condition – if the directory is created between the os.path.exists and the os.makedirs calls, the os.makedirs will fail with an OSError. Unfortunately, blanket-catching OSError and continuing is not foolproof, as it will ignore a failure to create the directory due to other factors, such as insufficient permissions, full disk, etc.
One option would be to trap the OSError and examine the embedded error code (see Is there a cross-platform way of getting information from Python’s OSError):
import os, errno
try:
os.makedirs(directory)
except OSError as e:
if e.errno != errno.EEXIST:
raise
Alternatively, there could be a second os.path.exists, but suppose another created the directory after the first check, then removed it before the second one – we could still be fooled.
Depending on the application, the danger of concurrent operations may be more or less than the danger posed by other factors such as file permissions. The developer would have to know more about the particular application being developed and its expected environment before choosing an implementation.
Modern versions of Python improve this code quite a bit, both by exposing FileExistsError (in 3.3+)...
try:
os.makedirs("path/to/directory")
except FileExistsError:
# directory already exists
pass
...and by allowing a keyword argument to os.makedirs called exist_ok (in 3.2+).
os.makedirs("path/to/directory", exist_ok=True) # succeeds even if directory exists.

Python 3.5+:
import pathlib
pathlib.Path('/my/directory').mkdir(parents=True, exist_ok=True)
pathlib.Path.mkdir as used above recursively creates the directory and does not raise an exception if the directory already exists. If you don't need or want the parents to be created, skip the parents argument.
Python 3.2+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
If using Python 3.4, even though it comes with pathlib, it is missing the useful exist_ok option. The backport is intended to offer a newer and superior implementation of mkdir which includes this missing option.
Using os:
import os
os.makedirs(path, exist_ok=True)
os.makedirs as used above recursively creates the directory and does not raise an exception if the directory already exists. It has the optional exist_ok argument only if using Python 3.2+, with a default value of False. This argument does not exist in Python 2.x up to 2.7. As such, there is no need for manual exception handling as with Python 2.7.
Python 2.7+:
Using pathlib:
If you can, install the current pathlib backport named pathlib2. Do not install the older unmaintained backport named pathlib. Next, refer to the Python 3.5+ section above and use it the same.
Using os:
import os
try:
os.makedirs(path)
except OSError:
if not os.path.isdir(path):
raise
While a naive solution may first use os.path.isdir followed by os.makedirs, the solution above reverses the order of the two operations. In doing so, it prevents a common race condition having to do with a duplicated attempt at creating the directory, and also disambiguates files from directories.
Note that capturing the exception and using errno is of limited usefulness because OSError: [Errno 17] File exists, i.e. errno.EEXIST, is raised for both files and directories. It is more reliable simply to check if the directory exists.
Alternative:
mkpath creates the nested directory, and does nothing if the directory already exists. This works in both Python 2 and 3. Note however that distutils has been deprecated, and is scheduled for removal in Python 3.12.
import distutils.dir_util
distutils.dir_util.mkpath(path)
Per Bug 10948, a severe limitation of this alternative is that it works only once per python process for a given path. In other words, if you use it to create a directory, then delete the directory from inside or outside Python, then use mkpath again to recreate the same directory, mkpath will simply silently use its invalid cached info of having previously created the directory, and will not actually make the directory again. In contrast, os.makedirs doesn't rely on any such cache. This limitation may be okay for some applications.
With regard to the directory's mode, please refer to the documentation if you care about it.

Using try except and the right error code from errno module gets rid of the race condition and is cross-platform:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
In other words, we try to create the directories, but if they already exist we ignore the error. On the other hand, any other error gets reported. For example, if you create dir 'a' beforehand and remove all permissions from it, you will get an OSError raised with errno.EACCES (Permission denied, error 13).

Starting from Python 3.5, pathlib.Path.mkdir has an exist_ok flag:
from pathlib import Path
path = Path('/my/directory/filename.txt')
path.parent.mkdir(parents=True, exist_ok=True)
# path.parent ~ os.path.dirname(path)
This recursively creates the directory and does not raise an exception if the directory already exists.
(just as os.makedirs got an exist_ok flag starting from python 3.2 e.g os.makedirs(path, exist_ok=True))
Note: when i posted this answer none of the other answers mentioned exist_ok...

I would personally recommend that you use os.path.isdir() to test instead of os.path.exists().
>>> os.path.exists('/tmp/dirname')
True
>>> os.path.exists('/tmp/dirname/filename.etc')
True
>>> os.path.isdir('/tmp/dirname/filename.etc')
False
>>> os.path.isdir('/tmp/fakedirname')
False
If you have:
>>> directory = raw_input(":: ")
And a foolish user input:
:: /tmp/dirname/filename.etc
... You're going to end up with a directory named filename.etc when you pass that argument to os.makedirs() if you test with os.path.exists().

Check os.makedirs: (It makes sure the complete path exists.)
To handle the fact the directory might exist, catch OSError.
(If exist_ok is False (the default), an OSError is raised if the target directory already exists.)
import os
try:
os.makedirs('./path/to/somewhere')
except OSError:
pass

Try the os.path.exists function
if not os.path.exists(dir):
os.mkdir(dir)

Insights on the specifics of this situation
You give a particular file at a certain path and you pull the directory from the file path. Then after making sure you have the directory, you attempt to open a file for reading. To comment on this code:
filename = "/my/directory/filename.txt"
dir = os.path.dirname(filename)
We want to avoid overwriting the builtin function, dir. Also, filepath or perhaps fullfilepath is probably a better semantic name than filename so this would be better written:
import os
filepath = '/my/directory/filename.txt'
directory = os.path.dirname(filepath)
Your end goal is to open this file, you initially state, for writing, but you're essentially approaching this goal (based on your code) like this, which opens the file for reading:
if not os.path.exists(directory):
os.makedirs(directory)
f = file(filename)
Assuming opening for reading
Why would you make a directory for a file that you expect to be there and be able to read?
Just attempt to open the file.
with open(filepath) as my_file:
do_stuff(my_file)
If the directory or file isn't there, you'll get an IOError with an associated error number: errno.ENOENT will point to the correct error number regardless of your platform. You can catch it if you want, for example:
import errno
try:
with open(filepath) as my_file:
do_stuff(my_file)
except IOError as error:
if error.errno == errno.ENOENT:
print 'ignoring error because directory or file is not there'
else:
raise
Assuming we're opening for writing
This is probably what you're wanting.
In this case, we probably aren't facing any race conditions. So just do as you were, but note that for writing, you need to open with the w mode (or a to append). It's also a Python best practice to use the context manager for opening files.
import os
if not os.path.exists(directory):
os.makedirs(directory)
with open(filepath, 'w') as my_file:
do_stuff(my_file)
However, say we have several Python processes that attempt to put all their data into the same directory. Then we may have contention over creation of the directory. In that case it's best to wrap the makedirs call in a try-except block.
import os
import errno
if not os.path.exists(directory):
try:
os.makedirs(directory)
except OSError as error:
if error.errno != errno.EEXIST:
raise
with open(filepath, 'w') as my_file:
do_stuff(my_file)

I have put the following down. It's not totally foolproof though.
import os
dirname = 'create/me'
try:
os.makedirs(dirname)
except OSError:
if os.path.exists(dirname):
# We are nearly safe
pass
else:
# There was an error on creation, so make sure we know about it
raise
Now as I say, this is not really foolproof, because we have the possiblity of failing to create the directory, and another process creating it during that period.

Check if a directory exists and create it if necessary?
The direct answer to this is, assuming a simple situation where you don't expect other users or processes to be messing with your directory:
if not os.path.exists(d):
os.makedirs(d)
or if making the directory is subject to race conditions (i.e. if after checking the path exists, something else may have already made it) do this:
import errno
try:
os.makedirs(d)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
But perhaps an even better approach is to sidestep the resource contention issue, by using temporary directories via tempfile:
import tempfile
d = tempfile.mkdtemp()
Here's the essentials from the online doc:
mkdtemp(suffix='', prefix='tmp', dir=None)
User-callable function to create and return a unique temporary
directory. The return value is the pathname of the directory.
The directory is readable, writable, and searchable only by the
creating user.
Caller is responsible for deleting the directory when done with it.
New in Python 3.5: pathlib.Path with exist_ok
There's a new Path object (as of 3.4) with lots of methods one would want to use with paths - one of which is mkdir.
(For context, I'm tracking my weekly rep with a script. Here's the relevant parts of code from the script that allow me to avoid hitting Stack Overflow more than once a day for the same data.)
First the relevant imports:
from pathlib import Path
import tempfile
We don't have to deal with os.path.join now - just join path parts with a /:
directory = Path(tempfile.gettempdir()) / 'sodata'
Then I idempotently ensure the directory exists - the exist_ok argument shows up in Python 3.5:
directory.mkdir(exist_ok=True)
Here's the relevant part of the documentation:
If exist_ok is true, FileExistsError exceptions will be ignored (same behavior as the POSIX mkdir -p command), but only if the last path component is not an existing non-directory file.
Here's a little more of the script - in my case, I'm not subject to a race condition, I only have one process that expects the directory (or contained files) to be there, and I don't have anything trying to remove the directory.
todays_file = directory / str(datetime.datetime.utcnow().date())
if todays_file.exists():
logger.info("todays_file exists: " + str(todays_file))
df = pd.read_json(str(todays_file))
Path objects have to be coerced to str before other APIs that expect str paths can use them.
Perhaps Pandas should be updated to accept instances of the abstract base class, os.PathLike.

fastest safest way to do it is:
it will create if not exists and skip if exists:
from pathlib import Path
Path("path/with/childs/.../").mkdir(parents=True, exist_ok=True)

Best way to do this in python
#Devil
import os
directory = "./out_dir/subdir1/subdir2"
if not os.path.exists(directory):
os.makedirs(directory)

In Python 3.4 you can also use the brand new pathlib module:
from pathlib import Path
path = Path("/my/directory/filename.txt")
try:
if not path.parent.exists():
path.parent.mkdir(parents=True)
except OSError:
# handle error; you can also catch specific errors like
# FileExistsError and so on.

For a one-liner solution, you can use IPython.utils.path.ensure_dir_exists():
from IPython.utils.path import ensure_dir_exists
ensure_dir_exists(dir)
From the documentation: Ensure that a directory exists. If it doesn’t exist, try to create it and protect against a race condition if another process is doing the same.
IPython is an extension package, not part of the standard library.

In Python3, os.makedirs supports setting exist_ok. The default setting is False, which means an OSError will be raised if the target directory already exists. By setting exist_ok to True, OSError (directory exists) will be ignored and the directory will not be created.
os.makedirs(path,exist_ok=True)
In Python2, os.makedirs doesn't support setting exist_ok. You can use the approach in heikki-toivonen's answer:
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise

The relevant Python documentation suggests the use of the EAFP coding style (Easier to Ask for Forgiveness than Permission). This means that the code
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST:
raise
else:
print "\nBE CAREFUL! Directory %s already exists." % path
is better than the alternative
if not os.path.exists(path):
os.makedirs(path)
else:
print "\nBE CAREFUL! Directory %s already exists." % path
The documentation suggests this exactly because of the race condition discussed in this question. In addition, as others mention here, there is a performance advantage in querying once instead of twice the OS. Finally, the argument placed forward, potentially, in favour of the second code in some cases --when the developer knows the environment the application is running-- can only be advocated in the special case that the program has set up a private environment for itself (and other instances of the same program).
Even in that case, this is a bad practice and can lead to long useless debugging. For example, the fact we set the permissions for a directory should not leave us with the impression permissions are set appropriately for our purposes. A parent directory could be mounted with other permissions. In general, a program should always work correctly and the programmer should not expect one specific environment.

I found this Q/A after I was puzzled by some of the failures and errors I was getting while working with directories in Python. I am working in Python 3 (v.3.5 in an Anaconda virtual environment on an Arch Linux x86_64 system).
Consider this directory structure:
└── output/ ## dir
├── corpus ## file
├── corpus2/ ## dir
└── subdir/ ## dir
Here are my experiments/notes, which provides clarification:
# ----------------------------------------------------------------------------
# [1] https://stackoverflow.com/questions/273192/how-can-i-create-a-directory-if-it-does-not-exist
import pathlib
""" Notes:
1. Include a trailing slash at the end of the directory path
("Method 1," below).
2. If a subdirectory in your intended path matches an existing file
with same name, you will get the following error:
"NotADirectoryError: [Errno 20] Not a directory:" ...
"""
# Uncomment and try each of these "out_dir" paths, singly:
# ----------------------------------------------------------------------------
# METHOD 1:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## no error but no dir created (missing tailing /)
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but no file created (os.makedirs creates dir, not files! ;-)
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# [2] https://docs.python.org/3/library/os.html#os.makedirs
# Uncomment these to run "Method 1":
#directory = os.path.dirname(out_dir)
#os.makedirs(directory, mode=0o777, exist_ok=True)
# ----------------------------------------------------------------------------
# METHOD 2:
# Re-running does not overwrite existing directories and files; no errors.
# out_dir = 'output/corpus3' ## works
# out_dir = 'output/corpus3/' ## works
# out_dir = 'output/corpus3/doc1' ## works
# out_dir = 'output/corpus3/doc1/' ## works
# out_dir = 'output/corpus3/doc1/doc.txt' ## no error but creates a .../doc.txt./ dir
# out_dir = 'output/corpus2/tfidf/' ## fails with "Errno 20" (existing file named "corpus2")
# out_dir = 'output/corpus3/tfidf/' ## works
# out_dir = 'output/corpus3/a/b/c/d/' ## works
# Uncomment these to run "Method 2":
#import os, errno
#try:
# os.makedirs(out_dir)
#except OSError as e:
# if e.errno != errno.EEXIST:
# raise
# ----------------------------------------------------------------------------
Conclusion: in my opinion, "Method 2" is more robust.
[1] How can I safely create a nested directory?
[2] https://docs.python.org/3/library/os.html#os.makedirs

You can use mkpath
# Create a directory and any missing ancestor directories.
# If the directory already exists, do nothing.
from distutils.dir_util import mkpath
mkpath("test")
Note that it will create the ancestor directories as well.
It works for Python 2 and 3.

In case you're writing a file to a variable path, you can use this on the file's path to make sure that the parent directories are created.
from pathlib import Path
path_to_file = Path("zero/or/more/directories/file.ext")
parent_directory_of_file = path_to_file.parent
parent_directory_of_file.mkdir(parents=True, exist_ok=True)
Works even if path_to_file is file.ext (zero directories deep).
See pathlib.PurePath.parent and pathlib.Path.mkdir.

Why not use subprocess module if running on a machine that supports command
mkdir with -p option ?
Works on python 2.7 and python 3.6
from subprocess import call
call(['mkdir', '-p', 'path1/path2/path3'])
Should do the trick on most systems.
In situations where portability doesn't matter (ex, using docker) the solution is a clean 2 lines. You also don't have to add logic to check if directories exist or not. Finally, it is safe to re-run without any side effects
If you need error handling:
from subprocess import check_call
try:
check_call(['mkdir', '-p', 'path1/path2/path3'])
except:
handle...

You have to set the full path before creating the directory:
import os,sys,inspect
import pathlib
currentdir = os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))
your_folder = currentdir + "/" + "your_folder"
if not os.path.exists(your_folder):
pathlib.Path(your_folder).mkdir(parents=True, exist_ok=True)
This works for me and hopefully, it will works for you as well

I saw Heikki Toivonen and A-B-B's answers and thought of this variation.
import os
import errno
def make_sure_path_exists(path):
try:
os.makedirs(path)
except OSError as exception:
if exception.errno != errno.EEXIST or not os.path.isdir(path):
raise

I use os.path.exists(), here is a Python 3 script that can be used to check if a directory exists, create one if it does not exist, and delete it if it does exist (if desired).
It prompts users for input of the directory and can be easily modified.

Use this command check and create dir
if not os.path.isdir(test_img_dir):
os.mkdir(test_img_dir)

Call the function create_dir() at the entry point of your program/project.
import os
def create_dir(directory):
if not os.path.exists(directory):
print('Creating Directory '+directory)
os.makedirs(directory)
create_dir('Project directory')

If you consider the following:
os.path.isdir('/tmp/dirname')
means a directory (path) exists AND is a directory. So for me this way does what I need. So I can make sure it is folder (not a file) and exists.

You can use os.listdir for this:
import os
if 'dirName' in os.listdir('parentFolderPath')
print('Directory Exists')

This may not exactly answer the question. But I guess your real intention is to create a file and its parent directories, given its content all in 1 command.
You can do that with fastcore extension to pathlib: path.mk_write(data)
from fastcore.utils import Path
Path('/dir/to/file.txt').mk_write('Hello World')
See more in fastcore documentation