This question already has answers here:
How do you properly determine the current script directory?
(16 answers)
Closed 6 months ago.
I'm building a simple helper script for work that will copy a couple of template files in our code base to the current directory. I don't, however, have the absolute path to the directory where the templates are stored. I do have a relative path from the script but when I call the script it treats that as a path relative to the current working directory. Is there a way to specify that this relative url is from the location of the script instead?
In the file that has the script, you want to do something like this:
import os
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
This will give you the absolute path to the file you're looking for. Note that if you're using setuptools, you should probably use its package resources API instead.
UPDATE: I'm responding to a comment here so I can paste a code sample. :-)
Am I correct in thinking that __file__ is not always available (e.g. when you run the file directly rather than importing it)?
I'm assuming you mean the __main__ script when you mention running the file directly. If so, that doesn't appear to be the case on my system (python 2.5.1 on OS X 10.5.7):
#foo.py
import os
print os.getcwd()
print __file__
#in the interactive interpreter
>>> import foo
/Users/jason
foo.py
#and finally, at the shell:
~ % python foo.py
/Users/jason
foo.py
However, I do know that there are some quirks with __file__ on C extensions. For example, I can do this on my Mac:
>>> import collections #note that collections is a C extension in Python 2.5
>>> collections.__file__
'/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-
dynload/collections.so'
However, this raises an exception on my Windows machine.
It's 2018 now, and Python has already evolved to the __future__ long time ago. So how about using the amazing pathlib coming with Python 3.4 to accomplish the task instead of struggling with os, os.path, glob , shutil, etc.
So we have 3 paths here (possibly duplicated):
mod_path: which is the path of the simple helper script
src_path: which contains a couple of template files waiting to be copied.
cwd: current directory, the destination of those template files.
and the problem is: we don't have the full path of src_path, only know its relative path to the mod_path.
Now let's solve this with the amazing pathlib:
# Hope you don't be imprisoned by legacy Python code :)
from pathlib import Path
# `cwd`: current directory is straightforward
cwd = Path.cwd()
# `mod_path`: According to the accepted answer and combine with future power
# if we are in the `helper_script.py`
mod_path = Path(__file__).parent
# OR if we are `import helper_script`
mod_path = Path(helper_script.__file__).parent
# `src_path`: with the future power, it's just so straightforward
relative_path_1 = 'same/parent/with/helper/script/'
relative_path_2 = '../../or/any/level/up/'
src_path_1 = (mod_path / relative_path_1).resolve()
src_path_2 = (mod_path / relative_path_2).resolve()
In the future, it's just that simple.
Moreover, we can select and check and copy/move those template files with pathlib:
if src_path != cwd:
# When we have different types of files in the `src_path`
for template_path in src_path.glob('*.ini'):
fname = template_path.name
target = cwd / fname
if not target.exists():
# This is the COPY action
with target.open(mode='wb') as fd:
fd.write(template_path.read_bytes())
# If we want MOVE action, we could use:
# template_path.replace(target)
you need os.path.realpath (sample below adds the parent directory to your path)
import sys,os
sys.path.append(os.path.realpath('..'))
As mentioned in the accepted answer
import os
dir = os.path.dirname(__file__)
filename = os.path.join(dir, '/relative/path/to/file/you/want')
I just want to add that
the latter string can't begin with the backslash , infact no string
should include a backslash
It should be something like
import os
dir = os.path.dirname(__file__)
filename = os.path.join(dir, 'relative','path','to','file','you','want')
The accepted answer can be misleading in some cases , please refer to this link for details
Consider my code:
import os
def readFile(filename):
filehandle = open(filename)
print filehandle.read()
filehandle.close()
fileDir = os.path.dirname(os.path.realpath('__file__'))
print fileDir
#For accessing the file in the same folder
filename = "same.txt"
readFile(filename)
#For accessing the file in a folder contained in the current folder
filename = os.path.join(fileDir, 'Folder1.1/same.txt')
readFile(filename)
#For accessing the file in the parent folder of the current folder
filename = os.path.join(fileDir, '../same.txt')
readFile(filename)
#For accessing the file inside a sibling folder.
filename = os.path.join(fileDir, '../Folder2/same.txt')
filename = os.path.abspath(os.path.realpath(filename))
print filename
readFile(filename)
See sys.path
As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.
Use this path as the root folder from which you apply your relative path
>>> import sys
>>> import os.path
>>> sys.path[0]
'C:\\Python25\\Lib\\idlelib'
>>> os.path.relpath(sys.path[0], "path_to_libs") # if you have python 2.6
>>> os.path.join(sys.path[0], "path_to_libs")
'C:\\Python25\\Lib\\idlelib\\path_to_libs'
Instead of using
import os
dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
as in the accepted answer, it would be more robust to use:
import inspect
import os
dirname = os.path.dirname(os.path.abspath(inspect.stack()[0][1]))
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
because using __file__ will return the file from which the module was loaded, if it was loaded from a file, so if the file with the script is called from elsewhere, the directory returned will not be correct.
These answers give more detail: https://stackoverflow.com/a/31867043/5542253 and https://stackoverflow.com/a/50502/5542253
From what suggest others and from pathlib documentation, a simple (but not ideal) solution is the following (suppose the file we need to refer to is Test/data/users.csv):
# Current file location: Tests/src/long/module/subdir/some_script.py
from pathlib import Path
# back to Tests/
PROJECT_ROOT = Path(__file__).parents[4]
# then down to Test/data/users.csv
CSV_USERS_PATH = PROJECT_ROOT / 'data' / 'users.csv'
with CSV_USERS_PATH.open() as users:
print(users.read())
This works but looks a bit odd because if you move some_script.py around, the path to the root of our project may change (and we would therefore need to change the parents[4] part).
I think I found a better solution that, based on the same idea.
We will use a file paths.py to store where the root of the project is, this file will remain at the same location compared to the root directory.
Tests
├── data
│ └── users.csv
└── src
├── long
│ └── module
│ └── subdir
│ └── some_script.py
├── main.py
└── paths.py
Where paths.py's only responsability is to provide PROJECT_ROOT:
from pathlib import Path
PROJECT_ROOT = Path(__file__).parents[1]
All scripts can now use paths.PROJECT_ROOT to express absolute paths from the root of the project. For example in src/long/module/subdir/some_script.py we could have:
from paths import PROJECT_ROOT
CSV_USERS_PATH = PROJECT_ROOT / 'data' / 'users.csv'
def hello():
with CSV_USERS_PATH.open() as f:
print(f.read())
And everything goes as expected:
~/Tests/src/$ python main.py
/Users/cglacet/Tests/data/users.csv
hello, user
~/Tests/$ python src/main.py
/Users/cglacet/Tests/data/users.csv
hello, user
The main.py script simply is:
from long.module.subdir import some_script
some_script.hello()
summary of the most important commands
>>> import os
>>> os.path.join('/home/user/tmp', 'subfolder')
'/home/user/tmp/subfolder'
>>> os.path.normpath('/home/user/tmp/../test/..')
'/home/user'
>>> os.path.relpath('/home/user/tmp', '/home/user')
'tmp'
>>> os.path.isabs('/home/user/tmp')
True
>>> os.path.isabs('/tmp')
True
>>> os.path.isabs('tmp')
False
>>> os.path.isabs('./../tmp')
False
>>> os.path.realpath('/home/user/tmp/../test/..') # follows symbolic links
'/home/user'
A detailed description is found in the docs.
These are linux paths. Windows should work analogous.
Hi first of all you should understand functions os.path.abspath(path) and os.path.relpath(path)
In short os.path.abspath(path) makes a relative path to absolute path. And if the path provided is itself a absolute path then the function returns the same path.
similarly os.path.relpath(path) makes a absolute path to relative path. And if the path provided is itself a relative path then the function returns the same path.
Below example can let you understand the above concept properly:
suppose i have a file input_file_list.txt which contains list of input files to be processed by my python script.
D:\conc\input1.dic
D:\conc\input2.dic
D:\Copyioconc\input_file_list.txt
If you see above folder structure, input_file_list.txt is present in Copyofconc folder and the files to be processed by the python script are present in conc folder
But the content of the file input_file_list.txt is as shown below:
..\conc\input1.dic
..\conc\input2.dic
And my python script is present in D: drive.
And the relative path provided in the input_file_list.txt file are relative to the path of input_file_list.txt file.
So when python script shall executed the current working directory (use os.getcwd() to get the path)
As my relative path is relative to input_file_list.txt, that is "D:\Copyofconc", i have to change the current working directory to "D:\Copyofconc".
So i have to use os.chdir('D:\Copyofconc'), so the current working directory shall be "D:\Copyofconc".
Now to get the files input1.dic and input2.dic, i will read the lines "..\conc\input1.dic" then shall use the command
input1_path= os.path.abspath('..\conc\input1.dic') (to change relative path to absolute path. Here as current working directory is "D:\Copyofconc", the file ".\conc\input1.dic" shall be accessed relative to "D:\Copyofconc")
so input1_path shall be "D:\conc\input1.dic"
This code will return the absolute path to the main script.
import os
def whereAmI():
return os.path.dirname(os.path.realpath(__import__("__main__").__file__))
This will work even in a module.
An alternative which works for me:
this_dir = os.path.dirname(__file__)
filename = os.path.realpath("{0}/relative/file.path".format(this_dir))
Example
Here's an example, tested in Python '3.9.5`:
your current directory: 'c:\project1\code\'
and you want to access the following folder: 'c:\project1\dataset\train\'.
Then you can access the folder using the following address: '../dataset/train/'
References
If you want some more information about path in Python, read this:
PEP - 355
PEP - 519
What worked for me is using sys.path.insert. Then I specified the directory I needed to go. For example I just needed to go up one directory.
import sys
sys.path.insert(0, '../')
I think to work with all systems use "ntpath" instead of "os.path". Today, it works well with Windows, Linux and Mac OSX.
import ntpath
import os
dirname = ntpath.dirname(__file__)
filename = os.path.join(dirname, 'relative/path/to/file/you/want')
A simple solution would be
import os
os.chdir(os.path.dirname(__file__))
From C:\Users\xyz\myFolder to C:\Users\xyz\testdata :
import os
working_dir = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
# C:\Users\xyz\myFolder
print(working_dir)
updated_working_dir = os.path.join(os.path.realpath(working_dir + '/../'), 'testdata')
# C:\Users\xyz\testdata
print(updated_working_dir)
Output
C:\Users\xyz\myFolder
C:\Users\xyz\testdata
Here is my sumup:
First, define the tool function named relpath, which convert a relative path to current file into a relative path to cwd
import os
relpath = lambda p: os.path.normpath(os.path.join(os.path.dirname(__file__), p))
Then we use it to wrap paths which is relative to current file
path1 = relpath('../src/main.py')
And you can also call sys.path.append() to import file relative to current file position
sys.path.append(relpath('..')) # so that you can import from upper dir
The full example code : https://gist.github.com/luochen1990/9b1ffa30f5c4a721dab5991e040e3eb1
Say the current archive named "Helper" and the upper directory named "Workshop", and the template files are in \Workshop\Templates, then the relative path in Python is "..\Templates".
This a simple way to add a relative path to the system path set . For example, for frequent case when the target directory is one level above (thus, '/../') the working directory:
import os
import sys
workingDir = os.getcwd()
targetDir = os.path.join(os.path.relpath(workingDir + '/../'),'target_directory')
sys.path.insert(0,targetDir)
This solution was tested for:
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021,
03:37:25) [MSC v.1916 64 bit (AMD64)]
I'm not sure if this applies to some of the older versions, but I believe Python 3.3 has native relative path support.
For example the following code should create a text file in the same folder as the python script:
open("text_file_name.txt", "w+t")
(note that there shouldn't be a forward or backslash at the beginning if it's a relative path)
Hi is there a function that copies a parent folder along with all its content to a specified destination in python.
I have used different functions but they seem to copy the contents excluding the parent folder.
Many thanks
shutil.copytree comes to mind immediately, but your issue is that copying directory foo in bar doesn't create bar/foo.
My proposal:
import shutil,os
def copytree2(source,dest):
os.mkdir(dest)
dest_dir = os.path.join(dest,os.path.basename(source))
shutil.copytree(source,dest_dir)
first create destination
then generate destination dir, which is the destination added source basename
perform copytree with the new destination, so source folder name level appears under dest
There's no subtle check about dest directory already exist or whatnot. I'll let you add that if needed (using os.path.isdir(dest) for instance)
Note that functions from shutil come with a note which encourages users to copy and modify them to better suit their needs.
In python 3.* we can use shutil
import shutil
old_folder = "D:/old_folder"
new_folder = "D:/new_folder"
shutil.copytree(old_folder, new_folder, dirs_exist_ok=True)
dirs_exist_ok=True is for ignoring the exception when the folder already exists in the new location.
Simply append the source directory you want to copy in the destination:
import shutil
shutil.copytree("source", "destination/source")
If you do not have fixed strings then use os.path.basename() to determine basename and combine it in the destination with os.path.join()
import os.path
import shutil
source = "/Projekte/python/source"
shutil.copytree(source, os.path.join("destination", os.path.basename(source)))
import shutil
shutil.copytree(srcDir, dst, symlinks=False, ignore=None)
I am trying to use shutils.py , make_archive function.
here: https://docs.python.org/2/library/shutil.html#archiving-operations
but I can't understand the difference between root_dir and base_dir.
Here's a simple code using make_archive:
#!user/bin/python
from os import path
from os import curdir
from shutil import make_archive
# Setting current Directory
current = path.realpath(curdir)
# Now Compressing
make_archive("Backup", "gztar", current)
This will create an archive named Backup.tar.gz which contains a . Directory inside it.
I don't want the . directory but the whole thing inside in the archive.
root_dir refers to base directory of output file, or working directory for your working script.
base_dir refers to content you want pack.
For example, if you have a directory tree like:
/home/apast/git/someproject
And you want to build a package for someproject folder, you can set:
root_dir="/home/apast/git"
base_dir="someproject"
If the contents of your tree is like following, for example:
/home/apast/git/someproject/test.py
/home/apast/git/someproject/model.py
The content of your package will acquire following structure:
someproject/test.py
someproject/model.py
And your package file will be stored at:
/home/apast/git/<packfile-name>
Like doc shows, by default, root_dir and base_dir are initialized for your current working directory (cwd, or curdir). But, you can use it in a more flexible way.
Let's consider following dir structure:
/home/apast/git/web/tornado.py
/home/apast/git/web/setup.py
/home/apast/git/core/service.py
/home/apast/git/mobile/gui.py
/home/apast/git/mobile/restfulapi.py
We will try two snippets to clarify examples:
1. Defining base_dir
2. Without defined base_dir
Defining base_dir, we specify which directory we will include on our file:
from shutil import make_archive
root_dir = "/home/apast/git/"
make_archive(base_name="/tmp/outputfile",
format="gztar",
root_dir=root_dir,
base_dir="web")
This code will generate a file called /tmp/outputfile.tar.gz with following struct:
web/tornado.py
web/setup.py
Running without base_dir, like following:
from shutil import make_archive
root_dir = "/home/apast/git/"
make_archive(base_name="/tmp/outputfile",
format="gztar",
root_dir=root_dir)
It will product a file containing:
web/tornado.py
web/setup.py
core/service.py
mobile/gui.py
mobile/restfulapi.py
To define specific folders, maybe it will be necessary use some other technique, like gzip lib directly.
cd in the root_dir... then tar the base_dir...
the docs makes me confuse too, read the code, that will make u clear.
It's a bit confusing if you read the documentation but if you see it visually, it can help quite a bit.
The root_dir is the directory that you will be storing the file in.
If I were storing a file in C:\Users\Elipzer\Desktop\MyFolder\, That would be my root_dir.
The base_dir is the part added onto the root_dir so if I were storing it under my MyFolder in ...\MyFolder\MySubFolder\, I would put that as the base_dir.
In many cases, there is no need to use these since you can just change the default directory to the directory that you want to store the file in and the make_archive function will just use the default directory as the root_dir and base_dir.
The data_files parameter for setup takes input in the following format:
setup(...
data_files = [(target_directory, [list of files to be put there])]
....)
Is there a way for me to specify an entire directory of data instead, so I don't have to name each file individually and update it as I change implementation in my project?
I attempted to use os.listdir(), but I don't know how to do that with relative paths, I couldn't use os.getcwd() or os.realpath(__file__) since those don't point to my repository root correctly.
karelv has the right idea, but to answer the stated question more directly:
from glob import glob
setup(
#...
data_files = [
('target_directory_1', glob('source_dir/*')), # files in source_dir only - not recursive
('target_directory_2', glob('nested_source_dir/**/*', recursive=True)), # includes sub-folders - recursive
# etc...
],
#...
)
import glob
for filename in glob.iglob('inner_dir/**/*', recursive=True):
print (filename)
Doing this, you get directly a list of files relative to current directory.
I don't know how to do that with relative paths
You need to get the path of the directory first, so...
Say you have this directory structure:
cur_directory
|- setup.py
|- inner_dir
|- file2.py
To get the directory of the current file (in this case setup.py), use this:
cur_directory_path = os.path.abspath(os.path.dirname(__file__))
Then, to get a directory path relative to current_directory, just join some other directories, eg:
inner_dir_path = os.path.join(cur_directory_path, 'inner_dir')
If you want to move up a directory, just use "..", for example:
parent_dir_path = os.path.join(current_directory, '..')
Once you have that path, you can do os.listdir
For completeness:
If you want the path of a file, in this case "file2.py" relative to setup.py, you could do:
file2_path = os.path.join(cur_directory_path, 'inner_dir', 'file2.py')
I ran into the same problem with directories containing nested subdirectories. The glob solutions didn't work, as they would include the directory in the list, which setup would blow up on, and in cases where I excluded matched directories, it still dumped them all in the same directory, which is not what I wanted, either. I ended up just falling back on os.walk():
def generate_data_files():
data_files = []
data_dirs = ('data', 'plugins')
for path, dirs, files in chain.from_iterable(os.walk(data_dir) for data_dir in data_dirs):
install_dir = os.path.join(sys.prefix, 'share/<my-app>/' + path)
list_entry = (install_dir, [os.path.join(path, f) for f in files if not f.startswith('.')])
data_files.append(list_entry)
return data_files
and then setting data_files=generate_data_files() in the setup() block.
With nested subdirectories, if you want to preserve the original directory structure, you can use os.walk(), as proposed in another answer.
However, an easier solution uses pbr library, which extends setuptools. See here for documentation on how to use it to install an entire directory structure:
https://docs.openstack.org/pbr/latest/user/using.html#files
I want to be able to write a script in python that will create a certain number of files/folders recursively to a certain location. So for example I want some sort of for loop that will create folder1, folder2, folder3 to a path C:\Temp. I assume a for loop is the best way to do this and using os also? Any help would be great!
Have a look at makedirs. It may help.
Here is how to create a folder recursively and then create a file in that folder.
from pathlib import Path
import os
folder_path = Path(os.getcwd() + os.path.join('\my\folders')) #define folder structure
if not os.path.exists(path): # create folders if not exists
os.makedirs(path)
file_path = os.path.join(path, 'file.xlsx') # add file to the folder path
f= open(file,"w+") # open in w+(write mode) or a+(append mode)
Since Python 3.4 you can use the pathlib to create a folder and all parent folders:
from pathlib import Path
Path("my/path/to/create").mkdir(parents=True, exist_ok=True)
If you want to raise an error on existent folder you can set exist_ok to False.