Let's say that I have a folder, x, containing the two folders foo and bar. Inside foo I have a Python script and inside bar, I have my data (let's say a couple of .txt files). Now, from within my script, I would like to access my data. I could simply use this command and call it a day:
fileList = glob.glob('/absolute/path/to/bar/*.txt')
However, using this method, my script would break if I moved my x folder.
What's the easiest way to specify the path in a relative manner that allows me to move around the parent folder, x, freely?
../ specifies your parent directory, so ../bar/*.txt.
Like John Gordon said, this works assuming you're inside x/foo/ and running the script from there.
If you're not executing the script from its own directory, you'll need to find out where your script currently is. This is not trivial. felipsmartins' solution of using os.path.dirname(os.path.dirname(__file__)) is generally fine, but doesn't work in all cases. See e.g. here: How to properly determine current script directory in Python?
I think __file__ is a nice approach:
import glob
import os
# "x" folder
base_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
files = glob.glob(os.path.join(base_dir, 'bar', '*.txt'))
Working from top directory is a lot easier. Django and other frameworks does it too.
Also, it solves issues when invoking script from command line and from different locations.
Another approch is usging os.chdir(), changing working directory:
import glob
import os
# same that: cd /absolute path/for/x folder
os.chdir(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
files = glob.glob(os.path.join('bar', '*.txt'))
Zend Framework (PHP) uses the above approach in bootstrap file.
Both solutions works very well for all cases.
Related
A typical situation for me is like this:
I have Python script which will during its run, open files from its dir.
In effect it will be something like.
actual_script_dir/
script.py
config.json
...
(somewhere in filesystem on my python path):
my_utilities/
module.py
...
#script.py
from my_utilities.module import Config
config_path='config.json'
conf=Config(file_path)
...
When I test it, I just usually run the file from its directory:
actual_scriptdir$ python3 script.py
and everything works.
However, the scripts are meant for crontab, which obviously has different directory when executing (/usr/bin):
0 3 * * * /usr/bin/python3 {actual_script_dir}/script.py
In which case I need to change config_path to absolute like:
config_path='{actual_script_dir}/config.json'
or it wil not be found.
The question is, can I somehow keep relative file paths in script.py regardless of working directory of execution? Ideally this would be part of Config module, which would know the actual path of the script it is loaded in.
I tried using __file__ and pathlib Path, but they either need to be used in script.py, which kinda defeats the purpose, or they show the my_utilities_module path, which is obviously not where config.json is located.
I would use the __file__ with pathlib.Path and then pass the full path into Config:
#script.py
from pathlib import Path
from my_utilities.module import Config
config_path='config.json'
THIS_DIR = Path(__file__).parent()
conf=Config(THIS_DIR / config_path)
I saw you mentioned in your original question that they "[...] need to be used in script.py, which kinda defeats the purpose" but I can't think where the problem might be (as per my comment).
EDIT:
Previous answer was missing .parent() which is required for the directory instead of the file.
While it doesn't really answer your question, I think it would be more flexible to accept the config path as the script's argument, instead of hard-coding it.
You could then specify both the script's path and the absolute path to the config in the CRON rule:
0 3 * * * /usr/bin/python3 {actual_script_dir}/script.py {actual_script_dir}/config.json
To read the argument, use sys.argc and sys.argv, or even better, the argparse module.
So, essentially I ended up passing __file__ to a special Config function.
class Config:
...
def this_dir(self,calling_file,json_file='config.json'):
self.script_dir=self.script_dir=os.path.dirname(Path(calling_file).absolute())
return os.path.join(self.script_dir,json_file)
...
Then, in calling program, I have to do:
conf=Config()
conf.load(conf.this_dir(__file__))
Inelegant, but the best I can do to avoid hardcoding paths when execing scripts both by hand and from crontab.
I'm writing a Python library which which you can load an object from a file and do something with it. For convenience, I'd like to make it so that people can provide three kinds of paths:
A path starting with a "/", which will be interpreted as an absolute path
A path starting with a "~/", which will be interpreted as relative to the user's home directory (for which I plan to use os.path.expanduser)
A path starting with neither, which would be interpreted as relative to the directory of the top-level script that is importing my library.
My question is: what is the best way of approaching the third case? The two ways I have come across on Stack Overflow are both kind of klugey:
1)
import __main__
if hasattr(__main__, "__file__"):
script_dir = os.path.dirname(os.path.abspath(__main__.__file__))
import inspect
frame_info = inspect.stack()[-1]
mod = inspect.getmodule(frame_info[0])
script_dir = os.path.dirname(mod.__file__)
Obviously both of these will fail in the case that we're running from an interactive terminal or something, and in that case I would want to fall back to just treating it as an absolute path.
Anyway, I get the sense that using inspect like this in a library is frowned upon, but the other way seems klugey as well. Is there a better way to do this that I'm unaware of?
Use the built-in package pathlib's getcwd method. It defaults to the root folder of the top package.
Will not work if you change the working directory. Although, an unprefixed path is common practice to be relative to the working directory.
import pathlib
print(pathlib.Path.cwd())
>>> A:\Programming\Python\generalfile
I have a Python module which uses some resources in a subdirectory of the module directory. After searching around on stack overflow and finding related answers, I managed to direct the module to the resources by using something like
import os
os.path.join(os.path.dirname(__file__), 'fonts/myfont.ttf')
This works fine when I call the module from elsewhere, but it breaks when I call the module after changing the current working directory. The problem is that the contents of __file__ are a relative path, which doesn't take into account the fact that I changed the directory:
>>> mymodule.__file__
'mymodule/__init__.pyc'
>>> os.chdir('..')
>>> mymodule.__file__
'mymodule/__init__.pyc'
How can I encode the absolute path in __file__, or barring that, how can I access my resources in the module no matter what the current working directory is? Thanks!
Store the absolute path to the module directory at the very beginning of the module:
package_directory = os.path.dirname(os.path.abspath(__file__))
Afterwards, load your resources based on this package_directory:
font_file = os.path.join(package_directory, 'fonts', 'myfont.ttf')
And after all, do not modify of process-wide resources like the current working directory. There is never a real need to change the working directory in a well-written program, consequently avoid os.chdir().
Building on lunaryorn's answer, I keep a function at the top of my modules in which I have to build multiple paths. This saves me repeated typing of joins.
def package_path(*paths, package_directory=os.path.dirname(os.path.abspath(__file__))):
return os.path.join(package_directory, *paths)
To build the path, call it like this:
font_file = package_path('fonts', 'myfont.ttf')
Or if you just need the package directory:
package_directory = package_path()
I've been working on a project that creates its own .py files that store handlers for the method, I've been trying to figure out how to store the Python files in folder and open them. Here is the code I'm using to create the files if they don't already exist, then importing the file:
if os.path.isfile("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py") == False:
TestPy = open("Btn"+ str(self.ButtonSet[self.IntBtnID].IntPID) +".py","w+")
try:
TestPy.write(StrHandler)
except Exception as Error:
print(Error)
TestPy.close()
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
self.IntBtnID += 1
when I change this line:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Btn" + str(self.IntBtnID))
to this:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons\\Btn" + str(self.IntBtnID))
the fill can't be found and ends up throwing an error because it can't find the file in the folder.
Do know why it doesn't work I just don't know how to get around the issue:/
My question is how do I open the .py when its stored in a folder?
There are a couple of unidiomatic things in your code that may be the cause of your issue. First of all, it is generally better to use the functions in os.path to manipulate paths to files. From your backslash usage, it appears you're working on Windows, but the os.path module ensures consistent behaviour across all platforms.
Also there is importlib.import_module, which is usually recommended over __import__. Furthermore, in case you want to load the generated module more than once during the lifetime of your program, you have to do that explicitly using imp.reload.
One last tip: I'd factor out the module path to avoid having to change it in more than one place.
You can't reference a path directory when you are importing files. Instead, you want to add the directory to your path and then import the name of the module.
import sys
sys.path.append( "Buttons" )
__import__("Btn"+str(self.IntBtnId))
See this so question for more information.
The first argument to the __import__() function is the name of the module, not a path to it. Therefore I think you need to use:
self.ButtonSet[self.IntBtnID].ImpHandler = __import__("Buttons.Btn" + str(self.IntBtnID))
You may also need to put an empty __init__.py file in the Buttons folder to indicate it's a package of modules.
My program is broken down into two parts: the engine, which deals with user interface and other "main program" stuff, and a set of plugins, which provide methods to deal with specific input.
Each plugin is written in its own module, and provides a function that will allow me to send and retrieve data to and from the plugin.
The name of this function is the same across all plugins, so all I need is to determine which one to call and then the plugin will handle the rest.
I've placed all of the plugins in a sub-folder, wrote an __ init__.py that imports each plugin, and then I import the folder (I think it's called a package?)
Anyways currently I explicitly tell it what to import (which is basically "import this", "import that"). Is there a way for me to write it so that it will import everything in that folder that is a plug-in so that I can add additional plugins without having to edit the init file?
Here is the code I use to do this:
def _loadPackagePlugins(package):
"Load plugins from a specified package."
ppath = package.__path__
pname = package.__name__ + "."
for importer, modname, ispkg in pkgutil.iter_modules(ppath, pname):
module = __import__(modname, fromlist = "dummy")
The main difference from Jakob's answer is that it uses pkgutil.iter_modules instead of os.listdir. I used to use os.listdir and changed to doing it this way, but I don't remember why. It might have been that os.listdir failed when I packaged my app with py2exe and py2app.
You could always have a dict called plugins, use __import__ to import the modules and store them that way.
e.g.
plugins = {}
for plugin in os.listdir('plugins'):
plugin = plugin.split()[0]
plugins[plugin] = __import__(plugin)
This is assuming that every plugin is a single file. Personally I would go with something that looks in each folder for a __run__.py file, like a __init__.py in a package it would indicate a plugin, that code would look more like something like this
for root, dirs, files in os.walk('.'):
for dir in dirs:
if "__run__.py" in os.listdir(os.path.join(root, dir)):
plugins[dir] = __import__(dir)
Code written without testing. YMMV