I want to inherit from a class in a file that lies in a directory above the current one.
Is it possible to relatively import that file?
from ..subpkg2 import mod
Per the Python docs: When inside a package hierarchy, use two dots, as the import statement doc says:
When specifying what module to import you do not have to specify the absolute name of the module. When a module or package is contained within another package it is possible to make a relative import within the same top package without having to mention the package name. By using leading dots in the specified module or package after from you can specify how high to traverse up the current package hierarchy without specifying exact names. One leading dot means the current package where the module making the import exists. Two dots means up one package level. Three dots is up two levels, etc. So if you execute from . import mod from a module in the pkg package then you will end up importing pkg.mod. If you execute from ..subpkg2 import mod from within pkg.subpkg1 you will import pkg.subpkg2.mod. The specification for relative imports is contained within PEP 328.
PEP 328 deals with absolute/relative imports.
import sys
sys.path.append("..") # Adds higher directory to python modules path.
#gimel's answer is correct if you can guarantee the package hierarchy he mentions. If you can't -- if your real need is as you expressed it, exclusively tied to directories and without any necessary relationship to packaging -- then you need to work on __file__ to find out the parent directory (a couple of os.path.dirname calls will do;-), then (if that directory is not already on sys.path) prepend temporarily insert said dir at the very start of sys.path, __import__, remove said dir again -- messy work indeed, but, "when you must, you must" (and Pyhon strives to never stop the programmer from doing what must be done -- just like the ISO C standard says in the "Spirit of C" section in its preface!-).
Here is an example that may work for you:
import sys
import os.path
sys.path.append(
os.path.abspath(os.path.join(os.path.dirname(__file__), os.path.pardir)))
import module_in_parent_dir
Import module from a directory which is exactly one level above the current directory:
from .. import module
How to load a module that is a directory up
preface: I did a substantial rewrite of a previous answer with the hopes of helping ease people into python's ecosystem, and hopefully give everyone the best change of success with python's import system.
This will cover relative imports within a package, which I think is the most probable case to OP's question.
Python is a modular system
This is why we write import foo to load a module "foo" from the root namespace, instead of writing:
foo = dict(); # please avoid doing this
with open(os.path.join(os.path.dirname(__file__), '../foo.py') as foo_fh: # please avoid doing this
exec(compile(foo_fh.read(), 'foo.py', 'exec'), foo) # please avoid doing this
Python isn't coupled to a file-system
This is why we can embed python in environment where there isn't a defacto filesystem without providing a virtual one, such as Jython.
Being decoupled from a filesystem lets imports be flexible, this design allows for things like imports from archive/zip files, import singletons, bytecode caching, cffi extensions, even remote code definition loading.
So if imports are not coupled to a filesystem what does "one directory up" mean? We have to pick out some heuristics but we can do that, for example when working within a package, some heuristics have already been defined that makes relative imports like .foo and ..foo work within the same package. Cool!
If you sincerely want to couple your source code loading patterns to a filesystem, you can do that. You'll have to choose your own heuristics, and use some kind of importing machinery, I recommend importlib
Python's importlib example looks something like so:
import importlib.util
import sys
# For illustrative purposes.
file_path = os.path.join(os.path.dirname(__file__), '../foo.py')
module_name = 'foo'
foo_spec = importlib.util.spec_from_file_location(module_name, file_path)
# foo_spec is a ModuleSpec specifying a SourceFileLoader
foo_module = importlib.util.module_from_spec(foo_spec)
sys.modules[module_name] = foo_module
foo_spec.loader.exec_module(foo_module)
foo = sys.modules[module_name]
# foo is the sys.modules['foo'] singleton
Packaging
There is a great example project available officially here: https://github.com/pypa/sampleproject
A python package is a collection of information about your source code, that can inform other tools how to copy your source code to other computers, and how to integrate your source code into that system's path so that import foo works for other computers (regardless of interpreter, host operating system, etc)
Directory Structure
Lets have a package name foo, in some directory (preferably an empty directory).
some_directory/
foo.py # `if __name__ == "__main__":` lives here
My preference is to create setup.py as sibling to foo.py, because it makes writing the setup.py file simpler, however you can write configuration to change/redirect everything setuptools does by default if you like; for example putting foo.py under a "src/" directory is somewhat popular, not covered here.
some_directory/
foo.py
setup.py
.
#!/usr/bin/env python3
# setup.py
import setuptools
setuptools.setup(
name="foo",
...
py_modules=['foo'],
)
.
python3 -m pip install --editable ./ # or path/to/some_directory/
"editable" aka -e will yet-again redirect the importing machinery to load the source files in this directory, instead copying the current exact files to the installing-environment's library. This can also cause behavioral differences on a developer's machine, be sure to test your code!
There are tools other than pip, however I'd recommend pip be the introductory one :)
I also like to make foo a "package" (a directory containing __init__.py) instead of a module (a single ".py" file), both "packages" and "modules" can be loaded into the root namespace, modules allow for nested namespaces, which is helpful if we want to have a "relative one directory up" import.
some_directory/
foo/
__init__.py
setup.py
.
#!/usr/bin/env python3
# setup.py
import setuptools
setuptools.setup(
name="foo",
...
packages=['foo'],
)
I also like to make a foo/__main__.py, this allows python to execute the package as a module, eg python3 -m foo will execute foo/__main__.py as __main__.
some_directory/
foo/
__init__.py
__main__.py # `if __name__ == "__main__":` lives here, `def main():` too!
setup.py
.
#!/usr/bin/env python3
# setup.py
import setuptools
setuptools.setup(
name="foo",
...
packages=['foo'],
...
entry_points={
'console_scripts': [
# "foo" will be added to the installing-environment's text mode shell, eg `bash -c foo`
'foo=foo.__main__:main',
]
},
)
Lets flesh this out with some more modules:
Basically, you can have a directory structure like so:
some_directory/
bar.py # `import bar`
foo/
__init__.py # `import foo`
__main__.py
baz.py # `import foo.baz
spam/
__init__.py # `import foo.spam`
eggs.py # `import foo.spam.eggs`
setup.py
setup.py conventionally holds metadata information about the source code within, such as:
what dependencies are needed to install named "install_requires"
what name should be used for package management (install/uninstall "name"), I suggest this match your primary python package name in our case foo, though substituting underscores for hyphens is popular
licensing information
maturity tags (alpha/beta/etc),
audience tags (for developers, for machine learning, etc),
single-page documentation content (like a README),
shell names (names you type at user shell like bash, or names you find in a graphical user shell like a start menu),
a list of python modules this package will install (and uninstall)
a defacto "run tests" entry point python ./setup.py test
Its very expansive, it can even compile c extensions on the fly if a source module is being installed on a development machine. For a every-day example I recommend the PYPA Sample Repository's setup.py
If you are releasing a build artifact, eg a copy of the code that is meant to run nearly identical computers, a requirements.txt file is a popular way to snapshot exact dependency information, where "install_requires" is a good way to capture minimum and maximum compatible versions. However, given that the target machines are nearly identical anyway, I highly recommend creating a tarball of an entire python prefix. This can be tricky, too detailed to get into here. Check out pip install's --target option, or virtualenv aka venv for leads.
back to the example
how to import a file one directory up:
From foo/spam/eggs.py, if we wanted code from foo/baz we could ask for it by its absolute namespace:
import foo.baz
If we wanted to reserve capability to move eggs.py into some other directory in the future with some other relative baz implementation, we could use a relative import like:
import ..baz
Here's a three-step, somewhat minimalist version of ThorSummoner's answer for the sake of clarity. It doesn't quite do what I want (I'll explain at the bottom), but it works okay.
Step 1: Make directory and setup.py
filepath_to/project_name/
setup.py
In setup.py, write:
import setuptools
setuptools.setup(name='project_name')
Step 2: Install this directory as a package
Run this code in console:
python -m pip install --editable filepath_to/project_name
Instead of python, you may need to use python3 or something, depending on how your python is installed. Also, you can use -e instead of --editable.
Now, your directory will look more or less like this. I don't know what the egg stuff is.
filepath_to/project_name/
setup.py
test_3.egg-info/
dependency_links.txt
PKG-INFO
SOURCES.txt
top_level.txt
This folder is considered a python package and you can import from files in this parent directory even if you're writing a script anywhere else on your computer.
Step 3. Import from above
Let's say you make two files, one in your project's main directory and another in a sub directory. It'll look like this:
filepath_to/project_name/
top_level_file.py
subdirectory/
subfile.py
setup.py |
test_3.egg-info/ |----- Ignore these guys
... |
Now, if top_level_file.py looks like this:
x = 1
Then I can import it from subfile.py, or really any other file anywhere else on your computer.
# subfile.py OR some_other_python_file_somewhere_else.py
import random # This is a standard package that can be imported anywhere.
import top_level_file # Now, top_level_file.py works similarly.
print(top_level_file.x)
This is different than what I was looking for: I hoped python had a one-line way to import from a file above. Instead, I have to treat the script like a module, do a bunch of boilerplate, and install it globally for the entire python installation to have access to it. It's overkill. If anyone has a simpler method than doesn't involve the above process or importlib shenanigans, please let me know.
Polished answer of #alex-martelli with pathlib:
import pathlib
import sys
_parentdir = pathlib.Path(__file__).parent.parent.resolve()
sys.path.insert(0, str(_parentdir))
import module_in_parent_dir
sys.path.remove(str(_parentdir))
To run python /myprogram/submodule/mymodule.py which imports /myprogram/mainmodule.py, e.g., via
from mainmodule import *
on Linux (e.g., in the python Docker image), I had to add the program root directory to PYTHONPATH:
export PYTHONPATH=/myprogram
It is 2022 and none of the answers really worked for me. Here is what worked in the end
import sys
sys.path.append('../my_class')
import my_class
My directory structure:
src
--my_class.py
notebooks
-- mynotebook.ipynb
I imported my_class from mynotebook.ipynb.
You can use the sys.path.append() method to add the directory containing the package to the list of paths searched for modules. For example, if the package is located two directories above the current directory, you can use the following code:
import sys
sys.path.append("../../")
if the package is location one directory above the current directory, you can use below code:
import sys
sys.path.append("..")
Python is a modular system
Python doesn't rely on a file system
To load python code reliably, have that code in a module, and that module installed in python's library.
Installed modules can always be loaded from the top level namespace with import <name>
There is a great sample project available officially here: https://github.com/pypa/sampleproject
Basically, you can have a directory structure like so:
the_foo_project/
setup.py
bar.py # `import bar`
foo/
__init__.py # `import foo`
baz.py # `import foo.baz`
faz/ # `import foo.faz`
__init__.py
daz.py # `import foo.faz.daz` ... etc.
.
Be sure to declare your setuptools.setup() in setup.py,
official example: https://github.com/pypa/sampleproject/blob/master/setup.py
In our case we probably want to export bar.py and foo/__init__.py, my brief example:
setup.py
#!/usr/bin/env python3
import setuptools
setuptools.setup(
...
py_modules=['bar'],
packages=['foo'],
...
entry_points={},
# Note, any changes to your setup.py, like adding to `packages`, or
# changing `entry_points` will require the module to be reinstalled;
# `python3 -m pip install --upgrade --editable ./the_foo_project
)
.
Now we can install our module into the python library;
with pip, you can install the_foo_project into your python library in edit mode,
so we can work on it in real time
python3 -m pip install --editable=./the_foo_project
# if you get a permission error, you can always use
# `pip ... --user` to install in your user python library
.
Now from any python context, we can load our shared py_modules and packages
foo_script.py
#!/usr/bin/env python3
import bar
import foo
print(dir(bar))
print(dir(foo))
I have a repo with Python code whose structure could be boiled down to this:
repo_root\
tool1\
tool1.py
tool1_aux_stuff.py
tool2\
tool2.py
tool2_aux_stuff.py
lib\
lib1\
lib1.py
lib1_aux_stuff.py
lib2\
lib2.py
lib2_aux_stuff.py
The following rules apply to the module usage:
Any tool could use the modules from any library and from its own package, but not from a different tool's one.
Any library could use the modules from any other library, and from its own package. Libraries never access the tool modules.
There must be a way to invoke any tool from any working directory, including those outside repo_root.
The question is: how do I import the lib modules from the tool ones?
I know that if I add __init__.py to each tool and lib directory and to the repo root, then I would be able to use absolute paths from the root, i.e. in tool1.py I could write
import lib.lib1, lib.lib2.lib2_aux_stuff
However, if I execute tool1.py from a random place, e.g.
machine_name: ~/random/place$ python /path/to/repo/tool1/tool1.py
I get the ModuleNotFoundError: No module named 'lib' found error.
I am aware of a workaround which could be implemented using the PYTHONPATH env variable by augmenting it with an absolute path to repo_root and supplying it to the invocation of the tool script, i.e.:
machine_name: ~/random/place$ PYTHONPATH=$PYTHONPATH:/path/to/repo python /path/to/repo/tool1/tool1.py
but I would really prefer something less clunky.
Any ideas how I could do it in a more straightforward way?
Add the path to lib to the scope using sys.path.append('/custom/path/to/modules'). It should then be callable as a module.
You do need to add __init__.py files in any directory that you want to import as if it were a module, otherwise Python doesn't treat them as modules and you'll get another ImportError
I have some doubts in relation to packages structure in a python project when I make the imports
These are some conventions
python-irodsclient_API = Project Name
I've defined python packages for each file, in this case are the following:
python-irodsclient_API/config/
python-irodsclient_API/connection/
These packages are well define as a packages and not as a directories really?
I have the file python-irodsclient_API/config/config.py in which I've defined some constants about of configuration for connect with my server:
And I have the python-irodsclient_API/connection/connection.py file:
In the last or previous image (highlighted in red) .. is this the right way of import the files?
I feel the sensation of this way is not better.
I know that the "imports" should be relatives and not absolutes (for the path) and that is necessary use "." instead "*"
In my case I don't know if this can be applied in relation to the I'm doing in the graphics.
I appreciate your help and orientation
Best Regards
There is a good tutorial about this in the Python module docs, which explains how to refer to packages under structured folders.
Basically, from x import y, where y is a submodule name, allows you to use y.z instead of x.y.z.
You have 2 options here:
1) make your project a package. Since it seems like your connection and config packages are interdependent, they should be modules within the same package. To make this happens, add a __init__.py files in python-irodsclient_API folder. Now you can use relative imports to import config into connection, as they are part of the same package:
from ..config import config
The .. part means import from one level above within the package structure (similar to how .. means parent directory in Unix)
2) if you don't want to make python-irodsclient_API a package for some reason, then the second option is to add that folder to the PYTHONPATH. You can do this dynamically per Tony Yang's answer, or do this from the bash command line as followed:
export PYTHONPATH=$PYTHONPATH:/path/to/python-irodsclient_API
I can invoke sys module to append python-irodsclient_API path.
import sys
sys.path.append('C:\..\python-irodsclient_API')
When you operate connection.py and want to invoke config, it's able to be successful.
I am writing a simple scheduling service. I don't want to hard-code all of the tasks it can schedule and instead would like to support plugins that can be dropped in a folder and loaded dynamically at runtime.
My plan is to have a JSON file (or any configuration file) that maps a task name to the location of a Python file (a module) which will have a class called Plugin. Pretty simple I thought. When someone schedules a task to run, they pass the task name and the time to run it. When the time elapses, the plugin is loaded (or reloaded) and is ran with any additional arguments passed to the scheduler.
I have been looking at the imp module to see how to load modules at runtime. I am not sure whether I want to list plugins using their physical location (file system path) or to use their module names like you'd see in a import statement. It seems imp wants to use physical location.
I got two different versions of this code "working". Here is one that uses importlib:
pluginName = self.__pluginLookup[pluginName]
module = import_module(pluginName)
module = reload(module) # force reload
plugin = module.Plugin()
return plugin
This is one I wrote using imp:
path = self.__pluginLookup[pluginName]
path, moduleName = split(path)
moduleName, extension = splitext(moduleName)
file, path, description = find_module(moduleName, [path])
with file:
module = load_module(moduleName, file, path, description)
plugin = module.Plugin()
return plugin
The problem I am running into is handling dependencies. If I have a plugin.py file that depends on a dependency.py file in the same folder, saying import dependency doesn't seem to work. Instead, it looks up the dependency from the PYTHONPATH.
How can I make the imports relative to the plugins themselves?
You could append path to sys.path:
import sys
sys.path.append(path)
where path is the directory containing the dependency.py.
If you have a plugins directory with an __init__.py, you can add that directory to sys.path. Then modules inside there can do from . import dependency to import another plugin. Or, if the plugin is itself a subpackage (i.e., a directory with its own __init__.py) then it can do from . import dep to import a dependency within the same plugin, or from .. import dep to import a dependency from the global plugins directory. With this setup you don't even need to use imp or the like; you can use the __import__ function, which works by module name.
One possible wrinkle, though, is you say the plugins directory will be "under the current working directory". What does that mean exactly? You mean you want people to be able to add plugins anywhere on the drive and still use them? It would be better to have one central plugins directory for your app, and add that to sys.path, and tell people to put their plugins there.
I have in my python workspace two Modules which need sip.pyd
Module1.pyd needs sip.pyd (which implements v 8.0-8.1)
Module2.pyd needs sip.pyd (another file, that implements v6.0)
So I can't just choose the newer one, it doesn't work: I have to keep them both!
(RuntimeError: the sip module implements API v6.0 but the fbx module requires API v8.1)
How can I import a module in .pyd extension (a python dll, not editable), and specify which sip.pyd to source?
As for a workaround, I manage to do that:
One sip.pyd is in my root site-packages location.
If I have to import the module that need the other sip.pyd, I remove root path form sys.path, and I append the precise folder path where the other sip.pyd are.
I can import my Module and restore previous sys.path.
Assuming you don't have a piece of code needing both files at once. I'd recommend the following:
install both files in 2 separate directories (call them e.g. sip-6.0 and sip-8.0), that you'll place in site-packages/
write a sip_helper.py file with code looking like
sip_helper.py contents:
import sys
import re
from os.path import join, dirname
def install_sip(version='6.0'):
assert version in ('6.0', '8.0'), "unsupported version"
keep = []
if 'sip' in sys.modules:
del sys.modules['sip']
for path in sys.path:
if not re.match('.*sip\d\.\d', path):
keep.append(path)
sys.path[:] = keep # remove other paths
sys.path.append(join(dirname(__file__), 'sip-%s' % version))
put sip_helper.py in site_packages (the parent directory of the sip-6.0 and sip-8.0 directories)
call sip_helper.install_sip at the startup of your programs
VirtualEnv is done to handle those case.
virtualenv is a tool to create isolated Python environments.
Using virtualenv, you will be able to create 2 environements, one with the sip.pyd in version 8.x another in version 6.0
I don't know if that works (if a module's name has to match its contents), but can't you just rename them to sip6.pyd resp. sip8.pyd and then do
if need6:
import sip6 as sip
else:
import sip8 as sip
?