Yet another python imports question.
I have a directory structure
thing
|- __init__.py
|- run.py
|- mod
|- __init__.py
|- what
|- __init__.py
|- yo.py
The contents of yo.py are
class Yo:
def __init__(self):
print("initialized What")
And the contents of my run.py are
from mod.what import yo
y = yo.Yo
print(y)
y()
Everything works great.
<class 'mod.what.yo.Yo'>
initialized What
But I need to import like this:
from mod import what
y = what.yo.Yo
print(y)
y()
and then I get
Traceback (most recent call last):
File "/Users/pavelkomarov/Desktop/thing/run.py", line 4, in <module>
y = what.yo.Yo
AttributeError: module 'mod.what' has no attribute 'yo'
Why? I've got __init__.pys out the wazoo. Shouldn't python be able to find the class? This structure is important to me because I have a lot of classes below a certain module and need to be able to access them all, preferably without having to do more granular imports of each one, which takes a ton of code.
Thanks to Maurice and the people over at that other thread.
I've found putting
import importlib
import pkgutil
for mod_info in pkgutil.walk_packages(__path__, __name__ + '.'):
mod = importlib.import_module(mod_info.name)
# Emulate `from mod import *`
try:
names = mod.__dict__['__all__']
except KeyError:
names = [k for k in mod.__dict__ if not k.startswith('_')]
in mod's __init__.py does the trick. It also works if you put it in what's __init__.py.
I disagree that this is anything close to Zen, but it works.
Related
This is the structure of my project at the moment.
classes/
├─ __init__.py
├─ card.py
├─ find.py
├─ player.py
├─ table.py
.gitignore
main.py
My __init__.py:
from .card import *
from .player import *
from .table import *
The problem is when I want to import from example player to table.py,
like this:
from classes import Player
from classes import Card, CardType, CardValue, HandValue
I get the following error message:
Traceback (most recent call last):
File "/home/adrian/Desktop/Programozas/Poker/classes/table.py", line 1, in <module>
from classes import Player
ModuleNotFoundError: No module named 'classes'
Do you have any idea?
Notice that this error is occurring in ".../classes/table.py", line 1, as you are trying to import classes.Player when both table.py and player.py are contained in classes. To do this in table.py, you would use from .player import Player just like you did in __init__.py (instead of from classes import Player).
Anything at the same level or above classes can be done using this method, but anything defined inside classes cannot call up to classes, as anything therein does not even know classes exists.
Edit: if you need to be able to run both main.py and table.py, you could do something like:
# in table.py
try:
from player import Player # this should work when running table.py directly
except:
from .player import Player # this should work when running main.py (it's a relative import)
I want to import constants from a constants module from two different modules, but I get the following error:
Traceback (most recent call last):
File "C:\Temp\tmp\pycircular\pycircular\pycircular.py", line 2, in <module>
from my_classes.foo import Foo
File "C:\Temp\tmp\pycircular\pycircular\my_classes\foo.py", line 1, in <module>
from pycircular.constants import ANOTHER_CONSTANT
File "C:\Temp\tmp\pycircular\pycircular\pycircular.py", line 2, in <module>
from my_classes.foo import Foo
ImportError: cannot import name 'Foo' from partially initialized module 'my_classes.foo' (most likely due to a circular import) (C:\Temp\tmp\pycircular\pycircular\my_classes\foo.py)
My project structure is the following:
|-constants.py
|-my_classes
| |-foo.py
| |-__init__.py
|-pycircular.py
|-__init__.py
# =============
# pycircular.py
# =============
from constants import SOME_CONSTANT
from my_classes.foo import Foo
def main():
print(SOME_CONSTANT)
my_foo = Foo()
my_foo.do_something()
if __name__ == "__main__":
main()
# =============
# foo.py
# =============
from pycircular.constants import ANOTHER_CONSTANT
class Foo:
def do_something(self):
print(ANOTHER_CONSTANT)
# =============
# constants.py
# =============
ANOTHER_CONSTANT = "ANOTHER"
SOME_CONSTANT = "CONSTANT"
I assume that it is the same problem as solved here https://stackoverflow.com/a/62303448/2021763.
But I really do not get why from my_classes.foo import Foo in pycircular.py is called a second time.
Update:
After renaming the package pycircular to pycircular_pack it worked in PyCharm.
But it only works because in Pycharm the option Add content roots to to PYTHONPATH is automatically set.
The output of sys.path is ['C:\\Temp\\tmp\\pycircular\\pycircular_pack', 'C:\\Temp\\tmp\\pycircular', 'C:\\Tools\\miniconda\\envs\\my_env\\python39.zip', 'C:\\Tools\\miniconda\\envs\\my_env\\DLLs', 'C:\\Tools\\miniconda\\envs\\my_env\\lib', 'C:\\Tools\\miniconda\\envs\\my_env', 'C:\\Tools\\miniconda\\envs\\my_env\\lib\\site-packages']
Without the option the output is ['C:\\Temp\\tmp\\pycircular\\pycircular_pack', 'C:\\Tools\\miniconda\\envs\\my_env\\python39.zip', 'C:\\Tools\\miniconda\\envs\\my_env\\DLLs', 'C:\\Tools\\miniconda\\envs\\my_env\\lib', 'C:\\Tools\\miniconda\\envs\\my_env', 'C:\\Tools\\miniconda\\envs\\my_env\\lib\\site-packages']
And without the option I only get it to work with absolute imports.
# pycircular.py
from constants import SOME_CONSTANT
from my_classes.foo import Foo
...
# foo.py
from constants import ANOTHER_CONSTANT
To elaborate based on the comments and edit:
After renaming the package pycircular to pycircular_pack it worked in PyCharm. But it only works because in Pycharm the option Add content roots to to PYTHONPATH is automatically set.
You should make sure the package directory is not set as a content root or source root. The directory hosting the package directory should be set as source root.
C:\Temp\tmp\pycircular # <- source root
|- pycircular_pack # <- not set as anything
| |- constants.py
| |- my_classes
| | |- foo.py
| | |- __init__.py
| |- pycircular.py
| |- __init__.py
|- other_file.py # <- for illustration's sake
Now your sys.path will be set to include C:\Temp\tmp\pycircular only and there will be exactly one way to import things from your module.
Namely,
other_file.py (outside the package) will be able to use the package as pycircular_pack
pycircular_pack/*.py can refer to modules in the pycircular_pack package by either
(e.g.) from .constants import ... (relative import from current package), or
(e.g.) from pycircular_pack.constants import ... (absolute import)
pycircular_pack/my_classes/*.py can refer to modules in the pycircular_pack package by either
(e.g.) from ..constants import ... (relative import from parent package), or
(e.g.) from pycircular_pack.constants import ... (absolute import)
If your pycircular_pack package would contain a runnable script, e.g. a CLI as pycircular_pack/cli.py, then the correct way to run that script on the command line would be to use python -m pycircular_pack.cli; this has Python set up the path just like we want here, where python pycircular_pack/cli.py would not do the right thing.
Consider the following package structure:
.
├── module
│ ├── __init__.py
│ └── submodule
│ ├── attribute.py
│ ├── data.txt
│ └── __init__.py
└── test.py
and the following piece of code:
import pkgutil
data = pkgutil.get_data('module.submodule', 'data.txt')
import module.submodule.attribute
retval = module.submodule.attribute.hello()
Running this will raise the error:
Traceback (most recent call last):
File "test.py", line 7, in <module>
retval = module.submodule.attribute.hello()
AttributeError: module 'module' has no attribute 'submodule'
However, if you run the following:
import pkgutil
import module.submodule.attribute
data = pkgutil.get_data('module.submodule', 'data.txt')
retval = module.submodule.attribute.hello()
or
import pkgutil
import module.submodule.attribute
retval = module.submodule.attribute.hello()
it works fine.
Why does running pkgutil.get_data disrupt the future import?
First of all, this was a great question and a great opportunity to learn something new about python's import system. So let's dig in!
If we look at the implementation of pkgutil.get_data we see something like this:
def get_data(package, resource):
spec = importlib.util.find_spec(package)
if spec is None:
return None
loader = spec.loader
if loader is None or not hasattr(loader, 'get_data'):
return None
# XXX needs test
mod = (sys.modules.get(package) or
importlib._bootstrap._load(spec))
if mod is None or not hasattr(mod, '__file__'):
return None
# Modify the resource name to be compatible with the loader.get_data
# signature - an os.path format "filename" starting with the dirname of
# the package's __file__
parts = resource.split('/')
parts.insert(0, os.path.dirname(mod.__file__))
resource_name = os.path.join(*parts)
return loader.get_data(resource_name)
And the answer to your question is in this part of the code:
mod = (sys.modules.get(package) or
importlib._bootstrap._load(spec))
It looks at the already loaded packages and if the package we're looking for (module.submodule in this example) exists it uses it and if not, then tries to load the package using importlib._bootstrap._load.
So let's look at the implementation of importlib._bootstrap._load to see what's going on.
def _load(spec):
"""Return a new module object, loaded by the spec's loader.
The module is not added to its parent.
If a module is already in sys.modules, that existing module gets
clobbered.
"""
with _ModuleLockManager(spec.name):
return _load_unlocked(spec)
Well, There's right there! The doc says "The module is not added to its parent."
It means the submodule module is loaded but it's not added to the module module. So when we try to access the submodule via module there's no connection, hence the AtrributeError.
It makes sense for the get_data method to use this function as it just wants some other file in the package and there is no need to import the whole package and add it to its parent and its parents' parent and so on.
to see it yourself I suggest using a debugger and setting some breakpoints. Then you can see what happens step by step along the way.
I am trying to add 'project_root' into __init__.py and all modules can use it, but it doesn't work.
Environment: Python 3.7.0 MACOS MOJAVE
file structure
·
├── __init__.py
└── a.py
the codes in __init__.py file:
import sys
project_root = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
sys.path.append(project_root)
and in another file
print(project_root)
If I run python a.py in the same dir ,or run python a.py out of the dir, error are the same below:
Traceback (most recent call last):
File "a.py", line 1, in <module>
print(project_root)
NameError: name 'project_root' is not defined
My question is why it doesn't work and how to fix it.
Another question is what if you want to share some variables for other modules in a same package, how to do it?
Let us try to understand by example.
Code and directory explanation:
Suppose we have the following directory and file structure:
dir_1
├── __init__.py
└── a.py
b.py
__init__.py contains:
import sys,os
# Just to make things clear
print("Print statement from init")
project_root = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
sys.path.append(project_root)
a.py contains:
def func():
print("func from a.py")
Let us start importing things:
Suppose you start with having the following code in b.py:
from dir_1.a import func
func()
Executing the above will give the following output:
Print statement from init
func from a.py
So, from the above, we understand that the print statement from __init__.py is being executed. Now, let's add print(project_root) in b.py:
from dir_1.a import func
func()
print(project_root)
Executing the above will result in an error saying:
...
NameError: name 'project_root' is not defined
This happened because we did not have to import the print statement from __init__.py it just gets executed. But that is not the case for a variable.
Let's try to import the variable and see what happens:
from dir_1.a import func
from dir_1 import project_root
func()
print(project_root)
Executing the above file will give the following output:
Print statement from init
func from a.py
/home/user/some/directory/name/dir_1
Long story short, you need to import the variable defined in __init__.py
or anywhere else in order to use it.
Hope this helps : )
You have to import the variable:
from dir import project_root
Where dir is the directory you are in
I'm trying to build a library with multiple sub-modules. The general structure looks like so:
package_test/
|___ setup.py
|___ README
|___ package_name/
|___|___ __init__.py
|___|___ submodule/
|___|___|___ __init__.py
|___|___|___ superawesome.py
|___|___ submodule2/
|___|___|___ __init__.py
|___|___|___ prettyokay.py
The goal is to allow a user to do something like:
from package_name.submodule import superawesome
from package_name.submodule2 import prettyokay
The __init__.py for each submodule currently contains (or the appropriate name for submodule2):
from .superawesome import superawesome
__all__ = ['superawesome']
The __init__.py for inside package_name contains:
__version__ = 'v0.0.alpha'
__all__ = ['submodule','submodule2']
When I attempt to use this, iPython shows this error:
From package_name.submodule import superawesome
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-1-03be4241edd5> in <module>()
----> 1 from package_name.submodule import superawesome
ModuleNotFoundError: No module named 'package_name.submodule'
My setup.py currently contains:
from distutils.core import setup
setup(
name = 'package_name',
packages = ['package_name'], # this must be the same as the name above
version = '0.1.1',
long_description=open('README').read(),
)
I need help figuring out how to do the layered importing structure in order to properly import the submodules as part of the whole module. This is clearly a contrived naming set, but in the real problem, each submodule contains a set of modules that belong together... but all of the submodules together are necessary for the library; so I really want to maintain this structure.