I originally had a Python module that was structured like this:
mymod
├── config.py
└── __init__.py
And the files contained this code:
# __init__.py
import mymod.config
mymod.config.foo()
# config.py
def foo():
print('here')
Running this with python3 -c "import mymod" will print "here", as expected.
Now as part of a refactor, the directory structure looks like this:
mymod
├── core
│ ├── config.py
│ └── __init__.py
└── __init__.py
I want the same initialization code in core/__init__.py to run when importing mymod, so the new root __init__.py looks like this:
# __init__.py
import mymod.core
And the other files just need to have the module paths fixed:
# core/__init__.py
import mymod.core.config
mymod.core.config.foo()
# core/config.py (no change)
def foo():
print('here')
But now I get an AttributeError relating to a partially initialized module:
$ python3 -c "import mymod"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/gsgx/code/mymod/__init__.py", line 1, in <module>
import mymod.core
File "/home/gsgx/code/mymod/core/__init__.py", line 3, in <module>
mymod.core.config.foo()
AttributeError: partially initialized module 'mymod' has no attribute 'core' (most likely due to a circular import)
I think I understand why: I'm trying to access mymod.core while in the middle of the original statement importing it. But I realized I could fix it by changing the import in config.py to use a from ... import instead:
# core/__init__.py
from mymod.core import config
config.foo()
My question is, while this technically fixes the issue, I was wondering if there's any other workaround for this. I don't want to have to go through all the imports and fix any name conflicts that result from doing this. It feels like I should be able to move a module into a subfolder without major refactoring and without running into partial initialization issues. I've read various answers on SO about Python imports, but wasn't able to find anything to help with this particular issue.
Related
I think I know what the issue is here but I don't know how to approach fixing it. This is the current structure I have in my project directory:
└── src
├── utils
└── utils.py
├── comparator
└── comparatorclass.py
In my comparatorclass.py I have the following import statements:
from utils.logging_service import LoggingService
from utils.utils import Utils
When I run comparatorclass.py, I get the error
Traceback (most recent call last):
File "comparator_gui.py", line 11, in <module>
import comparatorclass
File "/Users/sidu/Desktop/aaababa/src/comparator/comparatorclass.py", line 16, in <module>
from utils.logging_service import LoggingService
ModuleNotFoundError: No module named 'utils'
Of course, I'm pretty sure what's happening is that the comparatorclass.py is looking for the utils directory within the comparator directory when in fact it is one layer above that. The code works when I have a structure like this:
└── src
├── utils
└── utils.py
├── comparatorclass.py
This makes sense because now it is able to find the utils directory within the same parent, src. My question is how do I get it to work with the first scenario? I need a way to change what parent directory the import statement looks at.
I fixed the problem by adding the following line before importing the utilities:
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), '..'))
from utilities.logging_service import LoggingService
from utilities.comparator_utils import Utils
I don't know if this is the correct solution, but it did solve the problem for me.
Explanation
Use Case: Reference a top-level <module_name>.py module from a low-level (subfolder) <script_name>.py
I was under the impression one could import python scripts and their respective variables into other scripts within a project directory as long as they all had a main directory in common.
.
└── spam/ Main directory (git repo)
├── spam/ Main project directory
│ ├── __init__.py
│ ├── ham/ Directory within main project spam/
│ │ ├── __init__.py
│ │ └── eggs.py Script within sub directory ham/
│ └── config.py Config with contents eggs.py needs
└── README.md
(Note: Yes, the main directory and the main project directory have the same name.)
For example, I thought the above structure would allow this behavior: use contents from config with eggs.py
#! /spam/spam/ham/eggs.py
from spam import config
Well, not quite...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'spam'
So I took one step back.
#! /spam/spam/ham/eggs.py
import spam
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'spam'
I was wrong.
I am not so much looking for a solution at this point (since I found a few different ones including here and here (though the explanations did not really get through my thick skull). Would anyone like to explain it to me like I'm a 5th grader?
I will continue to collect what I consider to be valuable resources for finding the answers to my question. I appreciate all contributions that help clear up this picture.
Ongoing Discoveries
Exploratory Testing
Imports work for the above image.
# ../spam
# Direct import statements.
import spam
import config
# Indirect to get ham.
import spam.ham
from spam import ham
# Indirect to get eggs.
import spam.ham.eggs
from spam.ham import eggs
Imports work for the above image.
# ../spam/spam
# Direct import statements.
import ham
# Indirect to get eggs.
import ham.eggs
from ham import eggs
Imports work for the above image.
# ../spam/spam/ham
# Direct import statements.
import eggs
Observations
when doing import <module_name>, the current directory is checked but "backwards blind"
namespaces play a key role here
relative imports are only for use within module files (i.e., not within interactive interpreters)
Resources for Better Understanding
Stack Overflow "Relative imports for the billionth time
" Answer by BrenBarn
Stack Overflow Importing modules from parent folder Answer by np8*
*Solutions I test and find acceptable.
Tools and Versions
Python 3.8.6
A python module is not a folder, it's a python file (e.g. iamapythonfile.py)
Then your "spam" folder could not be resolved as a python module
you may use
import modulename.py
#or
from modulename.py import yourfunction
Then you should care about the path (absolute or relative)
More informations at : https://www.learnpython.org/en/Modules_and_Packages#:~:text=Modules%20in%20Python%20are%20simply,or%20variables%20defined%20and%20implemented.
EDIT :
i've just found this
https://realpython.com/absolute-vs-relative-python-imports/#:~:text=You%20need%20to%20have%20a,init__.py%20file).
, it's well explained and clear. Enjoy.
I'm on Python 3.7. I created a foo module, which has the following folder layout:
foo/
__init__.py
I put all the code for the foo package in the __init__.py file. I'm able to run import foo from other scripts as normal.
Later on, I wanted to add a util.py file to this package. The folder layout became:
foo/
__init__.py
util.py
My question is: Inside util.py, I'd like to access the classes, functions, etc. defined in __init__.py. How can I "import" __init__.py from the util.py file? Is this even possible? If I have to rearrange my code, then how?
I've tried to add the following to util.py:
from . import __init__
print(__init__.some_var_in_init)
But I get this error message:
Traceback (most recent call last):
File "util.py", line 3, in <module>
print(__init__.some_var_in_init)
AttributeError: 'method-wrapper' object has no attribute 'some_var_in_init'
So I'm guessing that's not the way to go about it.
import foo will work just fine.
I found the answer. It isn't enough to add import foo, but the parent folder must be added as well:
import os, sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import foo
print(foo.some_var_in_init)
I have read a ton of stackoverflow answers and a bunch of tutorials. In addition, I tried to read the Python documentation, but I cannot make this import work.
This is how the directory looks like:
myDirectory
├── __init__.py
├── LICENSE
├── project.py
├── README.md
├── stageManager.py
└── tests
├── __init__.py
└── test_project.py
There is a class in project.py called Project, and I want to import it in a file under tests directory. I have tried the following:
Relative import:
from ..project import Project
def print_sth():
print("something")
This gives me the following error: (running from the tests directory as python test_project.py and from myDirectory as python tests/test_project.py)
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from ..project import Project
SystemError: Parent module '' not loaded, cannot perform relative import
Absolute import with package name:
If I have something like the following, I get ImportError (with the same run command as above).
from project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from project import Project
ImportError: No module named 'project'
and this too:
from myDirectory.project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from myDirectory.project import Project
ImportError: No module named 'myDirectory'
Finally, I tried adding the if __name__ == '__main__' statement within the test_project.py file, but it still failed. I would really appreciate if anyone could help. If there is a solution where I do not have to write a verbose command, I would prefer that.
When you run a Python script by filename, the Python interpreter assumes that it is a top-level module (and it adds the directory the script is in to the module search path). If the script is in a package, that's not correct. Instead, you should run the module using the -m flag, which takes a module name in the same format as an import statement (dotted separators) and puts the current directory in the module search path.
So, you could run the test from myDirectory with: python -m tests.test_project. When you run the script this way, either of the kinds of imports you tried will work.
But if myDirectory is supposed to be a top-level package itself (as the __init__.py file suggests), you should probably go up one level further up, to myDirectory's parent, and run the script with two levels of package names: python -m myDirectory.tests.test_project. If you do this and want the test to use an absolute import you'd need to name the top level package that the project module is in: from myDirectory.project import Project.
Assume I have the following structure of a package:
parent/
package/
__init__.py
utils/
__init__.py
foo.py
apps/
__init__.py
bar.py
Now, I want to import the module foo from the module bar. What is the correct way to do it? I will welcome comments on the structure of my project :)
So far I only found answers where everything lives in the same directory... I tried something like:
from ..utils import foo
but I get:
Traceback (most recent call last):
File "./bar.py", line 4, in <module>
from ..utils import foo
ValueError: Attempted relative import in non-package
I think, it's better to set some environ variable or extend PYTHONPATH (it must be done in main script)