I have read a ton of stackoverflow answers and a bunch of tutorials. In addition, I tried to read the Python documentation, but I cannot make this import work.
This is how the directory looks like:
myDirectory
├── __init__.py
├── LICENSE
├── project.py
├── README.md
├── stageManager.py
└── tests
├── __init__.py
└── test_project.py
There is a class in project.py called Project, and I want to import it in a file under tests directory. I have tried the following:
Relative import:
from ..project import Project
def print_sth():
print("something")
This gives me the following error: (running from the tests directory as python test_project.py and from myDirectory as python tests/test_project.py)
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from ..project import Project
SystemError: Parent module '' not loaded, cannot perform relative import
Absolute import with package name:
If I have something like the following, I get ImportError (with the same run command as above).
from project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from project import Project
ImportError: No module named 'project'
and this too:
from myDirectory.project import Project
def print_sth():
print("something")
------------------------------------------------------
Traceback (most recent call last):
File "test_project.py", line 1, in <module>
from myDirectory.project import Project
ImportError: No module named 'myDirectory'
Finally, I tried adding the if __name__ == '__main__' statement within the test_project.py file, but it still failed. I would really appreciate if anyone could help. If there is a solution where I do not have to write a verbose command, I would prefer that.
When you run a Python script by filename, the Python interpreter assumes that it is a top-level module (and it adds the directory the script is in to the module search path). If the script is in a package, that's not correct. Instead, you should run the module using the -m flag, which takes a module name in the same format as an import statement (dotted separators) and puts the current directory in the module search path.
So, you could run the test from myDirectory with: python -m tests.test_project. When you run the script this way, either of the kinds of imports you tried will work.
But if myDirectory is supposed to be a top-level package itself (as the __init__.py file suggests), you should probably go up one level further up, to myDirectory's parent, and run the script with two levels of package names: python -m myDirectory.tests.test_project. If you do this and want the test to use an absolute import you'd need to name the top level package that the project module is in: from myDirectory.project import Project.
Related
I originally had a Python module that was structured like this:
mymod
├── config.py
└── __init__.py
And the files contained this code:
# __init__.py
import mymod.config
mymod.config.foo()
# config.py
def foo():
print('here')
Running this with python3 -c "import mymod" will print "here", as expected.
Now as part of a refactor, the directory structure looks like this:
mymod
├── core
│ ├── config.py
│ └── __init__.py
└── __init__.py
I want the same initialization code in core/__init__.py to run when importing mymod, so the new root __init__.py looks like this:
# __init__.py
import mymod.core
And the other files just need to have the module paths fixed:
# core/__init__.py
import mymod.core.config
mymod.core.config.foo()
# core/config.py (no change)
def foo():
print('here')
But now I get an AttributeError relating to a partially initialized module:
$ python3 -c "import mymod"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/gsgx/code/mymod/__init__.py", line 1, in <module>
import mymod.core
File "/home/gsgx/code/mymod/core/__init__.py", line 3, in <module>
mymod.core.config.foo()
AttributeError: partially initialized module 'mymod' has no attribute 'core' (most likely due to a circular import)
I think I understand why: I'm trying to access mymod.core while in the middle of the original statement importing it. But I realized I could fix it by changing the import in config.py to use a from ... import instead:
# core/__init__.py
from mymod.core import config
config.foo()
My question is, while this technically fixes the issue, I was wondering if there's any other workaround for this. I don't want to have to go through all the imports and fix any name conflicts that result from doing this. It feels like I should be able to move a module into a subfolder without major refactoring and without running into partial initialization issues. I've read various answers on SO about Python imports, but wasn't able to find anything to help with this particular issue.
I have the following file/folder structure:
testpackage (folder)
src (folder)
__init__.py
module1.py
tests
__init__.py
test_module1.py
just for clearance: the "module1.py" is under the "src" folder which is under the "testpakcage" folder.
"tests" is also under the "testpakcage" folder - same level as the "src" one.
module1.py has a class named "class1" as so:
class class1:
def method1 (self):
print('i am method1')
in test_module1.py I want to run tests on the above module. this is it's contents:
import unittest
from testpackage.src import module1
t = module1.class1()
t.method1()
this package is not installed, and I don't instead to install or submit it anywereh, I'm just trying to find the best structuring practice for me, for future packaging creation.
problem is: when I run the following either from the "tests" or "testpackage" folder:
/usr/bin/python3.6 -m unittest discover
I get the following error:
E
======================================================================
ERROR: test_module1 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_module1
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
something similar also happens when I just try to run "test_module1.py" from the "tests" folder:
Traceback (most recent call last):
File "test_module1.py", line 5, in <module>
from testpackage.src import *
ModuleNotFoundError: No module named 'testpackage'
so I tried changing the "import" line with a few alternatives but none of them work. each one of those was a different attempt (not all of them at once):
from testpackage.src import *
import testpackage.src.module1 as module1
import ..src.module1
from ..src.module1 import class1
searching stackoverflow I found solutions that worked for some but not for those using python 3 and above - which is my case.
any suggestions? I think what I'm trying to do is rather simple and I'm missing something really basic here.
I'm using python3.6 by way
I'm wondering if you saw different errors for some of those different import attempts you made. Not sure if this will solve your problem, but this is one way it would generally be accomplished.
First, I am not sure what the top level of your package is supposed to be. Do you reference testpackage in any of the code in the src folder? If so, that top folder should also contain an __init__.py file.
Now, at the very top of your test_module1.py file, add:
import sys
pkg_dir = ".." # if "src" if the top level folder else "..." if testpackage is the top level folder
sys.path.append(pkg_dir)
Note you must change pkg_dir depending on your module's structure.(which I cannot tell from your question).
What this code does is add the folder containing the top level folder of your package to the python import search tree. This can solve problems that the relative import in your example file will not: if the files in your module use module-level imports (e.g. import testpackage.src.module2 in module1.py). This is common in packages with multiple submodules that cross-import.
I thought I had understood the import system, however I'm struggling to understand an apparently trivial case: I have a very simple Python application with the following structure:
.
└── myapp
├── __init__.py
├── lib.py
└── myapp.py
The content of lib.py is a trivial function:
def funct():
print("hello from function in lib")
myapp.py is supposed to be the entrance point of the application:
import lib
def main():
lib.funct()
if __name__ == "__main__":
print("calling main")
main()
When I run the main script it works:
> python myapp/myapp.py
calling main
hello from function in lib
However, when I just import the package from IPython for instance, it fails:
In [2]: import myapp
---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
<ipython-input-2-cabddf3cb27d> in <module>
----> 1 import myapp
~/test/myapp/__init__.py in <module>
----> 1 import myapp.myapp
~/test/myapp/myapp.py in <module>
----> 1 import lib
2
3
4 def main():
5 lib.funct()
ModuleNotFoundError: No module named 'lib'
Why is this happening? Eventually, I'd like that my application is executable with python -m myapp, but also importable.
By default the import searches for module on the paths present in sys.path and one of the path present there is of current directory so
when you executed the main script:
python myapp/myapp.py
The import in myapp.py file searched for lib module in its current directory i.e "myapp" and as lib.py is in same directory, it executed perfectly
But when you imported myapp as a package in IPython,
The import in myapp.py file searches from IPython's path, where Lib.py is not present hence the classic no module found.
There are few things you can do here
use relevant path in myapp.py like
from . import lib
Note: This will generate error if you executed myapp.py directly as a script so handle accordingly
update the sys.path by appending lib.py's path (Not Recommended)
sys.path.append("....../lib.py")
watch this for clear understanding.
Also just to point out, __name__=="__main__" is only true when you execute the file directly. so the code inside if in myapp.py will not work when you'll use it as a module in package.
I hope this was helpful :)
I have the following set up.
~/python/pyct/lib/
├── printer.py
└── utils.py
~/apps/proj/
└── main.py
~/python/pyct/lib/utils.py
def printFunc(content):
print(content)
~/python/pyct/lib/printer.py
import utils # this breaks sometimes
# import pyct.lib.utils as utils # this works always
def printer(content):
utils.printFunc(content)
~/apps/proj/main.py
from pyct.lib.printer import printer
printer("hi")
Value of PYTHONPATH=$HOME/python
When I import printer.py in a file inside ~/python/pyct/lib/, everything runs as expected.
When I run main.py from ~/apps/proj/, I get the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
from pyct.lib.printer import printer
File "~/python/pyct/lib/printer.py", line 1, in <module>
import utils
ModuleNotFoundError: No module named 'utils'
I have tried using relative imports but that doesn't work. I have gone through this resource but could get the scenario working.
I am aware that the problem can be solved using absolute path in imports but I want to make a dir with multiple python files which will act as a library. Then use the core component else where outside the directory.
How can I achive the above scenario working.
Any resources to understand python imports will be very helpful.
The problem here is that ~/python/pyct/lib/printer.py looks for module named utils in its working directory - not in directory where it is placed itself.
You can use relative import in ~/python/pyct/lib/printer.py:
from . import utils
python will then look for module relative to the path of the importing one instead of working directory.
I have the project structure:
/hdfs-archiver
/logs
/qe
__init__.py
/tests
__init__.py
archiver.py
/utils
__init__.py
HdfsConnector.py
I am trying to run archiver.py but I am getting this error:
Traceback (most recent call last):
File "qe/tests/HdfsArchiver.py", line 8, in <module>
from qe.utils import HdfsConnector
ImportError: No module named qe.utils
I read around and it seemed like most people that come across this issue fixed it with __init__.py
when I pwd:
$ pwd
/Users/bli1/Development/QE/idea/hdfs-archiver
my PYTHONPATH in .bashrc
export PYTHONPATH=$PYTHONPATH:/Users/bli1/Development/QE/idea/hdfs-archiver
I also tried having my PYTHONPATH as
/Users/bli1/Development/QE/idea/hdfs-archiver/qe
You're trying to import HdfsConnector as a function or class. Include the HdfsConnector module as part of your absolute import:
from qe.utils.HdfsConnector import my_function
You can also import the module:
import qe.utils.HdfsConnector
# or
import qe.utils.HdfsConnector as HdfsConnector
Firstly, you could try a relative import such as
from ..utils import HdfsConnector
You'd also need to run the script as a module and not as a simple python script due to the __name__ being different. This wouldn't require you to modify the path.
You can find more info here.