I am having difficulties to understand how the python importer works..
I have a python script (fractalDimension.py) that imports a submodule ("greedyColoring.py") using:
from boxCovering.greedyColoring import *
It works fine when I call it directly:
python fractalDimension.py
The problem began when I moved the script to a folder and added a main script which imports the fractalDimension.py because now the import of the boxCovering sub module doesn't work.
In the main script I call the fractalDimension method:
import fractality.fractalDimension as fd
fd.calculate()
but when I run
python main.py
I get the following error
Traceback (most recent call last):
File "main.py", line 5, in <module>
import fractality.fractalDimension as fd
File "fractalDimension.py", line 11, in <module>
from boxCovering.greedyColoring import *
ImportError: No module named 'boxCovering'
Directory layout:
main.py
fractalDimension/
|
+-- __init__.py
+-- fractalDimension.py
+-- boxCovering/
|
+-- __init__.py
+-- greedyColoring.py
Both init files are empty
Why python cannot find the boxCovering module?
The error is due to the fact that python is searching for a top-level package called boxCovering inside the PYTHONPATH and none exists (you have only a sub-package inside the current directory, but this isn't searched).
When you want to import a subpackage/submodule you want to use a(n explicit) relative import:
from .boxCovering.greedyColoring import *
note the . (dot) at the beginning.
Alternatively use the absolute import:
from fractality.fractalDimension.boxCovering.greedyColoring import *
In python2 the import allowed an implicit relative import, which is what you are trying to do. In that case the import:
from boxCovering import X
done in the way you are doing could have two different meanings:
import of the subpackage boxCovering of the fractalDimension package
import of the top-level package boxCovering
In python3 (see the What's new? and relevant PEP 0328) they decided that such an import always has the second meaning. If you want a relative import you have to explicitly state so by using the relative syntax, which uses a dot at the beginning of the module name.
Each dot means go to one directory above so . means the current directory while .. means the parent directory (as in unix paths) etc.
You can enable this semantics on python2 adding:
from __future__ import absolute_import
at the top of your files.
Related
I have the following file/folder structure:
testpackage (folder)
src (folder)
__init__.py
module1.py
tests
__init__.py
test_module1.py
just for clearance: the "module1.py" is under the "src" folder which is under the "testpakcage" folder.
"tests" is also under the "testpakcage" folder - same level as the "src" one.
module1.py has a class named "class1" as so:
class class1:
def method1 (self):
print('i am method1')
in test_module1.py I want to run tests on the above module. this is it's contents:
import unittest
from testpackage.src import module1
t = module1.class1()
t.method1()
this package is not installed, and I don't instead to install or submit it anywereh, I'm just trying to find the best structuring practice for me, for future packaging creation.
problem is: when I run the following either from the "tests" or "testpackage" folder:
/usr/bin/python3.6 -m unittest discover
I get the following error:
E
======================================================================
ERROR: test_module1 (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_module1
Traceback (most recent call last):
File "/usr/lib64/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
something similar also happens when I just try to run "test_module1.py" from the "tests" folder:
Traceback (most recent call last):
File "test_module1.py", line 5, in <module>
from testpackage.src import *
ModuleNotFoundError: No module named 'testpackage'
so I tried changing the "import" line with a few alternatives but none of them work. each one of those was a different attempt (not all of them at once):
from testpackage.src import *
import testpackage.src.module1 as module1
import ..src.module1
from ..src.module1 import class1
searching stackoverflow I found solutions that worked for some but not for those using python 3 and above - which is my case.
any suggestions? I think what I'm trying to do is rather simple and I'm missing something really basic here.
I'm using python3.6 by way
I'm wondering if you saw different errors for some of those different import attempts you made. Not sure if this will solve your problem, but this is one way it would generally be accomplished.
First, I am not sure what the top level of your package is supposed to be. Do you reference testpackage in any of the code in the src folder? If so, that top folder should also contain an __init__.py file.
Now, at the very top of your test_module1.py file, add:
import sys
pkg_dir = ".." # if "src" if the top level folder else "..." if testpackage is the top level folder
sys.path.append(pkg_dir)
Note you must change pkg_dir depending on your module's structure.(which I cannot tell from your question).
What this code does is add the folder containing the top level folder of your package to the python import search tree. This can solve problems that the relative import in your example file will not: if the files in your module use module-level imports (e.g. import testpackage.src.module2 in module1.py). This is common in packages with multiple submodules that cross-import.
E:\Karya\Python <-- main.py inside here
E:\Karya\packages\extra< -- iota.py inside here
import sys
sys.path.append('E:\Karya')
sys.path.append('E:\Karya\Python')
from ..packages.extra import iota
print(iota.FunI())
Question:call function FunI() by import iota.py
ERROR :
$ C:/Users/ready/AppData/Local/Microsoft/WindowsApps/python3.9.exe e:/Karya/Python/main.py
Traceback (most recent call last):
File "e:\Karya\Python\main.py", line 5, in
from ..packages.extra import iota
ImportError: attempted relative import with no known parent package
detail:
tree directory, main.py inside python folder and iota.py inside extra folder
Error
Code
Just add packages directory to sys.path and import iota from extra
sys.path.append("E:\\Karya\\packages")
from extra import iota
Make sure that there is init.py file in extra directory (and I think it is needed in packages directory too)
I have the following set up.
~/python/pyct/lib/
├── printer.py
└── utils.py
~/apps/proj/
└── main.py
~/python/pyct/lib/utils.py
def printFunc(content):
print(content)
~/python/pyct/lib/printer.py
import utils # this breaks sometimes
# import pyct.lib.utils as utils # this works always
def printer(content):
utils.printFunc(content)
~/apps/proj/main.py
from pyct.lib.printer import printer
printer("hi")
Value of PYTHONPATH=$HOME/python
When I import printer.py in a file inside ~/python/pyct/lib/, everything runs as expected.
When I run main.py from ~/apps/proj/, I get the following error:
Traceback (most recent call last):
File "main.py", line 1, in <module>
from pyct.lib.printer import printer
File "~/python/pyct/lib/printer.py", line 1, in <module>
import utils
ModuleNotFoundError: No module named 'utils'
I have tried using relative imports but that doesn't work. I have gone through this resource but could get the scenario working.
I am aware that the problem can be solved using absolute path in imports but I want to make a dir with multiple python files which will act as a library. Then use the core component else where outside the directory.
How can I achive the above scenario working.
Any resources to understand python imports will be very helpful.
The problem here is that ~/python/pyct/lib/printer.py looks for module named utils in its working directory - not in directory where it is placed itself.
You can use relative import in ~/python/pyct/lib/printer.py:
from . import utils
python will then look for module relative to the path of the importing one instead of working directory.
I have the project structure:
/hdfs-archiver
/logs
/qe
__init__.py
/tests
__init__.py
archiver.py
/utils
__init__.py
HdfsConnector.py
I am trying to run archiver.py but I am getting this error:
Traceback (most recent call last):
File "qe/tests/HdfsArchiver.py", line 8, in <module>
from qe.utils import HdfsConnector
ImportError: No module named qe.utils
I read around and it seemed like most people that come across this issue fixed it with __init__.py
when I pwd:
$ pwd
/Users/bli1/Development/QE/idea/hdfs-archiver
my PYTHONPATH in .bashrc
export PYTHONPATH=$PYTHONPATH:/Users/bli1/Development/QE/idea/hdfs-archiver
I also tried having my PYTHONPATH as
/Users/bli1/Development/QE/idea/hdfs-archiver/qe
You're trying to import HdfsConnector as a function or class. Include the HdfsConnector module as part of your absolute import:
from qe.utils.HdfsConnector import my_function
You can also import the module:
import qe.utils.HdfsConnector
# or
import qe.utils.HdfsConnector as HdfsConnector
Firstly, you could try a relative import such as
from ..utils import HdfsConnector
You'd also need to run the script as a module and not as a simple python script due to the __name__ being different. This wouldn't require you to modify the path.
You can find more info here.
No matter how I structure the imports in the code files and in the __init__.py files, I can't seem to get it right for executing the program and running the tests using pytest. How do I need to write the imports when my project structure looks like this:
src/
__init__.py
VocableFileWriter.py
WordListHelper.py
WordListReader.py
XLDAttributeValueAdder.py
exceptions/
__init__.py
XMLInvalidException.py
XMLParseException.py
gui/
__init__.py
GTKSignal.py
XLDAttributeValueAdderWindow.py
test/
__init__.py
test_XLDAttributeValueAdder.py
xmlparser/
__init__.py
XMLParser.py
Currently I have them like this:
In the __init__.py files I have the imports like this (src/__init__.py):
from src import *
from src.exceptions import *
from src.xmlparser import *
and in a subpackage (src/xmlparser/__init__.py):
from src.xmlparser import *
So I guess those are project-absolute paths to the modules.
In the code files themselves I import like this:
import os
import sys
from VocableFileWriter import VocableFileWriter
from XLDAttributeValueAdder import XLDAttributeValueAdder
However, when I execute the code from the directory above src using:
./src/main.py
It tells me that:
Traceback (most recent call last):
File "./src/main.py", line 6, in <module>
from VocableFileWriter import VocableFileWriter
File "/home/xiaolong/Development/PycharmProjects/xld-attribute-value-adder/src/VocableFileWriter.py", line 2, in <module>
from XMLInvalidException import XMLInvalidException
ImportError: No module named 'XMLInvalidException'
It used to be a PyCharm project, but I couldn't get it to run with the imports structure PyCharm used when not using PyCharm but running it from the terminal, so I decided I wanted to take the whole import stuff into my own hands. So don't get confused by it being in a PyCharm directory.
I also want to be able to execute the tests using for example:
py.test src/test/test_XLDAttributeValueAdder.py
How do I solve this mess?
Edit#1:
I had the program running once, but then the test complained about missing modules, probably because it's in another subdirectory and I tried so many configurations, that I don't know how I had the program running anymore. If possible please add some explanation why a certain structure is correct and works for both, tests and the program itself.
EDIT#2:
I've managed to get the tests running now, but now the program doesn't run anymore.
I emptied all the __init__.py files and used only project-absolute paths in the code files like this (src/test/test_XLDAttributeValueAdder.py):
from src.VocableFileWriter import VocableFileWriter
from src.WordListHelper import WordListHelper
from src.WordListReader import WordListReader
from src.XLDAttributeValueAdder import XLDAttributeValueAdder
from src.exceptions.XMLInvalidException import XMLInvalidException
from src.exceptions.XMLParseException import XMLParserException
from src.xmlparser.XMLParser import XMLParser
from src.test.path_helper import go_up
from src.test.path_helper import go_in
and in the main.py:
from src.VocableFileWriter import VocableFileWriter
from src.XLDAttributeValueAdder import XLDAttributeValueAdder
the output of ./src/main.py:
./src/main.py
Traceback (most recent call last):
File "./src/main.py", line 6, in <module>
from src.VocableFileWriter import VocableFileWriter
ImportError: No module named 'src'
EDIT#3:
I've tried the relative import way like this:
main.py moved one directory up
The structure now looks like this:
main.py
src/
__init__.py
VocableFileWriter.py
WordListHelper.py
WordListReader.py
XLDAttributeValueAdder.py
exceptions/
__init__.py
XMLInvalidException.py
XMLParseException.py
gui/
__init__.py
GTKSignal.py
XLDAttributeValueAdderWindow.py
test/
__init__.py
test_XLDAttributeValueAdder.py
xmlparser/
__init__.py
XMLParser.py
__init__.py files empty
main.py file gets relative
Imports look like this:
import os
import sys
from .src.VocableFileWriter import VocableFileWriter
from .src.XLDAttributeValueAdder import XLDAttributeValueAdder
When I try to run the main.py file:
Traceback (most recent call last):
File "main.py", line 6, in <module>
from .src.VocableFileWriter import VocableFileWriter
SystemError: Parent module '' not loaded, cannot perform relative import
However, there is a __init__.py file in the same directory as the main.py file, also empty.
You'd better use relative imports.
When doing from src import * in src/__init__.py you'll only import anything you've defined before calling the import. Most likely that's just nothing. If you've defined the __all__ variable, you'll get the submodule from it, but maybe also an AttributeError if a variable hasn't been defined yet.
So instead import the modules you need explicitly where you need them like
from .VocableFileWriter import VocableFileWriter
from .exceptions.XMLInvalidException import XMLInvalidException
or lets say within src/gui/GTKSignal.py
from ..exceptions.XMLInvalidException import XMLInvalidException
Also you could use project-absolute paths like mentioned in your Edit#2.
Furthermore you've a problem with your path when calling ./src/main.py. In this case the directory containing src is not in the path. Instead ./src/ is in the path. Therefore python doesn't consider it a package anymore and since there's no other package with that name available, the import fails.
Instead put your main.py module in the same directory like the src package or call it as a module with python -m src.main. Doing so you could also rename it to src/main.py and call the package instad with python -m src.
When using the first approach, main should use an absolute import structure. Just think of it as a module you put anywhere on your computer, while your src package is somewhere else, where it can be found by Python, like any other module or package installed by the system. It's not different, when it's located besides the package itself, since the current directory is in sys.path.
main.py
import os
import sys
from src.VocableFileWriter import VocableFileWriter
from src.XLDAttributeValueAdder import XLDAttributeValueAdder