I am trying to build a Python package. For simplification, I will only mention the parts which are relevant to the problem:
The package (directory) is called moranpycess and inside it there are three relevant files:
The initfile for the whole package __init__.py:
from .Individual import Individual
from .MoranProcess import MoranProcess
A module called Individual which contains a class Individual.
A module called MoranProcess which contains a class MoranProcess. At the top it imports the previous module with: import Individual.
I install the package with python -m pip install .
Then I run a test to see if the package can be imported properly: python -c 'import moranpycess'.
I get:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/runner/work/angry-moran-simulator/angry-moran-simulator/moranpycess/__init__.py", line 18, in <module>
from .MoranProcess import MoranProcess
File "/home/runner/work/angry-moran-simulator/angry-moran-simulator/moranpycess/MoranProcess.py", line 22, in <module>
import Individual
ModuleNotFoundError: No module named 'Individual'
Error: Process completed with exit code 1.
This is strange to me since it seems that the Python interpreter can find the package, the package imports the respective classes but the interpreter also tries to execute the top-level module imports and (I don't know why) it does not find the module...
Am I doing something wrong here?
EDIT:
directory structure:
└── moranpycess
├── Individual.py
├── MoranProcess.py
└── __init__.py
Update
I am considering the solution proposed here:
https://stackoverflow.com/a/49375740/2340598
However I don't know if this is the "right" way to organize a package...
Relative imports are a bit harder to get right than absolute imports. They are also more brittle (e.g., if you move a file doing the import, it breaks), so unless you have a good reason to use relative imports I'd suggest to just use absolute ones.
Following that, your public API as it is defined in your topmost __init__.py would look like this:
from moranpycess.MoranProcess import MoranProcess
__all__ = ["MoranProcess"] # import hint that some tools use
While non-public objects can still be imported internally (e.g. in MoranProcess.py) in the following way:
from moranpycess.Individual import Individual
The good thing about absolute imports is that no matter where in your package (or outside of it) you are, they always look the same.
If done like this, users that have installed your package should be able to use your objects like this:
from moranpycess import MoranProcess # directly gets the object, not module
You should try to avoid messing with sys.path in order to get your imports to work, mainly because it isn't necessary. If all you're doing is writing regular python code that is meant to be used in a regular way, the builtin import mechanics should serve your use cases just fine.
If I were to import a package and noticed that sys.path changed, I'd suspect shenanigans and start looking for an alternative.
Related
I have a directory structure as follow:
evaluate.py
tools (folder)
-- ngram.py
-- bleu.py
In bleu.py, I import ngram. And, in evaluate.py, I import tools.bleu. However, an error occurs that ModuleNotFoundError: No module named 'ngram'. Where did I do wrong? Thanks~
If you intend for tools to be a package, you'll need to change the modules within it to use either absolute imports or explicit relative imports when they are importing each other.
That is, you need to change tools/bleu.py to do either:
import tools.ngram # or: from tools import ngram
Or:
from . import ngram
You should probably put an __init__.py file in the tools folder too (though it's not strictly necessary any more).
My Directory structure in /VM/repo/project is:
__init__.py
scripts/
getSomething.py
__init__.py
classes/
project.py
db.py
__init__.py
getSomething.py
from ..classes import project
from ..classes import db
project.py
class PROJECT:
def __init__(self):
stuff
db.py
class DB:
def __init__(self):
stuff
When I try to run
python getSomething.py
I get the error
Traceback (most recent call last):
File "scripts/getSomething.py", line 4, in < module >
from ..classes import project
ValueError: Attempted relative import in non-package
What am I missing here?
As stated in the error, you're running getSomething as a main module. But you can't do package relative imports when you aren't in a package. The main module is never in a package. So, if you were to import getSomething as part of a package...:
# /VM/repo/main.py
from project.scripts import getSomething
Then you would not have the import errors.
Perhaps it is helpful to have a quick discussion on python modules and packages. In general, a file that contains python source code and has the .py extension is a module. Typically that module's name is the name of the file (sans extension), but if you run it directly, that module's name is '__main__'. So far, this is all well known and documented. To import a module you just do import module or import package.module and so on. This last import statement refers to something else (a "package") which we'll talk about now...
Packages are directories that you can import. Many directories can't be imported (e.g. maybe they have no python source files -- modules -- in them). So to resolve this ambiguity, there is also the requirement that the directory has an __init__.py file. When you import a directory on your filesystem, python actually imports the __init__.py module and creates the associated package from the things in __init__.py (if there are any).
Putting all of this together shows why executing a file inside a directory that has an __init__.py is not enough for python to consider the module to be part of a package. First, the name of the module is __main__, not package.filename_sans_extension. Second, the creation of a package depends not just on the filesystem structure, but on whether the directory (and therefore __init__.py was actually imported).
You might be asking yourself "Why did they design it this way?" Indeed, I've asked myself that same question on occasion. I think that the reason is because the language designers want certain guarantees to be in place for a package. A package should be a unit of things which are designed to work together for a specific purpose. It shouldn't be a silo of scripts that by virtue of a few haphazard __init__.py files gain the ability to walk around the filesystem to find the other modules that they need. If something is designed to be run as a main module, then it probably shouldn't be part of the package. Rather it should be a separate module that imports the package and relies on it.
I'm still learning to code better in Python. Therefore I am trying to use some constructions in my programming. Things like name_conventions and structured packaging & modules.
That said I come across an issue which I cannot resolve, it seems.
The structure I chose is as follows:
Core
__init__.py
controller.py
Dashboard
Log
Plugins
Stats
__init__.py
controller.py
Plugin2
Plugin3
Etc..
main.py
In the Core controller.py i am trying to import the Stats(object) class.
I have tried several different ways. In my IDE (Pycharm) it works like a charm ;) When i try to start it via Command Line i get ImportErrors.
I have tried different approaches.
from Plugins.Stats.controller import Stats
from Plugins.Stats import controller
from Plugins.Stats import controller.Stats
from Plugins import Stats.controller
I also tried to put the Stats class into the Plugins.Stats.__init__.py and call it:
from Plugins.Stats import Stats
and I get this:
Traceback (most recent call last):
File "controller.py", line 4, in <module>
from Plugins.Stats.controller import Stats
ImportError: No module named 'Plugins'
The funny thing is... When i try to do the same from main.py.
It does work. So I am really confused. Am I doing something wrong?
Also are there special design patterns to handle imports?
You should add __init__.py file also in the Plugins directory so it will look like
Plugins
__init__.py
Stats
__init__.py
controller.py
You can read more about Python modules here - check out 6.4. Packages
you also can make your module you want to import visible for everyone by adding it to the pythonpath
sys.path.append("/path/to/main_folder")
before this modules are on another namespaces
You will find more info here
First put From plugin import stats, then from stats import controller and import _init_
Incase of running from the terminal it will only try to import from the current folder. You have to manually add the path of the main folder to this file using the following code at the beging of the file
import sys
sys.path.append("../")
If you want to back 2 folders add ../../ instead of ../
You should not start a submodule of a package directly as a script. This will mess up your hierarchy most of the time. Instead call it as a module:
python -m Core.controller
You can verify the following by adding this to the top of any of your scripts:
import sys
for p in sys.path:
print(repr(p))
The reason the imports work when invoked as:
python main.py
is python will add the directory of the script to the pythonpath (In this case the empty string '')
However, when running
python Core/controller.py
it will add Core to sys.path, rendering the rest of your project unimportable
You can instead use
python -m Core.controller
which'll invoke using runpy and again put the empty string on the python path.
You can read more about invocation options of the python interpreter here: https://docs.python.org/2/using/cmdline.html You can read about runpy here: https://docs.python.org/2/library/runpy.html
I am developing a package with the following structure on disk:
foo/
__init__.py
xml.py
bar.py
moo.py
The xml.py package provides a class that does some custom XML parsing and translation for the other package components using a SAX stream parser. So it has in it:
import xml.sax
import xml.sax.handler
But when I go to use foo.xml in an application I get:
Traceback (most recent call last):
File "testxmlparser.py", line 15, in <module>
import foo.xml
File "~/code/foo/xml.py", line 39, in <module>
import xml.sax
ImportError: No module named sax
I appear to have a namespace conflict. If I rename xml.py to something else like xmlparser.py everything works as expected. But this feels like the wrong thing to do. I feel like I'm missing something fundamental about package names and resolution in Python here.
Is there a proper way to make this work that doesn't involve me renaming the foo/xml.py file? Or is that really the only solution to the conflicting names?
Edit: The "avoid naming things the same as standard Python modules" seems...well..a mineshaft to me. That's a moving target, the standard module set, that's bound to change and grow over time. So unless you get really creative with your names the rename-things-until-you-find-something-that-doesn't-conflict solutions seems poor to me. Besides, I've got it in a unique package name with foo already (I'm not using foo, but something that is definitely unique), shouldn't that be enough?
As mentioned over here, use
from __future__ import absolute_import
and use relative imports if needed.
I know there are several similar questions, but I'm struggling to understand the error I'm getting and browsing the docs and similar questions hasn't helped yet. If anything, the similar questions make me feel like what I'm doing is right.
I have the following files:
src/main.py
from pack import pack
if __name__ == '__main__':
pack.exec("Hello Universe!")
src/pack/pack.py
import util
def exec(text):
util.write(text)
if __name__ == '__main__':
exec("Hello World!")
src/pack/util.py
def write(text):
print(text)
*src/pack/_init_.py*
EMPTY FILE
When I run python pack.py from the src/pack directory, it works (prints "Hello World!"). However when I run python main.py from the src directory I get the following exception:
Traceback (most recent call last):
File ".../src/main.py", line 1, in <module>
from pack import pack
File ".../src/pack/pack.py", line 1, in <module>
import util
ImportError: No module named util
If I change the import line in pack.py to from . import util as suggested, effectively the opposite occours. main.py runs successfully, however now pack.py fails, raising:
Traceback (most recent call last):
File ".../src/pack/pack.py", line 1, in <module>
from . import util
ValueError: Attempted relative import in non-package
I would have thought that imports are relative to the current location, and as such you ought to be able to construct a chain of imports like this. It seems very odd to me that module is supposed to import a sibling file differently depending on where the program starts.
Can someone explain why this error occurs in one way but not the other, and if there is some way to allow this file structure to run whether I want to run from main.py or pack.py?
You will have trouble making the import work in both cases. This is because in one case you are running pack.py as the main file and in another you run it as part of a package.
When you run it as a standalone script python pack.py, the "pack" directory gets added to the PYTHONPATH, meaning that you can import any module in it. Hence, import util will work.
When you run python main.py you add the src directory to your PYTHONPATH. This means that any module or package in src, for example the pack directory, now becomes importable. Hence from pack import pack. However, to access util.py you now need to do from pack import util. You can also do from . import util from within pack.py, as you noticed.
But you can't really do both at the same time. Either src/ is the main directory or src/pack is.
The obvious, but wrong solution, is to let the main.py add the src/pack directory to the PYTHONPATH. That will work, but it's not a good idea. The correct way to do this is to make up your mind. Is src/pack a module that should be imported via import pack or is it just a folder with a bunch of Python scripts? Decide! :-)
I think in this case its' obvious that src/pack should be a module. So then treat it like a module, and make sure it's available like a module. Then you can from pack import util even when running pack.py as a main script.
How do you do that? Well, basically you either install the pack module in your site-packages, or you add the src directory to the PYTHONPATH. That last one is what you want during development. You can either do it manually with export PYTHONPATH=<path> or you can let your testrunners do it for you. You don't have a testrunner? Well you should, but that's another question. :)
For installing it permanently once you don't do development anymore, take a look at Distribute. It includes a testrunner. ;)
Your link is to python 2.7 documentation, but it looks like you are using Python 3.x.
See here: http://docs.python.org/py3k/ for the correct docs.
Python 3 removes implicit relative import. That is, you cannot import packages within the same module in the same way anymore.
You need to use from . import util, this is an explicitly relative import and is allowed. import X no longer checks the current directory. Instead it only check the entries in sys.path. This includes the directory of the script was started and python's standard library.
You are missing __init__.py in your pack directory.
Create an empty file called __init__.py in your pack directory.
Nothing in the correct documentation supports your theory that this form of importing should work.