I am writing a python module neuralnet. It was working all fine in Python2, but in Python3 imports are failing.
This is my code structure.
neuralnet/
__init__.py
train.py # A wrapper to train (does not define new things)
neuralnet.py # Defines the workhorse class neuralnet
layer/
__init__.py
inlayer.py # Defines input layer class
hiddenlayer.py
application/ # A seperate application (not part of the package)
classify.py # Imports the neuralnet class from neuralnet.py
train.py needs to import neuralnet.py's neuralnet class.
neuralnet.py needs to import layers/inlayer.py etc.
(I prefer relative imports.)
I have a different application (classify.py) which needs to import this module.
Where I do...
from neuralnet.neuralnet import neuralnet
I have tried a few ways to import.
Either I get an error (mostly arcane like parent is not imported)
While running train.py (which is a part of the neuralnet module)
from . import layer # In file neuralnet.py
SystemError: Parent module '' not loaded, cannot perform relative import
Or
while running classify.py (which is outside the module).
from layer.inlayers import input_layer # In file neuralnet.py
ImportError: No module named 'layer'
My imports worked perfectly well for years in Python2. I am wondering what Python3 expects of me? Should I move train.py to outside my module (technically it is not a part of the module)? Please suggest best practice.
In Python 3, implicit relative imports are forbidden, see https://www.python.org/dev/peps/pep-0328/ and https://docs.python.org/release/3.0.1/whatsnew/3.0.html#removed-syntax:
The only acceptable syntax for relative imports is from .[module]
import name. All import forms not starting with . are interpreted as
absolute imports. (PEP 0328)
from .stuff import Stuff is an explicit relative import, which you "should" make use of whenever possible, and must use in Python 3, whenever possible. Head over to https://stackoverflow.com/a/12173406/145400 for a deeper analysis on relative imports.
Relative import usage has changed from python2 to python3,
The only acceptable syntax for relative imports is from .[module] import name. All import forms not starting with . are interpreted as absolute imports. (PEP 0328)
Python’s -m switch allows running a module as a script. When you ran a module that was located inside a package, relative imports didn’t work correctly.
The fix for Python 2.6 adds a __package__ attribute to modules. When this attribute is present, relative imports will be relative to the value of this attribute instead of the __name__ attribute. (PEP 0366)
To your questions:
You get SystemError: Parent module '' not loaded, cannot perform relative import or
in Python3.5 and forward you would get ImportError: attempted relative import with no known parent package when
you run python neuralnet.py and try to import from . import layers as your __package__ would be None and PYTHONPATH will only have current file(and not its parent), hence it can't find layer.
You may either run the module like this:
python -m neuralnet.neuralnet
Here your __package__ will be neuralnet, hence you will be able to import the neuralnet module which is within it.
Or you may do this workaround:
Update the __init__.py in neuralnet package to:
import os
import sys
sys.path.append(os.path.dirname(os.path.realpath(__file__)))
Then run your script neuralnet.py, the above lines will add neuralnet directory to the PYTHONPATH.
You get ImportError: No module named 'layer' when layer is not a module that is in your PYTHONPATH so either install it or add it to the PATH using
import sys
sys.path.append("/path/to/layer")
Background:
Part of a message from Guido(author of Python):
... in 2.4, we introduce the leading dot notation for relative import,
while still allowing relative import without a leading dot. In
2.5 we can start warning about relative import without a leading dot
(although that will undoubtedly get complaints from folks who have
code that needs to work with 2.3 still). In 3.0 we can retire ambiguous import.
The use case for multiple dots should be obvious: inside a highly
structured package, modules inside one subpackage must have a way to
do relative import from another subpackage of the same parent package.
There is the remaining issue of what exactly the syntax would be. I
propose to extend the from clause to allow one or more dots before the
dotted name, and to make the dotted name optional if at least one
leading dot is found. I propose not to change from-less import.
Examples:
from .foo import bar
from .foo.bar import xxx
from . import foobar as barfoo
from ..foo.bar import *
from ... import foobar, barfoo
Relavent PEP to read: PEP-328
Related
i have two modules under package 'code' .
First module in math_func.py
another one is main.py.
I am importing math_func.py in main.py. Earlier when i was running mypy ot was giving error
'''Cannot find implementation or library stub for module named "math_func"'''
but after reading mypy docs i added path MYPYPATH using command
(project_2) p0p029i#m-c02dt1k2md6n enviroment_2 % export MYPYPATH=/Users/p0p029i/Desktop/enviroment_2
i am again getting the same error. Attached screenshot for reference.
math_func is not a top-level module, as you have __init__.py under code/.
So you should write import code.math_func instead OR remove code/__init__.py.
Adding environment_2 to MYPYPATH does not solve it because it still doesn't allow mypy to discover math_func as a top-level module.
To have a better understanding of it, consider the difference between the following imports:
import .math_func # relative import
import math_func # look for a top-level module "math_func"
import code.math_func # look for a top-level module "code" that has a submodule math_func underneath
import re # look for a top-level module "re"
I have a directory structure as follow:
evaluate.py
tools (folder)
-- ngram.py
-- bleu.py
In bleu.py, I import ngram. And, in evaluate.py, I import tools.bleu. However, an error occurs that ModuleNotFoundError: No module named 'ngram'. Where did I do wrong? Thanks~
If you intend for tools to be a package, you'll need to change the modules within it to use either absolute imports or explicit relative imports when they are importing each other.
That is, you need to change tools/bleu.py to do either:
import tools.ngram # or: from tools import ngram
Or:
from . import ngram
You should probably put an __init__.py file in the tools folder too (though it's not strictly necessary any more).
It seems that I am still missing some basics of python. I was trying to understand submodules importing, which I feel I have not understood yet. But I have also stumbled upon something new.
I am having following two packages in two different PyDev projects:
package1
|
+--mod1.py
|
+--mod2.py
package2
|
+--__init__.py
|
+--modx.py
|
+--mody.py
In mod1, I can do import mod2. But in __init__ and modx, I cannot do import mody (Eclipse says "unresolved imports"). In __init__, I can do import .mody or from .mody import vary. In modx, I cannot do import .mody. (In fact I never saw use of . in import statement as prefix to the module. Earlier I only came across import mod and from mod import var, but never saw import .mod and from .mod import var.) Why this might be happening? I must be unaware some context which is causing this behaviour. But then I dont know what is it?
PS: I am using Python 3.4
There is a subtle different between how Python is treating both of those packages.
package1 is treated as a namespace package in that it does not contain an __init__.py file.
package2 is treated as a regular package in that it does contain an __init__.py file.
So I'll give a quick breakdown of why each step is happening:
In mod1, I can do import mod2.
This is happening due to how namespace packages are handled using absolute imports. You're most likely executing python mod1.py from the directory in which the file is stored, right (in my attempt to re-create your folder structure and test it myself locally, I did the same)? So package1 becomes your current working directory with your mod2 file being at the root of that directory.
With namespace packages, Python will default look to sys.path in an attempt to find the imports you have requested. Since your current working directory is automatically added to and included in sys.path, Python can successfully find your import mod2 request without any difficulty.
Thanks to ShadowRanger for correcting my initial response to this where I had misunderstood exactly how Python is including the current working directory in its search.
In init, I can do import .mody or from .mody import vary.
This is because Python is treating this as a regular package. The name of your regular package in this case is package2. When you use the . notation, you are asking Python to start searching for the import from the current package (which in this case is your parent package2). So you have to use import .mody to find the mody package within the current package.
If you used .. then it would import from the parent of the current package and so on.
The dot notation is useful as you are explicitly declaring that you wish to search from the current package only - so if there was another package2 package on your PYTHONPATH, Python would know which one to choose.
But in init and modx, I cannot do import mody (Eclipse says "unresolved imports").
With __init__.py this is because you have not used the dot notation and have not told Python that you wish to search for these modules in the current package. So it's looking to the Python standard library and to your PYTHONPATH for these packages and not finding them (hence your error in Eclipse). By using the dot notation, you are stating that you wish to include the current package in the search and, thus, Python will then be able to locate those files.
Using the dot notation like this, to import via from . import mody, is to use a relative import.
With modx you also have to use a relative import (see next section).
In modx, I cannot do import .mody. Why this might be happening?
This is because you're not using a relative / absolute import. You'll be using a relative import in this case. A relative import is the from . import mody syntax you've seen already. Using a relative or absolute import behaviour is default in Python.
It is now the default behaviour as, with the old Python import behaviour, suppose Python's own standard library had a package called mody. When you'd use import mody it would previously have imported mody from your package and not the standard library. This wasn't always desirable. What if you specifically wanted the standard library version?
So now your imports must be made using from . import mody or from .mody import vary syntax so as the import is very clear. If you use import and not the from... syntax, Python will assume it's a standard library or PYTHONPATH import.
By the way, sources for a lot of the above information came from the following sites:
https://docs.python.org/3/reference/import.html
https://docs.python.org/2.5/whatsnew/pep-328.html
Python modules are optional "additions" to Python that can be imported using the import command like so:
import package1
package1.mod1 # Can be accessed using this
To import individual parts of the package, use from like so:
from package1 import mod1
mod1 # Can be accessed using this
If you want to import every part of a module and use it without package., use:
from package1 i
I see many people use the following import methods in their projects:
from .module1 import a,b
from ..module2 import *
The module1 and module2 are a .py file but not a folder for package. What's the differences to the import module? Does it mean to import the module in current and ../ folder? But when I try to import another file in same folder, it said:
import .other
>>> SyntaxError: Invalid syntax
from .other import *
>>> ValueError: Attempted relative import in non-package
I'm curious on it. Thanks~
What you see is relative imports. They allow you to import modules by specifying their relative paths, without hard-coding the name of the package in which the modules are defined.
Does it mean to import the module in current and ../ folder?
Yes.
See PEP 328 for more details. Note it says:
Relative imports must always use from <> import; import <> is always
absolute.
which is why you get the SyntaxError when trying import .foo.
The ValueError is probably because you are running the importing file as a script (and it used to confuse me a lot). You need to run it as a package (using the -m switch) for relative imports to work. That is, suppose foo.py relative-imports other modules, you can't run it by
$ python foo.py # non-package error
Instead you do
$ python -m foo
See the related question: How to do relative imports in Python.
I'm writing a little package and I'm trying to include a demo script within it as an example. However, I can't seem to import the package cleanly from within as though I was outside of it.
With a directory structure like:
trainer/
__init__.py
helper.py
trainer.py
[...more files...]
demo.py
In demo.py I can't do from .. import trainer as it complains "Attempted relative import in non-package", despite the __init__.py. If I move the demo up a directory and import trainer it works fine, but I was trying to keep it together with the package.
The hack-looking import __init__ as trainer works, but eeeew.
Importing the various bits from all over the module directly also works, but makes for a messy example. Am I wholly misguided in my attempt or is there a better solution?
If you're trying to run demo.py as python demo.py, the problem that you're having is likely the same as here.
What's happening is that Python's relative import mechanism works by using the __name__ of the current module. When you execute a module directly, the __name__ gets set "__main__" regardless what the actual module name is. Thus, relative (in-package) imports don't work.
To remedy this, you can do the following:
Execute demo.py as a module within a package, like so: python -m trainer.demo. This should fix the error, but you'll still be importing the trainer.py module instead of the package.
Now add from __future__ import absolute_import to demo.py, which will cause your imports to be absolute-only by default, meaning that relative imports have to explicit (as in, from . import (...)). This is force import trainer to import the entire top-level package, instead of the module.
The way you organize the files, demo.py becomes part of the package, which might or might not be what you want. You can organize your files a little differently, moving demo.py outside of the trainer directory:
TopDir/
demo.py
trainer/
__init__.py
helper.py
trainer.py
[... more files ...]
Then, demo.py can do something like:
from trainer import trainer, helper