I don't understand the following from pep-0404
In Python 3, implicit relative imports within packages are no longer
available - only absolute imports and explicit relative imports are
supported. In addition, star imports (e.g. from x import *) are only
permitted in module level code.
What is a relative import?
In what other places star import was allowed in python2?
Please explain with examples.
Relative import happens whenever you are importing a package relative to the current script/package.
Consider the following tree for example:
mypkg
├── base.py
└── derived.py
Now, your derived.py requires something from base.py. In Python 2, you could do it like this (in derived.py):
from base import BaseThing
Python 3 no longer supports that since it's not explicit whether you want the 'relative' or 'absolute' base. In other words, if there was a Python package named base installed in the system, you'd get the wrong one.
Instead it requires you to use explicit imports which explicitly specify location of a module on a path-alike basis. Your derived.py would look like:
from .base import BaseThing
The leading . says 'import base from module directory'; in other words, .base maps to ./base.py.
Similarly, there is .. prefix which goes up the directory hierarchy like ../ (with ..mod mapping to ../mod.py), and then ... which goes two levels up (../../mod.py) and so on.
Please however note that the relative paths listed above were relative to directory where current module (derived.py) resides in, not the current working directory.
#BrenBarn has already explained the star import case. For completeness, I will have to say the same ;).
For example, you need to use a few math functions but you use them only in a single function. In Python 2 you were permitted to be semi-lazy:
def sin_degrees(x):
from math import *
return sin(degrees(x))
Note that it already triggers a warning in Python 2:
a.py:1: SyntaxWarning: import * only allowed at module level
def sin_degrees(x):
In modern Python 2 code you should and in Python 3 you have to do either:
def sin_degrees(x):
from math import sin, degrees
return sin(degrees(x))
or:
from math import *
def sin_degrees(x):
return sin(degrees(x))
For relative imports see the documentation. A relative import is when you import from a module relative to that module's location, instead of absolutely from sys.path.
As for import *, Python 2 allowed star imports within functions, for instance:
>>> def f():
... from math import *
... print sqrt
A warning is issued for this in Python 2 (at least recent versions). In Python 3 it is no longer allowed and you can only do star imports at the top level of a module (not inside functions or classes).
To support both Python 2 and Python 3, use explicit relative imports as below. They are relative to the current module. They have been supported starting from 2.5.
from .sister import foo
from . import brother
from ..aunt import bar
from .. import uncle
Added another case to Michał Górny's answer:
Note that relative imports are based on the name of the current module. Since the name of the main module is always "__main__", modules intended for use as the main module of a Python application must always use absolute imports.
Related
My project structure like this:
/project
main.py
/a_module
__init__.py
/sub_module
__init__.py
some_file.py
main.py
from a_module import main_api
a_module/__init__.py
from sub_module import sub_api
sub_module/__init__.py
from some_file import detail_api
In a_module/__init__.py gives Unable to import 'sub_module' error.
Why I cannot import 'sub_module'?
When I change to the relative path solve the error.
from .sub_module import sub_api
But I don't understand, does __init__.py design for public the API of the module? Why don't treat sub_module as a module instead of a directory? it's such a bad design to me...
__init__.py is executed when you import the package that contains it. But it's not your problem. Your problem is that module imports are always absolute unless explicitly relative. That means that they must chain from some directory in sys.path. By default this includes the working directory, so when you run main.py from within project, it can find a_module, and nothing else.
from sub_module import sub_api
In a_module/__init__.py doesn't work though, because imports are always absolute unless explicitly relative. So that import says "starting from some sys.path root, find a top level package named sub_module and import sub_api from it". Since no such module exists you get an error. from .sub_module import sub_api works because you opted into relative imports, so it doesn't start over from sys.path.
For an example of why you would do this, I'll give you something that broke in our own code back in the Python 2 days before absolute import by default was the law (from __future__ import absolute_import enabled the Py3 behavior, which is how we fixed it, but despite what the docs say, it was never enabled by default in Py2, the only enabled by default behavior was relative imports). Our layout was:
teamnamespace/
module.py
math/
mathrelatedsubmodule.py
othermathsubmodule.py
Now, we innocently thought hey, we'll put all our packages under a single shared top level namespace, and subpackages cover broad categories within them, and since we had a lot of additional utilities for basic mathematics, we put them under teamnamespace.math. Problem was, for the non-math modules, like teamnamespace.module, when they did:
import math # or
from math import ceil
it defaulted to relative lookup, and imported teamnamespace.math as math (a thoroughly useless import, since it was a namespace package only, all the functionality was in the sub-modules), not the built-in math module. In fact, without the Python 3 behavior, there was no reasonable way to get the built-in math module from a module under teamnamespace. Whereas with the Python 3 behavior, you can get either one or both (by aliasing one or the other with as, with no ambiguity:
# Gets built-in
import math
# Gets teamnamespace.math
from . import math
I have a number of small utils modules that i organize under a 'msa' namespace so I can use in a number of different research projects. Currently I have them organized like this:
# folder structure:
packages <-- in my pythonpath
--msa
----msa_utils.py
----msa_geom.py
----msa_pyglet.py
----msa_math.py
----etc
# imported and used this like
from msa import msa_pyglet
from msa import msa_math
msa_pyglet.draw_rect(msa_math.lerp(...))
However I would like to avoid the 'msa_' in the names and use like this:
# folder structure:
packages <-- in my pythonpath
--msa
----utils.py
----geom.py
----pyglet.py
----math.py
----etc
# imported and used this like
import msa.pyglet
import msa.math
msa.pyglet.draw_rect(msa.math.lerp(...))
This shouldn't cause name conflicts when importing from outside, however there are name conflicts when the modules themselves import modules with conflicting names. E.g. msa/pyglet needs to imports pyglet (the external one), but ends up trying to import itself. Likewise any module which tries to import the standard math library imports only my math module. Which is all understandable. But what is the usual pythonic way of dealing with this? Do I have to give each module file a globally unique name?
In Python 2, imports in packages without a package qualifier indeed first look for package local modules.
Thus, import pyglet will find msa.pyglet before the top-level pyglet is considered.
Switch to absolute imports to make the Python 3 behaviour the default, where unqualified names are always top level names:
from __future__ import absolute_import
Now import pyglet can only ever find the top-level name, never msa.pyglet. To reference other modules within your msa namespace, use from . import pyglet or from msa import pyglet.
See PEP 328 -- Imports: Multi-Line and Absolute/Relative for more details.
I'm trying to do the following in python 2.6.
my_module.py:-
from another_module import another_factory
def my_factory(name):
pass
another_module.py:-
from my_module import my_factory
def another_factory(name):
pass
Both modules in the same folder.
It gives me the error:
Error: cannot import name my_factory
As seen from the comments, you are trying to do a circle import which is impossible.
If in your module A you try to import something from the module B, and when loading the module B (to satisfy this dependency) you are trying to import something from the module A, you are where you started and you got a circle import: A needs B and B needs A!!, it is somehow like saying that A needs A, which is quite unlogic.
For instance:
# moduleA
from moduleB import functionB
...
So the interpreter tries to load the moduleB, which looks like the following:
# moduleB
from moduleA import functionA
...
And goes back to the moduleA, which tries again to import B, and, etc. Therefore python just raises the error and stops the insanity for a greater good.
Dependencies don't work like this. Define what module needs the other one, and just do a simple import. In your example, it seems that another_module needs my_module, so change my_module and eliminate the dependency on another_module.
If both modules actually need each other, it is a clear sign that they belong to the same logical concept, and should be merged.
PD: in some cases to avoid huge files, you can split a logical unit in two, and to avoid the circle dependencies, you write your imports inside of the functions (which are not executed at load time), so that there is not a circle. This is however in general something to avoid.
The real question is... do you consider each file as a module or are they part of a package ?
Trying to import modules outside a package is sometimes painful. You should rather build a package by simply creating an empty __init__.py module in the directory. Though, if you have
__init__.py
my_module.py
another_module.py
If you have te following function in my_module.py,
def my_factory(x):
return x * x
You should be able to access the my_factory() function from another_module.py by writing this :
from my_module import my_factory
But, if you don't have the __init__.py file/module, the import function will be (somehow) lost and will only use the sys.path for searching other modules. You may then add the following lines (before the import) in the another_module.py file :
sys.path.append(os.path.dirname(os.path.expanduser('.')))
You may also use the various packages available to help importing modules, like imp or import_file (see the documentation). Or you can decide to use load_source (also see the doc : https://docs.python.org/2/library/imp.html)
I am looking for a way to write python packages and modules that makes them Python 3 friendly and also it makes easy to import them.
Most common case is that you have one major class that you want to provide to the users, call it MyCode.
Your package would be named mycode and you would put the body of MyCode class into mycode/mycode.py.
Now, you would expect that people could do one of these:
import mycode
obj = mycode.MyClass()
or:
from mycode import MyClass()
obj = MyClass()
Now, the question is what you should put inside the __init__.py in order to make this work, in both python 2.6+ and 3.x.
Use absolute imports, it'll work fine across python versions. Inside mycode/__init__.py put:
from __future__ import absolute_import
from mycode.mycode import MyClass
where the __future__ import works from Python 2.5 and onwards; see PEP 328. Without the absolute_import import, import mycode is ambiguous; Python 3 will treat it as absolute and load the top-level package, Python 2 treats it as relative and import the nested module instead.
The alternative is to use a relative import:
from .mycode import MyClass
which will work across the same spectrum of versions.
Adding from mycode import * to the start of mycode/__init__.py is going to make the content of mycode/mycode.py importable from the mycode package like it was just a normal module like mycode.py, same as when you import mycode from the same directory as your mycode.py script (/mycode/) or do a relative import to mycode/mycode.py ...
all.
I'd think that this could be answered easily, but it isn't. As long as I've been searching for an answer, I keep thinking that I'm overlooking something simple.
I have a python workspace with the following package structure:
MyTestProject
/src
/TestProjectNamespace
__init__.py
Module_A.py
Module_B.py
SecondTestProject
/src
/SecondTestProjectNamespace
__init__.py
Module_1.py
Module_2.py
...
Module_10.py
Note that MyTestProjectNamespace has a reference to SecondTestProjectNamespace.
In MyTestProjectNamespace, I need to import everything in SecondTestProjectNamespace. I could import one module at a time with the following statement(s):
from SecondTestProjectNamespace.Module_A import *
from SecondTestProjectNamespace.Module_B import *
...but this isn't practical if the SecondTestProject has 50 modules in it.
Does Python support a way to import everything in a namespace / package? Any help would be appreciated.
Thanks in advance.
Yes, you can roll this using pkgutil.
Here's an example that lists all packages under twisted (except tests), and imports them:
# -*- Mode: Python -*-
# vi:si:et:sw=4:sts=4:ts=4
import pkgutil
import twisted
for importer, modname, ispkg in pkgutil.walk_packages(
path=twisted.__path__,
prefix=twisted.__name__+'.',
onerror=lambda x: None):
# skip tests
if modname.find('test') > -1:
continue
print(modname)
# gloss over import errors
try:
__import__(modname)
except:
print 'Failed importing', modname
pass
# show that we actually imported all these, by showing one subpackage is imported
print twisted.python
I have to agree with the other posters that star imports are a bad idea.
No. It is possible to set up SecondTestProject to automatically import everything in its submodules, by putting code in __init__.py to do the from ... import * you mention. It's also possible to automate this to some extent using the __import__ function and/or the imp module. But there is no quick and easy way to take a package that isn't set up this way and make it work this way.
It's probably not a good idea anyway. If you have 50 modules, importing everything from all of them into your global namespace is going to cause a proliferation of names, and very likely conflicts among those names.
As other had put it - it might not be a good idea. But there are ways of keeping your namespaces and therefore avoiding naming conflicts - and having all the modules/sub-packages in a module available to the package user with a single import.
Let's suppose I have a package named "pack", within it a module named "a.py" defining some "b" variable. All I want to do is :
>>> import pack
>>> pack.a.b
1
One way of doing this is to put in pack/__init__.py a line that says
import a - thus in your case you'd need fifty such lines, and keep them up to date.
Not that bad.
However, the documentation at http://docs.python.org/tutorial/modules.html#importing-from-a-package - says that if you have a string list named __all__ in your __init__.py file, all module/sub-package names in that list are imported when one does from pack import *
That alone would half-work - but would require users of your package to perform the not-recommended "from x import *" form.
But -- you can do the "... import *" inside __init__.py itself, after defining the __all__ variable - so all you have to do is to keep the __all__ up to date:
With the TestProjectNamespace/__init__.py being like this:
__all__ = ["Module_A", "Module_B", ...]
from TestProjectNamespace import *
your users would have
TestProjectNamespace.Module_A (and others) available upon import of TestProjectNamespace.
And, of course - you could automate the creation of __all__ - it is just a variable, after all - but I would not recommend that.
Does Python support a way to import everything in a namespace / package?
No. A package is not a super-module -- it's a collection of modules grouped together.
At least part of the reason is that it's not trivial to determine what 'everything' means inside a folder: there are problems like network drives, soft links, hard links, ...