Having already use flat packages, I was not expecting the issue I encountered with nested packages. Here is…
Directory layout
dir
|
+-- test.py
|
+-- package
|
+-- __init__.py
|
+-- subpackage
|
+-- __init__.py
|
+-- module.py
Content of init.py
Both package/__init__.py and package/subpackage/__init__.py are empty.
Content of module.py
# file `package/subpackage/module.py`
attribute1 = "value 1"
attribute2 = "value 2"
attribute3 = "value 3"
# and as many more as you want...
Content of test.py (3 versions)
Version 1
# file test.py
from package.subpackage.module import *
print attribute1 # OK
That's the bad and unsafe way of importing things (import all in a bulk), but it works.
Version 2
# file test.py
import package.subpackage.module
from package.subpackage import module # Alternative
from module import attribute1
A safer way to import, item by item, but it fails, Python don't want this: fails with the message: "No module named module". However …
# file test.py
import package.subpackage.module
from package.subpackage import module # Alternative
print module # Surprise here
… says <module 'package.subpackage.module' from '...'>. So that's a module, but that's not a module /-P 8-O ... uh
Version 3
# file test.py v3
from package.subpackage.module import attribute1
print attribute1 # OK
This one works. So you are either forced to use the overkill prefix all the time or use the unsafe way as in version #1 and disallowed by Python to use the safe handy way? The better way, which is safe and avoid unecessary long prefix is the only one which Python reject? Is this because it loves import * or because it loves overlong prefixes (which does not help to enforce this practice)?.
Sorry for the hard words, but that's two days I trying to work around this stupid‑like behavior. Unless I was totally wrong somewhere, this will leave me with a feeling something is really broken in Python's model of package and sub‑packages.
Notes
I don't want to rely on sys.path, to avoid global side effects, nor on *.pth files, which are just another way to play with sys.path with the same global effets. For the solution to be clean, it has to be local only. Either Python is able to handle subpackage, either it's not, but it should not require to play with global configuration to be able to handle local stuff.
I also tried use imports in package/subpackage/__init__.py, but it solved nothing, it do the same, and complains subpackage is not a known module, while print subpackage says it's a module (weird behavior, again).
May be I'm entirely wrong tough (the option I would prefer), but this make me feel a lot disappointed about Python.
Any other known way beside of the three I tried? Something I don't know about?
(sigh)
----- %< ----- edit ----- >% -----
Conclusion so far (after people's comments)
There is nothing like real sub‑package in Python, as all package references goes to a global dictionnary, only, which means there's no local dictionary, which implies there's is no way to manage local package reference.
You have to either use full prefix or short prefix or alias. As in:
Full prefix version
from package.subpackage.module import attribute1
# An repeat it again an again
# But after that, you can simply:
use_of (attribute1)
Short prefix version (but repeated prefix)
from package.subpackage import module
# Short but then you have to do:
use_of (module.attribute1)
# and repeat the prefix at every use place
Or else, a variation of the above.
from package.subpackage import module as m
use_of (m.attribute1)
# `m` is a shorter prefix, but you could as well
# define a more meaningful name after the context
Factorized version
If you don't mind about importing multiple entity all at once in a batch, you can:
from package.subpackage.module import attribute1, attribute2
# and etc.
Not in my first favorite taste (I prefer to have one import statement per imported entity), but may be the one I will personally favor.
Update (2012-09-14):
Finally appears to be OK in practice, except with a comment about the layout. Instead of the above, I used:
from package.subpackage.module import (
attribute1,
attribute2,
attribute3,
...) # and etc.
You seem to be misunderstanding how import searches for modules. When you use an import statement it always searches the actual module path (and/or sys.modules); it doesn't make use of module objects in the local namespace that exist because of previous imports. When you do:
import package.subpackage.module
from package.subpackage import module
from module import attribute1
The second line looks for a package called package.subpackage and imports module from that package. This line has no effect on the third line. The third line just looks for a module called module and doesn't find one. It doesn't "re-use" the object called module that you got from the line above.
In other words from someModule import ... doesn't mean "from the module called someModule that I imported earlier..." it means "from the module named someModule that you find on sys.path...". There is no way to "incrementally" build up a module's path by importing the packages that lead to it. You always have to refer to the entire module name when importing.
It's not clear what you're trying to achieve. If you only want to import the particular object attribute1, just do from package.subpackage.module import attribute1 and be done with it. You need never worry about the long package.subpackage.module once you've imported the name you want from it.
If you do want to have access to the module to access other names later, then you can do from package.subpackage import module and, as you've seen you can then do module.attribute1 and so on as much as you like.
If you want both --- that is, if you want attribute1 directly accessible and you want module accessible, just do both of the above:
from package.subpackage import module
from package.subpackage.module import attribute1
attribute1 # works
module.someOtherAttribute # also works
If you don't like typing package.subpackage even twice, you can just manually create a local reference to attribute1:
from package.subpackage import module
attribute1 = module.attribute1
attribute1 # works
module.someOtherAttribute #also works
The reason #2 fails is because sys.modules['module'] does not exist (the import routine has its own scope, and cannot see the module local name), and there's no module module or package on-disk. Note that you can separate multiple imported names by commas.
from package.subpackage.module import attribute1, attribute2, attribute3
Also:
from package.subpackage import module
print module.attribute1
If all you're trying to do is to get attribute1 in your global namespace, version 3 seems just fine. Why is it overkill prefix ?
In version 2, instead of
from module import attribute1
you can do
attribute1 = module.attribute1
Related
I would like a way to detect if my module was executed directly, as in import module or from module import * rather than by import module.submodule (which also executes module), and have this information accessible in module's __init__.py.
Here is a use case:
In Python, a common idiom is to add import statement in a module's __init__.py file, such as to "flatten" the module's namespace and make its submodules accessible directly. Unfortunately, doing so can make loading a specific submodule very slow, as all other siblings imported in __init__.py will also execute.
For instance:
module/
__init__.py
submodule/
__init__.py
...
sibling/
__init__.py
...
By adding to module/__init__.py:
from .submodule import *
from .sibling import *
It is now possible for users of the module to access definitions in submodules without knowing the details of the package structure (i.e. from module import SomeClass, where SomeClass is defined somewhere in submodule and exposed in its own __init__.py file).
However, if I now run submodule directly (as in import module.submodule, by calling python3 -m module.submodule, or even indirectly via pytest) I will also, unavoidably, execute sibling! If sibling is large, this can slow things down for no reason.
I would instead like to write module/__init__.py something like:
if __???__ == 'module':
from .submodule import *
from .sibling import *
Where __???__ gives me the fully qualified name of the import. Any similar mechanism would also work, although I'm mostly interested in the general case (detecting direct executing) rather than this specific example.
What is being desired is will result in undefined behavior (in the sense whether or not the flattened names be importable from module) when we consider how the import system actually works, if it were actually possible.
Hypothetically, if what you want to achieve is possible, where some __dunder__ that will disambiguate which import statement was used to import module/__init__.py (e.g. import module and from module import *, vs import module.submodule. For the first case, module may trigger the subsequent (slow) import to produce a "flattened" version of the desired imports, while the latter case (import module.submodule) will avoid that and thus module will not contain any assignments of the "flattened" imports.
To illustrate the example a bit more, say one may import SiblingClass from module.sibling.SiblingClass by simply doing from module import SiblingClass as the module/__init__.py file executes from .sibling import * statement to create that binding. But then, if executing import module.submodule resulting in the avoidance of that flatten import, we get the following scenario:
import module.submodule
# module.submodule gets imported
from module import SiblingClass
# ImportError will occur
Why is that? This is simply due to how Python imports a file - the source file is executed in its entirety once to assign imports, function and class declarations to the designated names, and be registered to sys.modules under its import name. Importing the module again will not execute the file again, thus if the from .sibling import * statement was not executed during its initial import (i.e. import module.submodule), it will never be executed again during subsequent import of the same module, as the copy produced by the initial import assigned to its module entry in sys.module is returned (unless the module was reloaded manually, the code for the module will be executed again).
You may verify this fact by putting in a print statement into a file, import the corresponding module to see the output produced, and see that no further output will be produced on subsequent import of that module (related: What happens when a module is imported twice?).
Effectively, the desired functionality as described in the question cannot be implemented in Python.
A related thread on this topic: How to only import sub module without exec __init__.py in the package
This is not a complete solution, but standalone py.test (ignore __init__.py files) proposes setting a global flag to detect when in test. This corrects the problem for tests at least, provided the concerned modules don't call each other.
i've run through many posts about this, but still doesn't seem to work. The deal is pretty cut. I've the got the following hierarchy.
main.py
DirA/
__init__.py
hello.py
DirB/
__init__.py
foo.py
bla.py
lol.py
The__init__.py at DirA is empty. The respective one at DirB just contains the foo module.
__all__.py = ["foo"]
The main.py has the following code
import DirA
import DirB
hey() #Def written at hello.py
foolish1() #Def written at foo.py
foolish2() #Def written at foo.py
Long story short, I got NameError: name 'foo' is not defined. Any ideas? Thanks in advance.
You only get what you import. Therefore, in you main, you only get DirA and DirB. You would use them in one of those ways:
import DirA
DirA.something_in_init_py()
# Importing hello:
import DirA.hello
DirA.hello.something_in_hello_py()
# Using a named import:
from DirA.hello import something_in_hello_py
something_in_hello_py()
And in DirB, just make the __init__.py empty as well. The only use of __all__ is for when you want to import *, which you don't want because, as they say, explicit is better than implicit.
But in case you are curious, it would work this way:
from DirB import *
something_in_dirb()
By default the import * will import everything it can find that does not start with an underscore. Specifying a __all__ restricts what it imported to the names defined in __all__. See this question for more details.
Edit: about init.
The __init__.py is not really connected to the importing stuff. It is just a special file with the following properties:
Its existence means the directory is a python package, with several modules in it. If it does not exist, python will refuse to import anything from the directory.
It will always be loaded before loading anything else in the directory.
Its content will be available as the package itself.
Just try it put this in DirA/__init__.py:
foo = 42
Now, in your main:
from DirA import foo
print(foo) # 42
It can be useful, because you can import some of your submodules in the __init__.py to hide the inner structure of your package. Suppose you build an application with classes Author, Book and Review. To make it easier to read, you give each class its own file in a package. Now in your main, you have to import the full path:
from myapp.author import Author
from myapp.book import Book
from myapp.review import Review
Clearly not optimal. Now suppose you put those exact lines above in your __init__.py, you may simplify you main like this:
from myapp import Author, Book, Review
Python will load the __init__.py, which will in turn load all submodules and import the classes, making them available on the package. Now your main does not need to know where the classes are actually implemented.
Have you tried something like this:
One way
from DirA import hello
Another way
from DirA.hello import hey
If those don't work then append a new system path
You need to import the function itself:
How to call a function from another file in Python?
In your case:
from DirA import foolish1, foolish2
I'm trying to do the following in python 2.6.
my_module.py:-
from another_module import another_factory
def my_factory(name):
pass
another_module.py:-
from my_module import my_factory
def another_factory(name):
pass
Both modules in the same folder.
It gives me the error:
Error: cannot import name my_factory
As seen from the comments, you are trying to do a circle import which is impossible.
If in your module A you try to import something from the module B, and when loading the module B (to satisfy this dependency) you are trying to import something from the module A, you are where you started and you got a circle import: A needs B and B needs A!!, it is somehow like saying that A needs A, which is quite unlogic.
For instance:
# moduleA
from moduleB import functionB
...
So the interpreter tries to load the moduleB, which looks like the following:
# moduleB
from moduleA import functionA
...
And goes back to the moduleA, which tries again to import B, and, etc. Therefore python just raises the error and stops the insanity for a greater good.
Dependencies don't work like this. Define what module needs the other one, and just do a simple import. In your example, it seems that another_module needs my_module, so change my_module and eliminate the dependency on another_module.
If both modules actually need each other, it is a clear sign that they belong to the same logical concept, and should be merged.
PD: in some cases to avoid huge files, you can split a logical unit in two, and to avoid the circle dependencies, you write your imports inside of the functions (which are not executed at load time), so that there is not a circle. This is however in general something to avoid.
The real question is... do you consider each file as a module or are they part of a package ?
Trying to import modules outside a package is sometimes painful. You should rather build a package by simply creating an empty __init__.py module in the directory. Though, if you have
__init__.py
my_module.py
another_module.py
If you have te following function in my_module.py,
def my_factory(x):
return x * x
You should be able to access the my_factory() function from another_module.py by writing this :
from my_module import my_factory
But, if you don't have the __init__.py file/module, the import function will be (somehow) lost and will only use the sys.path for searching other modules. You may then add the following lines (before the import) in the another_module.py file :
sys.path.append(os.path.dirname(os.path.expanduser('.')))
You may also use the various packages available to help importing modules, like imp or import_file (see the documentation). Or you can decide to use load_source (also see the doc : https://docs.python.org/2/library/imp.html)
I got the following files
1 ./run.py
2 ./code/util.py
3 ./code/__init__.py
and inside util.py I have
def funA():
print 'Hello World !'
and inside ./code/init.py I have
__all__=['util'];
from util import *
now I open python prompt (actually, ipython) in the current directory then I type
from code import *
and all I've got is the util module
util module <module 'code.util' from 'code/util.pyc'>
and I have to include the package name in order to use funA().
I expected that funA is now in my namespace and I can use it without the package name code.funA . However, this is not the case and I am wondering where the problem is.
I guess I am still somewhat confused to how exactly __ init __.py should be used.
The purpose of __all__ (as documented) is to indicate that you want only the names listed there to be available via from mymodule import *. By specifying only 'util', you are explicitly telling your package to not make anything but util available. If you remove that __all__, then everything you import from within util will also be available in code, and so if you do from code import *, then everything from util will also be available.
Whether this is a good idea is another matter. Importing * often leads to confusion.
all.
I'd think that this could be answered easily, but it isn't. As long as I've been searching for an answer, I keep thinking that I'm overlooking something simple.
I have a python workspace with the following package structure:
MyTestProject
/src
/TestProjectNamespace
__init__.py
Module_A.py
Module_B.py
SecondTestProject
/src
/SecondTestProjectNamespace
__init__.py
Module_1.py
Module_2.py
...
Module_10.py
Note that MyTestProjectNamespace has a reference to SecondTestProjectNamespace.
In MyTestProjectNamespace, I need to import everything in SecondTestProjectNamespace. I could import one module at a time with the following statement(s):
from SecondTestProjectNamespace.Module_A import *
from SecondTestProjectNamespace.Module_B import *
...but this isn't practical if the SecondTestProject has 50 modules in it.
Does Python support a way to import everything in a namespace / package? Any help would be appreciated.
Thanks in advance.
Yes, you can roll this using pkgutil.
Here's an example that lists all packages under twisted (except tests), and imports them:
# -*- Mode: Python -*-
# vi:si:et:sw=4:sts=4:ts=4
import pkgutil
import twisted
for importer, modname, ispkg in pkgutil.walk_packages(
path=twisted.__path__,
prefix=twisted.__name__+'.',
onerror=lambda x: None):
# skip tests
if modname.find('test') > -1:
continue
print(modname)
# gloss over import errors
try:
__import__(modname)
except:
print 'Failed importing', modname
pass
# show that we actually imported all these, by showing one subpackage is imported
print twisted.python
I have to agree with the other posters that star imports are a bad idea.
No. It is possible to set up SecondTestProject to automatically import everything in its submodules, by putting code in __init__.py to do the from ... import * you mention. It's also possible to automate this to some extent using the __import__ function and/or the imp module. But there is no quick and easy way to take a package that isn't set up this way and make it work this way.
It's probably not a good idea anyway. If you have 50 modules, importing everything from all of them into your global namespace is going to cause a proliferation of names, and very likely conflicts among those names.
As other had put it - it might not be a good idea. But there are ways of keeping your namespaces and therefore avoiding naming conflicts - and having all the modules/sub-packages in a module available to the package user with a single import.
Let's suppose I have a package named "pack", within it a module named "a.py" defining some "b" variable. All I want to do is :
>>> import pack
>>> pack.a.b
1
One way of doing this is to put in pack/__init__.py a line that says
import a - thus in your case you'd need fifty such lines, and keep them up to date.
Not that bad.
However, the documentation at http://docs.python.org/tutorial/modules.html#importing-from-a-package - says that if you have a string list named __all__ in your __init__.py file, all module/sub-package names in that list are imported when one does from pack import *
That alone would half-work - but would require users of your package to perform the not-recommended "from x import *" form.
But -- you can do the "... import *" inside __init__.py itself, after defining the __all__ variable - so all you have to do is to keep the __all__ up to date:
With the TestProjectNamespace/__init__.py being like this:
__all__ = ["Module_A", "Module_B", ...]
from TestProjectNamespace import *
your users would have
TestProjectNamespace.Module_A (and others) available upon import of TestProjectNamespace.
And, of course - you could automate the creation of __all__ - it is just a variable, after all - but I would not recommend that.
Does Python support a way to import everything in a namespace / package?
No. A package is not a super-module -- it's a collection of modules grouped together.
At least part of the reason is that it's not trivial to determine what 'everything' means inside a folder: there are problems like network drives, soft links, hard links, ...