I must be missing something very basic about building a package in Python. When I create a package following the guidelines of https://docs.python.org/2/tutorial/modules.html#packages and import it, Python does not find any of the modules. For example, say I create the package holygrail with the following structure:
holygrail/
__init__.py
knights.py
I leave __init__.py empty because the docs say that I can and I'm just trying to make a basic package to start. In knights.py I have:
def say():
print 'Ni!'
If I try import holygrail, Python doesn't give any errors, but holygrail.knights.say() results in Python telling me that the "'module' object [holygrail] has no attribute 'knights'." However, if I specifically import knights via from holygrail import knights, then knights.say() works. In addition, holygrail.knights.say() then also works.
I tried adding the line
__all__ = ['knights']
in the __init__.py file, but this did not change the behavior.
How do I construct a package such that import package loads in all of the modules, allowing statements like package.module.function()?
Python does not implicitly import the whole package hierarchy. You have to be explicit about what to import on what level of the package using the __init__.py files.
When you set __all__ = ['knights'] in the __init__.py it works only for the import all statements for that module, e.g.:
>>> import holygrail
>>> holygrail.knights.say()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'knights'
>>> from holygrail import *
>>> knights.say()
Ni!
It can also act as a filter on import all, importing from the module only what's specified.
To get knights automatically imported on import holygrail you have to put import knights or from . import knights (intra-package or relative import) to the __init__.py. You'll have to do the same for every module explicitly.
Add import knights into __init__.py.
The link you provided does state "In the simplest case, __init__.py can just be an empty file..." Your example is not the simplest case, that's all.
Related
I have a custom Python library ("common") that is being imported and used from several Python projects.
That central library has the following structure:
/common
/__init__.py
/wrapper.py
/util
/__init__.py
/misc.py
Our custom library resides in a central place /data/Development/Python, so in my Python projects I have an .env file in order to include our lib:
PYTHONPATH="/data/Development/Python"
That works fine, I can for example do something like:
from common.util import misc
However, now I want to make use of a class MyClass within common/wrapper.py from the code in common/util/misc.py. Thus I tried the following import in misc.py:
from ..wrapper import MyClass
But that leads to the following error:
Exception has occurred: ImportError
cannot import name 'MyClass'
Any ideas what I am doing wrong here?
PS: When I do an from .. import wrapper instead and then from code I use wrapper.MyClass, then it works fine. Does that make any sense?
It's finding wrapper, otherwise you'd get a different error:
>>> from wibble import myclass
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'wibble'
So it seems wrapper does not contain MyClass. Typo?
"However, now I want to make use of a class MyClass within common/wrapper.py from the code in common/util/misc.py. Thus I tried the following import in misc.py:"
If you do export PYTHONPATH=~/common
In order to import MyClass you will do:
from wrapper import MyClass
Because you have added common to your root folder, utils is already in your path since it is in you root folder.
Similarly, if you wanted to import from wrapper you would do from utils.misc import ThisClass since utils.misc is already in the root folder common
I have a top level namespace, working like an organization namespace. Let's name that myorg, where it's __init__.py looks like this :
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
This is being deployed under /opt where our Python lives, along with some developed python libraries - let's say mylib .
So on a Python2.7 interpreter this works
>>> import myorg
>>> import myorg.mylib
Apart from the libraries, we have some client code that gets deployed elsewhere in the system, in /bb/bin. For example for clientA I can have :
>>> import sys
>>> sys.path.append('/bb/bin')
>>> import clientA
and this works.
However, due to how our code is structured, we share the same namespace myorg - so clientB is also under myorg and there is a /bb/bin/myorg/__init__.py as well, with the same contants as the one in the /opt path.
Question is :
why this one works -
>>> import sys
>>> sys.path.append('/bb/bin')
>>> import myorg.clientB
but this one doesn't :
>>> import myorg # import from /opt location
>>> import sys
>>> sys.path.append('/bb/bin')
>>> import myorg.clientB # it should find the module under myorg in /bb/bin
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named clientB
This also does not work :
>>> import myorg.mylib
>>> import sys
>>> sys.path.insert(0,'/bb/bin') # Adding the /bb/bin path first
>>> import myorg.clientB
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named clientB
So, it's like the myorg namespace, if imported from the /opt then it will always try and find the modules under there.
Why is that? Can someone explain how to fix that issue and why that is happening?
I also dont understand why prepending that /bb/bin to my path does not make it work.
TL;DR
My intended behaviour or what I was hoping for is :
search in the /opt/ for the module
if not found go to other e.g. /bb/bin
This works on first case, but it seems if myorg has been imported from /opt then it only looks for modules inside there.
That's because "myorg" is already loaded - python loads modules/packages only once. If you load myorg (from /opt) first (or any sub-module as Python loads all "steps" of the import), then any myorg.something will be relative to that.
Can you try import as? PEP describing it.
If I understand how python works, this should work:
import myorg as whateveryouwant # import from /opt location
import sys
sys.path.append('/bb/bin')
import myorg.clientB
EDIT:
Noticed your comment that it doesn't work. :(
You can also try just appending /bb/bin/myorg and then importing just clientB instead of myorg.clientB. It's ugly but should work.
I'm baffled by the importing dynamics in __init__.py.
Say I have this structure:
package
├── __init__.py
└── subpackage
├── __init__.py
└── dostuff.py
I would like to import things in dostuff.py. I could do it like this: from package.subpackage.dostuff import thefunction, but I would like to remove the subpackage level in the import statement, so it would look like this:
from package.dostuff import thefunction
I tried putting this in package/__init__.py:
from .subpackage import dostuff
And what I don't understand is this:
# doing this works:
from package import dostuff
dostuff.thefunction()
# but this doesn't work:
from package.dostuff import thefunction
# ModuleNotFoundError: No module named 'package.dostuff'
Why is that, and how can I make from package.dostuff import thefunction work?
The only way I see to make what you intend would be to actually create a package/dostuff.py module and import all you need in it as from .subpackage.dostuff import thefunction.
The point is that when you use from .subpackage import dostuff in package/__init__.py, you do not rename the original module.
To be more explicit, here is an example of use with both your import and a package/dostuff.py file:
# We import the dostuff link from package
>>> from package import dostuff
>>> dostuff
<module 'package.subpackage.dostuff' from '/tmp/test/package/subpackage/dostuff.py'>
# We use our custom package.dostuff
>>> from package.dostuff import thefunction
>>> package.dostuff
<module 'package.dostuff' from '/tmp/test/package/dostuff.py'>
>>> from package import dostuff
>>> dostuff
<module 'package.dostuff' from '/tmp/test/package/dostuff.py'>
# The loaded function is the same
>>> dostuff.thefunction
<function thefunction at 0x7f95403d2730>
>>> package.dostuff.thefunction
<function thefunction at 0x7f95403d2730>
A clearer way of putting this is:
from X import Y only works when X is an actual module path.
Y on the contrary can be any item imported in this module.
This also applies to packages with anything being declared in their __init__.py. Here you declare the module package.subpackage.dostuff in package, hence you can import it and use it.
But if you try to use the module for a direct import, it has to exist on the filesystem
Resources:
Python documentation about module management in the import system:
https://docs.python.org/3/reference/import.html#submodules.
Python import system search behavior:
https://docs.python.org/3/reference/import.html#searching
https://docs.python.org/3/glossary.html#term-qualified-name
https://docs.python.org/2.0/ref/import.html
I hope that makes it clearer
You can in fact fake this quite easily by fiddling with Python's sys.modules dict. The question is whether you do really need this or whether it might be good to spend a second thought on your package structure.
Personally, I would consider this bad style, because it applies magic to the module and package names and people who might use and extend your package will have a hard time figuring out what's going on there.
Following your structure above, add the following code to your package/__init__.py:
import sys
from .subpackage import dostuff
# This will be package.dostuff; just avoiding to hard-code it.
_pkg_name = f"{__name__}.{dostuff.__name__.rsplit('.', 1)[1]}"
if _pkg_name not in sys.modules.keys():
dostuff.__name__ = _pkg_name # Will have no effect; see below
sys.modules[_pkg_name] = dostuff
This imports the dostuff module from your subpackage to the scope of package, changes its module path and adds it to the imported modules. Essentially, this just copies the binding of your module to another import path where member memory addresses remain the same. You just duplicate the references:
import package
print(package.dostuff)
print(package.subpackage.dostuff)
print(package.dostuff.something_to_do)
print(package.subpackage.dostuff.something_to_do)
... yields
<module 'package.subpackage.dostuff' from '/path/package/subpackage/dostuff.py'>
<module 'package.subpackage.dostuff' from '/path/package/subpackage/dostuff.py'>
<function something_to_do at 0x1029b8ae8>
<function something_to_do at 0x1029b8ae8>
Note that
The module name package.subpackage.dostuff has not changed even though being updated in package/__init__.py
The function reference is the same: 0x1029b8ae8
Now, you can also go
from package.dostuff import something_to_do
something_to_do()
However, be cautious. Changing the imported modules during import of a module might have unintended side-effects (also the order of updating sys.modules and importing other subpackages or submodules from within package might be relevant). Usually, you buy extra work and extra complexity by applying such kind of "improvement". Better yet set up a proper package structure and stick to it.
I am currently doing a personal coding project and I am trying to build a module, but I don't know why my structure doesn't work the way it's supposed to:
\mainModule
__init__.py
main.py
\subModule_1
__init__.py
someCode_1.py
someCode_2.py
\subModule_2
__init__.py
otherCode.py
I want to be able to run the following code from main.py:
>>> from subModule_1 import someCode_1
>>> someCode_1.function()
"Hey, this works!"
>>> var = someCode_2.someClass("blahblahblah")
>>> var.classMethod1()
>>> "blah blah blah"
>>> from subModule2 import otherCode
>>> otherCode("runCode","#ff281ba0")
However, when I try to import someCode_1, for example, it returns an AttributeError, and I'm not really sure why. Is it to do with the __init__.py file?
REVISIONS
Minimal, Complete and verifiable (I hope...)
\mainDir
__init__.py # blank file
main.py
\subDir
__init__.py # blank file
codeFile.py
Using this...
#main.py file
import subDir
subDir.codeFile.function()
And this...
#codeFile.py file
def function():
return "something"
...it returns the same problem mentioned above**.
** The exact error is:
Traceback (most recent call last):
File "C:\...\mainDir\main.py", line 2, in <module>
subDir.codeFile.function()
AttributeError: module 'subDir' has no attribute 'codeFile'
Credits to #jonrsharpe: Thanks for showing me how to use Stack Overflow correctly.
You have two options to make this work.
Either this:
from subdir import codeFile
codeFile.function()
Or:
import subdir.codeFile
subdir.codeFile.function()
When you import subDir, it does three things:
executes the code in mainDir/subDir/__init__.py (i.e. in this case does nothing, because this file is empty)
imports the resulting module under the name subDir locally, which will in turn make it an attribute of the mainDir module;
registers the new import globally in the sys.modules dictionary (because the import is being performed from a parent module mainDir, the name is completed to 'mainDir.subDir' for the purposes of this registration);
What it does not do, because it hasn't been told to, is import subDir.codeFile. Therefore, the code in codeFile.py has not been run and the name codeFile has not yet been imported into the namespace of mainDir.subDir. Hence the AttributeError when trying to access it. If you were to add the following line to mainDir/subDir/__init__.py then it would work:
import codeFile
Specifically, this will:
run the code in codeFile.py
add the resulting module as an attribute of the mainDir.subDir module
store a reference to it as yet another entry in sys.modules, this time under the name mainDir.subDir.codeFile.
You could also achieve the same effect from higher up the module hierarchy, by saying import subDir, subDir.codeFile instead of just import subDir in your mainDir.main source file.
NB: When you test this from the command line or IDE, make sure that your current working directory (queried by os.getcwd(), changed using os.chdir(wherever) ) is neither mainDir nor subDir. Work from somewhere else—e.g. the parent directory of mainDir. Working from inside a module will lead to unexpected results.
i've run through many posts about this, but still doesn't seem to work. The deal is pretty cut. I've the got the following hierarchy.
main.py
DirA/
__init__.py
hello.py
DirB/
__init__.py
foo.py
bla.py
lol.py
The__init__.py at DirA is empty. The respective one at DirB just contains the foo module.
__all__.py = ["foo"]
The main.py has the following code
import DirA
import DirB
hey() #Def written at hello.py
foolish1() #Def written at foo.py
foolish2() #Def written at foo.py
Long story short, I got NameError: name 'foo' is not defined. Any ideas? Thanks in advance.
You only get what you import. Therefore, in you main, you only get DirA and DirB. You would use them in one of those ways:
import DirA
DirA.something_in_init_py()
# Importing hello:
import DirA.hello
DirA.hello.something_in_hello_py()
# Using a named import:
from DirA.hello import something_in_hello_py
something_in_hello_py()
And in DirB, just make the __init__.py empty as well. The only use of __all__ is for when you want to import *, which you don't want because, as they say, explicit is better than implicit.
But in case you are curious, it would work this way:
from DirB import *
something_in_dirb()
By default the import * will import everything it can find that does not start with an underscore. Specifying a __all__ restricts what it imported to the names defined in __all__. See this question for more details.
Edit: about init.
The __init__.py is not really connected to the importing stuff. It is just a special file with the following properties:
Its existence means the directory is a python package, with several modules in it. If it does not exist, python will refuse to import anything from the directory.
It will always be loaded before loading anything else in the directory.
Its content will be available as the package itself.
Just try it put this in DirA/__init__.py:
foo = 42
Now, in your main:
from DirA import foo
print(foo) # 42
It can be useful, because you can import some of your submodules in the __init__.py to hide the inner structure of your package. Suppose you build an application with classes Author, Book and Review. To make it easier to read, you give each class its own file in a package. Now in your main, you have to import the full path:
from myapp.author import Author
from myapp.book import Book
from myapp.review import Review
Clearly not optimal. Now suppose you put those exact lines above in your __init__.py, you may simplify you main like this:
from myapp import Author, Book, Review
Python will load the __init__.py, which will in turn load all submodules and import the classes, making them available on the package. Now your main does not need to know where the classes are actually implemented.
Have you tried something like this:
One way
from DirA import hello
Another way
from DirA.hello import hey
If those don't work then append a new system path
You need to import the function itself:
How to call a function from another file in Python?
In your case:
from DirA import foolish1, foolish2