Say Ansible was installed by means of "pip install ansible". Right after the install the following import statement succeeds:
from ansible.module_utils.basic import AnsibleModule
Now, a local package named "ansible.module_utils.custom" is created. The directory structure:
ansible/
__init__.py
module_utils/
__init__.py
custom/
__init__.py
utils.py
As soon as this is put in place the aforementioned import statement fails. Claiming "basic" is undefined. The local package does indeed not declare a "basic" subpackage. Only the installed Ansible library does. It seems Python limited its search to the local package only.
I was under the impression Python would consider the complete system path before giving up on finding code. That it would backtrack out of the local package and finally hit the installed Ansible library.
Is this an incorrect assumption ? If so, is it possible at all to make the local package to coexist with the installed package ?
How Import works
import abc
The first thing Python will do is look up the name abc in sys.modules. This is a cache of all modules that have been previously imported.
If the name isn’t found in the module cache, Python will proceed to search through a list of built-in modules. These are modules that come pre-installed with Python and can be found in the Python Standard Library. If the name still isn’t found in the built-in modules, Python then searches for it in a list of directories defined by sys.path. This list usually includes the current directory, which is searched first.
When Python finds the module, it binds it to a name in the local scope. This means that abc is now defined and can be used in the current file without throwing a NameError.
If the name is never found, you’ll get a ModuleNotFoundError. You can find out more about imports in the Python documentation here!
Related
I am not seeing an answer to this out there, so apologies if this is a duplicate. Basically, I am trying to understand how to force my interpreter (2.7) to import a module from site packages if there is a conflict. For example imagine you are running python from a directory (top_level) that has the following structure:
top_level
----cool_mod
----init.py
----sweet_module.py
but you have already installed sweet module to site packages. When in this directory (but no others) if you run:
from cool_mod.sweet_module import *
you will import from the local module, not the global one. Can I change this somehow?
This situation might arise from the case:
top_level
setup.py
----cool_mod
----init.py
----sweet_module.py
You can run cool_mod.sweet_module before installing if you working directory is top_level. But after installing you can import cool_mod.sweet_module from anywhere. However, if you ever import from this directory, even after installation, you still import the local copy
Inserting the site package directory at the begining of sys.path, and then import.
Or, use imp.load_source to load a module from specified path.
Maybe it's not possible (I'm more used to Ruby, where this sort of thing is fine). I'm writing a library that provides additional functionality to docker-py, which provides the docker package, so you just import docker and then you get access to docker.Client etc.
Because it seemed a logical naming scheme, I wanted users to pull in my project with import docker.mymodule, so I've created a directory called docker with an __init__.py, and mymodule.py inside it.
When I try to access docker.Client, Python can't see it, as if my docker package has hidden it:
import docker
import docker.mymodule
docker.Client() # AttributeError: 'module' object has no attribute 'Client'
Is this possible, or do all top-level package names have to differ between source trees?
This would only be possible if docker was set up as a namespace package (which it isn't).
See zope.schema, zope.interface, etc. for an example of a namespace package (zope is the namespace package here). Because zope is declared as a namespace package in setup.py, it means that zope doesn't refer to a particular module or directory on the file system, but is a namespace shared by several packages. This also means that the result of import zope is pretty much undefined - it will simply import the top-level module of the first zope.* package found in the import path.
Therefore, when dealing with namespace packages, you need to explicitely import a specific one with import zope.schema or from zope import schema.
Unfortunately, namespace packages aren't that well documented. As noted by #Bakuriu in the comment, these are some resources that contain some helpful information:
Stackoverflow: How do I create a namespace package in Python?
Built-in support for namespace packages in Python 3.3
Namespace packages in the setuptools documentation
Post about namespace packages at sourceweaver.com
Okay, so in the past, I've made my own Python packages with Python 2.x (most recently, 2.7.5). It has worked fine. Let me explain how I did that, for reference:
Make a directory within the working directory. We'll call it myPackage.
Make a file called __init__.py in the directory myPackage.
Make sure all the modules that you want to be part of the package are imported within __init__.py. These modules are typically in the myPackage folder.
From a Python program in the working directory, type import myPackage (and it imports fine, and is usable).
However, in Python 3, I get errors with that. (ImportError: No module named 'Whatever the first imported module is')
I researched the problem and found the following:
Starred imports don't work in Python 3.3.
The __init__.py file is not required in Python 3.3.
So, I removed the stars from my imports, and leaving the __init__.py file in, I still got errors (ImportError: No module named 'Whatever the first imported module is'). So, I removed the __init__.py file, and I don't get any errors, but my package doesn't include any of my modules.
Okay, so I discovered by doing a web search for python3 __init__.py or some such that I can do the following, although I don't have any clue if this is the standard way of doing things:
In the modules in the package, make sure there are no plain imports (not just no starred ones). Only do from myModule import stuff. However, you need to put a . in front of myModule: e.g. from .myModule import stuff. Then, I can import myPackage.oneOfMyModules
I found that by following this rule in the __init__.py file, it also works.
Once again, I don't know if this is how it's supposed to work, but it seems to work.
I found this page that is supposed to have something to do with the changes in packages in Python 3.something, but I'm not sure how it relates to what I'm doing:
http://legacy.python.org/dev/peps/pep-0420/
So, what is the standard way to do this? Where is it documented (actually saying the syntax)? Is the way I'm doing it right? Can I do regular imports instead of from package import module?
After analyzing some Python 3 packages installed on my system (I should have tried that to start with!) I discovered that they often seem to do things a little differently. Instead of just doing from .myModule import stuff they would do from myPackage.myModule import stuff (inside the modules in the package). So, that works, too, I suppose, and seems to be more frequently used.
I have a package in my PYTHONPATH that looks something like this:
package/
__init__.py
module.py
print 'Loading module'
If I'm running Python from the package/ directory (or writing another module in this directory) and type
import module
it loads module.py and prints out "Loading module" as expected. However, if I then type
from package import module
it loads module.py and prints "Loading module" again, which I don't expect. What's the rationale for this?
Note: I think I understand technically why Python is doing this, because the sys.modules key for import module is just "module", but for from package import module it's "package.module". So I guess what I want to know is why the key is different here -- why isn't the file's path name used as the key so that Python does what one expects here?
Effectively, by running code from the package directory, you've misconfigured Python. You shouldn't have put that directory on sys.path, since it's inside a package.
Python doesn't use the filename as the key because it's not importing a file, it's importing a module. Allowing people to do 'import c:\jim\my files\projects\code\stuff' would encourage all kinds of nastiness.
Consider this case instead: what if you were in ~/foo/package/ and ~/bar were on PYTHONPATH - but ~/bar is just a symlink to ~/foo? Do you expect Python to resolve, then deduplicate the symbolic link for you? What if you put a relative directory on PYTHONPATH, then change directories? What if 'foo.py' is a symlink to 'bar.py'? Do you expect both of those to be de-duplicated too? What if they're not symlinks, but just exact copies? Adding complex rules to try to do something convenient in ambiguous circumstances means it does something highly inconvenient for other people. (Python zen 12: in the face of ambiguity, refuse the temptation to guess.)
Python does something simple here, and it's your responsibility to make sure that the environment is set up correctly. Now, you could argue that it's not a very good idea to put the current directory on PYTHONPATH by default - I might even agree with you - but given that it is there, it should follow the same consistent set of rules that other path entries do. If it's intended to be run from an arbitrary directory, your application can always remove the current directory from sys.path by starting off with sys.path.remove('').
It is a minor defect of the current module system.
When importing module, you do it from the current namespace, which has no name. the values inside this namespace are the same as those in package, but the interpreter cannot know it.
When importing package.module, you import module from the package namespace.
This the reason, that the main.py should be outside the package forlder.
Many modules have this organisation :
package /
main.py
package /
sub_package1/
sub_package2/
sub_package3/
module1.py
module2.py
Calling only main.py make sure the namespaces are correctly set, aka the current namespace is main.py's. Its makes impossible to call import module1.py in module2.py. You'ld need to call import package.module1. Makes things simpler and homogeneous.
And yes, import the current folder as the current nameless folder was a bad idea.
It is a PITA if you go beyond a few scripts. But as Python started there, it was not completely senseless.
I have a python package called zypp. It is generated via swig and the rpm package (called python-zypp) puts it in:
rpm -ql python-zypp
/usr/lib64/python2.6/site-packages/_zypp.so
/usr/lib64/python2.6/site-packages/zypp.py
Now, I have a different project which provides an additional sets of APIs. Pure python. Plus some scripts.
The layout is:
bin/script1
python
python/zypp
python/zypp/plugins.py
python/zypp/__init__.py
plugins.py contains a Plugin class. I intended to put this in an rpm, and put it into
/usr/lib64/python2.6/site-packages/zypp/plugins.py
script1 uses this Plugin class. But as I test it running from git, I would like it to find the module from git too if it is not installed. So it has something like:
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), '../python'))
from zypp.plugins import Plugin
However, it seems that if python-zypp is installed on /usr/lib64/python2.6/site-packages/zypp.py, then script1 won't find the plugins submodule anymore. If I uninstall python-zypp, it does.
So my question is if it is possible to extend a module by adding submodules, being the submodules being located in a different load path. Or will they always clash?
An analogy would be, I have a module foo. And I provide foo.extras in a different load path (which may use foo indeed). The script won't find foo.extras if foo is found first in the system load path. If I only use the custom load path, the script may not find foo module if foo.extras uses it.
I have more experience with ruby, but in ruby I could have installed:
/usr/lib64/ruby/gems/1.8/gems/foo-1.0/lib/foo/*
And I could have in my script:
bin/script
lib/foo/extras/*
I could do in script:
$: << File.join(File.dirname(__FILE__), "../lib"
and then my script could
require foo
require foo/extras
No mater if foo/extras is installed on the system or in the custom load path. They don't clash.
The other way around, I found out that with PYTHONPATH the local zypp.plugins is found first. But then the installed zypp module is not found:
import zypp # works, but seems to import the local one
from zypp.plugins import Plugin # works, PYTHONPATH finds it first
repoinfo = zypp.RepoInfo() # does not work
If I understand your question correctly, you want to use the development version of that module instead of the installed module. Therefore, you can use
PYTHONPATH
From the Module Search Path documentation:
When a module named spam is imported, the interpreter searches for a file named spam.py in the current directory, and then in the list of directories specified by the environment variable PYTHONPATH. This has the same syntax as the shell variable PATH, that is, a list of directory names. When PYTHONPATH is not set, or when the file is not found there, the search continues in an installation-dependent default path; on Unix, this is usually .:/usr/local/lib/python.
So, if the GIT tree of the module directory was "/home/username/some/path", you would change the PYTHONPATH to "/home/username/some/path". Or if the PYTHONPATH variable is already in use, you would append ":/home/username/some/path" to it (note the colon separator). In order to make this permanent, add the line "PYTHONPATH=value" to the file "/etc/environment".
sys.path.insert
If you have a start script for your program, you could override the module search path using sys.path.insert(0, "somepath"). This is similar to the sys.path.append call you described but inserts the path into the beginning of the list.