When I did a dir to find the list of methods in boltons I got the below output
>>> import boltons
>>> dir(boltons)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
When I explicitly did
>>> from boltons.strutils import camel2under
>>> dir(boltons)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'strutils']
found that strutils getting added to attribute of boltons
Why is strutils not showing before explicit import?
From the docs on what dir does:
With an argument, attempt to return a list of valid attributes for
that object.
When we import the boltons package we can see that strutils is not an attribute on the boltons object. Therefore we do not expect it to show up in dir(boltons).
>>>import boltons
>>>getattr(boltons, 'strutils')
AttributeError: module 'boltons' has no attribute 'strutils'
The docs on importing submodules say:
For example, if package spam has a submodule foo, after importing spam.foo, spam will have an attribute foo which is bound to the submodule.
Importing a submodule creates an attribute on the package. In your example:
>>>import boltons
>>>getattr(boltons, 'strutils')
AttributeError: module 'boltons' has no attribute 'strutils'
>>>from boltons.strutils import camel2under
>>>getattr(boltons, 'strutils')
<module 'boltons.strutils' from '/usr/local/lib/python3.5/site-packages/boltons/strutils.py'>
Therefore in this case we do expect strutils to show up in dir(boltons)
Related
i want to implement sort of plugin architecture that dynamically loads modules and calls a function from them
for instance plugin code looks like (in file "foo_func.py")
foo_local = []
def foo_add(a: int, b: int) -> int:
c = a + b
foo_local.append(c)
return c
def foo_print():
print(foo_local)
i need to support two plugins with the same code but with different memory state, so i created directory structure like this:
<ROOT_PROJECT>
app.py
bar/
apple/
foo/
foo_func.py
__init__.py
orange/
foo/
foo_func.py
__init__.py
code in "apple" and "orange" folders is the same.
then in app file i try to load modules and invoke functions from them
import importlib
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
apple = importlib.import_module('bar.apple.foo')
orange = importlib.import_module('bar.orange.foo')
apple_foo = getattr(apple, 'foo_func')
orange_foo = getattr(orange, 'foo_func')
apple_foo_add_my = getattr(apple_foo, 'foo_add')
apple_foo_print_my = getattr(apple_foo, 'foo_print')
apple_foo_add_my(1, 2)
apple_foo_print_my()
and this works fine, but you see these import lines at the top
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
they are not used in code (even pycharm complains about it)
but if i try to comment code and run it - then failure
AttributeError: module 'bar.apple.foo' has no attribute 'foo_func'
why ?
I suppose normal plugins should deal only with "importlib.import_module" and "getattr" and it must be enough ?
what is wrong here ?
Let's switch completely to direct imports for this explanation, because:
import something.whatever as name
is the same as:
name = importlib.import_module("something.whatever")
So let's rewrite your apple code:
apple = importlib.import_module('bar.apple.foo')
apple_foo = getattr(apple, 'foo_func')
will become:
import bar.apple.foo as apple
apple_foo = apple.foo_func
Now, the first line loads bar.apple.foo as a module. In case of packages, this means importing package's __init__.py code. And treating it as a module itself.
And what's the code in the package's init? Usually nothing! That's why the name lookup fails.
However, when you do any import my_package.whatever, the package gets its insides checked and the name becomes visible. You're basically pre-loading the module for interpreter to look at.
Why is pycharm giving you not used suggestion? Because it's not used as a variable anywhere. You're only using a side-effect + pycharm doesn't analyze strings for imports or attributes.
Visual example, with a part of standard library:
>>> import xml
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> xml.etree
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'xml' has no attribute 'etree'
>>>
>>> import xml.etree
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'etree']
And another example, what happens if there are multiple modules in the package:
>>> import dateutil
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version']
but:
>>> import dateutil.parser
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_common', '_version', 'parser', 'relativedelta', 'tz']
All sub-modules are now visible and usable with their qualified name.
tl;dr: import my_package == only my_package/__init__.py is looked at. import my_package.whatever == python now knows it's a package and registers its insides, all modules of my_package are visible and usable.
How to distinguish between attributes defined at global level and those imported from other modules programatically? For instance, I want to know which module HIGHEST_PROTOCOL and MY_HIGHEST_PROTOCOL defined in mymod.py belong to.
Contents of mymod.py:
from pickle import HIGHEST_PROTOCOL
MY_HIGHEST_PROTOCOL = 123
Inspecting in IPython.
In [2]: import mymod
In [3]: dir(mymod)
Out[3]:
['HIGHEST_PROTOCOL',
'MY_HIGHEST_PROTOCOL',
'__builtins__',
'__cached__',
'__doc__',
'__file__',
'__loader__',
'__name__',
'__package__',
'__spec__']
I'm trying to use the importlib library to verify whether the nmap library is installed on the computer executing the script in Python 3.5.2
I'm trying to use importlib.util.find_spec("nmap") but receive the following error.
>>> import importlib
>>> importlib.util.find_spec("nmap")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
Can someone tell me where I'm going wrong?
EDIT
I was able to get the function to work using the following code.
#!/usr/bin/pythonw
import importlib
from importlib import util
#check to see if nmap module is installed
find_nmap = util.find_spec("nmap")
if find_nmap is None:
print("Error")
Try this:
from importlib import util
util.find_spec("nmap")
I intend to investigate, but honestly I don't know why one works and the other doesn't. Also, observe the following interactive session:
>>> import importlib
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> from importlib import util
>>> util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
>>> importlib.util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
So...yeah. I am sure this makes perfect sense to someone, but not to me. I will update once I figure it out.
Update:
Comparing this to something like:
>>> import datetime
>>> datetime
<module 'datetime' from '/usr/lib/python3.5/datetime.py'>
>>> datetime.datetime
<class 'datetime.datetime'>
I think the difference is that in this case the first datetime is a module and the second is a class, while in the importlib.util case both are modules. So perhaps module.module is not OK unless the code from both modules has been loaded, while module.class is OK, because the class code is loaded when the module is imported.
Update #2
Nope, it seems like in many cases module.module is fine. For example:
>>> import urllib
>>> urllib
<module 'urllib' from '/usr/lib/python3.5/urllib/__init__.py'>
>>> urllib.error
<module 'urllib.error' from '/usr/lib/python3.5/urllib/error.py'>
So perhaps it is something specific to importlib.
Update #3
As #kfb pointed out in the comments, it does seem to be related to importlib specifically. See the following comment from the __init__.py for importlib:
# Until bootstrapping is complete, DO NOT import any modules that attempt
# to import importlib._bootstrap (directly or indirectly). Since this
# partially initialised package would be present in sys.modules, those
# modules would get an uninitialised copy of the source version, instead
# of a fully initialised version (either the frozen one or the one
# initialised below if the frozen one is not available).
importlib/util.py does import importlib._bootstrap so I would assume that this is realted. If my understanding is correct, when you do import importlib the submodules will be initialized, but are not initialized for the importlib module object that you have imported. At this point, if you do dir(importlib) you will not see util. Interestingly, after you have tried to access importlib.util and gotten an AttributeError, util (along with the other submodules) gets loaded/initialized, and now you can access importlib.util!
>>> import importlib
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_r_long', '_w_long', 'find_loader', 'import_module', 'invalidate_caches', 'reload', 'sys', 'types', 'warnings']
>>> importlib.util
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'importlib' has no attribute 'util'
>>> importlib.util
<module 'importlib.util' from '/usr/lib/python3.5/importlib/util.py'>
>>> dir(importlib)
['_RELOADING', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__import__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_bootstrap', '_bootstrap_external', '_imp', '_r_long', '_w_long', 'abc', 'find_loader', 'import_module', 'invalidate_caches', 'machinery', 'reload', 'sys', 'types', 'util', 'warnings']
I am reading about how import works in python.
When I do:
import A.B.C
A, A.B, A.B.C are put in sys.modules. Expected.
A's __init__, A.B's __init__ get executed. Expected.
But here is a surprise: When I print globals(), only A is put into the namespace, while 'A.B.C' is not. I expect 'A.B.C' to be in global namespace.
And this means, I can access A.x defined in A's __init__.
Why is import implemented this way?
Only objects/names are put in globals namespace. A.B.C is not a valid name.
In your above case, the object is the module object for A , and its name is A .
In this particular case, if you do -
dir(A)
You would see B inside it, and that means its an attribute of the module object A . If you do -
hasattr(A,'B')
It would return True.
And in the same way if you do - dir(A.B) , you would be able to see C in it , and C is an attribute of A.B .
A Very simple example to show this -
My directory structur -
shared/
__init__.py
pkg/
__init__.py
b.py
Then in code I do -
>>> import shared.pkg.b
>>> dir(shared)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'pkg']
>>> hasattr(shared,'pkg')
True
>>>
>>> dir(shared.pkg)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'b']
>>> hasattr(shared.pkg,'b')
True
B and C are reachable through A.
eg.
import A.B.C
print(A.B.C)
If you want B and C to appear directly in your current namespace then do
from A import B
from A.B import C
print(B, C)
i have installed a python package plivo using the sudo pip install plivo.
and interpreter i test it with some code like:
>>> import plivo
>>> p = plivo.RestAPI('xxx', 'yyy')
everything working fine in python interpreter.
exactly same code is not working in a python script test_plivio.py
giving error : AttributeError: 'module' object has no attribute 'RestAPI'
then i checked with dir()
in interpreter
>>> dir(plivo)
['Account', 'Application', 'Call', 'Carrier', 'Conference', 'ConferenceMember', 'EndPoint', 'Message', 'Number', 'PLIVO_VERSION', 'PlivoError', 'PlivoResponse', 'Pricing', 'Recording', 'RestAPI', 'SubAccount', 'XML', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'base64', 'hmac', 'json', 'requests', 'sha1', 'validate_signature']
RestAPI is there.
while in test_plivo.py dir(plivo) is like:
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'main']
clearly dir(plivo) in script is missing RestAPI with other functions.
why is that behavior and how to resolve that ?
You are importing a different module; on your path you have a different plivo.py (or plivo.pyc cached bytecode) file.
Print out the __file__ attribute to see what is imported instead:
print plivo.__file__
and rename that or move it somewhere else.