I am currently working on a library that patches several other modules upon loading. It mostly works fine, however in some cases it runs into the problem when the functions to be patched are contained in a sub-module that needs to be explicitely loaded. For instance in scikits-learn, the sub-module datasets has the following behavior:
>>> import sklearn
>>> dir(sklearn)
['__SKLEARN_SETUP__', '__all__', '__builtins__', '__check_build', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'base', 'clone', 'externals', 're', 'setup_module', 'sys', 'utils', 'warnings']
It only loads it in case when dataset was explicitely loaded:
>>> from sklearn import datasets
>>> dir(sklearn)
['__SKLEARN_SETUP__', '__all__', '__builtins__', '__check_build', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'base', 'clone', 'datasets', 'externals', 'feature_extraction', 'preprocessing', 're', 'setup_module', 'sys', 'utils', 'warnings']
How can I detect when dataset is explicitely imported in order to launch my patching only when this sub-module is loaded?
Related
i want to implement sort of plugin architecture that dynamically loads modules and calls a function from them
for instance plugin code looks like (in file "foo_func.py")
foo_local = []
def foo_add(a: int, b: int) -> int:
c = a + b
foo_local.append(c)
return c
def foo_print():
print(foo_local)
i need to support two plugins with the same code but with different memory state, so i created directory structure like this:
<ROOT_PROJECT>
app.py
bar/
apple/
foo/
foo_func.py
__init__.py
orange/
foo/
foo_func.py
__init__.py
code in "apple" and "orange" folders is the same.
then in app file i try to load modules and invoke functions from them
import importlib
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
apple = importlib.import_module('bar.apple.foo')
orange = importlib.import_module('bar.orange.foo')
apple_foo = getattr(apple, 'foo_func')
orange_foo = getattr(orange, 'foo_func')
apple_foo_add_my = getattr(apple_foo, 'foo_add')
apple_foo_print_my = getattr(apple_foo, 'foo_print')
apple_foo_add_my(1, 2)
apple_foo_print_my()
and this works fine, but you see these import lines at the top
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
they are not used in code (even pycharm complains about it)
but if i try to comment code and run it - then failure
AttributeError: module 'bar.apple.foo' has no attribute 'foo_func'
why ?
I suppose normal plugins should deal only with "importlib.import_module" and "getattr" and it must be enough ?
what is wrong here ?
Let's switch completely to direct imports for this explanation, because:
import something.whatever as name
is the same as:
name = importlib.import_module("something.whatever")
So let's rewrite your apple code:
apple = importlib.import_module('bar.apple.foo')
apple_foo = getattr(apple, 'foo_func')
will become:
import bar.apple.foo as apple
apple_foo = apple.foo_func
Now, the first line loads bar.apple.foo as a module. In case of packages, this means importing package's __init__.py code. And treating it as a module itself.
And what's the code in the package's init? Usually nothing! That's why the name lookup fails.
However, when you do any import my_package.whatever, the package gets its insides checked and the name becomes visible. You're basically pre-loading the module for interpreter to look at.
Why is pycharm giving you not used suggestion? Because it's not used as a variable anywhere. You're only using a side-effect + pycharm doesn't analyze strings for imports or attributes.
Visual example, with a part of standard library:
>>> import xml
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> xml.etree
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'xml' has no attribute 'etree'
>>>
>>> import xml.etree
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'etree']
And another example, what happens if there are multiple modules in the package:
>>> import dateutil
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version']
but:
>>> import dateutil.parser
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_common', '_version', 'parser', 'relativedelta', 'tz']
All sub-modules are now visible and usable with their qualified name.
tl;dr: import my_package == only my_package/__init__.py is looked at. import my_package.whatever == python now knows it's a package and registers its insides, all modules of my_package are visible and usable.
I am using ubuntu18.04. I have installed pygame module to play songs. But the time.Clock() function is not working. It is showing this following error after running the program: AttributeError: 'function' object has no attribute 'Clock'.
Here is the code:
def playSong(filename):
mixer.init()
mixer.music.load('/home/mjiabir/Music/rangamati songs/Roar.mp3')
mixer.music.play()
while mixer.music.get_busy:
time.Clock.tick(10)
mixer.music.stop()
Looks like Linux doesn't support this module. What should I do now?
I think you want to use pygame.time.Clock().tick(10).
One way to look up modules of a package or sub-package is to use dir(). For example:
# in a python interactive shell
import time
dir(time)
# output => ['CLOCK_BOOTTIME', 'CLOCK_MONOTONIC', 'CLOCK_MONOTONIC_RAW', 'CLOCK_PROCESS_CPUTIME_ID', 'CLOCK_REALTIME', 'CLOCK_THREAD_CPUTIME_ID', '_STRUCT_TM_ITEMS', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'altzone', 'asctime', 'clock', 'clock_getres', 'clock_gettime', 'clock_gettime_ns', 'clock_settime', 'clock_settime_ns', 'ctime', 'daylight', 'get_clock_info', 'gmtime', 'localtime', 'mktime', 'monotonic', 'monotonic_ns', 'perf_counter', 'perf_counter_ns', 'process_time', 'process_time_ns', 'pthread_getcpuclockid', 'sleep', 'strftime', 'strptime', 'struct_time', 'thread_time', 'thread_time_ns', 'time', 'time_ns', 'timezone', 'tzname', 'tzset']
import pygame
dir(pygame.time)
# output => ['Clock', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'delay', 'get_ticks', 'set_timer', 'wait']
Although you can use pygame.time.Clock().tick(), I would suggest constructing a Clock object, through clock = pygame.time.Clock() and clock.tick(10). Generally, it is good practice to utilize the object-oriented nature of Python, as opposed to calling construct methods directly.
How to distinguish between attributes defined at global level and those imported from other modules programatically? For instance, I want to know which module HIGHEST_PROTOCOL and MY_HIGHEST_PROTOCOL defined in mymod.py belong to.
Contents of mymod.py:
from pickle import HIGHEST_PROTOCOL
MY_HIGHEST_PROTOCOL = 123
Inspecting in IPython.
In [2]: import mymod
In [3]: dir(mymod)
Out[3]:
['HIGHEST_PROTOCOL',
'MY_HIGHEST_PROTOCOL',
'__builtins__',
'__cached__',
'__doc__',
'__file__',
'__loader__',
'__name__',
'__package__',
'__spec__']
Im trying to play around with the urllib on python 3.6.7, but my urllib does not contain the following classes:
error, parse, request, response
I've tried to use urllib3 but the request class there does not contain the urlopen() method
>>> import urllib
>>> dir(urllib)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> dir(urllib3)
['HTTPConnectionPool', 'HTTPResponse', 'HTTPSConnectionPool', 'PoolManager', 'ProxyManager', 'Retry', 'Timeout', '__all__', '__author__', '__builtins__', '__cached__', '__doc__', '__file__', '__license__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_collections', 'absolute_import', 'add_stderr_logger', 'connection', 'connection_from_url', 'connectionpool', 'disable_warnings', 'encode_multipart_formdata', 'exceptions', 'fields', 'filepost', 'get_host', 'logging', 'make_headers', 'packages', 'poolmanager', 'proxy_from_url', 'request', 'response', 'util', 'warnings']
i have installed a python package plivo using the sudo pip install plivo.
and interpreter i test it with some code like:
>>> import plivo
>>> p = plivo.RestAPI('xxx', 'yyy')
everything working fine in python interpreter.
exactly same code is not working in a python script test_plivio.py
giving error : AttributeError: 'module' object has no attribute 'RestAPI'
then i checked with dir()
in interpreter
>>> dir(plivo)
['Account', 'Application', 'Call', 'Carrier', 'Conference', 'ConferenceMember', 'EndPoint', 'Message', 'Number', 'PLIVO_VERSION', 'PlivoError', 'PlivoResponse', 'Pricing', 'Recording', 'RestAPI', 'SubAccount', 'XML', '__builtins__', '__doc__', '__file__', '__name__', '__package__', 'base64', 'hmac', 'json', 'requests', 'sha1', 'validate_signature']
RestAPI is there.
while in test_plivo.py dir(plivo) is like:
['__builtins__', '__doc__', '__file__', '__name__', '__package__', 'main']
clearly dir(plivo) in script is missing RestAPI with other functions.
why is that behavior and how to resolve that ?
You are importing a different module; on your path you have a different plivo.py (or plivo.pyc cached bytecode) file.
Print out the __file__ attribute to see what is imported instead:
print plivo.__file__
and rename that or move it somewhere else.