We Keep Coding

python importlib.import_module requires explicitly imported module

i want to implement sort of plugin architecture that dynamically loads modules and calls a function from them
for instance plugin code looks like (in file "foo_func.py")
foo_local = []
def foo_add(a: int, b: int) -> int:
c = a + b
foo_local.append(c)
return c
def foo_print():
print(foo_local)
i need to support two plugins with the same code but with different memory state, so i created directory structure like this:
<ROOT_PROJECT>
app.py
bar/
apple/
foo/
foo_func.py
__init__.py
orange/
foo/
foo_func.py
__init__.py
code in "apple" and "orange" folders is the same.
then in app file i try to load modules and invoke functions from them
import importlib
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
apple = importlib.import_module('bar.apple.foo')
orange = importlib.import_module('bar.orange.foo')
apple_foo = getattr(apple, 'foo_func')
orange_foo = getattr(orange, 'foo_func')
apple_foo_add_my = getattr(apple_foo, 'foo_add')
apple_foo_print_my = getattr(apple_foo, 'foo_print')
apple_foo_add_my(1, 2)
apple_foo_print_my()
and this works fine, but you see these import lines at the top
from bar.apple.foo.foo_func import foo_add as apple_foo_add, foo_print as apple_foo_print
from bar.orange.foo.foo_func import foo_add as orange_foo_add, foo_print as orange_foo_print
they are not used in code (even pycharm complains about it)
but if i try to comment code and run it - then failure
AttributeError: module 'bar.apple.foo' has no attribute 'foo_func'
why ?
I suppose normal plugins should deal only with "importlib.import_module" and "getattr" and it must be enough ?
what is wrong here ?

Let's switch completely to direct imports for this explanation, because:
import something.whatever as name
is the same as:
name = importlib.import_module("something.whatever")
So let's rewrite your apple code:
apple = importlib.import_module('bar.apple.foo')
apple_foo = getattr(apple, 'foo_func')
will become:
import bar.apple.foo as apple
apple_foo = apple.foo_func
Now, the first line loads bar.apple.foo as a module. In case of packages, this means importing package's __init__.py code. And treating it as a module itself.
And what's the code in the package's init? Usually nothing! That's why the name lookup fails.
However, when you do any import my_package.whatever, the package gets its insides checked and the name becomes visible. You're basically pre-loading the module for interpreter to look at.
Why is pycharm giving you not used suggestion? Because it's not used as a variable anywhere. You're only using a side-effect + pycharm doesn't analyze strings for imports or attributes.
Visual example, with a part of standard library:
>>> import xml
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> xml.etree
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: module 'xml' has no attribute 'etree'
>>>
>>> import xml.etree
>>> dir(xml)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'etree']
And another example, what happens if there are multiple modules in the package:
>>> import dateutil
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_version']
but:
>>> import dateutil.parser
>>> dir(dateutil)
['__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_common', '_version', 'parser', 'relativedelta', 'tz']
All sub-modules are now visible and usable with their qualified name.
tl;dr: import my_package == only my_package/__init__.py is looked at. import my_package.whatever == python now knows it's a package and registers its insides, all modules of my_package are visible and usable.

Distinguish between imported and globally defined module attributes

How to distinguish between attributes defined at global level and those imported from other modules programatically? For instance, I want to know which module HIGHEST_PROTOCOL and MY_HIGHEST_PROTOCOL defined in mymod.py belong to.
Contents of mymod.py:
from pickle import HIGHEST_PROTOCOL
MY_HIGHEST_PROTOCOL = 123
Inspecting in IPython.
In [2]: import mymod
In [3]: dir(mymod)
Out[3]:
['HIGHEST_PROTOCOL',
'MY_HIGHEST_PROTOCOL',
'__builtins__',
'__cached__',
'__doc__',
'__file__',
'__loader__',
'__name__',
'__package__',
'__spec__']

mpl_finance module import error in python

I have tried using from matplotlib.finance import quotes_historical_yahoo_ochl but found that it is deprecated. Hence, I have installed mpl_finance and tried this:
from mpl_finance import quotes_historical_yahoo_ochl
But I got the following error:
ImportError: cannot import name 'quotes_historical_yahoo_ochl'
I thought to check it:
dir(mpl_finance)
And found the following:
['Affine2D', 'Line2D', 'LineCollection', 'PolyCollection', 'Rectangle', 'TICKLEFT', 'TICKRIGHT', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_candlestick', '_check_input', '_plot_day_summary', 'absolute_import', 'candlestick2_ochl', 'candlestick2_ohlc', 'candlestick_ochl', 'candlestick_ohlc', 'division', 'index_bar', 'mcolors', 'np', 'plot_day_summary2_ochl', 'plot_day_summary2_ohlc', 'plot_day_summary_oclh', 'plot_day_summary_ohlc', 'print_function', 'unicode_literals', 'volume_overlay', 'volume_overlay2', 'volume_overlay3', 'xrange', 'zip']
Kindly, help me get rid of this issue. Please.

Because yahoo has ceased to provide its historical quotes service, the quotes_historical_yahoo_ochl function stopped working. It was hence removed from the mpl_finance code completely.

Python - submodule loading detection

I am currently working on a library that patches several other modules upon loading. It mostly works fine, however in some cases it runs into the problem when the functions to be patched are contained in a sub-module that needs to be explicitely loaded. For instance in scikits-learn, the sub-module datasets has the following behavior:
>>> import sklearn
>>> dir(sklearn)
['__SKLEARN_SETUP__', '__all__', '__builtins__', '__check_build', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'base', 'clone', 'externals', 're', 'setup_module', 'sys', 'utils', 'warnings']
It only loads it in case when dataset was explicitely loaded:
>>> from sklearn import datasets
>>> dir(sklearn)
['__SKLEARN_SETUP__', '__all__', '__builtins__', '__check_build', '__doc__', '__file__', '__name__', '__package__', '__path__', '__version__', 'base', 'clone', 'datasets', 'externals', 'feature_extraction', 'preprocessing', 're', 'setup_module', 'sys', 'utils', 'warnings']
How can I detect when dataset is explicitely imported in order to launch my patching only when this sub-module is loaded?

How can module be visible from one import and not visible from another?

So I've got an application that's using pymysql, the pure python mysql client implementation. Before I go into my response, I'd like to stress the fact that I am not open to using a different mysql driver.
I have a module implementing a data structure that's backed by MySQL. The gist of the module is as follows:
import pymysql
class Whatever:
def __init__(self):
# Debug statement
print dir(pymysql)
# use the cursors submodule
self.conn = pymysql.connect( ... , cursorclass=pymysql.cursors.DictCursor)
When I import this in my test file, everything is fine. Here is the output of the print statement. In particular, I draw your attention to the cursors module:
['BINARY', 'Binary', 'Connect', 'Connection', 'DATE', 'DATETIME', 'DBAPISet', 'DataError',
'DatabaseError', 'Date', 'DateFromTicks', 'Error', 'FIELD_TYPE', 'IntegrityError',
'InterfaceError', 'InternalError', 'MySQLError', 'NULL', 'NUMBER', 'NotSupportedError',
'OperationalError', 'ProgrammingError', 'ROWID', 'STRING', 'TIME', 'TIMESTAMP', 'Time',
'TimeFromTicks', 'Timestamp', 'TimestampFromTicks', 'VERSION', 'Warning', '__all__',
'__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__',
'__version__', 'apilevel', 'charset', 'connect', 'connections', 'constants', 'converters',
'cursors', 'err', 'escape_dict', 'escape_sequence', 'escape_string', 'get_client_info',
'install_as_MySQLdb', 'paramstyle', 'sys', 'thread_safe', 'threadsafety', 'times', 'util',
'version_info']
When I import the module from my main file, I get an AttributeError:
Traceback (most recent call last):
File "xxx.py", line 72, in <module>
passwd='', db='test_db')
File "yyy.py", line 26, in __init__
passwd=passwd, db=db, cursorclass=pymysql.cursors.DictCursor)
AttributeError: 'module' object has no attribute 'cursors'
The output of the dir print is as follows:
['BINARY', 'Binary', 'Connect', 'Connection', 'DATE', 'DATETIME', 'DBAPISet',
'DataError', 'DatabaseError', 'Date', 'DateFromTicks', 'Error', 'FIELD_TYPE',
'IntegrityError', 'InterfaceError', 'InternalError', 'MySQLError', 'NULL', 'NUMBER',
'NotSupportedError', 'OperationalError', 'ProgrammingError', 'ROWID', 'STRING', 'TIME',
'TIMESTAMP', 'Time', 'TimeFromTicks', 'Timestamp', 'TimestampFromTicks', 'VERSION',
'Warning', '__all__', '__builtins__', '__doc__', '__file__', '__name__', '__package__',
'__path__', '__version__', 'apilevel', 'charset', 'connect', 'constants', 'converters',
'err', 'escape_dict', 'escape_sequence', 'escape_string', 'get_client_info',
'install_as_MySQLdb', 'paramstyle', 'sys', 'thread_safe', 'threadsafety', 'times',
'version_info']
Notably, cursors is absent. Checking pymysql.__file__ is the same in both cases, and the output is:
__init__.py charset.py connections.py constants converters.pyc cursors.pyc err.pyc times.py util.py
__init__.pyc charset.pyc connections.pyc converters.py cursors.py err.py tests times.pyc util.pyc
Clearly cursors.py is there. So what gives?

You need to add an explicit import pymysql.cursors at the top of your file.
the cursors subpackage is not listed in the pymysql's __all__ and thus isn't imported when you do just import pymysql.