We Keep Coding

How to customize a module import in Python?

I would like to customize the behavior of my module when it is imported.
For example, let say I want my module to print an incremented number each time another file use import my_module. And when from my_module import some_string is used, it should print "some_string".
How could I do that?
I read several questions here and there but this does not seems to work.
# my_module.py
import sys
class MyImporter:
def find_module(self, module_name, package_path):
print(module_name, package_path)
return self
def load_module(self, module_name):
print(module_name)
return self
sys.meta_path.append(MyImporter())
# file.py
import my_module # Nothing happens

What you're asking for is to have Python work not like Python. Whenever it imports a module it parses and executes the 'opened' code only once so it can pick up the definitions, functions, classes, etc. - every subsequent import of the module just references the cached & parsed first import.
That's why even if you put something like vars()["counter"] = vars().get("counter", 0) + 1 at your module's 'root', the counter will never go above 1 indicating that the module was indeed executed only once. You can force module reload using reload() (or importlib.reload() on Python 3.6+) but then you'd lose your counter if you keep it in the module itself.
Of course, you can have an external counter to be called when your module is imported, but that would have to be a contract with the users of your module at which point the question becomes - can't you just contract your users to call a function to increase your counter whenever they import your module instead of having to reload it for you to capture the count? Reloading a module will also make it have a potentially different state in every context it was reloaded which will make Python behave unexpectedly and should be avoided at any cost.
So, a short answer would be - no, you cannot do that and you should not attempt to do it. If you want something that doesn't work like Python - use something that isn't Python.
However... If you have a really, REALLY good reason to do this (and you don't!) and you don't mind hacking how Python fundamentally behaves (and you should mind) then you might attempt to do this by wrapping the built-in import and checking whenever it gets fired for your module. Something like:
your_module.py:
# HERE BE DRAGONS!!!
import sys
try:
import builtins # Python 3.4+
except ImportError:
import __builtin__ as builtins # Python 2.6+
__builtin_import__ = builtins.__import__ # store a reference to the built-in import
def __custom_import__(name, *args, **kwargs):
# execute builtin first so that the import fails if badly requested
ret = __builtin_import__(name, *args, **kwargs)
if ret is sys.modules[__name__]: # we're trying to load this module
if len(args) > 1 and args[2]: # using the `from your_module import whatever` form
if "some_string" in args[2]: # if some_string is amongst requested properties
print("some_string")
else: # using the `import your_module` form...
print_counter() # increase and print the latest count
return ret # return back the actual import result
builtins.__import__ = __custom_import__ # override the built-in import with our method
counter = 0
# a convinience function, you can do all of this through the `__custom_import__` function
def print_counter():
global counter
counter += 1
print(counter)
print_counter() # call it immediately on the first import to print out the counter
some_string = "I'm holding some string value" # since we want to import this
# HAVE I FORGOT TO TELL YOU NOT TO DO THIS? WELL, DON'T!!!
Keep in mind that this will not account for the first import (be it in the pure import your_module or in the from your_module import whatever form) as the import override won't exist until your module is loaded - that's why it calls print_counter() immediately in hope that the first import of the module was in the form of import your_module and not in the from..import form (if not it will wrongly print out the count instead of some_string the first time). To solve the first-import issue, you can move this 'ovverride' to the __init__.py in the same folder so that the override loads before your module starts and then delegate the counter change / some_string print to the module once loaded, just make sure you do your module name check properly in that case (you need to account for the package as well) and make sure it doesn't automatically execute the counter.
You also, technically, don't need the some_string property at all - by moving the execution of the built-in import around you can do your from..import check first, find the position of some_string in args[2] and pop it before calling the builtin import, then return None in the same position once executed. You can also do your printing and counter incrementing from within the overriden import function.
Again, for the love of all things fluffy and the poor soul who might have to rely on your code one day - please don't do this!

Actually, it does look like it's possible to do what you're looking for in python3.5. It's probably a bad idea, and I've carefully written my code to demonstrate the concept without being polished enough to use as-is, because I'd think carefully before doing something like this in a production project.
If you need to look at a more-or-less production example of this, take a look at the SelfWrapper class in the sh module.
Meanwhile, you can override your own entry in sys.modules to be a subclass of Module. Then you can override getattribute and detect accesses to attributes.
As best I can tell:
Every subsiquent import of the module references spec so you could probably count accesses to spec to count total imports
Each from foo import bar accesses bar as an attribute. I don't think you can distinguish between "from foo import bar" and "import foo; foo.bar"
import sys, types
class Wrapper(types.ModuleType):
def __getattribute__(self, attr):
print(attr)
return super().__getattribute__(attr)
test = "test"
sys.modules[__name__].__class__ = Wrapper

Here is how you can dynamically import modules-
from importlib import import_module
def import_from(module, name):
module = import_module(module, name)
return getattr(module, name)
and use it like this-
funcObj = import_from("<file_name>", "<method_name>")
response = funcObj(arg1,arg2)

How do I detect if a class / variable was imported in Python 3?

This is the contents of script_one.py:
x = "Hello World"
This is the contents of script_two.py:
from script_one import x
print(x)
Now, if I ran script_two.py the output would be:
>>> Hello World
What I need is a way to detect if x was imported.
This is what I imagine the source code of script_one.py would look like:
x = "Hello World"
if x.has_been_imported:
print("You've just imported \"x\"!")
Then if I ran script_two.py the output "should" be:
>>> Hello World
>>> You've just imported "x"!
What is this called, does this feature exist in Python 3 and how do you use it?

You can't. Effort expended on trying to detect this are a waste of time, I'm afraid.
Python imports consist of the following steps:
Check if the module is already loaded by looking at sys.modules.
If the module hasn't been loaded yet, load it. This creates a new module object that is added to sys.modules, containing all objects resulting from executing the top-level code.
Bind names in the importing namespace. How names are bound depends on the exact import variant chosen.
import module binds the name module to the sys.modules[module] object
import module as othername binds the name othername to the sys.modules[module] object
from module import attribute binds the name attribute to the sys.modules[module].attribute object
from module import attribute as othername binds the name othername to the sys.modules[module].attribute object
In this context it is important to realise that Python names are just references; all Python objects (including modules) live on a heap and stand or fall with the number of references to them. See this great article by Ned Batchelder on Python names if you need a primer on how this works.
Your question then can be interpreted in two ways:
You want to know the module has been imported. The moment code in the module is executed (like x = "Hello World"), it has been imported. All of it. Python doesn't load just x here, it's all or nothing.
You want to know if other code is using a specific name. You'd have to track what other references exist to the object. This is a mammoth task involving recursively checking the gc.get_referrers() object chain to see what other Python objects might now refer to x.
The latter goal is made the harder all the further in any of the following scenarios:
import script_one, then use script_one.x; references like these could be too short-lived for you to detect.
from script_one import x, then del x. Unless something else still references the same string object within the imported namespace, that reference is now gone and can't be detected anymore.
import sys; sys.modules['script_one'].x is a legitimate way of referencing the same string object, but does this count as an import?
import script_one, then list(vars(script_one).values()) would create a list of all objects defined in the module, but these references are indices in a list, not named. Does this count as an import?

Looks like it is impossible previously. But ever since python 3.7+ introduces __getattr__ on module level, looks like it is possible now. At least we can distinguish whether a variable is imported by from module import varable or import module; module.variable.
The idea is to detect the AST node in the previous frame, whether it is an Attribute:
script_one.py
def _variables():
# we have to define the variables
# so that it dosen't bypass __getattr__
return {'x': 'Hello world!'}
def __getattr__(name):
try:
out = _variables()[name]
except KeyError as kerr:
raise ImportError(kerr)
import ast, sys
from executing import Source
frame = sys._getframe(1)
node = Source.executing(frame).node
if node is None:
print('`x` is imported')
else:
print('`x` is accessed via `script_one.x`')
return out
script_two.py
from script_one import x
print(x)
# `x` is imported
# 'Hello world!'
import script_one
print(script_one.x)
# `x` is accessed via `script_one.x`
# 'Hello world!'

Python: mock patch a module wherever it is imported from

I need to make sure that running unit tests won't trigger calling a heavy outer world function, say, this one:
# bigbad.py
def request(param):
return 'I searched the whole Internet for "{}"'.format(param)
Multiple modules use this function (bigbad.request) and they import it differently (in real-life it may be imported from an external library as well). Say, there are two modules, a and b, where b depends on a and both use the function:
# a.py, from...import
from bigbad import request
def routine_a():
return request('a')
# b.py, imports directly
import a
import bigbad
def routine_b():
resp_a = a.routine_a()
return 'resp_a: {}, resp_b=request(resp_a): {}'.format(resp_a, bigbad.request(resp_a))
Is there a way to make sure that bigbad.request is not ever called? This code mocks only one of the imports:
# test_b.py
import unittest
from unittest import mock
import b
with mock.patch('bigbad.request') as mock_request:
mock_request.return_value = 'mocked'
print(b.routine_b())
Obviously I could refactor b and change the imports but this way I cannot guarantee that someone during the future development is not going to break this provision. I believe tests should test behaviour than implementation details.

import bigbad
bigbad.request = # some dummy function
This will work as long as it runs before any module that does from bigbad import request is run/imported. That is, as long as they run after, they will receive the dummy function.

# a.py, from...import
from bigbad import request
To ensure that the original request is never called, you'll have to patch all the places where the reference is imported:
import mock
with mock.patch('a.request', return_value='mocked') as mock_request:
...
This is tedious, so if possible don't do from bigbad import request in your code, but use import bigbad; bigbad.request.
Another solution: if possible, change bigbad.py:
# bigbad.py
def _request(param):
return 'I searched the whole Internet for "{}"'.format(param)
def request(param):
return _request(param)
Then, even if some code does from bigbad import request, you'd be able to do with mock.patch('bigbad._request', return_value='mocked') as mock_request:.

For any people coming to this question from the future, I wrote a function to patch all imports of a given symbol.
This function returns a list of patchers for each import of the given symbol (a whole module, a specific function, or any other object). These patchers can then be started/stopped in your test fixture's setup/teardown areas (see the docstring for an example).
How it works:
Iterate through every currently visible module in sys.modules
If the module's name starts with match_prefix (optional) and does not contain skip_substring (optional), iterate through every local in the module
If the local is target_symbol, create a patcher for it, local to the module it's imported in
I recommend using an argument like skip_substring='test' so that you don't patch things imported by your test suite.
from typing import Any, Optional
import unittest.mock as mock
import sys
def patch_all_symbol_imports(
target_symbol: Any, match_prefix: Optional[str] = None,
skip_substring: Optional[str] = None
):
"""
Iterate through every visible module (in sys.modules) that starts with
`match_prefix` to find imports of `target_symbol` and return a list
of patchers for each import.
This is helpful when you want to patch a module, function, or object
everywhere in your project's code, even when it is imported with an alias.
Example:
::
import datetime
# Setup
patchers = patch_all_symbol_imports(datetime, 'my_project.', 'test')
for patcher in patchers:
mock_dt = patcher.start()
# Do stuff with the mock
# Teardown
for patcher in patchers:
patcher.stop()
:param target_symbol: the symbol to search for imports of (may be a module,
a function, or some other object)
:param match_prefix: if not None, only search for imports in
modules that begin with this string
:param skip_substring: if not None, skip any module that contains this
substring (e.g. 'test' to skip unit test modules)
:return: a list of patchers for each import of the target symbol
"""
patchers = []
# Iterate through all currently imported modules
# Make a copy in case it changes
for module in list(sys.modules.values()):
name_matches = (
match_prefix is None
or module.__name__.startswith(match_prefix)
)
should_skip = (
skip_substring is not None and skip_substring in module.__name__
)
if not name_matches or should_skip:
continue
# Iterate through this module's locals
# Again, make a copy
for local_name, local in list(module.__dict__.items()):
if local is target_symbol:
# Patch this symbol local to the module
patchers.append(mock.patch(
f'{module.__name__}.{local_name}', autospec=True
))
return patchers
For this question specifically, the following code could be used:
from bigbad import request
patchers = patch_all_symbol_imports(request, skip_substring='test')
for patcher in patchers:
mock_request = patcher.start()
mock_request.return_value = 'mocked'
print(b.routine_b())
for patcher in patchers:
patcher.stop()

Python: How to import all methods and attributes from a module dynamically

I'd like to load a module dynamically, given its string name (from an environment variable). I'm using Python 2.7. I know I can do something like:
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
This is roughly equivalent to
import my_settings
(where SETTINGS_MODULE = 'my_settings'). The problem is, I need something equivalent to
from my_settings import *
since I'd like to be able to access all methods and variables in the module. I've tried
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
from my_module import *
but I get a bunch of errors doing that. Is there a way to import all methods and attributes of a module dynamically in Python 2.7?

If you have your module object, you can mimic the logic import * uses as follows:
module_dict = my_module.__dict__
try:
to_import = my_module.__all__
except AttributeError:
to_import = [name for name in module_dict if not name.startswith('_')]
globals().update({name: module_dict[name] for name in to_import})
However, this is almost certainly a really bad idea. You will unceremoniously stomp on any existing variables with the same names. This is bad enough when you do from blah import * normally, but when you do it dynamically there is even more uncertainty about what names might collide. You are better off just importing my_module and then accessing what you need from it using regular attribute access (e.g., my_module.someAttr), or getattr if you need to access its attributes dynamically.

Not answering precisely the question as worded, but if you wish to have a file as proxy to a dynamic module, you can use the ability to define __getattr__ on the module level.
import importlib
import os
module_name = os.environ.get('CONFIG_MODULE', 'configs.config_local')
mod = importlib.import_module(module_name)
def __getattr__(name):
return getattr(mod, name)

My case was a bit different - wanted to dynamically import the constants.py names in each gameX.__init__.py module (see below), cause statically importing those would leave them in sys.modules forever (see: this excerpt from Beazley I picked from this related question).
Here is my folder structure:
game/
__init__.py
game1/
__init__.py
constants.py
...
game2/
__init__.py
constants.py
...
Each gameX.__init__.py exports an init() method - so I had initially a from .constants import * in all those gameX.__init__.py which I tried to move inside the init() method.
My first attempt in the lines of:
## -275,2 +274,6 ## def init():
# called instead of 'reload'
+ yak = {}
+ yak.update(locals())
+ from .constants import * # fails here
+ yak = {x: y for x,y in locals() if x not in yak}
+ globals().update(yak)
brec.ModReader.recHeader = RecordHeader
Failed with the rather cryptic:
SyntaxError: import * is not allowed in function 'init' because it contains a nested function with free variables
I can assure you there are no nested functions in there. Anyway I hacked and slashed and ended up with:
def init():
# ...
from .. import dynamic_import_hack
dynamic_import_hack(__name__)
Where in game.__init__.py:
def dynamic_import_hack(package_name):
print __name__ # game.init
print package_name # game.gameX.init
import importlib
constants = importlib.import_module('.constants', package=package_name)
import sys
for k in dir(constants):
if k.startswith('_'): continue
setattr(sys.modules[package_name], k, getattr(constants, k))
(for setattr see How can I add attributes to a module at run time? while for getattr How can I import a python module function dynamically? - I prefer to use those than directly access the __dict__)
This works and it's more general than the approach in the accepted answer cause it allows you to have the hack in one place and use it from whatever module. However I am not really sure it's the best way to implement it - was going to ask a question but as it would be a duplicate of this one I am posting it as an answer and hope to get some feedback. My questions would be:
why this "SyntaxError: import * is not allowed in function 'init'" while there are no nested functions ?
dir has a lot of warnings in its doc - in particular it attempts to produce the most relevant, rather than complete, information - this complete worries me a bit
is there no builtin way to do an import * ? even in python 3 ?

Dynamic Loading of Python Modules [duplicate]

This question already has answers here:
How can I import a module dynamically given its name as string?
(10 answers)
Closed 3 months ago.
I'm trying to dynamically load modules I've created.
Right now this works properly:
import structures.index
But if I try the same thing by importing it dynamically, it fails.
struct = __import__("structures.index")
Error supplied is:
Error ('No module named structures.index',)
Any ideas why?
Edit: When using full scope (it sort of works?):
struct = __import__("neoform.structures.index")
This doesn't throw any errors, however, it isn't loading the index module, it's loading the "neoform" module instead.
The result of "struct" is:
<module 'neoform' from '/neoform/__init__.py'>
Also, as a side question, how can I then instantiate a class within a dynamically loaded module? (assuming all the modules contain a common class name).
Edit: Solution: (thanks coonj & Rick) This ended up being what worked. Not sure why (yet), but the fromlist had to be something "anything apparently, since it worked when I put the letter "a" as a value (strange, given that the file only had 1 class in it).
def get_struct_module(self, name):
try:
return = __import__("neoform.structures." + name, fromlist='*')
except ImportError, e:
self.out.add("Could not load struct: neoform.structure." + name + "\n\n" + "Error " + str(e.args))

I'm not sure what "it fails" means, so I'll just mention that __import__('structures.index') should, in fact, work, but it doesn't assign the module name in the current scope. To do that (and then use a class in the dynamically imported module), you'll have to use:
structures = __import__('structures.index')
structures.index.SomeClass(...)
The complete details on __import__ are available here.
Edit: (based on question edit)
To import neoform.structures.index, and return the index module, you would do the following:
structures = __import__('neoform.structures.index',
fromlist=['does not in fact matter what goes here!'])
So if you have a list of package names packages, you can import their index modules and instantiate some MyClass class for each using the following code:
modules = [ __import__('neoform.%s.index' % pkg, fromlist=['a'])
for pkg in packages ]
objects = [ m.MyClass() for m in modules ]

To import sub-modules, you need to specify them in the fromlist arg of __import__() Fo example, the equivalent of:
import structures.index
is:
structures = __import__('structures', fromlist=['index'])
To do this in a map is a little more tricky...
import mod1.index
import mod2.index
import mod3.index
For those imports, you would want to define a new function to get the index sub-module from each module:
def getIndexMods(mod_names):
mod_list = map(lambda x: __import__(x, fromlist='index'))
index_mods = [mod.index for mod in mod_list]
return index_mods
Now, you can do this to get references to all index modules:
index_mods = getIndexMods(['mod1', 'mod2', 'mod3'])
Also, if you want to grab sub-modules that are not named 'index' then you could do this:
mod1, mod2, mod3 = map(lambda x,y: __import__(x, fromlist=y),
['mod1', 'mod2', 'mod3'], ['index1', 'index2', 'index3'])

Use full scope ("neoform.structures.index") with this helper method.
def import_module(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
module = import_module("neoform.structures.index")
# do stuff with module

>>> import imp
>>> fm = imp.find_module('index', ['./structures']) # for submodule
>>> mymod = imp.load_module('structures.index', *fm)
>>> mymod
<module 'structures.index' from './structures/index.pyc'>
>>> x = mymod.insideIndex()
Initialising index class...
Voila!

Java programmer here, but I think you need the imp module

Why on earth would you replace
import structures.index
with
map(__import__, ["structures.index"])
The first one (a) works, (b) is dynamic and (c) is directly supported. What possible use case is there for replacing easy-to-change, plain-text source with something more complex?
In short: don't do this. It doesn't have any value.
Edit
The "I'm getting the import from a database" is a noble effort, but still not sensible. What code block depends on those imports? That whole code block -- imports and all -- is what you want to execute. That whole code block -- imports, statements and everything -- should be a plain old python module file.
Import that block of code from the file system. Use the database to identify which file, the author of the file -- anything you want to use the database for. But simply import and execute the module the simplest possible way.

Really late post here. But I was searching for this question on google. I did some trial and error. Not sure if this snippet will help but here it is. Using it for Flask site.
modules = ['frontend', 'admin']
for module in modules:
mod = __init__('controllers.%s' % module, fromlist=[module])
app.register_blueprint(mod.blueprint_mod)
# or
from importlib import import_module
modules = ['frontend', 'admin']
for module in modules:
mod = import_module('controllers.%s' % module)
app.regitster_blueprint(mod.blueprint_mod)