I want to monkey patch a new class and then import that class but I get an error ModuleNotFoundError. At the same time I can use the new patched function. I believe that I miss how to add it in the "init" file.
See the following simplified example:
import numpy
class wrong_functions():
def wrong_add(num1, num2):
return num1 + num2 + 1
numpy.random.wrong_functions = wrong_functions
numpy.random.wrong_functions.wrong_add(1,1) # works
from numpy.random.wrong_functions import wrong_add # does not work
from numpy.random.wrong_functions import * # does not work
What do you think? Is that possible?
This is because of the import system.
Reading through the doc, you can find this paragraph:
[...] the statement from spam.ham import eggs, sausage as saus results in
_temp = __import__('spam.ham', globals(), locals(), ['eggs', 'sausage'], 0)
eggs = _temp.eggs
saus = _temp.sausage
The problem is: what does __import__() does?
The import statement combines two operations; it searches for the named module, then it binds the results of that search to a name in the local scope. [...]
A direct call to __import__() performs only the module search and, if found, the module creation operation.
So, when you re-import the module, your customization will be lost.
To ensure it stays on, you can import numpy as np and then, when using np - after you assing this new class - you can always access wrong_add.
>>> import numpy as np
>>> np.random.wrong_functions = wrong_functions
>>> np.random.wrong_function.wrong_add(1, 1)
3
EDIT: If you need/want to just call wrong_add instead of full package path to function, you can always assign it to a variable.
>>> wrong_add = np.random.wrong_function.wrong_add
>>> wrong_add(2, 2)
5
Related
Suppose I have a Python module "main.py":
import math # from the standard Python library
import my_own_module
...
foo = math.cos(bar)
And I also need to import the standard math module in "my_own_module.py":
import math
...
baz = math.sin(qux)
In this case I think import math in "main.py" is redundant and can be omitted.
What's best practice in this case:
Omit import math from "main.py" becuase it's redundant? Or,
Keep import math in "main.py" to clarify that the code in that module requires it?
The reference to math.cos in main.py means that import math is required in main.py, regardless of whether my_own_module.py imports it or not. It is not redundant, and it cannot be omitted (and if you try to omit it, you'll get an error).
import math
does something else than simply including the full text of one file into the other.
It introduces a new namespace with the name math, and this math name will be known in your current namespace.
If you omit the
import math
from your main.py file, your command
foo = math.cos(bar)
becomes illegal, as the math symbol will be not (recognized) in the main.py namespace.
This is not like, eg #include in C++. The import is not optional. Importing a module is required to be able to refer to its contents. This is true for every single file that does it.
A good question. The short answer is yes, if you use a math function in a py file then you need to import the module at the top regardless of how many times its imported elsewhere.
It gets interesting when we throw a thrid file into the mix, lets call this "explanation.py"
And lets suppose that your "main.py" becomes "my_functions.py" and contains a function called foo:
#my_functions.py
import math
import my_own_module
def foo(bar):
return math.cos(bar)
and in my_own_module.py:
#my_own_module.py
import math
def bar(foo):
return math.sin(foo)
and finally explanation.py (new main())
#main.py
import my_functions
import my_own_module
bar = my_functions.foo(10)
foo = my_own_module.bar(10)
print(foo)
print(bar)
Notice how you DO NOT need to add math if you call the functions imported from another file. I hope that might add further clarity to your enquiry :)
However it might be worth noting that this would exclude maths from the current namespace, therefore rendering any further calls to the math functions useless.
This question already has an answer here:
Is there a way to bypass the namespace/module name in Python?
(1 answer)
Closed last month.
I am using Canopy with the Jupyter notebook. I was wondering if there was a way to use function from a module without having to call the module. For example if I have
import numpy as np
print np.sin(2)
I would want to be able to just type
print sin(2)
The first thing that comes to mind is to add the numpy functions into whatever function library that Python is using. But I was wondering if this is feasible and, if so, how I could go about doing it. Note that I want to import all functions, not just a select few.
You can import specific objects from a module. Try:
from numpy import sin
print sin(2)
To import all objects from a module into the global namespace you can use import *.
from numpy import *
print sin(2)
But this is not recommended because you can easily end up with name clashes, e.g. if two modules define a function named sin which version of sin should be called?
>>> import math
>>> import numpy
>>> math.sin
<built-in function sin>
>>> numpy.sin
<ufunc 'sin'>
>>> from math import *
>>> sin
<built-in function sin>
>>> from numpy import *
>>> sin
<ufunc 'sin'>
You can see here that the second import from numpy replaced sin in the global namespace.
For this reason it is best to import the specific objects that you need if there are only a few, otherwise just import the module and use the module name as a prefix (as per your first example). In my example if you wanted to use both math.sin and nump.sin you would either need to import the modules only and prefix using the module name, or import the functions and rename them like this:
from numpy import sin as np_sin
from math import sin
from numpy import sin
print sin(2)
https://docs.python.org/2/tutorial/modules.html read this in details
I'd like to make a copy of an entire namespace while replacing some functions with dynamically constructed versions.
In other words, starting with namespace (import tensorflow as tf), I want to make a copy of it, replace some functions with my own versions, and update __globals__ of all the symbols to stay within the new namespace. This needs to be done in topological order of dependency.
I started doing something like it here but now I'm starting to wonder if I'm reinventing the wheel. Care is needed to deal with circular dependencies in system modules, functions/types/objects need to be updated differently, etc.
Can anyone point to existing code that solves a similar task?
To patch a set of functions while importing second instances of a set of functions, you can override the standard Python import hook and apply the patches directly at import time. This will make sure that no other module will ever see the unpatched versions of any of the modules, so even if they import functions from another module directly by name, they will only see the patched functions. Here is a proof-of-concept implementation:
import __builtin__
import collections
import contextlib
import sys
#contextlib.contextmanager
def replace_import_hook(new_import_hook):
original_import = __builtin__.__import__
__builtin__.__import__ = new_import_hook
yield original_import
__builtin__.__import__ = original_import
def clone_modules(patches, additional_module_names=None):
"""Import new instances of a set of modules with some objects replaced.
Arguments:
patches - a dictionary mapping `full.module.name.symbol` to the new object.
additional_module_names - a list of the additional modules you want new instances of, without
replacing any objects in them.
Returns:
A dictionary mapping module names to the new patched module instances.
"""
def import_hook(module_name, *args):
result = original_import(module_name, *args)
if module_name not in old_modules or module_name in new_modules:
return result
# The semantics for the return value of __import__() are a bit weird, so we need some logic
# to determine the actual imported module object.
if len(args) >= 3 and args[2]:
module = result
else:
module = reduce(getattr, module_name.split('.')[1:], result)
for symbol, obj in patches_by_module[module_name].items():
setattr(module, symbol, obj)
new_modules[module_name] = module
return result
# Group patches by module name
patches_by_module = collections.defaultdict(dict)
for dotted_name, obj in patches.items():
module_name, symbol = dotted_name.rsplit('.', 1) # Only allows patching top-level objects
patches_by_module[module_name][symbol] = obj
try:
# Remove the old module instances from sys.modules and store them in old_modules
all_module_names = list(patches_by_module)
if additional_module_names is not None:
all_module_names.extend(additional_module_names)
old_modules = {}
for name in all_module_names:
old_modules[name] = sys.modules.pop(name)
# Re-import modules to create new patched versions
with replace_import_hook(import_hook) as original_import:
new_modules = {}
for module_name in all_module_names:
import_hook(module_name)
finally:
sys.modules.update(old_modules)
return new_modules
And here some test code for this implementation:
from __future__ import print_function
import math
import random
def patched_log(x):
print('Computing log({:g})'.format(x))
return math.log(x)
patches = {'math.log': patched_log}
cloned_modules = clone_modules(patches, ['random'])
new_math = cloned_modules['math']
new_random = cloned_modules['random']
print('Original log: ', math.log(2.0))
print('Patched log: ', new_math.log(2.0))
print('Original expovariate: ', random.expovariate(2.0))
print('Patched expovariate: ', new_random.expovariate(2.0))
The test code has this output:
Computing log(4)
Computing log(4.5)
Original log: 0.69314718056
Computing log(2)
Patched log: 0.69314718056
Original expovariate: 0.00638038735379
Computing log(0.887611)
Patched expovariate: 0.0596108277801
The first two lines of output result from these two lines in random, which are executed at import time. This demonstrates that random sees the patched function right away. The rest of the output demonstrates that the original math and random still use the unpatched version of log, while the cloned modules both use the patched version.
A cleaner way of overriding the import hook might be to use a meta import hook as defined in PEP 302, but providing a full implementation of that approach is beyond the scope of StackOverflow.
Instead of trying to make a copy of the contents of a module and patch everything in it to use the correct globals, you could trick Python into importing everything you want to copy a second time. This will give you a newly initialized copy of all modules, so it won't copy any global state the modules might have (not sure whether you would need that).
import importlib
import sys
def new_module_instances(module_names):
old_modules = {}
for name in module_names:
old_modules[name] = sys.modules.pop(name)
new_modules = {}
for name in module_names:
new_modules[name] = importlib.import_module(name)
sys.modules.update(old_modules)
return new_modules
Note that we first delete all modules we want to replace from sys.modules, so they all get import a second time, and the dependencies between these modules are set up correctly automatically. At the end of the function, we restore the original state of sys.modules, so everything else continues to see the original versions of these modules.
Here's an example:
>>> import logging.handlers
>>> new_modules = new_module_instances(['logging', 'logging.handlers'])
>>> logging_clone = new_modules['logging']
>>> logging
<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>
>>> logging_clone
<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>
>>> logging is logging_clone
False
>>> logging is logging.handlers.logging
True
>>> logging_clone is logging_clone.handlers.logging
True
The last three expressions show that the two versions of logging are different modules, and both versions of the handlers module use the correct version of the logging module.
To my mind, you can do this easily:
import imp, string
st = imp.load_module('st', *imp.find_module('string')) # copy the module
def my_upper(a):
return "a" + a
def my_lower(a):
return a + "a"
st.upper = my_upper
st.lower = my_lower
print string.upper("hello") # HELLO
print string.lower("hello") # hello
print st.upper("hello") # ahello
print st.lower("hello") # helloa
And when you call st.upper("hello"), it will result in "hello".
So, you don't really need to mess with globals.
Is it possible to import a module with some parameter in python ?
All I mean by parameter is that there exists a variable in the module which is not initialized in that module, still I am using that variable in that module. In short, I want behavior similar to a function but unlike function, I want the variables of module to be exposed in the calling code.
eg a.py:
#lists like data, count, prob_distribution are constructed from training_pool (not initialized in this file)
x = pymc.Uniform('x', lower = 0, upper = 1)
rv = [ Multinomial("rv"+str(i), count[i], prob_distribution[i], value = data[i], observed=True) for i in xrange(0, len(count)) ]
b.py:
import a #I want some way tr pass value of training_pool
m = pymc.MCMC(a)
I want all random variables in a.py to be exposed to MCMC. I am open to a better approach for my problem at hand, but I would also like to know whether passing arguments to modules is possible in python or not.
there are various approaches to do so, here is just a silly and simple one:
main.py
"""A silly example - main supplies a parameter
"""
import sys,os
print os.path.basename(__file__)+":Push it by: --myModuleParam "+str(123)
sys.argv.append('--myModuleParam')
sys.argv.append(123)
import module
print os.path.basename(__file__)+":Pushed my param:"+str(module.displayMyParam)
module.py
"""A silly example - module consumes parameter
"""
import sys,os
displayMyParam = 'NotYetInitialized'
for px in sys.argv:
if px == '--myModuleParam':
idx = sys.argv.index(px)
sys.argv.pop(idx) # remove option
displayMyParam = sys.argv[idx]
sys.argv.pop(idx) # remove value
print os.path.basename(__file__)+":Got my param:"+str(displayMyParam)
#
# That's it...
#
As #otus already answered, there is no way to pass parameters to modules.
I think you are following some of the introductory examples for PyMC2, which use a pattern where a module wraps all the code for the nodes in a Bayesian model. This approach is good for getting started, but, as you have found, can be limiting, when you want to run your model with a range of variations.
Fortunately, PyMC2 can create an MCMC object from a list or a dictionary as well as a module. What I recommend in this case is just what #oleg-s suggested in the comments: use a function. You can end the function with return locals() to get a dictionary of everything that would have been in the module, and this is suitable input to the pymc.MCMC constructor. Here is an example:
# a.py
from pymc import *
count = [10, 10] # perhaps good to put this stuff in data.py
prob_distribution = [[.5, .5], [.1, .2, .7]]
data = [[2, 8], [2, 3, 5]]
def model(training_pool):
x = Uniform('x', lower = 0, upper = 1)
rv = [ Multinomial("rv"+str(i), count[i], prob_distribution[i], value = data[i], observed=True) for i in training_pool ]
return locals()
# b.py
import pymc, a
training_pool = [0]
m = pymc.MCMC(a.model(training_pool))
I found it helpful to define global variables, and allow these to be set by an init function.
def init(config_filename=CONFIG_FILENAME):
config = configparser.ConfigParser(interpolation=configparser.ExtendedInterpolation())
config.read(config_filename)
global YEARS
YEARS = config['DEFAULT']['YEARS']
global FEATURES
FEATURES = config['DEFAULT']['FEATURES']
Then all the user has to do is remember to initialize the module before using these methods:
import module
module.init('config.ini')
Note, I would NOT use this on a module that I expect to spread publicly. This is more for single-file modules for my own personal use.
There is no way to pass parameters to modules. However, you could use a global in a third module for this:
# a.py
parameter = None
# b.py
import a
a.parameter = 4
import c
# c.py
import a
# use a.parameter
Of course, this only works if nothing else imports c, because modules only get imported once.
Module-wide globals should be indeed enough for most uses, but what if
the parameter needs to be evaluated during module initialization, or
you need multiple versions of the module with different parameters
In recent versions of python, it is possible to load in two steps, first the spec, then exec. In the middle, you can set up extra variables.
import importlib
abstractModuleSpec=importlib.util.find_spec('myModule')
module4=importlib.util.module_from_spec(abstractModuleSpec)
module2=importlib.util.module_from_spec(abstractModuleSpec)
module2.parameter="you are version 2"
module4.parameter="you are version 4"
module4.__spec__.loader.exec_module(module4)
module2.__spec__.loader.exec_module(module2)
In the module you can check dir() or similar, to see if the variable is defined.
I really wonder nobody mentioned environment variables. That's the cleanest way I found:
a.py
import os
param = os.getenv('MY_PACKAGE_PARAM', None)
print(param)
b.py
import os
os.setenv('MY_PACKAGE_PARAM', 'Hello world!')
import a
There is no such way to pass parameters to the module, however you can revamp your code a bit and import the parameters from a module as global parameters.
I'd like to load a module dynamically, given its string name (from an environment variable). I'm using Python 2.7. I know I can do something like:
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
This is roughly equivalent to
import my_settings
(where SETTINGS_MODULE = 'my_settings'). The problem is, I need something equivalent to
from my_settings import *
since I'd like to be able to access all methods and variables in the module. I've tried
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
from my_module import *
but I get a bunch of errors doing that. Is there a way to import all methods and attributes of a module dynamically in Python 2.7?
If you have your module object, you can mimic the logic import * uses as follows:
module_dict = my_module.__dict__
try:
to_import = my_module.__all__
except AttributeError:
to_import = [name for name in module_dict if not name.startswith('_')]
globals().update({name: module_dict[name] for name in to_import})
However, this is almost certainly a really bad idea. You will unceremoniously stomp on any existing variables with the same names. This is bad enough when you do from blah import * normally, but when you do it dynamically there is even more uncertainty about what names might collide. You are better off just importing my_module and then accessing what you need from it using regular attribute access (e.g., my_module.someAttr), or getattr if you need to access its attributes dynamically.
Not answering precisely the question as worded, but if you wish to have a file as proxy to a dynamic module, you can use the ability to define __getattr__ on the module level.
import importlib
import os
module_name = os.environ.get('CONFIG_MODULE', 'configs.config_local')
mod = importlib.import_module(module_name)
def __getattr__(name):
return getattr(mod, name)
My case was a bit different - wanted to dynamically import the constants.py names in each gameX.__init__.py module (see below), cause statically importing those would leave them in sys.modules forever (see: this excerpt from Beazley I picked from this related question).
Here is my folder structure:
game/
__init__.py
game1/
__init__.py
constants.py
...
game2/
__init__.py
constants.py
...
Each gameX.__init__.py exports an init() method - so I had initially a from .constants import * in all those gameX.__init__.py which I tried to move inside the init() method.
My first attempt in the lines of:
## -275,2 +274,6 ## def init():
# called instead of 'reload'
+ yak = {}
+ yak.update(locals())
+ from .constants import * # fails here
+ yak = {x: y for x,y in locals() if x not in yak}
+ globals().update(yak)
brec.ModReader.recHeader = RecordHeader
Failed with the rather cryptic:
SyntaxError: import * is not allowed in function 'init' because it contains a nested function with free variables
I can assure you there are no nested functions in there. Anyway I hacked and slashed and ended up with:
def init():
# ...
from .. import dynamic_import_hack
dynamic_import_hack(__name__)
Where in game.__init__.py:
def dynamic_import_hack(package_name):
print __name__ # game.init
print package_name # game.gameX.init
import importlib
constants = importlib.import_module('.constants', package=package_name)
import sys
for k in dir(constants):
if k.startswith('_'): continue
setattr(sys.modules[package_name], k, getattr(constants, k))
(for setattr see How can I add attributes to a module at run time? while for getattr How can I import a python module function dynamically? - I prefer to use those than directly access the __dict__)
This works and it's more general than the approach in the accepted answer cause it allows you to have the hack in one place and use it from whatever module. However I am not really sure it's the best way to implement it - was going to ask a question but as it would be a duplicate of this one I am posting it as an answer and hope to get some feedback. My questions would be:
why this "SyntaxError: import * is not allowed in function 'init'" while there are no nested functions ?
dir has a lot of warnings in its doc - in particular it attempts to produce the most relevant, rather than complete, information - this complete worries me a bit
is there no builtin way to do an import * ? even in python 3 ?