This question already has answers here:
How can I import a module dynamically given its name as string?
(10 answers)
Closed 3 months ago.
I'm trying to dynamically load modules I've created.
Right now this works properly:
import structures.index
But if I try the same thing by importing it dynamically, it fails.
struct = __import__("structures.index")
Error supplied is:
Error ('No module named structures.index',)
Any ideas why?
Edit: When using full scope (it sort of works?):
struct = __import__("neoform.structures.index")
This doesn't throw any errors, however, it isn't loading the index module, it's loading the "neoform" module instead.
The result of "struct" is:
<module 'neoform' from '/neoform/__init__.py'>
Also, as a side question, how can I then instantiate a class within a dynamically loaded module? (assuming all the modules contain a common class name).
Edit: Solution: (thanks coonj & Rick) This ended up being what worked. Not sure why (yet), but the fromlist had to be something "anything apparently, since it worked when I put the letter "a" as a value (strange, given that the file only had 1 class in it).
def get_struct_module(self, name):
try:
return = __import__("neoform.structures." + name, fromlist='*')
except ImportError, e:
self.out.add("Could not load struct: neoform.structure." + name + "\n\n" + "Error " + str(e.args))
I'm not sure what "it fails" means, so I'll just mention that __import__('structures.index') should, in fact, work, but it doesn't assign the module name in the current scope. To do that (and then use a class in the dynamically imported module), you'll have to use:
structures = __import__('structures.index')
structures.index.SomeClass(...)
The complete details on __import__ are available here.
Edit: (based on question edit)
To import neoform.structures.index, and return the index module, you would do the following:
structures = __import__('neoform.structures.index',
fromlist=['does not in fact matter what goes here!'])
So if you have a list of package names packages, you can import their index modules and instantiate some MyClass class for each using the following code:
modules = [ __import__('neoform.%s.index' % pkg, fromlist=['a'])
for pkg in packages ]
objects = [ m.MyClass() for m in modules ]
To import sub-modules, you need to specify them in the fromlist arg of __import__() Fo example, the equivalent of:
import structures.index
is:
structures = __import__('structures', fromlist=['index'])
To do this in a map is a little more tricky...
import mod1.index
import mod2.index
import mod3.index
For those imports, you would want to define a new function to get the index sub-module from each module:
def getIndexMods(mod_names):
mod_list = map(lambda x: __import__(x, fromlist='index'))
index_mods = [mod.index for mod in mod_list]
return index_mods
Now, you can do this to get references to all index modules:
index_mods = getIndexMods(['mod1', 'mod2', 'mod3'])
Also, if you want to grab sub-modules that are not named 'index' then you could do this:
mod1, mod2, mod3 = map(lambda x,y: __import__(x, fromlist=y),
['mod1', 'mod2', 'mod3'], ['index1', 'index2', 'index3'])
Use full scope ("neoform.structures.index") with this helper method.
def import_module(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
module = import_module("neoform.structures.index")
# do stuff with module
>>> import imp
>>> fm = imp.find_module('index', ['./structures']) # for submodule
>>> mymod = imp.load_module('structures.index', *fm)
>>> mymod
<module 'structures.index' from './structures/index.pyc'>
>>> x = mymod.insideIndex()
Initialising index class...
Voila!
Java programmer here, but I think you need the imp module
Why on earth would you replace
import structures.index
with
map(__import__, ["structures.index"])
The first one (a) works, (b) is dynamic and (c) is directly supported. What possible use case is there for replacing easy-to-change, plain-text source with something more complex?
In short: don't do this. It doesn't have any value.
Edit
The "I'm getting the import from a database" is a noble effort, but still not sensible. What code block depends on those imports? That whole code block -- imports and all -- is what you want to execute. That whole code block -- imports, statements and everything -- should be a plain old python module file.
Import that block of code from the file system. Use the database to identify which file, the author of the file -- anything you want to use the database for. But simply import and execute the module the simplest possible way.
Really late post here. But I was searching for this question on google. I did some trial and error. Not sure if this snippet will help but here it is. Using it for Flask site.
modules = ['frontend', 'admin']
for module in modules:
mod = __init__('controllers.%s' % module, fromlist=[module])
app.register_blueprint(mod.blueprint_mod)
# or
from importlib import import_module
modules = ['frontend', 'admin']
for module in modules:
mod = import_module('controllers.%s' % module)
app.regitster_blueprint(mod.blueprint_mod)
Related
I'd like to load a module dynamically, given its string name (from an environment variable). I'm using Python 2.7. I know I can do something like:
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
This is roughly equivalent to
import my_settings
(where SETTINGS_MODULE = 'my_settings'). The problem is, I need something equivalent to
from my_settings import *
since I'd like to be able to access all methods and variables in the module. I've tried
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
from my_module import *
but I get a bunch of errors doing that. Is there a way to import all methods and attributes of a module dynamically in Python 2.7?
If you have your module object, you can mimic the logic import * uses as follows:
module_dict = my_module.__dict__
try:
to_import = my_module.__all__
except AttributeError:
to_import = [name for name in module_dict if not name.startswith('_')]
globals().update({name: module_dict[name] for name in to_import})
However, this is almost certainly a really bad idea. You will unceremoniously stomp on any existing variables with the same names. This is bad enough when you do from blah import * normally, but when you do it dynamically there is even more uncertainty about what names might collide. You are better off just importing my_module and then accessing what you need from it using regular attribute access (e.g., my_module.someAttr), or getattr if you need to access its attributes dynamically.
Not answering precisely the question as worded, but if you wish to have a file as proxy to a dynamic module, you can use the ability to define __getattr__ on the module level.
import importlib
import os
module_name = os.environ.get('CONFIG_MODULE', 'configs.config_local')
mod = importlib.import_module(module_name)
def __getattr__(name):
return getattr(mod, name)
My case was a bit different - wanted to dynamically import the constants.py names in each gameX.__init__.py module (see below), cause statically importing those would leave them in sys.modules forever (see: this excerpt from Beazley I picked from this related question).
Here is my folder structure:
game/
__init__.py
game1/
__init__.py
constants.py
...
game2/
__init__.py
constants.py
...
Each gameX.__init__.py exports an init() method - so I had initially a from .constants import * in all those gameX.__init__.py which I tried to move inside the init() method.
My first attempt in the lines of:
## -275,2 +274,6 ## def init():
# called instead of 'reload'
+ yak = {}
+ yak.update(locals())
+ from .constants import * # fails here
+ yak = {x: y for x,y in locals() if x not in yak}
+ globals().update(yak)
brec.ModReader.recHeader = RecordHeader
Failed with the rather cryptic:
SyntaxError: import * is not allowed in function 'init' because it contains a nested function with free variables
I can assure you there are no nested functions in there. Anyway I hacked and slashed and ended up with:
def init():
# ...
from .. import dynamic_import_hack
dynamic_import_hack(__name__)
Where in game.__init__.py:
def dynamic_import_hack(package_name):
print __name__ # game.init
print package_name # game.gameX.init
import importlib
constants = importlib.import_module('.constants', package=package_name)
import sys
for k in dir(constants):
if k.startswith('_'): continue
setattr(sys.modules[package_name], k, getattr(constants, k))
(for setattr see How can I add attributes to a module at run time? while for getattr How can I import a python module function dynamically? - I prefer to use those than directly access the __dict__)
This works and it's more general than the approach in the accepted answer cause it allows you to have the hack in one place and use it from whatever module. However I am not really sure it's the best way to implement it - was going to ask a question but as it would be a duplicate of this one I am posting it as an answer and hope to get some feedback. My questions would be:
why this "SyntaxError: import * is not allowed in function 'init'" while there are no nested functions ?
dir has a lot of warnings in its doc - in particular it attempts to produce the most relevant, rather than complete, information - this complete worries me a bit
is there no builtin way to do an import * ? even in python 3 ?
This question already has answers here:
How can I import a module dynamically given its name as string?
(10 answers)
How can I import a module dynamically given the full path?
(35 answers)
Closed 3 months ago.
So i'm trying to turn a bunch of "from x import x" statements, that look like this:
from class_foo import class_foo
into something dynamic. I'm trying to pass a path to a directory and have it import all of the modules therein.
def dynamicImport(dirPath):
filez = os.listdir(dirPath)
for file in filez:
if "class" in file:
oname = file[:-3] #cut off the file extension, trivial
imp_statement = "from " + oname + " import " + oname
#when I print imp_statement, I can verify it's being concatenated correctly
exec(imp_statement)
When I run this function and pass it a path, the statement strings are being created properly and it produces no errors, yet then later I'll try to access one of the imported objects, and this happens:
foo = class_foo()
NameError: name 'class_foo' is not defined
Clearly I'm doing something wrong. Any help would be appreciated.
You're execing your import statement in your function's local namespace, so that's where the names are defined. This namespace goes away when the function ends, leaving you with nothing. What you probably want is something like exec imp_statement in globals().
Why not just use __import__() instead of string-munging? Then you get a reference to your module. You can then fish out the class reference using getattr() on the module object and insert that into globals() (or just pass a dictionary back to the caller, who can then do globals().update() with it).
import sys, os
def getClasses(directory):
classes = {}
oldcwd = os.getcwd()
os.chdir(directory) # change working directory so we know import will work
for filename in os.listdir(directory):
if filename.endswith(".py"):
modname = filename[:-3]
classes[modname] = getattr(__import__(modname), modname)
os.setcwd(oldcwd)
return classes
globals().update(getClasses(r"C:\plugin_classes"))
Something like that. Or rather than updating globals() with your modules, which could clobber a global variable you care about, just leave the classes in the dictionary and reference them from there:
classes = getClasess(r"C:\plugin_classes")
for clas in classes.itervalues():
instance = clas(1, 2, 3) # instantiate
instance.dosomething_cool(42) # call method
Python >= 2.7 has importlib (you can pip install importlib to use importlib in earlier versions of python)
module = importlib.import_module("path.to.module")
MyClass = module.MyClass
Suppose your directory structure looks like this:
./ <- you are here
- main.py
- my_package/
- __init__.py
- my_module.py
and you want to dynamically import my_module.py, to use some of its functions, classes, whatever. Then, using importlib, you can use the following code in main.py:
import importlib
pack = "my_package"
mod = "my_module"
module = importlib.import_module("." + mod, pack)
# or, alternatively
module = importlib.import_module(".".join(pack, mod))
module.func("hello") # if my_package/my_module.py defines function "func"
obj = module.MyClass("world") # if my_package/my_module.py defines class "MyClass"
It's been a long time since I've worked with Python. But I think your problem might lay in the fact that "oname" is a string. The line from class_foo import class_foo is not a string. One painstaking option would be to have your code make a whole new .py file that would have all of the imports. So you would write all of your current file plus the new imports to, basically, a text file ending in .py
Take a look at the __import__ function
How can I make this import:
from module import *
with imp module?
Disclaimer:
I need use imp module because I need make several statements and I want to do it dynamicly
I need use * because I need that in the file I made the import, the variables and methods defined in module be available directly, i mean without module.method or module.variable. And I want import all variables and methods in the module because I don't know what methods or variables can be in the module in the future
Here!
def load_everything_from(module_names):
g = globals()
for module_name in module_names:
m = __import__(module_name)
names = getattr(m, '__all__', None)
if names is None:
names = [name for name in dir(m) if not name.startswith('_')]
for name in names:
g[name] = getattr(m, name)
I am kind of making things up there a little bit with trying to find an __all__ symbol first and then, if that files, doing a dir() and grabbing symbols that look non-private — you would have to look at the implementation of import * to know if that resembles Python's actual logic closely enough for your purposes.
If you are using django (as mentionned in comments), something like this should work
from django.utils import importlib
my_module = importlib.import_module('app.my_module')
from my_module import *
But I agree that it may be dangerous
You can do it by:
from imp import *
but remember:
Explicit is better than implicit.
(from The Zen of Python - read it by using the following command: import this)
I have a Python module that I want to dynamically import given only a string of the module name. Normally I use importlib or __import__ and this works quite well given that I know which objects I want to import from the module, but is there a way to do the equivalent of import * dynamically. Or is there a better approach?
I know in general its bad practice to use import * but the modules I'm trying to import are automatically generated on the fly and I have no way of knowing the exact module which contains the class I'm addressing.
Thanks.
Use update for dicts:
globals().update(importlib.import_module('some.package').__dict__)
Note, that using a_module.__dict__ is not the same as from a_module import *, because all names are "imported", not only those from __all__ or not starting with _.
I came up with some ugly hacky code, it works in python 2.6. I'm not sure if this is the smartest thing to do though, perhaps some other people here have some insight:
test = __import__('os',globals(),locals())
for k in dir(test):
globals()[k] = test.__dict__[k]
You probably want to put a check here to make sure you aren't overwriting anything in the global namespace. You could probably avoid the globals part and just look through each dynamically imported module for your class of interest. This would probably be much better than polluting the global namespace with everything you are importing.
For example, say your class is named Request from urllib2
test = __import__('urllib2',globals(),locals())
cls = None
if 'Request' in dir(test):
cls = test.__dict__['Request']
# you found the class now you can use it!
cls('http://test.com')
The following is highly sinful and will condemn you to purgatory or worse
# module_a.py
myvar = "hello"
# module_b.py
import inspect
def dyn_import_all(modpath):
"""Incredibly hackish way to load into caller's global namespace"""
exec('from ' + modpath + ' import *', inspect.stack()[1][0].f_globals)
# module_c.py
from module_b import dyn_import_all
def print_from(modpath):
dyn_import_all(modpath)
print(myvar)
Demo:
>>> import module_c
>>> module_c.print_from("module_a")
hello
In python you can do something like this to import a module using a string filename, and assign its namespace a variable on the local namespace.
x = __import__(str)
I'm wondering if there is a related function that will take take a string of Python code, instead of a path to a file with Python code, and return its namespace as a variable.
For example,
str = "a = 5";
x = importstr(str)
print x.a
#output is 5
I realize that I could write the string to a file, then use __import__ on it, but I'd like to skip the intermediate file if possible.
The reason for this is that I'm experimenting with metaprogramming in python, and it seems like a good solution to what I'm doing.
Here's an example of dynamically creating module objects using the imp module
Unfortunately, the imp module was recently deprecated (I have NO idea why).
Instead, you should do this:
from types import ModuleType
import sys
mod = ModuleType('my_module', 'doc string here')
exec('a = 1', mod.__dict__)
print(mod.a) # prints 1
# add to sys.modules
sys.modules['my_module'] = mod
Or you can use PyExt's RuntimeModule.from_string:
from pyext import RuntimeModule
mod = RuntimeModule.from_string('a = 1')
print(mod.a) # 1
Here is how to import a string as a module:
import sys,imp
my_code = 'a = 5'
mymodule = imp.new_module('mymodule')
exec my_code in mymodule.__dict__
so you can now access the module attributes (and functions, classes etc) as:
mymodule.a
>>> 5
To ignore any next attempt to import, add the module to sys:
sys.modules['mymodule'] = mymodule
types.ModuleType is not recommended according to Python documentation on module_from_spec():
importlib.util.module_from_spec(spec)
...
This function is preferred over using types.ModuleType to create a new module as spec is used to set as many import-controlled attributes on the module as possible.
Here is what I came up with to load the module from source code.
import importlib.util
spec = importlib.util.spec_from_loader('helper', loader=None)
helper = importlib.util.module_from_spec(spec)
exec('a = 5', helper.__dict__)
print(type(helper)) # prints "<class 'module'>"
helper.a # prints "5"
Is this something what you're looking for ?
my_namespace = {}
exec "a = 5" in my_namespace
print my_namespace["a"]