This question already has answers here:
How can I import a module dynamically given its name as string?
(10 answers)
How can I import a module dynamically given the full path?
(35 answers)
Closed 3 months ago.
So i'm trying to turn a bunch of "from x import x" statements, that look like this:
from class_foo import class_foo
into something dynamic. I'm trying to pass a path to a directory and have it import all of the modules therein.
def dynamicImport(dirPath):
filez = os.listdir(dirPath)
for file in filez:
if "class" in file:
oname = file[:-3] #cut off the file extension, trivial
imp_statement = "from " + oname + " import " + oname
#when I print imp_statement, I can verify it's being concatenated correctly
exec(imp_statement)
When I run this function and pass it a path, the statement strings are being created properly and it produces no errors, yet then later I'll try to access one of the imported objects, and this happens:
foo = class_foo()
NameError: name 'class_foo' is not defined
Clearly I'm doing something wrong. Any help would be appreciated.
You're execing your import statement in your function's local namespace, so that's where the names are defined. This namespace goes away when the function ends, leaving you with nothing. What you probably want is something like exec imp_statement in globals().
Why not just use __import__() instead of string-munging? Then you get a reference to your module. You can then fish out the class reference using getattr() on the module object and insert that into globals() (or just pass a dictionary back to the caller, who can then do globals().update() with it).
import sys, os
def getClasses(directory):
classes = {}
oldcwd = os.getcwd()
os.chdir(directory) # change working directory so we know import will work
for filename in os.listdir(directory):
if filename.endswith(".py"):
modname = filename[:-3]
classes[modname] = getattr(__import__(modname), modname)
os.setcwd(oldcwd)
return classes
globals().update(getClasses(r"C:\plugin_classes"))
Something like that. Or rather than updating globals() with your modules, which could clobber a global variable you care about, just leave the classes in the dictionary and reference them from there:
classes = getClasess(r"C:\plugin_classes")
for clas in classes.itervalues():
instance = clas(1, 2, 3) # instantiate
instance.dosomething_cool(42) # call method
Python >= 2.7 has importlib (you can pip install importlib to use importlib in earlier versions of python)
module = importlib.import_module("path.to.module")
MyClass = module.MyClass
Suppose your directory structure looks like this:
./ <- you are here
- main.py
- my_package/
- __init__.py
- my_module.py
and you want to dynamically import my_module.py, to use some of its functions, classes, whatever. Then, using importlib, you can use the following code in main.py:
import importlib
pack = "my_package"
mod = "my_module"
module = importlib.import_module("." + mod, pack)
# or, alternatively
module = importlib.import_module(".".join(pack, mod))
module.func("hello") # if my_package/my_module.py defines function "func"
obj = module.MyClass("world") # if my_package/my_module.py defines class "MyClass"
It's been a long time since I've worked with Python. But I think your problem might lay in the fact that "oname" is a string. The line from class_foo import class_foo is not a string. One painstaking option would be to have your code make a whole new .py file that would have all of the imports. So you would write all of your current file plus the new imports to, basically, a text file ending in .py
Take a look at the __import__ function
Related
I can import a python script using import_module. But, how can I call a function stored as a variable from that script? I've previously used getattr to work with dictionaries stored as variables, but I don't think this same method works with functions. Here's an example that does not currently work:
from importlib import import_module
file_list = ['file1','file2']
func_list = ['func1','func2']
for file in file_list:
test_file = import_module(file)
for func in func_list:
from test_file import func
file1:
def func1():
...
def func2():
...
file2:
def func1():
...
def func2():
...
I can import a python script using import_module.
When you do this, the result is a module object - just the same as an import statement provides.
from test_file import func
The reason this doesn't work is because it is looking for a test_file module - and it cares about module names as they appear in sys.path, not about your local variable names.
Fortunately, since you already have the module object, you presumably realized you could access the contents normally, as attributes, e.g. test_file.func.
I've previously used getattr to work with dictionaries stored as variables, but I don't think this same method works with functions
I'm not quite sure what you mean here. Attributes are attributes, whether they're plain data, functions, classes or anything else. test_file is a thing that has a func attribute, therefore getattr(test_file, 'func') gets that attribute.
The remaining issue is the variable-variables problem - you don't really want to be creating a name for that result dynamically. So yes, you can store that in a dict, if you want. But frankly it's easier to just use the module object. Unless perhaps for some reason you need/want to "trim" the contents and only expose a limited interface (for some other client); but you can't avoid loading the whole module. from X import Y does that anyway.
The module object that you got from the dynamic import is already working as a namespace, which you need here anyway because you're importing multiple modules that have overlapping attribute names.
tl;dr: if you want to call a function from that imported module, just do it the same way that you would have if you had imported the module (not a name from that module) normally. We can, for example, put the imported modules in a list:
modules = [import_module(f) for f in filenames]
and then call the appropriate method by looking it up within the appropriate module object:
modules[desired_module_id].desired_func()
basically You would run this code in a separate file and where it says the_file_where_this_is_needed.py You would insert the file where You want these import statement to be. (also probably You can run this code in the very file). it will be sort of like hardcoding but automatic
file_list = ['file1', 'file2']
func_list = ['func1', 'func2']
with open('the_file_where_this_is_needed.py', 'r') as file:
data = file.read()
string = ''
for file in file_list:
for func in func_list:
string += f'from {file} import {func}\n'
data = string + data
with open('the_file_where_this_is_needed.py', 'w') as file:
file.write(data)
I want to import some package depending on which value the user chooses.
The default is file1.py:
from files import file1
If user chooses file2, it should be :
from files import file2
In PHP, I can do this using variable variables:
$file_name = 'file1';
include($$file_name);
$file_name = 'file2';
include($$file_name);
How can I do this in Python?
Python doesn't have a feature that's directly equivalent to PHP's "variable variables". To get a "variable variable"'s value (or the value of any other expression) you can use the eval function.
foo = "Hello World"
print eval("foo")
However, this can't be used in an import statement.
It is possible to use the __import__ function to import using a variable.
package = "os"
name = "path"
imported = getattr(__import__(package, fromlist=[name]), name)
is equivalent to
from os import path as imported
Old thread, but I needed the answer, so someone else still might...
There's a cleaner way to do this in Python 2.7+:
import importlib
my_module = importlib.import_module("package.path.%s" % module_name)
As Fredrik Lundh states:
Anyway, here’s how these statements and functions work:
import X imports the module X, and creates a reference to that module
in the current namespace. Or in other words, after you’ve run this
statement, you can use X.name to refer to things defined in module X.
from X import * imports the module X, and creates references in the
current namespace to all public objects defined by that module (that
is, everything that doesn’t have a name starting with “_”). Or in
other words, after you’ve run this statement, you can simply use a
plain name to refer to things defined in module X. But X itself is not
defined, so X.name doesn’t work. And if name was already defined, it
is replaced by the new version. And if name in X is changed to point
to some other object, your module won’t notice.
from X import a, b, c imports the module X, and creates references in
the current namespace to the given objects. Or in other words, you can
now use a and b and c in your program.
Finally, X = __import__(‘X’) works like import X, with the difference
that you 1) pass the module name as a string, and 2) explicitly assign
it to a variable in your current namespace.
And by the way that's the last one method that you're intrested in.
Simply write (for example):
var = "datetime"
module = __import__(var)
Basing myself on mattjbray's answer:
from importlib import import_module
# lookup in a set is in constant time
safe_names = {"file1.py", "file2.py", "file3.py", ...}
user_input = ...
if user_input in safe_names:
file = import_module(user_input)
else:
print("Nope, not doing this.")
Saves a few lines of code, and allows you to set safe_names programmatically, or load multiple modules and assign them to a dict, for example.
It's probably a very bad idea to let the user choose what to import. Packages can execute code on import, so you're effectively allowing a user to arbitrarily execute code on your system! Much safer to do something like
if user_input == 'file1.py':
from files import file1 as file
elif user_input == 'file2.py':
from files import file2 as file
else:
file = None
print "Sorry, you can't import that file"
I'd like to load a module dynamically, given its string name (from an environment variable). I'm using Python 2.7. I know I can do something like:
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
This is roughly equivalent to
import my_settings
(where SETTINGS_MODULE = 'my_settings'). The problem is, I need something equivalent to
from my_settings import *
since I'd like to be able to access all methods and variables in the module. I've tried
import os, importlib
my_module = importlib.import_module(os.environ.get('SETTINGS_MODULE'))
from my_module import *
but I get a bunch of errors doing that. Is there a way to import all methods and attributes of a module dynamically in Python 2.7?
If you have your module object, you can mimic the logic import * uses as follows:
module_dict = my_module.__dict__
try:
to_import = my_module.__all__
except AttributeError:
to_import = [name for name in module_dict if not name.startswith('_')]
globals().update({name: module_dict[name] for name in to_import})
However, this is almost certainly a really bad idea. You will unceremoniously stomp on any existing variables with the same names. This is bad enough when you do from blah import * normally, but when you do it dynamically there is even more uncertainty about what names might collide. You are better off just importing my_module and then accessing what you need from it using regular attribute access (e.g., my_module.someAttr), or getattr if you need to access its attributes dynamically.
Not answering precisely the question as worded, but if you wish to have a file as proxy to a dynamic module, you can use the ability to define __getattr__ on the module level.
import importlib
import os
module_name = os.environ.get('CONFIG_MODULE', 'configs.config_local')
mod = importlib.import_module(module_name)
def __getattr__(name):
return getattr(mod, name)
My case was a bit different - wanted to dynamically import the constants.py names in each gameX.__init__.py module (see below), cause statically importing those would leave them in sys.modules forever (see: this excerpt from Beazley I picked from this related question).
Here is my folder structure:
game/
__init__.py
game1/
__init__.py
constants.py
...
game2/
__init__.py
constants.py
...
Each gameX.__init__.py exports an init() method - so I had initially a from .constants import * in all those gameX.__init__.py which I tried to move inside the init() method.
My first attempt in the lines of:
## -275,2 +274,6 ## def init():
# called instead of 'reload'
+ yak = {}
+ yak.update(locals())
+ from .constants import * # fails here
+ yak = {x: y for x,y in locals() if x not in yak}
+ globals().update(yak)
brec.ModReader.recHeader = RecordHeader
Failed with the rather cryptic:
SyntaxError: import * is not allowed in function 'init' because it contains a nested function with free variables
I can assure you there are no nested functions in there. Anyway I hacked and slashed and ended up with:
def init():
# ...
from .. import dynamic_import_hack
dynamic_import_hack(__name__)
Where in game.__init__.py:
def dynamic_import_hack(package_name):
print __name__ # game.init
print package_name # game.gameX.init
import importlib
constants = importlib.import_module('.constants', package=package_name)
import sys
for k in dir(constants):
if k.startswith('_'): continue
setattr(sys.modules[package_name], k, getattr(constants, k))
(for setattr see How can I add attributes to a module at run time? while for getattr How can I import a python module function dynamically? - I prefer to use those than directly access the __dict__)
This works and it's more general than the approach in the accepted answer cause it allows you to have the hack in one place and use it from whatever module. However I am not really sure it's the best way to implement it - was going to ask a question but as it would be a duplicate of this one I am posting it as an answer and hope to get some feedback. My questions would be:
why this "SyntaxError: import * is not allowed in function 'init'" while there are no nested functions ?
dir has a lot of warnings in its doc - in particular it attempts to produce the most relevant, rather than complete, information - this complete worries me a bit
is there no builtin way to do an import * ? even in python 3 ?
I want to import some package depending on which value the user chooses.
The default is file1.py:
from files import file1
If user chooses file2, it should be :
from files import file2
In PHP, I can do this using variable variables:
$file_name = 'file1';
include($$file_name);
$file_name = 'file2';
include($$file_name);
How can I do this in Python?
Python doesn't have a feature that's directly equivalent to PHP's "variable variables". To get a "variable variable"'s value (or the value of any other expression) you can use the eval function.
foo = "Hello World"
print eval("foo")
However, this can't be used in an import statement.
It is possible to use the __import__ function to import using a variable.
package = "os"
name = "path"
imported = getattr(__import__(package, fromlist=[name]), name)
is equivalent to
from os import path as imported
Old thread, but I needed the answer, so someone else still might...
There's a cleaner way to do this in Python 2.7+:
import importlib
my_module = importlib.import_module("package.path.%s" % module_name)
As Fredrik Lundh states:
Anyway, here’s how these statements and functions work:
import X imports the module X, and creates a reference to that module
in the current namespace. Or in other words, after you’ve run this
statement, you can use X.name to refer to things defined in module X.
from X import * imports the module X, and creates references in the
current namespace to all public objects defined by that module (that
is, everything that doesn’t have a name starting with “_”). Or in
other words, after you’ve run this statement, you can simply use a
plain name to refer to things defined in module X. But X itself is not
defined, so X.name doesn’t work. And if name was already defined, it
is replaced by the new version. And if name in X is changed to point
to some other object, your module won’t notice.
from X import a, b, c imports the module X, and creates references in
the current namespace to the given objects. Or in other words, you can
now use a and b and c in your program.
Finally, X = __import__(‘X’) works like import X, with the difference
that you 1) pass the module name as a string, and 2) explicitly assign
it to a variable in your current namespace.
And by the way that's the last one method that you're intrested in.
Simply write (for example):
var = "datetime"
module = __import__(var)
Basing myself on mattjbray's answer:
from importlib import import_module
# lookup in a set is in constant time
safe_names = {"file1.py", "file2.py", "file3.py", ...}
user_input = ...
if user_input in safe_names:
file = import_module(user_input)
else:
print("Nope, not doing this.")
Saves a few lines of code, and allows you to set safe_names programmatically, or load multiple modules and assign them to a dict, for example.
It's probably a very bad idea to let the user choose what to import. Packages can execute code on import, so you're effectively allowing a user to arbitrarily execute code on your system! Much safer to do something like
if user_input == 'file1.py':
from files import file1 as file
elif user_input == 'file2.py':
from files import file2 as file
else:
file = None
print "Sorry, you can't import that file"
This question already has answers here:
How can I import a module dynamically given its name as string?
(10 answers)
Closed 3 months ago.
I'm trying to dynamically load modules I've created.
Right now this works properly:
import structures.index
But if I try the same thing by importing it dynamically, it fails.
struct = __import__("structures.index")
Error supplied is:
Error ('No module named structures.index',)
Any ideas why?
Edit: When using full scope (it sort of works?):
struct = __import__("neoform.structures.index")
This doesn't throw any errors, however, it isn't loading the index module, it's loading the "neoform" module instead.
The result of "struct" is:
<module 'neoform' from '/neoform/__init__.py'>
Also, as a side question, how can I then instantiate a class within a dynamically loaded module? (assuming all the modules contain a common class name).
Edit: Solution: (thanks coonj & Rick) This ended up being what worked. Not sure why (yet), but the fromlist had to be something "anything apparently, since it worked when I put the letter "a" as a value (strange, given that the file only had 1 class in it).
def get_struct_module(self, name):
try:
return = __import__("neoform.structures." + name, fromlist='*')
except ImportError, e:
self.out.add("Could not load struct: neoform.structure." + name + "\n\n" + "Error " + str(e.args))
I'm not sure what "it fails" means, so I'll just mention that __import__('structures.index') should, in fact, work, but it doesn't assign the module name in the current scope. To do that (and then use a class in the dynamically imported module), you'll have to use:
structures = __import__('structures.index')
structures.index.SomeClass(...)
The complete details on __import__ are available here.
Edit: (based on question edit)
To import neoform.structures.index, and return the index module, you would do the following:
structures = __import__('neoform.structures.index',
fromlist=['does not in fact matter what goes here!'])
So if you have a list of package names packages, you can import their index modules and instantiate some MyClass class for each using the following code:
modules = [ __import__('neoform.%s.index' % pkg, fromlist=['a'])
for pkg in packages ]
objects = [ m.MyClass() for m in modules ]
To import sub-modules, you need to specify them in the fromlist arg of __import__() Fo example, the equivalent of:
import structures.index
is:
structures = __import__('structures', fromlist=['index'])
To do this in a map is a little more tricky...
import mod1.index
import mod2.index
import mod3.index
For those imports, you would want to define a new function to get the index sub-module from each module:
def getIndexMods(mod_names):
mod_list = map(lambda x: __import__(x, fromlist='index'))
index_mods = [mod.index for mod in mod_list]
return index_mods
Now, you can do this to get references to all index modules:
index_mods = getIndexMods(['mod1', 'mod2', 'mod3'])
Also, if you want to grab sub-modules that are not named 'index' then you could do this:
mod1, mod2, mod3 = map(lambda x,y: __import__(x, fromlist=y),
['mod1', 'mod2', 'mod3'], ['index1', 'index2', 'index3'])
Use full scope ("neoform.structures.index") with this helper method.
def import_module(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
module = import_module("neoform.structures.index")
# do stuff with module
>>> import imp
>>> fm = imp.find_module('index', ['./structures']) # for submodule
>>> mymod = imp.load_module('structures.index', *fm)
>>> mymod
<module 'structures.index' from './structures/index.pyc'>
>>> x = mymod.insideIndex()
Initialising index class...
Voila!
Java programmer here, but I think you need the imp module
Why on earth would you replace
import structures.index
with
map(__import__, ["structures.index"])
The first one (a) works, (b) is dynamic and (c) is directly supported. What possible use case is there for replacing easy-to-change, plain-text source with something more complex?
In short: don't do this. It doesn't have any value.
Edit
The "I'm getting the import from a database" is a noble effort, but still not sensible. What code block depends on those imports? That whole code block -- imports and all -- is what you want to execute. That whole code block -- imports, statements and everything -- should be a plain old python module file.
Import that block of code from the file system. Use the database to identify which file, the author of the file -- anything you want to use the database for. But simply import and execute the module the simplest possible way.
Really late post here. But I was searching for this question on google. I did some trial and error. Not sure if this snippet will help but here it is. Using it for Flask site.
modules = ['frontend', 'admin']
for module in modules:
mod = __init__('controllers.%s' % module, fromlist=[module])
app.register_blueprint(mod.blueprint_mod)
# or
from importlib import import_module
modules = ['frontend', 'admin']
for module in modules:
mod = import_module('controllers.%s' % module)
app.regitster_blueprint(mod.blueprint_mod)