A textbook suggests I should be able to do this:
d = {}
exec("C://Users//Dave//Desktop//Bot//bot_config_data.py", globals(), d)
File "<string>", line 1 C://Users//Dave//Desktop//Bot//bot_config_data.py
^
SyntaxError: invalid syntax
I can do this:
d = {}
exec('from bot_config_data import price_data', globals(), d)
But, I would like to do the former.
I'm trying to write a method which overrides config data from various files.
Am I completely off base here?
Update
The book was quite misleading. It posted part of the code, the result of the complete code block, and then gave the remainder. As I was referencing, rather than working through it cover to cover, I tripped over myself.
This is the code I now have:
data = {}
file = 'C:\\Users\\Dave\\Desktop\\Bot\\bot_config_data.py'
with open(file) as f:
code = compile(f.read(), file, "exec")
exec(code, globals(), data)
price_data = data["price_data"]
Update2
Using Mad Physicist's answer below, my code would be:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
filepath = 'C:\\Users\\Dave\\Desktop\\Bot\\bot_config_data.py'
module_data = os.path.basename(filepath)
spec = spec_from_loader(module_data, SourceFileLoader(module_data, filepath))
bot_config_data = module_from_spec(spec)
spec.loader.exec_module(bot_config_data)
price_data = bot_config_data.price_data
You are completely off-base and the textbook is giving you some very bad advice.
exec runs Python code. As in Python statements, not the name of a file. From the docs:
exec(object[, globals[, locals]])
This function supports dynamic execution of Python code. object must be either a string or a code object. If it is a string, the string is parsed as a suite of Python statements which is then executed (unless a syntax error occurs)
That is why your second statement works fine. If you want to run a Python module, import it. If you need to import using a dynamic name instead of a hard-coded import statement, you can use importlib.import_module:
price_data = importlib.import_module('bot_config_data').price_data
This will run the whole import machinery for you, including ensuring that bot_config_data ends up in sys.modules.
If you really need something even fancier, you can use the __import__ machinery. __import__ is the under-the-hood implementation of the import statement:
d = {}
bot_config_data = __import__('bot_config_data', locals=d, from_list=['price_data'])
price_data = bot_config_data.price_data
If you want full control over the process, you can use the low-level machinery described here: How to import a module given the full path?. In particular see my answer to that question, since it describes how to load random text files as Python scripts: https://stackoverflow.com/a/43602557/2988730:
from importlib.util import spec_from_loader, module_from_spec
from importlib.machinery import SourceFileLoader
spec = spec_from_loader("bot_config_data", SourceFileLoader("bot_config_data", "C:/Users/Dave/Desktop/Bot/bot_config_data.py"))
bot_config_data = module_from_spec(spec)
spec.loader.exec_module(bot_config_data)
price_data = bot_config_data.price_data
Final note: It looks like you blindly converted all the \\ to // in your path. Forward slashes do not need to be escaped, so you only need single forward slashes. If you want to avoid escaping your backslashes, put r in front of your string to make it a raw string.
If you need to import using a dynamic name instead of a hard-coded
import statement, you can use importlib.import_module:
price_data = importlib.import_module('bot_config_data').price_data
Huh? How is that "dynamic"? 'bot_config_data' is literally a string literal.
If you really want to import modules dynamically, contrary to what is widely believed, the only way to do this is with exec(). I challenge anyone to find or submit an example of importlib.import_module or __import__ being used to truly dynamically import modules.
To be clear: __import__ allows for the dynamic handling of imports, which is not the same as dynamically importing, as that is a chicken/egg type problem only exec() can solve.
The problem is that importlib.import_module() returns an object, which needs an assignment. (It doesn't just add to the global namespace as is often implied.) But you can't use the same argument on both sides of the assignment. So while it allows you to do handle imports in a more dynamic way, it provides no way of dynamically importing modules.
Related
I am in the habit of using raw_input(...) for certain debugging. However, in python3 this has changed to input(...). Is there a way to define an alias at the top of my project, such as:
# __init__.py
raw_input = input
I tried the above, but it only worked in the file I added it to, and not any other files in that directory. I'd like this to work basically in every file within my python repository.
You can define all aliases in a separate file (e.g. aliases.py) then import said file where needed (i.e. import aliases).
The con with this method that you'll be referencing the alias through aliases.alias unless you make the import stricter (i.e. from aliases import raw_input) or if you don't care about avoiding a wildcard import (i.e. from aliases import *).
Additionally, if you don't mind another import in the aliases file you can use the builtins namespace:
import builtins
builtins.raw_input = input
You still have to define all aliases separate file (e.g. aliases.py) then import said file where needed (i.e. import aliases) but the advantage of using the builtins namespace is that you can use that import exactly as given.
You can do it by creating a module for creating the renaming function and then importing it to every file you want to like this:
First the module function declaration in alias.py
def raw_input(a):
return input(a)
Secondly, import to another file:
from alias import raw_input
x = raw_input("hello world")
print(x)
Sadly, you will have to make the import of the module to every file you want to use the renamed function.
Hope it works for you!
Put this at the top, and you will get exactly what you want.
import builtins
builtins.raw_input = builtins.input
It is guaranteed to work, but generally considered a bad practice (everybody will be confused with where is that raw_input defined)
I found the following code snippet that I can't seem to make work for my scenario (or any scenario at all):
def load(code):
# Delete all local variables
globals()['code'] = code
del locals()['code']
# Run the code
exec(globals()['code'])
# Delete any global variables we've added
del globals()['load']
del globals()['code']
# Copy k so we can use it
if 'k' in locals():
globals()['k'] = locals()['k']
del locals()['k']
# Copy the rest of the variables
for k in locals().keys():
globals()[k] = locals()[k]
I created a file called "dynamic_module" and put this code in it, which I then used to try to execute the following code which is a placeholder for some dynamically created string I would like to execute.
import random
import datetime
class MyClass(object):
def main(self, a, b):
r = random.Random(datetime.datetime.now().microsecond)
a = r.randint(a, b)
return a
Then I tried executing the following:
import dynamic_module
dynamic_module.load(code_string)
return_value = dynamic_module.MyClass().main(1,100)
When this runs it should return a random number between 1 and 100. However, I can't seem to get the initial snippet I found to work for even the simplest of code strings. I think part of my confusion in doing this is that I may misunderstand how globals and locals work and therefore how to properly fix the problems I'm encountering. I need the code string to use its own imports and variables and not have access to the ones where it is being run from, which is the reason I am going through this somewhat over-complicated method.
You should not be using the code you found. It is has several big problems, not least that most of it doesn't actually do anything (locals() is a proxy, deleting from it has no effect on the actual locals, it puts any code you execute in the same shared globals, etc.)
Use the accepted answer in that post instead; recast as a function that becomes:
import sys, imp
def load_module_from_string(code, name='dynamic_module')
module = imp.new_module(name)
exec(code, mymodule.__dict__)
return module
then just use that:
dynamic_module = load_module_from_string(code_string)
return_value = dynamic_module.MyClass().main(1, 100)
The function produces a new, clean module object.
In general, this is not how you should dynamically import and use external modules. You should be using __import__ within your function to do this. Here's a simple example that worked for me:
plt = __import__('matplotlib.pyplot', fromlist = ['plt'])
plt.plot(np.arange(5), np.arange(5))
plt.show()
I imagine that for your specific application (loading from code string) it would be much easier to save the dynamically generated code string to a file (in a folder containing an __init__.py file) and then to call it using __import__. Then you could access all variables and functions of the code as parts of the imported module.
Unless I'm missing something?
I'm trying to build a sort of script system in python that will allow small snippets of code to be selected and executed at runtime inside python.
Essentially I want to be able to load a small python file like
for i in Foo: #not in a function.
print i
Where somewhere else in the program I assign what Foo will be. As if Foo served as a function argument to the entire loaded python file instead of a single function
So somewhere else
FooToPass = GetAFoo ()
TempModule = __import__ ("TheSnippit",<Somehow put {'Foo' : FooToPass} in the locals>)
It is considered bad style to have code with side effects at module level. If you want your module to do something, put that code in a function, make Foo a parameter of this function and call it with the desired value.
Python's import mechanism does not allow to preinitialise a module namespace. If you want to do this anyway (which is, in my opinion, confusing and unnecessary), you have to fiddle around with details of the import mechanism. Example implementation (untested):
import imp
import sys
def my_import(module_name, globals):
if module_name in sys.modules:
return sys.modules[module_name]
module = imp.new_module(module_name)
vars(module).update(globals)
f, module.__file__, options = imp.find_module(module_name)
exec f.read() in vars(module)
f.close()
sys.modules[module_name] = module
return module
I would like to load a .py file at runtime. This .py file is basically a config file with the following format:
var1=value
var2=value
predicate_function=func line : <return true or false>
Once this file is loaded, I would like to be able to access var1, var2 and predicate_function. For each line, I'll pass it to the predicate function, and if it returns false, I'll ignore it.
In any case, I'm not sure how to load a python file at runtime and access its variables.
Clarification: there may be any number of these config files that I need to pass to the main program and I won't know their names until runtime. Google tells me I should use __import__. I'm not sure how to correctly use that method and then access the variables of the imported file.
As written in the python official documentation, if you just want to import a module by name, you can look it up in the sys.modules dictionary after using __import__.
Supposing your configuration is in myproject.mymodule, you would do like that :
module_name = 'myproject.mymodule'
import sys
__import__(module_name)
mymodule = sys.modules[module_name]
# Then you can just access your variables and functions
print mymodule.var1
print mymodule.var2
# etc...
You can also use the return value of __import__ statement but you will have to understand fully how python works with namespaces and scopes.
You just need to be able to dynamically specify the imports and then dynamically get at the variables.
Let's say your config file is bar.py and looks like this:
x = 3
y = 4
def f(x): return (x<4)
Then your code should look like this:
import sys
# somehow modnames should be a list of strings that are the names of config files
#
# you can do this more dynamically depending on what you're doing
modnames = ['bar']
for modname in modnames:
exec('import %s' % modname)
for modname in modnames:
mod = sys.modules[modname]
for k in mod.__dict__:
if k[:2] != '__':
print modname, k, mod.__dict__[k]
I get this output:
bar f <function f at 0x7f2354eb4cf8>
bar x 3
bar y 4
Then you at least have all the variables and functions. I didn't quite get what you wanted from the predicate functions, but maybe you can get that on your own now.
To access another Python module, you import it. execfile has been mentioned by a couple people, but it is messy and dangerous. execfile clutters your namespace, possibly even messing up the code you are running. When you want to access another Python source file, use the import statement.
Even better would be not to use a Python file for configuration at all, but rather to use the builtin module ConfigParser or a serialization format like JSON. This way your configuration files don't allow execution of arbitrary (possibly malicious) code, doesn't require people to know Python to configure your program, and can easily be altered programatically.
If the imported module is on the regular search path, you can use __import__.
If you need to load the module from an arbitrary path in the filesystem, use imp.load_module.
Be sure to consider the security implications of loading arbitrary user-specified code.
In Python 2.*, execfile works (I recommend passing a specific dictionary and accessing the variables from there -- as the note in the docs says, execfile can't affect the calling function's locals() dictionary).
In Python 3.*, execfile has been removed, so do, instead:
with open('thefile.py') as f:
exec(f.read(), somedict)
Since the Python version hasn't been clearly mentioned, it is worth pointing out that the imp module has been deprecated in newer Python versions in favor of the importlib module. Example here.
I'm kinda late to the party, but I want to present an alternative answer nonetheless.
If you want to import code without affecting the global module namespace, you can create an anonymous module (using types.ModuleType) and load arbitrary code in it (using compile and exec). For instance, like this:
import types
filename = "/path/to/your/file.py"
with open(filename) as fp:
code = compile(fp.read(), filename, "exec")
config_module = types.ModuleType("<config>")
exec code in config_module.__dict__
You can then access the variables as config_module.var1, &c.
If you want to have a configuration file that will only be edited by the user when the program isn't running, just import it as a normal python file
ie.
main.py:
import config
print config.var1
config.py:
var="var12"
var2 = 100.5
try the imp module : http://docs.python.org/library/imp.html
This question already has answers here:
How can I import a module dynamically given its name as string?
(10 answers)
Closed 3 months ago.
I'm trying to dynamically load modules I've created.
Right now this works properly:
import structures.index
But if I try the same thing by importing it dynamically, it fails.
struct = __import__("structures.index")
Error supplied is:
Error ('No module named structures.index',)
Any ideas why?
Edit: When using full scope (it sort of works?):
struct = __import__("neoform.structures.index")
This doesn't throw any errors, however, it isn't loading the index module, it's loading the "neoform" module instead.
The result of "struct" is:
<module 'neoform' from '/neoform/__init__.py'>
Also, as a side question, how can I then instantiate a class within a dynamically loaded module? (assuming all the modules contain a common class name).
Edit: Solution: (thanks coonj & Rick) This ended up being what worked. Not sure why (yet), but the fromlist had to be something "anything apparently, since it worked when I put the letter "a" as a value (strange, given that the file only had 1 class in it).
def get_struct_module(self, name):
try:
return = __import__("neoform.structures." + name, fromlist='*')
except ImportError, e:
self.out.add("Could not load struct: neoform.structure." + name + "\n\n" + "Error " + str(e.args))
I'm not sure what "it fails" means, so I'll just mention that __import__('structures.index') should, in fact, work, but it doesn't assign the module name in the current scope. To do that (and then use a class in the dynamically imported module), you'll have to use:
structures = __import__('structures.index')
structures.index.SomeClass(...)
The complete details on __import__ are available here.
Edit: (based on question edit)
To import neoform.structures.index, and return the index module, you would do the following:
structures = __import__('neoform.structures.index',
fromlist=['does not in fact matter what goes here!'])
So if you have a list of package names packages, you can import their index modules and instantiate some MyClass class for each using the following code:
modules = [ __import__('neoform.%s.index' % pkg, fromlist=['a'])
for pkg in packages ]
objects = [ m.MyClass() for m in modules ]
To import sub-modules, you need to specify them in the fromlist arg of __import__() Fo example, the equivalent of:
import structures.index
is:
structures = __import__('structures', fromlist=['index'])
To do this in a map is a little more tricky...
import mod1.index
import mod2.index
import mod3.index
For those imports, you would want to define a new function to get the index sub-module from each module:
def getIndexMods(mod_names):
mod_list = map(lambda x: __import__(x, fromlist='index'))
index_mods = [mod.index for mod in mod_list]
return index_mods
Now, you can do this to get references to all index modules:
index_mods = getIndexMods(['mod1', 'mod2', 'mod3'])
Also, if you want to grab sub-modules that are not named 'index' then you could do this:
mod1, mod2, mod3 = map(lambda x,y: __import__(x, fromlist=y),
['mod1', 'mod2', 'mod3'], ['index1', 'index2', 'index3'])
Use full scope ("neoform.structures.index") with this helper method.
def import_module(name):
mod = __import__(name)
components = name.split('.')
for comp in components[1:]:
mod = getattr(mod, comp)
return mod
module = import_module("neoform.structures.index")
# do stuff with module
>>> import imp
>>> fm = imp.find_module('index', ['./structures']) # for submodule
>>> mymod = imp.load_module('structures.index', *fm)
>>> mymod
<module 'structures.index' from './structures/index.pyc'>
>>> x = mymod.insideIndex()
Initialising index class...
Voila!
Java programmer here, but I think you need the imp module
Why on earth would you replace
import structures.index
with
map(__import__, ["structures.index"])
The first one (a) works, (b) is dynamic and (c) is directly supported. What possible use case is there for replacing easy-to-change, plain-text source with something more complex?
In short: don't do this. It doesn't have any value.
Edit
The "I'm getting the import from a database" is a noble effort, but still not sensible. What code block depends on those imports? That whole code block -- imports and all -- is what you want to execute. That whole code block -- imports, statements and everything -- should be a plain old python module file.
Import that block of code from the file system. Use the database to identify which file, the author of the file -- anything you want to use the database for. But simply import and execute the module the simplest possible way.
Really late post here. But I was searching for this question on google. I did some trial and error. Not sure if this snippet will help but here it is. Using it for Flask site.
modules = ['frontend', 'admin']
for module in modules:
mod = __init__('controllers.%s' % module, fromlist=[module])
app.register_blueprint(mod.blueprint_mod)
# or
from importlib import import_module
modules = ['frontend', 'admin']
for module in modules:
mod = import_module('controllers.%s' % module)
app.regitster_blueprint(mod.blueprint_mod)