I have a python script that I'm picking up from someone else and am trying to understand what's happening when it runs.
I have a file in my current directory called __version__.py that contains the following line:
__version__ = "1.0"
In a separate script I have the following code:
import os
gdict = {}
curr_dir = os.path.abspath(os.path.dirname(__file__))
with open(os.path.join(curr_dir, "__version__.py")) as f:
exec(f.read(), gdict)
I'm puzzled about what's going on with the exec statement. My understanding from reading the documentation is that gdict is there to specify which global functions are available to the exec() function, and since it's empty there shouldn't be anything available to exec() beyond the built-in functions. I had thought that gdict would remain empty, but when I run this code and then look at gdict.keys() I see:
dict_keys(['__builtins__', '__version__'])
I understand that the f.read() is creating a global variable called "__version__" with a value of "1.0", but how is gdict being populated?
gdict from your program changes because python dictionaries are mutable. All available builtins will be added to the globals dict when you run exec(f.read(), gdict) and this is reflected when you view gdict.keys()
Related
I found the following code snippet that I can't seem to make work for my scenario (or any scenario at all):
def load(code):
# Delete all local variables
globals()['code'] = code
del locals()['code']
# Run the code
exec(globals()['code'])
# Delete any global variables we've added
del globals()['load']
del globals()['code']
# Copy k so we can use it
if 'k' in locals():
globals()['k'] = locals()['k']
del locals()['k']
# Copy the rest of the variables
for k in locals().keys():
globals()[k] = locals()[k]
I created a file called "dynamic_module" and put this code in it, which I then used to try to execute the following code which is a placeholder for some dynamically created string I would like to execute.
import random
import datetime
class MyClass(object):
def main(self, a, b):
r = random.Random(datetime.datetime.now().microsecond)
a = r.randint(a, b)
return a
Then I tried executing the following:
import dynamic_module
dynamic_module.load(code_string)
return_value = dynamic_module.MyClass().main(1,100)
When this runs it should return a random number between 1 and 100. However, I can't seem to get the initial snippet I found to work for even the simplest of code strings. I think part of my confusion in doing this is that I may misunderstand how globals and locals work and therefore how to properly fix the problems I'm encountering. I need the code string to use its own imports and variables and not have access to the ones where it is being run from, which is the reason I am going through this somewhat over-complicated method.
You should not be using the code you found. It is has several big problems, not least that most of it doesn't actually do anything (locals() is a proxy, deleting from it has no effect on the actual locals, it puts any code you execute in the same shared globals, etc.)
Use the accepted answer in that post instead; recast as a function that becomes:
import sys, imp
def load_module_from_string(code, name='dynamic_module')
module = imp.new_module(name)
exec(code, mymodule.__dict__)
return module
then just use that:
dynamic_module = load_module_from_string(code_string)
return_value = dynamic_module.MyClass().main(1, 100)
The function produces a new, clean module object.
In general, this is not how you should dynamically import and use external modules. You should be using __import__ within your function to do this. Here's a simple example that worked for me:
plt = __import__('matplotlib.pyplot', fromlist = ['plt'])
plt.plot(np.arange(5), np.arange(5))
plt.show()
I imagine that for your specific application (loading from code string) it would be much easier to save the dynamically generated code string to a file (in a folder containing an __init__.py file) and then to call it using __import__. Then you could access all variables and functions of the code as parts of the imported module.
Unless I'm missing something?
I am trying to make generic config, and thus config parser. There are two config files say A and B. I want to parse sections and make global values from them according to hardcoded list.
Here is an example:
in config file:
[section]
var1 = value1
var2 = value2
In global scope:
some_global_list = [ ["var1","var2"], ["var3","var4"] ]
in function to unpack this values, by ConfigParser:
configparser = ConfigParser.RawConfigParser()
configparser.read(some_config_filename)
for variables in some_global_list:
globals()[section]=dict()
for element in configparser.items(section):
globals()[section].update({element[0]:element[1]})
This works nicely...however. Scope of globals() seem to be limited to function which is obviously not what I intended. I can access variable only while in that function.
Could someone share better yet simple idea?
I know that i might move code to main and not to worry, but I don't think it is a good idea.
I thought also about making some generator (sorry for pseudocode here):
in global scope:
for x in some_global_list:
globals()[x] = x
also tried adding this to function:
for x in some_global_list[0]:
global x
but got nowhere.
Edit :
After discussion below, here it is:
Problem solved like this:
removed whole configuration from main script to module
imported (from module import somefunction) config from module
removed globals() in fact didnt need them, since function was changed a little like so:
in function:
def somefunction:
#(...)
configparser = ConfigParser.RawConfigParser()
configparser.read(some_config_filename)
temp_list=[]
for variables in some_global_list:
tmp=dict()
for element in configparser.items(section):
tmp.update({element[0]:element[1]})
temp_list.append (tmp)
return temp_list #this is pack for one file.
now in main script
tmp=[]
for i,conf_file in enumerate([args.conf1,args.conf2,args.conf3]):
if conf_file:
try:
tmp.append([function(params...)])
except:
#handling here
#but since i needed those config names as global variables
for j,variable_set in enumerate(known_variable_names[i]):
globals()[variable_set] = tmp[i][j]
so unfortunate hack presists. But seems to work. Thx for Your help guys.
I'm accepting (if thats possible) below answer since it gave me good idea :)
A simple way to solve this issue is in your application package within the __init__.py you can do something similar to the following:
app_config = read_config()
def read_config():
configparser = ConfigParser.RawConfigParser()
configparser.read(some_config_filename)
return configparser.as_dict() #An imaginery method which returns the vals as dict.
The "app_config" variable can be imported into any other module within the application.
I'm trying to build a sort of script system in python that will allow small snippets of code to be selected and executed at runtime inside python.
Essentially I want to be able to load a small python file like
for i in Foo: #not in a function.
print i
Where somewhere else in the program I assign what Foo will be. As if Foo served as a function argument to the entire loaded python file instead of a single function
So somewhere else
FooToPass = GetAFoo ()
TempModule = __import__ ("TheSnippit",<Somehow put {'Foo' : FooToPass} in the locals>)
It is considered bad style to have code with side effects at module level. If you want your module to do something, put that code in a function, make Foo a parameter of this function and call it with the desired value.
Python's import mechanism does not allow to preinitialise a module namespace. If you want to do this anyway (which is, in my opinion, confusing and unnecessary), you have to fiddle around with details of the import mechanism. Example implementation (untested):
import imp
import sys
def my_import(module_name, globals):
if module_name in sys.modules:
return sys.modules[module_name]
module = imp.new_module(module_name)
vars(module).update(globals)
f, module.__file__, options = imp.find_module(module_name)
exec f.read() in vars(module)
f.close()
sys.modules[module_name] = module
return module
(Important: See update below.)
I'm trying to write a function, import_something, that will important certain modules. (It doesn't matter which for this question.) The thing is, I would like those modules to be imported at the level from which the function is called. For example:
import_something() # Let's say this imports my_module
my_module.do_stuff() #
Is this possible?
Update:
Sorry, my original phrasing and example were misleading. I'll try to explain my entire problem. What I have is a package, which has inside it some modules and packages. In its __init__.py I want to import all the modules and packages. So somewhere else in the program, I import the entire package, and iterate over the modules/packages it has imported.
(Why? The package is called crunchers, and inside it there are defined all kinds of crunchers, like CruncherThread, CruncherProcess, and in the future perhaps MicroThreadCruncher. I want the crunchers package to automatically have all the crunchers that are placed in it, so later in the program when I use crunchers I know it can tell exactly which crunchers I have defined.)
I know I can solve this if I avoid using functions at all, and do all imports on the main level with for loops and such. But it's ugly and I want to see if I can avoid it.
If anything more is unclear, please ask in comments.
Functions have the ability to return something to where they were called. Its called their return value :p
def import_something():
# decide what to import
# ...
mod = __import__( something )
return mod
my_module = import_something()
my_module.do_stuff()
good style, no hassle.
About your update, I think adding something like this to you __init__.py does what you want:
import os
# make a list of all .py files in the same dir that dont start with _
__all__ = installed = [ name for (name,ext) in ( os.path.splitext(fn) for fn in os.listdir(os.path.dirname(__file__))) if ext=='.py' and not name.startswith('_') ]
for name in installed:
# import them all
__import__( name, globals(), locals())
somewhere else:
import crunchers
crunchers.installed # all names
crunchers.cruncherA # actual module object, but you can't use it since you don't know the name when you write the code
# turns out the be pretty much the same as the first solution :p
mycruncher = getattr(crunchers, crunchers.installed[0])
You can monkey with the parent frame in CPython to install the modules into the locals for that frame (and only that frame). The downsides are that a) this is really quite hackish and b) sys._getframe() is not guaranteed to exist in other python implementations.
def importer():
f = sys._getframe(1) # Get the parent frame
f.f_locals["some_name"] = __import__(module_name, f.f_globals, f.f_locals)
You still have to install the module into f_locals, since import won't actually do that for you - you just supply the parent frame locals and globals for the proper context.
Then in your calling function you can have:
def foo():
importer() # Magically makes 'some_name' available to the calling function
some_name.some_func()
Are you looking for something like this?
def my_import(*names):
for name in names:
sys._getframe(1).f_locals[name] = __import__(name)
then you can call it like this:
my_import("os", "re")
or
namelist = ["os", "re"]
my_import(*namelist)
According to __import__'s help:
__import__(name, globals={}, locals={}, fromlist=[], level=-1) -> module
Import a module. The globals are only used to determine the context;
they are not modified. ...
So you can simply get the globals of your parent frame and use that for the __import__ call.
def import_something(s):
return __import__(s, sys._getframe(1).f_globals)
Note: Pre-2.6, __import__'s signature differed in that it simply had optional parameters instead of using kwargs. Since globals is the second argument in both cases, the way it's called above works fine. Just something to be aware of if you decided to use any of the other arguments.
I would like to load a .py file at runtime. This .py file is basically a config file with the following format:
var1=value
var2=value
predicate_function=func line : <return true or false>
Once this file is loaded, I would like to be able to access var1, var2 and predicate_function. For each line, I'll pass it to the predicate function, and if it returns false, I'll ignore it.
In any case, I'm not sure how to load a python file at runtime and access its variables.
Clarification: there may be any number of these config files that I need to pass to the main program and I won't know their names until runtime. Google tells me I should use __import__. I'm not sure how to correctly use that method and then access the variables of the imported file.
As written in the python official documentation, if you just want to import a module by name, you can look it up in the sys.modules dictionary after using __import__.
Supposing your configuration is in myproject.mymodule, you would do like that :
module_name = 'myproject.mymodule'
import sys
__import__(module_name)
mymodule = sys.modules[module_name]
# Then you can just access your variables and functions
print mymodule.var1
print mymodule.var2
# etc...
You can also use the return value of __import__ statement but you will have to understand fully how python works with namespaces and scopes.
You just need to be able to dynamically specify the imports and then dynamically get at the variables.
Let's say your config file is bar.py and looks like this:
x = 3
y = 4
def f(x): return (x<4)
Then your code should look like this:
import sys
# somehow modnames should be a list of strings that are the names of config files
#
# you can do this more dynamically depending on what you're doing
modnames = ['bar']
for modname in modnames:
exec('import %s' % modname)
for modname in modnames:
mod = sys.modules[modname]
for k in mod.__dict__:
if k[:2] != '__':
print modname, k, mod.__dict__[k]
I get this output:
bar f <function f at 0x7f2354eb4cf8>
bar x 3
bar y 4
Then you at least have all the variables and functions. I didn't quite get what you wanted from the predicate functions, but maybe you can get that on your own now.
To access another Python module, you import it. execfile has been mentioned by a couple people, but it is messy and dangerous. execfile clutters your namespace, possibly even messing up the code you are running. When you want to access another Python source file, use the import statement.
Even better would be not to use a Python file for configuration at all, but rather to use the builtin module ConfigParser or a serialization format like JSON. This way your configuration files don't allow execution of arbitrary (possibly malicious) code, doesn't require people to know Python to configure your program, and can easily be altered programatically.
If the imported module is on the regular search path, you can use __import__.
If you need to load the module from an arbitrary path in the filesystem, use imp.load_module.
Be sure to consider the security implications of loading arbitrary user-specified code.
In Python 2.*, execfile works (I recommend passing a specific dictionary and accessing the variables from there -- as the note in the docs says, execfile can't affect the calling function's locals() dictionary).
In Python 3.*, execfile has been removed, so do, instead:
with open('thefile.py') as f:
exec(f.read(), somedict)
Since the Python version hasn't been clearly mentioned, it is worth pointing out that the imp module has been deprecated in newer Python versions in favor of the importlib module. Example here.
I'm kinda late to the party, but I want to present an alternative answer nonetheless.
If you want to import code without affecting the global module namespace, you can create an anonymous module (using types.ModuleType) and load arbitrary code in it (using compile and exec). For instance, like this:
import types
filename = "/path/to/your/file.py"
with open(filename) as fp:
code = compile(fp.read(), filename, "exec")
config_module = types.ModuleType("<config>")
exec code in config_module.__dict__
You can then access the variables as config_module.var1, &c.
If you want to have a configuration file that will only be edited by the user when the program isn't running, just import it as a normal python file
ie.
main.py:
import config
print config.var1
config.py:
var="var12"
var2 = 100.5
try the imp module : http://docs.python.org/library/imp.html