I'm trying to dynamically update code during runtime by reloading modules using importlib.reload. However, I need a specific module variable to be set before the module's code is executed. I could easily set it as an attribute after reloading but each module would have already executed its code (e.g., defined its default arguments).
A simple example:
# module.py
def do():
try:
print(a)
except NameError:
print('failed')
# main.py
import module
module.do() # prints failed
module.a = 'succeeded'
module.do() # prints succeeded
The desired pseudocode:
import_module_without_executing_code module
module.initialise(a = 'succeeded')
module.do()
Is there a way to control module namespace initialisation (like with classes using metaclasses)?
It's not usually a good idea to use reload other than for interactive debugging. For example, it can easily create situations where two objects of type module.A are not the same type.
What you want is execfile. Pass a globals dictionary (you don't need an explicit locals dictionary) to keep each execution isolated; anything you store in it ahead of time acts exactly like the "pre-set" variables you want. If you do want to have a "real" module interface change, you can have a wrapper module that calls (or just holds as an attribute) the most recently loaded function from your changing file.
Of course, since you're using Python 3, you'll have to use one of the replacements for execfile.
Strictly speaking, I don't believe there is a way to do what you're describing in Python natively. However, assuming you own the module you're trying to import, a common approach with Python modules that need some initializing input is to use an init function.
If all you need is some internal variables to be set, like a in you example above, that's easy: just declare some module-global variables and set them in your init function:
Demo: https://repl.it/MyK0
Module:
## mymodule.py
a = None
def do():
print(a)
def init(_a):
global a
a = _a
Main:
## main.py
import mymodule
mymodule.init(123)
mymodule.do()
mymodule.init('foo')
mymodule.do()
Output:
123
foo
Where things can get trickier is if you need to actually redefine some functions because some dynamic internal something is dependent on the input you give. Here's one solution, borrowed from https://stackoverflow.com/a/1676860. Basically, the idea is to grab a reference to the current module by using the magic variable __name__ to index into the system module dictionary, sys.modules, and then define or overwrite the functions that need it. We can define the functions locally as inner functions, then add them to the module:
Demo: https://repl.it/MyHT/2
Module:
## mymodule.py
import sys
def init(a):
current_module = sys.modules[__name__]
def _do():
try:
print(a)
except NameError:
print('failed')
current_module.do = _do
Related
I've run into a bit of a wall importing modules in a Python script. I'll do my best to describe the error, why I run into it, and why I'm tying this particular approach to solve my problem (which I will describe in a second):
Let's suppose I have a module in which I've defined some utility functions/classes, which refer to entities defined in the namespace into which this auxiliary module will be imported (let "a" be such an entity):
module1:
def f():
print a
And then I have the main program, where "a" is defined, into which I want to import those utilities:
import module1
a=3
module1.f()
Executing the program will trigger the following error:
Traceback (most recent call last):
File "Z:\Python\main.py", line 10, in <module>
module1.f()
File "Z:\Python\module1.py", line 3, in f
print a
NameError: global name 'a' is not defined
Similar questions have been asked in the past (two days ago, d'uh) and several solutions have been suggested, however I don't really think these fit my requirements. Here's my particular context:
I'm trying to make a Python program which connects to a MySQL database server and displays/modifies data with a GUI. For cleanliness sake, I've defined the bunch of auxiliary/utility MySQL-related functions in a separate file. However they all have a common variable, which I had originally defined inside the utilities module, and which is the cursor object from MySQLdb module.
I later realised that the cursor object (which is used to communicate with the db server) should be defined in the main module, so that both the main module and anything that is imported into it can access that object.
End result would be something like this:
utilities_module.py:
def utility_1(args):
code which references a variable named "cur"
def utility_n(args):
etcetera
And my main module:
program.py:
import MySQLdb, Tkinter
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
And then, as soon as I try to call any of the utilities functions, it triggers the aforementioned "global name not defined" error.
A particular suggestion was to have a "from program import cur" statement in the utilities file, such as this:
utilities_module.py:
from program import cur
#rest of function definitions
program.py:
import Tkinter, MySQLdb
db=MySQLdb.connect(#blahblah) ; cur=db.cursor() #cur is defined!
from utilities_module import *
But that's cyclic import or something like that and, bottom line, it crashes too. So my question is:
How in hell can I make the "cur" object, defined in the main module, visible to those auxiliary functions which are imported into it?
Thanks for your time and my deepest apologies if the solution has been posted elsewhere. I just can't find the answer myself and I've got no more tricks in my book.
Globals in Python are global to a module, not across all modules. (Many people are confused by this, because in, say, C, a global is the same across all implementation files unless you explicitly make it static.)
There are different ways to solve this, depending on your actual use case.
Before even going down this path, ask yourself whether this really needs to be global. Maybe you really want a class, with f as an instance method, rather than just a free function? Then you could do something like this:
import module1
thingy1 = module1.Thingy(a=3)
thingy1.f()
If you really do want a global, but it's just there to be used by module1, set it in that module.
import module1
module1.a=3
module1.f()
On the other hand, if a is shared by a whole lot of modules, put it somewhere else, and have everyone import it:
import shared_stuff
import module1
shared_stuff.a = 3
module1.f()
… and, in module1.py:
import shared_stuff
def f():
print shared_stuff.a
Don't use a from import unless the variable is intended to be a constant. from shared_stuff import a would create a new a variable initialized to whatever shared_stuff.a referred to at the time of the import, and this new a variable would not be affected by assignments to shared_stuff.a.
Or, in the rare case that you really do need it to be truly global everywhere, like a builtin, add it to the builtin module. The exact details differ between Python 2.x and 3.x. In 3.x, it works like this:
import builtins
import module1
builtins.a = 3
module1.f()
As a workaround, you could consider setting environment variables in the outer layer, like this.
main.py:
import os
os.environ['MYVAL'] = str(myintvariable)
mymodule.py:
import os
myval = None
if 'MYVAL' in os.environ:
myval = os.environ['MYVAL']
As an extra precaution, handle the case when MYVAL is not defined inside the module.
This post is just an observation for Python behaviour I encountered. Maybe the advices you read above don't work for you if you made the same thing I did below.
Namely, I have a module which contains global/shared variables (as suggested above):
#sharedstuff.py
globaltimes_randomnode=[]
globalist_randomnode=[]
Then I had the main module which imports the shared stuff with:
import sharedstuff as shared
and some other modules that actually populated these arrays. These are called by the main module. When exiting these other modules I can clearly see that the arrays are populated. But when reading them back in the main module, they were empty. This was rather strange for me (well, I am new to Python). However, when I change the way I import the sharedstuff.py in the main module to:
from globals import *
it worked (the arrays were populated).
Just sayin'
A function uses the globals of the module it's defined in. Instead of setting a = 3, for example, you should be setting module1.a = 3. So, if you want cur available as a global in utilities_module, set utilities_module.cur.
A better solution: don't use globals. Pass the variables you need into the functions that need it, or create a class to bundle all the data together, and pass it when initializing the instance.
The easiest solution to this particular problem would have been to add another function within the module that would have stored the cursor in a variable global to the module. Then all the other functions could use it as well.
module1:
cursor = None
def setCursor(cur):
global cursor
cursor = cur
def method(some, args):
global cursor
do_stuff(cursor, some, args)
main program:
import module1
cursor = get_a_cursor()
module1.setCursor(cursor)
module1.method()
Since globals are module specific, you can add the following function to all imported modules, and then use it to:
Add singular variables (in dictionary format) as globals for those
Transfer your main module globals to it
.
addglobals = lambda x: globals().update(x)
Then all you need to pass on current globals is:
import module
module.addglobals(globals())
Since I haven't seen it in the answers above, I thought I would add my simple workaround, which is just to add a global_dict argument to the function requiring the calling module's globals, and then pass the dict into the function when calling; e.g:
# external_module
def imported_function(global_dict=None):
print(global_dict["a"])
# calling_module
a = 12
from external_module import imported_function
imported_function(global_dict=globals())
>>> 12
The OOP way of doing this would be to make your module a class instead of a set of unbound methods. Then you could use __init__ or a setter method to set the variables from the caller for use in the module methods.
Update
To test the theory, I created a module and put it on pypi. It all worked perfectly.
pip install superglobals
Short answer
This works fine in Python 2 or 3:
import inspect
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
save as superglobals.py and employ in another module thusly:
from superglobals import *
superglobals()['var'] = value
Extended Answer
You can add some extra functions to make things more attractive.
def superglobals():
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals
def getglobal(key, default=None):
"""
getglobal(key[, default]) -> value
Return the value for key if key is in the global dictionary, else default.
"""
_globals = dict(inspect.getmembers(
inspect.stack()[len(inspect.stack()) - 1][0]))["f_globals"]
return _globals.get(key, default)
def setglobal(key, value):
_globals = superglobals()
_globals[key] = value
def defaultglobal(key, value):
"""
defaultglobal(key, value)
Set the value of global variable `key` if it is not otherwise st
"""
_globals = superglobals()
if key not in _globals:
_globals[key] = value
Then use thusly:
from superglobals import *
setglobal('test', 123)
defaultglobal('test', 456)
assert(getglobal('test') == 123)
Justification
The "python purity league" answers that litter this question are perfectly correct, but in some environments (such as IDAPython) which is basically single threaded with a large globally instantiated API, it just doesn't matter as much.
It's still bad form and a bad practice to encourage, but sometimes it's just easier. Especially when the code you are writing isn't going to have a very long life.
Here's my code:
import numpy as np
import matplotlib.pyplot as plt
import astropy
import matplotlib
%matplotlib ipympl
import scatterplot_with_hist as sc
badx=[]
bady=[]
import badcomp as bc
#things like data5 and list2 are defined in here--I know that code is functional so I'll omit it for brevity
bc.getlist(start = 2000, end = 2200)
The module code is as follows:
def getlist(start, end):
for f in range(1):
for i in range(1238):
for n in range(int(start),int(end)):
if ((data[n]['col1'] - list2[i]) == 0):
badx.append(data[n]['col2'])
bady.append(data[n]['col3'])
If I run this code in the regular space (instead of importing it and running it as a function) it works fine. When I run it as an imported function, it won't recognize variables like data5, list2, and badx and bady.
Why?
Each Python module has it's own global namespace. That means that code in different modules that each try to access global variables will see separate ones. You can access another module's global variables by importing the module and interacting with the attributes of the module object.
In your code, the getlist function in the badcomp module is trying to interact with several global variables, including badx and bady for output, and data and list2 for input. It's not working because you've defined those in the interactive session, which uses the namespace of a module with the special name __main__.
While you could import __main__ from badcomp and interact with the global variables defined there via the module's attributes, that would be a really bad design, since it won't work if the module gets imported in any other way (e.g. by a different module you write later). Instead, the function should probably use variables defined in its own global namespace. The __main__ module is already importing badcomp (as bc), and can access things like badx and bady as bc.badx and bc.bady if the definitions are moved into the module.
Or you might reconsider if global variables are the best way for this function to work. It's often much better to use arguments and return values to pass data in and out of a function, rather than global variables. Maybe badx and bady should be defined within getlist and returned at the end. Meanwhile, data and list2 could be added as arguments to the function.
When a module is imported, it does NOT have access to the global or local namespace of the module that called it. You can get around this by creating a function that creates a variable in the global namespace inside the imported module and run the function from the calling module with each of the variables you need.
Example code (really bad design, but it'll teach you hopefully):
Put THIS in the imported module:
def putVarsInNamespace(variable, variableNameToInject)
exec("global %s" % variableName)
exec("%s = variable" % variableName)
Put THIS in the calling module:
test = 5
from <MODULENAME> import putVarsInNamespace
putVarsInNamespace(test, "test")
How this works: variableNameToInject is the name that you want the injected variable to be called. It then runs global variableNameToInject but it uses the VALUE of variableNameToInject which is the name that the injected variable should be called. This is useful when you want to inject multiple variables without using multiple functions. It then sets the variable name (the value of variableNameToInject) to the value of variable, and just like that it's injected.
I have a project in which I want to repeatedly change code in a class and then run other modules to test the changes (verification..). Currently, after each edit I have to reload the code, the testing modules which run it, and then run the test. I want to reduce this cycle to one line, moreover, I will later want to test different classes, so I want to be able to receive the name of the tested class as a parameter - meaning I need dynamic imports.
I wrote a function for clean imports of any module, it seems to work:
def build_module_clean(module_string,attr_strings):
module = import_module(module_string)
module = reload(module)
for f in attr_strings:
globals()[f]=getattr(module,f)
Now, in the name of cleanliness, I want to keep this function in a wrapper module (which will contain the one-liner I want to rebuild and test all the code each time), and run it from the various modules, i.e. among the import statements of my ModelChecker module I would place the line
from wrapper import build_module_clean
build_module_clean('test_class_module',['test_class_name'])
however, when I do this, it seems the test class is added to the globals in the wrapper module, but not in the ModelChecker module (attempting to access globals()['test_class_name'] in ModelChecker gives a key error). I have tried passing globals or globals() as further parameters to build_module_clean, but globals is a function (so the test module is still loaded to the wrapper globals), and passing and then using globals() gives the error
TypeError: 'builtin_function_or_method' object does not support item assignment
So I need some way to edit one module's globals() from another module.
Alternatively, (ideally?) I would like to import the test_class module in the wrapper, in a manner that would make it visible to all the modules that use it (e.g. ModelChecker). How can I do that?
Your function should look like:
def build_module_clean(globals, module_string, attr_strings):
module = import_module(module_string)
module = reload(module)
globals[module_string] = module
for f in attr_strings:
globals[f] = getattr(module, f)
and call it like so:
build_module_clean(globals(), 'test_class_module', ['test_class_name'])
Explanation:
Calling globals() in the function call (build_module_clean(globals()...) grabs the module's __dict__ while still in the correct module and passes that to your function.
The function is then able to (re)assign the names to the newly-loaded module and it's current attributes.
Note that I also (re)assigned the newly-loaded module itself to the globals (you may not want that part).
I'm trying to build a sort of script system in python that will allow small snippets of code to be selected and executed at runtime inside python.
Essentially I want to be able to load a small python file like
for i in Foo: #not in a function.
print i
Where somewhere else in the program I assign what Foo will be. As if Foo served as a function argument to the entire loaded python file instead of a single function
So somewhere else
FooToPass = GetAFoo ()
TempModule = __import__ ("TheSnippit",<Somehow put {'Foo' : FooToPass} in the locals>)
It is considered bad style to have code with side effects at module level. If you want your module to do something, put that code in a function, make Foo a parameter of this function and call it with the desired value.
Python's import mechanism does not allow to preinitialise a module namespace. If you want to do this anyway (which is, in my opinion, confusing and unnecessary), you have to fiddle around with details of the import mechanism. Example implementation (untested):
import imp
import sys
def my_import(module_name, globals):
if module_name in sys.modules:
return sys.modules[module_name]
module = imp.new_module(module_name)
vars(module).update(globals)
f, module.__file__, options = imp.find_module(module_name)
exec f.read() in vars(module)
f.close()
sys.modules[module_name] = module
return module
(Important: See update below.)
I'm trying to write a function, import_something, that will important certain modules. (It doesn't matter which for this question.) The thing is, I would like those modules to be imported at the level from which the function is called. For example:
import_something() # Let's say this imports my_module
my_module.do_stuff() #
Is this possible?
Update:
Sorry, my original phrasing and example were misleading. I'll try to explain my entire problem. What I have is a package, which has inside it some modules and packages. In its __init__.py I want to import all the modules and packages. So somewhere else in the program, I import the entire package, and iterate over the modules/packages it has imported.
(Why? The package is called crunchers, and inside it there are defined all kinds of crunchers, like CruncherThread, CruncherProcess, and in the future perhaps MicroThreadCruncher. I want the crunchers package to automatically have all the crunchers that are placed in it, so later in the program when I use crunchers I know it can tell exactly which crunchers I have defined.)
I know I can solve this if I avoid using functions at all, and do all imports on the main level with for loops and such. But it's ugly and I want to see if I can avoid it.
If anything more is unclear, please ask in comments.
Functions have the ability to return something to where they were called. Its called their return value :p
def import_something():
# decide what to import
# ...
mod = __import__( something )
return mod
my_module = import_something()
my_module.do_stuff()
good style, no hassle.
About your update, I think adding something like this to you __init__.py does what you want:
import os
# make a list of all .py files in the same dir that dont start with _
__all__ = installed = [ name for (name,ext) in ( os.path.splitext(fn) for fn in os.listdir(os.path.dirname(__file__))) if ext=='.py' and not name.startswith('_') ]
for name in installed:
# import them all
__import__( name, globals(), locals())
somewhere else:
import crunchers
crunchers.installed # all names
crunchers.cruncherA # actual module object, but you can't use it since you don't know the name when you write the code
# turns out the be pretty much the same as the first solution :p
mycruncher = getattr(crunchers, crunchers.installed[0])
You can monkey with the parent frame in CPython to install the modules into the locals for that frame (and only that frame). The downsides are that a) this is really quite hackish and b) sys._getframe() is not guaranteed to exist in other python implementations.
def importer():
f = sys._getframe(1) # Get the parent frame
f.f_locals["some_name"] = __import__(module_name, f.f_globals, f.f_locals)
You still have to install the module into f_locals, since import won't actually do that for you - you just supply the parent frame locals and globals for the proper context.
Then in your calling function you can have:
def foo():
importer() # Magically makes 'some_name' available to the calling function
some_name.some_func()
Are you looking for something like this?
def my_import(*names):
for name in names:
sys._getframe(1).f_locals[name] = __import__(name)
then you can call it like this:
my_import("os", "re")
or
namelist = ["os", "re"]
my_import(*namelist)
According to __import__'s help:
__import__(name, globals={}, locals={}, fromlist=[], level=-1) -> module
Import a module. The globals are only used to determine the context;
they are not modified. ...
So you can simply get the globals of your parent frame and use that for the __import__ call.
def import_something(s):
return __import__(s, sys._getframe(1).f_globals)
Note: Pre-2.6, __import__'s signature differed in that it simply had optional parameters instead of using kwargs. Since globals is the second argument in both cases, the way it's called above works fine. Just something to be aware of if you decided to use any of the other arguments.