I have two code which really confused me.
def get_context():
__gc = globals()
__lc = locals()
def precompiler(code):
exec code in __lc
def compiler(script, scope):
return compile(script, scope, 'eval')
def executor(expr):
return eval(expr, __gc, __lc)
return precompiler, compiler, executor
maker1, compiler1, executor1 = get_context()
maker2, compiler2, executor2 = get_context()
maker1("abc = 123")
maker2("abc = 345")
expr1 = compiler1("abc == 123", "test.py")
print "executor1(abc == 123):", executor1(expr1)
print "executor2(abc == 123):", executor2(expr1)
the result is:
executor1(abc == 123): True
executor2(abc == 123): False
Why the compile execute in the closure only once and the byte-code could run in both?
And there is another code here:
def get_context():
__gc = globals()
__lc = locals()
test_var = 123
def compiler(script, scope):
return compile(script, scope, 'eval')
def executor(expr):
return eval(expr, __gc, __lc)
return compiler, executor
compiler1, executor1 = get_context()
compiler2, executor2 = get_context()
expr1 = compiler1("test_var == 123", "test.py")
print "executor1(test_var == 123):", executor1(expr1)
print "executor2(test_var == 123):", executor2(expr1)
the result is:
NameError: name 'test_var' is not defined
And how did this happen?
Why does the compile need to check the environment(variable or some others) of the closure while it is not dependent on the closure? This is what I confused!
In your first example, you are executing 'abc=123' in your first context, and 'abc=345' in your second context. So 'test_var==123' is true in your first context and false in your second context.
In your second example, you have caught an interesting situation where the interpreter has removed test_var from the context because test_var isn't referenced.
For your first question, compile just takes the python code and produces the bytecode. It it is not dependent in any way on the closure where you compiled it. Its not different then if you had produced say, a string. That string isn't permantely tied to the function where it was created and neither is the code object.
For your second question, locals() builds a dictionary of the local variables when it is called. Since you setup test_var after calling locals it doesn't have it. If you want test_var inside locals, you need to call it afterwards.
Related
Take into account the following code:
def main():
print('Calling methodA()')
methodA()
print('Calling methodB(True)')
methodB(True)
print('Calling methodB(False)')
try:
methodB(False)
except UnboundLocalError as error:
print(f'--> "UnboundLocalError" raised: {error}')
def methodA():
print('Running methodA()')
print('"method_original" in globals(): ' + str('method_original' in globals()))
method_original()
def methodB(patch_function):
print(f'Running methodB({patch_function})')
print('"method_original" in globals(): ' + str('method_original' in globals()))
if patch_function:
method_original=method_patched
method_original()
def method_original():
print('Running method_original()')
def method_patched():
print('Running method_patched()')
if __name__ == '__main__':
main()
It produces the following output:
Calling methodA()
Running methodA()
"method_original" in globals(): True
Running method_original()
Calling methodB(True)
Running methodB(True)
"method_original" in globals(): True
Running method_patched()
Calling methodA(False)
Running methodB(False)
"method_original" in globals(): True
--> "UnboundLocalError" raised: local variable 'method_original' referenced before assignment
Which makes no sense because "method_original" is in globals(). This error can be fixed simply adding global method_original at the beginning of the methodB() but in some cases we have a lot of functions and it could be a pain in the ass to put all of them at the beginning of every method.
Are there any rules to avoid this behavior?
//BR!
Let me explain it in a simpler example :
def fn(a):
if a % 2 == 0:
x = a
return x
print(fn(10)) # Fine
print(fn(9)) # UnboundLocalError: local variable 'x' referenced before assignment
In compile time, when interpreter reaches the function, it sees that there is an assignment to x, so it marks x as a "local" variable. Then in "runtime" interpreter tries to find it only in local namespace ! On the other hand, x is only defined, if a is even.
It doesn't matter if it presents in global namespace, now I want to add a global variable named x, to my example:
def fn(a):
if a % 2 == 0:
x = a
return x
x = 50
print(fn(10)) # Fine
print(fn(9)) # UnboundLocalError: local variable 'x' referenced before assignment
Nothing changed. Interpreter still tries to find x inside the function in local namespace.
Same thing happened in your example.
This is to show which variables are "local":
def fn(a):
if a % 2 == 0:
x = a
return x
print(fn.__code__.co_varnames)
co_varnames is a tuple containing the names of the local variables
(starting with the argument names)
Solution:
Either use global (which I see you don't like) , or do not do assignment inside the function, for example change your methodB to :
def methodB(patch_function):
print(f'Running methodB({patch_function})')
print('"method_original" in globals(): ' + str('method_original' in globals()))
if patch_function:
method_patched()
else:
method_original()
I'm doing code generation and I end up with a string of source that looks like this:
Source
import sys
import operator
def add(a,b):
return operator.add(a,b)
def mul(a,b):
return operator.mul(a,b)
def saveDiv(a,b):
if b==0:
return 0
else:
return a/b
def subtract(a,b):
return operator.sub(a,b)
def main(y,x,z):
y = int(y)
print y
x = int(x)
print x
z = int(z)
print z
ind = lambda y,x,z: mul(saveDiv(x, add(z, z)), 1)
return ind(y,x,z)
print main(**sys.argv)""
Execution
When I'm executing code using exec() and then piping it through stdoutIO()
Working
args={'x':"1",'y':"1",'z':"1"}
source = getSource()
sys.argv = args
with stdoutIO() as s:
exec source
s.getvalue
Not Working
class Coder():
def start(self):
args={'x':"1",'y':"1",'z':"1"}
source = getSource()
sys.argv = args
with stdoutIO() as s:
exec source
return s.getvalue
print "out:", Coder().start()
And the stdoutIO() is implemented like this:
class Proxy(object):
def __init__(self,stdout,stringio):
self._stdout = stdout
self._stringio = stringio
def __getattr__(self,name):
if name in ('_stdout','_stringio','write'):
object.__getattribute__(self,name)
else:
return getattr(self._stringio,name)
def write(self,data):
self._stdout.write(data)
self._stringio.write(data)
#contextlib.contextmanager
def stdoutIO(stdout=None):
old = sys.stdout
if stdout is None:
stdout = StringIO.StringIO()
sys.stdout = Proxy(sys.stdout,stdout)
yield sys.stdout
sys.stdout = old
Problem
If I execute the execution code outside of the class everything works however when I run it inside a class it breaks with this error. How can I fix it or avoid this problem?
File "<string>", line 29, in <module>
File "<string>", line 27, in main
File "<string>", line 26, in <lambda>
NameError: global name 'add' is not defined
Thanks
When you run exec expression, it executes the code contained in expression in the current scope (see here). Apparently inside a class, the function in your expression are dropping out of scope before main is run. I honestly have no idea why (it seems to me like it should work) but maybe someone can add a complete explanation in a comment.
Anyway, if you specifically provide a scope for the expression to be evaluated in, (which is good practice anyway so that you don't pollute your namespace), it works fine inside the class.
So, replace the line:
exec source
with
exec source in {}
and you should be right!
Here we provide an empty dictionary as a the globals() and locals() dctionaries during the evaluation of your expression. You can keep this dictionary if you want, or let it be garbage collected immediately as I have demonstrated in my code. This is all explained in the exec documentation in the link above.
I am writing a small app that has to perform some 'sanity checks' before entering execution. (eg. of a sanity check: test if a certain path is readable / writable / exists)
The code:
import logging
import os
import shutil
import sys
from paths import PATH
logging.basicConfig(level=logging.DEBUG)
log = logging.getLogger('sf.core.sanity')
def sanity_access(path, mode):
ret = os.access(path, mode)
logfunc = log.debug if ret else log.warning
loginfo = (os.access.__name__, path, mode, ret)
logfunc('%s(\'%s\', %s)==%s' % loginfo)
return ret
def sanity_check(bool_func, true_func, false_func):
ret = bool_func()
(logfunc, execfunc) = (log.debug, true_func) if ret else \
(log.warning, false_func)
logfunc('exec: %s', execfunc.__name__)
execfunc()
def sanity_checks():
sanity_check(lambda: sanity_access(PATH['userhome'], os.F_OK), \
lambda: None, sys.exit)
My question is related to the sanity_check function.
This function takes 3 parameters (bool_func, true_func, false_func). If the bool_func (which is the test function, returning a boolean value) fails, true_func gets executed, else the false_func gets executed.
1) lambda: None is a little lame , because for example if the sanity_access returns True, lambda: None gets executed, and the output printed will be:
DEBUG:sf.core.sanity:access('/home/nomemory', 0)==True
DEBUG:sf.core.sanity:exec: <lambda>
So it won't be very clear in the logs what function got executed. The log will only contain <lambda> . Is there a default function that does nothing and can be passed as a parameter ? Is it a way to return the name of the first function that is being executed inside a lambda ?
Or a way not to log that "exec" if 'nothing' is sent as a paramter ?
What's the none / do-nothing equivalent for functions ?
sanity_check(lambda: sanity_access(PATH['userhome'], os.F_OK), \
<do nothing, but show something more useful than <lambda>>, sys.exit)
Additional question, why is lambda: pass instead of lambda: None not working ?
What's with all the lambdas that serve no purpose? Well, maybe optional arguments will help you a bit:
def sanity_check( test, name='undefined', ontrue=None, onfalse=None ):
if test:
log.debug(name)
if ontrue is not None:
ontrue()
else:
log.warn( name )
if onfalse is not None:
onfalse()
def sanity_checks():
sanity_check(sanity_access(PATH['userhome'], os.F_OK), 'test home',
onfalse=sys.exit)
But you are really overcomplicating things.
update
I would normally delete this post because THC4k saw through all the complexity and rewrote your function correctly. However in a different context, the K combinator trick might come in handy, so I'll leave it up.
There is no builtin that does what you want AFIK. I believe that you want the K combinator (the link came up on another question) which can be encoded as
def K_combinator(x, name):
def f():
return x
f.__name__ = name
return f
none_function = K_combinator(None, 'none_function')
print none_function()
of course if this is just a one off then you could just do
def none_function():
return None
But then you don't get to say "K combinator". Another advantage of the 'K_combinator' approach is that you can pass it to functions, for example,
foo(call_back1, K_combinator(None, 'name_for_logging'))
as for your second statement, only expressions are allowed in lambda. pass is a statement. Hence, lambda: pass fails.
You can slightly simplify your call to sanity check by removing the lambda around the first argument.
def sanity_check(b, true_func, false_func):
if b:
logfunc = log.debug
execfunc = true_func
else:
logfunc = log.warning
execfunc = false_func
logfunc('exec: %s', execfunc.__name__)
execfunc()
def sanity_checks():
sanity_check(sanity_access(PATH['userhome'], os.F_OK),
K_combinator(None, 'none_func'), sys.exit)
This is more readable (largely from expanding the ternary operator into an if). the boolfunc wasn't doing anything because sanity_check wasn't adding any arguments to the call. Might as well just call instead of wrapping it in a lambda.
You might want to rethink this.
class SanityCheck( object ):
def __call__( self ):
if self.check():
logger.debug(...)
self.ok()
else:
logger.warning(...)
self.not_ok()
def check( self ):
return True
def ok( self ):
pass
def not_ok( self ):
sys.exit(1)
class PathSanityCheck(SanityCheck):
path = "/path/to/resource"
def check( self ):
return os.access( path, os.F_OK )
class AnotherPathSanityCheck(SanityCheck):
path = "/another/path"
def startup():
checks = ( PathSanityCheck(), AnotherPathSanityCheck() )
for c in checks:
c()
Callable objects can simplify your life.
>>> import dis
>>> f = lambda: None
>>> dis.dis(f)
1 0 LOAD_CONST 0 (None)
3 RETURN_VALUE
>>> g = lambda: Pass
>>>
>>>
>>> dis.dis(g)
1 0 LOAD_GLOBAL 0 (Pass)
3 RETURN_VALUE
>>> g = lambda: pass
File "<stdin>", line 1
g = lambda: pass
^
SyntaxError: invalid syntax
Actually, what you want is a function which does nothing, but has a __name__ which is useful to the log. The lambda function is doing exactly what you want, but execfunc.__name__ is giving "<lambda>". Try one of these:
def nothing_func():
return
def ThisAppearsInTheLog():
return
You can also put your own attributes on functions:
def log_nothing():
return
log_nothing.log_info = "nothing interesting"
Then change execfunc.__name__ to getattr(execfunc,'log_info', '')
Let's say we have a module m:
var = None
def get_var():
return var
def set_var(v):
var = v
This will not work as expected, because set_var() will not store v in the module-wide var. It will create a local variable var instead.
So I need a way of referring the module m from within set_var(), which itself is a member of module m. How should I do this?
def set_var(v):
global var
var = v
The global keyword will allow you to change global variables from within in a function.
As Jeffrey Aylesworth's answer shows, you don't actually need a reference to the local module to achieve the OP's aim. The global keyword can achieve this aim.
However for the sake of answering the OP title, How to refer to the local module in Python?:
import sys
var = None
def set_var(v):
sys.modules[__name__].var = v
def get_var():
return var
As a follow up to Jeffrey's answer, I would like to add that, in Python 3, you can more generally access a variable from the closest enclosing scope:
def set_local_var():
var = None
def set_var(v):
nonlocal var
var = v
return (var, set_var)
# Test:
(my_var, my_set) = set_local_var()
print my_var # None
my_set(3)
print my_var # Should now be 3
(Caveat: I have not tested this, as I don't have Python 3.)
I've been trying to learn Python, and while I'm enthusiastic about using closures in Python, I've been having trouble getting some code to work properly:
def memoize(fn):
def get(key):
return (False,)
def vset(key, value):
global get
oldget = get
def newget(ky):
if key==ky: return (True, value)
return oldget(ky)
get = newget
def mfun(*args):
cache = get(args)
if (cache[0]): return cache[1]
val = apply(fn, args)
vset(args, val)
return val
return mfun
def fib(x):
if x<2: return x
return fib(x-1)+fib(x-2)
def fibm(x):
if x<2: return x
return fibm(x-1)+fibm(x-2)
fibm = memoize(fibm)
Basically, what this is supposed to do is use closures to maintain the memoized state of the function. I realize there are probably many faster, easier to read, and in general more 'Pythonic' ways to implement this; however, my goal is to understand exactly how closures work in Python, and how they differ from Lisp, so I'm not interested in alternative solutions, just why my code doesn't work and what I can do (if anything) to fix it.
The problem I'm running into is when I try to use fibm - Python insists that get isn't defined:
Python 2.6.1 (r261:67515, Feb 1 2009, 11:39:55)
[GCC 4.0.1 (Apple Inc. build 5488)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import memoize
>>> memoize.fibm(35)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "memoize.py", line 14, in mfun
cache = get(args)
NameError: global name 'get' is not defined
>>>
Seeing as I'm new to Python, I don't know if I've done something wrong, or if this is just a limitation of the language. I'm hoping it's the former. :-)
The problem is in your scoping, not in your closures. If you're up for some heavy reading, then you can try http://www.python.org/dev/peps/pep-3104/.
If that's not the case, here's the simple explanation:
The problem is in the statement global get . global refers to the outermost scope, and since there isn't any global function get, it throws.
What you need, is an access specifier for variables in the enclosing scope, and not the global scope.
In python 3.0, as I've tested, the nonlocal keyword is exactly what you need, in the place of global.
nonlocal get
...
In python 2.x, I just removed the global get and the oldget references and it works properly.
def memoize(fn):
get = [lambda key: (False, None)]
def vset(args):
value = fn(*args)
oldget = get[0]
def newget(key):
if args == key:
return (True, value)
return oldget(key)
get[0] = newget
return value
def mfun(*args):
found, value = get[0](args)
if found:
return value
return vset(args)
return mfun
CALLS = 0
def fib(x):
global CALLS
CALLS += 1
if x<2: return x
return fib(x-1)+fib(x-2)
#memoize
def fibm(x):
global CALLS
CALLS += 1
if x<2: return x
return fibm(x-1)+fibm(x-2)
CALLS = 0
print "fib(35) is", fib(35), "and took", CALLS, "calls"
CALLS = 0
print "fibm(35) is", fibm(35), "and took", CALLS, "calls"
Output is:
fib(35) is 9227465 and took 29860703 calls
fibm(35) is 9227465 and took 36 calls
Similar to other answers, however this one works. :)
The important change from the code in the question is assigning to a non-global non-local (get); however, I also made some improvements while trying to maintain your *cough*broken*cough* closure use. Usually the cache is a dict instead of a linked list of closures.
You want to put global get at the beginning of every function (except get itself).
the def get is an assignment to the name get, so you want get to be declared global before that.
Putting global get in mfun and vset makes them work. I can't point to the scoping rules that makes this necessary, but it works ;-)
Your conses are quite lispy too... :)
Get is not global, but local to the surrounding function, that's why the global declaration fails.
If you remove the global, it still fails, because you can't assign to the captured variable name. To work around that, you can use an object as the variable captured by your closures and than just change properties of that object:
class Memo(object):
pass
def memoize(fn):
def defaultget(key):
return (False,)
memo = Memo()
memo.get = defaultget
def vset(key, value):
oldget = memo.get
def newget(ky):
if key==ky: return (True, value)
return oldget(ky)
memo.get = newget
def mfun(*args):
cache = memo.get(args)
if cache[0]: return cache[1]
val = apply(fn, args)
vset(args, val)
return val
return mfun
This way you don't need to assign to the captured variable names but still get what you wanted.
Probably because you want the global get while it isn't a global?
By the way, apply is deprecated, use fn(*args) instead.
def memoize(fn):
def get(key):
return (False,)
def vset(key, value):
def newget(ky):
if key==ky: return (True, value)
return get(ky)
get = newget
def mfun(*args):
cache = get(args)
if (cache[0]): return cache[1]
val = fn(*args)
vset(args, val)
return val
return mfun
def fib(x):
if x<2: return x
return fib(x-1)+fib(x-2)
def fibm(x):
if x<2: return x
return fibm(x-1)+fibm(x-2)
fibm = memoize(fibm)
I think the best way would be:
class Memoized(object):
def __init__(self,func):
self.cache = {}
self.func = func
def __call__(self,*args):
if args in self.cache: return cache[args]
else:
self.cache[args] = self.func(*args)
return self.cache[args]