This simple decorator works as expected:
def protect(*permissions):
def outer(f):
def inner(*args):
print permissions[0]
return f(*args)
return inner
return outer
#protect('protected')
def func(var):
return var
print func('something')
The output is:
protected
something
Moving beyond a Python shell and using the decorator in the larger scope of my project, something strange is happening: while inside of the inner function, permissions is not defined.
I'm thinking there must be some Python variable scoping/decorator subtleties I'm unaware of that could be causing this. Any insights are appreciated.
In my mind I could figure out what is going on - let me try to spell it out:
it has to do with Python not "perceiving" the "permissions" variable as existing on the scopes outside the "inner" function - since when "inner" itself is defined, "permissions" would long have been defined in the 'outsidemost' scope of protect. Thus, when compiling inner the variale is taken as being a global variable. (That is why the exact error message is needed - NameErrors can be for local variables used before definition, or for non-existing global variables - the exact message will tell much in this case)
In other words, you most likley have hit an implementation bug - please try to expose the minimum ammount of code that causes the issue and the exact python version you are using. If possible try with the latest micro version available for you - and ten it will be time to open an issue at bugs.python.org
I see two workarounds -- the first one would be a hack that would confirm my diagnosis - and might not work at all: Make a reading access to the permissions variable on the outer function, outside inner's body: that should make the interpretor "perceive" it as a non-local variable in outer, and propagate it into inner.
The other workaround is more solid and consistent - and maybe even better code-styling for yu in this case: to use a class as the decorator in this case, instead of relying on multiple nested functions and its closures.
The above snippet could be rewritten as:
class Protect(object):
def __init__(self, *permissions):
self.permissions = permissions
def __call__(self, f):
def inner(*args):
print self.permissions[0]
return f(*args)
return inner
#Protect('protected')
def func(var):
return var
print func('something')
This code does not rely on nested closures, thus avoidnign the bug you've hit. Besides, it follows the "flat is better than nested" and "explict is better than implicit" coding guidelines.
But please, help everyone to trace this bug, by giving us your version and code that actually triggers this behavior, if not opening an issue at python.org
I ran into this problem once, it was because you cannot set a variable that you put in a closure. Don't know if it's the case for you, but it's good to know.
Related
I have had PyCharm 2017.3 extract some code inside a top-level function to another top-level function, and it does a good job.
However, sometimes I would like to not put the extracted function on top level, but rather it should become a function nested inside the existing function. The rationale is re-using code that is only used inside a function, but several times there. I think that this "sub-function" should ideally not be accessible outside of the original function.
How can I do this? I have not seen any options in the refactoring dialog.
Example
a) Original code:
def foo():
a = ''
if a == '':
b = 'empty'
else:
b = 'not empty'
return b
b) What extracting does:
def foo():
a = ''
b = bar(a)
return b
def bar(a):
if a == '':
b = 'empty'
else:
b = 'not empty'
return b
c) What I would like to have:
def foo():
def bar():
if a == '':
b = 'empty'
else:
b = 'not empty'
return b
a = ''
b = bar(a)
return b
I am aware that bar's b will shadow foo's b unless it is renamed in the process. I also thought about completely accepting the shadowing by not returning or requesting b and just modifying it inside bar.
Please also hint me if what I want is not a good thing for any reason.
It is considered good practice to keep function boundaries isolated: get data as parameters and spit data as return values with as little side-effects as possible. That said, there are a few special cases where you break this rule; many of them when using closures. Closures are not as idiomatic in Python as they are in Javascript - personally I think it is good but many people disagree.
There is one place were closures are absolutely idiomatic in Python: decorators. For other cases where you would use a closure in order to avoid use of global variables and provide some form of data hiding there are other alternatives in Python. Although some people advocates using closure instead of a class when it has just one method, a plain function combined with functools.partial can be even better.
This is my guess about why there is no such feature in Pycharm: we almost never do it in Python, instead we tend to keep the function signature as foo(x) even when we can get x from the enclosing scope. Hell, in Python our methods receive self explicitly where most languages have an implicit this. If you write code this way then Pycharm already does everything that is needed when refactoring: it fixes the indentation when you cut & paste.
If you catch yourself doing this kind of refactoring a lot I guess you are coming from a language where closures are more idiomatic like Javascript or Lisp.
So my point is: this "nested to global" or "global to nested" function refactoring feature does not exist in Pycharm because nested functions relying on the enclosing scopes are not idiomatic in Python unless for closures - and even closures are not that idiomatic outside of decorators.
If you care enough go ahead and fill a feature request at their issue tracker or upvote some related tickets like #PY-12802 and #PY-2701 - as you can see those have not attracted a lot of attention possibly because of the reasons above.
I would like to write a function which receives a local namespace dictionary and update it. Something like this:
def UpdateLocals(local_dict):
d = {'a':10, 'b':20, 'c':30}
local_dict.update(d)
When I call this function from the interactive python shell it works all right, like this:
a = 1
UpdateLocals(locals())
# prints 20
print a
However, when I call UpdateLocals from inside a function, it doesn't do what I expect:
def TestUpdateLocals():
a = 1
UpdateLocals(locals())
print a
# prints 1
TestUpdateLocals()
How can I make the second case work like the first?
UPDATE:
Aswin's explanation makes sense and is very helpful to me. However I still want a mechanism to update the local variables. Before I figure out a less ugly approach, I'm going to do the following:
def LoadDictionary():
return {'a': 10, 'b': 20, 'c': 30}
def TestUpdateLocals():
a = 1
for name, value in LoadDictionary().iteritems():
exec('%s = value' % name)
Of course the construction of the string statements can be automated, and the details can be hidden from the user.
You have asked a very good question. In fact, the ability to update local variables is very important and crucial in saving and loading datasets for machine learning or in games. However, most developers of Python language have not come to a realization of its importance. They focus too much on conformity and optimization which is nevertheless important too.
Imagine you are developing a game or running a deep neural network (DNN), if all local variables are serializable, saving the entire game or DNN can be simply put into one line as print(locals()), and loading the entire game or DNN can be simply put into one line as locals().update(eval(sys.stdin.read())).
Currently, globals().update(...) takes immediate effect but locals().update(...) does not work because Python documentation says:
The default locals act as described for function locals() below:
modifications to the default locals dictionary should not be
attempted. Pass an explicit locals dictionary if you need to see
effects of the code on locals after function exec() returns.
Why they design Python in such way is because of optimization and conforming the exec statement into a function:
To modify the locals of a function on the fly is not possible without
several consequences: normally, function locals are not stored in a
dictionary, but an array, whose indices are determined at compile time
from the known locales. This collides at least with new locals added
by exec. The old exec statement circumvented this, because the
compiler knew that if an exec without globals/locals args occurred in
a function, that namespace would be "unoptimized", i.e. not using the
locals array. Since exec() is now a normal function, the compiler does
not know what "exec" may be bound to, and therefore can not treat is
specially.
Since global().update(...) works, the following piece of code will work in root namespace (i.e., outside any function) because locals() is the same as globals() in root namespace:
locals().update({'a':3, 'b':4})
print(a, b)
But this will not work inside a function.
However, as hacker-level Python programmers, we can use sys._getframe(1).f_locals instead of locals(). From what I have tested so far, on Python 3, the following piece of code always works:
def f1():
sys._getframe(1).f_locals.update({'a':3, 'b':4})
print(a, b)
f1()
However, sys._getframe(1).f_locals does not work in root namespace.
The locals are not updated here because, in the first case, the variable declared has a global scope. But when declared inside a function, the variable loses scope outside it.
Thus, the original value of the locals() is not changed in the UpdateLocals function.
PS: This might not be related to your question, but using camel case is not a good practice in Python. Try using the other method.
update_locals() instead of UpdateLocals()
Edit To answer the question in your comment:
There is something called a System Stack. The main job of this system stack during the execution of a code is to manage local variables, make sure the control returns to the correct statement after the completion of execution of the called function etc.,
So, everytime a function call is made, a new entry is created in that stack,
which contains the line number (or instruction number) to which the control has to return after the return statement, and a set of fresh local variables.
The local variables when the control is inside the function, will be taken from the stack entry. Thus, the set of locals in both the functions are not the same. The entry in the stack is popped when the control exits from the function. Thus, the changes you made inside the function are erased, unless and until those variables have a global scope.
OK, I know it's going to be obvious, but I cannot work out how to write a test for an internal function. Here's a trivial piece of code to illustrate the problem.
def high(x, y):
def low(x):
return x*2
return y*low(x)
class TestHigh(unittest.TestCase):
def test_high(self):
self.assertEqual(high(1,2),4)
def test_low(self):
self.assertEqual(low(3),6)
results in
Exception: NameError: global name 'low' is not defined
In the "real" case I want to be able to test the lower level function in isolation to make sure all the paths are exercised, which is cumbersome when testing only from the higher level.
low is nested within the high function, so it's not accessible from outside the function. The equivalent for your function would be high(3,1)
You write tests to ensure that the publicly visible interface performs according to its specification. You should not attempt to write tests for internal functionality that is not exposed.
If you cannot fully test low() through the results of high() then the untested parts of low() cannot matter to anything outside.
BAD: Try making a class and adding the functions as methods (or staticfunctions) to it.
(I'll leave this here as a reference for what NOT to do.)
GOOD: Write module level functions or accept that you can't test it if you nest it.
is it possible to add a local varible to a function, just before calling it ? if yes how ?
EDIT:REASON
i noticed that all my views in django are using
render_to_response(template_name,locals())
now i created a middleware and i wanted to add one more local variable using the
def process_view():
method of it .so that i don't have to modify the views .
The local scope for a function does not exist until the function is called, so it's not possible to do this. You could do this for a closure, but the next person to have to maintain the code would hunt you down and kill you.
Although I also think it is pretty useless, I thought that you may enclose the function in either a 'with' statement or another function, like the code below. Of course, this approach can be accomplished directly within the function of interest. In fact, you are adding the local variable 'during' the function declaration. See if this fits your needs!
#!/usr/bin/python
def my_funct(_local):
"""My function of interest
"""
print "Local argument was %s" % str(_local)
return "Finished"
def localize(fct, local_var):
"""
"""
return fct(_local = local_var)
## Use function to 'localize' variable
localize(my_funct, local_var="LOCAL_VARIABLE")
## Same effect without supplementary function :
my_funct(_local="LOCAL_VARIABLE")
try:
print local_var
except:
print "No such global variable"
Just some thoughts :)
Cheers
So if you’re one of those lazy
programmers and you like keeping code
particularly concise, you can take
advantage of a built-in Python
function called locals(). It returns a
dictionary mapping all local variable
names to their values, where “local”
means all variables that have been
defined within the current scope.
source
It is a trick in order to not have to explicitly list all of the variables you need to pass in to the function. In this case, you need to explicitly state a variable to pass in. Therefore, you should not be using locals() in the calls you are making in your middle-ware, as the trick was not designed to be used like that.
i mangaged to do that using decorators.
Does python have an equivalent to Tcl's uplevel command? For those who don't know, the "uplevel" command lets you run code in the context of the caller. Here's how it might look in python:
def foo():
answer = 0
print "answer is", answer # should print 0
bar()
print "answer is", answer # should print 42
def bar():
uplevel("answer = 42")
It's more than just setting variables, however, so I'm not looking for a solution that merely alters a dictionary. I want to be able to execute any code.
In general, what you ask is not possible (with the results you no doubt expect). E.g., imagine the "any code" is x = 23. Will this add a new variable x to your caller's set of local variables, assuming you do find a black-magical way to execute this code "in the caller"? No it won't -- the crucial optimization performed by the Python compiler is to define once and for all, when def executes, the exact set of local variables (all the barenames that get assigned, or otherwise bound, in the function's body), and turn every access and setting to those barenames into very fast indexing into the stackframe. (You could systematically defeat that crucial optimization e.g. by having an exec '' at the start of every possible caller -- and see your system's performance crash through the floor in consequence).
Except for assigning to the caller's local barenames, exec thecode in thelocals, theglobals may do roughly what you want, and the inspect module lets you get the locals and globals of the caller in a semi-reasonable way (in as far as deep black magic -- which would make me go postal on any coworker suggesting it be perpetrated in production code -- can ever be honored with the undeserved praise of calling it "semi-reasonable", that is;-).
But you do specify "I want to be able to execute any code." and the only solution to that unambiguous specification (and thanks for being so precise, as it makes answering easier!) is: then, use a different programming language.
Is the third party library written in Python? If yes, you could rewrite and rebind the function "foo" at runtime with your own implementation. Like so:
import third_party
original_foo = third_party.foo
def my_foo(*args, **kwds):
# do your magic...
original_foo(*args, **kwds)
third_party.foo = my_foo
I guess monkey-patching is slighly better than rewriting frame locals. ;)