This question already has answers here:
Short description of the scoping rules?
(9 answers)
Closed 6 months ago.
This message is a a bit long with many examples, but I hope it
will help me and others to better grasp the full story of variables
and attribute lookup in Python 2.7.
I am using the terms of PEP 227
(http://www.python.org/dev/peps/pep-0227/) for code blocks (such as
modules, class definition, function definitions, etc.) and
variable bindings (such as assignments, argument declarations, class
and function declaration, for loops, etc.)
I am using the terms variables for names that can be called without a
dot, and attributes for names that need to be qualified with an object
name (such as obj.x for the attribute x of object obj).
There are three scopes in Python for all code blocks, but the functions:
Local
Global
Builtin
There are four blocks in Python for the functions only (according to
PEP 227):
Local
Enclosing functions
Global
Builtin
The rule for a variable to bind it to and find it in a block is
quite simple:
any binding of a variable to an object in a block makes this variable
local to this block, unless the variable is declared global (in that
case the variable belongs to the global scope)
a reference to a variable is looked up using the rule LGB (local,
global, builtin) for all blocks, but the functions
a reference to a variable is looked up using the rule LEGB (local,
enclosing, global, builtin) for the functions only.
Let me know take examples validating this rule, and showing many
special cases. For each example, I will give my understanding. Please
correct me if I am wrong. For the last example, I don't understand the
outcome.
example 1:
x = "x in module"
class A():
print "A: " + x #x in module
x = "x in class A"
print locals()
class B():
print "B: " + x #x in module
x = "x in class B"
print locals()
def f(self):
print "f: " + x #x in module
self.x = "self.x in f"
print x, self.x
print locals()
>>>A.B().f()
A: x in module
{'x': 'x in class A', '__module__': '__main__'}
B: x in module
{'x': 'x in class B', '__module__': '__main__'}
f: x in module
x in module self.x in f
{'self': <__main__.B instance at 0x00000000026FC9C8>}
There is no nested scope for the classes (rule LGB) and a function in
a class cannot access the attributes of the class without using a
qualified name (self.x in this example). This is well described in
PEP227.
example 2:
z = "z in module"
def f():
z = "z in f()"
class C():
z = "z in C"
def g(self):
print z
print C.z
C().g()
f()
>>>
z in f()
z in C
Here variables in functions are looked up using the LEGB rule, but if
a class is in the path, the class arguments are skipped. Here again,
this is what PEP 227 is explaining.
example 3:
var = 0
def func():
print var
var = 1
>>> func()
Traceback (most recent call last):
File "<pyshell#102>", line 1, in <module>
func()
File "C:/Users/aa/Desktop/test2.py", line 25, in func
print var
UnboundLocalError: local variable 'var' referenced before assignment
We expect with a dynamic language such as python that everything is
resolved dynamically. But this is not the case for functions. Local
variables are determined at compile time. PEP 227 and
http://docs.python.org/2.7/reference/executionmodel.html describe this
behavior this way
"If a name binding operation occurs anywhere within a code block, all
uses of the name within the block are treated as references to the
current block."
example 4:
x = "x in module"
class A():
print "A: " + x
x = "x in A"
print "A: " + x
print locals()
del x
print locals()
print "A: " + x
>>>
A: x in module
A: x in A
{'x': 'x in A', '__module__': '__main__'}
{'__module__': '__main__'}
A: x in module
But we see here that this statement in PEP227 "If a name binding
operation occurs anywhere within a code block, all uses of the name
within the block are treated as references to the current block." is
wrong when the code block is a class. Moreover, for classes, it seems
that local name binding is not made at compile time, but during
execution using the class namespace. In that respect,
PEP227 and the execution model in the Python doc is misleading and for
some parts wrong.
example 5:
x = 'x in module'
def f2():
x = 'x in f2'
def myfunc():
x = 'x in myfunc'
class MyClass(object):
x = x
print x
return MyClass
myfunc()
f2()
>>>
x in module
my understanding of this code is the following. The instruction x = x
first look up the object the right hand x of the expression is referring
to. In that case, the object is looked up locally in the class, then
following the rule LGB it is looked up in the global scope, which is
the string 'x in module'. Then a local attribute x to MyClass is
created in the class dictionary and pointed to the string object.
example 6:
Now here is an example I cannot explain.
It is very close to example 5, I am just changing the local MyClass
attribute from x to y.
x = 'x in module'
def f2():
x = 'x in f2'
def myfunc():
x = 'x in myfunc'
class MyClass(object):
y = x
print y
return MyClass
myfunc()
f2()
>>>
x in myfunc
Why in that case the x reference in MyClass is looked up in the
innermost function?
In an ideal world, you'd be right and some of the inconsistencies you found would be wrong. However, CPython has optimized some scenarios, specifically function locals. These optimizations, together with how the compiler and evaluation loop interact and historical precedent, lead to the confusion.
Python translates code to bytecodes, and those are then interpreted by a interpreter loop. The 'regular' opcode for accessing a name is LOAD_NAME, which looks up a variable name as you would in a dictionary. LOAD_NAME will first look up a name as a local, and if that fails, looks for a global. LOAD_NAME throws a NameError exception when the name is not found.
For nested scopes, looking up names outside of the current scope is implemented using closures; if a name is not assigned to but is available in a nested (not global) scope, then such values are handled as a closure. This is needed because a parent scope can hold different values for a given name at different times; two calls to a parent function can lead to different closure values. So Python has LOAD_CLOSURE, MAKE_CLOSURE and LOAD_DEREF opcodes for that situation; the first two opcodes are used in loading and creating a closure for a nested scope, and the LOAD_DEREF will load the closed-over value when the nested scope needs it.
Now, LOAD_NAME is relatively slow; it will consult two dictionaries, which means it has to hash the key first and run a few equality tests (if the name wasn't interned). If the name isn't local, then it has to do this again for a global. For functions, that can potentially be called tens of thousands of times, this can get tedious fast. So function locals have special opcodes. Loading a local name is implemented by LOAD_FAST, which looks up local variables by index in a special local names array. This is much faster, but it does require that the compiler first has to see if a name is a local and not global. To still be able to look up global names, another opcode LOAD_GLOBAL is used. The compiler explicitly optimizes for this case to generate the special opcodes. LOAD_FAST will throw an UnboundLocalError exception when there is not yet a value for the name.
Class definition bodies on the other hand, although they are treated much like a function, do not get this optimization step. Class definitions are not meant to be called all that often; most modules create classes once, when imported. Class scopes don't count when nesting either, so the rules are simpler. As a result, class definition bodies do not act like functions when you start mixing scopes up a little.
So, for non-function scopes, LOAD_NAME and LOAD_DEREF are used for locals and globals, and for closures, respectively. For functions, LOAD_FAST, LOAD_GLOBAL and LOAD_DEREF are used instead.
Note that class bodies are executed as soon as Python executes the class line! So in example 1, class B inside class A is executed as soon as class A is executed, which is when you import the module. In example 2, C is not executed until f() is called, not before.
Lets walk through your examples:
You have nested a class A.B in a class A. Class bodies do not form nested scopes, so even though the A.B class body is executed when class A is executed, the compiler will use LOAD_NAME to look up x. A.B().f() is a function (bound to the B() instance as a method), so it uses LOAD_GLOBAL to load x. We'll ignore attribute access here, that's a very well defined name pattern.
Here f().C.z is at class scope, so the function f().C().g() will skip the C scope and look at the f() scope instead, using LOAD_DEREF.
Here var was determined to be a local by the compiler because you assign to it within the scope. Functions are optimized, so LOAD_FAST is used to look up the local and an exception is thrown.
Now things get a little weird. class A is executed at class scope, so LOAD_NAME is being used. A.x was deleted from the locals dictionary for the scope, so the second access to x results in the global x being found instead; LOAD_NAME looked for a local first and didn't find it there, falling back to the global lookup.
Yes, this appears inconsistent with the documentation. Python-the-language and CPython-the implementation are clashing a little here. You are, however, pushing the boundaries of what is possible and practical in a dynamic language; checking if x should have been a local in LOAD_NAME would be possible but takes precious execution time for a corner case that most developers will never run into.
Now you are confusing the compiler. You used x = x in the class scope, and thus you are setting a local from a name outside of the scope. The compiler finds x is a local here (you assign to it), so it never considers that it could also be a scoped name. The compiler uses LOAD_NAME for all references to x in this scope, because this is not an optimized function body.
When executing the class definition, x = x first requires you to look up x, so it uses LOAD_NAME to do so. No x is defined, LOAD_NAME doesn't find a local, so the global x is found. The resulting value is stored as a local, which happens to be named x as well. print x uses LOAD_NAME again, and now finds the new local x value.
Here you did not confuse the compiler. You are creating a local y, x is not local, so the compiler recognizes it as a scoped name from parent function f2().myfunc(). x is looked up with LOAD_DEREF from the closure, and stored in y.
You could see the confusion between 5 and 6 as a bug, albeit one that is not worth fixing in my opinion. It was certainly filed as such, see issue 532860 in the Python bug tracker, it has been there for over 10 years now.
The compiler could check for a scoped name x even when x is also a local, for that first assignment in example 5. Or LOAD_NAME could check if the name is meant to be a local, really, and throw an UnboundLocalError if no local was found, at the expense of more performance. Had this been in a function scope, LOAD_FAST would have been used for example 5, and an UnboundLocalError would be thrown immediately.
However, as the referenced bug shows, for historical reasons the behaviour is retained. There probably is code out there today that'll break were this bug fixed.
In two words, the difference between example 5 and example 6 is that in example 5 the variable x is also assigned to in the same scope, while not in example 6. This triggers a difference that can be understood by historical reasons.
This raises UnboundLocalError:
x = "foo"
def f():
print x
x = 5
f()
instead of printing "foo". It makes a bit of sense, even if it seems strange at first: the function f() defines the variable x locally, even if it is after the print, and so any reference to x in the same function must be to that local variable. At least it makes sense in that it avoids strange surprizes if you have by mistake reused the name of a global variable locally, and are trying to use both the global and the local variable. This is a good idea because it means that we can statically know, just by looking at a variable, which variable it means. For example, we know that print x refers to the local variable (and thus may raise UnboundLocalError) here:
x = "foo"
def f():
if some_condition:
x = 42
print x
f()
Now, this rule doesn't work for class-level scopes: there, we want expressions like x = x to work, capturing the global variable x into the class-level scope. This means that class-level scopes don't follow the basic rule above: we can't know if x in this scope refers to some outer variable or to the locally-defined x --- for example:
class X:
x = x # we want to read the global x and assign it locally
bar = x # but here we want to read the local x of the previous line
class Y:
if some_condition:
x = 42
print x # may refer to either the local x, or some global x
class Z:
for i in range(2):
print x # prints the global x the 1st time, and 42 the 2nd time
x = 42
So in class scopes, a different rule is used: where it would normally raise UnboundLocalError --- and only in that case --- it instead looks up in the module globals. That's all: it doesn't follow the chain of nested scopes.
Why not? I actually doubt there is a better explanation that "for historical reasons". In more technical terms, it could consider that the variable x is both locally defined in the class scope (because it is assigned to) and should be passed in from the parent scope as a lexically nested variable (because it is read). It would be possible to implement it by using a different bytecode than LOAD_NAME that looks up in the local scope, and falls back to using the nested scope's reference if not found.
EDIT: thanks wilberforce for the reference to http://bugs.python.org/issue532860. We may have a chance to get some discussion reactivated with the proposed new bytecode, if we feel that it should be fixed after all (the bug report considers killing support for x = x but was closed for fear of breaking too much existing code; instead what I'm suggesting here would be to make x = x work in more cases). Or I may be missing another fine point...
EDIT2: it seems that CPython did precisely that in the current 3.4 trunk: http://bugs.python.org/issue17853 ... or not? They introduced the bytecode for a slightly different reason and don't use it systematically...
Long story short, this is a corner case of Python's scoping that is a bit inconsistent, but has to be kept for backwards compatibility (and because it's not that clear what the right answer should be). You can see lots of the original discussion about it on the Python mailing list when PEP 227 was being implemented, and some in the bug for which this behaviour is the fix.
We can work out why there's a difference using the dis module, which lets us look inside code objects to see the bytecode a piece of code has been compiled to. I'm on Python 2.6, so the details of this might be slightly different - but I see the same behaviour, so I think it's probably close enough to 2.7.
The code that initialises each nested MyClass lives in a code object that you can get to via the attributes of the top-level functions. (I'm renaming the functions from example 5 and example 6 to f1 and f2 respectively.)
The code object has a co_consts tuple, which contains the myfunc code object, which in turn has the code that runs when MyClass gets created:
In [20]: f1.func_code.co_consts
Out[20]: (None,
'x in f2',
<code object myfunc at 0x1773e40, file "<ipython-input-3-6d9550a9ea41>", line 4>)
In [21]: myfunc1_code = f1.func_code.co_consts[2]
In [22]: MyClass1_code = myfunc1_code.co_consts[3]
In [23]: myfunc2_code = f2.func_code.co_consts[2]
In [24]: MyClass2_code = myfunc2_code.co_consts[3]
Then you can see the difference between them in bytecode using dis.dis:
In [25]: from dis import dis
In [26]: dis(MyClass1_code)
6 0 LOAD_NAME 0 (__name__)
3 STORE_NAME 1 (__module__)
7 6 LOAD_NAME 2 (x)
9 STORE_NAME 2 (x)
8 12 LOAD_NAME 2 (x)
15 PRINT_ITEM
16 PRINT_NEWLINE
17 LOAD_LOCALS
18 RETURN_VALUE
In [27]: dis(MyClass2_code)
6 0 LOAD_NAME 0 (__name__)
3 STORE_NAME 1 (__module__)
7 6 LOAD_DEREF 0 (x)
9 STORE_NAME 2 (y)
8 12 LOAD_NAME 2 (y)
15 PRINT_ITEM
16 PRINT_NEWLINE
17 LOAD_LOCALS
18 RETURN_VALUE
So the only difference is that in MyClass1, x is loaded using the LOAD_NAME op, while in MyClass2, it's loaded using LOAD_DEREF. LOAD_DEREF looks up a name in an enclosing scope, so it gets 'x in myfunc'. LOAD_NAME doesn't follow nested scopes - since it can't see the x names bound in myfunc or f1, it gets the module-level binding.
Then the question is, why does the code of the two versions of MyClass get compiled to two different opcodes? In f1 the binding is shadowing x in the class scope, while in f2 it's binding a new name. If the MyClass scopes were nested functions instead of classes, the y = x line in f2 would be compiled the same, but the x = x in f1 would be a LOAD_FAST - this is because the compiler would know that x is bound in the function, so it should use the LOAD_FAST to retrieve a local variable. This would fail with an UnboundLocalError when it was called.
In [28]: x = 'x in module'
def f3():
x = 'x in f2'
def myfunc():
x = 'x in myfunc'
def MyFunc():
x = x
print x
return MyFunc()
myfunc()
f3()
---------------------------------------------------------------------------
Traceback (most recent call last)
<ipython-input-29-9f04105d64cc> in <module>()
9 return MyFunc()
10 myfunc()
---> 11 f3()
<ipython-input-29-9f04105d64cc> in f3()
8 print x
9 return MyFunc()
---> 10 myfunc()
11 f3()
<ipython-input-29-9f04105d64cc> in myfunc()
7 x = x
8 print x
----> 9 return MyFunc()
10 myfunc()
11 f3()
<ipython-input-29-9f04105d64cc> in MyFunc()
5 x = 'x in myfunc'
6 def MyFunc():
----> 7 x = x
8 print x
9 return MyFunc()
UnboundLocalError: local variable 'x' referenced before assignment
This fails because the MyFunc function then uses LOAD_FAST:
In [31]: myfunc_code = f3.func_code.co_consts[2]
MyFunc_code = myfunc_code.co_consts[2]
In [33]: dis(MyFunc_code)
7 0 LOAD_FAST 0 (x)
3 STORE_FAST 0 (x)
8 6 LOAD_FAST 0 (x)
9 PRINT_ITEM
10 PRINT_NEWLINE
11 LOAD_CONST 0 (None)
14 RETURN_VALUE
(As an aside, it's not a big surprise that there should be a difference in how scoping interacts with code in the body of classes and code in a function. You can tell this because bindings at the class level aren't available in methods - method scopes aren't nested inside the class scope in the same way as nested functions are. You have to explicitly reach them via the class, or by using self. (which will fall back to the class if there's not also an instance-level binding).)
Related
I expected the following to result in y being a free variable:
def f(x):
return x + y
However, f.__code__.co_freevars is equal to (), f.__code__.co_varnames is equal to ('x',), and f.__code__.co_names is equal to ('y',).
So, what exactly will cause a variable use to populate co_freevars?
It seems that co_freevars is only populated when there is a closure, i.e. an inner function is defined in an outer function.
A module-level function has __closure__ set to None despite accessing a non-local variable:
y = 1
def foo(x):
return x + y
print(foo.__closure__)
print(foo.__code__.co_freevars)
Output:
None
()
As opposed to:
def bar():
y = 1
def baz(x):
return x + y
print(baz.__closure__)
print(baz.__closure__[0].cell_contents)
print(baz.__code__.co_freevars)
bar()
Output:
(<cell at 0x7fa8fbbf0700: int object at 0x7fa8fbc7a930>,)
1
('y',)
This is a supplement to #mportes's answer.
The documentation defines a free-variable thusly:
If a name is bound in a block, it is a local variable of that block, unless declared as nonlocal or global. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not defined there, it is a free variable.
A caveat from the following section:
The global statement has the same scope as a name binding operation in the same block. If the nearest enclosing scope for a free variable contains a global statement, the free variable is treated as a global.
And even further down, an explicit example:
Name resolution of free variables occurs at runtime, not at compile time. This means that the following code will print 42:
i = 10
def f():
print(i)
i = 42
f()
But the formal definition is irrelevant to how the term is used for co_freevars. That is explicitly listed in the documentation for inspect:
+------+-------------+------------------------------------------------------------------------+
| code | co_freevars | tuple of names of free variables (referenced via a function’s closure) |
+------+-------------+------------------------------------------------------------------------+
So code.co_freevars is a very specific subset of all free variables that reference the E portion of LEGB.
This question already has answers here:
Is it possible to modify a variable in python that is in an outer (enclosing), but not global, scope?
(9 answers)
Closed 5 months ago.
I am confused with the global keyword behavior in below code snippet, I was expecting 30, 30, 30 in all 3 prints.
def outer_function():
#global a ###commented intentionally
a = 20
def inner_function():
global a
a = 30
print('a =',a)
inner_function()
print('a =',a)
a = 10
outer_function()
print('a =',a)
#Output:
#30
#20 #Expecting 30 here
#30
All the confusion coming from "global a" after outer function definition. As my understanding at this point of time is " All the reference and assignment to variable become globally reflected on declaration of global keyword on that variable". If I am uncommenting that first global statement I am getting expected output 30,30,30.
Why global declaration inside inner_function and value change does not reflect on 2nd print i:e to outer_function(or outer scope), whereas got reflected in global namespace.
A common acronym to be familiar with is LEGB:
Local
Enclosed
Global
Built-in
This is the order in which Python will search the namespaces to find variable assignments.
Local
The local namespace is everything that happens within the current code block. Function definitions contain local variables that are the first thing that is found when Python looks for a variable reference. Here, Python will look in the local scope of foo first, find x with the assignment of 2 and print that. All of this happens despite x also being defined in the global namespace.
x = 1
def foo():
x = 2
print(x)
foo()
# prints:
2
When Python compiles a function, it decides whether each of the variables within the definition code block are local or global variables. Why is this important? Let's take a look at the same definition of foo, but flip the two lines inside of it. The result can be surprising
x = 1
def foo():
print(x)
x = 2
foo()
# raises:
UnboundLocalError: local variable 'x' referenced before assignment
This error occurs because Python compiles x as a local variable within foo due to the assignment of x = 2.
What you need to remember is that local variables can only access what is inside of their own scope.
Enclosed
When defining a multi-layered function, variables that are not compiled as local will search for their values in the next highest namespace. Here is a simple example.
x = 0
def outer_0():
x = 1
def outer_1():
def inner():
print(x)
inner()
outer_1()
outer_0()
# print:
1
When inner() is compiled, Python sets x as a global variable, meaning it will try to access other assignments of x outside of the local scope. The order in which Python searches for a value of x in moving upward through the enclosing namespaces. x is not contained in the namespace of outer_1, so it checks outer_0, finds a values and uses that assignment for the x within inner.
x --> inner --> outer_1 --> outer_0 [ --> global, not reached in this example]
You can force a variable to not be local using the keywords nonlocal and global (note: nonlocal is only available in Python 3). These are directives to the compiler about the variable scope.
nonlocal
Using the nonlocal keyword tells python to assign the variable to first instance found as it moves upward through the namespaces. Any changes made to the variable will be made in the variable's original namespace as well. In the example below, when 2 is assigned x, it is setting the value of x in the scope of outer_0 as well.
x = 0
def outer_0():
x = 1
def outer_1():
def inner():
nonlocal x
print('inner :', x)
x = 2
inner()
outer_1()
print('outer_0:', x)
outer_0()
# prints:
inner : 1
outer_0: 2
Global
The global namespace is the highest level namespace that you program is running in. It is also the highest enclosing namespace for all function definitions. In general it is not good practice to pass values in and out of variables in the global namespace as unexpected side effects can occur.
global
Using the global keyword is similar to non-local, but instead of moving upward through the namespace layers, it only searches in the global namespace for the variable reference. Using the same example from above, but in this case declaring global x tells Python to use the assignment of x in the global namespace. Here the global namespace has x = 0:
x = 0
def outer_0():
x = 1
def outer_1():
def inner():
global x
print('inner :', x)
inner()
outer_1()
outer_0()
# prints:
0
Similarly, if a variable is not yet defined in the global namespace, it will raise an error.
def foo():
z = 1
def bar():
global z
print(z)
bar()
foo()
# raises:
NameError: name 'z' is not defined
Built-in
Last of all, Python will check for built-in keywords. Native Python functions such as list and int are the final reference Python checks for AFTER checking for variables. You can overload native Python functions (but please don't do this, it is a bad idea).
Here is an example of something you SHOULD NOT DO. In dumb we overload the the native Python list function by assigning it to 0 in the scope of dumb. In the even_dumber, when we try to split the string into a list of letters using list, Python will find the reference to list in the enclosing namespace of dumb and try to use that, raising an error.
def dumb():
list = 0
def even_dumber():
x = list('abc')
print(x)
even_dumber()
dumb()
# raises:
TypeError: 'int' object is not callable
You can get back the original behavior by referencing the global definition of list using:
def dumb():
list = [1]
def even_dumber():
global list
x = list('abc')
print(x)
even_dumber()
dumb()
# returns:
['a', 'b', 'c']
But again, DO NOT DO THIS, it is bad coding practice.
I hope this helps bring to light some of how the namespaces work in Python. If you want more information, chapter 7 of Fluent Python by Luciano Ramalho has a wonderful in-depth walkthrough of namespaces and closures in Python.
From the documentation:
The global statement is a declaration which holds for the entire
current code block. It means that the listed identifiers are to be
interpreted as globals.
Note it only applies to current code block. So the global in inner_function only applies within inner_function. Outside of it, the identifier is not global.
Note how “identifier” is not the same as “variable”. So what it tells the interpreter is “when I use identifier a within this code block, do not apply normal scope resolution, I actually mean the module-level variable, ”.
Just uncomment your global command in the outer_function, otherwise you're declaring a local variable with value 20, changing a global variable then printing that same local variable.
It's not a good idea use global variabilities. If you want only reset the value of a variable, you just use this lines:
def outer_function():
a = 20
def inner_function():
a = 30
print('a =',a)
return a
a = inner_function()
print('a =',a)
return a
a = 10
a = outer_function()
print('a =',a)
def main():
x=2
def cool():
y=4
nonlocal x
print (x)
It is showing an error as --nonlocal x is a invalid syntax--.And if I dont declare it as nonlocal it says undefined x. So, how do i acess a variable which is in some other function ?Now how do i access x variable which is defined at main().
You can't.
You shouldn't. Doing so would just make it hard to read code because code from anywhere could read and modify your variables. By keeping access restricted to the one function it is easier to follow.
You do not. The variables local to a function only exist while that function is running; so once main has returned, its x does not exist. This also ties in to the fact that a separate call to the function gets a separate variable.
What you describe is a bit like reading the value of a static variable in C. The difference with static variables is that they're independent of the call; they still exist, and that makes most functions that use them non-reentrant. Sometimes this is emulated in Python by adding a default argument with a mutable value, with the same downsides.
In CPython, you actually can find out what the local variables of a pure Python function are by inspecting its code object, but their values will only exist in the call itself, normally on a call stack.
def func():
x=2
import dis
print func.__code__.co_varnames
dis.disassemble(func.__code__)
yields:
('x',)
2 0 LOAD_CONST 1 (2)
3 STORE_FAST 0 (x)
6 LOAD_CONST 0 (None)
9 RETURN_VALUE
So x is actually local variable 0.
I would suggest looking up a debugger for details on call stack inspection.
Once you start needing attributes on a function, turn it into a functor:
class MainFactory(object):
def __call__(self):
self.x = 4
main = MainFactory() # Create the function-like object main
main() # call it
print main.x # inspect internal attributes
A functor in python is simply a normal object whose class implements the special __call__ method. It can have an __init__ method as well, in which you may pre-set the values of attributes if you wish. Then you can create different "flavours" of your functor by supplying different arguments when instantiating:
class MainFactory(object):
def __init__(self, parameter=1):
self.parameter = parameter
def __call__(self):
self.x = 4 * self.parameter
Now main = MainFactory(2); main() will have main.x set to 8.
Obviously, you can keep unimportant variables of the function inaccessible by simply using local variables instead of attributes of self:
def __call__(self):
# i and p are not accessible from outside, self.x is
for i in range(10):
p = i ** self.parameter
self.x += p
x = 0
def main():
global x
x = 2 # accesses global x
main()
print x # prints 2
I am learning Python and just had this question. It possibly has no practical worth, I'm asking this out maybe because of a pedantic curiosity.
I have a function:
def f():
x = 12 #this is a local variable
global x #this is a global I declared before
x = 14 #changes global x
<How can I return the local x?>
I'm aware that this is ugly and I do get a Syntax Warning. Just wanted to know how to "get back" my local variable.
Thanks to everyone.
Your code isn't doing what you think it is and there's no way to change it do what you describe. You can't "revert" what globals does since it has no effect at run time.
The global keyword is interpreted at compile time so in the first line of f() where you set x = 12 this is modifying the global x since the compiler has decided the x refers to the x in the global namespace throughout the whole function, not just after the global statement.
You can test this easily:
>>> def g():
... x = 12
... y = 13
... global x
... print "locals:",locals()
...
<stdin>:4: SyntaxWarning: name 'x' is assigned to before global declaration
>>> g()
locals: {'y': 13}
The locals() function gives us a view of the local namespace and we can see that there's no x in there.
The documentation says the following:
Names listed in a global statement must not be used in the same code block textually preceding that global statement.
Names listed in a global statement must not be defined as formal parameters or in a for loop control target, class definition, function definition, or import statement.
The current implementation does not enforce the latter two restrictions, but programs should not abuse this freedom, as future implementations may enforce them or silently change the meaning of the program.
Just give the local variable a different name.
I very much doubt that it can be done.
>>> x = 1
>>> def f():
... x = 12
... global x
... print x
...
<stdin>:3: SyntaxWarning: name 'x' is assigned to before global declaration
>>> f()
12
>>> x
12
As you can see, even without an explicit assignment to x after the global keyword, python syncs up the the values of the two. To see why, you have to disassemble it.
>>> dis.dis(f) # different f with print statement removed for clarity
2 0 LOAD_CONST 1 (12)
3 STORE_GLOBAL 0 (x)
3 6 LOAD_CONST 0 (None)
9 RETURN_VALUE
>>>
The global keyword isn't represented here at all but the assignment statement gets compiled as STORE_GLOBAL. After you do it, it's done. Use globals with care and the global keyword with greater care.
It is not possible to undo the effect of a global statement.
A global statement applies for the entire duration of the function in which it appears, regardless of where in the function it is, and regardless of whether it's executed. So, for example, in the assignment in the following function:
def f():
x = 3
if False:
global x
x is still treated as global.
So it wouldn't make sense to have another statement to undo its effect afterwards, since a global "statement" is not so much a statement, that is executed at some point during the function, as a declaration about the behavior of the function as a whole.
I've been reading a lot about closures and I think I understand them, but without clouding the picture for myself and others, I am hoping someone can explain closures as succinctly and clearly as possible. I'm looking for a simple explanation that might help me understand where and why I would want to use them.
Closure on closures
Objects are data with methods
attached, closures are functions with
data attached.
def make_counter():
i = 0
def counter(): # counter() is a closure
nonlocal i
i += 1
return i
return counter
c1 = make_counter()
c2 = make_counter()
print (c1(), c1(), c2(), c2())
# -> 1 2 1 2
It's simple: A function that references variables from a containing scope, potentially after flow-of-control has left that scope. That last bit is very useful:
>>> def makeConstantAdder(x):
... constant = x
... def adder(y):
... return y + constant
... return adder
...
>>> f = makeConstantAdder(12)
>>> f(3)
15
>>> g = makeConstantAdder(4)
>>> g(3)
7
Note that 12 and 4 have "disappeared" inside f and g, respectively, this feature is what make f and g proper closures.
To be honest, I understand closures perfectly well except I've never been clear about what exactly is the thing which is the "closure" and what's so "closure" about it. I recommend you give up looking for any logic behind the choice of term.
Anyway, here's my explanation:
def foo():
x = 3
def bar():
print x
x = 5
return bar
bar = foo()
bar() # print 5
A key idea here is that the function object returned from foo retains a hook to the local var 'x' even though 'x' has gone out of scope and should be defunct. This hook is to the var itself, not just the value that var had at the time, so when bar is called, it prints 5, not 3.
Also be clear that Python 2.x has limited closure: there's no way I can modify 'x' inside 'bar' because writing 'x = bla' would declare a local 'x' in bar, not assign to 'x' of foo. This is a side-effect of Python's assignment=declaration. To get around this, Python 3.0 introduces the nonlocal keyword:
def foo():
x = 3
def bar():
print x
def ack():
nonlocal x
x = 7
x = 5
return (bar, ack)
bar, ack = foo()
ack() # modify x of the call to foo
bar() # print 7
I like this rough, succinct definition:
A function that can refer to environments that are no longer active.
I'd add
A closure allows you to bind variables into a function without passing them as parameters.
Decorators which accept parameters are a common use for closures. Closures are a common implementation mechanism for that sort of "function factory". I frequently choose to use closures in the Strategy Pattern when the strategy is modified by data at run-time.
In a language that allows anonymous block definition -- e.g., Ruby, C# -- closures can be used to implement (what amount to) novel new control structures. The lack of anonymous blocks is among the limitations of closures in Python.
I've never heard of transactions being used in the same context as explaining what a closure is and there really aren't any transaction semantics here.
It's called a closure because it "closes over" the outside variable (constant)--i.e., it's not just a function but an enclosure of the environment where the function was created.
In the following example, calling the closure g after changing x will also change the value of x within g, since g closes over x:
x = 0
def f():
def g():
return x * 2
return g
closure = f()
print(closure()) # 0
x = 2
print(closure()) # 4
# A Closure is a function object that remembers values in enclosing scopes even if they are not present in memory.
# Defining a closure
# This is an outer function.
def outer_function(message):
# This is an inner nested function.
def inner_function():
print(message)
return inner_function
# Now lets call the outer function and return value bound to name 'temp'
temp = outer_function("Hello")
# On calling temp, 'message' will be still be remembered although we had finished executing outer_function()
temp()
# Technique by which some data('message') that remembers values in enclosing scopes
# even if they are not present in memory is called closures
# Output: Hello
Criteria to met by Closures are:
We must have nested function.
Nested function must refer to the value defined in the enclosing function.
Enclosing function must return the nested function.
# Example 2
def make_multiplier_of(n): # Outer function
def multiplier(x): # Inner nested function
return x * n
return multiplier
# Multiplier of 3
times3 = make_multiplier_of(3)
# Multiplier of 5
times5 = make_multiplier_of(5)
print(times5(3)) # 15
print(times3(2)) # 6
Here's a typical use case for closures - callbacks for GUI elements (this would be an alternative to subclassing the button class). For example, you can construct a function that will be called in response to a button press, and "close" over the relevant variables in the parent scope that are necessary for processing the click. This way you can wire up pretty complicated interfaces from the same initialization function, building all the dependencies into the closure.
In Python, a closure is an instance of a function that has variables bound to it immutably.
In fact, the data model explains this in its description of functions' __closure__ attribute:
None or a tuple of cells that contain bindings for the function’s free variables. Read-only
To demonstrate this:
def enclosure(foo):
def closure(bar):
print(foo, bar)
return closure
closure_instance = enclosure('foo')
Clearly, we know that we now have a function pointed at from the variable name closure_instance. Ostensibly, if we call it with an object, bar, it should print the string, 'foo' and whatever the string representation of bar is.
In fact, the string 'foo' is bound to the instance of the function, and we can directly read it here, by accessing the cell_contents attribute of the first (and only) cell in the tuple of the __closure__ attribute:
>>> closure_instance.__closure__[0].cell_contents
'foo'
As an aside, cell objects are described in the C API documentation:
"Cell" objects are used to implement variables referenced by multiple
scopes
And we can demonstrate our closure's usage, noting that 'foo' is stuck in the function and doesn't change:
>>> closure_instance('bar')
foo bar
>>> closure_instance('baz')
foo baz
>>> closure_instance('quux')
foo quux
And nothing can change it:
>>> closure_instance.__closure__ = None
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: readonly attribute
Partial Functions
The example given uses the closure as a partial function, but if this is our only goal, the same goal can be accomplished with functools.partial
>>> from __future__ import print_function # use this if you're in Python 2.
>>> partial_function = functools.partial(print, 'foo')
>>> partial_function('bar')
foo bar
>>> partial_function('baz')
foo baz
>>> partial_function('quux')
foo quux
There are more complicated closures as well that would not fit the partial function example, and I'll demonstrate them further as time allows.
Here is an example of Python3 closures
def closure(x):
def counter():
nonlocal x
x += 1
return x
return counter;
counter1 = closure(100);
counter2 = closure(200);
print("i from closure 1 " + str(counter1()))
print("i from closure 1 " + str(counter1()))
print("i from closure 2 " + str(counter2()))
print("i from closure 1 " + str(counter1()))
print("i from closure 1 " + str(counter1()))
print("i from closure 1 " + str(counter1()))
print("i from closure 2 " + str(counter2()))
# result
i from closure 1 101
i from closure 1 102
i from closure 2 201
i from closure 1 103
i from closure 1 104
i from closure 1 105
i from closure 2 202
we all have used Decorators in python. They are nice examples to show what are closure functions in python.
class Test():
def decorator(func):
def wrapper(*args):
b = args[1] + 5
return func(b)
return wrapper
#decorator
def foo(val):
print val + 2
obj = Test()
obj.foo(5)
here final value is 12
Here, the wrapper function is able to access func object because wrapper is "lexical closure", it can access it's parent attributes.
That is why, it is able to access func object.
I would like to share my example and an explanation about closures. I made a python example, and two figures to demonstrate stack states.
def maker(a, b, n):
margin_top = 2
padding = 4
def message(msg):
print('\n’ * margin_top, a * n,
' ‘ * padding, msg, ' ‘ * padding, b * n)
return message
f = maker('*', '#', 5)
g = maker('', '♥’, 3)
…
f('hello')
g(‘good bye!')
The output of this code would be as follows:
***** hello #####
good bye! ♥♥♥
Here are two figures to show stacks and the closure attached to the function object.
when the function is returned from maker
when the function is called later
When the function is called through a parameter or a nonlocal variable, the code needs local variable bindings such as margin_top, padding as well as a, b, n. In order to ensure the function code to work, the stack frame of the maker function which was gone away long ago should be accessible, which is backed up in the closure we can find along with the 'message's function object.
For me, "closures" are functions which are capable to remember the environment they were created. This functionality, allows you to use variables or methods within the closure wich, in other way,you wouldn't be able to use either because they don't exist anymore or they are out of reach due to scope. Let's look at this code in ruby:
def makefunction (x)
def multiply (a,b)
puts a*b
end
return lambda {|n| multiply(n,x)} # => returning a closure
end
func = makefunction(2) # => we capture the closure
func.call(6) # => Result equal "12"
it works even when both, "multiply" method and "x" variable,not longer exist. All because the closure capability to remember.
The best explanation I ever saw of a closure was to explain the mechanism. It went something like this:
Imagine your program stack as a degenerate tree where each node has only one child and the single leaf node is the context of your currently executing procedure.
Now relax the constraint that each node can have only one child.
If you do this, you can have a construct ('yield') that can return from a procedure without discarding the local context (i.e. it doesn't pop it off the stack when you return). The next time the procedure is invoked, the invocation picks up the old stack (tree) frame and continues executing where it left off.