I have a function that I define inside another function.
def foo():
x = something
def bar(list=[]):
list.append(x)
return x
return bar
I have two questions:
As I return bar from foo, how does bar keep access to x? At this point, x is only known inside of foo, but when we exit foo, how will bar know what x is?
bar has a mutable default argument. When the returned function bar goes out of scope, is this argument, list, deleted from memory? It will grow and grow every time I call bar, and I want to make sure it is deleted when no longer used.
bar keeps access to x by creating a closure:
>>> def foo():
... x = 42
... def bar(list=[]):
... list.append(x)
... return x
... return bar
...
>>> func = foo()
>>> func
<function foo.<locals>.bar at 0x7fcae19ac050>
>>> func.__closure__
(<cell at 0x7fcae1b19910: int object at 0x1080d99a0>,)
As for the default argument, you ask "When the returned function bar goes out of scope, is this argument, list, deleted from memory?".
Functions are objects. Objects don't go out of scope, scope is a property of variables. If your function object is no longer referenced, then like any other object, it is available for garbage collection. In CPython, this is immediately after a reference count reaches zero. The default values of function arguments are just stored as an attribute, like in any other object:
>>> func.__defaults__
([],)
>>>
So yes, all of this will be cleaned up like normal. If the function object is the only object that references it, then when the function object ceases to exist, the list's reference count reaches zero, and then it becomes available for garbage collection.
You can show this to yourself by defining a verbose finalizer and using that as a default value:
>>> class Bar:
... def __del__(self):
... print('goodbye from ', self)
...
>>> def foo():
... def bar(x=Bar()):
... x.baz = 42
... return bar
...
>>> func = foo()
>>> func = foo()
goodbye from <__main__.Bar object at 0x7fcae1b19950>
>>> func = foo()
goodbye from <__main__.Bar object at 0x7fcae1b19910>
Related
My understanding of mutability and immutability in Python is, say we have a variable foo, if there exists a way to change how foo looks like (by using print) without changing its id, then foo is mutable. Otherwise, it's immutable.
For example, you can do this for a list,
foo = [1, 2, 3]
print(foo, id(foo))
foo[0] = 100
print(foo, id(foo))
but no way for int.
But what about function? First of all, is my definitions of mutability and immutability given above correct? If yes, can you find a way to mutate function without changing its id in order to prove it's mutable?
You can explicitly change the code of a function without affecting its id (here is code using python 2.7):
>>> def f():
... print "f"
...
>>> def g():
... print "g"
...
>>> id(f)
140305904690672
>>> f()
f
>>> f.func_code = g.func_code
>>> id(f)
140305904690672
>>> f()
g
# python3
def foo(a):
class A:
def say(self):
print(a)
return A
A = foo(1)
'__closure__' in dir(A.say) # True
a = A()
a.say.__closure__ # it returns the closure tuple
'__closure__' in dir(a.say) # False
'__closure__' in dir(a.say.__class__) # False
'__closure__' in dir(a.say.__class__.__class__) # False
In Python3, A.say is a function, and I know it has__closure__ attribute.
__closure__ not in dir(a.say) or its super class, but a.say.__closure__ returns the closure tuple. It makes me confuse.
Thanks.
I don't know in Python the internal implementation of objects with type instancemethod but I think it is how __getattr__ works in instance method objects.
My guess is when you say a.say.__closure__ it first looks up for __closure__ in dir(a.say) and then fallbacks on dir(a.say.im_func).
>>> a = foo(1)()
>>> print type(a.say)
>>> instancemethod
>>> a.say.im_func.__closure__
>>> (<cell at 0x10a00f980: int object at 0x7fef29e098b8>,)
>>> '__closure__' in dir(a.say.im_func)
>>> True
Say I have the following dict:
In [6]: scope
Out[6]: {'bar': <function bar>, 'foo': <function foo>}
And foo and bar are:
def foo():
return 5
def bar(x):
return foo() + x
I want to run bar(1), but it will need to find foo(). Is there any way to run bar() in the scope namespace so it finds foo()?
I don't know precisely which function in scope bar will need, so I need a general method of running bar in the scope namespace. I don't have the source code for and cannot modify either function to accept a dict.
It seems like functions have a __closure__ attribute, but it's immutable. There is also a __globals__ attribute, but that just points to globals(). I've seen some answers on SO that update locals() but I'd like to leave locals() untouched.
I tried an eval in scope, but get a NameError for foo:
In [12]: eval(scope['bar'](1), scope)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-12-f6305634f1da> in <module>()
----> 1 eval(scope['bar'](1), scope)
<string> in bar(x)
NameError: global name 'foo' is not defined
Is there some simple way I'm missing?
One approach is to create new functions with a shared globals dictionary:
from types import FunctionType
def scopify(**kwargs):
scoped = dict()
for name, value in kwargs.items():
if isinstance(value, FunctionType):
scoped[name] = FunctionType(value.__code__, scoped, name)
else:
scoped[name] = value
return scoped
scope = scopify(
foo = lambda: baz,
bar = lambda x: foo() + x,
baz = 5,
)
>>> scope['foo']
5
>>> scope['bar'](10)
15
>>> scope['baz'] = 100
>>> scope['bar'](10)
110
Using eval() will work, however there are two problems with the way you tried to use it. For starters, its first argument, expression, should be a string. Secondly, since you're referencing scope in this expression and passing it to eval() as the globals namespace argument, you'll need to add it to itself.
This illustrates what I mean:
def foo():
return 5
def bar(x):
return foo() + x
scope = {'scope': {'bar': bar, 'foo':foo}} # define self-referential scope
expression = "scope['bar'](42)" # explicit reference to `scope`
print('eval({!r}, scope) returns {}'.format(expression, eval(expression, scope)))
However, in this specific case, it would be simpler to just let eval() directly look-up the value of bar in the scope dictionary namespace itself (instead of via scope['scope']['bar']):
scope = {'bar': bar, 'foo':foo}
expression = 'bar(42)'
print('eval({!r}, scope) returns {}'.format(expression, eval(expression, scope)))
Is it possible to modify values for a dictionary inside a function without passing the dictionary as a parameter.
I would like not to return a dictionary, but only to modify its values.
That's possible, but not necessarily advisable, i can't imagine why you wouldn't like to pass or return the dictionary, if you merely don't want to return the dictionary, but could pass it, you can modify it to reflect in the original dictionary without having to return it, for example:
dict = {'1':'one','2':'two'}
def foo(d):
d['1'] = 'ONE'
print dict['1'] # prints 'one' original value
foo(dict)
print dict['1'] # prints 'ONE' ie, modification reflects in original value
# so no need to return it
However, if you absolutely cannot pass it for whatever reason, you can use a global dictionary as follows:
global dict # declare dictionary as global
dict = {'1':'one','2':'two'} # give initial value to dict
def foo():
global dict # bind dict to the one in global scope
dict['1'] = 'ONE'
print dict['1'] # prints 'one'
foo(dict)
print dict['1'] # prints 'ONE'
I'd recommend the first method demonstrated in the first code block, but feel free to use the second if absolutely necessary.
Enjoy :)
Yes you can, dictionary is an mutable object so they can be modified within functions, but it must be defined before you actually call the function.
To change the value of global variable pointing to an immutable object you must use the global statement.
>>> def func():
... dic['a']+=1
...
>>> dic = {'a':1} #dict defined before function call
>>> func()
>>> dic
{'a': 2}
For immutable objects:
>>> foo = 1
>>> def func():
... global foo
... foo += 3 #now the global variable foo actually points to a new value 4
...
>>> func()
>>> foo
4
Can someone explain why Python does the following?
>>> class Foo(object):
... bar = []
...
>>> a = Foo()
>>> b = Foo()
>>> a.bar.append(1)
>>> b.bar
[1]
>>> a.bar = 1
>>> a.bar
1
>>> b.bar
[1]
>>> a.bar = []
>>> a.bar
[]
>>> b.bar
[1]
>>> del a.bar
>>> a.bar
[1]
It's rather confusing!
This is because the way you have written it, bar is a class variable rather than an instance variable.
To define an instance variable, bind it in the constructor:
class Foo(object):
def __init__(self):
self.bar = []
Note that it now belongs to a single instance of Foo (self) rather than the Foo class, and you will see the results you expect when you assign to it.
As others have said the code as written creates a class variable rather than an instance variable. You need to assign in __init__ to create an instance variable.
Hopefully this annotated copy of your code is helpful in explaining what's going on at each stage:
>>> class Foo(object):
... bar = [] # defines a class variable on Foo (shared by all instances)
...
>>> a = Foo()
>>> b = Foo()
>>> a.bar.append(1) # appends the value 1 to the previously empty list Foo.bar
>>> b.bar # returns the value of the class variable Foo.bar
[1]
>>> a.bar = 1 # binds 1 to the instance variable a.bar, masking the access
>>> a.bar # you previously had to the class variable through a.bar
1
>>> b.bar # b doesn't have an instance variable 'bar' so this still
[1] # returns the class variable
>>> a.bar = [] # bind a's instance variable to to an empty list
>>> a.bar
[]
>>> b.bar # b doesn't have an instance variable 'bar' so this still
[1] # returns the class variable
>>> del a.bar # unbinds a's instance variable unmasking the class variable
>>> a.bar # so a.bar now returns the list with 1 in it.
[1]
Also, printing out the value of Foo.bar (the class variable accessed via the class rather than via an instance) after each of your statements might help clarify what is going on.
When you declare an element in the class like that it is shared by all instances of the class. To make a proper class member that belongs to each instance, separately, create it in __init__ like the following:
class Foo(object):
def __init__(self):
self.bar = []
In the beginning, bar is a class variable and it is shared between a and b, both a.bar and b.bar refer to the same object.
When you assign a new value to a.bar, this does not overwrite the class variable, it adds a new instance variable to the a object, hiding the class variable when you access a.bar. If you delete a.bar (the instance variable), then a.bar resolves again to the class variable.
b.bar on the other hand always refers to the class variable, it's not influenced by the additional bar on the a object or any values assigned to that.
To set the class variable you can access it through the class itself:
Foo.bar = 1
>>> class Foo(object):
... bar = []
...
bar is a shared class variable, not an instance variable. I believe that deals with most of your confusion. To make it a instance var, define it in class's __init__ per the other answers.
>>> a = Foo()
>>> b = Foo()
>>> a.bar.append(1)
>>> b.bar
[1]
This is the proof of that.
>>> a.bar = 1
>>> a.bar
1
>>> b.bar
[1]
Now you've redefined a.bar as a instance variable. That's what happens when you define variables externally by default.
>>> a.bar = []
>>> a.bar
[]
>>> b.bar
[1]
>>> del a.bar
>>> a.bar
[1]
Same again. b.bar is still the shared class variable.
On a related note, you should be aware of this pitfall that you might see sometime soon:
class A:
def __init__(self, mylist = []):
self.mylist = mylist
a = A()
a2 = A()
a.mylist.append(3)
print b.mylist #prints [3] ???
This confuses a lot of folks and has to do with how the code is interpreted. Python actually interprets the function headings first, so it evaluates __init__(self, mylist = []) and stores a reference to that list as the default parameter. That means that all instances of A will (unless provided their own list) reference the original list. The correct code for doing such a thing would be
class A:
def __init__(self, mylist=None):
if mylist:
self.mylist = mylist
else:
self.mylist = []
or if you want a shorter expression you can use the ternary syntax:
self.mylist = mylist if mylist else []