What happens to a view object in python after the original object have been deleted? For example,
a = {'foo': 1 , 'bar': 2, 'baz': 3 }
b = a.keys() # => dict_keys(['bar', 'foo', 'baz'])
At this point, if any changes are made to the dictionary, they are reflected in b. For example,
a['qux'] = 4
print(b) # => dict_keys(['bar', 'qux', 'foo', 'baz'])
However, when the dictionary is deleted, the dynamic variable still contains all the value of the keys from the deleted dictionary.
del a
a # NameError: name 'a' is not defined
print(b) # => dict_keys(['bar', 'qux', 'foo', 'baz'])
Question
In essence, I want to know if I have to always make sure I delete any variables with the keys values even after the dictionary is deleted. Can this will be a potential cause for a memory leak if the dictionary is large?
Any feedback will be appreciated.
Side Note:
And, yes I know I can put the keys in a list:
c = list(a.keys())
but I am using view objects because of their smaller memory footprint in comparison to list.
I would recommend not to use the term variable. Better use name that allows access to an object. For your example a and b are names for the objects dictionary and the keys of a dictionary. In Python 3 dict.keys() gives you a key-view object that reflects the changes in the underlying dictionary. Therefore, the key-view object keeps a reference to the dictionary.
So you don't delete the dictionary but rather the name pointing to it. Only if there are no more names (references to the dictionary) left, will the garbage collector remove the dictionary.
If you program in a structured way with functions that do not work on global objects, memory leaks are rather rare. In practice del is typically used sparingly.
del only decreases reference counter to an object in python's underhood. Dictionary itself will be deallocated from memory (disposed) far later, when no references left active in program.
When you save keys() product in a variable, you increase dict's reference counter, additionaly to dict's variable itself. Variable will not be deallocated while any reference left to it.
I can express your code in very simple, but absolutely same construction:
a = dict()
b = a # reference only
del(b)
# dict is not garbage-collected, as a still holds the reference
a != None # True
Related
class Object:
def __init__(self,dict):
self.dict = dict
a = Object({1:"hello",2:"lol"})
b = Object(a.dict)
b.dict.pop(1) #remove the element with key 1
print(a.dict, b.dict)
>>{2: 'lol'} {2: 'lol'}
for some reason the "a" object's dictionary gets modified too.
I've tried the same thing with a different attribute, like an int variable, and the problem didn't happen. I really don't know what to do :(
you are not copying the dictionary you are just pointing both a and b to the same dict, thats why its changing both.
you can use dict.copy() to create a shallow copy.
b = Object(a.dict.copy())
this will copy the dict but not any nested dictionaries, for that you need a deep copy.
import copy
b = copy.deepcopy(a.dict)
Here is my example code:
#!/usr/bin/env python3
myDict = {'a':1, 'b':2, 'c':3}
myKeys = myDict.keys()
# myDict = {} # toggle this line
myDict['y'] = 4
for myKey in myDict:
print('D ' + myKey)
for myKey in myKeys:
print('K ' + myKey)
If you run the program as shown here (with the line in the middle commented out), you get this output, which is exactly what I expected. The lines with the prefix D (loop over dictionary) have the same values as the lines with the prefix K (loop over keys of dictionary):
D a
D b
D c
D y
K a
K b
K c
K y
Now remove the hash and activate the line that was commented out. When you run the modified program, you get this:
D y
K a
K b
K c
But I expected one of these behaviors:
either:
After myDict = {} was executed, myKeys has become empty too (since it is a view object that always views the keys of its parent dictionary). Adding the item with the key 'y' should result in this output:
D y
K y
or:
After myDict = {} was executed, a new version of myDict was created and the previous version of myDict was destroyed, so myKeys has no longer any parent dictionary and therefor is pointing to null. So in this case looping over myKeys should throw an error.
But to me is looks as if the old version of myDict has become some kind of zombie, and myKeys is displaying the keys of this zombie.
So, here are my questions:
Is it true, that in my program myKeys shows keys of a zombie dictionary if the line in the middle is activated?
Is it reliable behavior, that myKeys will always show the keys of the previous version of the dictionary? (That would be very helpful for a program that I'm writing just now.)
Is there a way to revive the zombie dictionary? (I have all keys. Can I get the values too?)
The fundamental issue in your understanding here is that:
myDict = {}
Does nothing to the original dict, assignment never mutates, so the object that myDict was referring to is unmodified. Python objects will stay alive as long as something refers to them. CPython uses reference counting, and as an implementation detail, will reclaim objects immediately when their reference count reaches zero.
However, the dict_keys view object you created internally references the dictionary it is acting as a view over, so the original dictionary still has at least one reference to it, since the view object is alive. Note, though, the dict_keys API does not expose this reference, so if it is the only reference, you cannot really access your dict anymore (in any sane way).
Is it reliable behavior, that myKeys will always show the keys of the previous version of the dictionary?
You are misunderstanding the behavior. It is showing the keys of the same dictionary it has always been showing. There is no previous dictionary. The view is over the object, it doesn't care or know which variables happen to be referring to that object at any given time.
It's the same reason that if you do:
x = 'foo'
container = []
container.append(x)
x = 'bar'
print(container[0])
will still print "foo". Objects have no idea what names are referencing them at any given time, and should not care. And when one reference changes, the other references don't magically get updated.
As per this question on Stack Overflow, assignments in Python are always by value, hence you cannot change the original source. For example (from the same question),
locs = [ [1], [2] ]
for loc in locs:
loc = []
print locs
# prints => [ [1], [2] ]
However, dictionary view objects show an opposite behaviour
bike = {"Manufacturer":"Honda","Model":"CBR","cc":250,"price":70,"mileage":74}
keys = bike.keys()
print(keys)
bike["tyres"] = 2
print(keys)
This is the output :
dict_keys(['mileage', 'price', 'Model', 'Manufacturer', 'cc'])
dict_keys(['cc', 'Manufacturer', 'tyres', 'mileage', 'Model', 'price'])
Can they be treated as an exception to assignment by value nature? If yes, what are other such exceptions in Python3?
No, this is not an exception. Nothing is assigned to the dictionary view. Views are explicitly documented as being dynamic:
They provide a dynamic view on the dictionary’s entries, which means that when the dictionary changes, the view reflects these changes.
That's because they only store a reference to the original dictionary and provide you with direct access to just the keys, or just the values, or to (key, value) pairs in ways that differ from the dictionary API.
You can build your own views onto objects if you wanted, but note that such an object still needs a reference to the original object(s):
from collections.abc import Sequence
class ListReversedView(Sequence):
def __init__(self, lst):
self._lst = lst
def __getitem__(self, idx):
if idx < 0:
new = (-idx) - 1
else:
new = len(self) - idx - 1
if new < 0:
raise IndexError(new)
return self._lst[new]
def __len__(self):
return len(self._lst)
def __repr__(self):
return f"[{', '.join(map(repr, self))}]"
The above example gives you a different view on list contents; changes to the list are reflected in the 'view' this object provides. Nothing special needs to be done to Python's assignment model; assignments are still just references to objects, and the _lst attribute in this view object is no exception:
>>> foo = ['spam', 'ham', 'eggs']
>>> view = ListReversedView(foo)
>>> view
['eggs', 'ham', 'spam']
>>> foo[-1] = 'bacon'
>>> view
['bacon', 'ham', 'spam']
Circling back to your list loop; you can still assign back to the the list object itself; rebinding names may not work, but rebinding the indices works just fine:
for index in range(len(locs)):
locs[index] = []
In summary, Python objects all live on a heap, and names and attributes are just referencing to those objects. Multiple references can exist, and each such a reference will see changes made to the object (if permitted). Assignment changes only what a specific reference points to. Dict views are no exception here, they just continue to reference the dictionary from which they were created.
You may want to read up on the Python model; I strongly recommend the Facts and myths about Python names and values article by Ned Batchelder.
This question already has answers here:
How to copy a dictionary and only edit the copy
(23 answers)
Closed 8 years ago.
I'm expecting my frustration to be overridden with some enlightenment - here's a minimal version of the script to demonstrate the problem:
First I create a dictionary:
dic = {
'foo':{},
'bar':{}
}
Then we instantiate a template dictionary that can be iteratively appended
to keys of dic:
appendic= {
'is':'', # '' is a terminal value to be replaced later
}
So here we append appendic to each of the keys in dic:
dic['foo'] = appendic
dic['bar'] = appendic
Now we replace the terminal values, '', with something meaningful:
dic['foo']['is'] = 'foo'
dic['bar']['is'] = 'bar'
At this point, my intuition tells me that if we call:
print(dic['foo']['is']) we get 'foo'
But instead Python returns 'bar' ... to my un-trained mind that is counter-intuitive.
Questions:
How can I tell Python to keep the keys of dic independent?
Why is this the default behaviour? What use cases does this have?
When you assign a appendic to two different keys, Python doesn't make a copy. It assigns a reference instead.
As a result, both dic['please_make_me_Foo'] and dic['dont_make_him_Bar'] refer to the same object. These are not separate dictionaries, they are both the same object, the one appendic also references to.
If you expected these to be separate dictionaries, create a copy of appendic instead. The dict.copy() method creates a shallow copy of a dictionary:
dic['please_make_me_Foo']= appendic.copy()
dic['dont_make_him_Bar'] = appendic.copy()
Shallow means that a new dictionary is created and all references to keys and values contained are copied over.
If appendic itself contains values that are also dictionaries, these would not be copied. The new copy and appendic would both refer to the same values. In most cases, that's not a problem because most primitive values (strings, integers, etc.) are immutable, and you never notice references are shared as you replace such values with new ones.
You make a dict:
appendic= {
'Python_made_me':''
}
Add it to your other dict twice
dic['please_make_me_Foo']= appendic
dic['dont_make_him_Bar'] = appendic
And set the single dict's Python_made_me value twice
dic['please_make_me_Foo']['Python_made_me'] = 'Foo'
dic['dont_make_him_Bar']['Python_made_me'] = 'Bar'
But because they're the same dict, the second line overwrites the first
If you need to copy it, you need to use the copy method:
dic['please_make_me_Foo']= appendic.copy()
dic['dont_make_him_Bar'] = appendic.copy()
ok, I'm just going to write this as a complement to the other answers. When you manipulate a dictionary, you manipulate the reference to an instance, which is the root cause of your mistake. Using hex(id(foo)) you get the memory address of foo, so let's show the address of d instance in the following example to make that tangible:
>>> hex(id(d))
'0x10bd95e60'
>>> hex(id(e[1]))
'0x10bd95e60'
>>> hex(id(f[1]))
'0x10bd95e60'
so if you add or remove values from e[1], you're actually changing the same instance as the one pointed by d, and as a dictionary is mutable, i.e. you can change values within.
Now you're wondering why that does not happen when you're handling integers? Because, in fact it does, it's just that integers are not mutable:
>>> i = 1
>>> hex(id(i))
'0x10ba51e90'
>>> j = i
>>> hex(id(j))
'0x10ba51e90'
>>> i = 2
>>> hex(id(i))
'0x10ba51eb0'
i.e. i is pointing to another place in the memory.
It's possible to create a mutable integer though, by using a class:
>>> class Integer:
... def __init__(self, i):
... self.i = i
...
>>> i = Integer(2)
>>> hex(id(i))
'0x10bd9b410'
>>> j = i
>>> hex(id(j))
'0x10bd9b410'
>>> j.i = 2
>>> i.i
2
>>> hex(id(i))
'0x10bd9b410'
In order to create a new instance of the same dictionary, you need to use the copy() member of a dict:
>>> hex(id(d))
'0x10bd95e60'
>>> w = d.copy()
>>> x = d.copy()
>>> y = d.copy()
>>> hex(id(w))
'0x10bd96128'
>>> hex(id(x))
'0x10bd95f80'
>>> hex(id(y))
'0x10bd96098'
dic['please_make_me_Foo']= appendic
dic['dont_make_him_Bar'] = appendic
appendic is an object - you are assigning a reference to the same object to both keys in dic. So when you change one, you change both.
Try this instead:
dic['please_make_me_Foo']= appendic.copy()
dic['dont_make_him_Bar'] = appendic.copy()
My code will be more clear I think-
someList = list()
foo = {'a':'b'}
someList.append(foo)
print someList
>>> [{'a':'b'}]
defaultbazz = {'a':2, 'b':'t', 'c':'gg'}
for k, v in defaultbazz.iteritems():
foo[k] = v
print someList
>>> [{'a': 2, 'c': 'gg', 'b': 't'}]
Shouldn't the last print be [{'a':'b'}]? I didn't updated the someList, I want it as is..
It's seems to me uninterpreted behavior..
But if that's how python works, how can I find workaround? Even setting a new dict updates the original one dict.. I mean:
someList = list()
foo = {'a':'b'}
someList.append(foo)
print someList
>>> [{'a':'b'}]
bar = foo
defaultbazz = {'a':2, 'b':'t', 'c':'gg'}
for k, v in defaultbazz.iteritems():
bar[k] = v
print someList
>>> [{'a': 2, 'c': 'gg', 'b': 't'}]
I'll be thankful if someone can maybe explain me why it's happen..
It looks like you are expecting your dict to be copied when you add it to a list or assign it to a new variable, but that is not how Python operates. If you assign a dict -- actually, if you assign any object -- you are not creating a new object, but instead you are simply giving your object a new name. (An object can have multiple names.)
So, when you edit your object under the new name, the single instance of that object changes, and that change is visible when you access the object through any name.
If you want to copy your object, then you can do this:
bar = dict(foo)
or
bar = foo.copy()
To simplify:
a = {2: 3}
b = [a]
b contains a "reference" to a (and is a dict which is mutable) - so if a is modified then accessing a via the list b, will display the modified a.
You have to explicitly create a copy of a, which can be done in this case as:
b = [dict(a)]
But you should look at the copy module for copy.copy() and copy.deepcopy()
Dictionaries are mutable objects, hence the result of your script.
I guess you want a new object, i.e. a copy of the original one:
import copy
someList.append(copy.copy(foo))
Variables in Python are just names of objects. If you change the object from any name "attached" to it, you will see the changes from every other name. Python never creates copies automatically for you, in particular:
someList.append(foo)
doesn't create a copy of foo and put it on someList, it appends the object that the name foo refers to onto the list.
You can create a second name for this object
bar = foo
but this does not create a copy either. In particular
foo['x'] = 42
and
bar['x'] = 42
will then operate on exactly the same object. You can verify this by printing the memory address of the object:
print id(foo), id(bar)
and see that they are the same.
If you need a copy in Python, you'll need to create one explicitly. Depending on what you need, the copy module -- either copy.copy() or copy.deepcopy() -- will do what you want:
import copy
bar = copy.copy(foo)
print id(foo), id(bar)
should now print different memory locations.
Dicts are mutable, which means that they can change. It's because foo is inside of someList and you're changing foo in the for-loop. Take a look at this simple example:
a_dict = {'a':'b'}
a_list = [a_dict]
print a_list # [{'a':'b'}]
#change the dict
a_dict['a'] = 'c'
print a_list # [{'a':'c'}]