There are several threads on Python garbage collection in SO, and after reading about five, plus some doc on line, i am still not sure as to how garbage collection works and how i should manage objects which i am not using. In fact somewhere i read one should not do anything about collecting garbage, others tell one should del objects, while others again explain de-referencing an object is enough for Python to collect it as garbage.
So, at the risk of creating a duplicate, i will ask the question again, but differently, hoping to get more comprehensive and clearer information.
In my case i want to make a small simulation with objects representing people. Several instances of the Person() class will be created. It should exist for some time until it virtually "dies" while other instances will be created.
Now how do i make this Person() instance "die" (assuming many many of these instances will be created and i don't want these instances to hang out like ghosts)?
There are several ways i can reference an object:
john = Person('john')
or
people = []
people.append(Person('john'))
or
people = {}
people['john'] = Person('john')
What is the best way to keep my program clean, freeing resources optimally? And what is the best way then to reference my object so i can control the deletion of the object?
Maybe this also can help:
>>> # Create a simple object with a verbose __del__ to track gc.
>>> class C:
... def __del__(self):
... print "delete object"
...
>>> c = C()
>>> # Delete the object c successfully.
>>> del c
delete object
>>> # Deletion of an object when it go out of the scope where it was defined.
>>> def f():
... c = C()
...
>>> f()
delete object
>>> c = C()
>>> # Create another reference of the object.
>>> b = c
>>> # The object wasn't destructed the call of del only decremented the object reference.
>>> del c
>>> # Now the reference counter of the object reach 0 so the __del__ was called.
>>> del b
delete object
>>> # Create now a list that hold all the objects.
>>> l = [C(), C()]
>>> del l
delete object
delete object
>>> # Create an object that have a cyclic reference.
>>> class C:
... def __init__(self):
... self.x = self
... def __del__(self):
... print "delete object"
...
>>> c = C()
>>> # Run the garbage collector to collect object.
>>> gc.collect()
9
>>> # the gc.garbage contain object that the gc found unreachable and could not be freed.
>>> gc.garbage
[<__main__.C instance at 0x7ff588d84368>]
>>> # Break the cyclic reference.
>>> c.x = None
>>> # And now we can collect this object.
>>> del c
delete object
>>> # Create another object with cyclic reference.
>>> c = C()
>>> # When closing the interactive python interpreter the object will be collected.
delete object
Refrences : del method ; gc module ; weakref module
None of this really has anything to do with garbage collection.
Python's main method of memory management uses reference counting.
In all cases above, Python keeps a count of all the references to the object, and when there are none left, the object is deleted (similar to std::shared_pointer in C++).
References get decreased when
the object holding them is either explicitly deleted (via del)
or goes out of scope (see also here (esp. ex. 8)).
In your case, this applies to either the john object, or either of the people containers. They go out of scope at the end of the function that created them (assuming they are not returned to the calling function). The vast majority of the time, you can just let them go out of scope - it's only when you create really heavy objects or collections - say inside a big loop - that you might want to consider explicitly using del.
Garbage collection really only comes into play when there are reference cycles
- for instance, when an object refers to itself. Like:
a = []
a.append(a)
Again, this happens automatically, and you shouldn't need to do anything special.
I find that most programs create and dispose of objects quite naturally, so I never normally worry about it.
Some examples:
person = Person('john')
person = Person('james')
# Whoops! 'john' has died!
people = []
people.append(Person('john'))
# ...
# All 'Persons' live in people
people = []
# Now all 'Persons' are dead (including the list that referenced them)
class House():
def setOwner(self, person):
self.owner = person
house.setOwner(people[0])
# Now a House refers to a Person
people = []
# Now all 'Persons' are dead, except the one that house.owner refers to.
What I assume you are after is this:
people = {}
people['john'] = Person('john')
def removePerson(personName):
del people[personName]
removePerson('john')
In this case people is the master list and you can control when a Person gets added and removed from the list (its a dictionary).
You may have to think through the concept of a person being created and then dying very thoroughly: Once created how does the person first interact with the simulation. Upon death, how should you untangle the references? (Its ok for a person to refer to other stuff, its things like House in my example that would keep a person alive. You could have other objects hold on to just the name of the person).
Previous answers are correct but here is what is recommended according to python 3.7 Document:
"Python does automatic memory management (reference counting for most objects and garbage collection to eliminate cycles). The memory is freed shortly after the last reference to it has been eliminated."
If you really have to do this because of memory management issues you are experiencing with a particular case then import the gc library and just do this
del self.someInstanceOfyourClass
gc.collect()
here is a simple example https://github.com/nanoseconds/PythonTest/blob/master/test.py
Related
In my program I create an un-ending amount of class instances. the amount depends on how long the program is running. However I don't need the instances at all after a certain code being run. How could i remove them completely from memory?
Simple example code:
class Player:
def __init__(self, color):
self.color = color
for n in range(1000):
p = Player('black')
Would del p in this case completely remove that instance?
Python will remove them from memory for you when they are no longer referred to. If you have Player instances that refer to other Player instances (ex: p.teammates = [list of Players]) you could end up with circular references that may prevent them from being garbage collected. In this case you should consider the weakref module.
for example:
>>>sam = Player('blue')
>>>rob = Player('green')
>>>sam.team = [sam, rob]
>>>rob.team = [sam, rob]
>>> #sam and rob may not be deleted because they contain
>>> #references to eachother so the reference count cannot reach 0
>>>del sam #del is a way to manually dereference an object in an interactive prompt. Otherwise the interpreter cannot know you won't use it again unlike when the entire code is known at the beginning.
>>>print(rob.team[0].color) #this prints 'blue' proving that sam hasn't been deleted yet
blue
so how do we fix it?
>>>sam = Player('blue')
>>>rob = Player('green')
>>>sam.team = [weakref.ref(sam), weakref.ref(rob)]
>>>rob.team = [weakref.ref(sam), weakref.ref(rob)]
>>> #now sam and rob can be deleted, but we've changed the contents of `p.team` a bit:
>>> #if they both still exist:
>>>rob.team[0]() is sam #calling a `ref` object returns the object it refers to if it still exists
True
>>>del sam
>>>rob.team[0]() #calling a `ref` object that has been deleted returns `None`
None
>>>rob.team[0]().color #sam no longer exists so we can't get his color
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'color'
In this instance, del p would only remove the reference to the Player object, so that it can later be picked up by the garbage collector.
However, this also happens when it goes out-of-scope.
In most everyday Python, there is no need to use explicit del statements.
There is no way in Python to delete an instance. Instead you can delete references to the instance, and once they are all gone, the object is reclaimed.
I am trying this approach to delete an object in python. I read the documentation of Python stating that garbage collector will automatically delete the object that is not referenced.
def check():
class newOb():
def __init__(self,value):
self.value = value
print self.value
return None
class ob:
ins = {}
def cr(self,someuniqueid,value) :
newV = newOb(value)
ob.ins[someuniqueid] = newV ## saving this object refernce to the ob class ins dictionary
return newV
#### Accessing Object ###
someuniqueid = 12
c = ob()
d = c.cr(someuniqueid,123)
print d.value ## will print 123
# now deleting the associated object
del c.ins[someuniqueid]
check()
At the last step, I am removing the object reference from the memory
is using above procedure will delete the object from memory
If not then what is wrong with code and how to correct it
You would need to do del d as well, since d is also holding a reference to the same object. Calling del will only decrement the reference count and remove the particular reference from usage, but the actual in memory object is not garbage collected until the reference count hits 0.
I don't know what do you mean by writing:
If not then what is wrong with code and how to correct it
When you use del statement you delete a reference to an object. It will use up memory untill garbage collector is invoked. Remember that this can be a time-consuming process and not necessary if the process has enough memory to continue executing.
Generally speaking Python does not perform C++-like destructor bahaviour.
A quote from "Expert Python Programming":
The approach of such a memory manager is roughly based on a simple
statement: If a given object is not referenced anymore, it is removed.
In other words, all local references in a function are removed after
the interpreter:
• Leaves the function
• Makes sure the object is not being used anymore.
Under normal conditions, the collector will do a
nice job. But a del call can be used to help the garbage collector by
manually removing the references to an object manually.
So you don't manage memory by hand. You can help garbage collector, but it's better to leave memory managment behind the scenes.
I want to remove all objects, which refer to one memory location. How to do it if I do not know their all name?
class Foo(object):
pass
class Bar(object):
pass
a = Foo()
b = a
c = Bar()
c.a_ref = a
c.b_ref = b
for item in a, b, c.a_ref, c.b_ref:
print(id(item))
""" Out:
140035270075472
140035270075472
140035270075472
140035270075472
"""
UPD:
Ok. I want to remove link or port1.link and don't worry about all the other ports(port2.link) linked to it, they also have to disappear.
class LINK(object):
pass
class PORT(object):
def __init__(self, link=None):
self.link = link
def __repr__(self):
return str(id(self.link))
link = LINK()
port1 = PORT()
port2 = PORT()
port1.link = port2.link = link
print(id(link), port1, port2)
# (140586778512720, 140586778512720, 140586778512720)
del link
print(port1, port2)
# (140586778512720, 140586778512720)
# want: (None, None) ???
You can't explicitly free memory in Python. The garbage collector is responsible to do that for you.
What you can do is guarantee that all the references to your objects are cleaned. The garbage collector works with epochs (like e1, e2 and e3) and only the objects in e3 will be cleaned in the next iteration.
Objects only go from epoch eX to eY (where Y > X) if there is no reference to them. So your objects start in e1; if you "clean" all references, in the next iteration of garbage collector they will be moved to e2; in the next one they will be moved to e3 and finally the memory will be released.
You can change the periodicity that garbage collector is called, but I would not recommend that. Or you can force the calling of garbage collector using gc.collect() but as I said, your objects have to go through all epochs before they are cleaned (that's why the calling of gc.collector() didn't work for you). Just to clarify, the garbage collector is called once a threshold (number of objects in memory tracked by the gc) is achieved.
If you want to do a diagnoses of memory leaks in Python you can also use the objgraph library which is really good. It builds graphs with all links and references between objects and let you identify cycles. Cycles are the main reason for the objects not to be released by the garbage collector when you think that all references are cleaned. Here is a good tutorial http://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html
See the weakref module. It maintains a reference to an object without preventing it from being garbage collected if all its strong references are removed. Example:
import weakref
class LINK(object):
pass
class PORT(object):
def __init__(self, link=None):
if link is not None:
self.link = weakref.ref(link)
else:
self.link = None
def __repr__(self):
if self.link is None:
return 'None'
o = self.link()
return 'None' if o is None else str(id(o))
link = LINK()
port1 = PORT()
port2 = PORT()
port1.link = port2.link = weakref.ref(link)
print(id(link), port1, port2)
del link
print(port1, port2)
Output:
70741240 70741240 70741240
None None
Note, you may still have to call gc.collect() in some situations before the weak references report None.
You can't explicitly free memory. What you need to do is to make sure you don't keep references to objects. They will then be garbage collected, freeing the memory.
BTW, you can call gc.collect() to force a gc operation.
UPDATE:
You CAN NOT delete all the references to one object through only deleting one of its references, since you don't know who else is still using this object. And if so, there will be another question, how to prevent others to delete my object without acknowledging me.
I think best solution to your question is to delete attribute link separately.
del link
del port1.link
del port2.link
Then to get None after deleting, you should do this:
...
def __repr__(self):
repr_string = getattr(self, 'link', None)
return str(id(self.link)) if repr_string else None
...
Your real problem, to me, sounds like you have a graph where PORTs are nodes and LINKs are edges. And your requirement is that when you delete an edge, the two nodes that the edge connects should also be deleted. This can be done explicitly by overriding the __del__ method of the node so that deleting it deletes the edges which is connects to.
Python abstracts away the "memory" and using the fact that id gives you the memory location (which is really an implementation detail of CPython) is a flaky way of doing this.
I have two threads in my PyGTK app:
the main thread which runs the GTK loop and does all the GUI stuff
another thread which handles network requests, etc.
I need to have the second thread get some information from the first thread, so I call:
variable = None
gobject.idle_add( function_in_main_thread, variable )
In the main thread I have:
def function_in_main_thread( variable ):
variable = 1
The problem is that the variable in the second thread never gets set. It's value remains at None. So how can I get the main thread to actually modify the variable in the other thread?
Note: I have some thread synchronization code in the script in case anyone is concerned about modifying variables in other threads. I omitted it from this example because I felt it really didn't apply to the real issue.
In Python, assignment always creates the name in the current scope. The name you pass is indeed a reference to the variable you've created, but the assignment operator will try to find it in the current scope and, if it's not there, create it.
>>> def assign_to(name, value):
... name = value
...
>>> name = "nothing"
>>> assign_to(name, 5)
>>> name
'nothing'
You can instead pass a reference to some kind of instance, a type or maybe even a list, and then use some method of that instance.
>>> def append_to_list(my_list, value):
... my_list.append(value)
...
>>> my_list = []
>>> append_to_list(my_list, 5)
>>> my_list
[5]
The list could of course equally be just an empty object with a "value" member.
In short, if you want to do something akin to call by reference, avoid the assignment operator. Use some kind of method of the object instead.
There's a neat, and surprisingly terse guide to Python Namespaces by Shrutarshi Basu, in case you want to dig deeper. In order to understand what the assignment operator is doing, check out wp: Name Binding. It also explains why the assignment operator can't be overloaded in Python.
Also, as you pointed out, you need to call gobject.threads_init()
Supose you have something like:
x = "something"
b = x
l = [b]
How can you delete the object only having one reference, say x?
del x won't do the trick; the object is still reachable from b, for example.
No no no. Python has a garbage collector that has very strong territory issues - it won't mess with you creating objects, you don't mess with it deleting objects.
Simply put, it can't be done, and for a good reason.
If, for instance, your need comes from cases of, say, caching algorithms that keep references, but should not prevent data from being garbage collected once no one is using it, you might want to take a look at weakref.
The only solution I see right now is that you should make sure that you are holding the only reference to x, everyone else must not get x itself but a weak reference pointing to x. Weak references are implemented in the weakref module and you can use it this way:
>>> import weakref
>>> class TestClass(object):
... def bark(self):
... print "woof!"
... def __del__(self):
... print "destructor called"
...
>>> x = TestClass()
>>> b = weakref.proxy(x)
>>> b
<weakproxy at 0x7fa44dbddd08; to TestClass at 0x7fa44f9093d0>
>>> b.bark()
woof!
>>> del x
destructor called
>>> b.bark()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ReferenceError: weakly-referenced object no longer exists
However, note that not all classes can be weak-referenced. In particular, most built-in types cannot. Some built-in types can be weak-referenced if you subclass them (like dict), but others cannot (like int).
You don't. That's the entire point. Imagine if l is in a library outside of your control. It has every right to expect that the collection elements don't dissappear.
Also, imagine if it was otherwise. You'd have questions here on SO "How do I prevent others from deleting my objects?". As a language designer, you can't satisfy both demands.