Is there a way to remove all references to an object at once? I know that's unpythonic, so I'll explain what I'm trying to do and maybe someone knows a better way.
I'm writing an object-oriented wrapper around a SWIG wrapper for a C library. When a proxy for one of the C objects is deleted, it also deletes child objects (directly in C). I'd like that to also trigger deletion of their proxy objects in Python. Otherwise I run into a situation where there are Python objects carrying around invalid pointers which will segfault if they're accessed.
It looks sort of like this:
class Parent(object):
def __init__(self):
self.ptr = swig.createParent()
def __del__(self):
swig.deleteParent(self.ptr) # also deletes children
class Child(object):
def __init__(self, parent):
self.ptr = swig.createChild(parent)
def __del__(self):
swig.deleteChild(self.ptr)
And this is the situation I'm worried about:
p = Parent()
c = Child(parent)
del p
# accessing c.ptr now would be bad right?
If I understand you correctly, you are wrapping some C code, and the C code has a destructor that can be called. After that, any attempt to use the pointer to the C code object causes a fatal crash.
I am not sure of your exact situation, so I am going to give you two alternate answers.
0) If the C object can be freed for some reason out of your control, and you need to make sure your Python wrapper code doesn't crash, you need to make the Python wrapper know whether the C object is available or not. Make your Python object handle the pointer no longer being valid. You could raise a Python exception, return an error code, or just have the method functions become no-op functions, depending on what you are doing. The C object going away doesn't free the Python object, so you can handle this cleanly.
1) If the C object is only freed when the Python object is freed, you don't have a problem. Python references, when they go out of scope or you call del() on them, do not free the Python object; they just decrement the reference count on that object. When the reference count goes to zero, then the object is freed and your __del__() method function can call into the C code to free the C object.
You can watch how it works by running this code:
class DelTest(object):
def __init__(self):
print "__init__() called: object %08x created" % id(self)
def __del__(self):
print "__del__() called: object %08x destroyed" % id(self)
print "begin"
print "creating object, binding to name d"
d = DelTest()
print "adding reference bound to name x"
x = d
print "adding reference bound to lst[0]"
lst = []
lst.append(d)
print "deleting lst"
del(lst)
print "deleting x"
del(x)
print "deleting d"
del(d)
print "end"
Output from the above:
begin
creating object, binding to name d
__init__() called: object 01e4db50 created
adding reference bound to name x
adding reference bound to lst[0]
deleting lst
deleting x
deleting d
__del__() called: object 01e4db50 destroyed
end
A note about the behavior of __del__() method.
del x doesn’t directly call x.__del__() — the former decrements the reference count for x by one, and the latter is only called when x‘s reference count reaches zero.
Therefore even if you delete parent, it does not necessarily mean that __del__ is executed immediately until there are any references to it. Here is an example.
>>> class C(object):
... def __del__(self):
... print "deleting object of type: %s" %self.__class__
...
>>> class D(object):
... def __init__(self, parent):
... self.parent = parent
... def __del__(self):
... print "deleting object of type: %s" % self.__class__
...
>>> c = C()
>>> d = D(c)
>>> del c
>>> del d
deleting object of type: <class '__main__.D'>
deleting object of type: <class '__main__.C'>
Note that the __del__ method of C is called after the del d call.
Related
I was writing a class and at some point I decided that it would be nice to have a possibility to create an instance of the class using other instance of this same class. So basically the class would look like this:
class Test:
def __init__(self, a, b):
if type(b) == Test:
self = a / b
else:
self.a = a
self.b = b
def __rtruediv__(self, other):
return Test(other * self.b, self.a)
def __str__(self):
return f"{self.a = }, {self.b = }"
if __name__ == '__main__':
hf = Test(1, 2)
print(hf) # self.a = 1, self.b = 2
print(Test(1, hf)) # AttributeError: 'Test' object has no attribute 'a'
print(1 / Test(1, hf)) # AttributeError: 'Test' object has no attribute 'b'
However, when i tried to do it, I got the AttributeError (AttributeError: 'Test' object has no attribute 'a'). Interestingly enough, the code print(1/Test(1, hf)) gives the same attribute error about attribute b while going into the rtruediv func, so the object Test(1, hf) has methods that I defined. Why does that happen? Is something wrong with "self = ..."?
The reason this doesn't work is that self is just an ordinary variable like any other.
if type(b) == Test:
self = a / b
else:
self.a = a
self.b = b
In the if branch, we reassign a local variable, but it doesn't change the instance being constructed. It just makes a new instance, that will be discarded in a moment. The actual instance being constructed (the original value of self) is still there and remains in an uninitialized state (i.e. doesn't have a and b like it should).
I think your goal here is well-intentioned, though it will likely confuse Python programmers as it's not incredibly idiomatic. My recommendation is to simply set self.a and self.b in the constructor in all cases, as it's a more intuitive code design for the average Python coder.
However, what you want can be done. In Python, when you call Test(a, b), some complicated internal things happen.
First, we invoke __call__ on the metaclass of Test. In your example, the metaclass of Test isn't specified, so it defaults to [type], the built-in metaclass.
The default behavior of type.__call__ is to invoke the class method __new__ on your class (__new__ is implicitly a class method, even if you don't ask it to be, so you don't need the #classmethod decorator). Your class doesn't currently define __new__, so we inherit the default one from object.
object.__new__ actually creates the new object. This is written in low-level C and can't be replicated in Python. It truly is primitive.
Finally, object.__new__ calls Test.__init__ on the newly-constructed object and then returns the new object.
Aside from (3), which is truly a C-level primitive, we can easily inject into any part of this. We could write a metaclass and redefine __call__, but that's overkill for what we're doing. We're going to hook into (2) and define __new__. So rather than your __init__, consider
class Test:
def __new__(cls, a, b):
if type(b) == Test:
return a / b
else:
obj = object.__new__(cls)
obj.a = a
obj.b = b
return obj
In __init__, we're given the object to construct, and our only option is to construct that object (or throw an exception if we can't). But __new__ is lower-level. In __new__, it's our choice whether and how to create a new object. In the case where we're given two numbers, we delegate to the default __new__ on object (creating a new object, using the primitive C-level code discussed above), and then we initialize its instance variables. But in the case where b is a Test, we do something different and short-circuit the process entirely, calling a different method instead.
Again, I don't necessarily think this is super idiomatic Python code, but it is possible to do what you want.
As far I know in python 'self' represents the object of a class. Recently I found a code where in the constructor(__init__) a variable value is assigned to 'self' like below:
self.x = self
Can anyone please explain what kind of value is actually assigned to x?
It creates a circular reference. self is bound to the instance on which the method is called, so setting self.x = self just creates a reference to the instance on the instance.
This is a generally silly thing to do, and potentially harmful to the memory performance of your program. If the class also defines the object.__del__() method then this will prevent the object from being garbage collected, causing a memory leak in all CPython releases < 3.4 (which implements PEP 442):
>>> import gc
>>> class SelfReference(object):
... def __init__(self):
... self.x = self
... def __del__(self):
... pass
...
>>> s = SelfReference()
>>> s.x is s # the instance references itself
True
>>> del s # deleting the only reference should clear it from memory
>>> gc.collect()
25
>>> gc.garbage # yet that instance is *still here*
[<__main__.SelfReference object at 0x102d0b890>]
The gc.garbage list contains everything the garbage collector cannot clean up due to circular references and __del__ methods.
I suspect that you found one of the very few actual usecases for assigning self to a an attribute anyway, which is the usecase davidb mentions: setting self.__dict__ to self if self is a mapping object, to 'merge' attribute and subscription access into one namespace.
Even if this kind of assignments can generally seem not a good idea, yet there are cases where it is indeed useful and elegant.
Here is one of those cases:
class Dict(dict):
'''Dictionary subclass allowing to access an item using its key as an
attribute.
'''
def __init__(self, *args, **kwargs):
super(Dict, self).__init__(*args, **kwargs)
self.__dict__ = self
Here is a simple usage example:
>>> d = Dict({'one':1, 'two':2})
>>> d['one']
1
>>> d.one
1
Python 2.7 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
methods (both bound and unbound), ...
And Python 3.3 docs for weakref module say this:
Not all objects can be weakly referenced; those objects which can
include class instances, functions written in Python (but not in C),
instance methods, ...
To me, these indicate that weakrefs to bound methods (in all versions Python 2.7 - 3.3) should be good, and that weakrefs to unbound methods should be good in Python 2.7.
Yet in Python 2.7, creating a weakref to a method (bound or unbound) results in a dead weakref:
>>> def isDead(wr): print 'dead!'
...
>>> class Foo:
... def bar(self): pass
...
>>> wr=weakref.ref(Foo.bar, isDead)
dead!
>>> wr() is None
True
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Not what I would have expected based on the docs.
Similarly, in Python 3.3, a weakref to a bound method dies on creation:
>>> wr=weakref.ref(Foo.bar, isDead)
>>> wr() is None
False
>>> foo=Foo()
>>> wr=weakref.ref(foo.bar, isDead)
dead!
>>> wr() is None
True
Again not what I would have expected based on the docs.
Since this wording has been around since 2.7, it's surely not an oversight. Can anyone explain how the statements and the observed behavior are in fact not in contradiction?
Edit/Clarification: In other words, the statement for 3.3 says "instance methods can be weak referenced"; doesn't this mean that it is reasonable to expect that weakref.ref(an instance method)() is not None? and if it None, then "instance methods" should not be listed among the types of objects that can be weak referenced?
Foo.bar produces a new unbound method object every time you access it, due to some gory details about descriptors and how methods happen to be implemented in Python.
The class doesn't own unbound methods; it owns functions. (Check out Foo.__dict__['bar'].) Those functions just happen to have a __get__ which returns an unbound-method object. Since nothing else holds a reference, it vanishes as soon as you're done creating the weakref. (In Python 3, the rather unnecessary extra layer goes away, and an "unbound method" is just the underlying function.)
Bound methods work pretty much the same way: the function's __get__ returns a bound-method object, which is really just partial(function, self). You get a new one every time, so you see the same phenomenon.
You can store a method object and keep a reference to that, of course:
>>> def is_dead(wr): print "blech"
...
>>> class Foo(object):
... def bar(self): pass
...
>>> method = Foo.bar
>>> wr = weakref.ref(method, is_dead)
>>> 1 + 1
2
>>> method = None
blech
This all seems of dubious use, though :)
Note that if Python didn't spit out a new method instance on every attribute access, that'd mean that classes refer to their methods and methods refer to their classes. Having such cycles for every single method on every single instance in the entire program would make garbage collection way more expensive—and before 2.1, Python didn't even have cycle collection, so they would've stuck around forever.
#Eevee's answer is correct but there is a subtlety that is important.
The Python docs state that instance methods (py3k) and un/bound methods (py2.4+) can be weak referenced. You'd expect (naively, as I did) that weakref.ref(foo.bar)() would therefore be non-None, yet it is None, making the weak ref "dead on arrival" (DOA). This lead to my question, if the weakref to an instance method is DOA, why do the docs say you can weak ref a method?
So as #Eevee showed, you can create a non-dead weak reference to an instance method, by creating a strong reference to the method object which you give to weakref:
m = foo.bar # creates a *new* instance method "Foo.bar" and strong refs it
wr = weakref.ref(m)
assert wr() is not None # success
The subtlety (to me, anyways) is that a new instance method object is created every time you use Foo.bar, so even after the above code is run, the following will fail:
wr = weakref.ref(foo.bar)
assert wr() is not None # fails
because foo.bar is new instance of the "Foo instance" foo's "bar" method, different from m, and there is no strong ref to this new instance, so it is immediately gc'd, even if you have created a strong reference to it earlier (it is not the same strong ref). To be clear,
>>> d1 = foo.bla # assume bla is a data member
>>> d2 = foo.bla # assume bla is a data member
>>> d1 is d2
True # which is what you expect
>>> m1 = foo.bar # assume bar is an instance method
>>> m2 = foo.bar
>>> m1 is m2
False # !!! counter-intuitive
This takes many people by surprise since no one expects access to an instance member to be creating a new instance of anything. For example, if foo.bla is a data member of foo, then using foo.bla in your code does not create a new instance of the object referenced by foo.bla. Now if bla is a "function", foo.bla does create a new instance of type "instance method" representing the bound function.
Why the weakref docs (since python 2.4!) don't point that out is very strange, but that's a separate issue.
While I see that there's an accepted answer as to why this should be so, from a simple use-case situation wherein one would like an object that acts as a weakref to a bound method, I believe that one might be able to sneak by with an object as such. It's kind of a runt compared to some of the 'codier' things out there, but it works.
from weakref import proxy
class WeakMethod(object):
"""A callable object. Takes one argument to init: 'object.method'.
Once created, call this object -- MyWeakMethod() --
and pass args/kwargs as you normally would.
"""
def __init__(self, object_dot_method):
self.target = proxy(object_dot_method.__self__)
self.method = proxy(object_dot_method.__func__)
###Older versions of Python can use 'im_self' and 'im_func' in place of '__self__' and '__func__' respectively
def __call__(self, *args, **kwargs):
"""Call the method with args and kwargs as needed."""
return self.method(self.target, *args, **kwargs)
As an example of its ease of use:
class A(object):
def __init__(self, name):
self.name = name
def foo(self):
return "My name is {}".format(self.name)
>>> Stick = A("Stick")
>>> WeakFoo = WeakMethod(Stick.foo)
>>> WeakFoo()
'My name is Stick'
>>> Stick.name = "Dave"
>>> WeakFoo()
'My name is Dave'
Note that evil trickery will cause this to blow up, so depending on how you'd prefer it to work this may not be the best solution.
>>> A.foo = lambda self: "My eyes, aww my eyes! {}".format(self.name)
>>> Stick.foo()
'My eyes, aww my eyes! Dave'
>>> WeakFoo()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __call__
ReferenceError: weakly-referenced object no longer exists
>>>
If you were going to be replacing methods on-the-fly you might need to use a getattr(weakref.proxy(object), 'name_of_attribute_as_string') approach instead. getattr is a fairly fast look-up so that isn't the literal worst thing in the world, but depending on what you're doing, YMMV.
Is there any way to get the object name when the class name is known. If there are multiple objects for a class they also need to be printed.
Class A():
pass
Assume that some one have created objects for class A in some other files. So, I want to look all instances of 'Class A'
If you are the one creating the class you can simply store weak-references when instantiating the class:
import weakref
class A(object):
instances = []
def __init__(self):
A.instances.append(weakref.ref(self))
a, b, c = A(), A(), A()
instances = [ref() for ref in A.instances if ref() is not None]
Using weak-references allow the instances to be deallocated before the class.
See the weakref module for details on what it does.
Note that you may be able to use this technique even with classes that you didn't write. You simply have to monkey-patch the class.
For example:
def track_instances(cls):
def init(self, *args, **kwargs):
getattr(self, 'instances').append(weakref.ref(self))
getattr(self, '_old_init')(self, *args, **kwargs)
cls._old_init = cls.__init__
cls.__init__ = init
return cls
Then you can do:
track_instances(ExternalClass)
And all instances created after the execution of this statement will be found in ExternalClass.instances.
Depending on the class you may have to replace __new__ instead of __init__.
You can do this even without any special code in the class, simply using the garbage collector:
import gc
candidates = gc.get_referrers(cls_object)
instances = [candidate for candidate in candidates if isinstance(candidate, cls_object)]
And you can always obtain the class object since you can find it using object.__subclasses__ method:
cls_object = next(cls for cls in object.__subclasses__() if cls.__name__ == cls_name)
(assuming there is only a class with that name, otherwise you should try all of them)
However I cannot think of a situation where this is the right thing to do, so avoid this code in real applications.
I've done some testing and I believe that this solution may not work for built-in classes or classes defined in C extensions.
If you are in this case the last resort is to use gc.get_objects() to retrieve all tracked objects. However this will work only if the object support cyclic garbage collection, so there isn't a method that works in every possible situation.
Here the version getting the instances from memory, I wouldn't recommend using this in live code but it can be convenient for debugging:
import weakref
class SomeClass(object):
register = []
def __init__(self):
self.register.append(weakref.ref(self))
a = SomeClass()
b = SomeClass()
c = SomeClass()
# Now the magic :)
import gc
def get_instances(class_name):
# Get the objects from memory
for instance in gc.get_objects():
# Try and get the actual class
class_ = getattr(instance, '__class__', None)
# Only return if the class has the name we want
if class_ and getattr(class_, '__name__', None) == class_name:
yield instance
print list(get_instances('SomeClass'))
Python provides the types module that defined classes for built-in types and the locals() and globals() functions that return a list of local and global variables in the application.
One quick way to find objects by type is to do this.
import types
for varname, var_instance in locals().items():
if type(var_instance) == types.InstanceType and var_instance.__class__.__name__ == 'CLASS_NAME_YOU_ARE_LOOKING_FOR':
print "This instance was found:", varname, var_instance
It's worth going through the Python library documentation and read the docs for modules that work with the code directly. Some of which are inspect, gc, types, codeop, code, imp, ast. bdb, pdb. The IDLE source code is also very informative.
Instances are created within a namespace:
def some_function():
some_object = MyClass()
In this case, some_object is a name inside the "namespace" of the function that points at a MyClass instance. Once you leave the namespace (i.e., the function ends), Python's garbage collection cleans up the name and the instance.
If there would be some other location that also has a pointer to the object, the cleanup wouldn't happen.
So: no, there's no place where a list of instances is maintained.
It would be a different case where you to use a database with an ORM (object-relational mapper). In Django's ORM you can do MyClass.objects.all() if MyClass is a database object. Something to look into if you really need the functionality.
Update: See Bakuriu's answer. The garbage collector (which I mentioned) knows about all the instances :-) And he suggests the "weakref" module that prevents my won't-be-cleaned-up problem.
You cann get names for all the instances as they may not all have names, or the names they do have may be in scope. You may be able to get the instances.
If you are willing to keep track of the instances yourself, use a WeakSet:
import weakref
class SomeClass(object):
instances = weakref.WeakSet()
def __init__(self):
self.instances.add(self)
>>> instances = [SomeClass(), SomeClass(), SomeClass()]
>>> other = SomeClass()
>>> SomeClass.instances
<_weakrefset.WeakSet object at 0x0291F6F0>
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x028F0150>, <__main__.SomeClass object at 0x0291F210>]
Note that just deleting a name may not destroy the instance. other still exists until the garbage collected:
>>> del other
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x028F0150>, <__main__.SomeClass object at 0x0291F210>]
>>> import gc
>>> gc.collect()
0
>>> list(SomeClass.instances)
[<__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>, <__main__.SomeClass object at 0x0291F210>]
If you don't want to track them manually, then it is possible to use gc.get_objects() and filter out the instances you want, but that means you have to filter through all the objects in your program every time you do this. Even in the above example that means processing nearly 12,000 objects to find the 3 instances you want.
>>> [g for g in gc.get_objects() if isinstance(g, SomeClass)]
[<__main__.SomeClass object at 0x0291F210>, <__main__.SomeClass object at 0x0291F710>, <__main__.SomeClass object at 0x0291F730>]
>>> class TestClass:
... pass
...
>>> foo = TestClass()
>>> for i in dir():
... if isinstance(eval(i), TestClass):
... print(i)
...
foo
>>>
Finally found a way to get through.
As I know the class name, I would search for the object created for that class in garbage collector(gc) like this...
for instance in gc.get_objects():
if str(type(instance)).find("dict") != -1:
for k in instance.keys():
if str(k).find("Sample") != -1:
return k
The above code returns an instance of the class which will be like this. Unfortunately,its in String format which doesn't suit the requirement. It should be of 'obj' type.
<mod_example.Sample object at 0x6f55250>
From the above value, parse the id(0x6f55250) and get the object reference based on the id.
obj_id = 0x6f55250
for obj in gc.get_objects():
# Converting decimal value to hex value
if id(obj) == ast.literal_eval(obj_id):
required_obj = obj
Hence required_obj will hold the object reference exactly in the 'obj' format.
:-)
Please look at the simple example:
class A:
def __init__(self, flag):
self.flag = flag
def func(self):
print self.flag
a = A(1)
b = A(2)
callback_a = a.func
callback_b = b.func
callback_a()
callback_b()
The result is:
1
2
It runs as expected. But I have a question. In C, the callback function is passed as a pointer. In Python, it should have a similar way to do this, so the caller knows the address of the function. But in my example, not only the function pointer is passed, but also the parameter (self) is passed, because the same method of the same class prints different results. So my questions are:
Does such a method in Python only has one copy in memory? My meaning is that the code of any method only has one copy, and in my example the method won't be cloned itself. I think it should have only one copy, but here I still make this question in order to get more inputs.
I remember everything in Python is an object. So in my example, are there two function instances with different parameters but only one copy of code?
In Python, the callback is not simply a reference to a member function. Instead, it is "bound" to the object that it refers to when it was created. So a.func creates a callable that is bound to a, and b.func creates a callable that is bound to b.
Python only needs one implementation of func() in memory, but it will probably create one or more "trampoline" functions at runtime to accomplish the binding (I'm not certain of the internal details on this, and it would differ between Python implementations anyway).
If you print id(callback_a) and id(callback_b) you will get different results, showing that they are indeed different callable objects.
Specific to CPython, there is only one copy of the function object. During instance creation, the class wraps the unbound functions in its namespace as bound methods. But they all wrap the same function.
Here's your example expanded to show what's going on.
class A(object):
def __init__(self, flag):
self.flag = flag
def func(self):
print self.flag
a = A(1)
b = A(2)
callback_a = a.func
callback_b = b.func
print "typeof(callback_a) = {0}".format(type(callback_a))
print "typeof(callback_b) = {0}".format(type(callback_b))
print "typeof(callback_a.__func__) = {0}".format(type(callback_a.__func__))
print "typeof(callback_b.__func__) = {0}".format(type(callback_b.__func__))
print "'callback_a.__func__ is callback_b.__func__' is {0}".format(callback_a.__func__ is callback_b.__func__)
callback_a()
callback_b()
This code outputs
typeof(callback_a) = <type 'instancemethod'>
typeof(callback_b) = <type 'instancemethod'>
typeof(callback_a.__func__) = <type 'function'>
typeof(callback_b.__func__) = <type 'function'>
'callback_a.__func__ is callback_b.__func__' is True
You can clearly see, using the is operator, that both instancemethod classes are sharing the same function object.