Suppose there is a program with a couple of objects living in it at runtime.
Is the __del__ method of each object called when the programs ends?
If yes I could for example do something like this:
class Client:
__del__( self ):
disconnect_from_server()
There are many potential difficulties associated with using __del__.
Usually, it is not necessary, or the best idea to define it yourself.
Instead, if you want an object that cleans up after itself upon exit or an exception, use a context manager:
per Carl's comment:
class Client:
def __exit__(self,ext_type,exc_value,traceback):
self.disconnect_from_server()
with Client() as c:
...
original answer:
import contextlib
class Client:
...
#contextlib.contextmanager
def make_client():
c=Client()
yield c
c.disconnect_from_server()
with make_client() as c:
...
I second the general idea of using context managers and the with statement instead of relying on __del__ (for much the same reasons one prefers try/finally to finalizer methods in Java, plus one: in Python, the presence of __del__ methods can make cyclic garbage uncollectable).
However, given that the goal is to have "an object that cleans up after itself upon exit or an exception", the implementation by #~unutbu is not correct:
#contextlib.contextmanager
def make_client():
c=Client()
yield c
c.disconnect_from_server()
with make_client() as c:
...
If an exception is raised in the ... part, disconnect_from_server_ does not get called (since the exception propagates through make_client, being uncaught there, and therefore terminates it while it's waiting at the yield).
The fix is simple:
#contextlib.contextmanager
def make_client():
c=Client()
try: yield c
finally: c.disconnect_from_server()
Essentially, the with statement lets you almost forget about the good old try/finally statement... except when you're writing context managers with contextlib, and then it's really important to remember it!-)
Consider using with-statement to make cleanup explicit.
With circular references __del__ is not called:
class Foo:
def __del__(self):
self.p = None
print "deleting foo"
a = Foo()
b = Foo()
a.p = b
b.p = a
prints nothing.
Yes, the Python interpreter tidies up at shutdown, including calling the __del__ method of every object (except objects that are part of a reference cycle).
Although, as others have pointed out, __del__ methods are very fragile and should be used with caution.
Related
Take a look at the following example:
from contextlib import AbstractContextManager, contextmanager
class MyClass(AbstractContextManager):
_values = {}
#contextmanager
def using(self, name, value):
print(f'Allocating {name} = {value}')
self._values[name] = value
try:
yield
finally:
print(f'Releasing {name}')
del self._values[name]
def __enter__(self):
return self.using('FOO', 42).__enter__()
def __exit__(self, exc_type, exc_val, exc_tb):
pass
with MyClass():
print('Doing work...')
I would expect the above code to print the following:
Allocating FOO = 42
Doing work...
Releasing FOO
Instead, this is what is being printed:
Allocating FOO = 42
Releasing FOO
Doing work...
Why is FOO getting released eagerly?
You're creating two context managers here. Only one of those context managers is actually implemented correctly.
Your using context manager is fine, but you've also implemented the context manager protocol on MyClass itself, and the implementation on MyClass is broken. MyClass.__enter__ creates a using context manager, enters it, returns what that context manager's __enter__ returns, and then throws the using context manager away.
You don't exit the using context manager when MyClass() is exited. You never exit it at all! You throw the using context manager away. It gets reclaimed, and when it does, the generator gets close called automatically, as part of normal generator cleanup. That throws a GeneratorExit exception into the generator, triggering the finally block.
Python doesn't promise when this cleanup will happen (or indeed, if it will happen at all), but in practice, CPython's reference counting mechanism triggers the cleanup as soon as the using context manager is no longer reachable.
Aside from that, if _values is supposed to be an instance variable, it should be set as self._values = {} inside an __init__ method. Right now, it's a class variable.
class Package:
def __init__(self):
self.files = []
# ...
def __del__(self):
for file in self.files:
os.unlink(file)
__del__(self) above fails with an AttributeError exception. I understand Python doesn't guarantee the existence of "global variables" (member data in this context?) when __del__() is invoked. If that is the case and this is the reason for the exception, how do I make sure the object destructs properly?
I'd recommend using Python's with statement for managing resources that need to be cleaned up. The problem with using an explicit close() statement is that you have to worry about people forgetting to call it at all or forgetting to place it in a finally block to prevent a resource leak when an exception occurs.
To use the with statement, create a class with the following methods:
def __enter__(self)
def __exit__(self, exc_type, exc_value, traceback)
In your example above, you'd use
class Package:
def __init__(self):
self.files = []
def __enter__(self):
return self
# ...
def __exit__(self, exc_type, exc_value, traceback):
for file in self.files:
os.unlink(file)
Then, when someone wanted to use your class, they'd do the following:
with Package() as package_obj:
# use package_obj
The variable package_obj will be an instance of type Package (it's the value returned by the __enter__ method). Its __exit__ method will automatically be called, regardless of whether or not an exception occurs.
You could even take this approach a step further. In the example above, someone could still instantiate Package using its constructor without using the with clause. You don't want that to happen. You can fix this by creating a PackageResource class that defines the __enter__ and __exit__ methods. Then, the Package class would be defined strictly inside the __enter__ method and returned. That way, the caller never could instantiate the Package class without using a with statement:
class PackageResource:
def __enter__(self):
class Package:
...
self.package_obj = Package()
return self.package_obj
def __exit__(self, exc_type, exc_value, traceback):
self.package_obj.cleanup()
You'd use this as follows:
with PackageResource() as package_obj:
# use package_obj
The standard way is to use atexit.register:
# package.py
import atexit
import os
class Package:
def __init__(self):
self.files = []
atexit.register(self.cleanup)
def cleanup(self):
print("Running cleanup...")
for file in self.files:
print("Unlinking file: {}".format(file))
# os.unlink(file)
But you should keep in mind that this will persist all created instances of Package until Python is terminated.
Demo using the code above saved as package.py:
$ python
>>> from package import *
>>> p = Package()
>>> q = Package()
>>> q.files = ['a', 'b', 'c']
>>> quit()
Running cleanup...
Unlinking file: a
Unlinking file: b
Unlinking file: c
Running cleanup...
A better alternative is to use weakref.finalize. See the examples at Finalizer Objects and Comparing finalizers with __del__() methods.
As an appendix to Clint's answer, you can simplify PackageResource using contextlib.contextmanager:
#contextlib.contextmanager
def packageResource():
class Package:
...
package = Package()
yield package
package.cleanup()
Alternatively, though probably not as Pythonic, you can override Package.__new__:
class Package(object):
def __new__(cls, *args, **kwargs):
#contextlib.contextmanager
def packageResource():
# adapt arguments if superclass takes some!
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
yield package
package.cleanup()
def __init__(self, *args, **kwargs):
...
and simply use with Package(...) as package.
To get things shorter, name your cleanup function close and use contextlib.closing, in which case you can either use the unmodified Package class via with contextlib.closing(Package(...)) or override its __new__ to the simpler
class Package(object):
def __new__(cls, *args, **kwargs):
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
return contextlib.closing(package)
And this constructor is inherited, so you can simply inherit, e.g.
class SubPackage(Package):
def close(self):
pass
Here is a minimal working skeleton:
class SkeletonFixture:
def __init__(self):
pass
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
pass
def method(self):
pass
with SkeletonFixture() as fixture:
fixture.method()
Important: return self
If you're like me, and overlook the return self part (of Clint Miller's correct answer), you will be staring at this nonsense:
Traceback (most recent call last):
File "tests/simplestpossible.py", line 17, in <module>
fixture.method()
AttributeError: 'NoneType' object has no attribute 'method'
Hope it helps the next person.
I don't think that it's possible for instance members to be removed before __del__ is called. My guess would be that the reason for your particular AttributeError is somewhere else (maybe you mistakenly remove self.file elsewhere).
However, as the others pointed out, you should avoid using __del__. The main reason for this is that instances with __del__ will not be garbage collected (they will only be freed when their refcount reaches 0). Therefore, if your instances are involved in circular references, they will live in memory for as long as the application run. (I may be mistaken about all this though, I'd have to read the gc docs again, but I'm rather sure it works like this).
I think the problem could be in __init__ if there is more code than shown?
__del__ will be called even when __init__ has not been executed properly or threw an exception.
Source
Just wrap your destructor with a try/except statement and it will not throw an exception if your globals are already disposed of.
Edit
Try this:
from weakref import proxy
class MyList(list): pass
class Package:
def __init__(self):
self.__del__.im_func.files = MyList([1,2,3,4])
self.files = proxy(self.__del__.im_func.files)
def __del__(self):
print self.__del__.im_func.files
It will stuff the file list in the del function that is guaranteed to exist at the time of call. The weakref proxy is to prevent Python, or yourself from deleting the self.files variable somehow (if it is deleted, then it will not affect the original file list). If it is not the case that this is being deleted even though there are more references to the variable, then you can remove the proxy encapsulation.
It seems that the idiomatic way to do this is to provide a close() method (or similar), and call it explicitely.
A good idea is to combine both approaches.
To implement a context manager for explicit life-cycle handling. As well as handle cleanup in case the user forgets it or it is not convenient to use a with statement. This is best done by weakref.finalize.
This is how many libraries actually do it. And depending on the severity, you could issue a warning.
It is guaranteed to be called exactly once, so it is safe to call it at any time before.
import os
from typing import List
import weakref
class Package:
def __init__(self):
self.files = []
self._finalizer = weakref.finalize(self, self._cleanup_files, self.files)
#staticmethod
def _cleanup_files(files: List):
for file in files:
os.unlink(file)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
self._finalizer()
weakref.finalize returns a callable finalizer object which will be called when obj is garbage collected. Unlike an ordinary weak reference, a finalizer will always survive until the reference object is collected, greatly simplifying lifecycle management."
Unlike atexit.register the object is not held in memory until the interpreter is shut down.
And unlike object.__del__, weakref.finalize is guaranteed to be called at interpreter shutdown. So it is much more safe.
atexit.register is the standard way as has already been mentioned in ostrakach's answer.
However, it must be noted that the order in which objects might get deleted cannot be relied upon as shown in example below.
import atexit
class A(object):
def __init__(self, val):
self.val = val
atexit.register(self.hello)
def hello(self):
print(self.val)
def hello2():
a = A(10)
hello2()
a = A(20)
Here, order seems legitimate in terms of reverse of the order in which objects were created as program gives output as :
20
10
However when, in a larger program, python's garbage collection kicks in object which is out of it's lifetime would get destructed first.
I have an large script where i found out that lot of connections to a machine are left open and the reason was that for one of the class destructor was never getting called.
below is a simplified version of script manifesting the issue.
I tiered searching around and found out that it could be because of GC and weakref does help but in this case no help.
2 cases where i can see that the destructor is getting called are
If i call B_class object without passing A_class function
self.b = B_class("AA")
I call the make the B_class objects not global i.e not use self
b = B_class("AA",self.myprint)
b.do_something()
Both of these cases will cause further issues for my case. Last resort will be to close/del the objects at the end myself but i don't want to go that way.
can anybody suggest a better way out of this and help me understand this issue? Thanks in advance.
import weakref
class A_class:
def __init__(self,debug_level=1,version=None):
self.b = B_class("AA",self.myprint)
self.b.do_something()
def myprint(self, text):
print text
class B_class:
def __init__(self,ip,printfunc=None):
self.ip=ip
self.new_ip =ip
#self.printfunc = printfunc
self.printfunc = weakref.ref(printfunc)()
def __del__(self):
print("##B_Class Destructor called##")
def do_something(self,timeout=120):
self.myprint("B_Class ip=%s!!!" % self.new_ip)
def myprint(self,text):
if self.printfunc:
print ("ExtenalFUNC:%s" %text)
else:
print ("JustPrint:%s" %text)
def _main():
a = A_class()
if __name__ == '__main__':
_main()
You're not using the weakref.ref object properly. You're calling it immediately after it is created, which returns the referred-to object (the function passed in as printref).
Normally, you'd want to save the weak reference and only call it when you're going to use the reffered-to object (e.g. in myprint). However, that won't work for the bound method self.myprint you're getting passed in as printfunc, since the bound method object doesn't have any other references (every access to a method creates a new object).
If you're using Python 3.4 or later and you know that the object passed in will always be a bound method, you can use the WeakMethod class, rather than a regular ref. If you're not sure what kind of callable you're going to get, you might need to do some type checking to see if WeakMethod is required or not.
Use Python's "with" statement (http://www.python.org/dev/peps/pep-0343/).
It creates a syntactic scope and the __exit__ function which it creates is guaranteed to get called as soon as execution leaves the scope. You can also emulate "__enter__/__exit__" behavior by creating a generator with "contextmanager" decorator from the contextlib module (python 2.6+ or 2.5 using "from __future__ import with_statement" see PEP for examples).
Here's an example from the PEP:
import contextlib
#contextlib.contextmanger
def opening(filename):
f = open(filename) # IOError is untouched by GeneratorContext
try:
yield f
finally:
f.close() # Ditto for errors here (however unlikely)
and then in your main code, you write
with opening(blahblahblah) as f:
pass
# use f for something
# here you exited the with scope and f.close() got called
In your case, you'll want to use a different name (connecting or something) instead of "opening" and do socket connecting/disconnecting inside of your context manager.
self.printfunc = weakref.ref(printfunc)()
isn't actually using weakrefs to solve your problem; the line is effectively a noop. You create a weakref with weakref.ref(printfunc), but you follow it up with call parens, which converts back from weakref to a strong ref which you store (and the weakref object promptly disappears). Apparently it's not possible to store a weakref to the bound method itself (because the bound method is its own object created each time it's referenced on self, not a cached object whose lifetime is tied to self), so you have to get a bit hacky, unbinding the method so you can take a weakref on the object itself. Python 3.4 introduced WeakMethod to simplify this, but if you can't use that, then you're stuck doing it by hand.
Try changing it to (on Python 2.7, and you must import inspect):
# Must special case printfunc=None, since None is not weakref-able
if printfunc is None:
# Nothing provided
self.printobjref = self.printfuncref = None
elif inspect.ismethod(printfunc) and printfunc.im_self is not None:
# Handling bound method
self.printobjref = weakref.ref(printfunc.im_self)
self.printfuncref = weakref.ref(printfunc.im_func)
else:
self.printobjref = None
self.printfuncref = weakref.ref(printfunc)
and change myprint to:
def myprint(self,text):
if self.printfuncref is not None:
printfunc = self.printfuncref()
if printfunc is None:
self.printfuncref = self.printobjref = None # Ref died, so clear it to avoid rechecking later
elif self.printobjref is not None:
# Bound method not known to have disappeared
printobj = self.printobjref()
if printobj is not None:
print ("ExtenalFUNC:%s" %text) # To call it instead of just saying you have it, do printfunc(printobj, text)
return
self.printobjref = self.printfuncref = None # Ref died, so clear it to avoid rechecking later
else:
print ("ExtenalFUNC:%s" %text) # To call it instead of just saying you have it, do printfunc(text)
return
print ("JustPrint:%s" %text)
Yeah, it's ugly. You could factor out the ugly if you like (borrowing the implementation of WeakMethod from Python 3.4's source code would make sense, but names would have to change; __self__ is im_self in Py2, __func__ is im_func), but it's unpleasant even so. It's definitely not thread safe if the weakrefs could actually go dark, since the checks and clears of the weakref members aren't protected.
From the python docs:
It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.
Why not? What problems would occur if this guarantee were made?
I'm not convinced by the previous answers here.
Firstly note that the example given does not prevent __del__ methods being called during exit. In fact, the current CPythons will call the __del__ method given, twice in the case of Python 2.7 and once in the case of Python 3.4. So this can't be the "killer example" which shows why the guarantee is not made.
I think the statement in the docs is not motivated by a design principle that calling the destructors would be bad. Not least because it seems that in CPython 3.4 and up they are always called as you would expect and this caveat seems to be moot.
Instead I think the statement simply reflects the fact that the CPython implementation has sometimes not called all destructors on exit (presumably for ease of implementation reasons).
The situation seems to be that CPython 3.4 and 3.5 do always call all destructors on interpreter exit.
CPython 2.7 by contrast does not always do this. Certainly __del__ methods are usually not called on objects which have cyclic references, because those objects cannot be deleted if they have a __del__ method. The garbage collector won't collect them. While the objects do disappear when the interpreter exits (of course) they are not finalized and so their __del__ methods are never called. This is no longer true in Python 3.4 after the implementation of PEP 442.
However, it seems that Python 2.7 also does not finalize objects that have cyclic references, even if they have no destructors, if they only become unreachable during the interpreter exit.
Presumably this behaviour is sufficiently particular and difficult to explain that it is best expressed simply by a generic disclaimer - as the docs do.
Here's an example:
class Foo(object):
def __init__(self):
print("Foo init running")
def __del__(self):
print("Destructor Foo")
class Bar(object):
def __init__(self):
print("Bar1 init running")
self.bar = self
self.foo = Foo()
b = Bar()
# del b
With the del b commented out, the destructor in Foo is not called in Python 2.7 though it is in Python 3.4.
With the del b added, then the destructor is called (at interpreter exit) in both cases.
If you did some nasty things, you could find yourself with an undeletable object which python would try to delete forever:
class Phoenix(object):
def __del__(self):
print "Deleting an Oops"
global a
a = self
a = Phoenix()
Relying on __del__ isn't great in any event as python doesn't guarantee when an object will be deleted (especially objects with cyclic references). That said, perhaps turning your class into a context manager is a better solution ... Then you can guarantee that cleanup code is called even in the case of an exception, etc...
One example where the destructor is not called is, if you exit inside a method. Have a look at this example:
class Foo(object):
def __init__(self):
print("Foo init running")
def __del__(self):
print("Destructor Foo")
class Bar(object):
def __init__(self):
print("Bar1 init running")
self.bar = self
self.foo = Foo()
def __del__(self):
print("Destructor Bar")
def stop(self):
del self.foo
del self
exit(1)
b = Bar()
b.stop()
The output is:
Bar1 init running
Foo init running
Destructor Foo
As we destruct foo explicitly, the destructor is called, but not the destructor of bar!
And, if we do not delete foo explicitly, it is also not destructed properly:
class Foo(object):
def __init__(self):
print("Foo init running")
def __del__(self):
print("Destructor Foo")
class Bar(object):
def __init__(self):
print("Bar1 init running")
self.bar = self
self.foo = Foo()
def __del__(self):
print("Destructor Bar")
def stop(self):
exit(1)
b = Bar()
b.stop()
Output:
Bar1 init running
Foo init running
I don't think this is because doing the deletions would cause problems. It's more that the Python philosophy is not to encourage developers to rely on the use of object deletion, because the timing of these deletions cannot be predicted - it is up to the garbage collector when it occurs.
If the garbage collector may defer deleting unused objects for an unknown amount of time after they go out of scope, then relying on side effects that happen during the object deletion is not a very robust or deterministic strategy. RAII is not the Python way. Instead Python code handles cleanup using context managers, decorators, and the like.
Worse, in complicated situations, such as with object cycles, the garbage collector might not ever detect that objects can be deleted. This situation has improved as Python has matured. But because of exceptions to the expected GC behaviour like this, it is unwise for Python developers to rely on object deletion.
I speculate that interpreter exit is another complicated situation where the Python devs, especially for older versions of Python, were not completely strict about making sure the GC delete ran on all objects.
Likely because most of programmers would assume that destructors should only be called on dead (already unreachable) objects, and here on exit we would invoke them on live objects.
If it the developer has not been expecting a destructor call on the live object, some nasty UB may result. At least, something must be done to force-close the application after time out if it hangs. But then some destructors may not be called.
Java Runtime.runFinalizersOnExit has been deprecated because of the same reason.
I have two questions about creating thread safe types in python, and one related question about multiple inheritance.
1) Are there any problematic implications with using the following subclasses in my threaded application as a sort of "lazy" thread-safe type? I realize that whomever sets values which may be altered by other threads bears the responsibility to ensure those values are thread safe as well.
2) Another question I have is if there exists more prudent alternatives to these types within python in a typical installation.
Example:
from threading import Lock
from __future__ import with_statement
class safedict(dict):
def __init__(self,*args,**kwargs):
self.mylock=Lock();
super(safedict, self).__init__(*args, **kwargs)
def __setitem__(self,*args,**kwargs):
with self.mylock:
print " DEBUG: Overloaded __setitem__ has the lock now."
super(safedict,self).__setitem__(*args,**kwargs)
class safeobject(object):
mylock = Lock(); # a temporary useless lock, until we have a proper instance.
def __init__(self,*args,**kwargs):
self.mylock=Lock();
super(safeobject, self).__init__(*args, **kwargs)
def __setattr__(self,*args,**kwargs):
with self.mylock:
print " DEBUG: Overloaded __setattr__ has the lock now."
super(safeobject,self).__setattr__(*args,**kwargs)
3) If both of the types defined above could be considered reasonably safe, what negative implications would be faced by using multiple inheritance to create a type that supported a mixture of both of these modifications, and does my example inherit those classes in the optimal order?
Example:
class safedict2(safeobject,dict):
def __setitem__(self,*args,**kwargs):
with self.mylock:
print " DEBUG: Overloaded __setitem__ has the lock now."
super(safedict2,self).__setitem__(*args,**kwargs)
Edit:
Just another example of another type inheriting both of the former types, and testing using ipython.
In [304]: class safedict3(safeobject,safedict):
.....: pass
.....:
In [305]: d3 = safedict3()
DEBUG: Overloaded __setattr__ has the lock now.
DEBUG: Overloaded __setattr__ has the lock now.
In [306]: d3.a=1
DEBUG: Overloaded __setattr__ has the lock now.
In [307]: d3['b']=2
DEBUG: Overloaded __setitem__ has the lock now.
In [308]: d3
Out[308]: {'b': 2}
As to your first and second questions, the dict, list, etc. types are already thread-safe. You do not have to add thread safety to them. However you may find this useful. It's a decorator that basically implements the synchronized keyword from Java, using function scope to define a critical section. Using a similar approach it is possible to author a threading.Condition oriented decorator also.
import threading
def tryfinally(finallyf):
u"returns a decorator that adds try/finally behavior with given no-argument call in the finally"
def decorator(callable):
def execute(*args, **kwargs):
try: result = callable(*args, **kwargs)
finally: finallyf()
return result
return execute
return decorator
def usinglock(lock):
u"returns a decorator whose argument will acquire the given lock while executing"
def decorator(function):
body = tryfinally(lock.release)(function)
def execute(*args, **kwargs):
lock.acquire()
return body(*args, **kwargs)
return execute
return decorator
def synchronized(function):
u"decorator; only one thread can enter the decorated function at a time; recursion is OK"
return usinglock(threading.RLock())(function)
Use it like this (and beware deadlocks if you overuse it):
#synchronized
def foo(*args):
print 'Only one thread can enter this function at a time'
On the third question, the Python tutorial states that the search order for inherited attributes is depth-first, left-first. So if you inherit (myclass, dict) then the __setitem__ method from myclass should be used. (In older versions of Python, this same section in the tutorial implied that this choice was arbitrary, but nowadays it appears to be quite deliberate.)
I'm guessing from the Freudian slip of a semicolon in the posted source that you are new to Python but experienced in either Java or C#. If so you will need to keep in mind that attribute (method) resolution occurs at run time in Python, and that the classes as well as the instance are first-class objects that can be inspected/explored at run time.
First the instance attribute dictionary is searched, then the class attributes, and then the parent class search algorithm starts. This is done with (conceptually) the equivalent of repeated hasattr(class_or_instance, attribute) calls.
The below confirms that for "new-style" classes (classes that inherit from object, which in the 2.x language specification is optional), this resolution occurs each time the attribute is looked up. It is not done when the class (or subclass) is created, or when instances are created. (This was done in release 2.7.2.)
>>> class Foo(object):
... def baz(self):
... print 'Original Foo.baz'
...
>>> class Bar(Foo): pass
...
>>> def newprint(self):
... print 'New Foo.baz'
...
>>> x = Foo()
>>> y = Bar()
>>> Foo.baz = newprint
>>> a = Foo()
>>> b = Bar()
>>> map(lambda k: k.baz(), (x, y, a, b))
New Foo.baz
New Foo.baz
New Foo.baz
New Foo.baz
[None, None, None, None]
Replacing the method of class Foo changes the behavior of subclasses already defined and of instances already created.