I am wondering if there is a module-wide destructor such that we can make use of it to finalize or call some specific shut down functions in a module?
For example, some handlers of the module logbook are created and pust into the stack (ex, handler1.push_application() and it is better to pop up those handlers when your program exit. It would be great to have some sort of automatically function-calls to do this and module-wide destructor is one of possible candidates I can think :)
The atexit module allows you to register cleanup functions that Python will perform on interpreter termination.
Related
According to Python documentation:
It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.
I know that in older versions of Python cyclic referencing would be one of the examples for this behaviour, however as I understand it, in Python 3 such cycles will successfully be destroyed upon interpreter exit.
I'm wondering what are the cases (as close to exhaustive list as possible) when the interpreter would not destroy an object upon exit.
All examples are implementation details - Python does not promise whether or not it will call __del__ for any particular objects on interpreter exit. That said, one of the simplest examples is with daemon threads:
import threading
import time
def target():
time.sleep(1000)
class HasADel:
def __del__(self):
print('del')
x = HasADel()
threading.Thread(target=target, daemon=True).start()
Here, the daemon thread prevents the HasADel instance from being garbage collected on interpreter shutdown. The daemon thread doesn't actually do anything with that object, but Python can't clean up references the daemon thread owns, and x is reachable from references the daemon thread owns.
When the interpreter exits normally, in such ways as the program ending or sys.exit being called, not all objects are guaranteed to be destroyed. There is probably some amount of logic to this, but not very simple logic. After all, the __del__ method is for freeing memory resources, not other resources (like network connections) - that's what __enter__ and __exit__ are for.
Having said that, there are situtations in which __del__ will most certainly not be called. The parallel to this is atexit functions; they are usually run at exit. However:
Note: The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.
atexit documentation
So, there are situations in which clean-up functions, like __del__, __exit__, and functions registered with atexit will not be called:
The program is killed by a signal not handled by Python - If a program recieves a signal to stop, like SIGINT or SIGQUIT, and it doesn't handle the signal, then it will be stopped.
A Python fatal interpreter error occurs.
os._exit() is called - the documentation says:
Exit the process with status n, without calling cleanup handlers, flushing stdio buffers, etc.
So it is pretty clear that __del__ should not be called.
In conclusion, the interpreter does not guarantee __del__ being called, but there are situations in which it will definitely not be called.
After comparing the quoted sentence from documentation and your title, I thought you misunderstood what __del__ is and what it does.
You used the word "destroyed", and documentation said __del__ may not get called in some situations... The thing is "all" objects are get deleted after the interpreter's process finishes. __del__ is not a destructor and has nothing to do with the destruction of objects. Even if a memory leakage occurs in a process, operating systems(the ones I know at least: Linux, Windows,...) will eventually reclaim that memory for the process after it finishes. So everything is destroyed/deleted!(here and here)
In normal cases when these objects are about to get destroyed, __del__ (better known as finalizer) gets called in the very last step of destruction. In other cases mentioned by other answers, It doesn't get called.
That's why people say don't count on __del__ method for cleaning vital stuff and instead use a context manager. In some scenarios, __del__ may even revive the object by passing a reference around.
I'm doing some processing in a __del__() destructor of a Python object that I don't want to happen if the program exited via exception. Is there a way to check from __del__() if I'm in the middle of normal exit or exception unwinding?
Alternatively is there some way to check for the same condition in the atexit function?
Do not use a __del__ for this. The __del__ hook is not even guaranteed to be called:
It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.
Instead, manage the cache with a Context Manager and only mark the cache as reusable when the __exit__() method is called with the exc_type set to None.
I have a python cgi script that accepts user uploads (via sys.stdin.read).
After receiving the file (whether successfully or unsuccessfully), the script needs to do some cleanup. This works fine when upload finishes correctly, however if the user closes the client, the cgi script is silently killed on the server, and as a result no cleanup code gets executed. How can i force the script to always finish.
You can trap the exit signal with the signal module. Haven't tried this with mod_python though.
http://docs.python.org/library/signal.html
Note in the docs:
When a signal arrives during an I/O operation, it is possible that the I/O operation raises an exception after the signal handler returns. This is dependent on the underlying Unix system’s semantics regarding interrupted system calls.
You may need to catch I/O exceptions for the broken pipe and/or file write if you don't sys.exit from your handler.
The script is probably not killed silently; you just don't see the exception which python throws. I suggest to wrap the whole script in try-except and write any exception to a log file.
This way, you can see what really happens. The logging module is your friend.
You may be able to use the atexit module.
http://docs.python.org/library/atexit.html
From the documentation:
The atexit module defines a single function to register cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination.
Note: the functions registered via this module are not called when the program is killed by a signal, when a Python fatal internal error is detected, or when os._exit() is called.
This is an alternate interface to the functionality provided by the sys.exitfunc variable.
I keep a cache of transactions to flush (to persistent storage) on the event of a watermark or object finalization. Since __del__ is no longer guaranteed to be called on every object, is the appropriate approach to hook a similar function (or __del__ itself) into atexit.register (during initialization)?
If I'm not mistaken, this will cause the object to which the method is bound to hang around until program termination. This isn't likely to be a problem, but maybe there's a more elegant solution?
Note: I know using __del__ is non-ideal because it can cause uncatchable exceptions, but I can't think of another way to do this short of cascading finalize() calls all the way through my program. TIA!
If you have to handle ressources the prefered way is to have an explicit call to a close() or finalize() method. Have a look at the with statement to abstract that. In your case the weakref module might be an option. The cached object can be garbage collected by the system and have their __del__() method called or you finalize them if they are still alive.
I would say atexit or try and see if you can refactor the code into being able to be expressed using a with_statement which is in the __future__ in 2.5 and in 2.6 by default. 2.5 includes a module contextlib to simplify things a bit. I've done something like this when using Canonical's Storm ORM.
from future import with_statement
#contextlib.contextmanager
def start_transaction(db):
db.start()
yield
db.end()
with start_transaction(db) as transaction:
...
For a non-db case, you could just register the objects to be flushed with a global and then use something similar. The benefit of this approach is that it keeps things explicit.
If you don't need your object to be alive at the time you perform the flush, you could use weak references
This is similar to your proposed solution, but rather than using a real reference, store a list of weak references, with a callback function to perform the flush. This way, the references aren't going to keep those objects alive, and you won't run into any circular garbage problems with __del__ methods.
You can run through the list of weak references on termination to manually flush any still alive if this needs to be guaranteed done at a certain point.
Put the following in a file called destructor.py
import atexit
objects = []
def _destructor():
global objects
for obj in objects:
obj.destroy()
del objects
atexit.register(_destructor)
now use it this way:
import destructor
class MyObj(object):
def __init__(self):
destructor.objects.append(self)
# ... other init stuff
def destroy(self):
# clean up resources here
I think atexit is the way to go here.
Long story short, I have a substantial Python application that, among other things, does outcalls to "losetup", "mount", etc. on Linux. Essentially consuming system resources that must be released when complete.
If my application crashes, I want to ensure these system resources are properly released.
Does it make sense to do something like the following?
def main():
# TODO: main application entry point
pass
def cleanup():
# TODO: release system resources here
pass
if __name__ == "__main__":
try:
main()
except:
cleanup()
raise
Is this something that is typically done? Is there a better way? Perhaps the destructor in a singleton class?
I like top-level exception handlers in general (regardless of language). They're a great place to cleanup resources that may not be immediately related to resources consumed inside the method that throws the exception.
It's also a fantastic place to log those exceptions if you have such a framework in place. Top-level handlers will catch those bizarre exceptions you didn't plan on and let you correct them in the future, otherwise, you may never know about them at all.
Just be careful that your top-level handler doesn't throw exceptions!
A destructor (as in a __del__ method) is a bad idea, as these are not guaranteed to be called. The atexit module is a safer approach, although these will still not fire if the Python interpreter crashes (rather than the Python application), or if os._exit() is used, or the process is killed aggressively, or the machine reboots. (Of course, the last item isn't an issue in your case.) If your process is crash-prone (it uses fickle third-party extension modules, for instance) you may want to do the cleanup in a simple parent process for more isolation.
If you aren't really worried, use the atexit module.
Application wide handler is fine. They are great for logging. Just make sure that the application wide one is durable and is unlikely to crash itself.
if you use classes, you should free the resources they allocate in their destructors instead, of course. Use the try: on entire application just if you want to free resources that aren't already liberated by your classes' destructors.
And instead of using a catch-all except:, you should use the following block:
try:
main()
finally:
cleanup()
That will ensure cleanup in a more pythonic way.
That seems like a reasonable approach, and more straightforward and reliable than a destructor on a singleton class. You might also look at the "atexit" module. (Pronounced "at exit", not "a tex it" or something like that. I confused that for a long while.)
Consider writing a context manager and using the with statement.