I was wandering if there was a way to perform an action before the program closes. I am running a program over a long time and I do want to be able to close it and have the data be saved in a text file or something but there is no way of me interfering with the while True loop I have running, and simply saving the data each loop would be highly ineffective.
So is there a way that I can save data, say a list, when I hit the x or destroy the program? I have been looking at the atexit module but have had no luck, except when I set the program to finish at a certain point.
def saveFile(list):
print "Saving List"
with open("file.txt", "a") as test_file:
test_file.write(str(list[-1]))
atexit.register(saveFile(list))
That is my whole atexit part of the code and like I said, it runs fine when I set it to close through the while loop.
Is this possible, to save something when the application is terminated?
Your atexit usage is wrong. It expects a function and its arguments, but you're just calling your function right away and passing the result to atexit.register(). Try:
atexit.register(saveFile, list)
Be aware that this uses the list reference as it exists at the time you call atexit.register(), so if you assign to list afterwards, those changes will not be picked up. Modifying the list itself without reassigning should be fine, though.
You could use the handle_exit context manager from this ActiveState recipe:
http://code.activestate.com/recipes/577997-handle-exit-context-manager/
It handles SystemExit, KeyboardInterrupt, SIGINT, and SIGTERM, with a simple interface:
def cleanup():
print 'do some cleanup here'
def main():
print 'do something'
if __name__ == '__main__':
with handle_exit(cleanup):
main()
There's nothing you can in reaction to a SIGKILL. It kills your process immediately, without any allowed cleanup.
Catch the SystemExit exception at the top of your application, then rethrow it.
There are a a couple of approaches to this. As some have commented you could used signal handling ... your [Ctrl]+[C] from the terminal where this is running in the foreground is dispatching a SIGHUP signal to your process (from the terminal's drivers).
Another approach would be to use a non-blocking os.read() on sys.stdin.fileno such that you're polling your keyboard one during every loop to see if an "exit" keystroke or sequence has been entered.
A similarly non-blocking polling approach can be implemented using the select module's functionality. I've see that used with the termios and tty modules. (Seems inelegant that it needs all those to save, set changes to, and restore the terminal settings, and I've also seen some examples using os and fcntl; and I'm not sure when or why one would prefer one over the other if os.isatty(sys.stdin.fileno())).
Yet another approach would be to use the curses module with window.nodelay() or window.timeout() to set your desired input behavior and then either window.getch() or window.getkey() to poll for any input.
Related
I have a multithreaded program written in Python where I have a number of things happening at the same time:
reading raw data from an external source
organizing such data into different lists (parsing)
saving the post-parsed data into mass storage
some threads being used to flush some buffers from time to time and other optimizations
displaying parts of such data for real-time monitoring, using Curses.
The program is latency-sensitive, so I really need this to be multithreaded.
I got the curses thread to display correctly what I want to.
The problem is that while I had everything working without the curses thread, I had a "killswitch" in the main() function that terminated all activity at the press of a key.
I have this global variable called "killThreads" that goes into all functions who are called as threads, and all these functions only work as:
def oneThread():
while (not killThreads):
doStuff()
...
And then the main function defines the killThread as False, initializes all threads and turns the killThread as True after a raw_input():
killThreads=False
thisThread=threading.Thread(target=oneThread)
otherThread=threading.Thread(target=twoThread)
thisThread.setDaemon(True)
otherThread.setDaemon(True)
thisThread.start()
otherThread.start()
raw_input('Press to end the program')
killThreads=True
Everything ran fine until I ran a thread with the Curses module to display data.
It seems that while the Curses thread is on, it takes over all input commands. I tried to use getch() with no success. All I could do to keep everything running was to establish a timer within the Curses function:
def displayData():
screen=curses.initscr()
screen.nodelay(1)
timeKill=0
while (timeKill<80):
#stuff is drawn#
time.sleep(0.25)
timeKill+=1
Could anyone tell me how to go over Curses and get my keyboard input to "reach" the main function and kill all threads? Or do I always have to input to Curses and then make the Curses function alter the killThreads variable? If so, how do I do it (or where do I find the documentation for that)?
Thank you so much for your help.
Nice to meet you.
I'm trying to accomplish the same today. Look at this solution:
killThreads=False
thisThread=threading.Thread(target=oneThread)
otherThread=threading.Thread(target=twoThread)
thisThread.setDaemon(True)
otherThread.setDaemon(True)
thisThread.start()
otherThread.start()
raw_input('Press "q" to end the program')
key = ''
while key != ord('q'):
key = screen.getch()
killThreads=True
curses.nocbreak(); screen.keypad(0); curses.echo()
curses.endwin()
See, while will looping very fastly and waiting for q button be pressed, before switch your var killThreads to True.
It is pretty common practice. However, this while loop making thousands of idle loops in second, may be there can be more elegant way or better to embed into this while loop time.sleep(0.1) at least.
I am currently running a program, which i expect to go on for an hour or two. I need to break out of the loop right now, so that rest of the program continues.
This is a part of the code:
from nltk.corpus import brown
from nltk import word_tokenize, sent_tokenize
from operator import itemgetter
sentences = []
try:
for i in range(0,55000):
try:
sentences.append(brown.sents()[i])
print i
except:
break
except:
pass
the loop is currently around 30,000. I want to exit and continue with the code (not shown here). Please suggest me how to such that, the program doesn't break exit completely. (Not like keyboard interrupt)
Since it is already running, you can't modify the code. Unless you invoked it under pdb, you can't break into the Python debugger to alter the condition to leave the loop and continue with the rest of the program. So none of the normal avenues are open to you.
There is one outside solution, which requires intimate knowledge of the Python interpreter and runtime. You can attach the gdb debugger to the Python process (or VisualStudio if you are on Windows). Then when you break in, examine the stack trace of the main thread. You will see a whole series of nested PyEval_* calls and so on. If you can figure out where the loop is in the stack trace, then identify the loop. Then you will need to find the counter variable (an integer wrapped in a PyObject) and set it to a large enough value to trigger the end of the loop, then let the process continue. Not for the faint of heart! Some more info is here:
Tracing the Python stack in GDB
Realistically, you just need to decide if you either leave it alone to finish, or kill it and restart.
It's probably easiest to simply kill the process, modify your code so that the loop is interruptible (as #fedorSmirnov suggests) with the KeyboardInterrupt exception, then start again. You will lose the processing time you have invested already, but consider it a sunken cost.
There's lots of useful information here on how to add support to your program for debugging the running process:
Showing the stack trace from a running Python application
I think you could also put the for loop in a try block and catch the keyBoardInterrupt exception by proceeding with the rest of the program. With this approach, you should be able to break out of the loop by hitting ctrl + C while staying inside your program. The code would look similar to this:
try:
# your for loop
except KeyboardInterrupt:
print "interrupted"
# rest of your program
You can save the data with pickle before the break command. Next time load the data and continue the loop.
I have a python script I've written which uses atexit.register() to run a function to persist a list of dictionaries when the program exits. However, this code is also running when the script exits due to a crash or runtime error. Usually, this results in the data becoming corrupted.
Is there any way to block it from running when the program exits abnormally?
EDIT: To clarify, this involves a program using flask, and I'm trying to prevent the data persistence code from running on an exit that results from an error being raised.
You don't want to use atexit with Flask. You want to use Flask signals. It sounds like you are specifically looking for the request_finished signal.
from flask import request_finished
def request_finished_handler(sender, response, **extra):
sender.logger.debug('Request context is about to close down. '
'Response: %s', response)
# do some fancy storage stuff.
request_finished.connect(request_finished_handler, app)
The benefit of request_finished is that it only fires after a successful response. That means that so long as there isn't an error in another signal, you should be good.
One way: at global level in main program:
abormal_termination = False
def your_cleanup_function():
# Add next two lines at the top
if abnormal_termination:
return
# ...
# At end of main program:
try:
# your original code goes here
except Exception: # replace according to what *you* consider "abnormal"
abnormal_termination = True # stop atexit handler
Not pretty, but straightforward ;-)
How do you register a function with all the correct handlers etc to be called when the Python script is exitted (successfully or not)?
I have tried:
#atexit.register
def finalise():
'''
Function handles program close
'''
print("Tidying up...")
...
print("Closing")
...but this does not get called when the user closes the command prompt window for example (because #atexit.register decorated functions do not get called when the exitcode is non zero)
I am looking for a way of guaranteeing finalise() is called on program exit, regardless of errors.
For context, my Python program is a continually looping service program that aims to run all the time.
Thanks in advance
I don't think it can be done in pure Python. From documentation:
Note: the functions registered via this module are not called when the
program is killed by a signal not handled by Python, when a Python
fatal internal error is detected, or when os._exit() is called.
I think you may find this useful: How to capture a command prompt window close event in python
Have you tried to just catch all kinds of exceptions in your main function or code block?
For context, my Python program is a continually looping service program that aims to run all the time.
This is very hard to get right (see How do you create a daemon in Python?). You should use a library like http://pypi.python.org/pypi/python-daemon/ instead.
This one should work
works both on Ctrl-C and when the assertion fails. Maybe you can use a similar construct and pack it as a decorator, or whatever.
def main():
print raw_input('> ')
# do all your stuff here
if __name__ == '__main__':
try:
main()
finally:
print 'Bye!'
atexit.register(func)
func = lambda x: x
Use http://docs.python.org/2/library/atexit.html
I'm writing a program that adds normal UNIX accounts (i.e. modifying /etc/passwd, /etc/group, and /etc/shadow) according to our corp's policy. It also does some slightly fancy stuff like sending an email to the user.
I've got all the code working, but there are three pieces of code that are very critical, which update the three files above. The code is already fairly robust because it locks those files (ex. /etc/passwd.lock), writes to to a temporary files (ex. /etc/passwd.tmp), and then, overwrites the original file with the temporary. I'm fairly pleased that it won't interefere with other running versions of my program or the system useradd, usermod, passwd, etc. programs.
The thing that I'm most worried about is a stray ctrl+c, ctrl+d, or kill command in the middle of these sections. This has led me to the signal module, which seems to do precisely what I want: ignore certain signals during the "critical" region.
I'm using an older version of Python, which doesn't have signal.SIG_IGN, so I have an awesome "pass" function:
def passer(*a):
pass
The problem that I'm seeing is that signal handlers don't work the way that I expect.
Given the following test code:
def passer(a=None, b=None):
pass
def signalhander(enable):
signallist = (signal.SIGINT, signal.SIGQUIT, signal.SIGABRT, signal.SIGPIPE, signal.SIGALRM, signal.SIGTERM, signal.SIGKILL)
if enable:
for i in signallist:
signal.signal(i, passer)
else:
for i in signallist:
signal.signal(i, abort)
return
def abort(a=None, b=None):
sys.exit('\nAccount was not created.\n')
return
signalhander(True)
print('Enabled')
time.sleep(10) # ^C during this sleep
The problem with this code is that a ^C (SIGINT) during the time.sleep(10) call causes that function to stop, and then, my signal handler takes over as desired. However, that doesn't solve my "critical" region problem above because I can't tolerate whatever statement encounters the signal to fail.
I need some sort of signal handler that will just completely ignore SIGINT and SIGQUIT.
The Fedora/RH command "yum" is written is Python and does basically exactly what I want. If you do a ^C while it's installing anything, it will print a message like "Press ^C within two seconds to force kill." Otherwise, the ^C is ignored. I don't really care about the two second warning since my program completes in a fraction of a second.
Could someone help me implement a signal handler for CPython 2.3 that doesn't cause the current statement/function to cancel before the signal is ignored?
As always, thanks in advance.
Edit: After S.Lott's answer, I've decided to abandon the signal module.
I'm just going to go back to try: except: blocks. Looking at my code there are two things that happen for each critical region that cannot be aborted: overwriting file with file.tmp and removing the lock once finished (or other tools will be unable to modify the file, until it is manually removed). I've put each of those in their own function inside a try: block, and the except: simply calls the function again. That way the function will just re-call itself in the event of KeyBoardInterrupt or EOFError, until the critical code is completed.
I don't think that I can get into too much trouble since I'm only catching user provided exit commands, and even then, only for two to three lines of code. Theoretically, if those exceptions could be raised fast enough, I suppose I could get the "maximum reccurrsion depth exceded" error, but that would seem far out.
Any other concerns?
Pesudo-code:
def criticalRemoveLock(file):
try:
if os.path.isFile(file):
os.remove(file)
else:
return True
except (KeyboardInterrupt, EOFError):
return criticalRemoveLock(file)
def criticalOverwrite(tmp, file):
try:
if os.path.isFile(tmp):
shutil.copy2(tmp, file)
os.remove(tmp)
else:
return True
except (KeyboardInterrupt, EOFError):
return criticalOverwrite(tmp, file)
There is no real way to make your script really save. Of course you can ignore signals and catch a keyboard interrupt using try: except: but it is up to your application to be idempotent against such interrupts and it must be able to resume operations after dealing with an interrupt at some kind of savepoint.
The only thing that you can really to is to work on temporary files (and not original files) and move them after doing the work into the final destination. I think such file operations are supposed to be "atomic" from the filesystem prospective. Otherwise in case of an interrupt: restart your processing from start with clean data.