I want to do something when a process is crash or killed in a python script.
However I can't find anyway to know when a process is stop by lldb.
I've tried to catch a SIGKILL signal but no use.
import lldb
import signal
def debug(sig, frame):
print "stop!\n"
def listen():
signal.signal(signal.SIGKILL, debug) # Register handler
I've find that we can use this to handle a breakpoint hit, but it can't deal with my situation.
def breakpoint_function_wrapper(frame, bp_loc, dict):
Anyone has some solutions?
There's a little sample program in the lldb python examples that shows how to handle process events using the lldb library:
http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/process_events.py
That might help get you started.
Related
I have a python process, spawning multiple background processes. I am currently seeing one/many of the background process get stuck in deadloop or they are becoming irresponsive.
I want to attach a debugger to the background process. So, I can figure out what is going wrong. I have registered a signal handler for my background process (SIGUSR1) which I sets pdb trace.
I am sending the signal from another console to the background process, whenever it hangs. However, I don't see any terminal which can help me debug the code.
Am I doing something wrong? or is there a better way to attach a debugger to background python process.
I am running on MAC and so using gdb is not straight forward.
def installHandlers():
signal.signal(signal.SIGUSR1,debugHandle)
def debugHandle(sig,frame):
global processLog
processLog.info("got the SIGUSR1")
import pdb
pdb.Pdb().set_trace(frame)
-Thanks
I think it happens because Python closes stdin of the parent process in the multiprocessing.Process._bootstrap(). Therefore pdb.set_trace() doesn't work in child processes.
It usually fails with error. Maybe you don't see error because you redirect stdout somewhere?
There is a clone of pdb, imaginatively called pdb-clone, which allows for debugging of background processes.
You simply add from pdb_clone import pdbhandler; pdbhandler.register() to the code for the main process, and then you can start pdb with pdb-attach --kill --pid PID.
I want to submit my long running Python job using ampersand. I'm going to kick this process off from an interactive Python program by using a sub process call it.
How would I keep track of the submitted job programmatically in case I want to end the job from a menu option?
Example of interactive program:
Main Menu
1. Submit long running job &
2. End long running job
If you're using python's subprocess module, you don't really need to background it again with & do you? You can just keep your Popen object around to track the job, and it will run while the other python process continues.
If your "outer" python process is going to terminate what sort of track do you need to keep? Would pgrep/pkill be suitable? Alternately, you could have the long running job log its PID, often under /var/run somewhere, and use that to track if the process is still alive and/or signal it.
You could use Unix signals. Here we capture SIGUSR1 to tell the process to communicate some info to STDOUT.
#!/usr/bin/env python
import signal
import sys
def signal_handler(signal, frame):
print('Caught SIGUSR1!')
print("Current job status is " + get_job_status())
signal.signal(signal.SIGUSR1, signal_handler)
and then from the shell
kill <pid> --signal SIGUSR1
I have a server script that I need to be able to shutdown cleanly. While testing the usual try..except statements I realized that Ctrl-C didn't work the usual way. Normally I'd wrap long running tasks like this
try:
...
except KeyboardInterrupt:
#close the script cleanly here
so the task could be shutdown cleanly on Ctrl-C. I have never ran into any problems with this before, but somehow when I hit Ctrl-C when this particular script is running the script just exits without catching the Ctrl-C.
The initial version was implemented using Process from multiprocessing. I rewrote the script using Thread from threading, but same issue there. I have used threading many times before, but I am new to the multiprocessing library. Either way, I have never experienced this Ctrl-C behavior before.
Normally I have always implemented sentinels etc to close down Queues and Thread instances in an orderly fashion, but this script just exits without any response.
Last, I tried overriding signal.SIGINT as well like this
def handler(signal, frame):
print 'Ctrl+C'
signal.signal(signal.SIGINT, handler)
...
Here Ctrl+C was actually caught, but the handler doesn't execute, it never prints anything.
Besides the threading / multiprocessing aspect, parts of the script contains C++ SWIG objects. I don't know if that has anything to do with it. I am running Python 2.7.2 on OS X Lion.
So, a few questions:
What's going on here?
How can I debug this?
What do I need to learn in order to understand the root cause?
PLEASE NOTE: The internals of the script is proprietary so I can't give code examples. I am however very willing to receive pointers so I could debug this myself. I am experienced enough to be able to figure it out if someone could point me in the right direction.
EDIT: I started commenting out imports etc to see what caused the weird behavior, and I narrowed it down to an import of a C++ SWIG library. Any ideas why importing a C++ SWIG library 'steals' Ctrl-C? I am not the author of the guilty library however and my SWIG experience is limited so don't really know where to start...
EDIT 2: I just tried the same script on a windows machine, and in Windows 7 the Ctrl-C is caught as expected. I'm not really going to bother with the OS X part, the script will be run in an Windows environment anyway.
This might have to do with the way Python manages threads, signals and C calls.
In short - Ctrl-C cannot interrupt C calls, since the implementation requires that a python thread will handle the signal, and not just any thread, but the main thread (often blocked, waiting for other threads).
In fact, long operations can block everything.
Consider this:
>>> nums = xrange(100000000)
>>> -1 in nums
False (after ~ 6.6 seconds)
>>>
Now, Try hitting Ctrl-C (uninterruptible!)
>>> nums = xrange(100000000)
>>> -1 in nums
^C^C^C (nothing happens, long pause)
...
KeyboardInterrupt
>>>
The reason Ctrl-C doesn't work with threaded programs is that the main thread is often blocked on an uninterruptible thread-join or lock (e.g, any 'wait', 'join' or just a plain empty 'main' thread, which in the background causes python to 'join' on any spawned threads).
Try to insert a simple
while True:
time.sleep(1)
in your main thread.
If you have a long running C function, do signal handling in C-level (May the Force be with you!).
This is largely based on David Beazley's video on the subject.
It exits because something else is likely catching the KeyboardInterupt and then raising some other exception, or simply returning None. You should still get a traceback to help debug. You need to capture the stderr output or run your script with the -i commandline option so you can see traceback. Also, add another except block to catch all other exceptions.
If you suspect the C++ function call to be catching the CTRL+C try catching it's output. If the C function is not returning anything then there isn't much you can do except ask the author to add some exception handling, return codes, etc.
try:
#Doing something proprietary ...
#catch the function call output
result = yourCFuncCall()
#raise an exception if it's not what you expected
if result is None:
raise ValueError('Unexpected Result')
except KeyboardInterupt:
print('Must be a CTRL+C')
return
except:
print('Unhandled Exception')
raise
how about atexit?
http://docs.python.org/library/atexit.html#module-atexit
What's the best way to monitor a python daemon to determine the cause of it quitting unexpectedly? Is strace my best option or is there something Python specific that does the job?
I would generally start by adding logging to it. At a minimum, have whatever is launching it capture stdout/stderr so that any stack traces are saved. Examine your except blocks to make sure you're not capturing exceptions silently.
You can use pdb:
python -m pdb myscript.py
Running your program like this will cause it to enter post-mortem debugging if it exits abnormally. If you have an idea where the problem is you can also use import pdb; pdb.set_trace() at the point you want to start debugging. Also logging profusely helps.
As the answer above, try to add Logging, however be carefull if you are using python-daemon module it will not work with Logging module when logging to a file, so you should make the logging manually to a file.
Also, make your daemon restarts after it has failed by running it inside a loop and catch exceptions inside the loop.
Example:
while True:
try:
log_to_file("starting daemon")
run_daemon()
log_to_file("daemon stopped unexpectedly")
sleep(1)
except Exception as err:
log_to_file(err)
I am looking for a simple Python webserver that is easy to kill from within code. Right now, I'm playing with Bottle, but I can't find any way at all to kill it in code. If you know how to kill Bottle (in code, no Ctrl+C) that would be super, but I'll take anything that's Python, simple, and killable.
We use this.
import os
os._exit(3)
To crash in a 'controlled' way.
If you want to kill a process from Python, on a Unix-like platform, you can send signals equivalent to Ctrl-C at the console using Pythons os module e.g.
# Get this processes PID
pid_of_process = os.getpid()
# Send the interrupt signal to this process
os.kill(pid_of_process, signal.SIGINT)
Raise exeption and handle it in main or use sys.exit
Try putting
import sys
at the top and the command
sys.exit(0)
In the code that handles the "kill request".