I am currently writing a fuse using fuse-python. It's already doing what it should. However, after it's mounted for a few weeks, it's becoming noticeably slow. So I wanted to profile it. I know about a few point where it could be optimized. But these should not be the culprits.
However, fuse-python hangs in an infinite loop (see line 733 and 757 of the fuse source). If I run fuse in debug mode (using the -d switch), it will run in foreground. However, I cannot stop it with SIGINT nor with CTRL+C (which is anyway the same).
I tried to use the signal module to trap the signal in the main thread. But this does not work either. Interestingly, once I shoot the process down with SIGKILL, I see the KeyboardInterrupt on stdout. Also, after a SIGKILL, the signal handler is executed as expected.
This has repercussions on profiling. As the process never terminates normally, cProfile never gets the chance to save the stats file.
Any ideas?
Python installs a handler that raises KeyboardInterrupt on SIGINT. If a non-default signal handler is detected when fuse's main is called, it will not replace the handler with its own, which normally calls fuse_session_exit and cleans up. After you've called fuse's main, the KeyboardInterrupt is swallowed by CFUNCTYPE wrappers and you never see them.
Your options are to:
Send SIGQUIT by pressing Ctrl+\, or any other terminating signal other than SIGINT. However fuse will not exit cleanly.
Install the default signal handler to SIGINT before calling fuse's main, and restore the original when you're done.
old_handler =signal(SIGINT, SIG_DFL)
# call main
signal(SIGINT, old_handler)
I'd highly recommend you switch to an alternative binding also, fuse-python is terribly messy and difficult to work with. I've had a lot of luck with fusepy, and have submitted a few patches there.
When you're able to terminate your FUSE instance without using uncaught signals, the Python profiler will be able to save the stats as per normal.
Related
I have an event-driven application, written in python. After a while (usually >1 week) it appears to just stop responding to events. When this happens, I just ctrl-C and re-run and all is well-again. However, it's kind of annoying that this keeps happening and I have no idea what's causing it. Is there a way I can run my application that when this occurs and the application is no longer accepting connections, I can drop into a debugger and see what it's doing and why it's not taking connections?
I've used pdb before, but the way I've used it (if condition: pdb.set_trace()) doesn't really apply here, because I have no idea what it's doing in the code when it fails. My ideal situation would be instead of Ctrl-C maybe I hit Ctrl-somethingelse and that causes it to stop and drop into the debugger. Is such a thing easily done?
Triggering pdb in your case is probably not simple. However, whenever I need to debug such hangs, I inspect a "snapshot" of tracebacks of all the threads in the process, using the dumpstacks() function.
You can either use a timer to call it periodically and print the output to a log file, and refer to it when you notice the hanging, or harness some RPC mechanism (e.g. signals) to trigger the function call in your process on demand. I usually do the latter, because the processes in my system already listen to such RPC requests (using rpyc).
Using subprocess.Popen(), I'm launching a process that is supposed to take a long time. However, there is a chance that the process will fail shortly after it launches (producing a return code of 1). If that happens, I want to intercept the failure and present an explanatory message to the user. Is there a way to "listen" to the process and respond if it fails? I can't just use Popen.wait() because my python program has to keep running.
The hack I have in place right now is to time.sleep() my python program for .5 seconds (which should be enough time for the subprocess to to fail if it's going to do so). After the python program resumes, it polls the subprocess to determine if it has failed or not.
I imagine that a better solution might use threading and Popen.wait(), but I'm a relative beginner to python.
Edit:
The subprocess is a Java daemon that I'm launching. If another instance of the daemon is already running on the system, the Java subprocess will exit with a return code of 1, and I want to intercept the messy Java exception stack trace and present an understandable error message to the user.
Two approaches:
Call Popen.wait() on a thread as you suggested yourself, then call an error handler function if the exit code is non-zero. Make sure that the error handler is thread safe, preferably by dispatching the error message to the main thread if your application has an event loop.
Rewrite your application to use an event loop that already supports monitoring child processes, such as pyev. If you just want to monitor one subprocess, this is probably overkill.
KeyboardInterrupts in Idle work for me 90% of the time, but I was wondering why they don't always work. In Idle, if I do
import time
time.sleep(10)
and then attempt a KeyboardInterrupt using Ctrl+C, it does not interrupt the process until after sleeping for 10 seconds.
The same code and a KeyboardInterrupt via Ctrl+C works immediately in the shell.
A quick glance at the IDLE source reveals that KeyboardInterrupts have some special case handling: http://svn.python.org/view/python/tags/r267/Lib/idlelib/PyShell.py?annotate=88851
On top of that, code is actually executed in a separate process which the main IDLE gui process communicates with via RPC. You're going to get different behavior under that model - it's best to just test with the canonical interpreter (via command line, interactive, etc.)
============
Digging deeper...
The socket on the RPC server is managed in a secondary thread which is supposed to propagate a KeyboardInterrupt using a call to thread.interrupt_main() ( http://svn.python.org/view/python/tags/r267/Lib/idlelib/run.py?annotate=88851 ). Behavior is not as expected there... This posting hints that for some reason, interrupt_main doesn't provide the level of granularity that you would expect: http://bytes.com/topic/python/answers/38386-thread-interrupt_main-doesnt-seem-work
Async API functions in cPython are a little goofy (from my experience) due to how the interpreter loop is handled, so it doesn't surprise me. interrupt_main() calls PyErr_SetInterrupt() to asynchronously notify the interpreter to handle a SIGINT in the main thread. From http://docs.python.org/c-api/exceptions.html#PyErr_SetInterrupt:
This function simulates the effect of
a SIGINT signal arriving — the next
time PyErr_CheckSignals() is called,
KeyboardInterrupt will be raised
That would require the interpreter to go though whatever number of bytecode instructions before PyErr_CheckSignals() is called again - something that probably doesn't happen during a time.sleep(). I would venture to say that's a wart of simulating a SIGINT rather than actually signaling a SIGINT.
See This article:
I quote:
If you try to stop a CPython program
using Control-C, the interpreter
throws a KeyboardInterrupt exception.
It makes some sense, because the thread is asleep for 10 seconds and so exceptions cannot be thrown until the 10 seconds pass. However, ctrl + c always work in the shell because you are trying to stop a process, not throw a python KeyboardInterrupt exception.
Also, see this previously answered question.
I hope this helps!
In my program I have a bunch of threads running and I'm trying
to interrupt the main thread to get it to do something asynchronously.
So I set up a handler and send the main process a SIGUSR1 - see the code
below:
def SigUSR1Handler(signum, frame):
self._logger.debug('Received SIGUSR1')
return
signal.signal(signal.SIGUSR1, SigUSR1Handler)
[signal.signal(signal.SIGUSR1, signal.SIG_IGN)]
In the above case, all the threads and the main process stops - from a 'c'
point of view this was unexpected - I want the threads to continue as they
were before the signal. If I put the SIG_IGN in instead, everything continues
fine.
Can somebody tell me how to do this? Maybe I have to do something with the 'frame'
manually to get back to where it was..just a guess though
thanks in advance,
Thanks for your help on this.
To explain a bit more, I have thread instances writing string information to
a socket which is also output to a file. These threads run their own timers so they
independently write their outputs to the socket. When the program runs I also see
their output on stdout but it all stops as soon as I see the debug line from the signal.
I need the threads to constantly send this info but I need the main program to
take a command so it also starts doing something else (in parallel) for a while.
I thought I'd just be able to send a signal from the command line to trigger this.
Mixing signals and threads is always a little precarious. What you describe should not happen, however. Python only handles signals in the main thread. If the OS delivered the signal to another thread, that thread may be briefly interrupted (when it's performing, say, a systemcall) but it won't execute the signal handler. The main thread will be asked to execute the signalhandler at the next opportunity.
What are your threads (including the main thread) actually doing when you send the signal? How do you notice that they all 'stop'? Is it a brief pause (easily explained by the fact that the main thread will need to acquire the GIL before handling the signal) or does the process break down entirely?
I'll sort-of answer my own question:
In my first attempt at this I was using time.sleep(run_time) in the main
thread to control how long the threads ran until they were stopped. By adding
debug I could see that the sleep loop seemed to be exiting as soon as the
signal handler returned so everything was shutting down normally but early!
I've replaced the sleep with a while loop and that doesn't jump out after
the signal handler returns so my threads keep running. So it solves the
problem but I'm still a bit puzzled about sleep()'s behaviour.
You should probably use a threading.Condition variable instead of sending signals. Have your main thread check it every loop and perform its special operation if it's been set.
If you insist on using signals, you'll want to move to using subprocess instead of threads, as your problem is likely due to the GIL.
Watch this presentation by David Beazley.
http://blip.tv/file/2232410
It also explains some quirky behavior related to threads and signals (Python specific, not the general quirkiness of the subject :-) ).
http://pyprocessing.berlios.de/ Pyprocessing is a neat library that makes it easier to work with separate processes in Python.
I have a python cgi script that accepts user uploads (via sys.stdin.read).
After receiving the file (whether successfully or unsuccessfully), the script needs to do some cleanup. This works fine when upload finishes correctly, however if the user closes the client, the cgi script is silently killed on the server, and as a result no cleanup code gets executed. How can i force the script to always finish.
You can trap the exit signal with the signal module. Haven't tried this with mod_python though.
http://docs.python.org/library/signal.html
Note in the docs:
When a signal arrives during an I/O operation, it is possible that the I/O operation raises an exception after the signal handler returns. This is dependent on the underlying Unix system’s semantics regarding interrupted system calls.
You may need to catch I/O exceptions for the broken pipe and/or file write if you don't sys.exit from your handler.
The script is probably not killed silently; you just don't see the exception which python throws. I suggest to wrap the whole script in try-except and write any exception to a log file.
This way, you can see what really happens. The logging module is your friend.
You may be able to use the atexit module.
http://docs.python.org/library/atexit.html
From the documentation:
The atexit module defines a single function to register cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination.
Note: the functions registered via this module are not called when the program is killed by a signal, when a Python fatal internal error is detected, or when os._exit() is called.
This is an alternate interface to the functionality provided by the sys.exitfunc variable.