I'm working on a python project were I want the same behavior as in C for my threads. In C when the main thread exit, it kills all other threads.
The project contains a TCP error server that it used to get logs from other threads and other software .The TCP link is simplex.
Some errors must involve the end of the whole program.
For external software I can kill them using their PID.
For other threads I've tried sys._exit(), sometimes it works, and sometimes some threads remain.
If my other threads were looping I could use a semaphore or something like that, but it is only one iteration of a linear process.
I've thought about the design pattern Producer/Consumer or add a lot of lock.acquire()/lock.release() but I think it will add more complexity and it imply to break the linear thread.
I've had a look to other Stackoverflow question I've found those solutions:
Use sys._exit() but its success rate is not 100%.
Convert my threads into subprocess to kill them easily, but in my case I can't.
I'm looking for a solution, a design pattern or something else to solve it.
PS: I'm a C lover and each time I deal with Python I think to solutions as simple as to call exit() to kill all my threads.
If you make your worker threads daemon threads, they will die when all your non-daemon threads (e.g. the main thread) have exited.
http://docs.python.org/library/threading.html#threading.Thread.daemon
Thread daemon status isDaemon() is False, set it True by setDaemon(True)
Another solution :
To make the thread stop on Keyboard Interrupt signal (ctrl+c) you can catch the exception "KeyboardInterrup" and cleanup before exiting. Like this:
try:
start_thread() #And the rest of your main
except (KeyboardInterrupt, SystemExit):
cleanup_stop_thread();
sys.exit()
Related
I have read most of the similar questions in stackoverflow, but none see to solve my problem. I use ctypes to call a function from dll file. Therefore, I can't edit the source codes of the dll file to add any "end looping" conditions. Also, this function may last long (like some printing command). I need to design a "halt" command in case that something of emergency happens while printing is processed. The only way I can do is to kill the thread.
It is never good to forcibly kill a thread. Your program should be designed to cleanly exit from threads.
You can mark it as "daemon" before starting it. If you exit the main thread it will not wait on daemonized threads.
Terminating a thread can still be done in two ways. You can asynchronously raise a Python exception in a thread, via https://docs.python.org/2/c-api/init.html#c.PyThreadState_SetAsyncExc (as stated, this requires building a C module or using ctypes to make it work). The other approach on Windows is to call the Windows API TerminateThread():
TerminateThread is used to cause a thread to exit. When this occurs,
the target thread has no chance to execute any user-mode code. DLLs
attached to the thread are not notified that the thread is
terminating. The system frees the thread's initial stack.
[...]
TerminateThread is a dangerous function that should only be used in
the most extreme cases. You should call TerminateThread only if you
know exactly what the target thread is doing, and you control all of
the code that the target thread could possibly be running at the time
of the termination. For example, TerminateThread can result in the
following problems: ...
I think this should also be doable using ctypes.
You cannot safely terminate a thread without its cooperation. Threads are not isolated within a process, so unsafely terminating a thread contaminates the process. Please, don't go down this road.
If you need this kind of isolation, you need a process. You can safely terminate a process without its cooperation, though it may leave system objects (such as files) that the process was working on in an intermediate state. In your case, that may mean a print job half-done and a page halfway in the printer. Or it may mean temporary files that don't get removed.
I've been reading python's threadpool module's code.
It manipulates threads in this way : All workerThreads are created as daemonic thread. And it also have a dismiss mechanism that you can safely quit the worker thread by setting event, after all the job's done the dismissed threads will be joined in the main thread.
The python doc says that if worker threads were set daemonic, they will quit when main thread terminates. But it might be an ugly implementation, a better way is to make them non-daemonic and stop them with event.
Here is my question: Is it a good design to use both of the quit strategies? Is it better to set the threads non-daemonic and join them all before the main thread terminates?
In looking at this particular threadpool module, it appears to be designed to work either by allowing you to quit summarily, or waiting for the threads to complete. You would choose one or the other depending on how you want to handle requests currently in process:
If you don't care about whether threads die in the middle of processing requests, just let the program exit, and the daemon threads will be taken care of.
On the other hand, if you want to make sure a thread exits only between fully processing requests, either use dismissWorkers with do_join=True, or use dismissWorkers followed by joinAllDismissedWorkers.
That choice would vary depending on what you're processing and how. Note that the sample code that comes in the main routine does some of one and some of the other, which is probably not what you'd want to do in a real situation – the sample code is just designed to demonstrate capabilities.
You could argue that it's bad form to create daemon threads when you do care about how/when they exit, and it wouldn't be hard to fix the library so that daemon is an option for your worker threads when they are created, not a necessity. Currently, however, the module picks a default that favors ease of use over consistency.
I had some annoyances with spawning subprocesses, like getting correct output and so on. A wrapper library, envoy, solved all of my problems with an easy-to-use interface that gets rid of most problems.
Using thread, I sometimes struggle with hanging processes that does not end, external programs launched within threads that I can't reach anymore and so on.
Is there any "threading for dummies" python library out there? Thanks
Is there any "threading for dummies" python library out there?
No, there is not. threading is pretty simple to use in simple cases. You want to use it to introduce concurrency in your program. This means you want to use it whenever you want two or more actions to happen simultaneously, i.e. at the same time.
This is how you can let Peter build a house and let Igor drive to Moskow at the same time:
from threading import Thread
import time
def drive_bus():
time.sleep(1)
print "Igor: I'm Igor and I'm driving to... Moskow!"
time.sleep(9)
print "Igor: Yei, Moskow!"
def build_house():
print "Peter: Let's start building a large house..."
time.sleep(10.1)
print "Peter: Urks, we have no tools :-("
threads = [Thread(target=drive_bus), Thread(target=build_house)]
for t in threads:
t.start()
for t in threads:
t.join()
Isn't that simple? Define your function to be run in another thread. Create a threading.Thread instance with that function as target. Nothing happend so far, until you invoke start. It fires off the thread and immediately returns.
Before letting your main thread exit, you should wait for all the threads you have spawned to finish. This is what t.join() does: it blocks and waits for the thread t to finish. Only then it returns.
I would recommend reading more about the actual Python library - it is simple enough. Your problem with hanging threads, provided it prevents your application from exiting, may be solved by using daemon threads.
What kind of task are you trying to achieve? If you are trying to run a task in parallel without actual use of the custom threading, you may find the package multiprocessing useful. Furthermore, there is an interesting piece of information on the python wiki about parallel processing.
Could you elaborate a bit more on the task please?
I'm making a "python debug mapper" that shows a 'snapshot' of current python execution
Currently, I need to know a way to pause every other threads so that the 'capture' won't happen while other threads are running.
Are there any way to do:
PauseOtherThreads();
ResumeOtherThreads();
Thanks.
p.s: should I make any modifications to get the code working with Celery and Django?
Depending on if you want just to trace one thread while other threads are running or if you want to stop other threads I can think to two solutions. If other threads must run without tracing just make your trace command check the current thread id first and only do the trace operation if the thread is the one you are interested in:
def dotrace():
if tracing and threading.current_thread() == the_traced_thread:
... do the tracing ...
If instead other threads must stop while one is being traced you can make your tracing operation work as an halt for other threads adding something like:
def dotrace():
while tracing and threading.current_thread() != the_traced_thread:
time.sleep(0.01)
if tracing and threading.current_thread() == the_traced_thread:
... do the tracing ...
Of course only the trace operations will work as an halt in the last case, so other threads may keep running until they finish or they do anything that is traced.
Basically you will only stop other threads that you are monitoring and not all other threads. I'd say this is good because increases the probability that the program will still remain functional (some of the libraries and frameworks you use may need other threads to run for the thread being traced to actually work) but of course YMMV.
You could using sys.setcheckinterval(somebignumber)
setcheckinterval(...)
setcheckinterval(n)
Tell the Python interpreter to check for asynchronous events every
n instructions. This also affects how often thread switches occur.
In my program I have a bunch of threads running and I'm trying
to interrupt the main thread to get it to do something asynchronously.
So I set up a handler and send the main process a SIGUSR1 - see the code
below:
def SigUSR1Handler(signum, frame):
self._logger.debug('Received SIGUSR1')
return
signal.signal(signal.SIGUSR1, SigUSR1Handler)
[signal.signal(signal.SIGUSR1, signal.SIG_IGN)]
In the above case, all the threads and the main process stops - from a 'c'
point of view this was unexpected - I want the threads to continue as they
were before the signal. If I put the SIG_IGN in instead, everything continues
fine.
Can somebody tell me how to do this? Maybe I have to do something with the 'frame'
manually to get back to where it was..just a guess though
thanks in advance,
Thanks for your help on this.
To explain a bit more, I have thread instances writing string information to
a socket which is also output to a file. These threads run their own timers so they
independently write their outputs to the socket. When the program runs I also see
their output on stdout but it all stops as soon as I see the debug line from the signal.
I need the threads to constantly send this info but I need the main program to
take a command so it also starts doing something else (in parallel) for a while.
I thought I'd just be able to send a signal from the command line to trigger this.
Mixing signals and threads is always a little precarious. What you describe should not happen, however. Python only handles signals in the main thread. If the OS delivered the signal to another thread, that thread may be briefly interrupted (when it's performing, say, a systemcall) but it won't execute the signal handler. The main thread will be asked to execute the signalhandler at the next opportunity.
What are your threads (including the main thread) actually doing when you send the signal? How do you notice that they all 'stop'? Is it a brief pause (easily explained by the fact that the main thread will need to acquire the GIL before handling the signal) or does the process break down entirely?
I'll sort-of answer my own question:
In my first attempt at this I was using time.sleep(run_time) in the main
thread to control how long the threads ran until they were stopped. By adding
debug I could see that the sleep loop seemed to be exiting as soon as the
signal handler returned so everything was shutting down normally but early!
I've replaced the sleep with a while loop and that doesn't jump out after
the signal handler returns so my threads keep running. So it solves the
problem but I'm still a bit puzzled about sleep()'s behaviour.
You should probably use a threading.Condition variable instead of sending signals. Have your main thread check it every loop and perform its special operation if it's been set.
If you insist on using signals, you'll want to move to using subprocess instead of threads, as your problem is likely due to the GIL.
Watch this presentation by David Beazley.
http://blip.tv/file/2232410
It also explains some quirky behavior related to threads and signals (Python specific, not the general quirkiness of the subject :-) ).
http://pyprocessing.berlios.de/ Pyprocessing is a neat library that makes it easier to work with separate processes in Python.