I am trying to utilize threading.Thread with Celery to work like a daemon. On a larger scope, the threads will poll hardware sensors as a part of a web UI-powered thermostat, but narrowed-down, this is the bit I'm stuck on:
from celery import Celery
from celery.signals import worker_init
from celery.signals import worker_process_shutdown
from threading import Thread
from threading import Event
from time import sleep
class ThisClass(object):
def __init__(self):
self.shutdown = Event()
self.thread = Thread(target=self.BackgroundMethod)
def Start(self):
self.thread.start()
def Stop(self):
self.shutdown.set()
def BackgroundMethod(self):
while not self.shutdown.is_set():
print("Hello, world!")
sleep(1)
this_class = ThisClass()
celery_app = Celery("tasks", broker="amqp://guest#localhost//")
#worker_init.connect
def WorkerReady(**kwargs):
this_class.Start()
#worker_process_shutdown.connect
def StopPollingSensors(**kwargs):
this_class.Stop()
This Celery script is supposed to create an instance of ThisClass as this_class, and run this_class.Start() when Celery starts. When Celery is shutting down, it is supposed to call this_class.Stop(), which gracefully exits the Thread in ThisClass and Celery cleanly exits.
However, when I hit Ctrl-C in Celery to signal a SIGINT, this_class's thread continues to run and Celery does not exit, even after multiple SIGTERMs are issued. What confuses me is if I slip a print statement in ThisClass.Stop, I see it. Furthermore, if I add sleep(5); this_class.Stop() after this_class.Start(), the thread starts and stops as expected, and Celery will exit normally when issued a SIGINT.
How am I supposed to terminate threading.Thread instances in a Celery-based script?
Consider creating the Event object externally, and passing it to a Thread subclass. The thread loops on the event object. When an external function calls Event.set(), the thread loop exits out of the while loop cleanly.
import sys, time
from threading import Event, Thread, Timer
class ThisClass2(Thread):
def __init__(self, event):
super(ThisClass2,self).__init__()
self.event = event
def run(self):
print 'thread start'
while not self.event.wait(timeout=1.0):
print("thread: Hello, world!")
print 'thread done'
event2 = Event()
x = ThisClass2(event2)
x.start()
print 'okay'
time.sleep(1)
print 'signaling event'
event2.set()
print 'waiting'
x.join()
Example output:
thread start
okay
thread: Hello, world!
signaling event
waiting
thread done
Related
I have this code:
import threading
def printit():
print ("Hello, World!")
threading.Timer(1.0, printit).start()
threading.Timer(1.0, printit).start()
I am trying to have "Hello, World!" printed every second, however when I run the code nothing happens, the process is just kept alive.
I have read posts where exactly this code worked for people.
I am very confused by how hard it is to set a proper interval in python, since I'm used to JavaScript. I feel like I'm missing something.
Help is appreciated.
I don't see any issue with your current approach. It is working for me me in both Python 2.7 and 3.4.5.
import threading
def printit():
print ("Hello, World!")
# threading.Timer(1.0, printit).start()
# ^ why you need this? However it works with it too
threading.Timer(1.0, printit).start()
which prints:
Hello, World!
Hello, World!
But I'll suggest to start the thread as:
thread = threading.Timer(1.0, printit)
thread.start()
So that you can stop the thread using:
thread.cancel()
Without having the object to Timer class, you will have to shut your interpreter in order to stop the thread.
Alternate Approach:
Personally I prefer to write a timer thread by extending Thread class as:
from threading import Thread, Event
class MyThread(Thread):
def __init__(self, event):
Thread.__init__(self)
self.stopped = event
def run(self):
while not self.stopped.wait(0.5):
print("Thread is running..")
Then start thread with object of Event class as:
my_event = Event()
thread = MyThread(my_event)
thread.start()
You'll start seeing the below output in the screen:
Thread is running..
Thread is running..
Thread is running..
Thread is running..
To stop the thread, execute:
my_event.set()
This provides more flexibility in modifying the changes for the future.
What might be an issue is that you are creating a new thread each time you are running printit.
A better way may be just to create one thread that does whatever you want it to do and then you send an event to stop it when it is finished for some reason:
from threading import Thread,Event
from time import sleep
def threaded_function(evt):
while not evt.is_set():
print "running"
sleep(1)
if __name__ == "__main__":
e=Event()
thread = Thread(target = threaded_function, args = (e, ))
thread.start()
sleep(5)
e.set() # tells the thread to exit
thread.join()
print "thread finished...exiting"
I run it in python 3.6.It works ok as you expected .
I have used Python 3.6.0.
And I have used _thread and time package.
import time
import _thread as t
def a(nothing=0):
print('hi',nothing)
time.sleep(1)
t.start_new_thread(a,(nothing+1,))
t.start_new_thread(a,(1,))#first argument function name and second argument is tuple as a parameterlist.
o/p will be like
hi 1
hi 2
hi 3
....
I need to run a gstreamer pipeline to perform video streaming. The GStreamer pipeline requires a GObject.MainLoop object which has a run() method that does not terminate until quit() is called.
For this I create a process (P2) from my main application process (P1), which runs the GObject.MainLoop instance in its main thread. The problem is that loop goes on indefinitly within the process P2 and I'm unable to exit/quit it from the main application process (P1).
Following is the section of code that might help understanding the scenario.
'''
start() spawns a new process P2 that runs Mainloop within its main thread.
stop() is called from P1, but does not quit the Mainloop. This is probably because
processes do not have shared memory
'''
from multiprocessing import Process
import gi
from gi.repository import GObject
class Main:
def __init__(self):
self.process = None
self.loop = GObject.MainLoop()
def worker(self):
self.loop.run()
def start(self):
self.process=Process(target=self.worker, args=())
self.process.start()
def stop(self):
self.loop.quit()
Next, I tried using a multiprocessing Queue for sharing the 'loop' variable between the processes, but am still unable to quit the mainloop.
'''
start() spawns a new process and puts the loop object in a multiprocessing Queue
stop() calls get() from the loop and calls the quit() method, though it still does not quit the mainloop.
'''
from multiprocessing import Process, Queue
import gi
from gi.repository import GObject
class Main:
def __init__(self):
self.p=None
self.loop = GObject.MainLoop()
self.queue = Queue()
def worker(self):
self.queue.put(self.loop)
self.loop.run()
def start(self):
self.p=Process(target=self.worker, args=())
self.p.start()
def stop(self):
# receive loop instance shared by Child Process
loop=self.queue.get()
loop.quit()
How do I call the quit method for the MainLoop object which is only accessible within the child Process P2?
Ok firstly we need to be using threads not processes. Processes will be in a different address space.
What is the difference between a process and a thread?
Try passing the main loop object to a separate thread that does the actual work. This will make your main method in to nothing but a basic GLib event processing loop, but that is fine and the normal behavior in many GLib applciations.
Lastly, we need to handle the race condition of the child process finishing its work before the main loop activates. We do this with the while not loop.is_running() snippet.
from threading import Thread
import gi
from gi.repository import GObject
def worker(loop):
while not loop.is_running():
print("waiting for loop to run")
print("working")
loop.quit()
print("quitting")
class Main:
def __init__(self):
self.thread = None
self.loop = GObject.MainLoop()
def start(self):
self.thread=Thread(target=worker, args=(self.loop,))
self.thread.start()
self.loop.run()
def main():
GObject.threads_init()
m = Main()
m.start()
if __name__ =='__main__' : main()
I extended multiprocessing.Process module in my class Main and overridden its run() method to actually run the GObject.Mainloop instance inside another thread (T1) instead of its main thread. And then implemented a wait-notify mechanism which will make the main thread of Process (P2) to go under wait-notify loop and used multiprocessing.Queue to forward messages to the main thread of P2 and P2 will be notified at the same time. For eg, stop() method, which will send the quit message to P2 for which a handler is defined in the overridden run() method.
This module can be extended to parse any number of messages to the Child Process provided their handlers are to be defined also.
Following is the code snippet which I used.
from multiprocessing import Process, Condition, Queue
from threading import Thread
import gi
from gi.repository import GObject
loop=GObject.MainLoop()
def worker():
loop.run()
class Main(Process):
def __init__(self, target=None, args=()):
self.target=target
self.args=tuple(args)
print self.args
self.message_queue = Queue()
self.cond = Condition()
self.thread = None
self.loop = GObject.MainLoop()
Process.__init__(self)
def run(self):
if self.target:
self.thread = Thread(target=self.target, args=())
print "running target method"
self.thread.start()
while True:
with self.cond:
self.cond.wait()
msg = self.message_queue.get()
if msg == 'quit':
print loop.is_running()
loop.quit()
print loop.is_running()
break
else:
print 'message received', msg
def send_message(self, msg):
self.message_queue.put(msg)
with self.cond:
self.cond.notify_all()
def stop(self):
self.send_message("quit")
self.join()
def func1(self):
self.send_message("msg 1") # handler is defined in the overridden run method
# few others functions which will send unique messages to the process, and their handlers
# are defined in the overridden run method above
This method is working fine for my scenerio but suggestions are welcomed if there is a better way to do the same.
When I run a While True loop in thread and use time.sleep() function the loop stops looping.
I am using this code:
import threading
from time import sleep
class drive_worker(threading.Thread):
def __init__(self):
super(drive_worker, self).__init__()
self.daemon = True
self.start()
def run(self):
while True:
print('loop')
#some code
time.sleep(0.5)
To start the thread I am using this code:
thread = drive_worker()
The loop stops because you flagged the thread as daemon.
The program terminates when there are only daemon threads left running.
self.daemon = True # remove this statement and the code should work as expected
Or make the main thread wait for the the daemon thread to finish
dthread = drive_worker()
# no call to start method since your constructor does that
dthread.join() #now the main thread waits for the new thread to finish
You imported sleep as
from time import sleep
so you have to call sleep in run() as sleep(0.5) or you have to change import as
import time
which I do not recommend.
I have the following two threads:
myThread = threading.Thread(target=sender.mainloop.run, daemon=True)
myThread.start()
myThread2 = threading.Thread(target=receiver.mainloop.run, daemon=True)
myThread2.start()
The targets are GObject.Mainloop() methods.
Afterwards my main program is in an infinite loop.
My problem is that when the execution is terminated by CTRL-C, Keyboardexception is raised for both threads, but the main program does not terminate.
Any ideas how could both the main program and the two threads be terminated by CTRL-C?
ctrl-c issues a SIGINT signal, which you can capture in your main thread for a callback. You can then run whatever shutdown code you want in the callback, maybe a sender/receiver.mainloop.quit() or something.
import threading
import signal
import sys
def loop():
while True:
pass
def exit(signal, frame):
sys.exit(0)
myThread = threading.Thread(target=loop)
myThread.daemon = True
myThread.start()
myThread2 = threading.Thread(target=loop)
myThread2.daemon = True
myThread2.start()
signal.signal(signal.SIGINT, exit)
loop()
I am writing a python script that needs to run a thread which listens to a network socket.
I'm having trouble with killing it using Ctrl+c using the code below:
#!/usr/bin/python
import signal, sys, threading
THREADS = []
def handler(signal, frame):
global THREADS
print "Ctrl-C.... Exiting"
for t in THREADS:
t.alive = False
sys.exit(0)
class thread(threading.Thread):
def __init__(self):
self.alive = True
threading.Thread.__init__(self)
def run(self):
while self.alive:
# do something
pass
def main():
global THREADS
t = thread()
t.start()
THREADS.append(t)
if __name__ == '__main__':
signal.signal(signal.SIGINT, handler)
main()
Appreciate any advise on how to catch Ctrl+c and terminate the script.
The issue is that after the execution falls off the main thread (after main() returned), the threading module will pause, waiting for the other threads to finish, using locks; and locks cannot be interrupted with signals. This is the case in Python 2.x at least.
One easy fix is to avoid falling off the main thread, by adding an infinite loop that calls some function that sleeps until some action is available, like select.select(). If you don't need the main thread to do anything at all, use signal.pause(). Example:
if __name__ == '__main__':
signal.signal(signal.SIGINT, handler)
main()
while True: # added
signal.pause() # added
It's because signals can only be caught by main thread. And here main thread ended his life long time ago (application is waiting for your thread to finish). Try adding
while True:
sleep(1)
to the end of your main() (and of course from time import sleep at the very top).
or as Kevin said:
for t in THREADS:
t.join(1) # join with timeout. Without timeout signal cannot be caught.