I started learning python recently, and I am facing a situation that I do not even know if it is expected, or if something is wrong.
I am learning parallel threading to have two independent processes on the same program (UI control on one thread, image processing on another)
So, to test this I created this simple code:
(Camera is a custom class that connects to a usb webcam)
import thread
from vii.camera import Camera
class Process(object):
def __init__(self, width=800, height=600):
self._cam = Camera(width, height)
self._is_running = False
self._current_image = None
def start(self):
thread.start_new(self._run(), (self))
def _run(self):
self._cam.start()
self._is_running = True
while self._is_running:
self._current_image = self._cam.update()
self._current_image.show()
def get_image(self):
return self._current_image
def stop(self):
self._is_running = False
self._cam.close()
thread.exit()
process = Process()
process.start()
print("You will never see this output")
while (True):
key = raw_input()
if key == 'q':
process.stop()
break
The thread is created with success, and I am able to see the image. Now, I need to be able to affect it (stop it, get data from it) from the main thread. But the problem is that the code never enters in the while loop.
Is this behaviour expected? If it is, is there a way for me to achieve the functionality I need?
Related
I am pretty new to Python and have a question about threading.
I have one function that is called pretty often. This function starts another function in a new Thread.
def calledOften(id):
t = threading.Thread(target=doit, args=(id))
t.start()
def doit(arg):
while true:
#Long running function that is using arg
When calledOften is called everytime a new Thread is created. My goal is to always terminate the last running thread --> At all times there should be only one running doit() Function.
What I tried:
How to stop a looping thread in Python?
def calledOften(id):
t = threading.Thread(target=doit, args=(id,))
t.start()
time.sleep(5)
t.do_run = False
This code (with a modified doit Function) worked for me to stop the thread after 5 seconds.
but i can not call t.do_run = False before I start the new thread... Thats pretty obvious because it is not defined...
Does somebody know how to stop the last running thread and start a new one?
Thank you ;)
I think you can decide when to terminate the execution of a thread from inside the thread by yourself. That should not be creating any problems for you. You can think of a Threading manager approach - something like below
import threading
class DoIt(threading.Thread):
def __init__(self, id, stop_flag):
super().__init__()
self.id = id
self.stop_flag = stop_flag
def run(self):
while not self.stop_flag():
pass # do something
class CalledOftenManager:
__stop_run = False
__instance = None
def _stop_flag(self):
return CalledOftenManager.__stop_run
def calledOften(self, id):
if CalledOftenManager.__instance is not None:
CalledOftenManager.__stop_run = True
while CalledOftenManager.__instance.isAlive():
pass # wait for the thread to terminate
CalledOftenManager.__stop_run = False
CalledOftenManager.__instance = DoIt(id, CalledOftenManager._stop_flag)
CalledOftenManager.__instance.start()
# Call Manager always
CalledOftenManager.calledOften(1)
CalledOftenManager.calledOften(2)
CalledOftenManager.calledOften(3)
Now, what I tried here is to make a controller for calling the thread DoIt. Its one approach to achieve what you need.
In Python I have my main program which is basically a console 'gui' class. Its main method is a running thread that keeps the mains screen with options printed, and keeps waitin for user to input a choice.
The gui class has another object initialized that has its own running Threads. One of the Threads in this other object basically requires the main GUI thread to switch to a different mode / stop, but because it always 'pauses' on any input(), it will never switch.
So is there way, since I have access to the gui thread from the other object, to send an empty stdin to the gui thread so it gets over input()?
Example: (not actual code)
from threading import Thread
class Gui:
def __init__(self):
self.mainthread = Thread(target=self.console_loop, daemon=True)
self.server = Server(self.mainthread)
self.mainthread.start()
self.mainthread.join()
def console_loop(self):
while True:
if some_terminating_condition:
break
while self.server.guiloop:
"""
.
. various code
.
"""
a = input()
if a == "Some input":
self.server.guiloop = False
self.server.do_stuff()
while not self.server.guiloop:
"""
More code
"""
class Server:
def __init__(self, guithread):
self.guiloop = True
self.gui_thread = guithread
def do_stuff(self):
self.guiloop = False
"""
Code to stop gui waiting for input <-- What I need
"""
# Arbitrary code
Sorry for any mistakes, wrote it on the fly, since my code is too big to copy over
I have been making a GUI for a genetic algorithm I am working on and I made the mistake of leaving the threading so late simply because I did not (and still don't) know how to do it. So essentially when the start button is clicked the function 'run' starts the whole infinite loop process which actually happens in generation_loop. Each generation the loop checks to see if it should still be running. The idea is that if the stop or pause button has been clicked it will stop looping (with the stop button all the data is cleared with the pause button it remains and the unpause button just sets running to True and calls generation_loop)
So I need to work out a way to make my GUI responsive while generation_loop is running. Here is my code, I tried to minimise it but I am unsure what is important information for threading:
class Window(main_window, QMainWindow):
def __init__(self):
QMainWindow.__init__(self)
main_window.__init__(self)
self.setupUi(self)
self.scene = QGraphicsScene()
self.im_view.setScene(self.scene)
self.setWindowTitle('Fantasy Generator')
self.running = False
self.first_run = True
self.im = Image.new('RGBA', (400, 400), (0, 0, 0, 255))
self.saved_gens = deque([('A', self.im, self.im, self.im)])
self.set_save_amount(self.sb_saveamt.value())
self.population = []
self.btn_exit.clicked.connect(self.close)
self.actionQuit.triggered.connect(self.close)
self.btn_pauser.clicked.connect(self.pause_button)
self.sb_saveamt.valueChanged[int].connect(self.set_save_amount)
self.btn_restart.clicked.connect(self.start_button)
self.btn_loadimage.clicked.connect(self.get_image)
self.actionLoad_Image.triggered.connect(self.get_image)
self.gen_sldr.valueChanged[int].connect(self.display_gen)
self.cb_display.currentIndexChanged.connect(self.change_quality)
self.has_image = True
self.display_gen(0)
def get_image(self):
pass
# To save you time I removed the code here. It just sets self.im using a file dialog basically
def set_save_amount(self, amt):
if amt == -1:
self.saved_gens = deque(self.saved_gens)
else:
self.saved_gens = deque(self.saved_gens, amt + 1)
def pause_button(self):
if self.first_run:
self.run()
elif self.running:
self.running = False
self.btn_pauser.setText('Resume Execution')
# pause stuff goes here
else:
self.running = True
self.btn_pauser.setText('Pause Execution')
self.generation_loop()
# resume from pause stuff goes here
def start_button(self):
if self.first_run:
self.run()
else:
self.end()
# The run function should start the actual process
def run(self):
self.btn_restart.setText('End')
self.btn_pauser.setText('Pause Execution')
self.first_run = False
self.running = True
settings = dict(ind_per_gen=self.sb_ipg.value(), shapes_per_im=self.sb_spi.value(),
complexity=self.sb_complexity.value(), mut_rate=self.sb_mutation.value(),
cross_chance=self.sb_cross.value(), seed=self.sb_seed.value())
self.population = Population(self.im, **settings)
self.generation_loop()
# This is the loop I want to be able to exit out of using buttons
def generation_loop(self):
while self.running:
if self.first_run:
break
self.add_generation_data(self.population.next_gen())
def end(self):
self.btn_restart.setText('Start')
self.btn_pauser.setText('Start Execution')
self.first_run = True
self.running = False
self.saved_gens = deque([('A', self.im, self.im, self.im)])
self.set_save_amount()
self.display_gen(0)
def add_generation_data(self, data):
self.saved_gens.append(data)
self.gen_sldr.setMaximum(len(self.saved_gens) - 1)
self.gen_sldr.setValue(len(self.saved_gens) - 1)
self.display_gen(data[0] + 1)
def change_quality(self):
self.display_gen(self.gen_sldr.value())
def resizeEvent(self, e):
if self.has_image:
self.im_view.fitInView(QRectF(0, 0, self.width, self.height), Qt.KeepAspectRatio)
self.scene.update()
def display_image(self, image):
self.scene.clear()
if image.mode != 'RGBA':
image = image.convert('RGBA')
self.width, self.height = image.size
qim = ImageQt.ImageQt(image)
pixmap = QPixmap.fromImage(qim)
self.scene.addPixmap(pixmap)
self.im_view.fitInView(QRectF(0, 0, self.width, self.height), Qt.KeepAspectRatio)
self.scene.update()
def display_gen(self, index):
self.lcd_cur_gen.display(self.saved_gens[index][0])
if self.cb_display.currentIndex() == 0:
self.display_image(self.saved_gens[index][1])
elif self.cb_display.currentIndex() == 1:
self.display_image(self.saved_gens[index][2])
else:
self.display_image(self.saved_gens[index][3])
if __name__ == '__main__':
app = QApplication(sys.argv)
w = Window()
w.show()
sys.exit(app.exec_())
EDIT: I also just found at that I can't even change the graphics view from within the generation_loop but it works and changes if I limit the loop
In order to move your long running code to a thread, you need to first identify which parts of the long running code interact with the GUI and which parts don't. The key reason for this is that interacting with the GUI from a secondary thread is forbidden, and will lead to segfaults.
It looks like self.population.next_gen() is the long running bit of the code and doesn't interact with the GUI (although what this does is not provided so I can't be sure) while self.add_generation_data(...) updates the GUI which should be reasonably fast.
As such, this makes it reasonably simple to separate, which I'll show below.
Now, about threads. Python provides threads through the threading module (as the other answers show), however these not are recommended for use with a PyQt application if you want your thread to have any relation to the GUI (see here). PyQt also provides threading via the QThread object, which integrates support for sending and receiving Qt signals (which are thread safe). In short, the QThread has a separate event loop, and processes signals received asynchronously to the main thread, thus leaving the event loop in the main thread to process GUI events (like button clicks).
Typically you create a new class that inherits from QObject, instantiate it and move it to a QThread. Slots (aka methods) in the object that are triggered by a signal emission, then run in the thread.
So you'll want to do something like this
class MyWorker(QObject):
done = pyqtSignal(object) # you may need to update "object" to the type returned by Population.next_gen()
def __init__(self, settings):
# create the population object with whatever settings you need
# Note that this method runs in the parent thread as you have
# yet to move the object to a new thread. It shouldn't cause any
# problems, but may depend on what the Population class is/does.
# TODO: I've removed the reference to an image here...
#it may or may not be thread safe. I can't tell from your code.
self.population = Population(..., settings)
#pyqtSlot()
def next_gen(self):
new_gen = self.population.next_gen()
self.done.emit(new_gen)
class Window(....):
make_next_generation = pyqtSignal()
....
def run(self):
self.btn_restart.setText('End')
self.btn_pauser.setText('Pause Execution')
self.first_run = False
self.running = True
settings = dict(ind_per_gen=self.sb_ipg.value(), shapes_per_im=self.sb_spi.value(),
complexity=self.sb_complexity.value(), mut_rate=self.sb_mutation.value(),
cross_chance=self.sb_cross.value(), seed=self.sb_seed.value())
self.setupThread(settings)
def setupThread(self, settings):
self.thread = QThread()
self.worker = MyWorker(settings)
self.worker.moveToThread(self.thread)
# connect a signal in the main thread, to a slot in the worker.
# whenever you emit the signal, a new generation will be generated
# in the worker thread
self.make_next_generation.connect(self.worker.next_gen)
# connect the signal from the worker, to a slot in the main thread.
# This allows you to update the GUI when a generation has been made
self.worker.done.connect(self.process_generation)
# Start thread
self.thread.start()
# emit the signal to start the process!
self.make_next_generation.emit()
def process_generation(new_gen):
# run the GUI side of the code
# ignore the new generation if the "end" button was clicked
if not self.first_run:
self.add_generation_data(new_gen)
if self.running:
# make another generation in the thread!
self.make_next_generation.emit()
def pause_button(self):
if self.first_run:
self.run()
elif self.running:
self.running = False
self.btn_pauser.setText('Resume Execution')
# pause stuff goes here
else:
self.running = True
self.btn_pauser.setText('Pause Execution')
# make another generation in the thread!
self.make_next_generation.emit()
Things to note:
I haven't included all of your code in my answer. Merge as appropriate.
I'm unsure what self.im is. It's passed to Population so there might be some thread unsafe behaviour in your code that I can't see. I've left it to you to fix
I'm familiar with PyQt4, not PyQt5, so there is a possibility some things I've done don't work quite right. It should be easy for you to work out what to change from any error messages that are raised.
It's a bit messy recreating the thread and worker each time it is started from scratch. You might want to consider moving the instantiation of Population to a method in the worker (one that isn't __init__ and invoking it each time you want to start from scratch (in the same way we trigger a new generation). This would allow you to move pretty much all of setupThread to the Window.__init__ method and then when the start button was clicked, you'd just emit a signal to recreate Population followed by one to make the first generation.
You can use Threading events here.
from threading import Thread, Event
Once you detect the button click,
class MyThread(Thread):
def __init__(self, the_function, <any input param you want to provide>):
Thread.__init__(self)
self.stop_event = Event()
self.exec_func = the_function
def set_stop_flag(self, value):
if value:
self.stop_event.set()
else:
self.stop_event.clear()
def run(self):
while True:
try:
if not self.stop_event.is_set()
self.exec_func()
else:
break # once the event is set, you can break which will kill this thread.
# To stop busy waiting you can make this thread sleep for some seconds after each cycle.
import time
time.sleep(10) # 10 seconds wait before the next cycle.
except Exception, excep:
print "Got exception > ", str(excep)
Now in your code you embed this code piece and keep a reference for this thread.
Let's say
self.my_thread = MyThread(self.function_to_perform, <blabla>)
self.my_thread.setDaemon(True) # So that you don't have to worry about it when the Main process dies!
self.my_thread.start()
Now once you get a STOP button click event you call
self.my_thread.set_stop_flag(True) # Bingo! Your thread shall quit.
I've come across an unusual problem in regards to updating variables. I've built a simple class object to help me with some network sniffing. I wanted to make a parallel process which allows me to run some network tests and capture the traffic generated using python so I can extend the program to do amazing things. I'm using scapy's sniffing function to help with the interface sniffing.
Scapy's sniffer allows you to pass a function into itself function that allows you to create a 'stop sniffing' condition. In my case I've created function stop_filter and I wish to stop the Scapy sniff function by simply updating the self.stop_sniffing instance variable. I've presented the program output below, which shows self.stop_sniffing getting set to True in Function stop, but is then set back to False (or is not updated at all) when printed in stop_filter. I have no clue why this is happening and no solution comes to mind as it's such a weird problem.
If anyone with fresh eyes can see what insane thing I've done here it would be greatly appreciated!
from scapy.all import *
from multiprocessing import Process
class DatasetSniffer:
def __init__(self, iface, local_dir='.'):
self.iface = iface
self.master = None
self.local_dir = local_dir
self.stop_sniffing = False # Never updates! why!?
self.writer = PcapWriter(local_dir+"/master.pcap", append=True, sync=True)
def stop_filter(self, p):
# Note: 'p' gets passed in by Scapy function 'sniff'
print self.stop_sniffing
# Return 'True' to stop sniffer
return self.stop_sniffing
def sniff(self):
sniff(store=0, prn=self.writer.write, iface=self.iface, stop_filter=self.stop_filter)
def start(self):
self.master = Process(target=self.sniff)
self.master.start()
def stop(self):
self.stop_sniffing = True
# Shows that self.stop_sniffing is 'True'
print self.stop_sniffing
self.master.join()
if __name__ == "__main__":
interface = 'en3'
sniffer = DatasetSniffer(interface)
sniffer.start()
# some process
time.sleep(5)
sniffer.stop()
Shell output:
sudo python sniffing.py
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
False
True
False
False
False
False
The Problem
You are not using multiple threads in this example code you are using multiple processes.
Here you have two separate processes, that do not share memory:
the original process
a new process, started by multiprocessing.Process.start
this process will have been started by forking the original process, creating a copy of its memory at the time of the fork. They do not "share" memory.
Now, when you call DatasetSniffer.stop within your original process, this will not alter the value of stop_sniffing in the new ("master") process.
How to Communicate Then?
When using multiprocessing, you can communicate using a Pipe. Something like this:
readable_pipe, writable_pipe = multiprocessing.Pipe(duplex=False)
process = Process(target=do_something)
Now, our original process can send a message by writing to the pipe:
writable_pipe.send("stop")
while the new process can check for messages using:
if readable_pipe.poll():
msg = readable_pipe.recv()
Try working this into your code.
Thanks for all your suggestions. After a glass of inspiration I managed to knock up this script. Probably a nicer way to approach my problem without making too many changes. So this code allows the threads to use the stop function outside the class, thus allowing all the asynchronous tasks to use the stop_filter.
Found this information in the link below. Hopfully this post will be useful to someone else!
http://www.tutorialspoint.com/python/python_multithreading.htm
Cheers!
import threading
from scapy.all import *
from datetime import datetime
directory = str(datetime.now().strftime("%Y%m%d%H%M%S"))
os.makedirs(directory)
DatasetSnifferExit = 0
class DatasetSniffer(threading.Thread):
def __init__(self, iface, local_dir='.', filename=str(datetime.now())):
self.iface = iface
self.filename = filename
self.local_dir = local_dir
self.stop_sniffing = False
self.writer = PcapWriter(local_dir+"/"+filename+".pcap", append=True, sync=True)
threading.Thread.__init__(self)
def run(self):
sniff_interface(self.writer.write, self.iface)
def stop_filter(p):
if DatasetSnifferExit:
return True
else:
return False
def sniff_interface(write, iface):
sniff(store=0, prn=write, iface=iface, stop_filter=stop_filter)
if __name__ == "__main__":
DatasetSnifferExit = False
# Create new threads
pcap1 = DatasetSniffer('en3', directory, "master")
pcap2 = DatasetSniffer('en0', directory, "slave")
# Start new Threads
pcap1.start()
pcap2.start()
# Do stuff
time.sleep(10)
# Finished doing stuff
DatasetSnifferExit = True
first time SO user, please excuse any etiquette errors. I'm trying to implement a multithreaded program in python and am having troubles. This is no doubt due to a lack of understanding of how threading is implemented, but hopefully you can help me figure it out.
I have a basic program that continually listens for messages on a serial port and can then print/save/process/etc them, which works fine. It basically looks like this:
import serial
def main():
usb = serial.Serial('/dev/cu.usbserial-A603UBRB', 57600) #open serial w\ baud rate
while True:
line = usb.readline()
print(line)
However what I want to do is continually listen for the messages on a serial port, but not necessarily do anything with them. This should run in the background, and meanwhile in the foreground I want to have some kind of interface where the user can command the program to read/use/save these data for a while and then stop again.
So I created the following code:
import time
import serial
import threading
# this runs in the background constantly, reading the serial bus input
class serial_listener(threading.Thread):
def __init__(self, line, event):
super(serial_listener, self).__init__()
self.event = threading.Event()
self.line = ''
self.usb = serial.Serial('/dev/cu.usbserial-A603UBRB', 57600)
def run(self):
while True:
self.line = self.usb.readline()
self.event.set()
self.event.clear()
time.sleep(0.01)
# this lets the user command the software to record several values from serial
class record_data(threading.Thread):
def __init__(self):
super(record_data, self).__init__()
self.line = ''
self.event = threading.Event()
self.ser = serial_listener(self.line,self.event)
self.ser.start() #run thread
def run(self):
while(True):
user_input = raw_input('Record data: ')
if user_input == 'r':
event_counter = 0
while(event_counter < 16):
self.event.wait()
print(self.line)
event_counter += 1
# this is going to be the mother function
def main():
dat = record_data()
dat.start()
# this makes the code behave like C code.
if __name__ == '__main__':
main()
It compiles and runs, but when I order the program to record by typing r into the CLI, nothing happens. It doesn't seem to be receiving any events.
Any clues how to make this work? Workarounds are also fine, the only thing is that I can't constantly open and close the serial interface, it has to remain open the whole time, or else the device stops working until un/replugged.
Instead of using multiple threads, I would suggest using multiple processes. When you use threads, you have to think about the global interpreter lock. So you either listen to events or do something in your main thread. Both at the same time will not work.
When using multiple processes I would then use a queue to forward the events from your watchdog that you would like to handle. Or you could code your own event handler. Here you can find an example for multiprocess event handlers