We Keep Coding

How to simulate broadcast message passing between Thread

I'm writing a small concurrent program using Python 3.6. I have a question:
my program has a small Thread class (which simulates a thread);
this class has within it 3 methods that are executed as sub-threads:
class myThread(Thread):
def __init__(self, identifier):
super(myThread, self).__init__()
def fun1(self):
# broadcasts messages
def fun2(self):
# event that occurs when a message arrives
# do something
def fun3(self):
# event that occurs when a message arrives
# do something
def run(self):
t1 = Thread(target = self.fun1)
t2 = Thread(target = self.fun2)
t3 = Thread(target = self.fun3)
t1.start()
t2.start()
t3.start()
As you can see, fun1() sends broadcast messages (he sends objects) that the other 2 threads must receive. How can this thing be easily implemented in Python?
I have seen that the simplest way is to use Queue, but I have some doubts... where should I put this queue? How can a general method use the submitted object without emptying this queue (since the "broadcast" object must be used by the other methods)? How does a method perform its body every time a new object is added to the queue (as if it were an event)?

a good way to communicate between threads is using queue
it is better to use a designated queue for every thread
this is how you implement it in your code:
from queue import Queue
from threading import Thread
import time
# define some queues
fun2_q = Queue()
fun3_q = Queue()
class myThread(Thread):
def __init__(self, identifier):
super(myThread, self).__init__()
def fun1(self):
print('starting fun1')
# broadcasts messages
fun2_q.put('say something')
fun3_q.put('say something')
fun2_q.put('quit')
fun3_q.put('quit')
def fun2(self):
# event that occurs when a message arrives
# as a listener we should use infinite loop to monitor messages
# we will use non blocking way to read the queue using "if", also we can use fun2_q.get_nowait()
# instead of "if fun2_q.qsize() > 0:" statement
while True:
if fun2_q.qsize() > 0:
msg = fun2_q.get()
if msg == 'say something':
print('fun2 method saying hello')
elif msg == 'quit':
break # quit thread
# do other stuff below if no messages coming
time.sleep(0.1) # to stop while loop from abusing processor
print('fun2 terminating')
def fun3(self):
# event that occurs when a message arrives
# we will use a blocking way to read the queue
while True:
msg = fun3_q.get() # it will block here waiting for a message to come
if msg == 'say something':
print('fun3 method saying hello')
elif msg == 'quit':
break # quit thread
# can't do other stuff below if no messages coming, the loop will stuck waiting new message
# time.sleep(0.1) # no need for it since the loop will wait anyway
print('fun3 terminating')
def run(self):
t1 = Thread(target = self.fun1)
t2 = Thread(target = self.fun2)
t3 = Thread(target = self.fun3)
t1.start()
t2.start()
t3.start()
my_thread = myThread(1)
my_thread.run()
output:
starting fun1
fun2 method saying hello
fun3 method saying hello
fun3 terminating
fun2 terminating

multiprocessing script to scan for new values and put in queue not working

Here is my script:
# globals
MAX_PROCESSES = 50
my_queue = Manager().Queue() # queue to store our values
stop_event = Event() # flag which signals processes to stop
my_pool = None
def my_function(var):
while not stop_event.is_set():
#this script will run forever for each variable found
return
def var_scanner():
# Since `t` could have unlimited size we'll put all `t` value in queue
while not stop_event.is_set(): # forever scan `values` for new items
x = Variable.objects.order_by('var').values('var__var')
for t in x:
t = t.values()
my_queue.put(t)
time.sleep(10)
try:
var_scanner = Process(target=var_scanner)
var_scanner.start()
my_pool = Pool(MAX_PROCESSES)
while not stop_event.is_set():
try: # if queue isn't empty, get value from queue and create new process
var = my_queue.get_nowait() # getting value from queue
p = Process(target=my_function, args=("process-%s" % var))
p.start()
except Queue.Empty:
print "No more items in queue"
except KeyboardInterrupt as stop_test_exception:
print(" CTRL+C pressed. Stopping test....")
stop_event.set()
However I don't think this script is exactly what I want. Here's what I was looking for when I wrote the script. I want it to scan for variables in "Variables" table, add "new" variables if they don't already exists to the queue, run "my_function" for each variable in the queue.
I believe I have WAYYYY to many while not stop_event.is_set() functions. Because right now it just prints out "No more items in queue" about a million times.
Please HELP!! :)

Multi-processing... Running 2 scripts on 2 terminals

I have 2 scripts that start multiple processes. Right now I'm opening up two different terminals and running python start.py to start both the scripts. How can I achieve this with one command, or one running one script.
Start.py 1
# globals
my_queue = multiprocessing.Manager().Queue() # queue to store our values
stop_event = multiprocessing.Event() # flag which signals processes to stop
my_pool = None
def my_function(foo):
print("starting %s" % foo)
try:
addnews.varfoo)
except Exception,e:
print str(e)
MAX_PROCESSES = 50
my_pool = multiprocessing.Pool(MAX_PROCESSES)
x = Var.objects.order_by('name').values('link')
for t in x:
t = t.values()[0]
my_pool.apply_async(my_function, args=(t,))
my_pool.close()
my_pool.join()
Start1.py 2
# globals
MAX_PROCESSES = 50
my_queue = multiprocessing.Manager().Queue() # queue to store our values
stop_event = multiprocessing.Event() # flag which signals processes to stop
my_pool = None
def my_function(var):
var.run_main(var)
stop_event.set()
def var_scanner():
# Since `t` could have unlimited size we'll put all `t` value in queue
while not stop_event.is_set(): # forever scan `values` for new items
y = Var.objects.order_by('foo).values('foo__foo')
for t in y:
t = t.values()[0]
my_queue.put(t)
try:
var_scanner_process = multiprocessing.Process(target=var_scanner)
var_scanner_process.start()
my_pool = multiprocessing.Pool(MAX_PROCESSES)
#while not stop_event.is_set():
try: # if queue isn't empty, get value from queue and create new process
var = my_queue.get_nowait() # getting value from queue
p = multiprocessing.Process(target=my_function, args=(var,))
p.start()
except Queue.Empty:
print "No more items in queue"
time.sleep(1)
#stop_event.set()
except KeyboardInterrupt as stop_test_exception:
print(" CTRL+C pressed. Stopping test....")
stop_event.set()

You can run the first script in the background on the same terminal, using the & shell modifier.
python start.py &
python start1.py

python threads aren't terminating after target method has finished executing

I'm having some troubles with python threads. I'm writing a software package that plots data received from multiple devices. I have a plot thread that plots the data once it has received a set of data from all devices, and a data retrieval thread for each device. The application plots data continuously (as fast as data can be retrieved from the device) until the user hits a button. I have a threading.Event() self.stop_thread that is checked frequently to back out of the threaded loops. The threads hit the check, break out of the loop, but are still 'running' according to my debugger and threading.active_count(). Does anyone know why this is happening and how can I get it to stop? I need to know these threads are gone before I move on to another function of the application. The following three methods are where the issues arise.
# initalizes startup settings, starts a thread to carry out
# plotting and a seperate thread to carry out data retrieval
def start_plot_threads(self):
if not self.abstraction.connected:
self.connect_to_device()
if not self.abstraction.connected:
return
self.stop_thread.clear()
self.pause_thread.clear()
for device in self.devices:
device.pause_thread.clear()
device.stop_thread.clear()
device.change_units.set()
self.presentation.enable_derivative()
self.presentation.show_average_button.SetValue(False)
self.presentation.show_average_button.Disable()
self.abstraction.multi_plot_data = {}
try:
if self.plot_thread.is_alive():
return
except Exception:
pass
self.plot_thread = Thread(target=self.plot_data)
self.plot_thread.daemon = True
self.plot_thread.start()
for device in self.devices:
thread = Thread(target=self.retrieve_data,
kwargs={'device': device},
name="Data Retrieval Thread %s" % device.instr_id)
thread.daemon = True
thread.start()
# waits for plot data to be thrown on a thread safe queue by the data
# retrieval thread and plots it. data comes in as a tuple of the form
# (y_data, label, x_data)
def plot_data(self):
multiplot = False
if len(self.devices) > 1:
multiplot = True
plot_data = []
while not self.stop_thread.is_set():
try:
data = self.plot_data_queue.get()
except Empty:
pass
else:
if multiplot:
scan = {}
scan['y_data'] = [data[0]]
scan['labels'] = [data[1]]
scan['x_data'] = data[2]
plot_data.append(scan)
if len(plot_data) == len(self.devices):
self.presentation.plot_multiline(plot_data, average=False)
self.abstraction.multi_plot_data = plot_data
plot_data = []
else:
self.presentation.plot_signal(data[0], data[1])
# the intent is that the data retrieval thread stays in this loop while
# taking continuous readings
def retrieve_data(self, device):
while True:
if device.stop_thread.is_set():
return
while device.pause_thread.is_set():
if device.stop_thread.is_set():
return
sleep(0.1)
y = self.get_active_signal_data(device)
if not y:
return
self.plot_data_queue.put(
(y, device.name, device.x_data))
self.abstraction.y_data = [y]
try:
self.update_spectrum(device)
except DeviceCommunicationError, data:
self.presentation.give_connection_error(data)
self.presentation.integ_time = device.prev_integ
I apologize for the extra bulk in the methods. They are straight from my code base.

The reason why your threads continue running is unknown- device.stop_thread.is_set(): (What is doing the setting??)
However you can guarantee that all your threads have stopped by retaining a handler on each thread ( by appending each thread object to a list) and once you have started all your threads you can then proceed to thread.join() them.
threads = []
for job in batch:
thr = threading.Thread(target=do_job, args = (job))
thr.start()
threads.append(thr)
#join all the threads
for thr in threads:
thr.join()
Join will wait for the thread to complete before moving on.
Python Docs:
https://docs.python.org/2/library/threading.html

Controlling a python thread with a function

Thanks to those who helped me figure out I needed to use threading to run a loop in a control script I have run, I now have an issue to try and control the thread - by starting or stopping it based on a function:
I want to start a process to get a motor to cycle through a movement based on a 'start' parameter sent to the controlling function, also I want to send a 'stop' parameter to stop the thread too - here's where I got to:
def looper():
while True:
print 'forward loop'
bck.ChangeDutyCycle(10)
fwd.ChangeDutyCycle(0)
time.sleep(5)
print 'backwards loop'
bck.ChangeDutyCycle(0)
fwd.ChangeDutyCycle(20)
time.sleep(5)
def looper_control(state):
t = threading.Thread(target=looper)
if state == 'start':
t.start()
elif state == 'stop':
t.join()
print 'looper stopped!!'
This starts the thread okay when I call looper_control('start') but throws an error when looper_control('stop'):
File "/usr/lib/python2.7/threading.py", line 657, in join
raise RuntimeError("cannot join thread before it is started")
RuntimeError: cannot join thread before it is started
EDIT: looper_control called from here
if "motor" in tmp:
if tmp[-1:] == '0':
#stop both pin
MotorControl('fwd',0,0)
print 'stop motors'
looper_control('stop')
elif tmp[-1:] == '2':
#loop the motor
print 'loop motors'
looper_control('start')
UPDATE: Ive not been able to stop the thread using the method suggested - I thought I had it!
here's where I am:
class sliderControl(threading.Thread):
def __init__(self,stop_event):
super(sliderControl,self).__init__()
self.stop_event = stop_event
def run(self):
while self.stop_event:
print 'forward loop'
bck.ChangeDutyCycle(10)
fwd.ChangeDutyCycle(0)
time.sleep(5)
print 'backwards loop'
bck.ChangeDutyCycle(0)
fwd.ChangeDutyCycle(20)
time.sleep(5)
def looper_control(state,stop_event):
if state == 'start':
t = sliderControl(stop_event=stop_event)
t.start()
elif state == 'stop':
#time.sleep(3)
stop_event.set()
#t.join()
print 'looper stopped!!'
called via:
if tmp[-1:] == '0':
#stop both pin
MotorControl('fwd',0,0)
print 'stop motors'
#stop_thread_event = threading.Event()
print 'stopping thread'
print stop_thread_event
looper_control('stop',stop_thread_event)
elif tmp[-1:] == '2':
#loop the motor
print 'loop motors'
global stop_thread_event
stop_thread_event = threading.Event()
print stop_thread_event
looper_control('start', stop_thread_event)
It looked like a separate thread event was being called by loop and stop, so I thought a global would sort it out but its just not playing ball. When I start the loop - it runs, but when I try to stop it, I get looper stopped!! , but the process just keeps running

Your top-level thread routine will need to become an event handler that listens to a Queue object (as in from Queue import Queue) for messages, then handles them based on state. One of those messages can be a shutdown command, in which case the worker thread function simply exits, allowing the main thread to join it.
Instead of time.sleep, use threading.Timer with the body of the timer sending a message into your event queue.
This is a substantial refactoring. But especially if you plan on adding more conditions, you'll need it. One alternative is to use a package that handles this kind of thing for you, maybe pykka.

To stop a python thread you can use threading.Event()
try this:
class YourClass(threading.Thread):
def __init__(self, stop_event):
super(YourClass, self).__init__()
self.stop_event = stop_event
def run(self):
while not self.stop_event.is_set():
# do what you need here (what you had in looper)
def looper_control(state, stop_event):
if state == 'start':
t = YourClass(stop_event=stop_event)
t.start()
elif state == 'stop':
stop_event.set()
and call to looper_control:
stop_thread_event = threading.Event()
looper_control(state, stop_thread_event)

you only can "start" once a thread
but you can lock and unlock the thread.
the best way to stop and start a thread is with mutex, Example:
#!/usr/bin/python
import threading
from time import sleep
mutex2 = threading.Lock()
#This thread add values to d[]
class Hilo(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
while True:
mutex2.acquire()
#Add values to d[]
d.append("hi from Peru")
mutex2.release()
sleep(1)
d=[];
hilos = [Hilo()]
#Stop Thread
#If you have more threads you need make a mutex for every thread
mutex2.acquire()
#Start treades, but the thread is lock
for h in hilos:
h.start()
#so you need do
#unlock THREAD<
mutex2.release()
#>START THREAD
#Sleep for 4 seconds
sleep(4)
#And print d[]
print d
print "------------------------------------------"
#WAIT 5 SECONDS AND STOP THE THREAD
sleep(5)
try:
mutex2.acquire()
except Exception, e:
mutex2.release()
mutex2.acquire()
#AND PRINT d[]
print d
#AND NOW YOUR TRHEAD IS STOP#
#When the thread is lock(stop), you only need call: mutex2.release() for unlock(start)
#When your thread is unlock(start) and you want lock(stop):
#try:
# mutex2.acquire()
#except Exception, e:
# mutex2.release()
# mutex2.acquire()