I have a class that, calls a method from other and execute it in parallel, the function that is executed has a for loop that iterates over a dataframe. How can I display a progress bar for each thread?
class App:
otherClass = OtherClass()
n_cpus = os.cpu_count()
def call_multiprocessing(self):
with mp.Pool(processes=self.n_cpus) as executor:
for result in executor.map(otherClass.some_method, self.dataframes):
pass
class OtherClass()
def some_method(self, df):
for index, row in tqdm(dataframe.iterrows(), desc="Progress: ", total= len(df)):
#For each thread that this function is executing
#I want to display a progress bar and keep her in the screen
The way that I'm doing each time a progress bar from a thread is updated it replaces the one that is in console. I want to show then simultaneously, is it possible? Example: If the PC has 4 CPUs, 4 progress bars.
Progress bars should only run in the main thread. Use a queue or similar to pass the status to the main thread and display from there.
Here is an example using Enlighten and multiprocessing, but you should be able to achieve something similar using the threading and queue modules.
Related
I am developing an app in Kivy and have one function that seems to take a long time to finish. Therefore, when the button that calls this function is pressed, i first open a modalview with a progress bar. I now want to update the value in the progressbar every 500ms while the main function is executing
My first attempt was to use Clock.schedule_interval(). Pseudo code looks something like this:
Class MainClass(Screen):
def button_callback_function(self):
#open modalview with progress bar
view = ModalView(autodismiss=True)
view.open()
Clock.schedule_interval(self.update_progress_bar_in_modalview,0.5)
self.function_that_takes_too_long()
def function_that_takes_too_long(self):
/* code here*/
def update_progress_bar_in_modalview(self,dt):
/*updates value of progress bar*/
With this code, the progress bar does indeed get updated but only after function_that_takes_too_long() finishes, not parallel to it.
My second attempt was to use python threads:
Class MainClass(Screen):
def button_callback_function(self):
#open modalview with progress bar
view = ModalView(autodismiss=True)
view.open()
x=threading.Thread(target=self.update_progress_bar_in_modalview)
x.start()
self.function_that_takes_too_long()
def function_that_takes_too_long(self):
/* code here*/
def update_progress_bar_in_modalview(self):
/*updates value of progress bar*/
timer.sleep(0.5)
Here the second thread to update the progress bar is never even started. It seems the main thread has to pause to give the second thread a chance to start.
So is there any way to call update_progress_bar_in_modalview() while function_that_takes_too_long() is still executing? Something like periodic interrupts in micro controllers. Or maybe start the thread that updates the progress bar on a separate core?
I appreciate any hints.
Alex
i have some trouble to make a gui responsive while plotting live data. So that the GUI doesn't freeze, I try to thread all of my activities. I want to realize the following:
Recording data through a serial port
Calculations for the later plot in a thread
Plotting in a thread (currently via a QTimer, but when i drag the window, there is always a "small" freeze and the plot does not update on the drag)
1 and 2 is done, but now im not sure how to update my plot in a seperate thread.
My PlotWidget get initiated looks like this:
self.plottingDQ = [queue.deque(maxlen=100), queue.deque(maxlen=100), queue.deque(maxlen=100)]
self.graph = pg.PlotWidget()
self.barItem = pg.BarGraphItem(x0=self.plottingDQ[0], y0=self.plottingDQ[1], width=self.plottingDQ[2], height=1)
self.graph.addItem(self.barItem)
starting my threads is done via a button which is connected to this function. Writer-Thread is not relevant because it has no dependency on the plot. But the Calculator-Thread calculates the data for updating the plot
def startPlotting(self):
# not relevant for the example
self.csvData = queue.Queue()
self.csv = Writer(self.csvData)
self.csv.setDaemon(True)
self.csv.start()
self.calcData = queue.Queue()
self.calcPlot = Calculator(self.calcData, self.plottingDQ)
self.calcPlot.setDaemon(True)
self.calcPlot.start()
# Timer to update plot every x ms
self.timer = QTimer()
self.timer.timeout.connect(self.updatePlot)
self.timer.start(500)
Right now im updating my plot within the Qtimer every 500ms
def updatePlot(self):
print("update")
self.barItem.setOpts()
So every time i get some input from my serial port i just pass the data to my thread and call something like this:
def fromPort(self, data):
self.csvData.put(data)
self.calcData.put(data)
Inside my Calculator-Thread the data will be calculated and handed over to the plottingDQ which is connected to the BarGraphItem
class Calculator(threading.Thread):
def __init__(self, calcData, plottingDQ):
threading.Thread.__init__(self)
self.calcData = calcData
self.plottingDQ = plottingDQ
self.a = threading.Event()
self.a.set()
def run(self):
while self.a.isSet():
# Do some stuff here ...
# After the calculations, i write the data into the plottingDQ
self.plottingDQ[0].append(x)
self.plottingDQ[1].append(y)
self.plottingDQ[2].append(duration)
Is this the correct way to pass my calculated data from my Calcualtor-Thread into the deques which are used in the BarGraphItem? How can i update my BarGraphItem inside a thread?
The way you programmed this looks fine.
The root cause of the "stutter" seems to be an "update block" during the dragging process.
Try to enforce updates by adding pg.QtGui.QApplication.processEvents() to your updatePlot function, as described here:
def updatePlot(self):
print("update")
self.barItem.setOpts()
pg.QtGui.QApplication.processEvents()
It may be better to use Qthread to deal with this, having a worker thread doing the plot update inside the app loop.
I'm writing a tkinter app.
I want to use Thread to avoid the tkinter window freezing but actually I did not find solution.
A quick part of my code (simplify):
from threading import Thread
import tkinter as tk
class App(tk.Tk):
def __init__(self):
super().__init__()
search_button = tk.Button(self, text='Print', command=self.Running)
search_button.grid(row=0, column=0)
def funct1(self):
print('One')
def funct2(self):
print('Two')
def CreateThread(self, item):
self.item = item
t = Thread(target=self.item)
t.start()
def Running(self):
self.CreateThread(self.funct1)
# How to wait for the end of self.CreateThread(self.funct1) ?
self.CreateThread(self.funct2)
if __name__ == '__main__':
myGUI = App()
myGUI.mainloop()
How to wait for the self.CreateThread(self.funct1) ending before running self.CreateThread(self.funct2).
With a queue ?
With something else ?
I already have take a look to Thread.join() but it freez the tkinter window.
Hope you can help me :)
IMO you should think differently about what "Thread" means. A thread is not a thing that you run. A thread is a thing that runs your code. You have two tasks (i.e., things that need to be done), and you want those tasks to be performed sequentially (i.e., one after the other).
The best way to do things sequentially is to do them in the same thread. Instead of creating two separate threads, why not create a single thread that first calls funct1() and then calls funct2()?
def thread_main(self):
funct1()
funct2()
def Running(self):
Threead(target=self.thread_main).start()
P.S.: This could be a mistake:
def CreateThread(self, item):
self.item = item
t = Thread(target=self.item)
t.start()
The problem is, both of the threads are going to assign and use the same self.item attribute, and the value that is written by the first thread may be over-written by the second thread before the first thread gets to used it. Why not simply do this?
def CreateThread(self, item):
Thread(target=item).start()
Or, since the function body reduces to a single line that obviously creates and starts a thread, why even bother to define CreateThread(...) at all?
You can synchronize threads using locks. Hard to give a concrete answer without knowing what needs to be done with these threads. But, locks will probably solve your problem. Here's an article on synchronizing threads.
I have a function that reads in two large files and merges them, and I would like to add a progress bar using TK to show the user that the function is still running.
def makeDF():
# DF SETUP---
print("Starting dataframe creation")
qData = pd.read_excel("fileOne.xlsx", sheet_name='Sheet1')
yData= pd.read_excel(fileTwo.xlsx", sheet_name='Sheet1')
allData = pd.merge(qData, yData, on=['Var1','Var2','Var3'], how="outer")
I have read through various threads and not found how to link a progress bar to a function that isn't looping or incrementing anything. I just want to start the progress bar, execute the function, have the progress bar update/show activity while the function runs, then end the progress bar when the function is complete. So far I have seen articles on threading, but they always involve some list of items that is being processed or worked through in a queue, not just one lengthy operation.
Thank you!
I would like to update a progress bar that I have on a main window with the progress of a task that I am doing on another sub-routine, would it be possible??
To be as clear as possible, I would have 2 files:
In my Mainwindow.py I would have something like:
import Calculations
#some code
self.ui.progressBar
Calculations.longIteration("parameters")
Then I would have a separate file for the calculations: Calculations.py
def longIteration("parameters")
#some code for the loop
"here I would have a loop running"
"And I would like to update the progressBar in Mainwindow"
Is that possible?
Or it should be done on a different way?
Thanks.
The simplest of methods would be to simply pass the GUI object:
self.ui.progressBar
Calculations.longIteration("parameters", self.ui.progressBar)
and update progressBar on Calculations. This has two problems, though:
You're mixing GUI code with Calculations, who probably shouldn't know anything about it
if longIteration is a long running function, as its name implies, you're blocking your GUI main thread, which will make many GUI frameworks unhappy (and your application unresponsive).
Another solution is running longIteration in a thread, and pass a callback function that you use to update your progress bar:
import threading
def progress_callback():
#update progress bar here
threading.Thread(target=Calculations.longIteration, args=["parameters", progress_callback]).run()
then, inside longIteration, do:
def longIteration( parameters, progress_callback ):
#do your calculations
progress_callback() #call the callback to notify of progress
You can modify the progress_callback to take arguments if you need them, obviously