I am creating a PyQt5 application and want to run some ML code in a separate thread when a button is clicked. I am unable to do so. This is my threading code:
newthread = QtCore.QThread()
runthread = RunThread()
runthread.moveToThread(newthread)
runthread.finished.connect(newthread.quit)
newthread.started.connect(runthread.long_run)
newthread.start()
And the class RunThread is as follows:
class RunThread(QtCore.QObject):
finished = QtCore.pyqtSignal()
def __init__(self):
super().__init__()
#pyqtSlot()
def long_run(self):
#ML code called in the line below
simulation.start_simulation()
self.finished.emit()
Running it normally doesnt work. Pycharm quits with the following error:
process finished with exit code -1073740791 (0xc0000409)
Running it in debug mode throws thousands of warnings thrown by sklearn saying:
Multiprocessing-backed parallel loops cannot be nested below threads,
setting n_jobs=1
The code runs eventually, but it takes a lot more time (at least 4x times) to do so.
Could someone please let me know what the issue here is?
The warning from sklearn is pretty explicit here. Basically you cannot train sklearn models with hyperparameter n_jobs set to anything greater than 1 when executing from a nested thread.
That being said, without seeing what is going on inside of simulation.start_simulation() it's hard to conjecture.
My best guess would be to look for anything in start_simulation() that is using multiple jobs and see what happens when you set that to n_jobs=1.
Alternatively, you may try writing your ML code as a standalone script and execute it using QProcess. That may allow you to utilize n_jobs greater than 1.
Related
Currently i am working on a GUI program in PyQt5. My program has several steps which run after each other. After finish each step i want to print for example Step1 finished and then Step2 finished and so on. Since the main window freezes the print does not work.
It would be very helpful if you give me some solutions.
A simple solution would be a thread based solution, where you set an event flag after finishing the task:
import threading
done_flag = threading.Event()
def print_status():
while True:
done_flag.wait()
print("I'm done!")
done_flag.clear()
Inside your function, you have to call .set_flag. This functions blocks until your function have finished (if you always call set_flag).
It's by the way always good practice to use threads in GUI applications. This avoids freezing cause the main loop etc. cannot run.
By the way, instead of raw printing, I would suggest using logging which gives more useful information.
I am trying to parallelize the load of the data and the learning in a pytorch project. From main thread I create 2 threads, one loading the next batch while the other is learning from the current batch. The loading threading is transferring the loaded data through a Queue.
My problem is: the program suddenly stops at random state of execution without any error message (debug execution or not). Sometimes at the first epoch, sometimes after 7 epoch (50min)... Is it possible to have an error message somehow ? Does anyone have encountered that problem ?
It makes me think about memory leakage, but I check all the code about the shared data. I have also seen things about prints not being thread safe so I removed them... Please also note that the code didn't had this problem before parallelization.
I am using:
conda environement.
Threading.thread
pytorch
Windows server
Update: Apparently the pytorch code referencing cuda learning doesn't like to be called from a separate thread. If I keep my cuda learning in the main thread, it stays alive...
Code: Since I have less unexpected crashes with the cuda learning in the main thread. I only kept one thread. (it also makes more sense)
part of the main:
dataQueue = queue.Queue()
dataAvailable = threading.Event()
doneComputing = threading.Event()
# Create new thread
loadThread = LoadingThread(1,dataQueue)
#learningThread.daemon = True
loadThread.daemon=True
# Add threads to thread list
threads = []
threads.append(loadThread)
print(" >> Parrallel process start")
# Start new Thread
loadThread.start()
doneComputing.set()
# Learning process
for i_import in range(import_init['n_import_train']):
# Wait for data loaded
dataAvailable.wait()
data = dataQueue.get()
dataAvailable.clear()
# DO learning
net,net_prev,y_all,yhat_all = doLearning(data, i_import, net, net_prev, y_all, yhat_all)
doneComputing.set()
# Wait for all threads to complete
for t in threads:
t.join()
One interesting fact is that it seems that the program crashes more often if the model sent to cuda is heavy ... Could tat be only a cuda problem?
Currently i'm trying to use proper threading to execute a bunch of scripts.
They are sorted like that:
Main Thread (Runs the Flask app)
-Analysis Thread (Runs the analysis script which invokes all needed scripts)
-3 different functions executed as thread (Divided in 3 parts so the analysis runs quicker)
My problem is i have a global variable with the analysis thread to be able to determine after the call wether the thread is running or not. The first time it does start and running just fine. Then you can call that endpoint as often as you like it wont do anything because i return a 423 to state that the thread (the analysis) is still running. After all scripts are finished, the if clause with analysis_thread.isAlive() returns false as it should and tries to start the analysis again with analysis_thread.start() but that doesn't work, it throws an exception saying the thread is already active and can't be started twice.
Is there a way to achieve that the script can be started and while it is running it returns another code but when it is finished i can start it again ?
Thanks for reading and for all your help
Christoph
The now hopefully working solution is to never stop the thread and just let it wait.
in the analysis script i have a global variable which indicates the status it is set to False by default.
inside the function it runs two whiles:
while True:
while not thread_status:
time.sleep(30)
execution of the other scripts.
thread_status = False # to ensure the execution runs just once.
I then just set the flag to True from the Controller class so it starts executing
I am attempting to convert code from Summerfield's article on (old-style) PyQt Signals/Slots to new-style PySide code. One example is a pure console application, which I have never worked with before. Unfortunately, when I try to run it multiple times, I am told that the previous application is still running.
It is a simple app: it basically lets you set a number, and reports back if the number is new:
from PySide import QtCore
import sys
class TaxRate(QtCore.QObject):
rateChangedSig=QtCore.Signal(float)
def __init__(self):
QtCore.QObject.__init__(self)
self.rate = 17.5
def getRate(self):
return self.rate
def setRate(self, newRate):
if newRate != self.rate:
self.rate = newRate
self.rateChangedSig.emit(self.rate) #was self.emit(SIGNAL("rateChanged"), self.rate)
#QtCore.Slot() #technically not really needed
def rateChangedSlot(value):
print("Tax rate changed to {0:.2f} %".format(value))
if __name__=="__main__":
qtApp = QtCore.QCoreApplication(sys.argv) #origional had QtGui.QApplication, but there is no GUI
vat = TaxRate()
vat.rateChangedSig.connect(rateChangedSlot) #was vat.connect(vat, SIGNAL("rateChanged"), rateChanged)
vat.setRate(8.5) # A change will occur (new rate is different)
qtApp.quit()
sys.exit(qtApp.exec_())
Overall, it works as expected, except the final two lines do not kill the process. When I try to run the program twice, the second time my IDE (Spyder) always tells me that it is already running in a separate process. If I try running it from the command line, the window just hangs.
Strangely, when I comment out the last two lines I do not get this warning. This is the opposite of what I expect (based on previous experience with PySide GUI applications and the documentation for quit()).
Following the Closing a window example at Zetcode, I tried replacing qtApp.quit() with qtApp.instance().quit(), which yielded the same non-killing result.
So, how do I kill this thing?
One idea is that I shouldn't have even started it in the first place (as suggested here). Even though it is a pure console app, Summerfield's original program initializes with app=QtGui.QApplication(sys.argv), and it does not contain the last two lines. Things run fine, multiple times. However, isn't there a concern that each run would create a new process, so his program seems to be effectively multiplying processes without warning? (Note in practice I don't think this is happening on my system, so the answer seems to be 'No' for reasons I don't understand).
What is the correct way to control/initialize/kill a console app using PySide?
(This is ignoring the question, for the time being, why one would ever use PySide for a pure console application in Python as has been pointed out previously. But if anyone were to be interested in answering that separate question, I could start a separate question for it).
Potentially relevant post:
Pyside applications not closing properly
The problem is because you call QCoreApplication.quit() before you call QCoreApplication.exec_(). The call to quit is not queued up in the event loop, it happens immediately. The call to QCoreApplication.exec_() starts the event loop which only ends when a call to QCoreApplication.exit() (or QCoreApplication.quit()) is made while the event loop is running.
This is somewhat explained in the Qt documentation of QCoreApplication but it is very easy to miss.
I imagine you don't really need to call exec_() as you aren't using any events in your current code (most events are to do with window/mouse/keyboard though you might conceivably use some in the future like those generated by QTimer). It really depends what you want to do with the program in the future.
If you don't call exec_(), then your script will exit as you would normally expect any Python script to do (the only blocking function in your code is the call to exec_(), remove that and nothing keeps it running.)
I am creating an application which automates loading webpages and creating screenshots (no, I cannot use one of the existing solutions) using PySide. One part of the app gets a list of URLs and loads each URL in turn using a new QWebPage object. After the page is loaded a screenshot is taken and the QWebPage object is deleted.
Every now and then, given enough runs, I get the following error from PySide, as a RuntimeError exception:
Invalid Signal signature: loadStarted()
Failed to connect signal loadStarted().
The first line is printed to STDERR (probably by Qt?) and the second line is the Python exception.
loadStarted() is a built-in QWebPage signal, not something I created. This works 90% of the time, and I could not figure out what makes it fail occasionally.
To be honest, this app is quite unusual in design, as it hooks PySide/Qt into a uWSGI served web app - this means that for example I am not using the QApplication event loop but rather a local event loop for each page load. I am also not experienced in either Qt or Python so it's possible I'm making a lot of mistakes, but I can't figure out what those are.
I am thinking this post might have something to do with it, but I'm not sure.
Any suggestions on where to look next?
UPDATE: the signal is connected through the following code:
class MyWebPage(QWebPage):
def __init__(self, parent=None):
super(MyWebPage, self).__init__(parent)
self.loadStarted.connect(self.started)
self.loadFinished[bool].connect(self.finished)
MyWebPage objects are created as children of a different single QObject instance which is not deleted until the process shuts down. They are deleted by calling page.deleteLater() once I am done with them. Since I am running a local event loop, I trigger deferred deletions to happen after exiting local event loop by calling:
# self.eventLoop is the local event loop, which at this stage is not running
self.eventLoop.processEvents()
# self.app is the QApplication instance
self.app.sendPostedEvents(None, QEvent.DeferredDelete)
I was having the same problem (I'd get these errors every once in a while, but I couldn't consistently reproduce it). I think you may be right that it has something to do with methods not existing when you try to connect the signals to them - just to test that, I put the .connect calls in a separate method, and the errors went away. For example:
EDIT:
(a few hours later) I guess I spoke too soon - I just got the error again.
UPDATE:
(a few weeks later)
I played around with the syntax a lot and occasionally even got a RuntimeError (possibly this bug in PySide?). I'm still not completely sure why, but as the error happens inconsistently, you're probably safe in forcing it like this:
class MyWebPage(QWebPage):
def __init__(self, parent=None):
super(MyWebPage, self).__init__(parent)
success = False
while not success:
try:
success = self.loadStarted.connect(self.started)
except RuntimeError:
success = False
success = False
while not success:
try:
success = self.loadFinished[bool].connect(self.finished)
except RuntimeError:
success = False
If you really want to be safe, you could maybe keep a loop counter and just crash the program if the signal doesn't connect correctly before some threshold.
For what its worth, this and other issues were finally solved in a decent way after I upgraded to PySide 1.2.1.