We Keep Coding

Perform modifications in the scene and update custom window from a QThread in Maya

Context
I'm creating a PySide2 tool running in Maya. The tool is executing a lot of long tasks, some modifying the scene (cleaning tasks), some creating files (exporting tasks).
Because this is a long task, I'd like to display feedback (progress bar) while it's running.
Problems
Unfortunately, so far, the whole UI does not seem to be updated during the executing.
Also, because I had odd behaviors (Maya freezing forever) in the real code, I'm guessing this is not a safe use of threads.
Example code
Here is a simplified bit of code showing where I am so far. Is this the right way to use QThread? I'm from a CG Artist background, not a professional programmer, so I'm probably misusing or misunderstanding the concepts I'm trying to use (threads, PySide...)
import time
from PySide2.QtGui import *
from PySide2.QtCore import *
from PySide2.QtWidgets import *
import maya.cmds as cmds
class Application(object):
def __init__(self):
self.view = View(self)
def do_something(self, callback):
start = int(cmds.playbackOptions(q=True, min=True))
end = int(cmds.playbackOptions(q=True, max=True))
# First operation
for frame in xrange(start, end + 1):
cmds.currentTime(frame, edit=True)
# Export ...
callback(33)
time.sleep(1)
# Second operation
for frame in xrange(start, end + 1):
cmds.currentTime(frame, edit=True)
# Export ...
callback(66)
time.sleep(1)
# Third operation
for frame in xrange(start, end + 1):
cmds.currentTime(frame, edit=True)
# Export ...
callback(100)
time.sleep(1)
class View(QWidget):
def __init__(self, controller):
super(View, self).__init__()
self.controller = controller
self.thread = None
self.setLayout(QVBoxLayout())
self.progress = QLabel()
self.layout().addWidget(self.progress)
self.button = QPushButton('Do something')
self.layout().addWidget(self.button)
self.button.clicked.connect(self.do_something)
self.show()
def do_something(self):
self.thread = DoSomethingThread(self.controller)
self.thread.updated.connect(lambda progress: self.progress.setText(str(progress) + '%'))
self.thread.run()
class DoSomethingThread(QThread):
completed = Signal()
updated = Signal(int)
def __init__(self, controller, parent=None):
super(DoSomethingThread, self).__init__(parent)
self.controller = controller
def run(self):
self.controller.do_something(self.update_progress)
self.completed.emit()
def update_progress(self, progress):
self.updated.emit(int(progress))
app = Application()

Threads are difficult to use correctly in Maya Python (you can see this from the number of questions listed here)
Generally there are two hard rules to observe:
all work that touches the Maya scene (say selecting or moving an object) has to happen in the main thread
all work that touches Maya GUI also has to happen in the main thread.
"main thread" here is the thread you get when you run a script from the listener, not on you're creating for yourself
This obviously makes a lot of things hard to do. Generally a solution will involve the a controlling operation running on the main thread while other work that does not touch Maya GUI or scene objects is happening elsewhere. A thread-safe container (like a python Queue can be used to move completed work out of a worker thread into a place where the main thread can get to it safely, or you can use QT signals to safely trigger work in the main thread.... all of which is a bit tricky if you're not far along in your programming career.
The good news is -- if all the work you want to do in Maya is in the scene you aren't losing much by not having threads. Unless the work is basically non-Maya work -- like grabbing data of the web using an HTTP request, or writing a non-Maya file to disk, or something else that does not deal with Maya-specific data -- adding threads won't get you any additional performance. It looks like your example is advancing the time line, doing work, and then trying to update a PySide GUI. For that you don't really need threads at all (you also don't need a separate QApplication -- Maya is already a QApplication)
Here's a really dumb example.
from PySide2.QtCore import *
from PySide2.QtGui import *
from PySide2.QtWidgets import *
import maya.cmds as cmds
class DumbWindow(QWidget):
def __init__(self):
super(DumbWindow, self).__init__()
#get the maya app
maya_app = QCoreApplication.instance()
# find the main window for a parent
for widget in maya_app.topLevelWidgets():
if 'TmainWindow' in widget.metaObject().className():
self.setParent(widget)
break
self.setWindowTitle("Hello World")
self.setWindowFlags(Qt.Window)
self.layout = QVBoxLayout()
self.setLayout(self.layout)
start_button = QPushButton('Start', self)
stop_button = QPushButton('Stop', self)
self.layout.addWidget(start_button)
self.layout.addWidget(stop_button)
self.should_cancel = False
self.operation = None
self.job = None
# hook up the buttons
start_button.clicked.connect(self.start)
stop_button.clicked.connect(self.stop)
def start(self):
'''kicks off the work in 'this_is_the_work'''
self.operation = self.this_is_the_work()
self.should_cancel = False
self.job = cmds.scriptJob(ie=self.this_makes_it_tick)
def stop(self):
''' cancel before the next step'''
self.should_cancel = True
def this_is_the_work(self):
print "--- started ---"
for frame in range(100):
cmds.currentTime(frame, edit=True)
yield "advanced", frame
print "--- DONE ----"
def bail(self):
self.operation = None
def kill_my_job():
cmds.scriptJob(k=self.job)
print "job killed"
cmds.scriptJob(ie = kill_my_job, runOnce=True)
def this_makes_it_tick(self):
'''
this is called whenever Maya is idle and thie
'''
# not started yet
if not self.operation:
return
# user asked to cancel
if self.should_cancel:
print "cancelling"
self.bail()
return
try:
# do one step. Here's where you can update the
# gui if you need to
result = next(self.operation)
print result
# example GUI update
self.setWindowTitle("frame %i" % result[-1])
except StopIteration:
# no more stpes, we're done
print "completed"
self.bail()
except Exception as e:
print "oops", e
self.bail()
test = DumbWindow()
test.show()
Hitting start creates a maya scriptJob that will try to run whatever operation is in the function called this_is_the_work(). It will run to the next yield statement and then check to make sure the user hasn't asked to cancel the job. Between yields Maya will be busy (just as it would if you entered some lines in the listener) but if you're interacting with Maya when a yield comes up, the script will wait for you instead. This allows for safe user interaction without a separate thread, though of course it's not as smooth as a completely separate thread either.
You'll notice that this kicks off a second scriptJob in the bail() method -- that's because a scriptJob can't kill itself, so we create another one which will run during the next idle event and kill the one we don't want.
This trick is basically how most of the Maya's MEL-based UI works under the hood -- if you run cmds.scriptJob(lj=True) in the listener you'll usually see a lot of scriptJobs that represent UI elements keeping track of things.

Adding list widget items async Pyside

I am taking photos using gphoto2 and would like to them to a list widget asynchronously as the photos are taken but for some reason it isn't working as intended. It takes the photo on a QThread but is not adding the photo to the list until all photos have been taken (like a bulk add). How would I go about this?
Here is the relevant source code (it won't compile because as there is too many dependencies to fit within the question):
class DownloadThread(QThread):
data_downloaded = Signal(object)
def __init__(self, photo_name):
QThread.__init__(self)
self.photo_name = photo_name
def run(self):
image_location = capture_image.take_photo(self.photo_name)
image = QImage(image_location)
to_pixmap = QPixmap.fromImage(image).scaled(200, 200)
to_qicon = QIcon(to_pixmap)
self.data_downloaded.emit(QListWidgetItem(to_qicon, image_location))
class MainWindow(QMainWindow, Ui_MainWindow):
def take_photo(self):
import time
for x in range(2):
photo_name = str(x) +'.jpg'
downloader = DownloadThread(photo_name)
downloader.data_downloaded.connect(self.on_photo_ready)
downloader.start()
time.sleep(5)
def on_photo_ready(self, photo):
print "WHY"
self.listWidget.addItem(photo)
I have some simple print statement in the function being called so the terminal looks like this:
Photo
Photo
Photo
Photo
Photo
Photo
WHY
WHY
WHY
WHY
WHY
WHY
Meaning it waits to actually call emit until the for loop is complete and not on its own thread as intended. Any help would be AWESOME!

You are telling the main thread to sleep while the secondary thread works. This queues all of your signals so that they arrive at once. Remove time.sleep(5) and change
downloader = ...
To
self.downloader = ...
And you should be fine.
That said, the worker model is a Good Thing. See this question or this one for details.

There are a few problems
1. Your thread isn't actually running
You need to call QThread.start() to actually run QThread.run(). That being said, you probably don't want to design your application like this. There's no reason to create dozens or hundreds of different threads -- one for each image download. It would be far more efficient to create one worker thread that downloads all the images in a queue. See below for an example.
2. You can't create QPixmaps or GUI items in a secondary thread
You can't create QPixmap's outside the main thread. You can't create QListWidgetItem's either, or any GUI element for that matter; they can only be created (and safely manipulated) in the main thread. You can use other similar elements (like QImage), but really, the only thing you need to pass back to the main thread is the downloaded filepath; the main thread can handle the QPixmap and item creation.
class DownloadWorker(QObject):
data_downloaded = Signal(object)
#QtCore.Slot(str)
def download_image(self, name):
image_location = capture_image.take_photo(name)
self.data_downloaded.emit(image_location)
class MainWindow(QMainWindow, Ui_MainWindow):
request_download = QtCore.Signal(str)
def __init__(self, ...)
...
self.worker = DownloadWorker()
self.thread = QThread(self)
self.request_download.connect(self.worker.download_image)
self.worker.data_downloaded.connect(self.on_photo_ready)
self.worker.moveToThread(self.thread)
self.thread.start()
self.timer = QTimer(self)
self.timer.timeout.connect(self.take_photo)
self.timer.start(5000)
def take_photo(self):
import time
photo_name = str(time.time()) +'.jpg'
self.request_download.emit(photo_name)
#QtCore.Slot(str)
def on_photo_ready(self, filepath):
item = QListWidgetItem(QIcon(filepath))
self.listWidget.addItem(item)

GUI not responding after child processes finishes due to redirection of Stdout

I've spent the past few days reading various threads about making tkinter thread-safe and running children without blocking the main thread. I thought I had arrived at a solution that allowed my code to run as I wanted it to, but now my main thread becomes non-responsive when my child process finishes. I can move the window around but the GUI part shows a loading cursor, whites out, and says "Not Responding" in the title of the window. I can let it sit like that forever and nothing will happen. I know what part of the code is causing the problem but I am not sure why it's causing the GUI to freeze. I'm using Windows.
I want my GUI to run another process using multiprocess. I have sys.stdout and sys.stderr routed to a queue and I use threading to create a thread that holds an automatic queue checker that updates the GUI every 100 ms so my GUI updates in "real time". I have tried every way of sending the child's stdout/stderr to the GUI and this is the only way that works the way I want it to (except for the freezing bit), so I would like to find out why it's freezing. Or I would like help setting up a proper way of sending the child's output to the GUI. I have tried every method I could find and I could not get them to work.
My main thread:
#### _______________IMPORT MODULES_________________###
import Tkinter
import multiprocessing
import sys
from threading import Thread
import qBMPchugger
###____________Widgets__________________###
class InputBox(Tkinter.Tk):
def __init__(self,parent):
Tkinter.Tk.__init__(self, parent)
self.parent = parent
self.initialize()
def initialize(self):
# Styles
self.grid()
# Approval
self.OKbutton = Tkinter.Button(self, text=u"OK", command=self.OKgo, anchor="e")
self.OKbutton.pack(side="right")
self.view = Tkinter.Text(self)
self.view.pack(side="left")
self.scroll = Tkinter.Scrollbar(self, orient=Tkinter.VERTICAL)
self.scroll.config(command=self.view.yview)
self.view.config(yscrollcommand=self.scroll.set)
self.scroll.pack(side="left")
def write(self, text):
self.view.insert("end", text)
def OKgo(self):
sys.stdout = self
sys.stderr = self
checker = Thread(target=self._update)
checker.daemon = True
checker.start()
self.view.delete(1.0, "end")
self.update_idletasks()
print("Loading user-specified inputs...")
path = "C:/"
inarg = (q, path)
print("Creating the program environment and importing modules...")
# Starts the text monitor to read output from the child process, BMPchugger
p = multiprocessing.Process(target=qBMPchugger.BMPcode, args=inarg)
p.daemon = 1
p.start()
def _update(self):
msg = q.get()
self.write(msg)
self.update_idletasks()
self.after(100, self._update)
if __name__ == "__main__":
app = InputBox(None)
app.title("File Inputs and Program Settings")
q = multiprocessing.Queue()
app.mainloop()
My child process (qBMPchugger):
#### _______________INITIALIZE_________________###
import os
import sys
import tkMessageBox
import Tkinter
class BadInput(Exception):
pass
def BMPcode(q, path):
# Create root for message boxes
boxRoot = Tkinter.Tk()
boxRoot.withdraw()
# Send outputs to the queue
class output:
def __init__(self, name, queue):
self.name = name
self.queue = queue
def write(self, msg):
self.queue.put(msg)
def flush(self):
sys.__stdout__.flush()
class error:
def __init__(self, name, queue):
self.name = name
self.queue = queue
def write(self, msg):
self.queue.put(msg)
def flush(self):
sys.__stderr__.flush()
sys.stdout = output(sys.stdout, q)
sys.stderr = error(sys.stderr, q)
print("Checking out the Spatial Analyst extension from GIS...")
# Check out extension and overwrite outputs
### _________________VERIFY INPUTS________________###
print("Checking validity of specified inputs...")
# Check that the provided file paths are valid
inputs = path
for i in inputs:
if os.path.exists(i):
pass
else:
message = "\nInvalid file path: {}\nCorrect the path name and try again.\n"
tkMessageBox.showerror("Invalid Path", message.format(i))
print message.format(i)
raise BadInput
print("Success!")
It's the part under # Send outputs to the queue (starting with the output class and ending with sys.stderr = error(sys.stderr, q)) that is causing my program to freeze. Why is that holding up my main thread when the child process finishes executing? EDIT: I think the freezing is being caused by the queue remaining open when the child process closes... or something. It's not the particular snippet of code like I thought it was. It happens even when I change the print statements to q.put("text") in either the parent or the child.
What is a better way to send the output to the queue? If you link me to a topic that answers my question, PLEASE show me how to implement it within my code. I have not been successful with anything I've found so far and chances are that I've already tried that particular solution and failed.

Use a manager list or dictionary to communicate between processes https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes . You can have a process update the dictionary and send it to the GUI/some code outside the processes, and vice versa. The following is a simple, and a little sloppy, example of doing it both ways.
import time
from multiprocessing import Process, Manager
def test_f(test_d):
""" frist process to run
exit this process when dictionary's 'QUIT' == True
"""
test_d['2'] = 2 ## add as a test
while not test_d["QUIT"]:
print "P1 test_f", test_d["QUIT"]
test_d["ctr"] += 1
time.sleep(1.0)
def test_f2(test_d):
""" second process to run. Runs until the for loop exits
"""
for j in range(0, 10):
## print to show that changes made anywhere
## to the dictionary are seen by this process
print " P2", j, test_d
time.sleep(0.5)
print "second process finished"
if __name__ == '__main__':
##--- create a dictionary via Manager
manager = Manager()
test_d = manager.dict()
test_d["ctr"] = 0
test_d["QUIT"] = False
##--- start first process and send dictionary
p = Process(target=test_f, args=(test_d,))
p.start()
##--- start second process
p2 = Process(target=test_f2, args=(test_d,))
p2.start()
##--- sleep 2 seconds and then change dictionary
## to exit first process
time.sleep(2.0)
print "\nterminate first process"
test_d["QUIT"] = True
print "test_d changed"
print "dictionary updated by processes", test_d
##--- may not be necessary, but I always terminate to be sure
time.sleep(5.0)
p.terminate()
p2.terminate()

For my particular problem, the main thread was trying to read from the queue when the queue was empty and not having anything else put into it. I don't know the exact details as to why the main loop got hung up on that thread (self._update in my code) but changing _update to the following stopped making the GUI non-responsive when the child finished:
def _update(self):
if q.empty():
pass
else:
msg = q.get()
self.write(msg)
self.update_idletasks()

how to create an extra infinite loop in an gtk/python program?

New at this, I know that the program is supposed to enter an infinite loop when you call gtk.main. The loop in gtk.main will break when self.quit is called.
But I need another loop active that would check a log file for changes, and keep updating the changes, into a gtk.Textbuffer, to be shown in a gtk.Textbox. So where can i add this loop in the following code.
class MessageBox:
def __init__(self):
builder = gtk.Builder()
builder.add_from_file("mbx.glade")
self.window = builder.get_object("window")
dic = { "on_buttonSend_clicked" : self.sendmsg,
"on_entry_activate" : self.sendmsg,
"on_buttonWhois_clicked" : self.sendwhois,
"on_buttonIdleTime_clicked" : self.sendidletime,
"on_window_destroy" : self.exitfunc}
builder.connect_signals(dic)
self.entry = builder.get_object("entry")
self.display = builder.get_object("display")
self.displaybuff=self.display.get_buffer()
def exitfunc(self, widget):
def sendmsg(self, widget):
def sendwhois (self, widget):
def sendidletime (self, widget):
if __name__ == "__main__":
msgbox = MessageBox()
msgbox.window.show()
gtk.main()

Only exists one mainloop in gtk at the same time, that is, gtk.main enters into a new level and gtk.main_quit exists from that level.
Usually, what you'd need to do is is a create worker thread that keeps working in parallel and updates the widgets when it gets new information. However, according to the problem you describe, I believe you just need to monitor a file for changes with gio as follows:
monitor = gio.File(filename).monitor()
monitor.connect('changed', file_changed_cb)
Whenever the file changes a signal will be emitted and the file_changed_cb callback method will be executed. There you can update the widgets to match the new information from the file that is being monitored. However, note that if your callback method takes too long to complete, your application might look unresponive, then you have to go for the thread strategy.

Help adding threading for GUI progress

I have an FTP function that traces the progress of running upload but my understanding of threading is limited and i have been unable to implement a working solution... I'd like to add a GUI progress bar to my current Application by using threading. Can someone show me a basic function using asynchronous threads that can be updated from another running thread?
def ftpUploader():
BLOCKSIZE = 57344 # size 56 kB
ftp = ftplib.FTP()
ftp.connect(host)
ftp.login(login, passwd)
ftp.voidcmd("TYPE I")
f = open(zipname, 'rb')
datasock, esize = ftp.ntransfercmd(
'STOR %s' % os.path.basename(zipname))
size = os.stat(zipname)[6]
bytes_so_far = 0
print 'started'
while 1:
buf = f.read(BLOCKSIZE)
if not buf:
break
datasock.sendall(buf)
bytes_so_far += len(buf)
print "\rSent %d of %d bytes %.1f%%\r" % (
bytes_so_far, size, 100 * bytes_so_far / size)
sys.stdout.flush()
datasock.close()
f.close()
ftp.voidresp()
ftp.quit()
print 'Complete...'

Here's a quick overview of threading, just in case :) I won't go into too much detail into the GUI stuff, other than to say that you should check out wxWidgets. Whenever you do something that takes a long time, like:
from time import sleep
for i in range(5):
sleep(10)
You'll notice that to the user, the entire block of code seems to take 50 seconds. In those 5 seconds, your application can't do anything like update the interface, and so it looks like it's frozen. To solve this problem, we use threading.
Usually there are two parts to this problem; the overall set of things you want to process, and the operation that takes a while, that we'd like to chop up. In this case, the overall set is the for loop and the operation we want chopped up is the sleep(10) function.
Here's a quick template for the threading code, based on our previous example. You should be able to work your code into this example.
from threading import Thread
from time import sleep
# Threading.
# The amount of seconds to wait before checking for an unpause condition.
# Sleeping is necessary because if we don't, we'll block the os and make the
# program look like it's frozen.
PAUSE_SLEEP = 5
# The number of iterations we want.
TOTAL_ITERATIONS = 5
class myThread(Thread):
'''
A thread used to do some stuff.
'''
def __init__(self, gui, otherStuff):
'''
Constructor. We pass in a reference to the GUI object we want
to update here, as well as any other variables we want this
thread to be aware of.
'''
# Construct the parent instance.
Thread.__init__(self)
# Store the gui, so that we can update it later.
self.gui = gui
# Store any other variables we want this thread to have access to.
self.myStuff = otherStuff
# Tracks the paused and stopped states of the thread.
self.isPaused = False
self.isStopped = False
def pause(self):
'''
Called to pause the thread.
'''
self.isPaused = True
def unpause(self):
'''
Called to unpause the thread.
'''
self.isPaused = False
def stop(self):
'''
Called to stop the thread.
'''
self.isStopped = True
def run(self):
'''
The main thread code.
'''
# The current iteration.
currentIteration = 0
# Keep going if the job is active.
while self.isStopped == False:
try:
# Check for a pause.
if self.isPaused:
# Sleep to let the os schedule other tasks.
sleep(PAUSE_SLEEP)
# Continue with the loop.
continue
# Check to see if we're still processing the set of
# things we want to do.
if currentIteration < TOTAL_ITERATIONS:
# Do the individual thing we want to do.
sleep(10)
# Update the count.
currentIteration += 1
# Update the gui.
self.gui.update(currentIteration,TOTAL_ITERATIONS)
else:
# Stop the loop.
self.isStopped = True
except Exception as exception:
# If anything bad happens, report the error. It won't
# get written to stderr.
print exception
# Stop the loop.
self.isStopped = True
# Tell the gui we're done.
self.gui.stop()
To call this thread, all you have to do is:
aThread = myThread(myGui,myOtherStuff)
aThread.start()