I have two processes communicating over queues. One is altering the application state, the other one is running the pyglet event-loop. (just a side note, this is likely not a multiprocessing issue). I only want to draw when the state has changed or possibly with interpolated steps in between.
I'm confused about two things here.
I could write my own event-loop and check in there, if the state has changed. However even the minimal one from the docs is way less performant than the default app.run(). Why is that?
This is the loop from the documentation:
while True:
pyglet.clock.tick()
for window in pyglet.app.windows:
window.switch_to()
window.dispatch_events()
window.dispatch_event('on_draw')
window.flip()
If I use app.run() (for said performance reasons) and I check for state-changes inside of on_draw(), then I get weird stuttering images between frames. It looks like the window is changing back and forth between multiple frames while I'm not drawing anything. Why?
My on_draw() would look something like this:
def on_draw(self):
if self.check_if_state_changed():
self.draw_the_new_state()
I know that I could prevent the stuttering like this:
def on_draw(self):
if self.check_if_state_changed():
self.draw_the_new_state()
else:
self.draw_the_old_state_again()
But I don't want to do that, because re-drawing the old state requires time and performance is crucial for this application.
I did not find out why the minimal loop is slower than app.run(), but it seems like the documentation is contradicting itself. The recommended way to change the event-loop is actually to subclass app.EventLoop and override the idle() method.
The stuttering is due to the window being double buffered, which is probably for the better.
My solution to both problems is this, which will repeatedly call on_draw() as fast as possible:
class CustomLoop(app.EventLoop):
def idle(self):
dt = self.clock.update_time()
self.clock.call_scheduled_functions(dt)
# Redraw all windows
for window in app.windows:
window.switch_to()
window.dispatch_event('on_draw')
window.flip()
window._legacy_invalid = False
# no timout (sleep-time between idle()-calls)
return 0
app.event_loop = CustomLoop()
app.run() # locks the thread
I think their documentation should add a best-practice example for changing the event-loop.
Related
I am trying to implement a simple spinner (using code adapted from this answer) below a progress bar for a long-running function.
[######## ] x%
/ Compressing filename
I have the compression and progress bar running in the main thread of my script and the spinner running in another thread, so it can actually spin while compression takes place. However, I am using curses for both the progress bar and the spinner, and both use curses.refresh()
Sometimes the terminal will randomly output gibberish, and I'm not sure why. I think it is due to the multi-threaded nature of the spinner, as when I disable the spinner the problem goes away.
Here is the pseudocode of the spinner:
def start(self):
self.busy = True
global stdscr
stdscr = curses.initscr()
curses.noecho()
curses.cbreak()
threading.Thread(target=self.spinner_task).start()
def spinner_task(self):
while self.busy:
stdscr.addstr(1, 0, next(self.spinner_generator))
time.sleep(self.delay)
stdscr.refresh()
And here is the pseudocode for the progress bar:
progress_bar = "\r[{}] {:.0f}%".format("#" * block + " " * (bar_length - block), round(progress * 100, 0))
progress_file = " {} {}".format(s, filename)
stdscr.clrtoeol()
stdscr.addstr(1, 1, " ")
stdscr.clrtoeol()
stdscr.addstr(0, 0, progress_bar)
stdscr.addstr(1, 1, progress_file)
stdscr.refresh()
And called from main() like:
spinner.start()
for each file:
update_progress_bar
compress(file)
spinner.stop()
Why would the output sometimes become corrupted? Is it because of the separate threads? If so, any suggestions on a better way to design this?
The curses libraries that Python's curses module relies on are not threadsafe.
ncurses has a curs_threads feature, which has apparently been there since 5.7 about a decade ago. But it requires changing the way you do a few API calls, and linking against -lncursest, and it's still not trivial, and… almost nobody ever uses it.
As far as I know, no standard installer or distro package ever builds Python curses to link ncursest—even if the distro includes ncursest in the first place, which they often won't. And even if they did, there are no bindings for the threadsafe functions, so you still wouldn't be able to safely access things like setting the tabsize.
In my (possibly out-of-date, and possibly platform-limited) experience, you can nevertheless get away with things, but you need to:
Obviously only one thread can ever call stuff like getch and getmouse.
Add a global Lock, then make sure every batch of updates ends with a refresh, and the whole batch is inside the Lock.
Avoid the Python wrappers around the functionality mentioned in curs_threads—e.g., don't change the escdelay or the tabsize.
Init (and close) the screen from the main thread, before starting (after exiting) the other threads.
If at all possible, make sure you also create all of the windows you need in the main thread. (Hopefully you didn't want any dynamic popup subwindows or anything…)
But the safe way to do this is to do the same kind of thing you do with tkinter or other GUI libraries that don't understand threads. It's not identical, but the idea is similar. The simplest version is:
Move your main thread's work to another background thread.
Add a queue.Queue so that your background threads can ask for curses commands to be run. (You don't need anything complicated to represent a "command", it's just a (func, *args) tuple, because Python.)
Make the main thread loop around popping commands off that queue and calling them.
If your background threads need to call functions that return a value, obviously you need to make this slightly more complicated. You can look at how multiprocessing.dummy.AsyncResult and concurrent.futures.Future work. Or you can even steal Future for your own purposes. But you probably don't need anything as complicated as either.
If you're looping around input, you'll probably also want your main thread to do that (this means picking a "frame rate" and alternating between waiting on the queue and the input, with a timeout) and dispatch it, even if you're always dispatching to the same thread.
You could even write an mtTkinter-style wrapper that reproduces the curses interface (or even monkeypatches the curses module) but replaces each function with a call to put the function and args on a queue. But I'm not sure this would be worth the effort.
If this is the only place where you're using the curses module, the best solution will be to stop using it.
The only functionality of curses that you're really using here is its ability to clear the screen and move the cursor. This can easily be replicated by outputting the appropriate control sequences directly, e.g:
sys.stdout.write("\x1b[f\x1b[J" + progress_bar + "\n" + progress_file)
The \x1b[f sequence moves the cursor to 1,1, and \x1b[J clears all content from the cursor position to the end of the screen.
No additional calls are needed to refresh the screen, or to reset it when you're done. You can output "\x1b[f\x1b[J" again to clear the screen if you want.
This approach does, admittedly, assume that the user is using a VT100-compatible terminal. However, terminals which do not implement this standard are effectively extinct, so this is probably a safe assumption.
I'm building a Python GUI app using tkinter.
Basically I'm starting and integrating with a different thread, while communication goes using input and output queues.
In the GUI side (the main thread where tkinter's mainloop() goes) I want to add a function which will be called on every iteration of the mainloop (I'm processing and displaying information on real-time).
So my function does something like that:
def loop(self):
try:
output_type, data = wlbt.output_q.get_nowait()
pass # if got something out of the queue, display it!
except Queue.Empty:
pass
self.loop_id = self.after(1, self.loop)
While when starting the program I just call self.loop_id = self.after(1, self.loop).
So two things that bother me:
The loop function raise the CPU usage by 30%-50%. If I disable it then it's good.
I want to be able to use after_idle() to maximize the refresh-rate, but I wasn't able to just replace it - got and error.
I'm sensing there's something I don't fully understand. What can be done to address these issues?
When you call self.after(1, self.loop) you are asking for a function to be run roughly once per millisecond. It's not at all surprising that the CPU usage goes up since you are making 1000 function calls per second.
Given that humans cannot perceive that many changes, if all you're doing is updating the display then there's no reason to do that more than 20-30 times per second.
I recently downloaded fogleman's excellent "Minecraft in 500 lines" demo from https://github.com/fogleman/Craft. I used the 2to3 tool and corrected some details by hand to make it runnable under python3. I am now wondering about a thing with the call of self.clear() in the render method. This is my modified rendering method that is called every frame by pyglet:
def on_draw(self):
""" Called by pyglet to draw the canvas.
"""
frameStart = time.time()
self.clear()
clearTime = time.time()
self.set_3d()
glColor3d(1, 1, 1)
self.model.batch.draw()
self.draw_focused_block()
self.set_2d()
self.draw_label()
self.draw_reticle()
renderTime = time.time()
self.clearBuffer.append(str(clearTime - frameStart))
self.renderBuffer.append(str(renderTime - clearTime))
As you can see, I took the execution times of self.clear() and the rest of the rendering method. The call of self.clear() calls this method of pyglet, that can be found at .../pyglet/window/__init__.py:
def clear(self):
'''Clear the window.
This is a convenience method for clearing the color and depth
buffer. The window must be the active context (see `switch_to`).
'''
gl.glClear(gl.GL_COLOR_BUFFER_BIT | gl.GL_DEPTH_BUFFER_BIT)
So I basically do make a call to glClear().
I noticed some frame drops while testing the game (at 60 FPS), so I added the above code to measure the execution time of the commands, and especially that one of glClear(). I found out that the rendering itself never takes longer than 10 ms. But the duration of glClear() is a bit of a different story, here is the distribution for 3 measurements under different conditions:
Duration of glClear() under different conditions.
The magenta lines show the time limit of a frame. So everything behind the first line means there was a frame drop.
The execution time of glClear() seems to have some kind of "echo" after the first frame expires. Can you explain me why? And how can I make the call faster?
Unfortunately I am not an OpenGL expert, so I am thankful for every advice guys. ;)
Your graph is wrong. Well, at least it's not a suitable graph for the purpose of measuring performance. Don't ever trust a gl* function to execute when you tell it to, and don't ever trust it to execute as fast as you'd expect it to.
Most gl* functions aren't executed right away, they couldn't be. Remember, we're dealing with the GPU, telling it to do stuff directly is slow. So, instead, we write a to-do list (a command queue) for the GPU, and dump it into VRAM when we really need the GPU to output something. This "dump" is part of a process called synchronisation, and we can trigger one with glFlush. Though, OpenGL is user friendly (compared to, say, Vulkan, at least), and as such it doesn't rely on us to explicitly flush the command queue. Many gl* functions, which exactly depending on your graphics driver, will implicitly synchronise the CPU and GPU state, which includes a flush.
Since glClear usually initiates a frame, it is possible that your driver thinks it'd be good to perform such an implicit synchronisation. As you might imagine, synchronisation is a very slow process and blocks CPU execution until it's finished.
And this is probably what's going on here. Part of the synchronisation process is to perform memory transactions (like glBufferData, glTexImage*), which are probably queued up until they're flushed with your glClear call. Which makes sense in your example; the spikes that we can observe are probably the frames after you've uploaded a lot of block data.
But bear in mind, this is just pure speculation, and I'm not a total expert on this sort of stuff either, so don't trust me on the exact details. This page on the OpenGL wiki is a good resource on this sort of thing.
Though one thing is certain, your glClear call does not take as long as your profiler says it does. You should be using a profiler dedicated to profiling graphics.
Sometimes, I do live updates to a plot in a loop. Normally, this works fine, but when the processing within the loop takes a long time, the plot 'greys out'/sleeps, for all but the first 10 seconds of this time. This can be quite annoying, as it makes it typically not possible to distinguish the curves (I could use dotted lines of course, but...). I'm using Ubuntu, and about 10 seconds is the threshold where this starts to happen for me.
Below is some toy code to reproduce the problem, and some pictures to demonstrate what happens.
Is there an easy way to prevent this 'greying out' behaviour?
import numpy as np
import pylab as p
import time
def create_data(i):
time.sleep(10) # INCREASE THIS VALUE TO MAKE THE PLOT GREY OUT WHILE IT WAITS
return np.sin(np.arange(i) * 0.1)
def live_plot(y):
p.cla()
p.plot(y)
p.plot(y**2)
p.draw()
p.pause(0.01)
for i in xrange(1000):
y = create_data(i)
live_plot(y)
The issue is that the GUI window becomes non-responsive from the PoV of the window manager so it 'helpfully' grays it out to tell you this. There may be a setting in gnome/unity (not sure which you are using) to disable this. (system dependent, maybe impossible :( )
One solution is to push the computation to another thread/process which will allow you to use a blocking show to leave the GUI main loop responsive. (complex :( )
Another solution is to in your slow loop periodically poke the GUI loop. I think calling fig.canvas.flush_events() should be sufficient. (dirty :( )
In short, GUIs (and asynchronous stuff in general) are hard to get right.
In Ubuntu 18.04, go to "search program" and type in "compiz" in the search box. When the ccsm window comes up, look for and uncheck the "fade window" box.
I've been fighting for three hours now to get this process multithreaded, so that I can display a progress box. I finally got it working, insomuch as the process completes as expected, and all the functions call, including the ones to update the progress indicator on the window.
However, the window never actually displays. This is a PyGObject interface designed in Glade. I am not having fun.
def runCompile(obj):
compileWindow = builder.get_object("compilingWindow")
compileWindow.show_all()
pool = ThreadPool(processes=1)
async_result = pool.apply_async(compileStrings, ())
output = async_result.get()
#output = compileStrings() #THIS IS OLD
compileWindow.hide()
return output
As I mentioned, everything works well, except for the fact that the window doesn't appear. Even if I eliminate the compileWindow.hide() command, the window never shows until the process is done. In fact, the whole stupid program freezes until the process is done.
I'm at the end of my rope. Help?
(By the way, the "recommended" processes of using generators doesn't work, as I HAVE to have a return from the "long process".)
I'm not a pyGobject expert and i don't really understand your code. I think that you should post more. Why are you calling the builder in a function? you can call it at the init of the GUI?
Anyways.. It seems that you are having the common multithread problems..
are you using at the startup GObject.threads_init() and Gdk.threads_init() ?
Then, if you want to show a window from a thread you need to use Gdk.threads_enter() and Gdk.threads_leave().
here is an useful doc
I changed the overall flow of my project, so that may affect it. However, it is imperative that Gtk be given a chance to go through its own main loop, by way of...
if Gtk.events_pending():
Gtk.main_iteration()
In this instance, I only want to call it once, to ensure the program doesn't hang.
(The entire program source code can be found on SourceForge. The function in question is on line 372 as of this posting, in function compileModel().