part-time C# programmer here trying to learn Python.
I am looking to be able to, from another thread, set up a function that will be run on the GUI thread in WxPython. Currently the only way I can think of doing this is to push the function onto some cross-thread-synchronized list, then have a timer firing every few milliseconds (on the GUI thread) that checks the list and runs any functions that have been pushed onto it. I was wondering if there was instead a more canonical way of doing this, similar to BeginInvoke in .NET.
wx.CallLater and wx.CallAfter().
Related
Whilst creating a project in python, I wanted to create a GUI, so I decided, to use the module tkinter. After trying to implement a while True loop in my code, it seems the GUI had frozen and was unusable, so I am here to ask is this a problem with the GUI modules themselves, or is it a problem with Python itself? Is this the same in other languages?
It is the nature of most GUI frameworks. For a GUI to work, it must be able to receive a steady flow of events to be processed. These events can be button clicks, typing on a keyboard, and system-generated events where the OS tells the window it needs to be refreshed.
The code that processes these events is typically called the event loop - it is a global loop that waits for events, compares them to registered handlers, and then calls the handlers. In tkinter, calling the mainloop function starts this event loop.
When you put a long or infinite loop in the same thread that the GUI is operating, while that code is running the GUI framework is unable to process these events. Thus, the GUI appears to be frozen because it can't refresh itself.
Some toolkits might run the event loop in a separate thread. That is not the case with tkinter, and maybe other python-based GUI frameworks. With tkinter, on some platforms it's best to run the event loop in the same thread that created the widgets. On OSX, this is a strict requirement.
If you need to create your own loops in a GUI program, or any long running function, it's best that you create a separate thread for any such code.
I'm a bot developer but I'm new to Python. I've been reading for hours now, planning the design of a new bot.
I'd like your opinion on performance issues with running a GUI and a very fast core loop to keep a modest array of game entities updated.
The bot consists of a main array in an infinite loop which continually updates and also runs calculations. I know from my past bots that GUIs provide a problem with performance as they need to wait to detect events.
I've looked at the possibility of a second thread, but I read that tkinter doesn't like more than one thread.
I've checked out the idea of using .after to run the core array loop from the tkinter mainloop but my gut tells me that this would be bad practice.
Right now I feel all I can do is try to contain the GUI and the core array all in one loop but this has never been good for performance.
Are there better ways of designing the structure of this bot that I have not discovered?
Edit
I decided on removing the mainloop from tinker and simply using .update() to update any gui components I have, right now, this only consists of some labels which overlay the game screen.
Both the labels and the bot's functions run great so far.
Using tkinter and .after, I wrote a little single-threaded program that displays 1000 ovals that move randomly across the screen, updating every 50ms, and I don't see any lag. At 10ms I think I maybe see a tiny bit of lag.
These are simple objects with very little math to calculate the new positions, and there's no collision detection except against the edges of the window.
The GUI seems responsive. I am able to type into a text window in the same GUI as fast as I can.
I don't know how that compares to what you want to do.
If you want to run a CPU-intensive work in a separate Python thread, it will starve other threads of CPU. Python threads are only good for efficient waiting for I/O (the cause being the Global Interpreter Lock).
I'd either try multiprocessing, which may be fine depending on the amount of data you need to pass between processes, or a different language.
Not wanting my GUI to freeze, I've decided to use a thread for a file operation. Currently I'm using thread.start_new_thread(self.openFile, (filepath, self.openedFile)) to do so, with self.openedFile being my callback method. Inside self.openFile it is just invoked using callback(success).
But unfortunately I couldn't figure out how to execute my callback in the main thread instead of the newly created one. This is required as the GUI cannot be modified from another thread.
I really appreciate all your help!
The thread owning the GUI will have to periodically check a Queue.Queue instance on which other threads can put work requests (e.g a callback function and arguments for it).
How easy or hard it is to insert such checks within a GUI's main loop is 100% dependent on exactly what GUI framework you're using.
For example if you're using Tkinter, the after method of widgets lets you do such periodic checks, as explained e.g at Run an infinite loop in the backgroung in Tkinter .
My apologies beforehand for the length of the question, I didn't want to leave anything out.
Some background information
I'm trying to automate a data entry process by writing a Python application that uses the Windows API to simulate keystrokes, mouse movement and window/control manipulation. I have to resort to this method because I do not (yet) have the security clearance required to access the datastore/database directly (e.g. using SQL) or indirectly through a better suited API. Bureaucracy, it's a pain ;-)
The data entry process involves the correction of sales orders due to changes in article availability. The unavailable articles are either removed from the order or replaced by another suitable article.
Initially I want a human to be able to monitor the automatic data entry process to make sure everything goes right. To achieve this I slow down the actions on the one hand but also inform the user of what is currently going on through a pinned window.
The actual question
To allow the user to halt the automation process I'm registering the Pause/Break key as a hotkey and in the handler I want to pause the automation functionality. However, I'm currently struggling to figure out a way to properly pause the execution of the automation functionality. When the pause function is invoked I want the automation process to stop dead in its tracks, no matter what it is doing. I don't want it to even execute another keystroke.
UPDATE [23/01]: I actually want to do more than just pause, I want to be able to communicate with the automation process while it is running and request it to pause, skip the current sales order, give up completely and perhaps even more.
Can anybody show me The Right Way (TM) to achieve what I want?
Some more information
Here's an example of how the automation works (I'm using the pywinauto library):
from pywinauto import application
app = application.Application()
app.start_("notepad")
app.Notepad.TypeKeys("abcdef")
UPDATE [25/01]: After a few days of working on my application I've noticed I don't really use pywinauto that much, right now I'm only using it for finding window and then I directly use SendKeysCtypes.SendKeys to simulate keyboard input and win32api functions to simulate mouse input.
What I've found out so far
Here are a few methods I've come across so far in my search for an answer:
I could separate the automation functionality and the interface + hotkey listener in two separate processes. Let's refer to the former as "automator" and the latter as "manager". The manager can then pause the execution of the automator by sending the process a SIGSTOP signal and unpause it using the SIGCONT signal (or the Windows equivalents through SuspendThread/ResumeThread).
To be able to update the user interface the automator will need to inform the manager of its progression through some sort of an IPC mechanism.
Cons:
Would using SIGSTOP not be a little harsh? Would it even work properly? Lots of people seem to be advising against it and even calling it "dangerous".
I am worried that implementing the IPC mechanism is going to be a bit complicated. On the other hand, I have worked with DBus which wouldn't be too hard to implement.
The second method and one that lots of people seem to be suggesting involves using threads and essentially boils down to the following (simplified):
while True:
if self.pause: # pause
# Do the work...
However, doing it this way it seems it will only pause after there is no more work to do. The only way I see this method would work would be to divide the work (the entire automation process) into smaller work segments (i.e. tasks). Before starting on a new task the worker thread would check if it should pause and wait.
Cons:
Seems like an implementation to divide the work into smaller segments, such as the one above, would be very ugly code wise (aesthetically).
The way I imagine it, all statements would be transformed to look something like: queue.put((function, args)) (e.g. queue.put((app.Notepad.TypeKeys, "abcdef"))) and you'd have the automating process thread running through the tasks and continuously checking for the pause state before starting a task. That just can't be right...
The program would not actually stop dead in its tracks, but would first finish a task (however small) before actually pausing.
Progress made
UPDATE [23/01]: I've implemented a version of my application using the first method through the mentioned SuspendThread/ResumeThread functionality. So far this seems to work very nicely and also allows me to write the automation stuff just like you'd write any other script. The only quirk I've come across is that keyboard modifiers (CTRL, ALT, SHIFT) get "stuck" while paused. Something I can probably easily work around.
I've also written a test using the second method (threads and signals/message passing) and implemented the pause functionality. However, it looks really ugly (both checking for the pause flag and everything related to the "doing the work"). So if anybody can show me a proper example of something similar to the second method I'd appreciate it.
Related questions
Pausing a process?
Pausing a thread using threading class
Alex Martelli posted an answer saying:
There is no method for other threads to forcibly pause a thread (any more than there is for other threads to kill that thread) -- the target thread must cooperate by occasionally checking appropriate "flags" (a threading.Condition might be appropriate for the pause/unpause case).
He then referred to the multiprocessing module and SIGSTOP/SIGCONT.
Is there a way to indefinitely pause a thread?
Pausing a process in Windows
An answer to this question quotes the MSDN documentation regarding SuspendThread:
This function is primarily designed for use by debuggers. It is not intended to be used for thread synchronization. Calling SuspendThread on a thread that owns a synchronization object, such as a mutex or critical section, can lead to a deadlock if the calling thread tries to obtain a synchronization object owned by a suspended thread. To avoid this situation, a thread within an application that is not a debugger should signal the other thread to suspend itself. The target thread must be designed to watch for this signal and respond appropriately.
Is there any way to kill a Thread in Python?
How do I pass an exception between threads in python
Keep in mind that although in your level of abstraction, "executing a keystroke" is a single atomic operation, it's implemented on the machine as a rather complicated sequence of machine instructions. So, pausing a thread at arbitrary points could lead to things being in an indeterminate state. Sending SIGSTOP is the same level of dangerous as pausing a thread at an arbitrary point. Depending on where you are in a particular step, though, your automation could potentially be broken. For example, if you pause in the middle of a timing-dependent step.
It seems to me that this problem would be best solved at the level of the automation library. I'm not very familiar with the automation library that you're using. It might be worth contacting the developers of the library to see if they have any suggestions for pausing the execution of automation steps at safe sub-step levels.
I don't know pywinauto. But I'll assume that you have something like an Application class which you obtain and have methods like SendKeys/SendMouseEvent/etc to do things.
Create your own MyApplication class which holds a reference to pywinauto's application class. Provide the same methods but before each method check whether a pause event has occurred. If it has, you can jump into code which handles the pause event. That way you are checking for a pause every time you cause an event, but this all is handled by the one class without putting pause all over your code.
Once you've detected the pause you can handle it any way you like. For example, you can throw an exception to force giving up on the current task.
Separating the functionality and the interface thread/process is definately the best option imho, the second solution is quicker and easier but definately not better.
Perhaps using multiple threads and an exception would be a better idea than using multiple processes. But if you're using multiple processes than SIGSTOP might be your only way to get it to work.
Is there anything against using 2 threads for this?
1 thread for actually executing
1 thread for reading the user input
I use Python but not pywinauto; for this sort of tasks I use AutoHotKey . One way to implement a simple pause in an AutoHotkey script may be using a "toggle" key like ScrollLock and testing the key state in the script. Also, the script can restore the key state after switching the internal pause setting on / off.
I have a wx.python application that takes some files and processes them when a button is clicked. I need to process them in parallel.
I use this code inside the bound button function:
my_pool = multiprocessing.Pool(POOLSIZE)
results=[digest_pool.apply_async(self.fun, [args]) for file in list_files() ]
my_pool.close()
my_pool.join()
for result in results :
print result.get()
But it seems this code is not run at all, even if I print something on fun. I didn't get any result and my GUI application got stuck. Could someone help? What is the problem here and how can I solve it using the pool multiprocessing module inside my wx frame class?
It looks like you're running up against a pretty common problem encountered by people attempting to use threading with GUI toolkits. The core of the issue is that you must never block the main GUI thread in your code. The graphical toolkit needs to be able to constantly respond to events. When you do the my_pool.join() call, you're putting the main thread to sleep and the result is that your entire process will appear to lock up.
I'm not particularly familiar with wxWidgets but I'm sure there are a few patterns out there for how to use threads with it. It's easy to spin off background threads from the GUI thread but getting the results back is usually the trick. You'll need some sort of asynchronous "work done" event that you can send to the main GUI thread when the background operation completes. Exactly how that's done differs from toolkit to toolkit. I'm not sure what the mechanism is for wxWidgets but I'd expect a quick google search would turn up an answer (or perhaps a kind commenter will provide a link ;-)