I built a small system tray app based on Simon Brunning's example.
According to How can I use win32gui in order to have an app run in the background?,
calling win32gui.PumpMessages() runs a loop by itself, but I need to make an HTTP request every ~15 min and if necessary update the system tray icon and tooltip.
I do this by creating a separate process with an infinite loop, sleeping for 900 s after every request, which updates a dictionary shared between the two (main app, requesting) processes. The main app runs an infinite loop checking the shared dictionary for updates and win32gui.PumpWaitingMessages() on every cycle, which I would like to keep so I can get a tooltip immediately on hovering or the context menu on clicking.
This is a simplified version of the relevant part of the code, with hopefully sufficient detail to show how it works:
from multiprocessing import Process, Manager
def get_data(data):
while True:
new_data = make_request()
data["value"] = new_data
data["refresh"] = True
sleep(900)
class SysTrayIcon():
def __init__():
self.data = Manager().dict({"value": "foo", "refresh": False})
self.update_process(target=get_data, args=(self.data))
self.update_process.start()
while True:
if self.data["refresh"]:
self.refresh_icon()
self.data["refresh"] = False
win32gui.PumpWaitingMessages()
I see a pretty high CPU load (~10%) which is probably due to the loop running very quickly and checking self.data["refresh"] repeatedly.
What would be an elegant way to avoid this and/or reduce CPU load? I could probably just add a simple sleep(0.03) on every iteration which would still give me a sufficiently quick response for interactions with the app, but maybe there is a better solution?
Related
I am trying to do an app using tkinter that gets information from some internet sources after entering stuff in an input box and then clicking a "load" button.
The button function reads the stuff string from the input box. After its loaded it retrieves some info from the internet (which is blocking), and then update some labels with this information.
Obviously, when clicking the load button, the app freezes a micro second due to the requests blocking the flow of the program.
Once the info is retrieved, and the labels updated, some other labels would need to keep retrieving data from the internet constantly. To do this I have them done a bit like this:
Note: the print statements are done for testing, so I can see them on the console
def update_price_label(self):
# TODO fix .after() duplicate
print("Updating stuff")
price = self.get_formatted_price(self.stuff) # this is another function being called, passing an argument of the stuff that has been loaded by the load button, this function returns price
self.PriceValue.configure(text=price) # updates the price label with the price obtained from the function above
self.PriceValue.after(1000, self.update_price_label) # sets a scheduler to run this function to update the price label each second
Above there is a function that is called upon clicking "load" for a label that needs to be updated all the time, this function calls another function that receives an argument, and then returns the price. Then the label is updated, and then its scheduled in an endless loop using the after() method of the priceValue label control. This way, the price gets updated all the time
def get_formatted__price(self, stuff):
price = RETRIEVE PRICE # this is not a real function, but this is the request being done to the server to get the price
return f"{price:.12f} # sets decimal numbers
This function is called by update_price_label(), receives an argument and returns the price
As you can see I have divided the label update functions vs the actual functions that actually retrieve the info from a server. So the first function is responsible for calling another function to retrieve the information, update the label, and then reschedule itself again using the after() method of each label widget.
There are like 5 functions like this that need to update several labels on the app, connecting to different sources and keeping the information up to date. All of them are scheduled using after() and run on the same intervals (1 second).
Obviously, due to not using any threading, the app freezes a lot when information is being requested due to them being blocking in nature.
So I need to implement threading, or any form of concurrency. And I am unable to find any good tutorials on this, or at least that fits my needs for an app that fetches information from sources on a regular basis.
I am still grasping the concepts of threading and concurrency in general, and maybe there are other ways such as asynchronism or other methods of concurrency that I don't know yet and might be better suited. But Threading seems to be the one that seems to be mostly used with Tkinter.
I assume each of these request functions would need a thread. Like this:
get_formatted_price_thread = Thread(target=get_formatted_price, args=(stuff), daemon=True) # calling a thread with an argument on the function and setting daemon to true, so it closes when I close the app
So I have tried to create threads on one of them as an example and I have found a few limitations such as:
There's no way to get the returned value of the get_formatted_price() function directly. So the other way could be just letting the function in a thread to change the label value. Or wrapping the whole label update function in a thread. But as I read everywhere, Tkinter is not thread safe. Meaning that updating tkinter widgets may work fine on some operative systems but not in others.
Besides that, I seem to struggle on to how to turn the structure of this app to something that works well with threading or any kind of concurrency. As I need to have all the labels updated as soon as new info is retrieved.
My other concern is that threads are controlled by the operative system. As it is the one that decides when threads are launched, and how that could impact my apps performance when fetching data.
I have also checked queues and the Queue library, but I am not sure if this what would help me, as each price update would be put into the queue by the thread, and retrieved by the label widget. But the information could be outdated as the queue gets the first element of the queue.
So my question here is what would I need to change here to achieve what I need. And if threading is what I need to go on, or if maybe I would need to try another approach.
Any example source code of apps that do what I need to would be very appreciated. After all, retrieving information, and keeping widgets up to date with that information should be like a pretty common use case.
Another approach I have thought of is creating a data structure, such as a Python dictionary, or an object. Each server fetching function would run in a thread in an endless loop as a daemon, and would write to the dictionary. Then the label widget update functions, since they are scheduled, would read the data on the dictionary and update the labels accordingly. But I think this approach might be messy, and probably there would be a delay on updating the labels vs the information on the dictionaries, unless a smaller after() scheduler timer is set. Or maybe all solutions are messy by default
Thank you.
I would solve this by creating a data structure, creating a function that can update the display based on the current values in the data structure, and then binding to an event that calls this function. Then, create a thread that updates this data structure and emits the event when the data changes.
Here's a contrived example that calls a web service once a second and updates a simple data structure with the time and timezone information. Whenever the data changes it emits a <<Tick>> event that triggers an update of the display.
I'm not an expert on writing threaded tkinter code, and it's my understanding that except in a very few circumstances it is unsafe to run any tkinter code in a thread other than the one where the widgets were created. One exception is that it's safe for additional threads to generate events, since the events themselves get handled in the main GUI thread. I'm guessing it's also safe to call the winfo_exists function since it doesn't modify any internal data structures.
This example kills itself after 10 seconds so as to not hammer the server for too long.
import requests
import tkinter as tk
from tkinter.font import Font
from threading import Thread
import time
class ThreadedClock(tk.Frame):
data = {"time": "", "tz": ""}
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.time_label = tk.Label(self, width=12, font=Font(size=32))
self.tz_label = tk.Label(self, text="GMT")
self.time_label.pack(side="top", fill="x")
self.tz_label.pack(side="top", fill="x")
# call the refresh function on every <<Tick>> event
self.bind("<<Tick>>", self.refresh)
# start a thread to update the data and generate <<Tick>> events
self.thread = Thread(target=self.get_data, daemon=True)
self.running = True
self.thread.start()
def get_data(self):
while self.winfo_exists():
now = time.time()
response = requests.get(
"https://timeapi.io/api/Time/current/zone?timeZone=GMT"
)
t = response.json()
timestr = f"{t['hour']:02}:{t['minute']:02}:{t['seconds']:02}"
self.data = {"time": timestr, "tz": t["timeZone"]}
self.event_generate("<<Tick>>")
delta = time.time() - now
time.sleep(delta)
def refresh(self, event=None):
self.time_label.configure(text=self.data["time"])
self.tz_label.configure(text=f"timezone: {self.data['tz']}")
if __name__ == "__main__":
root = tk.Tk()
root.after(10000, root.destroy)
clock = ThreadedClock(root)
clock.pack(fill="both", expand=True)
root.mainloop()
It all began last night when I was making a script that required 8 or so packages including pygame.mixer which on my computer importing this takes a few seconds.
This meant that before the script even started I had to wait 10 or so seconds for all the imports to load. Because I want the script to obviously be as fast as possible could I start running the script while getting the imports with something like this:
import threading
def import_modules():
import tkinter as tk
from pygame import mixer
import json
import webbrowser
print('imports finished')
a = threading.Thread(target=import_modules)
a.start()
for i in range(10000):
print('Getting Modules')
So my question is:
Is this considered bad practice and will it cause problems?
If so are there alternatives I could use?
Or is it OK to do this?
If you are using CPython, this might not yield as much improvement as you'd expect.
CPython has a Global Interpreter Lock ("GIL") that ensures that only one thread at a time can be executing Python bytecode.
So whenever the import thread is executing Python code, the other thread is not running. The GIL is released by a thread when it is e.g. waiting on I/O. So there will be some time savings because of that.
There is a difference of opinion as to whether tkinter is truly thread-safe. It is still considered wise to run the tkinter main loop in the original thread, and to not invoke tkinter calls from other threads, because that can lead to crashes.
The GIL also can cause problems for GUI programs. If you are using a second thread for a long-running calculation, the user interface might become less responsive. There are at least two possible solutions. The first one is to split the long-running calculation up into small pieces which are each executed by a after method. The second is to run the calculation in a different process.
Follow-up questions from the comments:
is there anything else to speed up execution time?
The first thing you must to do is measure; what exactly causes the problem. Then you can look into the problem areas and try to improve them.
For example module load times. Run your app under a profiler to see how long the module loads take and why.
If pygame.mixer takes too long to load, you could use your platform's native mixer. UNIX-like operating systems generally have a /dev/mixer device, while ms-windows has different API's for it. Using those definitely won't take 10 seconds.
There is a cost associated with this: you will loose portability between operating systems.
What are the alternatives
Using multiple cores is a usual tactic to try and speed things up. Currently on CPython the only general way get code to run in parallel on multiple cores is with multiprocessing or concurrent.futures.
However it depends on the nature of your problem if this tactic can work.
If your problem involves doing the same calculations over a huge set of data, that is relatively easy to parallelize. In that case you can expect a maximal speedup roughly equivalent to the numbers of cores you use.
It could be that your problem consists of multiple steps, each of which depends on the result of a previous step. Such problems are serial in nature and are much harder to execute in parallel.
Other ways to possible speed things up could be to use another Python implementation like Pypy. Or you could use cython together with type hints to convert performance-critical parts to compiled C code.
I understand this is an old thread but i was looking for a way to minimize the loading time of my application, and wanted the user to see the gui so he can interact with it while other module being imported in background
i have read some answers suggesting a lazy import techniques, which i found complicated "for me", then i stumbled here with a suggest to use threading to import modules in background, then i gave it a shot, and found out it is the most brilliant idea that fits my needs
below is a code for an example gui application using PySimpleGUI which ask the user to enter a url and it will open it in the default browser window, the only module required to do so is webbrowser, so this job could be done while other modules loading
I added comments in this code to explain mostly all parts, hope it will help someone,
tested on python 3.6, windows10.
please note: this is just a dummy code as a showcase.
# import essentials first
import PySimpleGUI as sg
import time, threading
# global variable names to reference to the imported modules, this way will
# solve the problem of importing inside a function local namespace
pg = None
js = None
wb = None
progress = 0 # for our progress bar
def importer():
# we will simulate a time consuming modules by time.sleep()
global progress
progress = 10
start = time.time()
global pg, js, wb
import pygame as pg
time.sleep(3)
print(f'done importing pygame mixer in {time.time()-start} seconds')
progress = 40
start = time.time()
import webbrowser as wb
time.sleep(2)
print(f'done importing webbrowser in {time.time()-start} seconds')
progress = 70
start = time.time()
import json as js
time.sleep(10)
print(f'done importing json in {time.time()-start} seconds')
progress = 100
print('imports finished')
# start our importer in a separate thread
threading.Thread(target=importer).start()
# main app
def main():
# window layout
layout = [[sg.Text('Enter url:', size=(15,1)), sg.Input(default_text='https://google.com', size=(31, 1), key='url')],
[sg.Text('Loading modules:', size=(15,1), key='status'),
sg.ProgressBar(max_value=100, orientation='horizontal', size=(20,10), key='progress')],
[sg.Button('Open url', disabled=True, key='open_url'), sg.Button('joysticks', disabled=True, key='joysticks'), sg.Cancel()]]
window = sg.Window('test application for lazy imports', layout=layout) # our window
while True: # main application loop
event, values = window.Read(timeout=10) # non blocking read from our gui
if event in [None, 'Cancel']:
window.Close()
break
elif event == 'open_url':
wb.open(values['url'])
print('helllooooooooooooo')
elif event == 'joysticks':
# show joystics number currently connected
pg.init()
n = pg.joystick.get_count() # Get count of joysticks
sg.Popup(f'joysticks number currently connected to computer = {n}')
# open url button is disabled by default and will be enabled once our webbrowser module imported
if wb:
window.Element('open_url').Update(disabled= False)
if pg:
window.Element('joysticks').Update(disabled= False)
# progress bar
window.Element('progress').UpdateBar(progress)
if progress >= 100:
window.Element('status').Update('Loading completed', background_color='green')
main()
In my application, I have a process which runs on another thread, takes a few seconds to complete and only needs to be done once. I also have a loading window which would let users know that the application is still running and let them cancel the process. This loading window calls a function every 0.5s to update the message : Processing., Processing.. or Processing... in a cycle.
The problem I have is that the computing time increases significantly with the loading window. Here are the 2 different implementation :
Without loading window :
processing_thread.start()
processing_thread.join()
With loading window :
processing_thread.start()
loading_window = LoadingWindow()
while processing_thread.is_alive():
try:
loading_window.window.update_idletasks()
loading_window.window.update()
except TclError:
return
Note that I don't use mainloop but an equivalent implementation which enables me the check if my process is still running - a bit like join and mainloop merged together (Tkinter understanding mainloop). I also tested it with mainloop() and it still didn't reduce the processing time in a significant way.
For now, the quick fix was to slow down the loop and add more idle time in the main thread :
processing_thread.start()
loading_window = LoadingWindow()
while processing_thread.is_alive():
try:
loading_window.window.update_idletasks()
loading_window.window.update()
time.sleep(0.5)
except TclError:
return
This reduced the time to something similar to what I have without the loading window but it brings me 2 problems (as far as I could see) :
Response time is slower (0.5s at worst)
The application will end some time after the process ended (0.5s at worst)
Is there a way to implement this without these drawbacks?
Would multiprocessing (https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing) solve this?
Thank you
I have built a desktop kivy application with some methods that perform changes in my app as well as other calculations that take some time (around 30 seconds). When I call these methods, the app freezes during their execution, which is not desirable. Is there a way to prevent the app from freezing and instead display some kind of popup with an animated gif, so that the user knows the app is processing something and did not crash/stop responding?
I have tried to use multiprocessing to run these methods in the background while an informative Popup appears, but these processes do not share memory and data structures with the main app process. So, even though the method run successfully and the popup was shown, there were no changes in the app...
Anyway, this is the multiprocessing code:
def run_in_background(self, func, *args):
def run_func(m, a):
# Execute the function with args
m(*a)
def check_process(proc, dt):
if not proc.is_alive():
Clock.unschedule(check_func)
self.dismiss_popup()
# Create background process and start it
p = multiprocessing.Process(target=run_func, args=(func, args))
p.start()
# Create waiting dialog
content = CrunchData()
# Show waiting popup
self.show_popup(title="Some title", content=content, size=(400, 300))
# Create schedulled check for process state
check_func = partial(check_process, p)
Clock.schedule_interval(check_func, .1)
Any help would be much appreciated.
You've got a good start to your problem, but polling is_alive is not what you want to do. Since you are only doing one background task and showing a dialog, you could use either threading OR multiprocessing for this. Their API is very similar.
To avoid polling, you can use a processing pool (easier to deal with) and pass a callback to pool.apply_async.
This callback will not be run on the main thread, though, so if you need to run code on the main thread you will need to schedule it in your callback.
An alternative is just to use a separate thread instead of process, and use Kivy's clock to schedule a function call at the end of the thread function. See kivy threading docs.
This SO answer is also an option for transferring data from a thread to the main thread: https://stackoverflow.com/a/22033066. It's from a Kivy dev.
I'm learning to program with PyQT4 and Python. I'm trying to code a simple app that will display the current CPU usage in a QLCD widget. For CPU usage I'm using psutils module.
The problem is that the CPU usage is not updated all the time - it only records the CPU usage at the moment the app has been launched (I'm guessing), and then it just stops. So, I'm looking for some sort of a loop equivalent that will hopefully not take too much of CPU power to process.
This is what I have so far:
self.wpCpuUsage.display(cpu_percent(interval=1))
and this is within __init__ of the QMainWindow class.
I've tried putting it in a for loop, but then it iterates over it, and basically waits for it to iterate and then executes the program.
Help?
You can use a QTimer[reference] object with a callback.
Something like that should work:
def call_this():
self.wpCpuUsage.display(cpu_percent(interval=1))
self.my_timer = QtCore.QTimer()
self.my_timer.timeout.connect(call_this)
self.my_timer.start(1000) #1 second interval