It all began last night when I was making a script that required 8 or so packages including pygame.mixer which on my computer importing this takes a few seconds.
This meant that before the script even started I had to wait 10 or so seconds for all the imports to load. Because I want the script to obviously be as fast as possible could I start running the script while getting the imports with something like this:
import threading
def import_modules():
import tkinter as tk
from pygame import mixer
import json
import webbrowser
print('imports finished')
a = threading.Thread(target=import_modules)
a.start()
for i in range(10000):
print('Getting Modules')
So my question is:
Is this considered bad practice and will it cause problems?
If so are there alternatives I could use?
Or is it OK to do this?
If you are using CPython, this might not yield as much improvement as you'd expect.
CPython has a Global Interpreter Lock ("GIL") that ensures that only one thread at a time can be executing Python bytecode.
So whenever the import thread is executing Python code, the other thread is not running. The GIL is released by a thread when it is e.g. waiting on I/O. So there will be some time savings because of that.
There is a difference of opinion as to whether tkinter is truly thread-safe. It is still considered wise to run the tkinter main loop in the original thread, and to not invoke tkinter calls from other threads, because that can lead to crashes.
The GIL also can cause problems for GUI programs. If you are using a second thread for a long-running calculation, the user interface might become less responsive. There are at least two possible solutions. The first one is to split the long-running calculation up into small pieces which are each executed by a after method. The second is to run the calculation in a different process.
Follow-up questions from the comments:
is there anything else to speed up execution time?
The first thing you must to do is measure; what exactly causes the problem. Then you can look into the problem areas and try to improve them.
For example module load times. Run your app under a profiler to see how long the module loads take and why.
If pygame.mixer takes too long to load, you could use your platform's native mixer. UNIX-like operating systems generally have a /dev/mixer device, while ms-windows has different API's for it. Using those definitely won't take 10 seconds.
There is a cost associated with this: you will loose portability between operating systems.
What are the alternatives
Using multiple cores is a usual tactic to try and speed things up. Currently on CPython the only general way get code to run in parallel on multiple cores is with multiprocessing or concurrent.futures.
However it depends on the nature of your problem if this tactic can work.
If your problem involves doing the same calculations over a huge set of data, that is relatively easy to parallelize. In that case you can expect a maximal speedup roughly equivalent to the numbers of cores you use.
It could be that your problem consists of multiple steps, each of which depends on the result of a previous step. Such problems are serial in nature and are much harder to execute in parallel.
Other ways to possible speed things up could be to use another Python implementation like Pypy. Or you could use cython together with type hints to convert performance-critical parts to compiled C code.
I understand this is an old thread but i was looking for a way to minimize the loading time of my application, and wanted the user to see the gui so he can interact with it while other module being imported in background
i have read some answers suggesting a lazy import techniques, which i found complicated "for me", then i stumbled here with a suggest to use threading to import modules in background, then i gave it a shot, and found out it is the most brilliant idea that fits my needs
below is a code for an example gui application using PySimpleGUI which ask the user to enter a url and it will open it in the default browser window, the only module required to do so is webbrowser, so this job could be done while other modules loading
I added comments in this code to explain mostly all parts, hope it will help someone,
tested on python 3.6, windows10.
please note: this is just a dummy code as a showcase.
# import essentials first
import PySimpleGUI as sg
import time, threading
# global variable names to reference to the imported modules, this way will
# solve the problem of importing inside a function local namespace
pg = None
js = None
wb = None
progress = 0 # for our progress bar
def importer():
# we will simulate a time consuming modules by time.sleep()
global progress
progress = 10
start = time.time()
global pg, js, wb
import pygame as pg
time.sleep(3)
print(f'done importing pygame mixer in {time.time()-start} seconds')
progress = 40
start = time.time()
import webbrowser as wb
time.sleep(2)
print(f'done importing webbrowser in {time.time()-start} seconds')
progress = 70
start = time.time()
import json as js
time.sleep(10)
print(f'done importing json in {time.time()-start} seconds')
progress = 100
print('imports finished')
# start our importer in a separate thread
threading.Thread(target=importer).start()
# main app
def main():
# window layout
layout = [[sg.Text('Enter url:', size=(15,1)), sg.Input(default_text='https://google.com', size=(31, 1), key='url')],
[sg.Text('Loading modules:', size=(15,1), key='status'),
sg.ProgressBar(max_value=100, orientation='horizontal', size=(20,10), key='progress')],
[sg.Button('Open url', disabled=True, key='open_url'), sg.Button('joysticks', disabled=True, key='joysticks'), sg.Cancel()]]
window = sg.Window('test application for lazy imports', layout=layout) # our window
while True: # main application loop
event, values = window.Read(timeout=10) # non blocking read from our gui
if event in [None, 'Cancel']:
window.Close()
break
elif event == 'open_url':
wb.open(values['url'])
print('helllooooooooooooo')
elif event == 'joysticks':
# show joystics number currently connected
pg.init()
n = pg.joystick.get_count() # Get count of joysticks
sg.Popup(f'joysticks number currently connected to computer = {n}')
# open url button is disabled by default and will be enabled once our webbrowser module imported
if wb:
window.Element('open_url').Update(disabled= False)
if pg:
window.Element('joysticks').Update(disabled= False)
# progress bar
window.Element('progress').UpdateBar(progress)
if progress >= 100:
window.Element('status').Update('Loading completed', background_color='green')
main()
Related
So basically, i have made a tkinter app that has a reminder utiliy in specific to generate notifications at the scheduled time. Everything works fine until I run the app module and another module having the notification generating function one at a time , but when I call the notification generating function intto the app module, my app doesnt work but the notification works. I want the app to run such that the notification generating function kind of runs in the background until the app module is open.
github link: https://github.com/click-boom/Trella
Looking into chatgpt i found terms like threading and multiprocessing, but i have no concept of that and still tried but didnt work.
Sure enough what you are looking for is multithreading.
Here is a simple example of how multithreading works (sorry for my lack of drawing skills).
This is how all monothread programs work. In most programming languages this is the default behaviour.
So in this example Second Task will have to wait for First Task to complete.
If you want several tasks to run concurrently, you can use multithreading.
This is how you could implement this in Python.
Monothreading:
from time import sleep
def firstTask():
time = 10
for i in range(time):
sleep(1)
print(f'I have been running for {i}s')
def secondTask():
print('All I want to do is run once')
firstTask()
secondTask()
Here, secondTask will only run after firstTask is done (i.e after 10 seconds).
Multithreading:
from threading import Thread
from time import sleep
def firstTask():
time = 10
for i in range(time):
sleep(1)
print(f'I have been running for {i}s')
def secondTask():
print('All I want to do is run once')
first_thread = Thread(target=firstTask)
second_thread = Thread(target=secondTask)
first_thread.start()
second_thread.start()
I hope this will be a help to someone !
I'm making a chess program using a monte carlo tree search which will in the future be guided by a neural network. The problem at the moment is that despite all my efforts to maximise the efficiency of the system, the engine still takes 3 minutes to play through the 800 games required for the search. To combat this I have tried to implement some multiprocessing into the system so that it can play more than one game at once.
Unfortunately, despite being in a separate file which is imported into my main function, the multiprocessing still opens extra pygame windows with my menu system.
Here is the relevant part of my code:
import multiprocessing
import time
import pygame
pygame.init()
screen = pygame.display.set_mode([800, 400])
def play_game():
time.sleep(3)
print("done")
if __name__ == "__main__":
for i in range(3):
game1 = multiprocessing.Process(target=play_game)
game2 = multiprocessing.Process(target=play_game)
game1.start()
game2.start()
game1.join()
game2.join()
I'd expect this to open the pygame window at the beginning, then run the play_games after and leave the pygame code alone.
However, it opens two extra windows, one for each process. Why is this and how can I stop it from happening?
I believe this encounters the same problem as you, and there is a solution there:
Python Multiprocessing: Execute code serially before and after parallel execution
Basically you have to move screen = pygame.display.set_mode([800, 400]) into __main__ so that it only got executed once.
Hope this somewhat helps!
In my application, I have a process which runs on another thread, takes a few seconds to complete and only needs to be done once. I also have a loading window which would let users know that the application is still running and let them cancel the process. This loading window calls a function every 0.5s to update the message : Processing., Processing.. or Processing... in a cycle.
The problem I have is that the computing time increases significantly with the loading window. Here are the 2 different implementation :
Without loading window :
processing_thread.start()
processing_thread.join()
With loading window :
processing_thread.start()
loading_window = LoadingWindow()
while processing_thread.is_alive():
try:
loading_window.window.update_idletasks()
loading_window.window.update()
except TclError:
return
Note that I don't use mainloop but an equivalent implementation which enables me the check if my process is still running - a bit like join and mainloop merged together (Tkinter understanding mainloop). I also tested it with mainloop() and it still didn't reduce the processing time in a significant way.
For now, the quick fix was to slow down the loop and add more idle time in the main thread :
processing_thread.start()
loading_window = LoadingWindow()
while processing_thread.is_alive():
try:
loading_window.window.update_idletasks()
loading_window.window.update()
time.sleep(0.5)
except TclError:
return
This reduced the time to something similar to what I have without the loading window but it brings me 2 problems (as far as I could see) :
Response time is slower (0.5s at worst)
The application will end some time after the process ended (0.5s at worst)
Is there a way to implement this without these drawbacks?
Would multiprocessing (https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing) solve this?
Thank you
I'm wanting to play a sound while some text is being printed out, but how do I go about doing this at the same time? Here is what I have to work with.
import winsound as win
import sys
import time
text = "I like Pizza!"
for letter in text:
sys.stdout.write(letter)
sys.stdout.flush()
time.sleep(0.5)
gametheme = "Place where the sound is stored"
win.PlaySound(gametheme, win.SND_FILENAME)
(Copied) Threading in python is used to run multiple threads (tasks, function calls) at the same time. Note that this does not mean that they are executed on different CPUs. Python threads will NOT make your program faster if it already uses 100 % CPU time. In that case, you probably want to look into parallel programming.
Programs run line by line, so it is difficult to run it at the same time. It is only possible with threads. A thread is another program, linked to another program, which run at the same time, side-by-side. I have never used them before, but I could tell you some websites.
docs.python.org
stackoverflow :)
I have a piece of code that displays a Gui which has a QTextEdit Field. I would like to print to this field in real time similar to how the print function outputs to the console.
I have tried using multiple instances of the append function. Ex:
self.textEdit.append(_translate("MainWindow", ">>> Text", None))
The problem is that no matter where they are in the code, they seem to only show after the program is executed. My goal is to have them show in line like the print function does on the console.
I feel like this is an easy answer, but I have had no luck searching.. I am fairly new to Python and any help or guidance will be appreciated.
Thanks in advance!
Indeed, as mata mentioned the freezing comes from doing all your work in the same (main) thread, which also handles UI updates. One way to solve your responsiveness issue is indeed to frequently use QApplication.processEvents() in your blocking code. That will give the impression of a responsive GUI to the user if frequent enough.
However, using threads in Python (whether native or QThread) will not always work. That is because of the existence of the Global Interpreter Lock (GIL, the wiki has a good short intro). In short, Python does not allow more than one thread to execute code at the same time.
If your background task is light, or is based on IO, you can get around this as most IO-heavy modules for Python release the GIL while doing their job. However, if you are performing heavy computations in Python, the GIL will be locked by your processes, and as such your UI will still be unresponsive.
Consider the following example, built using PySide:
import sys, random
from threading import Thread
from time import sleep
from urllib import urlopen
from PySide import QtCore, QtGui
class Window(QtGui.QMainWindow):
update_signal = QtCore.Signal(int)
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.progress_bar = QtGui.QProgressBar(self)
self.progress_bar.setRange(0, 10)
self.setCentralWidget(self.progress_bar)
self.update_signal[int].connect(self.progress_bar.setValue)
self.show()
self.t = Thread(target=self.worker)
self.t.start()
def worker(self):
while self.progress_bar.value() < 10:
self.update_signal.emit(self.progress_bar.value()+1)
print "Starting Sleep"
sleep(5)
print "End of Sleep"
if __name__ == '__main__':
qapp = QtGui.QApplication(sys.argv)
win = Window()
sys.exit(qapp.exec_())
Then, try replacing the worker function to:
def worker(self):
while self.progress_bar.value() < 10:
self.update_signal.emit(self.progress_bar.value()+1)
v = 0
print "Starting Add"
for i in xrange(5000000):
v = v+random.uniform(0, 100)
print "End of Add"
The first case maintains a responsive UI, as the call to sleep() releases the GIL. But the second example does not, as the computationally-intense algorithm keeps the lock.
One solution could be using the multiprocessing package.
From the docs:
multiprocessing is a package that supports spawning processes using an
API similar to the threading module. The multiprocessing package
offers both local and remote concurrency, effectively side-stepping
the Global Interpreter Lock by using subprocesses instead of threads.
Due to this, the multiprocessing module allows the programmer to fully
leverage multiple processors on a given machine.
A simple, illustrative example of using python multipocessing.
Also, if you have further interest, this blog post about multi-processing techniques might be of interest.
This means that you're probably doing all your work in the GUI thread. This is a common mistake, and it means that the GUI will freeze and not respond while there is something else going on.
You can add a call to QApplication.processEvents() to allow for the GUI to update after you change the text in your QTextEdit, but that will only partially solve the problem, the GUI will nevertheless freeze up between those calls.
The solution is simple: do the work in a separate thread. You should read Threading Basics from the Qt documentation, that should get you started.