We Keep Coding

How to use multi-threading to speed up Python imports? [duplicate]

It all began last night when I was making a script that required 8 or so packages including pygame.mixer which on my computer importing this takes a few seconds.
This meant that before the script even started I had to wait 10 or so seconds for all the imports to load. Because I want the script to obviously be as fast as possible could I start running the script while getting the imports with something like this:
import threading
def import_modules():
import tkinter as tk
from pygame import mixer
import json
import webbrowser
print('imports finished')
a = threading.Thread(target=import_modules)
a.start()
for i in range(10000):
print('Getting Modules')
So my question is:
Is this considered bad practice and will it cause problems?
If so are there alternatives I could use?
Or is it OK to do this?

If you are using CPython, this might not yield as much improvement as you'd expect.
CPython has a Global Interpreter Lock ("GIL") that ensures that only one thread at a time can be executing Python bytecode.
So whenever the import thread is executing Python code, the other thread is not running. The GIL is released by a thread when it is e.g. waiting on I/O. So there will be some time savings because of that.
There is a difference of opinion as to whether tkinter is truly thread-safe. It is still considered wise to run the tkinter main loop in the original thread, and to not invoke tkinter calls from other threads, because that can lead to crashes.
The GIL also can cause problems for GUI programs. If you are using a second thread for a long-running calculation, the user interface might become less responsive. There are at least two possible solutions. The first one is to split the long-running calculation up into small pieces which are each executed by a after method. The second is to run the calculation in a different process.
Follow-up questions from the comments:
is there anything else to speed up execution time?
The first thing you must to do is measure; what exactly causes the problem. Then you can look into the problem areas and try to improve them.
For example module load times. Run your app under a profiler to see how long the module loads take and why.
If pygame.mixer takes too long to load, you could use your platform's native mixer. UNIX-like operating systems generally have a /dev/mixer device, while ms-windows has different API's for it. Using those definitely won't take 10 seconds.
There is a cost associated with this: you will loose portability between operating systems.
What are the alternatives
Using multiple cores is a usual tactic to try and speed things up. Currently on CPython the only general way get code to run in parallel on multiple cores is with multiprocessing or concurrent.futures.
However it depends on the nature of your problem if this tactic can work.
If your problem involves doing the same calculations over a huge set of data, that is relatively easy to parallelize. In that case you can expect a maximal speedup roughly equivalent to the numbers of cores you use.
It could be that your problem consists of multiple steps, each of which depends on the result of a previous step. Such problems are serial in nature and are much harder to execute in parallel.
Other ways to possible speed things up could be to use another Python implementation like Pypy. Or you could use cython together with type hints to convert performance-critical parts to compiled C code.

I understand this is an old thread but i was looking for a way to minimize the loading time of my application, and wanted the user to see the gui so he can interact with it while other module being imported in background
i have read some answers suggesting a lazy import techniques, which i found complicated "for me", then i stumbled here with a suggest to use threading to import modules in background, then i gave it a shot, and found out it is the most brilliant idea that fits my needs
below is a code for an example gui application using PySimpleGUI which ask the user to enter a url and it will open it in the default browser window, the only module required to do so is webbrowser, so this job could be done while other modules loading
I added comments in this code to explain mostly all parts, hope it will help someone,
tested on python 3.6, windows10.
please note: this is just a dummy code as a showcase.
# import essentials first
import PySimpleGUI as sg
import time, threading
# global variable names to reference to the imported modules, this way will
# solve the problem of importing inside a function local namespace
pg = None
js = None
wb = None
progress = 0 # for our progress bar
def importer():
# we will simulate a time consuming modules by time.sleep()
global progress
progress = 10
start = time.time()
global pg, js, wb
import pygame as pg
time.sleep(3)
print(f'done importing pygame mixer in {time.time()-start} seconds')
progress = 40
start = time.time()
import webbrowser as wb
time.sleep(2)
print(f'done importing webbrowser in {time.time()-start} seconds')
progress = 70
start = time.time()
import json as js
time.sleep(10)
print(f'done importing json in {time.time()-start} seconds')
progress = 100
print('imports finished')
# start our importer in a separate thread
threading.Thread(target=importer).start()
# main app
def main():
# window layout
layout = [[sg.Text('Enter url:', size=(15,1)), sg.Input(default_text='https://google.com', size=(31, 1), key='url')],
[sg.Text('Loading modules:', size=(15,1), key='status'),
sg.ProgressBar(max_value=100, orientation='horizontal', size=(20,10), key='progress')],
[sg.Button('Open url', disabled=True, key='open_url'), sg.Button('joysticks', disabled=True, key='joysticks'), sg.Cancel()]]
window = sg.Window('test application for lazy imports', layout=layout) # our window
while True: # main application loop
event, values = window.Read(timeout=10) # non blocking read from our gui
if event in [None, 'Cancel']:
window.Close()
break
elif event == 'open_url':
wb.open(values['url'])
print('helllooooooooooooo')
elif event == 'joysticks':
# show joystics number currently connected
pg.init()
n = pg.joystick.get_count() # Get count of joysticks
sg.Popup(f'joysticks number currently connected to computer = {n}')
# open url button is disabled by default and will be enabled once our webbrowser module imported
if wb:
window.Element('open_url').Update(disabled= False)
if pg:
window.Element('joysticks').Update(disabled= False)
# progress bar
window.Element('progress').UpdateBar(progress)
if progress >= 100:
window.Element('status').Update('Loading completed', background_color='green')
main()

python tkinter window going unresponsive and hanging

MyButton1 =Button(master, text='Quit',bg="grey",width=20,
command=master.quit)
MyButton1.place(x=200, y=100)
MyButton2 =Button(master, text='Propagate', bg="grey",width=20,
command=mainmethod)
MyButton2.place(x=1000, y=100)
master.geometry("1500x1500")
master.mainloop( )
In the above code after pressing propagate button mainmethod is invoking..
I wrote my logic in main method where this method alone taking 2minutes to execute in the mean time GUI going unresponsive state for few min and later displaying all my required output on text box i inserted
whether any away to avoid the unresponsive issue apart from using multi threading
and i am looking such that after pressing propagate button button should disabled and window should not go unresponsive and display text.insert statements continuously which i added in main method ?????

To prevent hanging, you need to separate the calculations in the mainmethod from Tkinter's main loop by executing them in different threads. However, threading system in Python is not that well-developed as in other languages (AFAIK) because of GIL (Global Interpreter Lock), but there is an alternative - using processes just like threads. This is possible with multiprocessing library.
In order to just prevent hanging, you could create another function
from multiprocessing import Process
def mainmethodLaunch():
global mainmethodProcess
mainmethodProcess = Process(target=mainmethod)
mainmethodProcess.start()
And bind this function to MyButton2 instead of mainmethod itself.
Docs: https://docs.python.org/2/library/multiprocessing.html#the-process-class
You can see p.join in the example. join method will cause your main process to wait for the other one to complete, which you don't want.
So when you press the button, mainmethodLaunch function will be invoked, and it will create another process executing mainmethod. mainmethodLaunch function's own run duration should be insignificant. Due to usage of another process, Tkinter window will not hang. However, if you do just this, you will not be able to interact with mainmethod process in any kind while it will be working.
In order to let these processes communicate with each other, you could use pipes (https://docs.python.org/2/library/multiprocessing.html#exchanging-objects-between-processes)
I guess the example is quite clear.
In order to receive some data from the mainmethod process over time, you will have to poll the parent_conn once a little time, let's say, second. This can be achieved with Tkinter's after method
(tkinter: how to use after method)
IMPORTANT NOTE: when using multiprocessing, you MUST initialize the program in if __name__ == '__main__': block. I mean, there should be no doing-something code outside functions and this block, no doing-something code with zero indent.
This is because multiprocessing is going to fork the same Python executable file, and it will have to distinguish the main process from the forked one, and not do initializing stuff in the forked one.
Check twice if you have done that because if you make such a mistake, it can cost you hanging of not just Tkinter window, but the whole system :)
Because the process will be going to fork itself endlessly, consuming all RAM you have, regardless of how much you have.

Python's Tkinter after() function jumps CPU

I'm building a Python GUI app using tkinter.
Basically I'm starting and integrating with a different thread, while communication goes using input and output queues.
In the GUI side (the main thread where tkinter's mainloop() goes) I want to add a function which will be called on every iteration of the mainloop (I'm processing and displaying information on real-time).
So my function does something like that:
def loop(self):
try:
output_type, data = wlbt.output_q.get_nowait()
pass # if got something out of the queue, display it!
except Queue.Empty:
pass
self.loop_id = self.after(1, self.loop)
While when starting the program I just call self.loop_id = self.after(1, self.loop).
So two things that bother me:
The loop function raise the CPU usage by 30%-50%. If I disable it then it's good.
I want to be able to use after_idle() to maximize the refresh-rate, but I wasn't able to just replace it - got and error.
I'm sensing there's something I don't fully understand. What can be done to address these issues?

When you call self.after(1, self.loop) you are asking for a function to be run roughly once per millisecond. It's not at all surprising that the CPU usage goes up since you are making 1000 function calls per second.
Given that humans cannot perceive that many changes, if all you're doing is updating the display then there's no reason to do that more than 20-30 times per second.

PyGObject (Glade) Window Never Showing (Multithreaded)

I've been fighting for three hours now to get this process multithreaded, so that I can display a progress box. I finally got it working, insomuch as the process completes as expected, and all the functions call, including the ones to update the progress indicator on the window.
However, the window never actually displays. This is a PyGObject interface designed in Glade. I am not having fun.
def runCompile(obj):
compileWindow = builder.get_object("compilingWindow")
compileWindow.show_all()
pool = ThreadPool(processes=1)
async_result = pool.apply_async(compileStrings, ())
output = async_result.get()
#output = compileStrings() #THIS IS OLD
compileWindow.hide()
return output
As I mentioned, everything works well, except for the fact that the window doesn't appear. Even if I eliminate the compileWindow.hide() command, the window never shows until the process is done. In fact, the whole stupid program freezes until the process is done.
I'm at the end of my rope. Help?
(By the way, the "recommended" processes of using generators doesn't work, as I HAVE to have a return from the "long process".)

I'm not a pyGobject expert and i don't really understand your code. I think that you should post more. Why are you calling the builder in a function? you can call it at the init of the GUI?
Anyways.. It seems that you are having the common multithread problems..
are you using at the startup GObject.threads_init() and Gdk.threads_init() ?
Then, if you want to show a window from a thread you need to use Gdk.threads_enter() and Gdk.threads_leave().
here is an useful doc

I changed the overall flow of my project, so that may affect it. However, it is imperative that Gtk be given a chance to go through its own main loop, by way of...
if Gtk.events_pending():
Gtk.main_iteration()
In this instance, I only want to call it once, to ensure the program doesn't hang.
(The entire program source code can be found on SourceForge. The function in question is on line 372 as of this posting, in function compileModel().

How do I close a loading screen right when the other thread is finished in Python using Tk?

I have a Python GUI that uses Tkinter. I have to SSH into another place to get data. I start a new thread to do this so that the GUI doesn't hang. During this time, I want to pop up a screen that lets the user know it is loading. Once the program is finished getting the data, I want to close the loading screen. What must I do to have my main loop recognize that the thread is done? I've tried to use that thread to close the loading screen that exists in the main loop, but then I discovered that doesn't work.
I have seen some producer consumer models that don't use GUIs, and they have while loops. This doesn't help me though. I also don't want to download and install other packages, but imports are ok. Thank you for your help!

Have your thread set a flag when it is done. Have the GUI periodically check for that flag and dismiss the window when it is set.
You can check for the flag by creating a function that checks for the flag, and if it's not set it uses after to have itself run again a few hundred ms later. The window won't go away immediately after the thread exits, but as long as the lag isn't more than a couple hundred ms the user will never notice.