We Keep Coding

How to use multi-threading to speed up Python imports? [duplicate]

It all began last night when I was making a script that required 8 or so packages including pygame.mixer which on my computer importing this takes a few seconds.
This meant that before the script even started I had to wait 10 or so seconds for all the imports to load. Because I want the script to obviously be as fast as possible could I start running the script while getting the imports with something like this:
import threading
def import_modules():
import tkinter as tk
from pygame import mixer
import json
import webbrowser
print('imports finished')
a = threading.Thread(target=import_modules)
a.start()
for i in range(10000):
print('Getting Modules')
So my question is:
Is this considered bad practice and will it cause problems?
If so are there alternatives I could use?
Or is it OK to do this?

If you are using CPython, this might not yield as much improvement as you'd expect.
CPython has a Global Interpreter Lock ("GIL") that ensures that only one thread at a time can be executing Python bytecode.
So whenever the import thread is executing Python code, the other thread is not running. The GIL is released by a thread when it is e.g. waiting on I/O. So there will be some time savings because of that.
There is a difference of opinion as to whether tkinter is truly thread-safe. It is still considered wise to run the tkinter main loop in the original thread, and to not invoke tkinter calls from other threads, because that can lead to crashes.
The GIL also can cause problems for GUI programs. If you are using a second thread for a long-running calculation, the user interface might become less responsive. There are at least two possible solutions. The first one is to split the long-running calculation up into small pieces which are each executed by a after method. The second is to run the calculation in a different process.
Follow-up questions from the comments:
is there anything else to speed up execution time?
The first thing you must to do is measure; what exactly causes the problem. Then you can look into the problem areas and try to improve them.
For example module load times. Run your app under a profiler to see how long the module loads take and why.
If pygame.mixer takes too long to load, you could use your platform's native mixer. UNIX-like operating systems generally have a /dev/mixer device, while ms-windows has different API's for it. Using those definitely won't take 10 seconds.
There is a cost associated with this: you will loose portability between operating systems.
What are the alternatives
Using multiple cores is a usual tactic to try and speed things up. Currently on CPython the only general way get code to run in parallel on multiple cores is with multiprocessing or concurrent.futures.
However it depends on the nature of your problem if this tactic can work.
If your problem involves doing the same calculations over a huge set of data, that is relatively easy to parallelize. In that case you can expect a maximal speedup roughly equivalent to the numbers of cores you use.
It could be that your problem consists of multiple steps, each of which depends on the result of a previous step. Such problems are serial in nature and are much harder to execute in parallel.
Other ways to possible speed things up could be to use another Python implementation like Pypy. Or you could use cython together with type hints to convert performance-critical parts to compiled C code.

I understand this is an old thread but i was looking for a way to minimize the loading time of my application, and wanted the user to see the gui so he can interact with it while other module being imported in background
i have read some answers suggesting a lazy import techniques, which i found complicated "for me", then i stumbled here with a suggest to use threading to import modules in background, then i gave it a shot, and found out it is the most brilliant idea that fits my needs
below is a code for an example gui application using PySimpleGUI which ask the user to enter a url and it will open it in the default browser window, the only module required to do so is webbrowser, so this job could be done while other modules loading
I added comments in this code to explain mostly all parts, hope it will help someone,
tested on python 3.6, windows10.
please note: this is just a dummy code as a showcase.
# import essentials first
import PySimpleGUI as sg
import time, threading
# global variable names to reference to the imported modules, this way will
# solve the problem of importing inside a function local namespace
pg = None
js = None
wb = None
progress = 0 # for our progress bar
def importer():
# we will simulate a time consuming modules by time.sleep()
global progress
progress = 10
start = time.time()
global pg, js, wb
import pygame as pg
time.sleep(3)
print(f'done importing pygame mixer in {time.time()-start} seconds')
progress = 40
start = time.time()
import webbrowser as wb
time.sleep(2)
print(f'done importing webbrowser in {time.time()-start} seconds')
progress = 70
start = time.time()
import json as js
time.sleep(10)
print(f'done importing json in {time.time()-start} seconds')
progress = 100
print('imports finished')
# start our importer in a separate thread
threading.Thread(target=importer).start()
# main app
def main():
# window layout
layout = [[sg.Text('Enter url:', size=(15,1)), sg.Input(default_text='https://google.com', size=(31, 1), key='url')],
[sg.Text('Loading modules:', size=(15,1), key='status'),
sg.ProgressBar(max_value=100, orientation='horizontal', size=(20,10), key='progress')],
[sg.Button('Open url', disabled=True, key='open_url'), sg.Button('joysticks', disabled=True, key='joysticks'), sg.Cancel()]]
window = sg.Window('test application for lazy imports', layout=layout) # our window
while True: # main application loop
event, values = window.Read(timeout=10) # non blocking read from our gui
if event in [None, 'Cancel']:
window.Close()
break
elif event == 'open_url':
wb.open(values['url'])
print('helllooooooooooooo')
elif event == 'joysticks':
# show joystics number currently connected
pg.init()
n = pg.joystick.get_count() # Get count of joysticks
sg.Popup(f'joysticks number currently connected to computer = {n}')
# open url button is disabled by default and will be enabled once our webbrowser module imported
if wb:
window.Element('open_url').Update(disabled= False)
if pg:
window.Element('joysticks').Update(disabled= False)
# progress bar
window.Element('progress').UpdateBar(progress)
if progress >= 100:
window.Element('status').Update('Loading completed', background_color='green')
main()

Multithreading curses output in Python

I am trying to implement a simple spinner (using code adapted from this answer) below a progress bar for a long-running function.
[######## ] x%
/ Compressing filename
I have the compression and progress bar running in the main thread of my script and the spinner running in another thread, so it can actually spin while compression takes place. However, I am using curses for both the progress bar and the spinner, and both use curses.refresh()
Sometimes the terminal will randomly output gibberish, and I'm not sure why. I think it is due to the multi-threaded nature of the spinner, as when I disable the spinner the problem goes away.
Here is the pseudocode of the spinner:
def start(self):
self.busy = True
global stdscr
stdscr = curses.initscr()
curses.noecho()
curses.cbreak()
threading.Thread(target=self.spinner_task).start()
def spinner_task(self):
while self.busy:
stdscr.addstr(1, 0, next(self.spinner_generator))
time.sleep(self.delay)
stdscr.refresh()
And here is the pseudocode for the progress bar:
progress_bar = "\r[{}] {:.0f}%".format("#" * block + " " * (bar_length - block), round(progress * 100, 0))
progress_file = " {} {}".format(s, filename)
stdscr.clrtoeol()
stdscr.addstr(1, 1, " ")
stdscr.clrtoeol()
stdscr.addstr(0, 0, progress_bar)
stdscr.addstr(1, 1, progress_file)
stdscr.refresh()
And called from main() like:
spinner.start()
for each file:
update_progress_bar
compress(file)
spinner.stop()
Why would the output sometimes become corrupted? Is it because of the separate threads? If so, any suggestions on a better way to design this?

The curses libraries that Python's curses module relies on are not threadsafe.
ncurses has a curs_threads feature, which has apparently been there since 5.7 about a decade ago. But it requires changing the way you do a few API calls, and linking against -lncursest, and it's still not trivial, and… almost nobody ever uses it.
As far as I know, no standard installer or distro package ever builds Python curses to link ncursest—even if the distro includes ncursest in the first place, which they often won't. And even if they did, there are no bindings for the threadsafe functions, so you still wouldn't be able to safely access things like setting the tabsize.
In my (possibly out-of-date, and possibly platform-limited) experience, you can nevertheless get away with things, but you need to:
Obviously only one thread can ever call stuff like getch and getmouse.
Add a global Lock, then make sure every batch of updates ends with a refresh, and the whole batch is inside the Lock.
Avoid the Python wrappers around the functionality mentioned in curs_threads—e.g., don't change the escdelay or the tabsize.
Init (and close) the screen from the main thread, before starting (after exiting) the other threads.
If at all possible, make sure you also create all of the windows you need in the main thread. (Hopefully you didn't want any dynamic popup subwindows or anything…)
But the safe way to do this is to do the same kind of thing you do with tkinter or other GUI libraries that don't understand threads. It's not identical, but the idea is similar. The simplest version is:
Move your main thread's work to another background thread.
Add a queue.Queue so that your background threads can ask for curses commands to be run. (You don't need anything complicated to represent a "command", it's just a (func, *args) tuple, because Python.)
Make the main thread loop around popping commands off that queue and calling them.
If your background threads need to call functions that return a value, obviously you need to make this slightly more complicated. You can look at how multiprocessing.dummy.AsyncResult and concurrent.futures.Future work. Or you can even steal Future for your own purposes. But you probably don't need anything as complicated as either.
If you're looping around input, you'll probably also want your main thread to do that (this means picking a "frame rate" and alternating between waiting on the queue and the input, with a timeout) and dispatch it, even if you're always dispatching to the same thread.
You could even write an mtTkinter-style wrapper that reproduces the curses interface (or even monkeypatches the curses module) but replaces each function with a call to put the function and args on a queue. But I'm not sure this would be worth the effort.

If this is the only place where you're using the curses module, the best solution will be to stop using it.
The only functionality of curses that you're really using here is its ability to clear the screen and move the cursor. This can easily be replicated by outputting the appropriate control sequences directly, e.g:
sys.stdout.write("\x1b[f\x1b[J" + progress_bar + "\n" + progress_file)
The \x1b[f sequence moves the cursor to 1,1, and \x1b[J clears all content from the cursor position to the end of the screen.
No additional calls are needed to refresh the screen, or to reset it when you're done. You can output "\x1b[f\x1b[J" again to clear the screen if you want.
This approach does, admittedly, assume that the user is using a VT100-compatible terminal. However, terminals which do not implement this standard are effectively extinct, so this is probably a safe assumption.

python tkinter window going unresponsive and hanging

MyButton1 =Button(master, text='Quit',bg="grey",width=20,
command=master.quit)
MyButton1.place(x=200, y=100)
MyButton2 =Button(master, text='Propagate', bg="grey",width=20,
command=mainmethod)
MyButton2.place(x=1000, y=100)
master.geometry("1500x1500")
master.mainloop( )
In the above code after pressing propagate button mainmethod is invoking..
I wrote my logic in main method where this method alone taking 2minutes to execute in the mean time GUI going unresponsive state for few min and later displaying all my required output on text box i inserted
whether any away to avoid the unresponsive issue apart from using multi threading
and i am looking such that after pressing propagate button button should disabled and window should not go unresponsive and display text.insert statements continuously which i added in main method ?????

To prevent hanging, you need to separate the calculations in the mainmethod from Tkinter's main loop by executing them in different threads. However, threading system in Python is not that well-developed as in other languages (AFAIK) because of GIL (Global Interpreter Lock), but there is an alternative - using processes just like threads. This is possible with multiprocessing library.
In order to just prevent hanging, you could create another function
from multiprocessing import Process
def mainmethodLaunch():
global mainmethodProcess
mainmethodProcess = Process(target=mainmethod)
mainmethodProcess.start()
And bind this function to MyButton2 instead of mainmethod itself.
Docs: https://docs.python.org/2/library/multiprocessing.html#the-process-class
You can see p.join in the example. join method will cause your main process to wait for the other one to complete, which you don't want.
So when you press the button, mainmethodLaunch function will be invoked, and it will create another process executing mainmethod. mainmethodLaunch function's own run duration should be insignificant. Due to usage of another process, Tkinter window will not hang. However, if you do just this, you will not be able to interact with mainmethod process in any kind while it will be working.
In order to let these processes communicate with each other, you could use pipes (https://docs.python.org/2/library/multiprocessing.html#exchanging-objects-between-processes)
I guess the example is quite clear.
In order to receive some data from the mainmethod process over time, you will have to poll the parent_conn once a little time, let's say, second. This can be achieved with Tkinter's after method
(tkinter: how to use after method)
IMPORTANT NOTE: when using multiprocessing, you MUST initialize the program in if __name__ == '__main__': block. I mean, there should be no doing-something code outside functions and this block, no doing-something code with zero indent.
This is because multiprocessing is going to fork the same Python executable file, and it will have to distinguish the main process from the forked one, and not do initializing stuff in the forked one.
Check twice if you have done that because if you make such a mistake, it can cost you hanging of not just Tkinter window, but the whole system :)
Because the process will be going to fork itself endlessly, consuming all RAM you have, regardless of how much you have.

What magic prevents Tkinter programs from blocking in interactive shell?

Note: This is somewhat a follow-up on the question: Tkinter - when do I need to call mainloop?
Usually when using Tkinter, you call Tk.mainloop to run the event loop and ensure that events are properly processed and windows remain interactive without blocking.
When using Tkinter from within an interactive shell, running the main loop does not seem necessary. Take this example:
>>> import tkinter
>>> t = tkinter.Tk()
A window will appear, and it will not block: You can interact with it, drag it around, and close it.
So, something in the interactive shell does seem to recognize that a window was created and runs the event loop in the background.
Now for the interesting thing. Take the example from above again, but then in the next prompt (without closing the window), enter anything—without actually executing it (i.e. don’t press enter). For example:
>>> t = tkinter.Tk()
>>> print('Not pressing enter now.') # not executing this
If you now try to interact with the Tk window, you will see that it completely blocks. So the event loop which we thought would be running in the background stopped while we were entering a command to the interactive shell. If we send the entered command, you will see that the event loop continues and whatever we did during the blocking will continue to process.
So the big question is: What is this magic that happens in the interactive shell? What runs the main loop when we are not doing it explicitly? And why does it need to halt when we enter commands (instead of halting when we execute them)?
Note: The above works like this in the command line interpreter, not IDLE. As for IDLE, I assume that the GUI won’t actually tell the underlying interpreter that something has been entered but just keep the input locally around until it’s being executed.

It's actually not being an interactive interpreter that matters here, but waiting for input on a TTY. You can get the same behavior from a script like this:
import tkinter
t = tkinter.Tk()
input()
(On Windows, you may have to run the script in pythonw.exe instead of python.exe, but otherwise, you don't have to do anything special.)
So, how does it work? Ultimately, the trick comes down to PyOS_InputHook—the same way the readline module works.
If stdin is a TTY, then, each time it tries to fetch a line with input(), various bits of the code module, the built-in REPL, etc., Python calls any installed PyOS_InputHook instead of just reading from stdin.
It's probably easier to understand what readline does: it tries to select on stdin or similar, looping for each new character of input, or every 0.1 seconds, or every signal.
What Tkinter does is similar. It's more complicated because it has to deal with Windows, but on *nix it's doing something pretty similar to readline. Except that it's calling Tcl_DoOneEvent each time through the loop.
And that's the key. Calling Tcl_DoOneEvent repeatedly is exactly the same thing that mainloop does.
(Threads make everything more complicated, of course, but let's assume you haven't created any background threads. In your real code, if you want to create background threads, you'll just have a thread for all the Tkinter stuff that blocks on mainloop anyway, right?)
So, as long as your Python code is spending most of its time blocked on TTY input (as the interactive interpreter usually is), the Tcl interpreter is chugging along and your GUI is responding. If you make the Python interpreter block on something other than TTY input, the Tcl interpreter is not running and the your GUI is not responding.
What if you wanted to do the same thing manually in pure Python code? You'd of need to do that if you want to, e.g., integrate a Tkinter GUI and a select-based network client into a single-threaded app, right?
That's easy: Drive one loop from the other.
You can select with a timeout of 0.02s (the same timeout the default input hook uses), and call t.dooneevent(Tkinter.DONT_WAIT) each time through the loop.
Or, alternatively, you can let Tk drive by calling mainloop, but use after and friends to make sure you call select often enough.

Threaded Tkinter script crashes when creating the second Toplevel widget

I have a Python script which uses Tkinter for the GUI. My little script should create a Toplevel widget every X seconds. When I run my code, the first Toplevel widget is created successfully, but when it tries to create a second one the program crashes.
What I am doing is using the after method to call the function startCounting every 5 seconds alongside root's mainloop. Every time this function is called, I append a Toplevel widget object into a list and start a new thread which hopefully will be running the new mainloop.
I would be very grateful if someone could figure this problem out. By the way, this is just a little script that I am currently using to solve my problem, which is preventing me from going on with my real school project.
The code:
import threading,thread
from Tkinter import *
def startCounting():
global root
global topLevelList
global classInstance
topLevelList.append (Toplevel())
topLevelList[len(topLevelList)-1].title("Child")
classInstance.append(mainLoopThread(topLevelList[len(topLevelList)-1]))
root.after(5000,startCounting)
class mainLoopThread(threading.Thread):
def __init__(self,toplevelW):
self.toplevelW = toplevelW
threading.Thread.__init__(self)
self.start()
def run(self):
self.toplevelW.mainloop()
global classInstance
classInstance = []
global topLevelList
topLevelList = []
global root
root = Tk()
root.title("Main")
startCounting()
root.mainloop()

Tkinter is designed to run from the main thread, only. See the docs:
Just run all UI code in the main
thread, and let the writers write to a
Queue object; e.g.
...and a substantial example follows, showing secondary threads writing requests to a queue, and the main loop being exclusively responsible for all direct interactions with Tk.
Many objects and subsystems don't like receiving requests from multiple various threads, and in the case of GUI toolkit it's not rare to need specfically to use the main thread only.
The right Python architecture for this issue is always to devote a thread (the main one, if one must) to serving the finicky object or subsystem; every other thread requiring interaction with said subsystem or object must them obtain it by queueing requests to the dedicated thread (and possibly waiting on a "return queue" for results, if results are required as a consequence of some request). This is also a very sound Python architecture for general-purpose threading (and I expound on it at length in "Python in a Nutshell", but that's another subject;-).

Tkinter has issues dealing with input from multiple threads, I use mtTkinter instead, you won't need to change any code and everything will work fine. Just import mtTkinter instead of Tkinter.
You can get it here:
http://tkinter.unpythonic.net/wiki/mtTkinter

Is there a reason you want (or think you need) one event loop per toplevel window? A single event loop is able to handle dozens (if not hundreds or thousands) of toplevel windows. And, as has been pointed out in another answer, you can't run this event loop in a separate thread.
So, to fix your code you need to only use a single event loop, and have that run in the main thread.