All,
In my GUI I'm using multiprocessing to run a function. But the pool start multiple GUI.
I have read that peoples add if __name__ == '__main__': in their code and it seems to work.
But I don't know if this trick will work in my case, and where I have to insert this code.
The function run_func() is launched by a button in the GUI.
How can I block this multiple start?
I have a second question:
How can I do to unimport setup at the end of the exec?
Thanks a lot !
#pyqtSlot()
def run_func():
run="""
import os
import sys
from setup import *
print('toto')
print('titi')
"""
from multiprocessing import Pool
pool = Pool(processes=4)
asyncResult = pool.apply_async(exec(run),{},{}),range(1)
You don't provide much context for your question. Anyway I made a test removing the from setup import * part from the run string. And all run well, hence, is not a PyQT problem, is more like at some point you're executing again the module/function that runs the GUI.
For the first question:
I recommend you to use a debugger and set some breakpoints where the GUI is launched, then you can use the call-stack for figureout who is calling your GUI. That way yuo will know where the 'main' code block goes, and even if it is really necessary.
As debugger use pdb with in-code breakpoints (remember you're running in multiprocess) put the line:
import pdb; pdb.set_trace()
where ever yuo want to set a breakpoint.
For the second question:
See, How do I unload (reload) a Python module?
Related
So basically, i have made a tkinter app that has a reminder utiliy in specific to generate notifications at the scheduled time. Everything works fine until I run the app module and another module having the notification generating function one at a time , but when I call the notification generating function intto the app module, my app doesnt work but the notification works. I want the app to run such that the notification generating function kind of runs in the background until the app module is open.
github link: https://github.com/click-boom/Trella
Looking into chatgpt i found terms like threading and multiprocessing, but i have no concept of that and still tried but didnt work.
Sure enough what you are looking for is multithreading.
Here is a simple example of how multithreading works (sorry for my lack of drawing skills).
This is how all monothread programs work. In most programming languages this is the default behaviour.
So in this example Second Task will have to wait for First Task to complete.
If you want several tasks to run concurrently, you can use multithreading.
This is how you could implement this in Python.
Monothreading:
from time import sleep
def firstTask():
time = 10
for i in range(time):
sleep(1)
print(f'I have been running for {i}s')
def secondTask():
print('All I want to do is run once')
firstTask()
secondTask()
Here, secondTask will only run after firstTask is done (i.e after 10 seconds).
Multithreading:
from threading import Thread
from time import sleep
def firstTask():
time = 10
for i in range(time):
sleep(1)
print(f'I have been running for {i}s')
def secondTask():
print('All I want to do is run once')
first_thread = Thread(target=firstTask)
second_thread = Thread(target=secondTask)
first_thread.start()
second_thread.start()
I hope this will be a help to someone !
It all began last night when I was making a script that required 8 or so packages including pygame.mixer which on my computer importing this takes a few seconds.
This meant that before the script even started I had to wait 10 or so seconds for all the imports to load. Because I want the script to obviously be as fast as possible could I start running the script while getting the imports with something like this:
import threading
def import_modules():
import tkinter as tk
from pygame import mixer
import json
import webbrowser
print('imports finished')
a = threading.Thread(target=import_modules)
a.start()
for i in range(10000):
print('Getting Modules')
So my question is:
Is this considered bad practice and will it cause problems?
If so are there alternatives I could use?
Or is it OK to do this?
If you are using CPython, this might not yield as much improvement as you'd expect.
CPython has a Global Interpreter Lock ("GIL") that ensures that only one thread at a time can be executing Python bytecode.
So whenever the import thread is executing Python code, the other thread is not running. The GIL is released by a thread when it is e.g. waiting on I/O. So there will be some time savings because of that.
There is a difference of opinion as to whether tkinter is truly thread-safe. It is still considered wise to run the tkinter main loop in the original thread, and to not invoke tkinter calls from other threads, because that can lead to crashes.
The GIL also can cause problems for GUI programs. If you are using a second thread for a long-running calculation, the user interface might become less responsive. There are at least two possible solutions. The first one is to split the long-running calculation up into small pieces which are each executed by a after method. The second is to run the calculation in a different process.
Follow-up questions from the comments:
is there anything else to speed up execution time?
The first thing you must to do is measure; what exactly causes the problem. Then you can look into the problem areas and try to improve them.
For example module load times. Run your app under a profiler to see how long the module loads take and why.
If pygame.mixer takes too long to load, you could use your platform's native mixer. UNIX-like operating systems generally have a /dev/mixer device, while ms-windows has different API's for it. Using those definitely won't take 10 seconds.
There is a cost associated with this: you will loose portability between operating systems.
What are the alternatives
Using multiple cores is a usual tactic to try and speed things up. Currently on CPython the only general way get code to run in parallel on multiple cores is with multiprocessing or concurrent.futures.
However it depends on the nature of your problem if this tactic can work.
If your problem involves doing the same calculations over a huge set of data, that is relatively easy to parallelize. In that case you can expect a maximal speedup roughly equivalent to the numbers of cores you use.
It could be that your problem consists of multiple steps, each of which depends on the result of a previous step. Such problems are serial in nature and are much harder to execute in parallel.
Other ways to possible speed things up could be to use another Python implementation like Pypy. Or you could use cython together with type hints to convert performance-critical parts to compiled C code.
I understand this is an old thread but i was looking for a way to minimize the loading time of my application, and wanted the user to see the gui so he can interact with it while other module being imported in background
i have read some answers suggesting a lazy import techniques, which i found complicated "for me", then i stumbled here with a suggest to use threading to import modules in background, then i gave it a shot, and found out it is the most brilliant idea that fits my needs
below is a code for an example gui application using PySimpleGUI which ask the user to enter a url and it will open it in the default browser window, the only module required to do so is webbrowser, so this job could be done while other modules loading
I added comments in this code to explain mostly all parts, hope it will help someone,
tested on python 3.6, windows10.
please note: this is just a dummy code as a showcase.
# import essentials first
import PySimpleGUI as sg
import time, threading
# global variable names to reference to the imported modules, this way will
# solve the problem of importing inside a function local namespace
pg = None
js = None
wb = None
progress = 0 # for our progress bar
def importer():
# we will simulate a time consuming modules by time.sleep()
global progress
progress = 10
start = time.time()
global pg, js, wb
import pygame as pg
time.sleep(3)
print(f'done importing pygame mixer in {time.time()-start} seconds')
progress = 40
start = time.time()
import webbrowser as wb
time.sleep(2)
print(f'done importing webbrowser in {time.time()-start} seconds')
progress = 70
start = time.time()
import json as js
time.sleep(10)
print(f'done importing json in {time.time()-start} seconds')
progress = 100
print('imports finished')
# start our importer in a separate thread
threading.Thread(target=importer).start()
# main app
def main():
# window layout
layout = [[sg.Text('Enter url:', size=(15,1)), sg.Input(default_text='https://google.com', size=(31, 1), key='url')],
[sg.Text('Loading modules:', size=(15,1), key='status'),
sg.ProgressBar(max_value=100, orientation='horizontal', size=(20,10), key='progress')],
[sg.Button('Open url', disabled=True, key='open_url'), sg.Button('joysticks', disabled=True, key='joysticks'), sg.Cancel()]]
window = sg.Window('test application for lazy imports', layout=layout) # our window
while True: # main application loop
event, values = window.Read(timeout=10) # non blocking read from our gui
if event in [None, 'Cancel']:
window.Close()
break
elif event == 'open_url':
wb.open(values['url'])
print('helllooooooooooooo')
elif event == 'joysticks':
# show joystics number currently connected
pg.init()
n = pg.joystick.get_count() # Get count of joysticks
sg.Popup(f'joysticks number currently connected to computer = {n}')
# open url button is disabled by default and will be enabled once our webbrowser module imported
if wb:
window.Element('open_url').Update(disabled= False)
if pg:
window.Element('joysticks').Update(disabled= False)
# progress bar
window.Element('progress').UpdateBar(progress)
if progress >= 100:
window.Element('status').Update('Loading completed', background_color='green')
main()
How do I make a function which can restart my entire program, without using a while loop......
I have tried and tried again (many times) and have not been able to make it work with a GUI (tkinter)
You can use execl, which replaces the current process with a new process. You can give it the exact same arguments as the current process, effectively restarting your program:
import sys, os
python = sys.executable
os.execl(python, python, * sys.argv)
Pay close attention to the documentation: this function does not return, and it doesn't close any connections or open files. Make sure you prepare your app for an immediate exit before calling.
You can use os.execl for this purpose.
There's a snippet over here
You can find more information about those functions in the documentation
I’m trying to write a program in Python. What I want to write is a script which immediately returns a friendly message to the user, but spawns a long subprocess in the background that takes with several different files and writes them to a granddaddy file. I’ve done several tutorials on threading and processing, but what I’m running into is that no matter what I try, the program waits and waits until the subprocess is done before it displays the aforementioned friendly message to the user. Here’s what I’ve tried:
Threading example:
#!/usr/local/bin/python
import cgi, cgitb
import time
import threading
class TestThread(threading.Thread):
def __init__(self):
super(TestThread, self).__init__()
def run(self):
time.sleep(5)
fileHand = open('../Documents/writable/output.txt', 'w')
fileHand.write('Big String Goes Here.')
fileHand.close()
print 'Starting Program'
thread1 = TestThread()
#thread1.daemon = True
thread1.start()
I’ve read these SO posts on multithreading
How to use threading in Python?
running multiple threads in python, simultaneously - is it possible?
How do threads work in Python, and what are common Python-threading specific pitfalls?
The last of these says that running threads concurrently in Python is actually not possible. Fair enough. Most of those posts also mention the multiprocessing module, so I’ve read up on that, and it seems fairly straightforward. Here’s the some of the resources I’ve found:
How to run two functions simultaneously
Python Multiprocessing Documentation Example
https://docs.python.org/2/library/multiprocessing.html
So here’s the same example translated to multiprocessing:
#!/usr/local/bin/python
import time
from multiprocessing import Process, Pipe
def f():
time.sleep(5)
fileHand = open('../Documents/writable/output.txt', 'w')
fileHand.write('Big String Goes Here.')
fileHand.close()
if __name__ == '__main__':
print 'Starting Program'
p = Process(target=f)
p.start()
What I want is for these programs to immediately print ‘Starting Program’ (in the web-browser) and then a few seconds later a text file shows up in a directory to which I’ve given write privileges. However, what actually happens is that they’re both unresponsive for 5 seconds and then they print ‘Starting Program’ and create the text file at the same time. I know that my goal is possible because I’ve done it in PHP, using this trick:
//PHP
exec("php child_script.php > /dev/null &");
And I figured it would be possible in Python. Please let me know if I’m missing something obvious or if I’m thinking about this in the completely wrong way. Thanks for your time!
(System information: Python 2.7.6, Mac OSX Mavericks. Python installed with homebrew. My Python scripts are running as CGI executables in Apache 2.2.26)
Ok- I think I found the answer. Part of it was my own misunderstanding. A python script can't simply return message to a client-side (ajax) program but still be executing a big process. The very act of responding to the client means that the program has finished, threads and all. The solution, then, is to use the python version of this PHP trick:
//PHP
exec("php child_script.php > /dev/null &");
And in Python:
#Python
subprocess.call(" python worker.py > /dev/null &", shell=True)
It starts an entirely new process outside the current one, and it will continue after the current one has ended. I'm going to stick with Python because at least we're using a civilized api function to start the worker script instead of the exec function, which always made me uncomfortable.
I have a piece of code that displays a Gui which has a QTextEdit Field. I would like to print to this field in real time similar to how the print function outputs to the console.
I have tried using multiple instances of the append function. Ex:
self.textEdit.append(_translate("MainWindow", ">>> Text", None))
The problem is that no matter where they are in the code, they seem to only show after the program is executed. My goal is to have them show in line like the print function does on the console.
I feel like this is an easy answer, but I have had no luck searching.. I am fairly new to Python and any help or guidance will be appreciated.
Thanks in advance!
Indeed, as mata mentioned the freezing comes from doing all your work in the same (main) thread, which also handles UI updates. One way to solve your responsiveness issue is indeed to frequently use QApplication.processEvents() in your blocking code. That will give the impression of a responsive GUI to the user if frequent enough.
However, using threads in Python (whether native or QThread) will not always work. That is because of the existence of the Global Interpreter Lock (GIL, the wiki has a good short intro). In short, Python does not allow more than one thread to execute code at the same time.
If your background task is light, or is based on IO, you can get around this as most IO-heavy modules for Python release the GIL while doing their job. However, if you are performing heavy computations in Python, the GIL will be locked by your processes, and as such your UI will still be unresponsive.
Consider the following example, built using PySide:
import sys, random
from threading import Thread
from time import sleep
from urllib import urlopen
from PySide import QtCore, QtGui
class Window(QtGui.QMainWindow):
update_signal = QtCore.Signal(int)
def __init__(self):
QtGui.QMainWindow.__init__(self)
self.progress_bar = QtGui.QProgressBar(self)
self.progress_bar.setRange(0, 10)
self.setCentralWidget(self.progress_bar)
self.update_signal[int].connect(self.progress_bar.setValue)
self.show()
self.t = Thread(target=self.worker)
self.t.start()
def worker(self):
while self.progress_bar.value() < 10:
self.update_signal.emit(self.progress_bar.value()+1)
print "Starting Sleep"
sleep(5)
print "End of Sleep"
if __name__ == '__main__':
qapp = QtGui.QApplication(sys.argv)
win = Window()
sys.exit(qapp.exec_())
Then, try replacing the worker function to:
def worker(self):
while self.progress_bar.value() < 10:
self.update_signal.emit(self.progress_bar.value()+1)
v = 0
print "Starting Add"
for i in xrange(5000000):
v = v+random.uniform(0, 100)
print "End of Add"
The first case maintains a responsive UI, as the call to sleep() releases the GIL. But the second example does not, as the computationally-intense algorithm keeps the lock.
One solution could be using the multiprocessing package.
From the docs:
multiprocessing is a package that supports spawning processes using an
API similar to the threading module. The multiprocessing package
offers both local and remote concurrency, effectively side-stepping
the Global Interpreter Lock by using subprocesses instead of threads.
Due to this, the multiprocessing module allows the programmer to fully
leverage multiple processors on a given machine.
A simple, illustrative example of using python multipocessing.
Also, if you have further interest, this blog post about multi-processing techniques might be of interest.
This means that you're probably doing all your work in the GUI thread. This is a common mistake, and it means that the GUI will freeze and not respond while there is something else going on.
You can add a call to QApplication.processEvents() to allow for the GUI to update after you change the text in your QTextEdit, but that will only partially solve the problem, the GUI will nevertheless freeze up between those calls.
The solution is simple: do the work in a separate thread. You should read Threading Basics from the Qt documentation, that should get you started.