In a standalone Python script, I have a functioning GLib mainloop.
# myloop.py
import gobject as GLib
def main():
mainloop = GLib.MainLoop()
# do some DBus related stuffs
mainloop.run()
if __name__ == '__main__':
main()
I now need to run this from another Python project,
where multiple processes are used to implement different features.
from multiprocessing import Process
from myloop import main as myloop_main
def do something():
# do somethng else
if __name__ == '__main__':
something_proc = Process(target = do_something)
myloop_proc = Process(target = myloop_main)
something_proc.start()
myloop_proc.start()
something_proc.join()
myloop_proc.join()
This runs without error, but the mainloop is not really running
and the feature implemented in the mainloop is not working.
How to run the GLib's mainloop in a process?
Related
I have a simple function that I intend to run in Parallel using the Python multiprocessing module. However I get the following error RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase. The error suggests that I add this:
if __name__ == '__main__':
freeze_support()
And most posts online suggest the same like this SO answer.
I added it and it works but I don't seem to understand why it's necessary for such a simple piece of code.
Code without __name__=="__main__" (throws RuntimeError)
import multiprocessing
import time
start = time.perf_counter()
def do_something():
print('Sleeping 1 second...')
time.sleep(1)
print('Done sleeping...')
p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
finish = time.perf_counter()
print(f'Finished in {round(finish - start, 2)} second(s)')
Code with __name__=="__main__" (doesn't throw RuntimeError)
import multiprocessing
import time
start = time.perf_counter()
def do_something():
print('Sleeping 1 second...')
time.sleep(1)
print('Done sleeping...')
def main():
p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)
p1.start()
p2.start()
finish = time.perf_counter()
print(f'Finished in {round(finish - start, 2)} second(s)')
if __name__ == "__main__":
main()
In Windows, multiprocessing.Process executes a fresh copy of python to run the code. It has to get the code you want to execute to load in that process so it pickles a snapshot of your current environment to expand in the child. For that to work, the child needs to reimport modules used by the parent. In particular, it needs to import the main script as a module. When you import, any code residing at module level executes.
So lets make the simplest case
foo.py
import multiprocessing as mp
process = mp.Process(target=print, args=('foo',))
process.start()
process.join()
process.start() executes a new python which imports foo.py. And there's the problem. That new foo will create another subprocess which will again import foo.py. So yet another process is created.
The would go on until you blow up your machine except that python detects the problem and raises the exception.
THE FIX
Python modules have the __name__ attribute. If you run your program as a script, __name__ is "main", otherwise, __name__ is the name of your module. So, when a multiprocessing process is importing your main script to setup your environment, its name is not __main__. You can use that to make sure that your MP work is only done in the parent module.
import multiprocessing as mp
if __name__ == "__main__":
# run as top level script, but not as imported module
process = mp.Process(target=print, args=('foo',))
process.start()
process.join()
I have one module which uses python "threading" for concurrency, and "signal" for shutdown hook:
signal.signal(signal.SIGINT, self.shutdownhook)
I have another module which uses dbus and gobject
dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
....
GObject.threads_init()
mainloop = GObject.MainLoop()
mainloop.run()
When I run them seperately, they both operate as expected and ctrl+c causes termination via "KeyboardInterrupt".
However, when I run them together the mainloop terminates but the shutdown hook is never called - the process does not terminate without kill -9 pid.
Can someone please explain why this occurs, and how best to integrate the two models
Here is a working example which highlights my problem. I cannot exit the program with just a CTRL+C and the shutdown hook is not called in this case either.
import threading
import signal
import sys
from gi.repository import GObject
def runMainloop():
print('running mainloop')
mainloop.run()
def shutdown():
print('shutdown')
def readInput():
print('readInput')
print(sys.stdin.readline())
if __name__ == '__main__':
signal.signal(signal.SIGINT, shutdown)
signal.signal(signal.SIGTERM, shutdown)
GObject.threads_init()
mainloop = GObject.MainLoop()
mainloopThread = threading.Thread(name='mainloop', target=runMainloop)
mainloopThread.setDaemon(True)
mainloopThread.start()
print('started')
inputThread = threading.Thread(name='input', target=readInput)
inputThread.start()
print('started input')
No one is interested, so let me try.
Just to be on the same page:
import signal
from gi.repository import GObject
GObject.threads_init()
mainloop = GObject.MainLoop()
signal.signal(signal.SIGINT, lambda n, f: mainloop.quit())
mainloop.run()
This code works:
import signal
from gi.repository import GObject
signal.signal(signal.SIGINT, lambda n, f: print("kill"))
GObject.threads_init()
mainloop = GObject.MainLoop()
mainloop.run()
I've first registered signal handler, and then initiated the loop. Strange is that it is not called. However result is - as expected...
As a side note - according to their doc... mainloop is deprecated. That is first thing.
Edit
Here is example with reading from stdin inside MainLoop:
import signal
import sys
from gi.repository import GObject, GLib
GObject.threads_init()
def readInput():
print('readInput\n')
while True:
input = sys.stdin.readline()
print(input)
if input.strip() == 'exit':
print('closing main loop')
mainloop.quit()
print('terminating thread')
return
if __name__ == '__main__':
signal.signal(signal.SIGINT, signal.SIG_DFL)
mainloop = GObject.MainLoop.new(None, False)
GObject.timeout_add(1000, readInput)
# inputThread = threading.Thread(name='input', target=readInput)
# inputThread.start()
# print('started input')
print('running mainloop\n')
try:
mainloop.run()
except KeyboardInterrupt:
mainloop.quit()
Adding .new(None, False) allows CTRL-C working normally. Took it from here, also here is another thread about integrating pulse audio controller with GLib/GObject loop. There are samples about integration dbus with the loop, but I'm not sure which way you wish to go...
Adding a timer makes the loop receive Unix signals.
import gi
from gi.repository import GLib
import signal
GLib.threads_init()
mainloop = GLib.MainLoop()
signal.signal(signal.SIGTERM, lambda signum, frame: mainloop.quit())
GLib.timeout_add(1000, lambda *args: (print("tick") or True))
try:
mainloop.run()
except KeyboardInterrupt:
print()
import Queue
from multiprocessing.managers import BaseManager
BaseManager.register('get_queue', callable=lambda: Queue.Queue())
manager = BaseManager(address=('', 5000), authkey='abc')
manager.start()
manager.shutdown()
This code will throw a exception
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.
This probably means that you are on Windows and you have
forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
freeze_support()
...
The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce a Windows executable.
but after I add if __name__ == '__main__': freeze_support()
It will throw anther exception, how to fix it?
My os is window7
This error message is displayed when using multiprocessing with the 'spawn' start method (default on platforms lacking fork like windows), and not protecting your code with a if __name__ = '__main__' guard.
The reason is that with the 'spawn' start method a new python process is spawned, which then in turn has to import the __main__ module before it can proceed to do it's work. If your program does not have the mentioned guard, that subprocess would try to execute the same code as the parent process again, spawning another process and so on, until your program (or computer) crashes.
The message is not ment to tell you to add the freeze_support() line, but to guard your program:
import Queue
from multiprocessing.managers import BaseManager
def main():
BaseManager.register('get_queue', callable=lambda: Queue.Queue())
manager = BaseManager(address=('', 5000), authkey='abc')
manager.start()
manager.shutdown()
if __name__ == '__main__':
# freeze_support() here if program needs to be frozen
main() # execute this only when run directly, not when imported!
I have a script I wrote out of modifications of helloworld for gkt, and cmd.
#!/usr/bin/python
import cmd
from gi.repository import Gtk
import threading
class GtkInterface(object):
def __init__(self):
win = Gtk.Window()
win.connect("delete-event", Gtk.main_quit)
win.show_all()
self.window = win;
def create_button(self):
self.button = Gtk.Button(label="Click Here")
self.button.connect("clicked", self.on_button_clicked)
self.window.add(self.button)
self.window.show_all()
def on_button_clicked(self, widget):
print 'something happened'
return
class HelloWorld(cmd.Cmd):
#Simple command processor example.
prompt='>'
def __init__(self, gtk_object):
cmd.Cmd.__init__(self)
# or cmd.Cmd.__init__(self)
self.gtk_object = gtk_object
def do_greet(self, line):
print "hello"
def do_setbutton(self, line):
self.gtk_object.create_button()
def do_exit(self, line):
return True
gtk_o = GtkInterface()
hello = HelloWorld(gtk_o)
def worker(num):
"""thread worker function"""
#print 'Worker: %s' % num
hello.cmdloop()
return
def worker2(num):
Gtk.main()
threads = []
t = threading.Thread(target=worker, args=(1,))
threads.append(t)
t2 = threading.Thread(target=worker2, args=(2,))
threads.append(t2)
if __name__ == '__main__':
#HelloWorld().cmdloop()
#Gtk.main()
t.start()
t2.start()
This works. What I'd like to know is this ok? Are there issues to look out for? This is my first time trying this so there are a lot of unknowns for me. I understand that both cmd, and gtk are blocking. The Gtk.main, and cmd loops work flawlessly so far. I'm just being cautious.
My first time using threading too. When cmd gets the command to setbutton the button is set. When the button is clicked 'something happened' prints. The command line continues as if nothing out of the ordinary happened. I was really surprised at how seamless it all works. Yet I am still a little worried.
GTK has its' own threading library, and you need to be careful I think with complex applications : http://faq.pygtk.org/index.py?req=show&file=faq20.006.htp - for instance when you have threads which update your GUI indepedently of the main thread.
In your example you do have a threaded application, although in fact your entire GTK application is running in a single thread - so you are ok.
GTK+ is not thread safe and there are a few problems with the example which may cause instability. The example loads GTK+ in the main thread, runs the GTK+ main loop in a different thread and creates GTK+ widgets in yet another thread. All the GTK+ API calls should occur in the main thread with other threads communicating back to the main GUI thread by adding idle or timer callbacks. Have a read through the PyGObject threading wiki [1].
https://wiki.gnome.org/Projects/PyGObject/Threading
I have an application (Gtk) that has an embedded server (using circuits). Both components (The GUI and Server) have infinite loops. How can I run both loops simultaneously ?
I also need the server loop to end when the gtk loop ends.
The code for the example server
from circuits.web import Server, Controller
import os
class MyServer(Controller):
def index(self):
return "Hello World"
server = Server(8000)
server += MyServer()
server.run()
and the code for example gtk application
import gtk
class App:
def __init__(self):
self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
self.window.connect("destroy",gtk.main_quit)
self.window.show_all()
gtk.main()
if __name__ == '__main__':
app = App()
You could use the multiprocessing module to do this:
from multiprocessing import Process
def run_app():
... run the app ...
def run_server():
... run the server ...
def main():
app = Process(target=run_app)
app.start()
server = Process(target=run_server)
server.start()
app.join()
server.terminate()
if __name__ == "__main__":
main()
Otherwise, if you're using Python < 2.6 on Unix, you could fiddle around with os.fork() to do the same sort of thing. Threading might work, but I don't know how well GTK or circuits plays with threads.
You could run the web server from another thread:
from threading import Thread
# ...
server = Server(8000)
server += MyServer()
web_server_thread = Thread(target=server.run)
web_server_thread.start()
gtk.main()