I have an application (Gtk) that has an embedded server (using circuits). Both components (The GUI and Server) have infinite loops. How can I run both loops simultaneously ?
I also need the server loop to end when the gtk loop ends.
The code for the example server
from circuits.web import Server, Controller
import os
class MyServer(Controller):
def index(self):
return "Hello World"
server = Server(8000)
server += MyServer()
server.run()
and the code for example gtk application
import gtk
class App:
def __init__(self):
self.window = gtk.Window(gtk.WINDOW_TOPLEVEL)
self.window.connect("destroy",gtk.main_quit)
self.window.show_all()
gtk.main()
if __name__ == '__main__':
app = App()
You could use the multiprocessing module to do this:
from multiprocessing import Process
def run_app():
... run the app ...
def run_server():
... run the server ...
def main():
app = Process(target=run_app)
app.start()
server = Process(target=run_server)
server.start()
app.join()
server.terminate()
if __name__ == "__main__":
main()
Otherwise, if you're using Python < 2.6 on Unix, you could fiddle around with os.fork() to do the same sort of thing. Threading might work, but I don't know how well GTK or circuits plays with threads.
You could run the web server from another thread:
from threading import Thread
# ...
server = Server(8000)
server += MyServer()
web_server_thread = Thread(target=server.run)
web_server_thread.start()
gtk.main()
Related
In a standalone Python script, I have a functioning GLib mainloop.
# myloop.py
import gobject as GLib
def main():
mainloop = GLib.MainLoop()
# do some DBus related stuffs
mainloop.run()
if __name__ == '__main__':
main()
I now need to run this from another Python project,
where multiple processes are used to implement different features.
from multiprocessing import Process
from myloop import main as myloop_main
def do something():
# do somethng else
if __name__ == '__main__':
something_proc = Process(target = do_something)
myloop_proc = Process(target = myloop_main)
something_proc.start()
myloop_proc.start()
something_proc.join()
myloop_proc.join()
This runs without error, but the mainloop is not really running
and the feature implemented in the mainloop is not working.
How to run the GLib's mainloop in a process?
tl,dr: How can I programmably execute a python module (not function) as a separate process from a different python module?
On my development laptop, I have a 'server' module containing a bottle server. In this module, the name==main clause starts the bottle server.
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
if __name__ == '__main__':
serve(bt_app, port=localhost:8080)
I also have a 'test_server' module containing pytests. In this module, the name==main clause runs pytest and displays the results.
def test_something():
_rtn = some_server_function()
assert _rtn == desired
if __name__ == '__main__':
_rtn = pytest.main([__file__])
print("Pytest returned: ", _rtn)
Currently, I manually run the server module (starting the web server on localhost), then I manually start the pytest module which issues html requests to the running server module and checks the responses.
Sometimes I forget to start the server module. No big deal but annoying. So I'd like to know if I can programmatically start the server module as a separate process from the pytest module (just as I'm doing manually now) so I don't forget to start it manually.
Thanks
There is my test cases dir tree:
test
├── server.py
└── test_server.py
server.py start a web server with flask.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
test_server.py make request to test.
import sys
import requests
import subprocess
import time
p = None # server process
def start_server():
global p
sys.path.append('/tmp/test')
# here you may want to do some check.
# whether the server is already started, then pass this fucntion
kwargs = {} # here u can pass other args needed
p = subprocess.Popen(['python','server.py'], **kwargs)
def test_function():
response = requests.get('http://localhost:5000/')
print('This is response body: ', response.text)
if __name__ == '__main__':
start_server()
time.sleep(3) # waiting server started
test_function()
p.kill()
Then you can do python test_server to start the server and do test cases.
PS: Popen() needs python3.5+. if older version, use run instead
import logging
import threading
import time
def thread_function(name):
logging.info("Thread %s: starting", name)
time.sleep(2)
logging.info("Thread %s: finishing", name)
if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
datefmt="%H:%M:%S")
threads = list()
for index in range(3):
logging.info("Main : create and start thread %d.", index)
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for index, thread in enumerate(threads):
logging.info("Main : before joining thread %d.", index)
thread.join()
logging.info("Main : thread %d done", index)
With threading you can run multiple processes at once!
Wim baasically answered this question. I looked into the subprocess module. While reading up on it, I stumbled on the os.system function.
In short, subprocess is a highly flexible and functional program for running a program. os.system, on the other hand, is much simpler, with far fewer functions.
Just running a python module is simple, so I settled on os.system.
import os
server_path = "python -m ../src/server.py"
os.system(server_path)
Wim, thanks for the pointer. Had it been a full fledged answer I would have upvoted it. Redo it as a full fledged answer and I'll do so.
Async to the rescue.
import gevent
from gevent import monkey, spawn
monkey.patch_all()
from gevent.pywsgi import WSGIServer
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
def test_something():
_rtn = some_server_function()
assert _rtn == desired
print("Pytest returned: ",_rtn)
sleep(0)
if __name__ == '__main__':
spawn(test_something) #runs async
server = WSGIServer(("0.0.0.0", 8080, bt_app)
server.serve_forever()
I am working on Python's Flask server side code where there is a background task which runs periodically and executes a function (note that 'periodically' is not so hardline, execute once and then after x seconds also works). But I also need it to execute the same function immediately when the server receives a request (and then resume the background task).
This kind of reminds me of the SELECT system call in C, where the system waits for a timeout or until a packet arrives.
Here is what I came up minimally after looking up a lot of answers.
from flask import Flask, request
import threading, os, time
POOL_TIME = 2
myThread = threading.Thread()
def pollAndExecute(a='|'):
time.sleep(1)
print(time.time(), a)
# time.sleep(1)
myThread = threading.Timer(POOL_TIME, pollAndExecute)
myThread.start()
def startWork():
global myThread
myThread = threading.Timer(POOL_TIME, pollAndExecute)
myThread.start()
app = Flask(__name__)
#app.route('/ping', methods=['POST'])
def ping():
global myThread
myThread.cancel()
pollAndExecute("#")
return "Hello"
if __name__ == '__main__':
app.secret_key = os.urandom(12)
startWork()
app.run(port=5001)
Output:
But the output clearly says that it is not behaving properly after there is a request (sent using curl -X POST http://localhost:5001/ping)
Please guide me as to how to correct this or are there any other ways to do it. Just FYI, in the original code, there are various database updates in the pollAndExecute() as well and I need to take care that there are no race conditions between polling and ping. Needless to say, only one copy of the function should execute at a particular time (preferably in a single thread).
Here is the solution I made for your problem. I used a priority queue that takes in data to be run with the printTime function. The background and flask functions are two different threads that push data into the priority queue, which should prioritize the flask call over the background one. Notice how it now waits for the current thread to finish before executing another one.
from flask import Flask, request
import threading, os, time
from threading import Thread, Lock
from queue import PriorityQueue
POOL_TIME = 2
lock = Lock()
def printTime(a='|'):
time.sleep(1) # Simulate process taking 1 sec
print(time.time(), a)
jobs = PriorityQueue()
class Queue(Thread):
def __init__(self):
Thread.__init__(self)
self.daemon = True
self.start()
def run(self):
while True:
_, data = jobs.get()
printTime(data)
class backGroundProcess(Thread):
def __init__(self):
Thread.__init__(self)
self.daemon = True
self.start()
def run(self):
while True:
time.sleep(2) # Background process enqueues a job every 2 secs
jobs.put((0,"|"))
class flaskProcess(Thread):
def __init__(self):
Thread.__init__(self)
self.start()
def run(self):
jobs.put((1,"#"))
app = Flask(__name__)
#app.route('/ping', methods=['POST'])
def ping():
flaskThread = flaskProcess()
return "Hello"
if __name__ == '__main__':
backGroundProcess()
Queue()
app.secret_key = os.urandom(12)
app.run(port=5001)
The above snippet may be a little verbose because I used classes, but this should get you started.
I am building a flask app and need some background processes to run. I decided to go with multiprocess, but it's producing two processes when running within Flask. Does anyone know why this would happen? I've tested it on OS X and Ubuntu 12.04, with the same results. Here is an example:
import time
import multiprocessing
from flask import Flask
app = Flask(__name__)
backProc = None
def testFun():
print('Starting')
while True:
time.sleep(3)
print('looping')
time.sleep(3)
print('3 Seconds Later')
#app.route('/')
def root():
return 'Started a background process with PID ' + str(backProc.pid) + " is running: " + str(backProc.is_alive())
#app.route('/kill')
def kill():
backProc.terminate()
return 'killed: ' + str(backProc.pid)
#app.route('/kill_all')
def kill_all():
proc = multiprocessing.active_children()
for p in proc:
p.terminate()
return 'killed all'
#app.route('/active')
def active():
proc = multiprocessing.active_children()
arr = []
for p in proc:
print(p.pid)
arr.append(p.pid)
return str(arr)
#app.route('/start')
def start():
global backProc
backProc = multiprocessing.Process(target=testFun, args=(), daemon=True)
backProc.start()
return 'started: ' + str(backProc.pid)
if __name__ == '__main__':
app.run(port=int("7879"))
This is a problem with the Flask auto-reload feature, which is used during development to automatically restart the webserver when changes in code is detected, in order to serve up the new code without requiring a manual restart.
In the guide, the “app.run()” call is always placed within an “if __name__ == ‘__main__’” condition, since the reloader is set to on by default. When using multiprocessing, this condition will result in false, so you have to instead disable the Flask autoreload when using it in a function like so:
def startWebserver():
app.run(debug=True, use_reloader=False)
Link for reference:
http://blog.davidvassallo.me/2013/10/23/nugget-post-python-flask-framework-and-multiprocessing/
I have a script I wrote out of modifications of helloworld for gkt, and cmd.
#!/usr/bin/python
import cmd
from gi.repository import Gtk
import threading
class GtkInterface(object):
def __init__(self):
win = Gtk.Window()
win.connect("delete-event", Gtk.main_quit)
win.show_all()
self.window = win;
def create_button(self):
self.button = Gtk.Button(label="Click Here")
self.button.connect("clicked", self.on_button_clicked)
self.window.add(self.button)
self.window.show_all()
def on_button_clicked(self, widget):
print 'something happened'
return
class HelloWorld(cmd.Cmd):
#Simple command processor example.
prompt='>'
def __init__(self, gtk_object):
cmd.Cmd.__init__(self)
# or cmd.Cmd.__init__(self)
self.gtk_object = gtk_object
def do_greet(self, line):
print "hello"
def do_setbutton(self, line):
self.gtk_object.create_button()
def do_exit(self, line):
return True
gtk_o = GtkInterface()
hello = HelloWorld(gtk_o)
def worker(num):
"""thread worker function"""
#print 'Worker: %s' % num
hello.cmdloop()
return
def worker2(num):
Gtk.main()
threads = []
t = threading.Thread(target=worker, args=(1,))
threads.append(t)
t2 = threading.Thread(target=worker2, args=(2,))
threads.append(t2)
if __name__ == '__main__':
#HelloWorld().cmdloop()
#Gtk.main()
t.start()
t2.start()
This works. What I'd like to know is this ok? Are there issues to look out for? This is my first time trying this so there are a lot of unknowns for me. I understand that both cmd, and gtk are blocking. The Gtk.main, and cmd loops work flawlessly so far. I'm just being cautious.
My first time using threading too. When cmd gets the command to setbutton the button is set. When the button is clicked 'something happened' prints. The command line continues as if nothing out of the ordinary happened. I was really surprised at how seamless it all works. Yet I am still a little worried.
GTK has its' own threading library, and you need to be careful I think with complex applications : http://faq.pygtk.org/index.py?req=show&file=faq20.006.htp - for instance when you have threads which update your GUI indepedently of the main thread.
In your example you do have a threaded application, although in fact your entire GTK application is running in a single thread - so you are ok.
GTK+ is not thread safe and there are a few problems with the example which may cause instability. The example loads GTK+ in the main thread, runs the GTK+ main loop in a different thread and creates GTK+ widgets in yet another thread. All the GTK+ API calls should occur in the main thread with other threads communicating back to the main GUI thread by adding idle or timer callbacks. Have a read through the PyGObject threading wiki [1].
https://wiki.gnome.org/Projects/PyGObject/Threading