We Keep Coding

Python - how can I run separate module (not function) as a separate process?

tl,dr: How can I programmably execute a python module (not function) as a separate process from a different python module?
On my development laptop, I have a 'server' module containing a bottle server. In this module, the name==main clause starts the bottle server.
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
if __name__ == '__main__':
serve(bt_app, port=localhost:8080)
I also have a 'test_server' module containing pytests. In this module, the name==main clause runs pytest and displays the results.
def test_something():
_rtn = some_server_function()
assert _rtn == desired
if __name__ == '__main__':
_rtn = pytest.main([__file__])
print("Pytest returned: ", _rtn)
Currently, I manually run the server module (starting the web server on localhost), then I manually start the pytest module which issues html requests to the running server module and checks the responses.
Sometimes I forget to start the server module. No big deal but annoying. So I'd like to know if I can programmatically start the server module as a separate process from the pytest module (just as I'm doing manually now) so I don't forget to start it manually.
Thanks

There is my test cases dir tree:
test
├── server.py
└── test_server.py
server.py start a web server with flask.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
test_server.py make request to test.
import sys
import requests
import subprocess
import time
p = None # server process
def start_server():
global p
sys.path.append('/tmp/test')
# here you may want to do some check.
# whether the server is already started, then pass this fucntion
kwargs = {} # here u can pass other args needed
p = subprocess.Popen(['python','server.py'], **kwargs)
def test_function():
response = requests.get('http://localhost:5000/')
print('This is response body: ', response.text)
if __name__ == '__main__':
start_server()
time.sleep(3) # waiting server started
test_function()
p.kill()
Then you can do python test_server to start the server and do test cases.
PS: Popen() needs python3.5+. if older version, use run instead

import logging
import threading
import time
def thread_function(name):
logging.info("Thread %s: starting", name)
time.sleep(2)
logging.info("Thread %s: finishing", name)
if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
datefmt="%H:%M:%S")
threads = list()
for index in range(3):
logging.info("Main : create and start thread %d.", index)
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for index, thread in enumerate(threads):
logging.info("Main : before joining thread %d.", index)
thread.join()
logging.info("Main : thread %d done", index)
With threading you can run multiple processes at once!

Wim baasically answered this question. I looked into the subprocess module. While reading up on it, I stumbled on the os.system function.
In short, subprocess is a highly flexible and functional program for running a program. os.system, on the other hand, is much simpler, with far fewer functions.
Just running a python module is simple, so I settled on os.system.
import os
server_path = "python -m ../src/server.py"
os.system(server_path)
Wim, thanks for the pointer. Had it been a full fledged answer I would have upvoted it. Redo it as a full fledged answer and I'll do so.

Async to the rescue.
import gevent
from gevent import monkey, spawn
monkey.patch_all()
from gevent.pywsgi import WSGIServer
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
def test_something():
_rtn = some_server_function()
assert _rtn == desired
print("Pytest returned: ",_rtn)
sleep(0)
if __name__ == '__main__':
spawn(test_something) #runs async
server = WSGIServer(("0.0.0.0", 8080, bt_app)
server.serve_forever()

Running bokeh server programmatically to show in browser locally

I am making an interactive data manipulation with bokeh (0.12.6) utility that I will deploy within a package. The idea is that a user can run a some routine module.utility() that will start the bokeh server, launch the application in a browser, and when the tab or browser are closed, the server will be stopped.
My application launches fine if I run bokeh serve --show myapp, but it hangs when connecting to the localhost using my method described below. I've inspected the handlers, everything looks as it should.
Is this a reasonable thing to do, and am I going about it correctly?
Directory format
<installed module path>/myapp
└── main.py
Where ./myapp will reside in venv/lib/python3.5/site-packages/mymodule etc.
main.py
from bokeh.io import curdoc
from bokeh.layouts import column
from bokeh.plotting import figure
from bokeh.models.sources import ColumnDataSource
source = ColumnDataSource(dict(x=list(range(5)), y=list(range(5))))
p = figure(width=300, height=300, tools=[], toolbar_location=None)
p.line(x='x', y='y', source=source)
curdoc().add_root(column(p, sizing_mode='scale_width'))
Run script
def run_single_server(abs_app_path, port=5000):
'''Run bokeh application for single session from server`'''
from bokeh.application import Application
from bokeh.application.handlers import DirectoryHandler
from bokeh.server.server import Server
import os
app_name = os.path.split(abs_app_path)[1]
url = '/{}'.format(app_name)
# Start a bokeh server
apps = {url:Application(DirectoryHandler(filename=abs_app_path))}
server = Server(apps, port=port)
server.start()
server.show(url)
# somehow wait for session to end, perhaps using `server_lifecycle.py`
server.stop()
return
def utility():
import mymodule
module_path = os.path.split(mymodule.__file__)[0]
abs_app_path = os.path.join(module_path, 'myapp')
run_single_server(abs_app_path, port=5000)
return
Perhaps have that routine in the main __init__.py, and have it work like this:
import mymodule
mymodule.utility()
# 1. Browser launches
# 2. user does stuff
# 3. user closes window
# 4. bokeh server is shutdown
Update
I found the build_single_handler_application routine and tried that, but it also appears to hang.
from bokeh.command.util import build_single_handler_application
import os
app_name = os.path.split(abs_app_path)[1]
url = '/{}'.format(app_name)
# Start a bokeh server
apps = build_single_handler_application(abs_app_path)

It looks like I had a couple of problems. I ended up finding and adapting some code that I found on the mail group here for my use-case.
I managed to get everything to work by using separate process for 1) starting the server, 2) launching the app urls with webbrowser, and 3) checking for closed connections and shutting down.
I think I could perhaps do away with initiating the tornado server instance as was done in the flask example I adapted, but I'm happy here.
Note: this example uses single file apps, but you can pass the paths of directory formatted apps as well.
def create_bokeh_server(io_loop, files, argvs, host, port):
'''Start bokeh server with applications paths'''
from bokeh.server.server import Server
from bokeh.command.util import build_single_handler_applications
# Turn file paths into bokeh apps
apps = build_single_handler_applications(files, argvs)
# kwargs lifted from bokeh serve call to Server, with created io_loop
kwargs = {
'io_loop':io_loop,
'generate_session_ids':True,
'redirect_root':True,
'use_x_headers':False,
'secret_key':None,
'num_procs':1,
'host': host,
'sign_sessions':False,
'develop':False,
'port':port,
'use_index':True
}
server = Server(apps,**kwargs)
return server
def run_single_app(files, port=5000, new='tab'):
def start_bokeh(io_loop):
'''Start the `io_loop`'''
io_loop.start()
return None
def launch_app(host, app_name, new):
'''Lauch app in browser
Ideally this would `bokeh.util.browser.view()`, but it doesn't work
'''
import webbrowser
# Map method strings to webbrowser method
options = {'current':0, 'window':1, 'tab':2}
# Concatenate url and open in browser, creating a session
app_url = 'http://{}/{}'.format(host, app_name)
print('Opening `{}` in browser'.format(app_url))
webbrowser.open(app_url, new=options[new])
return None
def server_loop(server, io_loop):
'''Check connections once session created and close on disconnect'''
import time
connected = [True,]
session_loaded = False
while any(connected):
# Check if no session started on server
sessions = server.get_sessions()
if not session_loaded:
if sessions:
session_loaded = True
# Once 1+ sessions started, check for no connections
else:
# List of bools for each session
connected = [True,]*len(sessions)
# Set `connected` item false no connections on session
for i in range(len(sessions)):
if sessions[i].connection_count == 0:
connected[i] = False
# Keep the pace down
time.sleep(2)
# Stop server once opened session connections closed
io_loop.stop()
return None
import os
import threading
import tornado.ioloop
import tornado.autoreload
import time
# Initialize some values, sanatize the paths to the bokeh plots
argvs = {}
app_names = []
for path in files:
argvs[path] = None
app_names.append(os.path.splitext(os.path.split(path)[1])[0])
# Concate hostname/port for creating handlers, launching apps
host = 'localhost:{}'.format(port)
# Initialize the tornado server
io_loop = tornado.ioloop.IOLoop.instance()
tornado.autoreload.start(io_loop)
# Add the io_loop to the bokeh server
server = run_bokeh_server(io_loop, files, argvs, host, port)
print('Starting the server on {}'.format(host))
args = (io_loop,)
th_startup = threading.Thread(target=start_bokeh, args=args)
th_startup.start()
# Launch each application in own tab or window
th_launch = [None,]*len(app_names)
for i in range(len(app_names)):
args = (host, app_names[i], new)
th_launch[i] = threading.Thread(target=launch_app, args=args)
th_launch[i].start()
# Delay to allow tabs to open in same browser window
time.sleep(2)
# Run session connection test, then stop `io_loop`
args = (server, io_loop)
th_shutdown = threading.Thread(target=server_loop, args=args)
th_shutdown.start()
return None
if __name__ == "__main__":
import os
files = [os.path.join('bokeh', fname) for fname in ['ex1.py','ex2.py']]
run_single_app(files, port=5006)

SocketServer does not release port on exit

I have the following issue with Python (2.7) socketserver:
import wx
import socket
from SocketServer import BaseRequestHandler, ThreadingTCPServer
from threading import Thread
from wx.lib.pubsub import pub
from GUI import GUI
class LogHandler(BaseRequestHandler):
def handle(self):
data = self.request.recv(1024)
wx.CallAfter(pub.sendMessage, 'logmsg', msg=data)
self.request.close()
class MyTestGUI(GUI):
def __init__(self):
super(MyTestGUI, self).__init__(None)
pub.subscribe(self.AppendLog, 'logmsg')
self.LogServer = ThreadingTCPServer(('', 10010), LogHandler)
self.LogServer.allow_reuse_address = True
self.thd = Thread(target=self.LogServer.serve_forever)
self.thd.daemon = True
self.thd.start()
def AppendLog(self, msg):
# Append the mesage
print(msg)
def AppClose(self, event):
self.LogServer.shutdown()
self.LogServer.server_close()
exit()
if __name__ == '__main__':
app = wx.App()
frame = MyTestGUI()
frame.Show(True)
app.MainLoop()
This server is supposed to receive messages from a device (which closes the socket upon message sent). On the first run the code works ok, but after restart I get the following exception:
self.LogServer = ThreadingTCPServer(('', 10010), LogHandler)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 420, in __init__
self.server_bind()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 434, in server_bind
self.socket.bind(self.server_address)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 48] Address already in use
LogServer.allow_reuse_address = True / False, does not change a thing.

I had the same problem in Python 3.7.2 with a (non-threading) socketserver.TCPServer instance. ( I use anaconda on opensuse, in case it matters )
My solution:
during server initialization I set the third parameter of the __init__ function
false: bind_and_activate = False.
After initialization I set the .allow_reuse_address flag to True, and the .server_bind() and .server_activate() funs were called only after this flag setting.
I think that the important thing is to set the bind_and_activate flag before bindig and activating the server, since if the bind_and_activate parameter during TCPServer initialization is True or not set, then the TCPServer.__init__() calls both the server_bind() and server_activate() funtion. And in this case I could not change the value of the .allow_reuse_address flag - I could see it from the print() calls .
(Note that the code below does not work since the request handler definition is missing from it. I also use timeout, but that is independent of the problem )
def set_up_server_for_one_request(host, port, reqHandler, timeout = 600):
''' set up a server for one request. there is timeout as well.
'''
with socketserver.TCPServer((host, port), reqHandler, \
bind_and_activate= False) as serverInstance:
print('reuse adress attribute before flag setting: ',\
serverInstance.socket.getsockopt(socket.SOL_SOCKET,\
socket.SO_REUSEADDR))#print flagVal before setting
serverInstance.allow_reuse_address = True
print('reuse adress attribute after flag setting: ',\
serverInstance.socket.getsockopt(socket.SOL_SOCKET, \
socket.SO_REUSEADDR)) #print flagVal after setting
print('server adress: ', serverInstance. server_address)
serverInstance.server_bind()
serverInstance.server_activate()
serverInstance.timeout = timeout #set server's timeout
serverInstance.handle_request()
print('server finished, and exits')
if __name__ == '__main__':
set_up_server_for_one_request(host = 'localhost', port = 9998,\
reqHandler = My_EchoRH, timeout = 20)

Twisted makeService() async

I need to create a MultiService that loads asynchronously in a twisted plugin, is there a way of achieving that ?
Here what I've done so far:
app/plugins.py
class Options(usage.Options):
synopsis = '[options]'
longdesc = (
'Starts app modules services. The modules must '
'be configured and enabled for the current server '
'before being started. To see the list installed modules '
'use the --list switch.')
optFlags = [['list', 'l', 'Display the list of installed modules.']]
def makeService(options):
from twisted.internet import reactor
debug = options['debug']
return app.ModuleService.load(reactor)
twisted/plugins/app_plugins.py
TestService = ServiceMaker(
'test',
'app.plugins',
'Test service.',
'test')
app.py
class ModuleService(service.MultiService):
def __init__(self, reactor=None):
# Twisted old style classes
service.MultiService.__init__(self)
if reactor is None:
from twisted.internet import reactor
self._reactor = reactor
#classmethod
#defer.inlineCallbacks
def load(cls, reactor=None):
modules = yield get_modules()
service = cls(reactor)
for module in modules:
# module tcp server
mod_endpoint = endpoints.TCP4ServerEndpoint(reactor, module.port)
module_service = internet.StreamServerEndpointService(
mod_endpoint,
pb.PBServerFactory(spread.RunnerServer()))
module_service.setServiceParent(service)
defer.returnValue(service)
So my problem is that the MultiService is loaded asynchronously and therefore cannot be used in the makeService function, someone can help me with this ?

Services in twisted are not supposed to start asynchronously. Service start is done before reactoru startup. You should change your application architecture. I am currently studying how :D

how to deal with Python BaseHTTPServer killed,but the port is still be occupied?

I use python BaseHTTPServer,it can handle do_GET,do_POST methods,in do_POST method,i execute linux shell using os.system,when i kill the python script,but the listening port still occupied,so i can't run the script again, the netstat -antp|grep 80 show that the bash/tail is occupied the port 80
import sys
import os
import traceback
import time
import logging.handlers
import logging
from threading import *
from datetime import datetime
import urllib2
reload(sys)
sys.setdefaultencoding('utf-8')
class WebRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
server_version = 'python httpserver'
sys_version = 'b'
backup_dir = None
def do_HEAD(s):
s.send_response(200)
s.send_header("Content-type", "text/html")
s.end_headers()
def do_GET(self):
if self.path == '/foo':
self.send_response(200)
os.system('nohup tail -f a.log &')
else:
self.send_error(404)
if __name__ == "__main__":
try:
server = BaseHTTPServer.HTTPServer(('',80), WebRequestHandler)
server.serve_forever()
except KeyboardInterrupt:
server.socket.close()

File descriptors are inherited by default by child processes, so the socket listening on port 80 is inherited by the command you have launched using your system() call.
To avoid that, you should set the FD_CLOEXEC flag on the listening socket. This can be done by adding (and using) a specific HTTPServer class.
class WebServer(BaseHTTPServer.HTTPServer):
def __init__(self, *args, **kwargs):
BaseHTTPServer.HTTPServer.__init__(self, *args, **kwargs)
# Set FD_CLOEXEC flag
flags = fcntl.fcntl(self.socket.fileno(), fcntl.F_GETFD)
flags |= fcntl.FD_CLOEXEC
fcntl.fcntl(self.socket.fileno(), fcntl.F_SETFD, flags)
Related :
Process started from system command in C inherits parent fd's