We Keep Coding

Django global variables with threading

I'm building a web-app to control some hardware connected to a Raspberry Pi.
The goal:
On server startup do some tasks once and store information globally (open COM Ports and store as objects)
On POST/GET call from a view start a thread with the serialports from 1) plus some data (names as Strings) from the POST/GET call. This thread is some game logic for the hardware and runs for around one hour each
Have possibility to terminate this thread anytime from a view and from within the thread
Have possibility to get variables from thread and display on web-app
At the moment I'm struggeling with the general design of this task. I'm new to threading and Django and can't quite tie these together. My Implementation right now only has a simple endless loop which should print out some debugging info if a global variable has changed.
On startup of the server, I'm starting a runner script in a new thread. The global var is stored in cfg.py and imported in apps.py and views.py. All three files are in the root directory of the app.
cfg.py
RUNNING = False
views.py
from django.shortcuts import render
from .models import Player
from . import cfg
def index(request):
player = Player.objects.all()
if request.method == 'POST':
try:
print('got POST request')
cfg.RUNNING = True
except Exception as exc:
print(exc)
return render(request, 'mygame/index.html', {
'player' : player
})
apps.py
import os
import threading
import datetime
import time
from django.apps import AppConfig
from . import cfg
class MyGameConfig(AppConfig):
default_auto_field = 'django.db.models.BigAutoField'
name = 'mygame'
def ready(self):
run_once = os.environ.get('CMDLINERUNNER_RUN_ONCE')
if run_once is not None:
return
os.environ['CMDLINERUNNER_RUN_ONCE'] = 'True'
#### Code to execute on startup
try:
bp_thread = threading.Thread(target=runner, args=('John', 'James', ), daemon=True)
bp_thread.start()
except Exception as exc:
raise exc
def runner(p1, p2, *args, **kwargs):
# dummy function running in background
starttime = datetime.datetime.now()
stroftime = starttime.isoformat()
print(f'process started at {stroftime}')
while True:
if cfg.RUNNING:
print('RUNNING = True')
print('setting back to False for testing')
cfg.RUNNING = False
else:
pass
time.sleep(1)
Why can't my def index(request) function change the val of the cfg.RUNNING var? I get the debugging print with got POST request but the runner function is not recognizing the changed state of cfg.RUNNING.
I also tried an aproach with global RUNNING without success. If it was only strings I have to pass I'd use the model database but since the serial ports are objects I can't store in the SQLite3 db, this doesn't seem to suit my needs. Also the vals should be reset on server restart.
Maybe my whole design is flawed and I'm open to any suggestions to achive my goal. Thanks a lot in advance!

Python - how can I run separate module (not function) as a separate process?

tl,dr: How can I programmably execute a python module (not function) as a separate process from a different python module?
On my development laptop, I have a 'server' module containing a bottle server. In this module, the name==main clause starts the bottle server.
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
if __name__ == '__main__':
serve(bt_app, port=localhost:8080)
I also have a 'test_server' module containing pytests. In this module, the name==main clause runs pytest and displays the results.
def test_something():
_rtn = some_server_function()
assert _rtn == desired
if __name__ == '__main__':
_rtn = pytest.main([__file__])
print("Pytest returned: ", _rtn)
Currently, I manually run the server module (starting the web server on localhost), then I manually start the pytest module which issues html requests to the running server module and checks the responses.
Sometimes I forget to start the server module. No big deal but annoying. So I'd like to know if I can programmatically start the server module as a separate process from the pytest module (just as I'm doing manually now) so I don't forget to start it manually.
Thanks

There is my test cases dir tree:
test
├── server.py
└── test_server.py
server.py start a web server with flask.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
test_server.py make request to test.
import sys
import requests
import subprocess
import time
p = None # server process
def start_server():
global p
sys.path.append('/tmp/test')
# here you may want to do some check.
# whether the server is already started, then pass this fucntion
kwargs = {} # here u can pass other args needed
p = subprocess.Popen(['python','server.py'], **kwargs)
def test_function():
response = requests.get('http://localhost:5000/')
print('This is response body: ', response.text)
if __name__ == '__main__':
start_server()
time.sleep(3) # waiting server started
test_function()
p.kill()
Then you can do python test_server to start the server and do test cases.
PS: Popen() needs python3.5+. if older version, use run instead

import logging
import threading
import time
def thread_function(name):
logging.info("Thread %s: starting", name)
time.sleep(2)
logging.info("Thread %s: finishing", name)
if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
datefmt="%H:%M:%S")
threads = list()
for index in range(3):
logging.info("Main : create and start thread %d.", index)
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for index, thread in enumerate(threads):
logging.info("Main : before joining thread %d.", index)
thread.join()
logging.info("Main : thread %d done", index)
With threading you can run multiple processes at once!

Wim baasically answered this question. I looked into the subprocess module. While reading up on it, I stumbled on the os.system function.
In short, subprocess is a highly flexible and functional program for running a program. os.system, on the other hand, is much simpler, with far fewer functions.
Just running a python module is simple, so I settled on os.system.
import os
server_path = "python -m ../src/server.py"
os.system(server_path)
Wim, thanks for the pointer. Had it been a full fledged answer I would have upvoted it. Redo it as a full fledged answer and I'll do so.

Async to the rescue.
import gevent
from gevent import monkey, spawn
monkey.patch_all()
from gevent.pywsgi import WSGIServer
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
def test_something():
_rtn = some_server_function()
assert _rtn == desired
print("Pytest returned: ",_rtn)
sleep(0)
if __name__ == '__main__':
spawn(test_something) #runs async
server = WSGIServer(("0.0.0.0", 8080, bt_app)
server.serve_forever()

How Do You Thread an External Hanging API Call in Flask?

Getting the specifics out of the way, I'm writing an open source P2P social network over IPFS and Flask -- I know, it's been done. I'm choosing Flask because pyinstaller can put it in an exe file.
I am attempting to update my IPNS every 10 minutes to publish all status updates I've added to the network during said 10 minutes. The cron function from setup class (from library.py) is where that updater function is stored. At first, I threaded the cron function from init of setup. The server hung. Then I moved the threading process over to app.before_first_request. The server still hangs.
https://pastebin.com/bXHTuH83 (main.py)
from flask import Flask, jsonify
from library import *
#=========================TO BE DELETED=========================================
def pretty(json):
json = dumps(loads(json), indent=4, sort_keys=True)
return json
#===============================================================================
app = Flask(__name__)
GANN = setup()
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
#app.route("/")
def home():
return "Hello World!!!"
if __name__ == "__main__":
app.run(port="80", debug=True, threaded=True)
https://pastebin.com/W5P8Tpvd (library.py)
from threading import Thread
from time import time, sleep
import urllib.request
from json import loads, dumps
def api(*argv, **kwargs):
url = "http://127.0.0.1:5001/api/v0/"
for arg in argv:
arg = arg.replace(" ", "/")
if arg[:-1] != "/":
arg += "/"
url += arg
url = url[0:-1]
if kwargs:
url+="?"
for val in kwargs:
url = url + val + "=" + kwargs[val] + "&"
url = url[0:-1]
print(url)
try:
with urllib.request.urlopen(url, timeout=300) as response:
return response.read()
except:
return b"""{"ERROR": "CANNOT CONNECT TO IPFS!"}"""
class setup():
def __init__(self):
api("files", "mkdir", arg="/GANN", parents="True")
self.root_hash = ""
def update_root(self):
try:
for entry in loads(api("files", "ls", l="True").decode())["Entries"]:
if entry["Name"] == "GANN":
self.root_hash = entry["Hash"]
except:
return """{"ERROR": "CANNOT FIND ROOT DIRECTORY"}"""
def publish_root(self):
api("name", "publish", arg=self.root_hash)
def cron(self):
while True:
print("CRON Thread Started!")
self.update_root()
self.publish_root()
sleep(600)
I have searched the web for a couple days and have yet to find a threading technique that will split from the main process and not hang the server from taking other requests. I believe I'm on a single stream connection, as IPFS blocks connections to every other device in my home when it's started. It takes a couple minutes for the CLI IPNS update to go through, so I set urllib's timeout to 300 seconds.

Well what I think the threading code is not correct.
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
Here you created a Thread object. The argument must be callable, but you called your method already here. so the right way would be
Thread(target=GANN.cron)
So the thread can call the cron function later. having said that, the Thread must be started, so it will call the function target you gave. So it must be ike
thread_cron = Thread(target=GANN.cron)
thread_cron.start()
Since you called the GANN.cron() , the method starts executing and your app hung!

Running bokeh server programmatically to show in browser locally

I am making an interactive data manipulation with bokeh (0.12.6) utility that I will deploy within a package. The idea is that a user can run a some routine module.utility() that will start the bokeh server, launch the application in a browser, and when the tab or browser are closed, the server will be stopped.
My application launches fine if I run bokeh serve --show myapp, but it hangs when connecting to the localhost using my method described below. I've inspected the handlers, everything looks as it should.
Is this a reasonable thing to do, and am I going about it correctly?
Directory format
<installed module path>/myapp
└── main.py
Where ./myapp will reside in venv/lib/python3.5/site-packages/mymodule etc.
main.py
from bokeh.io import curdoc
from bokeh.layouts import column
from bokeh.plotting import figure
from bokeh.models.sources import ColumnDataSource
source = ColumnDataSource(dict(x=list(range(5)), y=list(range(5))))
p = figure(width=300, height=300, tools=[], toolbar_location=None)
p.line(x='x', y='y', source=source)
curdoc().add_root(column(p, sizing_mode='scale_width'))
Run script
def run_single_server(abs_app_path, port=5000):
'''Run bokeh application for single session from server`'''
from bokeh.application import Application
from bokeh.application.handlers import DirectoryHandler
from bokeh.server.server import Server
import os
app_name = os.path.split(abs_app_path)[1]
url = '/{}'.format(app_name)
# Start a bokeh server
apps = {url:Application(DirectoryHandler(filename=abs_app_path))}
server = Server(apps, port=port)
server.start()
server.show(url)
# somehow wait for session to end, perhaps using `server_lifecycle.py`
server.stop()
return
def utility():
import mymodule
module_path = os.path.split(mymodule.__file__)[0]
abs_app_path = os.path.join(module_path, 'myapp')
run_single_server(abs_app_path, port=5000)
return
Perhaps have that routine in the main __init__.py, and have it work like this:
import mymodule
mymodule.utility()
# 1. Browser launches
# 2. user does stuff
# 3. user closes window
# 4. bokeh server is shutdown
Update
I found the build_single_handler_application routine and tried that, but it also appears to hang.
from bokeh.command.util import build_single_handler_application
import os
app_name = os.path.split(abs_app_path)[1]
url = '/{}'.format(app_name)
# Start a bokeh server
apps = build_single_handler_application(abs_app_path)

It looks like I had a couple of problems. I ended up finding and adapting some code that I found on the mail group here for my use-case.
I managed to get everything to work by using separate process for 1) starting the server, 2) launching the app urls with webbrowser, and 3) checking for closed connections and shutting down.
I think I could perhaps do away with initiating the tornado server instance as was done in the flask example I adapted, but I'm happy here.
Note: this example uses single file apps, but you can pass the paths of directory formatted apps as well.
def create_bokeh_server(io_loop, files, argvs, host, port):
'''Start bokeh server with applications paths'''
from bokeh.server.server import Server
from bokeh.command.util import build_single_handler_applications
# Turn file paths into bokeh apps
apps = build_single_handler_applications(files, argvs)
# kwargs lifted from bokeh serve call to Server, with created io_loop
kwargs = {
'io_loop':io_loop,
'generate_session_ids':True,
'redirect_root':True,
'use_x_headers':False,
'secret_key':None,
'num_procs':1,
'host': host,
'sign_sessions':False,
'develop':False,
'port':port,
'use_index':True
}
server = Server(apps,**kwargs)
return server
def run_single_app(files, port=5000, new='tab'):
def start_bokeh(io_loop):
'''Start the `io_loop`'''
io_loop.start()
return None
def launch_app(host, app_name, new):
'''Lauch app in browser
Ideally this would `bokeh.util.browser.view()`, but it doesn't work
'''
import webbrowser
# Map method strings to webbrowser method
options = {'current':0, 'window':1, 'tab':2}
# Concatenate url and open in browser, creating a session
app_url = 'http://{}/{}'.format(host, app_name)
print('Opening `{}` in browser'.format(app_url))
webbrowser.open(app_url, new=options[new])
return None
def server_loop(server, io_loop):
'''Check connections once session created and close on disconnect'''
import time
connected = [True,]
session_loaded = False
while any(connected):
# Check if no session started on server
sessions = server.get_sessions()
if not session_loaded:
if sessions:
session_loaded = True
# Once 1+ sessions started, check for no connections
else:
# List of bools for each session
connected = [True,]*len(sessions)
# Set `connected` item false no connections on session
for i in range(len(sessions)):
if sessions[i].connection_count == 0:
connected[i] = False
# Keep the pace down
time.sleep(2)
# Stop server once opened session connections closed
io_loop.stop()
return None
import os
import threading
import tornado.ioloop
import tornado.autoreload
import time
# Initialize some values, sanatize the paths to the bokeh plots
argvs = {}
app_names = []
for path in files:
argvs[path] = None
app_names.append(os.path.splitext(os.path.split(path)[1])[0])
# Concate hostname/port for creating handlers, launching apps
host = 'localhost:{}'.format(port)
# Initialize the tornado server
io_loop = tornado.ioloop.IOLoop.instance()
tornado.autoreload.start(io_loop)
# Add the io_loop to the bokeh server
server = run_bokeh_server(io_loop, files, argvs, host, port)
print('Starting the server on {}'.format(host))
args = (io_loop,)
th_startup = threading.Thread(target=start_bokeh, args=args)
th_startup.start()
# Launch each application in own tab or window
th_launch = [None,]*len(app_names)
for i in range(len(app_names)):
args = (host, app_names[i], new)
th_launch[i] = threading.Thread(target=launch_app, args=args)
th_launch[i].start()
# Delay to allow tabs to open in same browser window
time.sleep(2)
# Run session connection test, then stop `io_loop`
args = (server, io_loop)
th_shutdown = threading.Thread(target=server_loop, args=args)
th_shutdown.start()
return None
if __name__ == "__main__":
import os
files = [os.path.join('bokeh', fname) for fname in ['ex1.py','ex2.py']]
run_single_app(files, port=5006)

Multiprocess within flask app spinning up 2 processes

I am building a flask app and need some background processes to run. I decided to go with multiprocess, but it's producing two processes when running within Flask. Does anyone know why this would happen? I've tested it on OS X and Ubuntu 12.04, with the same results. Here is an example:
import time
import multiprocessing
from flask import Flask
app = Flask(__name__)
backProc = None
def testFun():
print('Starting')
while True:
time.sleep(3)
print('looping')
time.sleep(3)
print('3 Seconds Later')
#app.route('/')
def root():
return 'Started a background process with PID ' + str(backProc.pid) + " is running: " + str(backProc.is_alive())
#app.route('/kill')
def kill():
backProc.terminate()
return 'killed: ' + str(backProc.pid)
#app.route('/kill_all')
def kill_all():
proc = multiprocessing.active_children()
for p in proc:
p.terminate()
return 'killed all'
#app.route('/active')
def active():
proc = multiprocessing.active_children()
arr = []
for p in proc:
print(p.pid)
arr.append(p.pid)
return str(arr)
#app.route('/start')
def start():
global backProc
backProc = multiprocessing.Process(target=testFun, args=(), daemon=True)
backProc.start()
return 'started: ' + str(backProc.pid)
if __name__ == '__main__':
app.run(port=int("7879"))

This is a problem with the Flask auto-reload feature, which is used during development to automatically restart the webserver when changes in code is detected, in order to serve up the new code without requiring a manual restart.
In the guide, the “app.run()” call is always placed within an “if __name__ == ‘__main__’” condition, since the reloader is set to on by default. When using multiprocessing, this condition will result in false, so you have to instead disable the Flask autoreload when using it in a function like so:
def startWebserver():
app.run(debug=True, use_reloader=False)
Link for reference:
http://blog.davidvassallo.me/2013/10/23/nugget-post-python-flask-framework-and-multiprocessing/