I have a scenario where I want to show output of a long running script through a Flask API. I followed an example given for Flask and it works. I get dmesg steam in my browser.
import subprocess
import time
from flask import Flask, Response
app = Flask(__name__)
#app.route('/yield')
def index():
def inner():
proc = subprocess.Popen(
['dmesg'], # call something with a lot of output so we can see it
shell=True,
stdout=subprocess.PIPE
)
for line in iter(proc.stdout.readline,''):
time.sleep(1) # Don't need this just shows the text streaming
yield line.rstrip() + '<br/>\n'
return Response(inner(), mimetype='text/html') # text/html is required for most browsers to show this
The thing is, I have been using Flask-Restful from a long time. So I want to do the streaming using it. I tried it and it's not working.
import subprocess
import time
from flask import Response
from flask_restful import Resource
class CatalogStrings(Resource):
def get(self):
return Response(inner(), mimetype='text/html')
def inner():
proc = subprocess.Popen(
['dmesg'], # call something with a lot of output so we can see it
shell=True,
stdout=subprocess.PIPE
)
for line in iter(proc.stdout.readline, ''):
time.sleep(1) # Don't need this just shows the text streaming
yield line.rstrip() + '<br/>\n'
Please help
Related
So I have a situation where I have an Flask endpoint A, endpoint B and two other scripts foo.py and bar.py.
When I call endpoint A, I will do a call for foo.py with Popen and store its PID.
On foo.py, it makes a call to bar.py using Popen, which makes another call, again, using Popen. The process opened on bar.py is a server (to be more specific, it's a tf-serving server), which will be hanging forever when I do an p.wait(). Later on, I would like to use endpoint B to end the whole process triggered by A.
The situation can be something like:
Flask's endpoints:
import os
import json
import signal
from subprocess import Popen
from flask import current_app
from flask import request, jsonify
#app.route('/A', methods=['GET'])
def a():
p = Popen(['python', '-u','./foo.py'])
current_app.config['FOO_PID'] = p.pid
return jsonify({'message': 'Started successfully'}), 200
#inspection.route('/B', methods=['GET'])
def b():
os.kill(current_app.config['FOO_PID'], signal.SIGTERM)
return jsonify({'message': 'Stopped successfully'}), 200
foo.py:
p = Popen(['python' ,'-u', './bar.py', '--serve'])
while True:
continue
bar.py:
command = f'tensorflow_model_server --rest_api_port=8501 --model_name=obj_det --model_base_path=./model'
p = subprocess.Popen(command, shell=True, stderr=sys.stderr, stdout=sys.stdout)
p.wait()
Unfortunately when I kill foo.py using endpoint B, the process created by bar.py (ie. the server) will not end. How can I kill the server?
Please consider a solution that is OS agnostic.
Using a package like psutil allows you to recursively iterate and access of all child processes related to a certain PID. This will effectively allow you to kill all nested processes. Documentation for psutil https://github.com/giampaolo/psutil.
import json
import signal
from subprocess import Popen
from flask import current_app
from flask import request, jsonify
from psutil import Process
#app.route('/A', methods=['GET'])
def a():
p = Popen(['python', '-u','./foo.py'])
current_app.config['FOO_PID'] = p.pid
return jsonify({'message': 'Started successfully'}), 200
#inspection.route('/B', methods=['GET'])
def b():
pid = current_app.config['FOO_PID']
parent = Process(pid)
for child in parent.children(recursive=True):
child.kill()
parent.kill()
return jsonify({'message': 'Stopped successfully'}), 200
Getting the specifics out of the way, I'm writing an open source P2P social network over IPFS and Flask -- I know, it's been done. I'm choosing Flask because pyinstaller can put it in an exe file.
I am attempting to update my IPNS every 10 minutes to publish all status updates I've added to the network during said 10 minutes. The cron function from setup class (from library.py) is where that updater function is stored. At first, I threaded the cron function from init of setup. The server hung. Then I moved the threading process over to app.before_first_request. The server still hangs.
https://pastebin.com/bXHTuH83 (main.py)
from flask import Flask, jsonify
from library import *
#=========================TO BE DELETED=========================================
def pretty(json):
json = dumps(loads(json), indent=4, sort_keys=True)
return json
#===============================================================================
app = Flask(__name__)
GANN = setup()
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
#app.route("/")
def home():
return "Hello World!!!"
if __name__ == "__main__":
app.run(port="80", debug=True, threaded=True)
https://pastebin.com/W5P8Tpvd (library.py)
from threading import Thread
from time import time, sleep
import urllib.request
from json import loads, dumps
def api(*argv, **kwargs):
url = "http://127.0.0.1:5001/api/v0/"
for arg in argv:
arg = arg.replace(" ", "/")
if arg[:-1] != "/":
arg += "/"
url += arg
url = url[0:-1]
if kwargs:
url+="?"
for val in kwargs:
url = url + val + "=" + kwargs[val] + "&"
url = url[0:-1]
print(url)
try:
with urllib.request.urlopen(url, timeout=300) as response:
return response.read()
except:
return b"""{"ERROR": "CANNOT CONNECT TO IPFS!"}"""
class setup():
def __init__(self):
api("files", "mkdir", arg="/GANN", parents="True")
self.root_hash = ""
def update_root(self):
try:
for entry in loads(api("files", "ls", l="True").decode())["Entries"]:
if entry["Name"] == "GANN":
self.root_hash = entry["Hash"]
except:
return """{"ERROR": "CANNOT FIND ROOT DIRECTORY"}"""
def publish_root(self):
api("name", "publish", arg=self.root_hash)
def cron(self):
while True:
print("CRON Thread Started!")
self.update_root()
self.publish_root()
sleep(600)
I have searched the web for a couple days and have yet to find a threading technique that will split from the main process and not hang the server from taking other requests. I believe I'm on a single stream connection, as IPFS blocks connections to every other device in my home when it's started. It takes a couple minutes for the CLI IPNS update to go through, so I set urllib's timeout to 300 seconds.
Well what I think the threading code is not correct.
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
Here you created a Thread object. The argument must be callable, but you called your method already here. so the right way would be
Thread(target=GANN.cron)
So the thread can call the cron function later. having said that, the Thread must be started, so it will call the function target you gave. So it must be ike
thread_cron = Thread(target=GANN.cron)
thread_cron.start()
Since you called the GANN.cron() , the method starts executing and your app hung!
I am trying to connect to a backend (other python script) where a response is getting generated. This takes a little and seems like its crashing the response of bottle. Please have a look here:
#!/usr/bin/env python
# -*- coding: utf-8 -*-s
from bottle import route, run, response, request
import subprocess
sherlockQueryScriptPath = ".\sherlock-backend\Sherlock_Queries.py"
emptyJsonString = "{}"
def main():
#print prepareJson("test")
#sys.exit(0)
run(host='localhost', port=8080, threaded=True)
def prepareJson(query):
queryProcess = subprocess.Popen(
[sherlockQueryScriptPath, '-j', '-q', query],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=True
)
queryProcess.wait() # <== seems to cause problems
result, error = queryProcess.communicate()
if error:
print "Error: ", error
return emptyJsonString
return result.strip()
def cors(func):
def wrapper(*args, **kwargs):
response.content_type = 'application/json; charset=UTF8'
response.set_header("Access-Control-Allow-Origin", "*")
response.set_header("Access-Control-Allow-Methods", "GET, POST, OPTIONS")
response.set_header("Access-Control-Allow-Headers", "Origin, Content-Type")
return func(*args, **kwargs)
return wrapper
#route('/sherlockQuery.json')
#cors
def respondToGet():
query = request.query.q
return prepareJson(query)
#route('/sherlockQuery.json', method='POST')
#cors
def respondToPost():
query = request.forms.get('q')
return prepareJson(query)
if __name__ == "__main__":
main()
Could someone tell my how to make its work? Or hold the connection till response is ready and then send it?
I think a quick code snippet is better to explain my problem, so please have a look at this:
from flask import Flask
from flask.ext.socketio import SocketIO
from threading import Thread
import subprocess
import threading
from eventlet.green.subprocess import Popen
app = Flask(__name__)
socketio = SocketIO(app)
def get_tasks_and_emit():
instance = Popen(["tasklist"], stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1)
lines_iterator = iter(instance.stdout.readline, b"")
data = ""
for line in lines_iterator:
data += line.decode("utf8")
socketio.emit("loaded", data)
print("::: DEBUG - returned tasks with thread")
#app.route("/")
def index():
html = "<!DOCTYPE html>"
html += "<script src=https://code.jquery.com/jquery-2.2.0.min.js></script>"
html += "<script src=https://cdn.socket.io/socket.io-1.4.5.js></script>"
html += "<script>"
html += "var socket = io.connect(window.location.origin);"
html += "socket.on('loaded', function(data) {alert(data);});"
html += "function load_tasks_threaded() {$.get('/tasks_threaded');}"
html += "function load_tasks_nonthreaded() {$.get('/tasks');}"
html += "</script>"
html += "<button onclick='load_tasks_nonthreaded()'>Load Tasks</button>"
html += "<button onclick='load_tasks_threaded()'>Load Tasks (Threaded)</button>"
return html
#app.route("/tasks")
def tasks():
get_tasks_and_emit()
print("::: DEBUG - returned tasks without thread")
return ""
#app.route("/tasks_threaded")
def tasks_threaded():
threading.Thread(target=get_tasks_and_emit).start()
return ""
if __name__ == "__main__":
socketio.run(app, port=7000, debug=True)
I am running this code on Windows using eventlet, if I don't use eventlet everything is fine (but of course much slower due to the werkzeug threading mode). (And I just checked and it's not working on Linux either)
I hope someone can point me into the right direction. (My Python version is 3.5.1 by the way)
I found the problem. Apparently you have to monkey patch the threading module, so I added
import eventlet
eventlet.monkey_patch(thread=True)
and then I also had a problem with long running programs. I had the same problem as the guy in this StackOverflow post:
Using Popen in a thread blocks every incoming Flask-SocketIO request
So I added
eventlet.sleep()
to the for loop that processes the pipes.
EDIT:
As temoto pointed out, alternatively one can also just use the threading module from eventlet.green like this:
from eventlet.green import threading
I have the following situation:
I receive a request on a socketio server. I answer it (socket.emit(..)) and then start something with heavy computation load in another thread.
If the heavy computation is caused by subprocess.Popen (using subprocess.PIPE) it totally blocks every incoming request as long as it is being executed although it happens in a separate thread.
No problem - in this thread it was suggested to asynchronously read the result of the subprocess with a buffer size of 1 so that between these reads other threads have the chance to do something. Unfortunately this did not help for me.
I also already monkeypatched eventlet and that works fine - as long as I don't use subprocess.Popen with subprocess.PIPE in the thread.
In this code sample you can see that it only happens using subprocess.Popen with subprocess.PIPE. When uncommenting #functionWithSimulatedHeavyLoad() and instead comment functionWithHeavyLoad() everything works like charm.
from flask import Flask
from flask.ext.socketio import SocketIO, emit
import eventlet
eventlet.monkey_patch()
app = Flask(__name__)
socketio = SocketIO(app)
import time
from threading import Thread
#socketio.on('client command')
def response(data, type = None, nonce = None):
socketio.emit('client response', ['foo'])
thread = Thread(target = testThreadFunction)
thread.daemon = True
thread.start()
def testThreadFunction():
#functionWithSimulatedHeavyLoad()
functionWithHeavyLoad()
def functionWithSimulatedHeavyLoad():
time.sleep(5)
def functionWithHeavyLoad():
from datetime import datetime
import subprocess
import sys
from queue import Queue, Empty
ON_POSIX = 'posix' in sys.builtin_module_names
def enqueueOutput(out, queue):
for line in iter(out.readline, b''):
if line == '':
break
queue.put(line)
out.close()
# just anything that takes long to be computed
shellCommand = 'find / test'
p = subprocess.Popen(shellCommand, universal_newlines=True, shell=True, stdout=subprocess.PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target = enqueueOutput, args = (p.stdout, q))
t.daemon = True
t.start()
t.join()
text = ''
while True:
try:
line = q.get_nowait()
text += line
print(line)
except Empty:
break
socketio.emit('client response', {'text': text})
socketio.run(app)
The client receives the message 'foo' after the blocking work in the functionWithHeavyLoad() function is completed. It should receive the message earlier, though.
This sample can be copied and pasted in a .py file and the behavior can be instantly reproduced.
I am using Python 3.4.3, Flask 0.10.1, flask-socketio1.2, eventlet 0.17.4
Update
If I put this into the functionWithHeavyLoad function it actually works and everything's fine:
import shlex
shellCommand = shlex.split('find / test')
popen = subprocess.Popen(shellCommand, stdout=subprocess.PIPE)
lines_iterator = iter(popen.stdout.readline, b"")
for line in lines_iterator:
print(line)
eventlet.sleep()
The problem is: I used find for heavy load in order to make the sample for you more easily reproducable. However, in my code I actually use tesseract "{0}" stdout -l deu as the sell command. This (unlike find) still blocks everything. Is this rather a tesseract issue than eventlet? But still: how can this block if it happens in a separate thread where it reads line by line with context switch when find does not block?
Thanks to this question I learned something new today. Eventlet does offer a greenlet friendly version of subprocess and its functions, but for some odd reason it does not monkey patch this module in the standard library.
Link to the eventlet implementation of subprocess: https://github.com/eventlet/eventlet/blob/master/eventlet/green/subprocess.py
Looking at the eventlet patcher, the modules that are patched are os, select, socket, thread, time, MySQLdb, builtins and psycopg2. There is absolutely no reference to subprocess in the patcher.
The good news is that I was able to work with Popen() in an application very similar to yours, after I replaced:
import subprocess
with:
from eventlet.green import subprocess
But note that the currently released version of eventlet (0.17.4) does not support the universal_newlines option in Popen, you will get an error if you use it. Support for this option is in master (here is the commit that added the option). You will either have to remove that option from your call, or else install the master branch of eventlet direct from github.