Getting the specifics out of the way, I'm writing an open source P2P social network over IPFS and Flask -- I know, it's been done. I'm choosing Flask because pyinstaller can put it in an exe file.
I am attempting to update my IPNS every 10 minutes to publish all status updates I've added to the network during said 10 minutes. The cron function from setup class (from library.py) is where that updater function is stored. At first, I threaded the cron function from init of setup. The server hung. Then I moved the threading process over to app.before_first_request. The server still hangs.
https://pastebin.com/bXHTuH83 (main.py)
from flask import Flask, jsonify
from library import *
#=========================TO BE DELETED=========================================
def pretty(json):
json = dumps(loads(json), indent=4, sort_keys=True)
return json
#===============================================================================
app = Flask(__name__)
GANN = setup()
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
#app.route("/")
def home():
return "Hello World!!!"
if __name__ == "__main__":
app.run(port="80", debug=True, threaded=True)
https://pastebin.com/W5P8Tpvd (library.py)
from threading import Thread
from time import time, sleep
import urllib.request
from json import loads, dumps
def api(*argv, **kwargs):
url = "http://127.0.0.1:5001/api/v0/"
for arg in argv:
arg = arg.replace(" ", "/")
if arg[:-1] != "/":
arg += "/"
url += arg
url = url[0:-1]
if kwargs:
url+="?"
for val in kwargs:
url = url + val + "=" + kwargs[val] + "&"
url = url[0:-1]
print(url)
try:
with urllib.request.urlopen(url, timeout=300) as response:
return response.read()
except:
return b"""{"ERROR": "CANNOT CONNECT TO IPFS!"}"""
class setup():
def __init__(self):
api("files", "mkdir", arg="/GANN", parents="True")
self.root_hash = ""
def update_root(self):
try:
for entry in loads(api("files", "ls", l="True").decode())["Entries"]:
if entry["Name"] == "GANN":
self.root_hash = entry["Hash"]
except:
return """{"ERROR": "CANNOT FIND ROOT DIRECTORY"}"""
def publish_root(self):
api("name", "publish", arg=self.root_hash)
def cron(self):
while True:
print("CRON Thread Started!")
self.update_root()
self.publish_root()
sleep(600)
I have searched the web for a couple days and have yet to find a threading technique that will split from the main process and not hang the server from taking other requests. I believe I'm on a single stream connection, as IPFS blocks connections to every other device in my home when it's started. It takes a couple minutes for the CLI IPNS update to go through, so I set urllib's timeout to 300 seconds.
Well what I think the threading code is not correct.
#app.before_first_request
def cron_job():
Thread(target=GANN.cron())
Here you created a Thread object. The argument must be callable, but you called your method already here. so the right way would be
Thread(target=GANN.cron)
So the thread can call the cron function later. having said that, the Thread must be started, so it will call the function target you gave. So it must be ike
thread_cron = Thread(target=GANN.cron)
thread_cron.start()
Since you called the GANN.cron() , the method starts executing and your app hung!
Related
tl,dr: How can I programmably execute a python module (not function) as a separate process from a different python module?
On my development laptop, I have a 'server' module containing a bottle server. In this module, the name==main clause starts the bottle server.
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
if __name__ == '__main__':
serve(bt_app, port=localhost:8080)
I also have a 'test_server' module containing pytests. In this module, the name==main clause runs pytest and displays the results.
def test_something():
_rtn = some_server_function()
assert _rtn == desired
if __name__ == '__main__':
_rtn = pytest.main([__file__])
print("Pytest returned: ", _rtn)
Currently, I manually run the server module (starting the web server on localhost), then I manually start the pytest module which issues html requests to the running server module and checks the responses.
Sometimes I forget to start the server module. No big deal but annoying. So I'd like to know if I can programmatically start the server module as a separate process from the pytest module (just as I'm doing manually now) so I don't forget to start it manually.
Thanks
There is my test cases dir tree:
test
├── server.py
└── test_server.py
server.py start a web server with flask.
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello, World!'
if __name__ == '__main__':
app.run()
test_server.py make request to test.
import sys
import requests
import subprocess
import time
p = None # server process
def start_server():
global p
sys.path.append('/tmp/test')
# here you may want to do some check.
# whether the server is already started, then pass this fucntion
kwargs = {} # here u can pass other args needed
p = subprocess.Popen(['python','server.py'], **kwargs)
def test_function():
response = requests.get('http://localhost:5000/')
print('This is response body: ', response.text)
if __name__ == '__main__':
start_server()
time.sleep(3) # waiting server started
test_function()
p.kill()
Then you can do python test_server to start the server and do test cases.
PS: Popen() needs python3.5+. if older version, use run instead
import logging
import threading
import time
def thread_function(name):
logging.info("Thread %s: starting", name)
time.sleep(2)
logging.info("Thread %s: finishing", name)
if __name__ == "__main__":
format = "%(asctime)s: %(message)s"
logging.basicConfig(format=format, level=logging.INFO,
datefmt="%H:%M:%S")
threads = list()
for index in range(3):
logging.info("Main : create and start thread %d.", index)
x = threading.Thread(target=thread_function, args=(index,))
threads.append(x)
x.start()
for index, thread in enumerate(threads):
logging.info("Main : before joining thread %d.", index)
thread.join()
logging.info("Main : thread %d done", index)
With threading you can run multiple processes at once!
Wim baasically answered this question. I looked into the subprocess module. While reading up on it, I stumbled on the os.system function.
In short, subprocess is a highly flexible and functional program for running a program. os.system, on the other hand, is much simpler, with far fewer functions.
Just running a python module is simple, so I settled on os.system.
import os
server_path = "python -m ../src/server.py"
os.system(server_path)
Wim, thanks for the pointer. Had it been a full fledged answer I would have upvoted it. Redo it as a full fledged answer and I'll do so.
Async to the rescue.
import gevent
from gevent import monkey, spawn
monkey.patch_all()
from gevent.pywsgi import WSGIServer
#bt_app.post("/")
def server_post():
<< Generate response to 'http://server.com/' >>
def test_something():
_rtn = some_server_function()
assert _rtn == desired
print("Pytest returned: ",_rtn)
sleep(0)
if __name__ == '__main__':
spawn(test_something) #runs async
server = WSGIServer(("0.0.0.0", 8080, bt_app)
server.serve_forever()
I am working on Python's Flask server side code where there is a background task which runs periodically and executes a function (note that 'periodically' is not so hardline, execute once and then after x seconds also works). But I also need it to execute the same function immediately when the server receives a request (and then resume the background task).
This kind of reminds me of the SELECT system call in C, where the system waits for a timeout or until a packet arrives.
Here is what I came up minimally after looking up a lot of answers.
from flask import Flask, request
import threading, os, time
POOL_TIME = 2
myThread = threading.Thread()
def pollAndExecute(a='|'):
time.sleep(1)
print(time.time(), a)
# time.sleep(1)
myThread = threading.Timer(POOL_TIME, pollAndExecute)
myThread.start()
def startWork():
global myThread
myThread = threading.Timer(POOL_TIME, pollAndExecute)
myThread.start()
app = Flask(__name__)
#app.route('/ping', methods=['POST'])
def ping():
global myThread
myThread.cancel()
pollAndExecute("#")
return "Hello"
if __name__ == '__main__':
app.secret_key = os.urandom(12)
startWork()
app.run(port=5001)
Output:
But the output clearly says that it is not behaving properly after there is a request (sent using curl -X POST http://localhost:5001/ping)
Please guide me as to how to correct this or are there any other ways to do it. Just FYI, in the original code, there are various database updates in the pollAndExecute() as well and I need to take care that there are no race conditions between polling and ping. Needless to say, only one copy of the function should execute at a particular time (preferably in a single thread).
Here is the solution I made for your problem. I used a priority queue that takes in data to be run with the printTime function. The background and flask functions are two different threads that push data into the priority queue, which should prioritize the flask call over the background one. Notice how it now waits for the current thread to finish before executing another one.
from flask import Flask, request
import threading, os, time
from threading import Thread, Lock
from queue import PriorityQueue
POOL_TIME = 2
lock = Lock()
def printTime(a='|'):
time.sleep(1) # Simulate process taking 1 sec
print(time.time(), a)
jobs = PriorityQueue()
class Queue(Thread):
def __init__(self):
Thread.__init__(self)
self.daemon = True
self.start()
def run(self):
while True:
_, data = jobs.get()
printTime(data)
class backGroundProcess(Thread):
def __init__(self):
Thread.__init__(self)
self.daemon = True
self.start()
def run(self):
while True:
time.sleep(2) # Background process enqueues a job every 2 secs
jobs.put((0,"|"))
class flaskProcess(Thread):
def __init__(self):
Thread.__init__(self)
self.start()
def run(self):
jobs.put((1,"#"))
app = Flask(__name__)
#app.route('/ping', methods=['POST'])
def ping():
flaskThread = flaskProcess()
return "Hello"
if __name__ == '__main__':
backGroundProcess()
Queue()
app.secret_key = os.urandom(12)
app.run(port=5001)
The above snippet may be a little verbose because I used classes, but this should get you started.
So I have been playing around with Captcha, Multiprocessing and Flask.
What have I done so far:
Basically what I have done is that I have created own Mutltiprocessing where I enter how many tasks/processers I want to run during this script. It will say if I enter 3 then it will give me a process of 3 "threads" which works fine.
Also whenever webbrowser.open('http://Thrillofit.baller.com:5000/solve') opens, It works aswell. Also get a captcha that is able to solve.
Basically meaning the captcha does work and also the multiprocessing:
What's the issue then?
The issue is sitting now whenever I run the program it gets stuck when solving, basically I can solve how many captchas as I want but it will not continue which I can't get a grip of why? A GIF pic of what is happening: https://i.gyazo.com/d5f183471f20be5eda6be939d255a157.mp4
In the video you can see that i am trying to solve the captcha but nothing happeneds to the program like it gets stuck.
MY thoughts is that it might be a issue between Multiprocessing and Captcha/flask but I can't see the issue and I have been completely blind at this point. Maybe there is someone that might see the problem?
The goals is to whenever I solve a captcha, It should print out a token (Inside the def passwd(): method but nothing prints out and just gets stuck pretty much...
import requests, time, json, re, sys, os, timeit, random, multiprocessing, traceback, threading, names, logging, webbrowser, _thread
from flask import Flask, render_template, request, redirect
tokens = []
def captureToken(token):
expiry = datetime.now().timestamp() + 115
tokenDict = {
'expiry': expiry,
'token': token
}
tokens.append(tokenDict)
return
def sendToken():
while not tokens:
pass
token = tokens.pop(0)
return token['token']
def manageTokens():
while True:
for item in tokens:
if item['expiry'] < datetime.now().timestamp():
tokens.remove(item)
time.sleep(5)
app = Flask(__name__)
log = logging.getLogger('werkzeug')
log.setLevel(logging.ERROR)
#app.route('/', methods=['GET'])
def base():
return redirect("http://Thrillofit.baller.com:5000/solve", code=302)
#app.route('/solve', methods=['GET'])
def solve():
sitekey = "6LdyFRkUAAAAAF2YmQ9baZ6ytpVnbVSAymVpTXKi"
return render_template('index.html', sitekey=sitekey)
#app.route('/submit', methods=['POST'])
def submit():
token = request.form.get('g-recaptcha-response', '')
captureToken(token)
return redirect("http://Thrillofit.baller.com:5000/solve", code=302)
def info(thread):
global prod
prod = int(thread) + 1
passwd()
def passwd():
lilcapttoken = sendToken()
print(lilcaptoken)
def main():
webbrowser.open('http://Thrillofit.baller.com:5000/solve')
user_input = 0
while True:
try:
user_input = int(input(Fore.WHITE + 'How many tasks do you wanna run? [NUMBERS] \n' + Fore.RESET))
except ValueError:
print(Fore.RED + "Stop being stupid" + Fore.RESET)
continue
else:
HowManyThread = user_input
print()
i = 0
jobs = []
for i in range(HowManyThread):
p = multiprocessing.Process(target=info, args=(str(i),))
jobs.append(p)
time.sleep(.5)
p.start()
for p in jobs:
p.join()
sys.exit()
if __name__ == '__main__':
try:
_thread.start_new_thread(app.run, ())
_thread.start_new_thread(manageTokens, ())
main()
except Exception as e:
print(e)
print(traceback.print_exc())
print(traceback)
This question already has answers here:
Why does running the Flask dev server run itself twice?
(7 answers)
Closed 5 years ago.
I am running into a seemingly bizarre problem that I can't figure out - I am trying to create a single object, however for some reason, two are always made.
The two files control a robot i am building.
App.py
A flask server which receives in coming commands through a socket.io connection with clients
Hypervisor.py
Robot Controller, takes in user commands, sensor data, current agenda, and feeds commands to the MotionController.
I am trying to create a single instance of a the Hypervisor class from within the Flask server app.py, however two are consistently created.
Below is my code and console output showing the double object creation. Why is this happeneing!!?!?!?
App.py
#!/usr/bin/env python
from flask import Flask, render_template, session, request, send_from_directory, send_file
from flask_socketio import SocketIO, emit, join_room, leave_room, close_room, rooms, disconnect
import time
import json
import datetime
import logging
import platform
from bColors import bcolors
from RobotSystem.Hypervisor import Hypervisor
from RobotSystem.Services.Utilities.RobotUtils import RobotUtils
async_mode = None
app = Flask(__name__, static_url_path='/static')
app.config['SECRET_KEY'] = 'secret!'
socketio = SocketIO(app, async_mode=async_mode)
log = logging.getLogger("werkzeug")
log.setLevel(logging.ERROR)
thread = None
connections = 0
#app.route('/', methods=['GET', 'POST'])
def index():
return render_template('index.html', async_mode=socketio.async_mode)
def background_thread():
while True:
socketio.sleep(1)
#socketio.on('valueUpdate')
def valueUpdateHandler(message):
RobotUtils.ColorPrinter("app.py",'Value update fired ', 'OKBLUE')
quadbot.inputData(message)
data = {}
data['Recieved'] = True
return json.dumps(data)
#socketio.on('connect')
def test_connect():
global connections
connections+=1
print_str = "Client connected. "+ str(connections)+ " current connections"
RobotUtils.ColorPrinter("app.py",print_str, 'OKBLUE')
global thread, quadbotThread
if thread is None:
print "init"
thread = socketio.start_background_task(target=background_thread)
#socketio.on('disconnect')
def test_disconnect():
global connections
connections -= 1
RobotUtils.ColorPrinter("app.py",str( 'Client disconnected. ' +str(connections)+ " current connections" ), 'OKBLUE')
if __name__ == '__main__':
global quadbot
quadbot = Hypervisor()
socketio.run(app, debug=True)
Hypervisor.py
#!/usr/bin/python
from Services import *
import time
import math
import json
import sys
import threading
import os
from Queue import Queue,Empty
class Hypervisor():
def __init__(self):
if RobotUtils.LIVE_TESTING:
self.pwm = PWM()
self.pwm.setPWMFreq(RobotUtils.FREQUENCY)
else:
self.pwm = None
self.inputQueue = Queue()
self.agendaThread = threading.Thread(group=None,target=self.updateAgendaLoop,name="agendaThread")
self.agendaThread.start()
self.data_file_name = RobotUtils.DATA_FILE
self.front_left = None
self.front_right = None
self.back_left = None
self.back_right = None
self.TURN_LEFT = RobotUtils.TURN_LEFT
self.TURN_RIGHT = RobotUtils.TURN_RIGHT
self.FORWARD = RobotUtils.FORWARD
self.BACKWARD = RobotUtils.BACKWARD
self.STOP = RobotUtils.STOP
self.AUTONOMOUS = RobotUtils.AUTONOMOUS
self.INVALID_DATA_ERROR = RobotUtils.INVALID_DATA_ERROR
self.horizVidMotor = Motor(50, RobotUtils.HORIZONTAL_VID_PIN, RobotUtils.HORIZONTAL_VID_MIN_VAL, RobotUtils.HORIZONTAL_VID_MAX_VAL, 0, "horizontal video motor", self.pwm)
self.vertVidMotor = Motor( 50, RobotUtils.VERTICAL_VID_PIN, RobotUtils.VERTICAL_VID_MIN_VAL, RobotUtils.VERTICAL_VID_MAX_VAL, 0, "vertical video motor", self.pwm)
self.setup()
self.motors = [self.front_left, self.front_right,self.back_left,self.back_right, self.horizVidMotor, self.vertVidMotor ]
self.MotionController = MotionController(self.TURN_LEFT, self.TURN_RIGHT, self.FORWARD, self.BACKWARD, self.STOP,self.AUTONOMOUS,self.INVALID_DATA_ERROR,
self.motors, RobotUtils
)
self.stand()
RobotUtils.ColorPrinter(self.__class__.__name__, '__init__() finished. Robot Created with id ' +str(id(self)), 'OKBLUE')
# loads json data and creates Leg objects with add_leg()
def setup(self):
with open(self.data_file_name) as data_file:
data = json.load(data_file)
constants = data["constants"]
for i in range(len(data["legs"])):
self.add_leg(data["legs"][i],constants)
# reads dictuanary values from input, creates a Leg object, and adds it to leg variables
def add_leg(self,legData,constants):
leg_name = legData["name"]
body_pin = legData["motors"]["body"]["pinValue"]
body_offset = legData["motors"]["body"]["offset"]
body_center = constants["bodyCenterValue"] + body_offset
body_min = constants["bodyRange"]["min"]
body_max = constants["bodyRange"]["max"]
mid_horiz_value = legData["motors"]["middle"]["horizValue"]
middle_pin = legData["motors"]["middle"]["pinValue"]
middle_min = constants["middleRange"]["min"]
middle_max = constants["middleRange"]["max"]
middle_offset_to_center = constants["midOffsetFromHoriz"]
leg_horiz_value = legData["motors"]["leg"]["horizValue"]
leg_pin = legData["motors"]["leg"]["pinValue"]
leg_min = constants["legRange"]["min"]
leg_max = constants["legRange"]["max"]
leg_offset_to_center = constants["legOffsetFromHoriz"]
leg = Leg( self.pwm, leg_name, body_pin, body_min, body_max, body_center, mid_horiz_value, middle_pin, middle_min, middle_max, middle_offset_to_center, leg_horiz_value, leg_pin, leg_min, leg_max, leg_offset_to_center)
if leg_name == "FR":
self.front_right = leg
elif leg_name == "FL":
self.front_left = leg
elif leg_name == "BL":
self.back_left = leg
elif leg_name == "BR":
self.back_right = leg
else:
print "ERROR: LEG CANNOT BE IDENTIFIED"
# Called by server when a change in user data is detected
def inputData(self,data):
self.inputQueue.put(data)
def updateAgendaLoop(self):
while True:
try:
data = self.inputQueue.get_nowait()
self.updateAgenda(data)
except Empty:
pass
time.sleep(RobotUtils.AGENDA_UPDATE_SPEED)
print '\033[94m' + "Robot: QUEUE READING FINISHED" + '\033[0m'
sys.exit()
# acts as central coordinator for the robot - raeads incoming data + state of the bot and calls methods accordingly
def updateAgenda(self,data):
self.MotionController.updateCameras(data)
nextMove = self.MotionController.NextMove(data)
if nextMove == self.INVALID_DATA_ERROR:
print "Fix this"
else:
self.MotionController.MakeMove(nextMove)
Console Output
First of all a little soapboxing:
If you provide a SSCCE (Short, Self-Contained, Correct (Compilable), Example) then you are more likely to get a response. Also by trimming the example to the minimum required to reproduce you may very well identify the source of the problem yourself. For example the following would be a SSCCE for your issue:
Required dependancies:
pip install flask
pip install flask-socketio
Code:
import logging
from flask import Flask
from flask_socketio import SocketIO
logging.basicConfig(level=logging.ERROR)
app = Flask(__name__)
socketio = SocketIO(app)
class Hypervisor():
def __init__(self):
print('Hypervisor initialized')
if __name__ == '__main__':
quadbot = Hypervisor()
socketio.run(app, debug=True)
Output:
Hypervisor initialized
Hypervisor initialized
Explanation
If you use a debugger the answer presents itself fairly easily. You can use the debugger tools included in your IDE, or you can always use The Python Debugger from the python standard library (a.k.a. pdb).
While it's beyond the scope of this answer to provide a complete tutorial of pdb, the method we will use to debug is to break into the debugger by importing pdb and inserting the following pdb.set_trace() at the point you want to begin debugging.
Since the issue is with Hypervisor creation the logical point to insert the debugger is just before the initialization of Hypervisor like so:
if __name__ == '__main__':
import pdb
global quadbot
pdb.set_trace() # debugging begins here
quadbot = Hypervisor()
socketio.run(app, debug=True)
From this point when you run your app, it will drop into pdb right before the initialization of Hypervisor. Before you do so though, you will need to know two commands for using pdb.
The first is n for next which continues execution until to the next line of code in the current function.
The second is s for step which will step into the current code stopping at first opportunity.
Or as the docs explain it:
The difference between next and step is that step stops inside a called function, while next executes called functions at (nearly) full speed, only stopping at the next line in the current function.
So armed with this knowledge run your app, and you will be presented with the pdb prompt which should look something like this:
-> quadbot = Hypervisor()
(Pdb)
The -> shows the current line of code about to be executed. (Pdb) is the prompt awaiting input. At this point let's initialize the Hypervisor by proceeding to the next line of code by typing n and enter. At this point you should see a single Hypervisor has been created.
[4/3/2017 20:02:46 ] Hypervisor: __init__() finished. Robot Created with id 4218654299
And be returned to the pdb prompt for the next line:
-> socketio.run(app, debug=True)
(Pdb)
So since there's only one more line of code left to run, the issue is somewhere in socketio.run. So this time we will step into the current line of code by typing s and enter which will bring you to:
-> def run(self, app, host=None, port=None, **kwargs):
(Pdb)
From this point, keep stepping to the next line of code until you see the second Hypervisor initialization. Looking at the trace you should see something like the following:
-> app.run(host=host, port=port, threaded=True,
(Pdb) n
-> use_reloader=use_reloader, **kwargs)
(Pdb) n
-> quadbot = Hypervisor()
(Pdb) n
[4/3/2017 20:03:52 ] Hypervisor: __init__() finished. Robot Created with id 4367452293
This shows you that right after the app.run executes (it takes two n's to execute since the command spans two lines of code), it returns to the quadbot = Hypervisor() line of code we started at. So a closer look at app.run shows an argument called use_reloader. If you haven't guessed already, looking at the Flask-SocketIO docs tells us:
use_reloader - True to enable the Flask reloader, False to disable it.
With a little more digging we can find the following two nuggets of wisdom in the Flask docs:
debug
The debug flag. Set this to True to enable debugging of the application. In debug mode the debugger will kick in when an unhandled exception occurs and the integrated server will automatically reload the application if changes in the code are detected.
and
run...
Flask will suppress any server error with a generic error page unless it is in debug mode. As such to enable just the interactive debugger without the code reloading, you have to invoke run() with debug=True and use_reloader=False. Setting use_debugger to True without being in debug mode won’t catch any exceptions because there won’t be any to catch.
So armed with this information you can choose to either disable debug, or pass the option use_reloader=False to socketio.run.
TL;DR
You most likely want to disable the Flask reloader by passing use_reloader=False to socketio.run
if __name__ == '__main__':
global quadbot
quadbot = Hypervisor()
socketio.run(app, debug=True, use_reloader=False)
I have a scenario where I want to show output of a long running script through a Flask API. I followed an example given for Flask and it works. I get dmesg steam in my browser.
import subprocess
import time
from flask import Flask, Response
app = Flask(__name__)
#app.route('/yield')
def index():
def inner():
proc = subprocess.Popen(
['dmesg'], # call something with a lot of output so we can see it
shell=True,
stdout=subprocess.PIPE
)
for line in iter(proc.stdout.readline,''):
time.sleep(1) # Don't need this just shows the text streaming
yield line.rstrip() + '<br/>\n'
return Response(inner(), mimetype='text/html') # text/html is required for most browsers to show this
The thing is, I have been using Flask-Restful from a long time. So I want to do the streaming using it. I tried it and it's not working.
import subprocess
import time
from flask import Response
from flask_restful import Resource
class CatalogStrings(Resource):
def get(self):
return Response(inner(), mimetype='text/html')
def inner():
proc = subprocess.Popen(
['dmesg'], # call something with a lot of output so we can see it
shell=True,
stdout=subprocess.PIPE
)
for line in iter(proc.stdout.readline, ''):
time.sleep(1) # Don't need this just shows the text streaming
yield line.rstrip() + '<br/>\n'
Please help