We Keep Coding

Run Flask webserver parallel from main code?

Context:
For the Raspberry Pi i am developing some home automation tools.
At one side i have my main application that reads a CSV file, that consists of date+time entries with a GPIO port number and a duration it needs to send a signal to that port.
My main app reads this CSV, creates a small list of entries of this and then basically checks every 60 seconds if there is any job to do.
So far so good, this works like a charm.
Now on the other half, i am trying to run a Flask webservice so i can directly interact with this schedule, overwrite, push to refresh the csv, and so on.
Later on (future music) i am thinking of making some android app that has a nice GUI that talks with this webservice.
But i keep struggling to start the webservice and then kick off the main app (read csv; execute loop)
some code snipit:
import threading
from flask import Flask, render_template, request
from dwe_homeautomation_app import runMainWorker
app = Flask(__name__)
# Some routing samples
#app.route('/app/breakLoop')
def breakLoop():
m_worker.breakLoop = True # set global var to exit the 60 sec loop
return "break!"
if __name__ == '__main__':
# TODO: how to run this parallel ?
t1 = threading.Thread(target=app.run(debug=True, use_reloader=False, port=5000, host='0.0.0.0')) # Flask webserver
t2 = threading.Thread(target=runMainWorker()) # The main app that reads the csv and executes the 60 sec loop
t1.start()
t2.start()
As i was reading some topics trough google and stack overflow, but i couldn't really figure out how to get this working in my code; i saw some advice about multi threading, though the info and advice doesn't seem to be very in sync with eachother.
For some reason t1 (the webservice) starts, but t2 doesn't start at all.
Im relative new to Python, so i might be missing the obvious here.
Any advice, pointing me in the right direction, or pointing me my mistake in the code sample is much appreciated.

Try that:
from flask import Flask, render_template, request
from threading import Thread
app = Flask(__name__)
# Some routing samples
#app.route('/app/breakLoop')
def breakLoop():
m_worker.breakLoop = True # set global var to exit the 60 sec loop
return "break!"
def runApp():
app.run(debug=True, use_reloader=False, port=5000, host='0.0.0.0')
if __name__ == '__main__':
# TODO: how to run this parallel ?
Thread(target = runApp).start()
Thread(target = runMainWorker).start()

Check the threading.Thread docs:
https://docs.python.org/3/library/threading.html#thread-objects
You have to pass the target without brackets and args/kwargs as defined in the docs.

Use zerorpc inside Flask app throws error "operation would block forever"

I have a RPC Server using zerorpc in Python, written this way
import zerorpc
from service import Service
print('RPC server - loading')
def main():
print('RPC server - main')
s = zerorpc.Server(Service())
s.bind("tcp://*:4242")
s.run()
if __name__ == "__main__" : main()
It works fine when I create a client
import zerorpc, sys
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
name = sys.argv[1] if len(sys.argv) > 1 else "dude"
print(client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm'))
and runs it. The print() outputs what this 'videos' function returns.
But when I try to use it this same code inside route from a Flask app, I receive the following error:
File "src/gevent/__greenlet_primitives.pxd", line 35, in
gevent.__greenlet_primitives._greenlet_switch
gevent.exceptions.LoopExit: This operation would block forever Hub:
The flask method/excerpt
import zerorpc, sys
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
#app.route('/videos', methods=['POST'])
def videos():
global client_rpc
client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm')
I can't find out what might be happening. I'm quite new to Python and I understand that this may have some relation with Flask and how it handles the thread, but I can't figure out how to solve it.

zerorpc depends on gevent, which provides async IO with cooperative coroutines. This means your flask application must use gevent for all IO operations.
In your specific case, you are likely starting your application with a standard blocking IO WSGI server.
Here is a snippet using the WSGI server from gevent:
import zerorpc
from gevent.pywsgi import WSGIServer
app = Flask(__name__)
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
#app.route('/videos', methods=['POST'])
def videos():
global client_rpc
client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm')
# ...
if __name__ == "__main__":
http = WSGIServer(('', 5000), app)
http.serve_forever()
Excerpt from https://sdiehl.github.io/gevent-tutorial/#chat-server

flask REST API with multithreading is not working

I have designed a REST API which receives inputs through POST requests and then applies some logic to the inputs and returns to the callback uri which is part of the inputs.
This design was working fine for single input, but then i want to implement multithreading so that i can handle multiple POST requests. I have tried using 'app.run(threaded=True)' but was not successful.
I am running this code on linux platform. Not sure what is wrong in the following code, and am not so good at using threads in python, would appreciate if someone can let me know where the issue is:
I am able to get the '200' response once there is a POST request and the inputs are appended to 'inp_params', after which there is no processing in the thread.
from flask import Flask, jsonify, request
import time
import json
import os
import threading
import Queue
import test_func_module as tf
app = Flask(__name__)
inp_params = []
# Create the queue and threader
q = Queue.Queue()
#app.route('/', methods = ['GET', 'POST'] )
def get_data():
if request.method == 'GET':
return 'RESTful API'
elif request.method == 'POST':
global inp_params
inputs = {"fileName": request.json["fileName"], "fileId": request.json["fileId"], "ModuleId": request.json["ModuleId"], "WorkflowId": request.json["WorkflowId"],"Language": request.json["Language"], "callbackuri": request.json["callbackuri"]}
inp_params.append(inputs)
return '200'
def test_integrate(worker):
TF_output = tf.test_func(worker)
return TF_output
def threader():
while True:
# gets an worker from the queue
worker = q.get()
# Run the example job with the avail worker in queue (thread)
test_integrate(worker)
# completed with the job
q.task_done()
if __name__ == '__main__':.
for worker in inp_params:
q.put(worker)
for x in range(4): #4 cores
t = threading.Thread(target=threader)
# classifying as a daemon, so they will die when the main dies
t.daemon = True
# begins, must come after daemon definition
t.start()
# wait until the thread terminates.
q.join()
app.run(threaded=True)

#Shilparani Since you mentioned
I have tried using 'app.run(threaded=True)' but was not successful.
May not be exact answer for your question but I would like to share my experience for achieving concurrency through uwsgi/gunicorn :
Keep it simple by coding Flask for REST endpoints and move Multithreading , MultiProcessing logic to gunicorn or uwsgi where you can mention threads and workers which help for achieving concurrency , parallelism if that's what you are trying to achieve.
gunicorn -b localhost:8080 -w 4 -t 4 app:app
Based on your need and operations:
If tasks are CPU intensive try to keep #workers as #CPU-cores
If tasks are I/O intensive may be safe to try with more threads

How to schedule a function to run every hour on Flask?

I have a Flask web hosting with no access to cron command.
How can I execute some Python function every hour?

You can use BackgroundScheduler() from APScheduler package (v3.5.3):
import time
import atexit
from apscheduler.schedulers.background import BackgroundScheduler
def print_date_time():
print(time.strftime("%A, %d. %B %Y %I:%M:%S %p"))
scheduler = BackgroundScheduler()
scheduler.add_job(func=print_date_time, trigger="interval", seconds=60)
scheduler.start()
# Shut down the scheduler when exiting the app
atexit.register(lambda: scheduler.shutdown())
Note that two of these schedulers will be launched when Flask is in debug mode. For more information, check out this question.

I'm a little bit new with the concept of application schedulers, but what I found here for APScheduler v3.3.1 , it's something a little bit different. I believe that for the newest versions, the package structure, class names, etc., have changed, so I'm putting here a fresh solution which I made recently, integrated with a basic Flask application:
#!/usr/bin/python3
""" Demonstrating Flask, using APScheduler. """
from apscheduler.schedulers.background import BackgroundScheduler
from flask import Flask
def sensor():
""" Function for test purposes. """
print("Scheduler is alive!")
sched = BackgroundScheduler(daemon=True)
sched.add_job(sensor,'interval',minutes=60)
sched.start()
app = Flask(__name__)
#app.route("/home")
def home():
""" Function for test purposes. """
return "Welcome Home :) !"
if __name__ == "__main__":
app.run()
I'm also leaving this Gist here, if anyone have interest on updates for this example.
Here are some references, for future readings:
APScheduler Doc: https://apscheduler.readthedocs.io/en/latest/
daemon=True: https://docs.python.org/3.4/library/threading.html#thread-objects

You could make use of APScheduler in your Flask application and run your jobs via its interface:
import atexit
# v2.x version - see https://stackoverflow.com/a/38501429/135978
# for the 3.x version
from apscheduler.scheduler import Scheduler
from flask import Flask
app = Flask(__name__)
cron = Scheduler(daemon=True)
# Explicitly kick off the background thread
cron.start()
#cron.interval_schedule(hours=1)
def job_function():
# Do your work here
# Shutdown your cron thread if the web process is stopped
atexit.register(lambda: cron.shutdown(wait=False))
if __name__ == '__main__':
app.run()

I've tried using flask instead of a simple apscheduler what you need to install is
pip3 install flask_apscheduler
Below is the sample of my code:
from flask import Flask
from flask_apscheduler import APScheduler
app = Flask(__name__)
scheduler = APScheduler()
def scheduleTask():
print("This test runs every 3 seconds")
if __name__ == '__main__':
scheduler.add_job(id = 'Scheduled Task', func=scheduleTask, trigger="interval", seconds=3)
scheduler.start()
app.run(host="0.0.0.0")

For a simple solution, you could add a route such as
#app.route("/cron/do_the_thing", methods=['POST'])
def do_the_thing():
logging.info("Did the thing")
return "OK", 200
Then add a unix cron job that POSTs to this endpoint periodically. For example to run it once a minute, in terminal type crontab -e and add this line:
* * * * * /opt/local/bin/curl -X POST https://YOUR_APP/cron/do_the_thing
(Note that the path to curl has to be complete, as when the job runs it won't have your PATH. You can find out the full path to curl on your system by which curl)
I like this in that it's easy to test the job manually, it has no extra dependencies and as there isn't anything special going on it is easy to understand.
Security
If you'd like to password protect your cron job, you can pip install Flask-BasicAuth, and then add the credentials to your app configuration:
app = Flask(__name__)
app.config['BASIC_AUTH_REALM'] = 'realm'
app.config['BASIC_AUTH_USERNAME'] = 'falken'
app.config['BASIC_AUTH_PASSWORD'] = 'joshua'
To password protect the job endpoint:
from flask_basicauth import BasicAuth
basic_auth = BasicAuth(app)
#app.route("/cron/do_the_thing", methods=['POST'])
#basic_auth.required
def do_the_thing():
logging.info("Did the thing a bit more securely")
return "OK", 200
Then to call it from your cron job:
* * * * * /opt/local/bin/curl -X POST https://falken:joshua#YOUR_APP/cron/do_the_thing

You could try using APScheduler's BackgroundScheduler to integrate interval job into your Flask app. Below is the example that uses blueprint and app factory (init.py) :
from datetime import datetime
# import BackgroundScheduler
from apscheduler.schedulers.background import BackgroundScheduler
from flask import Flask
from webapp.models.main import db
from webapp.controllers.main import main_blueprint
# define the job
def hello_job():
print('Hello Job! The time is: %s' % datetime.now())
def create_app(object_name):
app = Flask(__name__)
app.config.from_object(object_name)
db.init_app(app)
app.register_blueprint(main_blueprint)
# init BackgroundScheduler job
scheduler = BackgroundScheduler()
# in your case you could change seconds to hours
scheduler.add_job(hello_job, trigger='interval', seconds=3)
scheduler.start()
try:
# To keep the main thread alive
return app
except:
# shutdown if app occurs except
scheduler.shutdown()
Hope it helps :)
Ref :
https://github.com/agronholm/apscheduler/blob/master/examples/schedulers/background.py

Another alternative might be to use Flask-APScheduler which plays nicely with Flask, e.g.:
Loads scheduler configuration from Flask configuration,
Loads job definitions from Flask configuration
More information here:
https://pypi.python.org/pypi/Flask-APScheduler

You may use flask-crontab module, which is quite easy.
Step 1: pip install flask-crontab
Step 2:
from flask import Flask
from flask_crontab import Crontab
app = Flask(__name__)
crontab = Crontab(app)
Step 3:
#crontab.job(minute="0", hour="6", day="*", month="*", day_of_week="*")
def my_scheduled_job():
do_something()
Step 4: On cmd, hit
flask crontab add
Done. now simply run your flask application, and you can check your function will call at 6:00 every day.
You may take reference from Here (Official DOc).

A complete example using schedule and multiprocessing, with on and off control and parameter to run_job()
the return codes are simplified and interval is set to 10sec, change to every(2).hour.do()for 2hours. Schedule is quite impressive it does not drift and I've never seen it more than 100ms off when scheduling. Using multiprocessing instead of threading because it has a termination method.
#!/usr/bin/env python3
import schedule
import time
import datetime
import uuid
from flask import Flask, request
from multiprocessing import Process
app = Flask(__name__)
t = None
job_timer = None
def run_job(id):
""" sample job with parameter """
global job_timer
print("timer job id={}".format(id))
print("timer: {:.4f}sec".format(time.time() - job_timer))
job_timer = time.time()
def run_schedule():
""" infinite loop for schedule """
global job_timer
job_timer = time.time()
while 1:
schedule.run_pending()
time.sleep(1)
#app.route('/timer/<string:status>')
def mytimer(status, nsec=10):
global t, job_timer
if status=='on' and not t:
schedule.every(nsec).seconds.do(run_job, str(uuid.uuid4()))
t = Process(target=run_schedule)
t.start()
return "timer on with interval:{}sec\n".format(nsec)
elif status=='off' and t:
if t:
t.terminate()
t = None
schedule.clear()
return "timer off\n"
return "timer status not changed\n"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
You test this by just issuing:
$ curl http://127.0.0.1:5000/timer/on
timer on with interval:10sec
$ curl http://127.0.0.1:5000/timer/on
timer status not changed
$ curl http://127.0.0.1:5000/timer/off
timer off
$ curl http://127.0.0.1:5000/timer/off
timer status not changed
Every 10sec the timer is on it will issue a timer message to console:
127.0.0.1 - - [18/Sep/2018 21:20:14] "GET /timer/on HTTP/1.1" 200 -
timer job id=b64ed165-911f-4b47-beed-0d023ead0a33
timer: 10.0117sec
timer job id=b64ed165-911f-4b47-beed-0d023ead0a33
timer: 10.0102sec

You might want to use some queue mechanism with scheduler like RQ scheduler or something more heavy like Celery (most probably an overkill).

Run a function once on bottle.py startup

I have a bottle app that I eventually wan't to deploy on apache (just fyi in case that's important).
Now I need to run a function once after the bottle app is started. I can't just put it into a routed function because it has to run even if no user has accessed the site yet.
Any best pratice to do this ?
The function starts a APScheduler Instance and adds a jobstore to it.

Here's what I do.
def initialize():
//init whatever you need.
if __name__ == '__main__':
initialize()
#bottle.run(port='8080', yatta yatta)

Honestly your problem is simply a sync vs async issue. Use gevent to easily convert to microthreads, and then launch each separately. You can even add a delay either in your function or before with gevent.sleep if you want to wait for the web server to finish launching.
import gevent
from gevent import monkey, signal, spawn, joinall
monkey.patch_all()
from gevent.pywsgi import WSGIServer
from bottle import Bottle, get, post, request, response, template, redirect, hook, abort
import bottle
#get('/')
def mainindex():
return "Hello World"
def apScheduler():
print "AFTER SERVER START"
if __name__ == "__main__":
botapp = bottle.app()
server = WSGIServer(("0.0.0.0", 80), botapp)
threads = []
threads.append(spawn(server.serve_forever))
threads.append(spawn(apScheduler))
joinall(threads)

Create an APScheduler class.
Look at examples of object use and creation in this same site bacause it's too general to give an especific example to copy.
I don't know if this helps.
class Shed(object):
def __init__(self): # this to start it
# instruccions here
def Newshed(self, data):
# Call from bottle
# more methods ...
...
# init
aps = Shed() # this activates Shed.__init__()
...
# in the #router
x = aps.Newshed(data) # or whatever
Anyway I'm still learning this stuff and it's just an idea.

import threading
import bottle
def init_app():
def my_function_on_startup():
# some code here
pass
app = bottle.app()
t = threading.Thread(target=my_function_on_startup)
t.daemon = True
t.start()
return app
app = init_app()
#app.route("/")
def hello():
return "App is running"
if __name__ == "__main__":
bottle.run(app, host='localhost', port=8080, debug=True)