I've been trying threads recently in my webapp, and I've come across an issue that I cannot seem to solve.
The issue is that I have an index page, and every time the user enters that page, a new thread is being started which checks for changes in my database in a while loop, although I only want one thread to be on at that moment. Is there a way to "kill" a thread when the index page is accessed the second time and then start a new one?
I did try to use is_alive() to check if a thread is already running, but had no success since all the time they're different.
Code example below:
#app.route("/")
def index():
#copy_current_request_context
def check_for_updates():
while True:
...... # Query for information
if something_changed:
socketio.emit('new_notifications', {'data': new_data})
if index_opened_again:
break
sleep(5)
index_thread = threading.Thread(target=check_for_updates, daemon=True)
index_thread.start()
return render_template("index.html")
I user the below code to kill threads when I am existing a server ,
My suggestion is to kill all active threadss before opening a new one
for id, thread in threading._active.items():
ctypes.pythonapi.PyThreadState_SetAsyncExc(thread_id, ctypes.py_object(SystemExit))
In my example I use a global variable in combination with a lock. It's certainly not optimal, but you can check if a thread is already running.
If you're using flask-socketio anyway, I recommend you take a look at start_background_task. The functionality is compatible with that of a standard thread.
from threading import Lock
from flask import Flask, render_template
from flask_socketio import SocketIO
app = Flask(__name__)
app.secret_key = 'your secret here'
sio = SocketIO(app)
thread = None
thread_lock = Lock()
def background_task():
while True:
# ...
sleep(5)
#app.route('/')
def index():
global thread
with thread_lock:
if thread is None:
thread = sio.start_background_task(background_task)
return render_template('index.html')
if __name__ == '__main__':
sio.run(app)
The following scenario:
I have a Raspberry Pi running as a server. Currently I am using a Python script with Flask and I can also access the Raspberry Pi from my PC. (The flask server runs an react app.)
But the function should be extended. It should look like the following:
2nd Python script is running all the time. This Python script fetches data from an external API every second and processes it. If certain conditions are met, the data should be processed and then the data should be communicated to the Python Flask server. And the Flask server then forwards the data to the website running on the computer.
How or which method is best to program this "interprocess communication". Are there any libraries? I tried Celery, but then it throws up my second Python script whenever I want to access the external API, so I don't know if this is the right choice.
What else would be the best approach? Threading? Direct interprocess communication?
If important, this is how my server application looks so far:
from gevent import monkey
from flask import Flask, render_template
from flask_socketio import SocketIO
monkey.patch_all()
app = Flask(__name__, template_folder='./build', static_folder='./build/static')
socket_io = SocketIO(app)
#app.route('/')
def main():
return render_template('index.html')
#socket_io.on('fromFrontend')
def handleInput(input):
print('Input from Frontend: ' + input)
send_time()
#socket_io.on('time')
def send_time():
socket_io.emit('time', {'returnTime': "some time"})
if __name__ == '__main__':
socket_io.run(app, host='0.0.0.0', port=5000, debug=True)
Well i found a solution for my specific problem i implemented it with a thread as follows:
import gevent.monkey
gevent.monkey.patch_all()
from flask import Flask, render_template
from flask_socketio import SocketIO
import time
import requests
from threading import Thread
app = Flask(__name__, template_folder='./build', static_folder='./build/static')
socket_io = SocketIO(app)
#app.route('/')
def main():
thread = Thread(target=backgroundTask)
thread.daemon = True
thread.start()
return render_template('index.html')
#socket_io.on('fromFrontend')
def handleInput(input):
print('Input from Frontend: ' + input)
#socket_io.on('time')
def send_time():
socket_io.emit('time', {'returnTime': 'hi frontend'})
def backgroundTask():
# do something here
# access socket to push some data
socket_io.emit('time', {'returnTime': "some time"})
if __name__ == '__main__':
socket_io.run(app, host='0.0.0.0', port=5000, debug=True)
I have a RPC Server using zerorpc in Python, written this way
import zerorpc
from service import Service
print('RPC server - loading')
def main():
print('RPC server - main')
s = zerorpc.Server(Service())
s.bind("tcp://*:4242")
s.run()
if __name__ == "__main__" : main()
It works fine when I create a client
import zerorpc, sys
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
name = sys.argv[1] if len(sys.argv) > 1 else "dude"
print(client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm'))
and runs it. The print() outputs what this 'videos' function returns.
But when I try to use it this same code inside route from a Flask app, I receive the following error:
File "src/gevent/__greenlet_primitives.pxd", line 35, in
gevent.__greenlet_primitives._greenlet_switch
gevent.exceptions.LoopExit: This operation would block forever Hub:
The flask method/excerpt
import zerorpc, sys
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
#app.route('/videos', methods=['POST'])
def videos():
global client_rpc
client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm')
I can't find out what might be happening. I'm quite new to Python and I understand that this may have some relation with Flask and how it handles the thread, but I can't figure out how to solve it.
zerorpc depends on gevent, which provides async IO with cooperative coroutines. This means your flask application must use gevent for all IO operations.
In your specific case, you are likely starting your application with a standard blocking IO WSGI server.
Here is a snippet using the WSGI server from gevent:
import zerorpc
from gevent.pywsgi import WSGIServer
app = Flask(__name__)
client_rpc = zerorpc.Client()
client_rpc.connect("tcp://127.0.0.1:4242")
#app.route('/videos', methods=['POST'])
def videos():
global client_rpc
client_rpc.videos('138cd9e5-3c4c-488a-9b6f-49907b55a040.webm')
# ...
if __name__ == "__main__":
http = WSGIServer(('', 5000), app)
http.serve_forever()
Excerpt from https://sdiehl.github.io/gevent-tutorial/#chat-server
I have been trying to follow the tutorials to get flask apps to run on Heroku, like this one: https://dev.to/emcain/how-to-set-up-a-twitter-bot-with-python-and-heroku-1n39.
They all tell you to put this in your code in a file server.py:
from flask import Flask
app = Flask(__name__)
app.run(host='0.0.0.0')
And then run the app via the following command:
python3 server.py
But the tutorials don't explain how to connect the actual function you want to run using the app. In my case, I have a File testbot.py that has the function test(arg1) that contains the code I want to execute:
def test(arg1):
while(1):
#do stuff with arg1 on twitter
I want to do something like this:
from flask import Flask
from testbot import test
from threading import Thread
app = Flask(__name__)
app.addfunction(test(arg1='hardcodedparameter'))
app.run(host='0.0.0.0')
So that when the app runs my test() function executes with the argument. Right now my server is starting, but nothing is happening.
Am I thinking about this correctly?
*Edit: I got it working with the solution, so my server.py now looks like this:
from flask import Flask
from testbot import test
def main_process():
test("hardcodeparam")
app = Flask(__name__)
Thread(target=main_process).start()
app.run(debug=True,host='0.0.0.0')
And now test runs as expected.
Before app.run, register the function with a path, e.g.
#app.route('/')
def test(): # no argument
... do one iteration
return 'ok'
Then visiting the URL will trigger the function. Sites such as https://cron-job.org/ can automate that visiting on a regular basis for free, as suggested here.
If the regular intervals aren't good enough, then you could try:
#app.route('/')
def index(): # no argument
return 'ok'
def test():
while True:
# do stuff
from threading import Thread
Thread(target=test).start()
app.run(...)
You will probably still need to have a job regularly visiting the URL so that Heroku sees that the server is alive and in use.
I have a Flask web hosting with no access to cron command.
How can I execute some Python function every hour?
You can use BackgroundScheduler() from APScheduler package (v3.5.3):
import time
import atexit
from apscheduler.schedulers.background import BackgroundScheduler
def print_date_time():
print(time.strftime("%A, %d. %B %Y %I:%M:%S %p"))
scheduler = BackgroundScheduler()
scheduler.add_job(func=print_date_time, trigger="interval", seconds=60)
scheduler.start()
# Shut down the scheduler when exiting the app
atexit.register(lambda: scheduler.shutdown())
Note that two of these schedulers will be launched when Flask is in debug mode. For more information, check out this question.
I'm a little bit new with the concept of application schedulers, but what I found here for APScheduler v3.3.1 , it's something a little bit different. I believe that for the newest versions, the package structure, class names, etc., have changed, so I'm putting here a fresh solution which I made recently, integrated with a basic Flask application:
#!/usr/bin/python3
""" Demonstrating Flask, using APScheduler. """
from apscheduler.schedulers.background import BackgroundScheduler
from flask import Flask
def sensor():
""" Function for test purposes. """
print("Scheduler is alive!")
sched = BackgroundScheduler(daemon=True)
sched.add_job(sensor,'interval',minutes=60)
sched.start()
app = Flask(__name__)
#app.route("/home")
def home():
""" Function for test purposes. """
return "Welcome Home :) !"
if __name__ == "__main__":
app.run()
I'm also leaving this Gist here, if anyone have interest on updates for this example.
Here are some references, for future readings:
APScheduler Doc: https://apscheduler.readthedocs.io/en/latest/
daemon=True: https://docs.python.org/3.4/library/threading.html#thread-objects
You could make use of APScheduler in your Flask application and run your jobs via its interface:
import atexit
# v2.x version - see https://stackoverflow.com/a/38501429/135978
# for the 3.x version
from apscheduler.scheduler import Scheduler
from flask import Flask
app = Flask(__name__)
cron = Scheduler(daemon=True)
# Explicitly kick off the background thread
cron.start()
#cron.interval_schedule(hours=1)
def job_function():
# Do your work here
# Shutdown your cron thread if the web process is stopped
atexit.register(lambda: cron.shutdown(wait=False))
if __name__ == '__main__':
app.run()
I've tried using flask instead of a simple apscheduler what you need to install is
pip3 install flask_apscheduler
Below is the sample of my code:
from flask import Flask
from flask_apscheduler import APScheduler
app = Flask(__name__)
scheduler = APScheduler()
def scheduleTask():
print("This test runs every 3 seconds")
if __name__ == '__main__':
scheduler.add_job(id = 'Scheduled Task', func=scheduleTask, trigger="interval", seconds=3)
scheduler.start()
app.run(host="0.0.0.0")
For a simple solution, you could add a route such as
#app.route("/cron/do_the_thing", methods=['POST'])
def do_the_thing():
logging.info("Did the thing")
return "OK", 200
Then add a unix cron job that POSTs to this endpoint periodically. For example to run it once a minute, in terminal type crontab -e and add this line:
* * * * * /opt/local/bin/curl -X POST https://YOUR_APP/cron/do_the_thing
(Note that the path to curl has to be complete, as when the job runs it won't have your PATH. You can find out the full path to curl on your system by which curl)
I like this in that it's easy to test the job manually, it has no extra dependencies and as there isn't anything special going on it is easy to understand.
Security
If you'd like to password protect your cron job, you can pip install Flask-BasicAuth, and then add the credentials to your app configuration:
app = Flask(__name__)
app.config['BASIC_AUTH_REALM'] = 'realm'
app.config['BASIC_AUTH_USERNAME'] = 'falken'
app.config['BASIC_AUTH_PASSWORD'] = 'joshua'
To password protect the job endpoint:
from flask_basicauth import BasicAuth
basic_auth = BasicAuth(app)
#app.route("/cron/do_the_thing", methods=['POST'])
#basic_auth.required
def do_the_thing():
logging.info("Did the thing a bit more securely")
return "OK", 200
Then to call it from your cron job:
* * * * * /opt/local/bin/curl -X POST https://falken:joshua#YOUR_APP/cron/do_the_thing
You could try using APScheduler's BackgroundScheduler to integrate interval job into your Flask app. Below is the example that uses blueprint and app factory (init.py) :
from datetime import datetime
# import BackgroundScheduler
from apscheduler.schedulers.background import BackgroundScheduler
from flask import Flask
from webapp.models.main import db
from webapp.controllers.main import main_blueprint
# define the job
def hello_job():
print('Hello Job! The time is: %s' % datetime.now())
def create_app(object_name):
app = Flask(__name__)
app.config.from_object(object_name)
db.init_app(app)
app.register_blueprint(main_blueprint)
# init BackgroundScheduler job
scheduler = BackgroundScheduler()
# in your case you could change seconds to hours
scheduler.add_job(hello_job, trigger='interval', seconds=3)
scheduler.start()
try:
# To keep the main thread alive
return app
except:
# shutdown if app occurs except
scheduler.shutdown()
Hope it helps :)
Ref :
https://github.com/agronholm/apscheduler/blob/master/examples/schedulers/background.py
Another alternative might be to use Flask-APScheduler which plays nicely with Flask, e.g.:
Loads scheduler configuration from Flask configuration,
Loads job definitions from Flask configuration
More information here:
https://pypi.python.org/pypi/Flask-APScheduler
You may use flask-crontab module, which is quite easy.
Step 1: pip install flask-crontab
Step 2:
from flask import Flask
from flask_crontab import Crontab
app = Flask(__name__)
crontab = Crontab(app)
Step 3:
#crontab.job(minute="0", hour="6", day="*", month="*", day_of_week="*")
def my_scheduled_job():
do_something()
Step 4: On cmd, hit
flask crontab add
Done. now simply run your flask application, and you can check your function will call at 6:00 every day.
You may take reference from Here (Official DOc).
A complete example using schedule and multiprocessing, with on and off control and parameter to run_job()
the return codes are simplified and interval is set to 10sec, change to every(2).hour.do()for 2hours. Schedule is quite impressive it does not drift and I've never seen it more than 100ms off when scheduling. Using multiprocessing instead of threading because it has a termination method.
#!/usr/bin/env python3
import schedule
import time
import datetime
import uuid
from flask import Flask, request
from multiprocessing import Process
app = Flask(__name__)
t = None
job_timer = None
def run_job(id):
""" sample job with parameter """
global job_timer
print("timer job id={}".format(id))
print("timer: {:.4f}sec".format(time.time() - job_timer))
job_timer = time.time()
def run_schedule():
""" infinite loop for schedule """
global job_timer
job_timer = time.time()
while 1:
schedule.run_pending()
time.sleep(1)
#app.route('/timer/<string:status>')
def mytimer(status, nsec=10):
global t, job_timer
if status=='on' and not t:
schedule.every(nsec).seconds.do(run_job, str(uuid.uuid4()))
t = Process(target=run_schedule)
t.start()
return "timer on with interval:{}sec\n".format(nsec)
elif status=='off' and t:
if t:
t.terminate()
t = None
schedule.clear()
return "timer off\n"
return "timer status not changed\n"
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5000)
You test this by just issuing:
$ curl http://127.0.0.1:5000/timer/on
timer on with interval:10sec
$ curl http://127.0.0.1:5000/timer/on
timer status not changed
$ curl http://127.0.0.1:5000/timer/off
timer off
$ curl http://127.0.0.1:5000/timer/off
timer status not changed
Every 10sec the timer is on it will issue a timer message to console:
127.0.0.1 - - [18/Sep/2018 21:20:14] "GET /timer/on HTTP/1.1" 200 -
timer job id=b64ed165-911f-4b47-beed-0d023ead0a33
timer: 10.0117sec
timer job id=b64ed165-911f-4b47-beed-0d023ead0a33
timer: 10.0102sec
You might want to use some queue mechanism with scheduler like RQ scheduler or something more heavy like Celery (most probably an overkill).