I'm making a website, and on startup, I want to launch another process that starts loading an embedding model because this takes a long time and will be needed by the user eventually. This is my code:
from flask import Flask, render_template
from flask_socketio import SocketIO, send
import bot
import sys
sys.path = sys.path + ['filepath']
from BigLearnPy import BigLearn
from multiprocessing import Process
app = Flask(__name__)
app.config['SECRET_KEY'] = 'password'
socketio = SocketIO(app)
def loadModel():
BigLearn.LoadEmbeddingEngine()
emb = BigLearn.EmbeddingEngine('filepath')
#app.route('/')
def index():
return render_template('index.html')
#socketio.on('message')
def handleMessage(msg):
send(msg, broadcast=True)
p1.join()
send('0' + bot.getResponse(msg, emb), broadcast=True)
send('2' + bot.getKB(msg, emb), broadcast=True)
if __name__ == '__main__':
emb = None
p1 = Process(target=loadModel)
p1.start()
socketio.run(app)
I start the process to load the model right before I start running the app (penultimate line). I join the process in the handleMessage function right before I need the value of emb. So that I can access emb outside of the loadModel function, I declared it right before creating the process. However, when I run the code, I get an error saying emb is a NoneType object. This seems like a scoping issue but no matter where I sayemb = None, I either get that emb is None or undefined when I try to use it. How can I load the model in a different process then access the model? Thanks.
You cannot load the model from a different process. That is not how multi-processing works.
At the fork, each process get its own copy of memory (conceptually; in practice there are tricks to prevent copying everything). Any change in variables after the fork will only be visible in the process that changed it, not in its parent.
If you want to share memory you need to use threads, not processes. But mutating memory that is shared between threads in a safe way is fairly complicated. In any case it might not help you that much because Python has a Global Interpreter Lock: only one Python thread can run at a time.
If you want to experiment with threads or processes I would recommend starting with simpler examples.
As for your problem, I would start by trying to optimize the loading code so it is faster. Without knowing what it does it is hard to make more specific suggestions.
Related
This question already has answers here:
Are global variables thread-safe in Flask? How do I share data between requests?
(4 answers)
Closed 18 days ago.
Context
I'm trying to write a python flask server that answers a simple request. The data to be returned in the response is queried from a backend service. Since this query may take some time to complete, I don't want to do it synchronously, I want to have it run periodically, in a background thread, where I can explicitly control the frequency. It should update some data structure that is shared with the flask view function, so the GET requests gets its answer from the shared data.
I'm including two sample codes below. In both codes, cnt is a global variable, starting at 0, and increased inside a separate thread. The index.html file displays the value of cnt:
<h1>cnt = {{ cnt }}</h1>
The issue I'm facing
When the view function is inside the same module, it works: every time I refresh the page with F5, the value of cnt has changed, I can see it increasing.
But when I put the view functions in a separate routes module (which I import at the end of my hello.py file), it no longer works: I can see in the server traces that cnt is being increased by the background thread, but when I refresh the page I always see
cnt = 1
It's as if I now have two different copies of the cnt variable, even though the variable has been imported into the routes module.
Note
I've found countless question on SO on this topic, but none that really addresses this specific concern. Also, I'm perfectly aware that in my examples below, there is no lock protecting the shared data (which is a simple cnt variable) and I'm not handling thread termination. This is being deliberately ignored for now, in order to keep the sample code minimal.
Here are the codes.
Single module, it works
Here's the main hello.py file, with everything inside the same module:
from flask import Flask, render_template
import threading as th
from time import sleep
cnt = 0
app = Flask(__name__)
# Run in background thread
def do_stuff():
global cnt
while True:
cnt += 1
print(f'do_stuff: cnt={cnt}')
sleep(1)
# Create the separate thread
t = th.Thread(target=do_stuff)
t.start()
#app.route("/")
def hello():
return render_template('index.html', cnt=cnt)
The variable is indeed shared between the background thread and the view function, as expected.
Separate modules, it no longer works
Here's the main hello.py module, without the view function :
from flask import Flask
import threading as th
from time import sleep
cnt = 0
app = Flask(__name__)
# Run in background thread
def do_stuff():
global cnt
while True:
cnt += 1
print(f'do_stuff: cnt={cnt}')
sleep(1)
# Create the separate thread
t = th.Thread(target=do_stuff)
t.start()
import routes
And here is the separate routes.py file (see import at the end of hello.py above):
# routes.py
from hello import app, cnt
from flask import render_template
#app.route("/")
def hello():
return render_template('index.html', cnt=cnt)
With this code, the web page always displays cnt = 1, as if the two modules had two distinct instances of the cntvariable.
I feel like I'm missing some basic insight into python modules, or threads, or their interaction. Any help will be appreciated, and my apologies for such a long question.
globals are not shared between modules
when you say
from moduleA import count you have imported a non mutable number
if something in another file changes count in module A it will not update anything else... its overwritten that count you imported with a new variable named count effectively
if count was a mutable object than any changes would indeed reflect back ... so if it was a list or dictionary or a class with fields etc..
instead you can import the module which is mutable
import moduleA
now when moduleA changes count it is changing the mutable module field and you can see it in another place by printing moduleA.count
We have a rather big project that is doing a lot of networking (API calls, Websocket messages) and that also has a lot of internal jobs running in intervals in threads. Our current architecture involves spawning a lot of threads and the app is not working very well when the system is under a big load, so we've decided to give asyncio a try.
I know that the best way would be to migrate the whole codebase to async code, but that is not realistic in the very near future because of the size of the codebase and the limited development resources. However, we would like to start migrating parts of our codebase to use asyncio event loop and hopefully, we will be able to convert the whole project at some point.
The problem we have encountered so far is that the whole codebase has sync code and in order to add non-blocking asyncio code inside, the code needs to be run in different thread since you can't really run async and sync code in the same thread.
In order to combine async and sync code, I came up with this approach of running the asyncio code in a separate thread that is created on app start. Other parts of the code add jobs to this loop simply by calling add_asyncio_task.
import threading
import asyncio
_tasks = []
def threaded_loop(loop):
asyncio.set_event_loop(loop)
global _tasks
while True:
if len(_tasks) > 0:
# create a copy of needed tasks
needed_tasks = _tasks.copy()
# flush current tasks so that next tasks can be easily added
_tasks = []
# run tasks
task_group = asyncio.gather(*needed_tasks)
loop.run_until_complete(task_group)
def add_asyncio_task(task):
_tasks.append(task)
def start_asyncio_loop():
loop = asyncio.get_event_loop()
t = threading.Thread(target=threaded_loop, args=(loop,))
t.start()
and somewhere in app.py:
start_asyncio_loop()
and anywhere else in the code:
add_asyncio_task(some_coroutine)
Since I am new to asyncio, I am wondering if this is a good approach in our situation or if this approach is considered an anti-pattern and has some problems that will hit us later down the road? Or maybe asyncio already has some solution for this and I'm just trying to invent the wheel here?
Thanks for your inputs!
The approach is fine in general. You have some issues though:
(1) Almost all asyncio objects are not thread safe
(2) Your code is not thread safe on its own. What if a task appears after needed_tasks = _tasks.copy() but before _tasks = []? You need a lock here. Btw making a copy is pointless. Simple needed_tasks = _tasks will do.
(3) Some asyncio constructs are thread safe. Use them:
import threading
import asyncio
# asyncio.get_event_loop() creates a new loop per thread. Keep
# a single reference to the main loop. You can even try
# _loop = asyncio.new_event_loop()
_loop = asyncio.get_event_loop()
def get_app_loop():
return _loop
def asyncio_thread():
loop = get_app_loop()
asyncio.set_event_loop(loop)
loop.run_forever()
def add_asyncio_task(task):
asyncio.run_coroutine_threadsafe(task, get_app_loop())
def start_asyncio_loop():
t = threading.Thread(target=asyncio_thread)
t.start()
I'm currently creating a web app using Python Flask and I've run into a road block and I'm not sure if I'm even thinking about it correctly.
So my website's homepage is just a simple landing page with text input that is required to perform the websites function. What I am trying to accomplish is for the web app to perform two things after the text is input. First, the server takes the username input and performs a function that doesn't return anything to the user but creates a bunch of data that is logged into an sqlite database, and used later on in the process. Then, the server returns the web page for a survey that has to be taken after the username is input. However, the function that the server performs can take upwards of 2 minutes depending on the user. The way I currently have it coded, the server performs the function, then once it has finished, it returns the web page, so the user is stuck at a loading screen for up to 2 minutes.
#app.route("/survey")
def main(raw_user):
raw_user = request.args.get("SteamID") <
games = createGameDict(user_obj) <----- the function
tag_lst = get_tags(games) <
return render_template("survey_page.html")
Since the survey doesn't depend on the user input, instead of having the user sitting at a loading screen, I would like them to be able to start the survey while the functions works in the background, is that possible, and how would I do that?
Update: I've had to solve this problem a number of times in Flask, so I wrote a small Flask extension called Flask-Executor to do it for me. It's a wrapper for concurrent.futures that provides a few handy features, and is my preferred way of handling background tasks that don't require distribution in Flask.
For more complex background tasks, something like celery is your best bet. For simpler use cases however, what you want is the threading module.
Consider the following example:
from flask import Flask
from time import sleep
app = Flask(__name__)
def slow_function(some_object):
sleep(5)
print(some_object)
#app.route('/')
def index():
some_object = 'This is a test'
slow_function(some_object)
return 'hello'
if __name__ == '__main__':
app.run()
Here, we create a function, slow_function() that sleeps for five seconds before returning. When we call it in our route function it blocks the page load. Run the example and hit http://127.0.0.1:5000 in your browser, and you'll see the page wait five seconds before loading, after which the test message is printed in your terminal.
What we want to do is to put slow_function() on a different thread. With just a couple of additional lines of code, we can use the threading module to separate out the execution of this function onto a different thread:
from flask import Flask
from time import sleep
from threading import Thread
app = Flask(__name__)
def slow_function(some_object):
sleep(5)
print(some_object)
#app.route('/')
def index():
some_object = 'This is a test'
thr = Thread(target=slow_function, args=[some_object])
thr.start()
return 'hello'
if __name__ == '__main__':
app.run()
What we're doing here is simple. We're creating a new instance of Thread and passing it two things: the target, which is the function we want to run, and args, the argument(s) to be passed to the target function. Notice that there are no parentheses on slow_function, because we're not running it - functions are objects, so we're passing the function itself to Thread. As for args, this always expects a list. Even if you only have one argument, wrap it in a list so args gets what it's expecting.
With our thread ready to go, thr.start() executes it. Run this example in your browser, and you'll notice that the index route now loads instantly. But wait another five seconds and sure enough, the test message will print in your terminal.
Now, we could stop here - but in my opinion at least, it's a bit messy to actually have this threading code inside the route itself. What if you need to call this function in another route, or a different context? Better to separate it out into its own function. You could make threading behaviour a part of slow function itself, or you could make a "wrapper" function - which approach you take depends a lot on what you're doing and what your needs are.
Let's create a wrapper function, and see what it looks like:
from flask import Flask
from time import sleep
from threading import Thread
app = Flask(__name__)
def slow_function(some_object):
sleep(5)
print(some_object)
def async_slow_function(some_object):
thr = Thread(target=slow_function, args=[some_object])
thr.start()
return thr
#app.route('/')
def index():
some_object = 'This is a test'
async_slow_function(some_object)
return 'hello'
if __name__ == '__main__':
app.run()
The async_slow_function() function is doing pretty much exactly what we were doing before - it's just a bit neater now. You can call it in any route without having to rewrite your threading logic all over again. You'll notice that this function actually returns the thread - we don't need that for this example, but there are other things you might want to do with that thread later, so returning it makes the thread object available if you ever need it.
I want to implement slack slash command that has to process fucntion pipeline which takes roughly 30 seconds to process. Now since Slack slash commands only allows 3 seconds to respond, how to go about implementing this. I referred this but don't how to implement it.
Please hold up with me. I am doing this first time.
This is what I have tried. I know how to respond with ok status within 3 seconds but I don't understand how to again call pipeline
import requests
import json
from bottle import route, run, request
from S3_download import s3_download
from index import main_func
#route('/action')
def action():
pipeline()
return "ok"
def pipeline():
s3_download()
p = main_func()
print (p)
if __name__ == "__main__":
run(host='0.0.0.0', port=8082, debug=True)
I came across this article. Is using AWS lambda the only solution?
Can't we do this completely in python?
Something like this:
from boto import sqs
#route('/action', method='POST')
def action():
#retrieving all the required request example
params = request.forms.get('response_url')
sqs_queue = get_sqs_connection(queue_name)
message_object = sqs.message.Message()
message_object.set_body(params)
mail_queue.write(message_object)
return "request under process"
and you can have another process which processes the queue and call long running function:
sqs_queue = get_sqs_connection(queue_name)
for sqs_msg in sqs_queue.get_messages(10, wait_time_seconds=5):
processed_msg = json.loads(sqs_msg.get_body())
response = pipeline(processed_msg)
if response:
sqs_queue.delete_message(sqs_msg)
you can run this 2nd process maybe in a diff standalone python file, as a daemon process or cron.
I`v used sqs Amazon Queue here, but there are different options available.
You have an option or two for doing this in a single process, but it's fraught with peril. If you spin up a new Thread to handle the long process, you might end up deploying or crashing in the middle and losing it.
If durability is important to you, look into background-task workers like SQS, Lambda, or even a Celery task queue backed with Redis. A separate task has some interesting failure modes, and these tools will help you deal with them better than just spawning a thread.
Is there a good way to share a multiprocessing Lock between gunicorn workers? I am trying to write a json API with Flask. Some of the API calls will interact a python class that manages a running process (like ffmpeg for video conversion). When I scale up my number of web workers to more than 1, how can I ensure that only 1 worker is interacting with the class at the same time?
My initial thought was to use multiprocessing.Lock so the start() function can be atomic. I don't think I've figured out the right place to create a Lock so that one is shared across all the workers:
# runserver.py
from flask import Flask
from werkzeug.contrib.fixers import ProxyFix
import dummy
app = Flask(__name__)
#app.route('/')
def hello():
dummy.start()
return "ffmpeg started"
app.wsgi_app = ProxyFix(app.wsgi_app)
if __name__ == '__main__':
app.run()
Here is my dummy operation:
# dummy.py
from multiprocessing import Lock
import time
lock = Lock()
def start():
lock.acquire()
# TODO do work
for i in range(0,10):
print "did work %s" % i
time.sleep(1)
lock.release()
When I refresh the page a few times, I see the output from each call woven together.
Am I barking up the wrong tree here? Is there an easier way to make sure that only copy of the processing class (here just the dummy start() method) gets run at the same time? I think I might need something like celery to run tasks (and just use only 1 worker) but that seems a bit overkill for my small project.
I tried something, and it seems to work. I put preload_app = True in my gunicorn.conf and now the lock seems to be shared. I am still looking into exactly what's happening here but for now this is good enough, YMMV.
Follow peterw's answer, the workers can share the lock resource.
But, It is better to use try-finally block to ensure the lock will always be released.
# dummy.py
from multiprocessing import Lock
import time
lock = Lock()
def start():
lock.acquire()
try:
# TODO do work
for i in range(0,10):
print "did work %s" % i
time.sleep(1)
finally:
lock.release()
Late addition:
If for some reason, using preload_app is not feasible, then you need to use a named lock. This ensures that all processes are using the same lock object.
Using mp.Lock() will create a different object for each process, negating any value.
I saw this package but did not use it yet. It supplies a named lock in the scope of one machine; that means that all processes within the same machine will use the same lock, but outside the boundaries of one machine this solution is not appropriate.