I made an API for my AI model but I would like to not have any down time when I update the model. I search a way to load in background and once it's loaded I switch the old model with the new. I tried passing values between sub process but doesn't work well. Do you have any idea how can I do that ?
You can place the serialized model in a raw storage, like an S3 bucket if you're on AWS. In S3's case, you can use bucket versioning which might prove helpful. Then setup some sort of trigger. You can definitely get creative here, and I've thought about this a lot. In practice, the best options I've tried are:
Set up an endpoint that when called will go open the new model at whatever location you store it at. Set up a webhook on the storage/S3 bucket that will send a quick automated call to the given endpoint and auto-load that new item
Same thing as #1, but instead you just manually load it. In both cases you'll really want some security on that endpoint or anyone that finds your site can just absolutely abuse your stack.
Set a timer at startup that calls a given function nightly, internally running within the application itself. The function is invoked and then goes and reloads.
Could be other ideas I'm not smart enough (yet!) to use, just trying to start some dialogue.
Found a way to do it with async and multiprocessing
import asyncio
import random
from uvicorn import Server, Config
from fastapi import FastAPI
import time
from multiprocessing import Process, Manager
app = FastAPI()
value = {"latest": 1, "b": 2}
#app.get("/")
async def root():
global value
return {"message": value}
def background_loading(d):
time.sleep(2)
d["test"] = 3
async def update():
while True:
global value
manager = Manager()
d = manager.dict()
p1 = Process(target=background_loading, args=(d,))
p1.daemon = True
p1.start()
while p1.is_alive():
await asyncio.sleep(5)
print(f'Update to value to {d}')
value = d
if __name__ == "__main__":
loop = asyncio.new_event_loop()
config = Config(app=app, loop=loop)
server = Server(config)
loop.create_task(update())
loop.run_until_complete(server.serve())
Related
Assuming I have this listener defined in my Sanic app:
#app.before_server_start
async def db_setup(*args):
# ... set up the DB as I wish for the app
If I want to unit test this function (with pytest) and I import it in a unit test file with from my.app import db_setup, it seems the test actually starts serving the app, as pytest outputs:
[INFO] Goin' Fast # http://0.0.0.0:8000
[INFO] Starting worker [485]
Now, I know that I can remove the effects of the decorator by doing db_setup = db_setup.__wrapped__, but in order to do this I actually need to import db_setup, which is where the Sanic server fires up.
Is there a way of removing the effects of the decorator at import time?
LE: I've tried patching the Sanic app as follows:
async def test_stuff(mocker):
mocker.patch('myapp.app.app') # last '.app' being `app = Sanic('MyApp')`
imp = importlib.import_module('myapp.app')
db_setup = getattr(imp, 'db_setup')
await db_setup()
but now I get a RuntimeError: Cannot run the event loop while another loop is running for the mocker.patch('myapp.app.app') line.
I am going to make a few assumptions here, so I may need to amend this answer if there are some clarifications.
Before starting, it should be noted that the decorator itself will not start your web server. That will run in one of two scenarios:
You are running app.run() somewhere in the global scope
You are using the Sanic TestClient, which specifically operates by running your application's web server
Now, from what I can understand, you are trying to run db_setup in a test manually by calling it as a function, but you do not want it to attach as a listener to the application in your tests.
You can get access to all of your application instance's listeners in the app.listeners property. Therefore one solution would be something like this:
# conftest.py
import pytest
from some.place import app as myapp
#pytest.fixture
def app():
myapp.listeners = {}
return myapp
Like I said earlier, this will just empty out your listeners. It does not actually impact your application starting, so I am not sure it has the utility that you are looking for.
You should be able to have something like this:
from unittest.mock import Mock
import pytest
from sanic import Request, Sanic, json
app = Sanic(__name__)
#app.get("/")
async def handler(request: Request):
return json({"foo": "bar"})
#app.before_server_start
async def db_setup(app, _):
app.ctx.db = 999
#pytest.mark.asyncio
async def test_sample():
await db_setup(app, Mock())
assert app.ctx.db == 999
For the sake of ease, it is all in the same scope, but even if the test functions, the application instance, and the listener are spread across different modules, the end result is the same: You can run db_setup as any other function and it does not matter if it is registered as a listener or not.
TLDR; I am trying to run a client that fetches data from a web-sockets URI, and then use Flask to serve the fetched data from the sockets.
My intended workflow looks like: Read data from web-sockets (async) -> write data to a Python dict (continuously) -> read data from Python dict using GET requests on Flask (continuously)
The problem I am facing is that I am using a Python dictionary for storage, but when I read that dictionary from Flask, it does not show me updated values.
I have created an illustration with code to my problem
Client:
import asyncio
import websockets
import json
SOME_URI = "ws://localhost:8080/foo"
connections = set()
connections.add(SOME_URI)
class Storage:
storage = dict() # local storage dict for simplicity
#staticmethod
def store(data): # here I store the value
a, b, c = data
Storage.storage[a] = c
#staticmethod
def show(): # Here I just show the value to be used in the GET
return Storage.storage
async def consumer_handler(uri):
async with websockets.connect(uri) as websocket:
async for message in websocket:
await consumer(message)
async def consumer(message):
line = json.loads(message)
Storage.store(line) # adds message to dict
async def main():
await asyncio.wait([consumer_handler(uri) for uri in connections])
if __name__ == "__main__":
asyncio.run(main())
App:
from flask import Flask
from client import Storage
app = Flask(__name__)
app.debug = True
#app.route('/bar', methods=['GET'])
def get_instruments():
res = Storage.show() # I expected here to see updated value for the dict as it fills up from websockets
return res, 200
if __name__ == "__main__":
app.run()
Whenever I try to make a GET request to the Flask page, I get the value of an Empty dict (does not reflect the changes I add from the web socket). I was expecting to get an updated value of the dict with every GET request.
The problem is that you have two separate Python processes involved here -- one is the async script for ingesting data and the other is running flask. As these are separate processes, the dict updates that you're doing on the first process is not visible to the second process.
You need some form of IPC (Inter-Process Communication) mechanism here for data portability and/or persistence. One obvious and easier option to try is to use a named FIFO or a plain file. Depending on the level of complexity you want to handle for this, using Redis (or alike) is also another option.
I'm trying to delete a lot of files in s3. I am planning on using a multiprocessing.Pool for doing all these deletes, but I'm not sure how to keep the s3.client alive between jobs. I'm wanting to do something like
import boto3
import multiprocessing as mp
def work(key):
s3_client = boto3.client('s3')
s3_client.delete_object(Bucket='bucket', Key=key)
with mp.Pool() as pool:
pool.map(work, lazy_iterator_of_billion_keys)
But the problem with this is that a significant amount of time is spent doing the s3_client = boto3.client('s3') at the start of each job. The documentation says to make a new resource instance for each process so I need a way to make a s3 client for each process.
Is there any way to make a persistent s3 client for each process in the pool or cache the clients?
Also, I am planning on optimizing the deletes by sending batches of keys and using s3_client.delete_objects, but showed s3_client.delete_object in my example for simplicity.
Check this snippet from the RealPython concurrency tutorial. They create a single request Session for each process since you cannot share resources because each pool has its own memory space. Instead, they create a global session object to initialize the multiprocessing pool, otherwise, each time the function is called it would instantiate a Session object which is an expensive operation.
So, following that logic, you could instantiate the boto3 client that way and you would only create one client per process.
import requests
import multiprocessing
import time
session = None
def set_global_session():
global session
if not session:
session = requests.Session()
def download_site(url):
with session.get(url) as response:
name = multiprocessing.current_process().name
print(f"{name}:Read {len(response.content)} from {url}")
def download_all_sites(sites):
with multiprocessing.Pool(initializer=set_global_session) as pool:
pool.map(download_site, sites)
if __name__ == "__main__":
sites = [
"https://www.jython.org",
"http://olympus.realpython.org/dice",
] * 80
start_time = time.time()
download_all_sites(sites)
duration = time.time() - start_time
print(f"Downloaded {len(sites)} in {duration} seconds")
I ended up solving this using functools.lru_cache and a helper function for getting the s3 client. An LRU cache will stay consistent in a process, so it will preserve the connection. The helper function looks like
from functools import lru_cache
#lru_cache()
def s3_client():
return boto3.client('s3')
and then that is called in my work function like
def work(key):
s3_client = s3_client()
s3_client.delete_object(Bucket='bucket', Key=key)
I was able to test this and benchmark it in the following way:
import os
from time import time
def benchmark(key):
t1 = time()
s3 = get_s3()
print(f'[{os.getpid()}] [{s3.head_object(Bucket='bucket', Key=key)}] :: Total time: {time() - t1} s')
with mp.Pool() as p:
p.map(benchmark, big_list_of_keys)
And this result showed that the first function call for each pid would take about 0.5 seconds and then subsequent calls for the same pid would take about 2e-6 seconds. This was proof enough to me that the client connection was being cached and working as I expected.
Interestingly, if I don't have #lru_cache() on s3_client() then subsequent calls would take about 0.005 seconds, so there must be some internal caching that happens automatically with boto3 that I wasn't aware of.
And for testing purposes, I benchmarked Milton's answer in the following way
s3 = None
def set_global_session():
global s3
if not s3:
s3 = boto3.client('s3')
with mp.Pool(initializer=set_global_session) as p:
p.map(benchmark, big_list_of_keys)
And this also had averaging 3e-6 seconds per job, so pretty much the same as using functools.lru_cache on a helper function.
I'm currently building an application that allows users to collaborate together and create things, as I require a sort of discord like group chatfeed. I need to be able to subscribe logged in users to a project for notifications.
I have a method open_project that retrieves details from a project that has been selected by the user, which I use to subscribe him to any updates for that project.
So I can think of 2 ways of doing this. I have created a instance variable in my connect function, like this:
def connect(self):
print("connected to projectconsumer...")
self.accept()
self.projectSessions = {}
And here is the open_project method:
def open_project(self, message):
p = Project.objects.values("projectname").get(id=message)
if len(self.projectSessions) == 0:
self.projectSessions[message] = []
pass
self.projectSessions[message] = self.projectSessions[message].append(self)
print(self.projectSessions[message])
self.userjoinedMessage(self.projectSessions[message])
message = {}
message["command"] = "STC-openproject"
message["message"] = p
self.send_message(json.dumps(message))
Then when the user opens a project, he is added to the projectSessions list, this however doesn't work (I think) whenever a new user connects to the websocket, he gets his own projectconsumer.
The second way I thought of doing this is to create a managing class that only has 1 instance and keeps track of all the users connected to a project. I have not tried this yet as I would like some feedback on if I'm even swinging in the right ball park. Any and all feedback is appreciated.
EDIT 1:
I forgot to add the userjoinedMessage method to the question, this method is simply there to mimic future mechanics and for feedback to see if my solution actually works, but here it is:
def userjoinedMessage(self, pointer):
message = {}
message["command"] = "STC-userjoinedtest"
message["message"] = ""
pointer.send_message(json.dumps(message))
note that i attempt to reference the instance of the consumer.
I will also attempt to implement a consumer manager that has the role of keeping track of what consumers are browsing what projects and sending updates to the relevant channels.
From the question, the issue is how to save projectSessions and have it accessible across multiple instances of the consumer. Instead of trying to save it in memory, you can save in a database. It is a dictionary with project as key. You can make it a table with ForeignKey to the Project model.
In that way, it is persisted and there would be no issue retrieving it even across multiple channels server instances if you ever decide to scale your channels across multiple servers.
Also, if you feel that a traditional database will slow down the retrieval of the sessions, then you can use faster storage systems like redis
Right, this is probably a horrible way of doing things and i should be taken out back and shot for doing it but i have a fix for my problem. I have made a ProjectManager class that handles subscriptions and updates to the users of a project:
import json
class ProjectManager():
def __init__(self):
if(hasattr(self, 'projectSessions')):
pass
else:
self.projectSessions = {}
def subscribe(self, projectid, consumer):
print(projectid not in self.projectSessions)
if(projectid not in self.projectSessions):
self.projectSessions[projectid] = []
self.projectSessions[projectid].append(consumer)
self.update(projectid)
def unsubscribe(self, projectid, consumer):
pass
def update(self, projectid):
if projectid in self.projectSessions:
print(self.projectSessions[projectid])
for consumer in self.projectSessions[projectid]:
message = {}
message["command"] = "STC-userjoinedtest"
message["message"] = ""
consumer.send_message(json.dumps(message))
pass
in my apps.py file i initialize the above ProjectManager class and assign it to a variable.
from django.apps import AppConfig
from .manager import ProjectManager
class ProjectConfig(AppConfig):
name = 'project'
manager = ProjectManager()
Which i then use in my consumers.py file. I import the manager from the projectconfig class and assign it to a instance variable inside the created consumer whenever its connected:
def connect(self):
print("connected to projectconsumer...")
self.accept()
self.manager = ProjectConfig.manager
and whenever i call open_project i subscribe to that project with the given project id recieved from the front-end:
def open_project(self, message):
p = Project.objects.values("projectname").get(id=message)
self.manager.subscribe(message, self)
message = {}
message["command"] = "STC-openproject"
message["message"] = p
self.send_message(json.dumps(message))
as i said i in no way claim that this is the correct way of doing it and i am also aware that channel_layers supposedly does this for you in a neat way. i however don't really have the time to get into channel_layers and will therefore be using this.
I am still open to suggestions ofcourse and am always happy to learn more.
How can I run background tasks on App Engine?
You may use the Task Queue Python API.
GAE is very useful tool to build scalable web applications. Few of the limitations pointed out by many are no support for background tasks, lack of periodic tasks and strict limit on how much time each HTTP request takes, if a request exceeds that time limit the operation is terminated, which makes running time consuming tasks impossible.
How to run background task ?
In GAE the code is executed only when there is a HTTP request. There is a strict time limit (i think 10secs) on how long the code can take. So if there are no requests then code is not executed. One of the suggested work around was use an external box to send requests continuously, so kind of creating a background task. But for this we need an external box and now we dependent on one more element. The other alternative was sending 302 redirect response so that client re-sends the request, this also makes us dependent on external element which is client. What if that external box is GAE itself ? Everyone who has used functional language which does not support looping construct in the language is aware of the alternative ie recursion is the replacement to loop. So what if we complete part of the computation and do a HTTP GET on the same url with very short time out say 1 second ? This creates a loop(recursion) on php code running on apache.
<?php
$i = 0;
if(isset($_REQUEST["i"])){
$i= $_REQUEST["i"];
sleep(1);
}
$ch = curl_init("http://localhost".$_SERVER["PHP_SELF"]."?i=".($i+1));
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_TIMEOUT, 1);
curl_exec($ch);
print "hello world\n";
?>
Some how this does not work on GAE. So what if we do HTTP GET on some other url say url2 which does HTTP GET on the first url ? This seem to work in GAE. Code for this looks like this.
class FirstUrl(webapp.RequestHandler):
def get(self):
self.response.out.write("ok")
time.sleep(2)
urlfetch.fetch("http://"+self.request.headers["HOST"]+'/url2')
class SecondUrl(webapp.RequestHandler):
def get(self):
self.response.out.write("ok")
time.sleep(2)
urlfetch.fetch("http://"+self.request.headers["HOST"]+'/url1')
application = webapp.WSGIApplication([('/url1', FirstUrl), ('/url2', SecondUrl)])
def main():
run_wsgi_app(application)
if __name__ == "__main__":
main()
Since we found out a way to run background task, lets build abstractions for periodic task (timer) and a looping construct which spans across many HTTP requests (foreach).
Timer
Now building timer is straight forward. Basic idea is to have list of timers and the interval at which each should be called. Once we reach that interval call the callback function. We will use memcache to maintain the timer list. To find out when to call callback, we will store a key in memcache with interval as expiration time. We periodically (say 5secs) check if that key is present, if not present then call the callback and again set that key with interval.
def timer(func, interval):
timerlist = memcache.get('timer')
if(None == timerlist):
timerlist = []
timerlist.append({'func':func, 'interval':interval})
memcache.set('timer-'+func, '1', interval)
memcache.set('timer', timerlist)
def checktimers():
timerlist = memcache.get('timer')
if(None == timerlist):
return False
for current in timerlist:
if(None == memcache.get('timer-'+current['func'])):
#reset interval
memcache.set('timer-'+current['func'], '1', current['interval'])
#invoke callback function
try:
eval(current['func']+'()')
except:
pass
return True
return False
Foreach
This is needed when we want to do long taking computation say doing some operation on 1000 database rows or fetch 1000 urls etc. Basic idea is to maintain list of callbacks and arguments in memcache and each time invoke callback with the argument.
def foreach(func, args):
looplist = memcache.get('foreach')
if(None == looplist):
looplist = []
looplist.append({'func':func, 'args':args})
memcache.set('foreach', looplist)
def checkloops():
looplist = memcache.get('foreach')
if(None == looplist):
return False
if((len(looplist) > 0) and (len(looplist[0]['args']) > 0)):
arg = looplist[0]['args'].pop(0)
func = looplist[0]['func']
if(len(looplist[0]['args']) == 0):
looplist.pop(0)
if((len(looplist) > 0) and (len(looplist[0]['args']) > 0)):
memcache.set('foreach', looplist)
else:
memcache.delete('foreach')
try:
eval(func+'('+repr(arg)+')')
except:
pass
return True
else:
return False
# instead of
# foreach index in range(0, 1000):
# someoperaton(index)
# we will say
# foreach('someoperaton', range(0, 1000))
Now building a program which fetches list of urls every one hour is straight forward. Here is the code.
def getone(url):
try:
result = urlfetch.fetch(url)
if(result.status_code == 200):
memcache.set(url, '1', 60*60)
#process result.content
except :
pass
def getallurl():
#list of urls to be fetched
urllist = ['http://www.google.com/', 'http://www.cnn.com/', 'http://www.yahoo.com', 'http://news.google.com']
fetchlist = []
for url in urllist:
if (memcache.get(url) is None):
fetchlist.append(url)
#this is equivalent to
#for url in fetchlist: getone(url)
if(len(fetchlist) > 0):
foreach('getone', fetchlist)
#register the timer callback
timer('getallurl', 3*60)
complete code is here http://groups.google.com/group/httpmr-discuss/t/1648611a54c01aa
I have been running this code on appengine for few days without much problem.
Warning: We make heavy use of urlfetch. The limit on no of urlfetch per day is 160000. So be careful not to reach that limit.
You can find more about cron jobs in Python App Engine here.
Up and coming version of runtime will have some kind of periodic execution engine a'la cron. See this message on AppEngine group.
So, all the SDK pieces appear to work, but my testing indicates this isn't running on the production servers yet-- I set up an "every 1 minutes" cron that logs when it runs, and it hasn't been called yet
Hard to say when this will be available, though...
Using the Deferred Python Library is the easiest way of doing background task on Appengine using Python which is built on top of TaskQueue API.
from google.appengine.ext import deferred
def do_something_expensive(a, b, c=None):
logging.info("Doing something expensive!")
# Do your work here
# Somewhere else
deferred.defer(do_something_expensive, "Hello, world!", 42, c=True)
If you want to run background periodic tasks, see this question (AppEngine cron)
If your tasks are not periodic, see Task Queue Python API or Task Queue Java API
There is a cron facility built into app engine.
Please refer to:
https://developers.google.com/appengine/docs/python/config/cron?hl=en
Use the Task Queue - http://code.google.com/appengine/docs/java/taskqueue/overview.html