We Keep Coding

How to include heavy model training in a FastAPI endpoint?

There are many resources online describing usage of fastapi for prediction using a pre-trained model that is static. What if we would like to have an endpoint to initiate retraining of new data? It seems that fastapi's BackgroundTasks is not suitable for a training of several minutes that would take up a lot of CPU.
I am wondering how the following skeleton code could roughly be extended, assuming the underling model is a deep learning model (so quite long training).
import numpy as np
import pickle
from fastapi import FastAPI
from pydantic import BaseModel
import os
app = FastAPI(title="dummy app")
# represents a particular datapoint
class Datapoint(BaseModel):
feature1: float
feature2: float
feature3: float
#app.on_event("startup")
def load_model():
# load model from pickle file
#print(os.getcwd())
#print(os.listdir())
with open("../app/model.pkl", "rb") as file:
global model
model = pickle.load(file)
#app.get("/")
async def root():
return {"message": "Hello World"}
#app.post("/predict")
def predict(data: Datapoint):
data_point = np.array(
[
[
data.feature1,
data.feature2,
data.feature3,
]
]
)
pred = model.predict(data_point).tolist()
pred = pred[0]
print(pred)
return {"Prediction": pred}

Tensorflow: Can I run my tensorflow model directly in flask?

I have created a tensorflow model, saved it and have tested it. I really don't know how to use tensorflow serving and I'm not sure about the input and output nodes of my model to convert it into protobuf and then use tensorflow serving. So, I wanted to know if I can directly use the prediction function on flask and load the model there to make the predictions? I am really confused as to why we have to use only tensorflow serving to deploy tensorflow models? Is there any easier direct way?

You can, but you will need to set up a TensorFlow Serving server. Then you send a post request to the server.
Reference link here: Deploying-keras-models-using-tensorflow-serving-and-flask
Reference link here: Serving-TensorFlow flask client
"""This script wraps the client into a Flask server. It receives POST request with
prediction data, and forward the data to tensorflow server for inference.
"""
from flask import Flask, render_template, request, url_for, jsonify,Response
import json
import tensorflow as tf
import numpy as np
import os
import argparse
import sys
from datetime import datetime
from grpc.beta import implementations
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2
tf.app.flags.DEFINE_string('server', 'localhost:9000', 'PredictionService host:port')
FLAGS = tf.app.flags.FLAGS
app = Flask(__name__)
class mainSessRunning():
def __init__(self):
host, port = FLAGS.server.split(':')
channel = implementations.insecure_channel(host, int(port))
self.stub = prediction_service_pb2.beta_create_PredictionService_stub(channel)
self.request = predict_pb2.PredictRequest()
self.request.model_spec.name = 'example_model'
self.request.model_spec.signature_name = 'prediction'
def inference(self, val_x):
# temp_data = numpy.random.randn(100, 3).astype(numpy.float32)
temp_data = val_x.astype(np.float32).reshape(-1, 3)
print("temp_data is:", temp_data)
data, label = temp_data, np.sum(temp_data * np.array([1, 2, 3]).astype(np.float32), 1)
self.request.inputs['input'].CopyFrom(
tf.contrib.util.make_tensor_proto(data, shape=data.shape))
result = self.stub.Predict(self.request, 5.0)
return result, label
run = mainSessRunning()
print("Initialization done. ")
# Define a route for the default URL, which loads the form
#app.route('/inference', methods=['POST'])
def inference():
request_data = request.json
input_data = np.expand_dims(np.array(request_data), 0)
result, label = run.inference(input_data)
di={"result":str(result),'label': label[0].tolist()}
return Response(json.dumps(di), mimetype='application/json')

Image Recognition with Tensorflow + keras + Flask

I was trying to create a web application (REST API) for Image Recognition with Tensorflow + keras in Flask. I have tried to follow some resource available in internet and came up with a script as below
from imageai.Prediction import ImagePrediction
import os
execution_path = os.getcwd()
prediction = ImagePrediction()
prediction.setModelTypeAsResNet()
prediction.setModelPath( execution_path + "\\resnet50_weights_tf_dim_ordering_tf_kernels.h5")
prediction.loadModel()
for i in range(3):
predictions, percentage_probabilities = prediction.predictImage("C:\\Users\\Administrator\\Downloads\\pics\\banana"+str(i)+".jpg", result_count=5)
for index in range(len(predictions)):
print(predictions[index] , " : " , percentage_probabilities[index])
This worked fine as standalone script. Then I tried to convert the same to Flask Application and it starts failing .
import flask
from imageai.Prediction import ImagePrediction
import os
import json
import keras
# initialize our Flask application and the Keras model
app = flask.Flask(__name__)
def init():
global model
execution_path = os.getcwd()
model = ImagePrediction()
model.setModelTypeAsResNet()
model.setModelPath( os.path.join(os.getcwd(),"models","resnet50_weights_tf_dim_ordering_tf_kernels.h5"))
model.loadModel()
# API for prediction
#app.route("/predict", methods=["GET"])
def predict():
predictions, percentage_probabilities = model.predictImage( os.path.join(os.getcwd(),"pics","banana.jpg"), result_count=5)
mylist = []
for index in range(len(predictions)):
mydict = {}
mydict[predictions[index]]=percentage_probabilities[index]
mylist.append(mydict)
keras.backend.clear_session()
return sendResponse(json.dumps(mylist))
# Cross origin support
def sendResponse(responseObj):
response = flask.jsonify(responseObj)
response.headers.add('Access-Control-Allow-Origin', '*')
response.headers.add('Access-Control-Allow-Methods', 'GET')
response.headers.add('Access-Control-Allow-Headers', 'accept,content-type,Origin,X-Requested-With,Content-Type,access_token,Accept,Authorization,source')
response.headers.add('Access-Control-Allow-Credentials', True)
return response
# if this is the main thread of execution first load the model and then start the server
if __name__ == "__main__":
print(("* Loading Keras model and Flask starting server..."
"please wait until server has fully started"))
init()
app.run(threaded=True)

Keras: Save Model In Suitable Format for Google Cloud ML Engine (missing functions)

I'm trying to deploy a recently trained Keras model on Google Cloud ML Engine. I googled around to see what format the saved model needs to be for ML Engine and found this:
import keras.backend as K
import tensorflow as tf
from keras.models import load_model, Sequential
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import tag_constants, signature_constants
from tensorflow.python.saved_model.signature_def_utils_impl import predict_signature_def
# reset session
K.clear_session()
sess = tf.Session()
K.set_session(sess)
# disable loading of learning nodes
K.set_learning_phase(0)
# load model
model = load_model('model.h5')
config = model.get_config()
weights = model.get_weights()
new_Model = Sequential.from_config(config)
new_Model.set_weights(weights)
# export saved model
export_path = 'YOUR_EXPORT_PATH' + '/export'
builder = saved_model_builder.SavedModelBuilder(export_path)
signature = predict_signature_def(inputs={'NAME_YOUR_INPUT': new_Model.input},
outputs={'NAME_YOUR_OUTPUT': new_Model.output})
with K.get_session() as sess:
builder.add_meta_graph_and_variables(sess=sess,
tags=[tag_constants.SERVING],
signature_def_map={
signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: signature})
builder.save()
However, in Keras 2.1.3, keras.backend no longer seems to have clear_session(), set_session(), or get_session(). What's the modern way of handling this issue? Do those functions live elsewhere now?
Thanks!

Tensor is not an element of this graph; deploying Keras model

Im deploying a keras model and sending the test data to the model via a flask api. I have two files:
First: My Flask App:
# Let's startup the Flask application
app = Flask(__name__)
# Model reload from jSON:
print('Load model...')
json_file = open('models/model_temp.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
keras_model_loaded = model_from_json(loaded_model_json)
print('Model loaded...')
# Weights reloaded from .h5 inside the model
print('Load weights...')
keras_model_loaded.load_weights("models/Model_temp.h5")
print('Weights loaded...')
# URL that we'll use to make predictions using get and post
#app.route('/predict',methods=['GET','POST'])
def predict():
data = request.get_json(force=True)
predict_request = [data["month"],data["day"],data["hour"]]
predict_request = np.array(predict_request)
predict_request = predict_request.reshape(1,-1)
y_hat = keras_model_loaded.predict(predict_request, batch_size=1, verbose=1)
return jsonify({'prediction': str(y_hat)})
if __name__ == "__main__":
# Choose the port
port = int(os.environ.get('PORT', 9000))
# Run locally
app.run(host='127.0.0.1', port=port)
Second: The file Im using to send the json data sending to the api endpoint:
response = rq.get('api url has been removed')
data=response.json()
currentDT = datetime.datetime.now()
Month = currentDT.month
Day = currentDT.day
Hour = currentDT.hour
url= "http://127.0.0.1:9000/predict"
post_data = json.dumps({'month': month, 'day': day, 'hour': hour,})
r = rq.post(url,post_data)
Im getting this response from Flask regarding Tensorflow:
ValueError: Tensor Tensor("dense_6/BiasAdd:0", shape=(?, 1), dtype=float32) is not an element of this graph.
My keras model is a simple 6 dense layer model and trains with no errors.
Any ideas?

Flask uses multiple threads. The problem you are running into is because the tensorflow model is not loaded and used in the same thread. One workaround is to force tensorflow to use the gloabl default graph .
Add this after you load your model
global graph
graph = tf.get_default_graph()
And inside your predict
with graph.as_default():
y_hat = keras_model_loaded.predict(predict_request, batch_size=1, verbose=1)

It's so much simpler to wrap your keras model in a class and that class can keep track of it's own graph and session. This prevents the problems that having multiple threads/processes/models can cause which is almost certainly the cause of your issue. While other solutions will work this is by far the most general, scalable and catch all. Use this one:
import os
from keras.models import model_from_json
from keras import backend as K
import tensorflow as tf
import logging
logger = logging.getLogger('root')
class NeuralNetwork:
def __init__(self):
self.session = tf.Session()
self.graph = tf.get_default_graph()
# the folder in which the model and weights are stored
self.model_folder = os.path.join(os.path.abspath("src"), "static")
self.model = None
# for some reason in a flask app the graph/session needs to be used in the init else it hangs on other threads
with self.graph.as_default():
with self.session.as_default():
logging.info("neural network initialised")
def load(self, file_name=None):
"""
:param file_name: [model_file_name, weights_file_name]
:return:
"""
with self.graph.as_default():
with self.session.as_default():
try:
model_name = file_name[0]
weights_name = file_name[1]
if model_name is not None:
# load the model
json_file_path = os.path.join(self.model_folder, model_name)
json_file = open(json_file_path, 'r')
loaded_model_json = json_file.read()
json_file.close()
self.model = model_from_json(loaded_model_json)
if weights_name is not None:
# load the weights
weights_path = os.path.join(self.model_folder, weights_name)
self.model.load_weights(weights_path)
logging.info("Neural Network loaded: ")
logging.info('\t' + "Neural Network model: " + model_name)
logging.info('\t' + "Neural Network weights: " + weights_name)
return True
except Exception as e:
logging.exception(e)
return False
def predict(self, x):
with self.graph.as_default():
with self.session.as_default():
y = self.model.predict(x)
return y

Just after loading the model add model._make_predict_function()
`
# Model reload from jSON:
print('Load model...')
json_file = open('models/model_temp.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
keras_model_loaded = model_from_json(loaded_model_json)
print('Model loaded...')
# Weights reloaded from .h5 inside the model
print('Load weights...')
keras_model_loaded.load_weights("models/Model_temp.h5")
print('Weights loaded...')
keras_model_loaded._make_predict_function()

It turns out this way does not need a clear_session call and is at the same time configuration friendly, using the graph object from configured session session = tf.Session(config=_config); self.graph = session.graph and the prediction by the created graph as default with self.graph.as_default(): offers a clean approach
from keras.backend.tensorflow_backend import set_session
...
def __init__(self):
config = self.keras_resource()
self.init_model(config)
def init_model(self, _config, *args):
session = tf.Session(config=_config)
self.graph = session.graph
#set configured session
set_session(session)
self.model = load_model(file_path)
def keras_resource(self):
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
return config
def predict_target(self, to_predict):
with self.graph.as_default():
predict = self.model.predict(to_predict)
return predict

I had the same problem. it was resolved by changing TensorFlow-1 version to TensorFlow-2.
just uninstall ver-1 and install ver-2.

Ya their is a bug when you predict from model with keras. Keras will not be able to build graph due to some error. Try to predict images from model with the help of tensor flow. Just replace this line of code
Keras code:
features = model_places.predict( img )
tensorflow code:
import tensorflow as tf
graph = tf.get_default_graph()
import this library in your code and replace.
with graph.as_default():
features = model_places.predict( img ).tolist()
If Problem still not solved :
if still problem not solved than try to refresh the graph.
As your code is fine, running with a clean environment should solve it.
Clear keras cache at ~/.keras/
Run on a new environment, with the right packages (can be done easily with anaconda)
Make sure you are on a fresh session, keras.backend.clear_session() should remove all existing tf graphs.
Keras Code:
keras.backend.clear_session()
features = model_places.predict( img )
TensorFlow Code:
import tensorflow as tf
with tf.Session() as sess:
tf.reset_default_graph()

Simplest solution is to use tensorflow 2.0. Run your code in Tensorflow 2.0 environment and it will work.
I was facing same issues while exposing a pre-trained model via REST server. I was loading the model at the server startup and later using the loaded model to make predictions via POST/GET request. While predicting it was generating error as session not saved between the predict call. Though when I was loading the model every time prediction is made it was working fine.
Then to avoid this issue with the session I just ran the code in TF=2.0 environment and it ran fine.