I have a pytorch model in NLP and a script for use it in python. Now i want to use this script in C#. I tried run python script from C# and it worked. I get user sentence in C#, pass it to python and its outputs use in C#. The problem is that i want to do this work in a loop until user select exit but every time it goes to python code, it must load pytorch model and its time consuming.
Is there a way to load model one time and then in a loop get input from user and inference with loaded model?
you can export the model in ONNX format and then use the OpenCV DNN module or tensorrt for inference purposes. It will give you a significant boost in speed and your whole code will be in C#.
Related
I am very new to machine learning. I have a python file with a very simple TensorFlow model that I need to deploy on Android using Google's ML Kit (which would be creating tflite file). I absolutely don't understand what should be the structure of my python file and Google's documentation doesn't make it any easier. Maybe someone has a good example of converting CUSTOM MODEL WRITTEN FROM SCRATCH and then using it in Java. I need to pass a string from Android's text field and get a predicted answer.
You need to first train your model on whatever the dataset you have. The layers in the model must comply with the supported layers by the TFLite library. Here is a list of layers that are supported and unsupported.
Once you have trained it, based on how you saved it (Let's say using kerasmodel.save). Convert it to TFLite following this tutorial or other tutorials on this page.
Now you can use this .tflite model in Android studio. For this you can follow this good tutorial.
Hi I have some problem about Keras with python 3.6
My enviroment is keras with Python and Only CPU.
but the problem is when I iterate same Keras model for predict some diferrent input, its getting slower and slower..
my code is so simple just like that
for i in range(100):
model.predict(x)
the First run is fast. it takes 2 seconds may be. but second run takes 3 seconds and Third takes 5 seconds... its getting slower and slower even if I use same input.
what can I iterate predict keras model hold fast? I don't want any getting slower.. it will be very critical.
How can I Fix IT??
Try using the __call__ method directly. The documentation of the predict method states the following:
For small numbers of inputs that fit in one batch, directly use __call__() for faster execution, e.g., model(x).
I see the performance is critical in this case. So, if it doesn't help, you could use OpenVINO which is optimized for Intel hardware but it should work with any CPU. Your performance should be much better than using Keras directly.
It's rather straightforward to convert the Keras model to OpenVINO. The full tutorial on how to do it can be found here. Some snippets below.
Install OpenVINO
The easiest way to do it is using PIP. Alternatively, you can use this tool to find the best way in your case.
pip install openvino-dev[tensorflow2]
Save your model as SavedModel
OpenVINO is not able to convert HDF5 model, so you have to save it as SavedModel first.
import tensorflow as tf
from custom_layer import CustomLayer
model = tf.keras.models.load_model('model.h5', custom_objects={'CustomLayer': CustomLayer})
tf.saved_model.save(model, 'model')
Use Model Optimizer to convert SavedModel model
The Model Optimizer is a command-line tool that comes from OpenVINO Development Package. It converts the Tensorflow model to IR, which is a default format for OpenVINO. You can also try the precision of FP16, which should give you better performance without a significant accuracy drop (change data_type). Run in the command line:
mo --saved_model_dir "model" --data_type FP32 --output_dir "model_ir"
Run the inference
The converted model can be loaded by the runtime and compiled for a specific device e.g. CPU or GPU (integrated into your CPU like Intel HD Graphics). If you don't know what is the best choice for you, use AUTO.
# Load the network
ie = Core()
model_ir = ie.read_model(model="model_ir/model.xml")
compiled_model_ir = ie.compile_model(model=model_ir, device_name="CPU")
# Get output layer
output_layer_ir = compiled_model_ir.output(0)
# Run inference on the input image
result = compiled_model_ir([input_image])[output_layer_ir]
Disclaimer: I work on OpenVINO.
If your model calls the fit function in batches, there are different samples in the same batch with slightly different times over the course of the iteration, and then you try again and again to get more and more groups of predictive model performance time will be longer and longer.
My use case is I am training machine learners in python to find which model is most performant. But once I have finished choosing a model, I need to recapitulate that model in C for the shippable product.
What we have been planning to do is pick apart the model in python, save all its parameters to a file, and load these in to a different model with the same structure in C. But this strikes me as unnecessarily complex if there is actually a way to save a trained model as some kind of black-box, compiled library with a predict() function.
So is there a way to compile an object with all its data to a machine-code library? All I find about saving python objects is pickling them, which is clearly not what I need.
I'm having trouble figuring out how to use a support vector machine trained on Weka for real time processing with python.
For example when you train a back propagation algorithm on Matlab, you can extract the weights and biases and use them to replicate the network on other programs (e.g python) in feed-forward.
Thanks for your suggestions.
Assuming you want to continue using Python and Weka, the easiest way is to just call the Weka command line using subprocess (see https://docs.python.org/2/library/subprocess.html). You can then train and save your models and use them as needed. See this reference: https://weka.wikispaces.com/Saving+and+loading+models
What I have read about the tutorials is that you create your data then write the model using protobuf and then you write the solver file. Finally you train the model and you get your generated file. All this is done though command line. Now there are two questions
1) Suppose I have the generated model now how do I load a new image not in the test folder and perform a forward pass. Should it be done though command line or from some language(c++, python) ?
2) I guess above was one way of doing it. What is the best way to train the classifier (command line train/ or though coding) and how to use the generated model file(after training) in your code.
I want to interface caffe with my code but I am not able to find a short tutorial which will give me step by step on any database say mnist and the model doesn't need to be as complicated as LeNet but a simple Fully connected layer will also do. But can anyone tell me how to just write a simple code using C++ or python and train any dataset from scratch.
A sample C++/python code for training a classifier and using it to predict new data using caffe would also be appreciated.
Training is best done using the command line. See this tutorial.
Once you trained a model and you have a myModel.caffemodel file (a binary file storing the wieghts of the different layers) and a deploy.prototxt file (a text file describing your net), you can use python interface to classify images.
You can run python script classify.py to classify image(s) from command line. This script wraps around classifier.py - a python object that holds a trained net and allows you to perform forward passes in python.