What I have read about the tutorials is that you create your data then write the model using protobuf and then you write the solver file. Finally you train the model and you get your generated file. All this is done though command line. Now there are two questions
1) Suppose I have the generated model now how do I load a new image not in the test folder and perform a forward pass. Should it be done though command line or from some language(c++, python) ?
2) I guess above was one way of doing it. What is the best way to train the classifier (command line train/ or though coding) and how to use the generated model file(after training) in your code.
I want to interface caffe with my code but I am not able to find a short tutorial which will give me step by step on any database say mnist and the model doesn't need to be as complicated as LeNet but a simple Fully connected layer will also do. But can anyone tell me how to just write a simple code using C++ or python and train any dataset from scratch.
A sample C++/python code for training a classifier and using it to predict new data using caffe would also be appreciated.
Training is best done using the command line. See this tutorial.
Once you trained a model and you have a myModel.caffemodel file (a binary file storing the wieghts of the different layers) and a deploy.prototxt file (a text file describing your net), you can use python interface to classify images.
You can run python script classify.py to classify image(s) from command line. This script wraps around classifier.py - a python object that holds a trained net and allows you to perform forward passes in python.
Related
I am currently working on a Record Linkage (identifying data sets, which describe the same entity in the real world) Program. Herefore, I am using the Python Record Linkage Toolkit (https://recordlinkage.readthedocs.io/en/latest/ref-classifiers.html#classifiers) and the provided ECMClassifier. I get correct results, but right now I need to train the Classifier again every time I run my script. The relevant codelines are:
ecm = recordlinkage.ECMClassifier(binarize=0.4)
matches = ecm.fit_predict(comparison_vectors)
Now, my question is, can I just save the Classifier after the training and reload it when I run the script the next time ? This way I could save training time and maybe train the Classifier more and more each time.
Saving ML-models is not new and there are easy python packages to do this like pickle5 (as described here: https://www.projectpro.io/recipes/save-trained-model-in-python ).
What I'm concerned about is:
Does the classifier itself change/learn with each use or is everything happening only in the _fit_predict function and therefore the progress can not be saved ?
Is it a problem, that training and prediction happen in one method ? Is it not useful, to save the classifier after both steps have already been undertaken ?
Are there other things to consider when saving/reloading a classifer that are not in line with the default way pickle saves and loads objects?
I am using Python version: 3.8.13.
Thanks in advance!
I have a pytorch model in NLP and a script for use it in python. Now i want to use this script in C#. I tried run python script from C# and it worked. I get user sentence in C#, pass it to python and its outputs use in C#. The problem is that i want to do this work in a loop until user select exit but every time it goes to python code, it must load pytorch model and its time consuming.
Is there a way to load model one time and then in a loop get input from user and inference with loaded model?
you can export the model in ONNX format and then use the OpenCV DNN module or tensorrt for inference purposes. It will give you a significant boost in speed and your whole code will be in C#.
At the beginning of the year I started to look into Tensorflow and ML.
A friend of mine with some code sample to train some classes for custom objects using FASTER_RCNN_RESNET_101. Because I had no GPU and the training was taking quite some time on my machine he was running the training on his machine and he was providing me the files needed for me in the image detection code.
Meanwhile I have bought a GPU and I tried to do all the steps on my machine.
I am doing all the steps I was doing with him but now when I run the detection no objects are detected after the training.
The step that I have no documentation is the one that generates the .pb file.
I tried to use models\research\object_detection\export_inference_graph.py to generate the file like:
python models\research\object_detection\export_inference_graph.py --input_type image_tensor --pipeline_config_path data\pipeline_vio.config --trained_checkpoint_prefix data\train_folder\model.ckpt-497416 --output_directory train_folder\exported_graphs
I encounter a problem:
Using the .pb file generated no object is recognized but if I use the file he provided the objects are recognized.
Last time I did the training using python train.py --logtostderr --train_dir=data\train_folder --pipeline_config_path=data\pipeline_vio.config
Is there a method to see the difference between the 2 graphs or something to understand why one of the graphs works but not the other
I am teaching myself to code Convolutional Neural Networks. In particular I am looking at the "Dogs vs. Cats" challenge (https://medium.com/#mrgarg.rajat/kaggle-dogs-vs-cats-challenge-complete-step-by-step-guide-part-2-e9ee4967b9). I am using PyCharm.
In PyCharm, is there a way of using the trained model to make a prediction on the test data without having to run the entire file each time (and thus retrain the model each time)? Additionally, is there a way to skip the part of the script that prepares the data for input into the CNN? In a similar manner, does PyCharm store variables- can I print individual variables after the script has been run.
Would it be better if I used a different IDLE?
You can use sklearn joblib to save the trained model as a pickle and use it later for predictions.
from sklearn.externals import joblib
# Save the model as a pickle in a file
joblib.dump(knn, 'filename.pkl')
# Load the model from the file
knn_from_joblib = joblib.load('filename.pkl')
# Use the loaded model to make predictions
knn_from_joblib.predict(X_test)
I'm having trouble figuring out how to use a support vector machine trained on Weka for real time processing with python.
For example when you train a back propagation algorithm on Matlab, you can extract the weights and biases and use them to replicate the network on other programs (e.g python) in feed-forward.
Thanks for your suggestions.
Assuming you want to continue using Python and Weka, the easiest way is to just call the Weka command line using subprocess (see https://docs.python.org/2/library/subprocess.html). You can then train and save your models and use them as needed. See this reference: https://weka.wikispaces.com/Saving+and+loading+models