We Keep Coding

Recieving output text when a TensorFlow model name != x

I have made a model which detects when a person has their face to the right, left, or in the middle. I am making a prediction using the following code:
from keras.models import load_model
from PIL import Image, ImageOps
import numpy as np
# Disable scientific notation for clarity
np.set_printoptions(suppress=True)
# Load the model
model = load_model('models//keras_Model.h5', compile=False)
# Load the labels
class_names = open('models//labels.txt', 'r').readlines()
# Create the array of the right shape to feed into the keras model
# The 'length' or number of images you can put into the array is
# determined by the first position in the shape tuple, in this case 1.
data = np.ndarray(shape=(1, 224, 224, 3), dtype=np.float32)
# Replace this with the path to your image
image = Image.open('IMG.png').convert('RGB')
# resize the image to a 224x224 with the same strategy as in TM2:
# resizing the image to be at least 224x224 and then cropping from the center
size = (224, 224)
image = ImageOps.fit(image, size, Image.Resampling.LANCZOS)
# turn the image into a numpy array
image_array = np.asarray(image)
# Normalize the image
normalized_image_array = (image_array.astype(np.float32) / 127.0) - 1
# Load the image into the array
data[0] = normalized_image_array
# run the inference
prediction = model.predict(data)
index = np.argmax(prediction)
class_name = class_names[index]
confidence_score = prediction[0][index]
print('Class:', class_name, end='')
print('Confidence score:', confidence_score)
That code gives me the proper prediction. Here is the labels.txt file:
0 Left_Side_Face
1 Middle_Face
2 Right_Side_Face
What I want to do now is say for example the person has turned their head to the right side, I want to print the message 'face the computer' but when I tried doing it I failed.
Here is the code I tried:
if class_name == "2 Right_Side_Face":
print('face the computer')
if class_name == "0 Left_Side_Face":
print('face the computer')
else:
print('Invalid class name')
print('exiting.....')
The problem is that whenever I run the code above it goes to the else and print 'Invalid class name' and when I remove the else there are no errors in the code, it just skips the if statements.
Question:
How can I do something (print a message, alarm, etc.) when a specific class is detected by a TensorFlow model?
EDIT:
Here is the output of the script when I feed the model with a right side face image:
Class: 2 Right_Side_Face
Confidence score: 0.84736574
Invalid class name
exiting....

I think this is because class_name is not a complete string but an int + a string: maybe do: if str(class_name) == "2 Right_Side_Face": instead of if class_name == "2 Right_Side_Face": to make sure it is a complete string before comparing or just use the index : if index == 2: print('face the computer')

Forward method error in DNN module of OpenCV (Ptyhon) using .onnx model

I wanted to test a pretrained model downloaded from here to perform an ocr task. Link to download, its name is CRNN_VGG_BiLSTM_CTC.onnx. This model is extracted from here. The sample-image.png can be download from here (see the code bellow).
When I do the forward of the neural network to predict (ocr) in the blob I get the following error:
error: OpenCV(4.4.0) /tmp/pip-req-build-xgme2194/opencv/modules/dnn/src/layers/convolution_layer.cpp:348: error: (-215:Assertion failed) ngroups > 0 && inpCn % ngroups == 0 && outCn % ngroups == 0 in function 'getMemoryShapes'
Feel free to read the code bellow. I tried many things, it's weird because this model does not require a predetermined input shape. If you know any way to read this model and do the forward it is also going to be helpful but I'd rather solve using OpenCV.
import cv2 as cv
# The model is downloaded from here https://drive.google.com/drive/folders/1cTbQ3nuZG-EKWak6emD_s8_hHXWz7lAr
# model path
modelRecognition = os.path.join(MODELS_PATH,'CRNN_VGG_BiLSTM_CTC.onnx')
# read net
recognizer = cv.dnn.readNetFromONNX(modelRecognition)
# Download sample_image.png from https://i.ibb.co/fMmCB7J/sample-image.png (image host website)
sample_image = cv.imread('sample-image.png')
# Height , Width and number of channels of the image
H, W, C = sample_image.shape
# Create a 4D blob from cropped image
blob = cv.dnn.blobFromImage(sample_image, size = (H, W))
recognizer.setInput(blob)
# Here is where i get the errror that I mentioned before
result = recognizer.forward()
Thank you so much in advance.

Your problem is actually that the input data you feed to your model doesn't match the shape of the data the model was trained on.
I used this answer to inspect your onnx model and it appears that it expects an input of shape (1, 1, 32, 100). I modified your code to reshape the image to 1 x 32 x 100 pixels and the inference actually runs without error.
EDIT
I've added some code to interpret the result of the inference. We now display the image and the inferred OCR text.
This doesn't seem to be working, but reading the tutorial on OpenCV, there should be two models:
one that detects where there is text in the image. This network accepts images of various sizes, it returns the locations of text within the image and then cropped parts of the image, of sizes 100x32 are passed to the second
one that actually does the "reading" and given patches of image, returns the characters. For this, there a file alphabet_36.txt that is provided together with the pre-trained models.
It isn't clear to me though which network to use for text detection. Hope the edited code below helps you develop your application further.
import cv2 as cv
import os
import numpy as np
import matplotlib.pyplot as plt
# The model is downloaded from here https://drive.google.com/drive/folders/1cTbQ3nuZG-EKWak6emD_s8_hHXWz7lAr
# model path
MODELS_PATH = './'
modelRecognition = os.path.join(MODELS_PATH,'CRNN_VGG_BiLSTM_CTC.onnx')
# read net
recognizer = cv.dnn.readNetFromONNX(modelRecognition)
# Download sample_image.png from https://i.ibb.co/fMmCB7J/sample-image.png (image host website)
sample_image = cv.imread('sample-image.png', cv.IMREAD_GRAYSCALE)
sample_image = cv.resize(sample_image, (100, 32))
sample_image = sample_image[:,::-1].transpose()
# Height and Width of the image
H,W = sample_image.shape
# Create a 4D blob from image
blob = cv.dnn.blobFromImage(sample_image, size=(H,W))
recognizer.setInput(blob)
# network inference
result = recognizer.forward()
# load alphabet
with open('alphabet_36.txt') as f:
alphabet = f.readlines()
alphabet = [f.strip() for f in alphabet]
# interpret inference results
res = []
for i in range(result.shape[0]):
ind = np.argmax(result[i,0])
res.append(alphabet[ind])
ocrtxt = ''.join(res)
# show image and detected OCR characters
plt.imshow(sample_image)
plt.title(ocrtxt)
plt.show()
Hope it helps.
Cheers

How to fix "TypeError: 'int' object is not subscriptable in the rfind(os.sep.path) + 1:"

I am trying to classify my input image using pre-trained labels and matching that with the label inserted into the input image file name like : "my_name.png" ; which is also there in the label.pickle "my_name"
def onClassify(self):
IMAGE_DIMS = (96, 96, 3)
image = cv2.imread("C:/Users/Jimit Vaghela/PycharmProjects/Image Classification/gui /cap_images/fold_1/jimit_Image_202084T132225.png")
output = image.copy()
# pre-process the image for classification
image = cv2.resize(image, (IMAGE_DIMS[1], IMAGE_DIMS[0]))
image = image.astype("float") / 255.0
image = img_to_array(image)
image = np.expand_dims(image, axis=0)
# load the trained convolutional neural network and the label
# binarizer
print("[INFO] loading network...")
model = load_model("C:/Users/Jimit Vaghela/PycharmProjects/Image Classification/gui/classifier/")
lb = pickle.loads(open("C:/Users/Jimit Vaghela/PycharmProjects/Image Classification/gui/labelbin.pickle", "rb").read())
# classify the input image
print("[INFO] classifying image...")
proba = model.predict(image)[0]
idx = np.argmax(proba)
label = lb.classes_[idx]
# we'll mark our prediction as "correct" of the input image filename
# contains the predicted label text (obviously this makes the
# assumption that you have named your testing image files this way)
filename = "jimit_Image_202084T132225.png".rfind(os.path.sep)[+1:]
correct = "correct" if filename.rfind(label) != -1 else "incorrect"
# build the label and draw the label on the image
label = "{}: {:.2f}% ({})".format(label, proba[idx] * 100, correct)
output = imutils.resize(output, width=400)
cv2.putText(output, label, (10, 25), cv2.FONT_HERSHEY_SIMPLEX,
0.7, (0, 255, 0), 2)
# show the output image
print("[INFO] {}".format(label))
self.label.setPixmap(QPixmap.fromImage(output)) ```
The error is in this line: filename = "jimit_Image_202084T132225.png".rfind(os.path.sep)[+1:]
What am I doing wrong here? What should I add in front of .rfind() method.
Desired output as asked by #MisterMiyagi:
As you can see the input image was named as "charmander_counter.png" in the directory "/examples/charmander_counter.png" and I am using my "my_name.png" replacing the original code.
To clarify, I have already trained the model with various images of mine in a class named as "my_name"

if you are trying to get the file name without the extension then try this
filename = "jimit_Image_202084T132225.png".rsplit(".")[-2]
which would give you
jimit_Image_202084T132225
assuming you only have file names, if the filenname might also include path, something like this would work.
filepath = "path/to/file/jimit_Image_202084T132225.png"
filename_with_extension = filepath..rsplit(os.path.sep)[-1] # this will extract the filename, giving you jimit_Image_202084T132225.png
filename = filename_with_extension.rsplit(".")[-2] # this will remove the extension giving you jimit_Image_202084T132225.
or in one line
filename = "path/to/file/jimit_Image_202084T132225.png".rsplit(os.path.sep)[-1].rsplit(".")[-2]
note that this relies on the file having an extension so if your original file was jimit_Image_202084T132225 (no .png) then it won't work.
this would also work if you had multiple file extensions like jimit_Image_202084T132225.new.png you'd get jimit_Image_202084T132225.new, if that is not desired then comment and I'll change it to the desired one
Edit: as mentioned by #MisterMiyagi os.path contains tools for doing this, so you can use this to achieve the same thing
os.path.splitext(os.path.basename("path/to/file/jimit_Image_202084T132225.new.png"))[0]

Why OpenCV SimpleBlobDetector only works on the image saved by using matplotlib.pyplot.savefig() but not the one by cv2.imwrite()?

First of all, I am trying to generate images of the simulation environment OpenAI Gym. What I get is a 128 x 128 x 3 numpy.ndarray, which is assigned to the variable state. This image contains a target object, and I want to detect the object by using the SimpleBlobDetector in cv2 (opencv).
So far, I have tried three ways to achieve this goal.
1. use the variable state as the input of SimpleBlobDetector
2. use matplotlib.pyplot.savefig() to save the variable state as test_mat.png, then restore the image as the input of SimpleBlobDetector.
3. Do the same as 2. except for using cv2.imwrite() to save the image.
The following is how I store the images:
import gym
import random
import matplotlib.pyplot as plt
import cv2
env = gym.make('Reacher-v2')
state = env.reset()
print("State:",state.shape) # (128, 128, 3)
# save the image with matplotlib
%matplotlib inline
plt.axis('off')
plt.imshow(state)
plt.savefig('test_mat.png', bbox_inches='tight', pad_inches=0)
plt.show()
# save the image with cv2
cv2.imwrite('test_cv2.png', state)
Then I tried reading in the image test_mat.png using cv2.imread() and the blob detector worked.
im_cv2 = cv2.imread("test_mat.png")
detector = cv2.SimpleBlobDetector_create()
print("im_cv2: ", im_cv2.shape, type(im_cv2))
keypoints = detector.detect(im_cv2)
print("Keypoints: ", keypoints)
if keypoints:
print(keypoints[0].pt)
The output looks like:
im_cv2: (237, 243, 3) <class 'numpy.ndarray'>
Keypoints: [<KeyPoint 0x128a5b2a0>]
(102.55021667480469, 96.66270446777344)
However, when I tried using cv2.imread() to restore test_cv2.png (which is saved by using cv2.imwrite()), the blob detector failed.
im_cv2 = cv2.imread("test_cv2.png")
detector = cv2.SimpleBlobDetector_create()
print("im_cv2: ", im_cv2.shape, type(im_cv2))
keypoints = detector.detect(im_cv2)
print("Keypoints: ", keypoints)
if keypoints:
print(keypoints[0].pt)
The output:
im_cv2: (128, 128, 3) <class 'numpy.ndarray'>
Keypoints: []
Also, using the variable state as input of SimpleBlobDetector didn't work either.

How to feed an image ROI into session.run() of Tensorflow?

I am trying to feed my image roi into the Tensorflow classifier I took from here. The idea is to first run a simple filter, get rectangle candidates, and then check (using the network) whether each rectangle(roi) is actually what I am looking for.
class ScrewDetector:
def __init__(self):
self.session = None # an internal variable needed for inception network
# to keep the screw data in
self.screw_data = dict()
# load the labels of the classification: screw / non-screw
self.class_labels = [line.rstrip() for line in tf.gfile.GFile(home + "/imagine_weights/screw_detector/retrained_labels.txt")]
# prepare the network
with tf.gfile.FastGFile(home + "/weights/screw_detector/retrained_graph.pb", 'rb') as f:
graph_def = tf.GraphDef() ## the graph-graph_def is a saved copy of a TensorFlow graph, object initialization
graph_def.ParseFromString(f.read()) # parse serialized protocol buffer data into variable
_ = tf.import_graph_def(graph_def, name='') # import a serialized TensorFlow GraphDef protocol buffer, extract objects in the GraphDef as tf.Tensor
# start the session
with tf.Session() as self.session:
self.softmax_tensor = self.session.graph.get_tensor_by_name('final_result:0')
def detect_screw(self):
# get a copy and resize it
img_raw = self.cv_image.copy()
resized_img = cv2.resize(img_raw, (0,0), fx=RESIZE_FACTOR, fy=RESIZE_FACTOR)
# grayscale it
gray = cv2.cvtColor(resized_img, cv2.COLOR_BGR2GRAY)
# detect circles in the image
circles = cv2.HoughCircles(gray, cv2.HOUGH_GRADIENT, 1, 100, param1=50,param2=35,minRadius=15,maxRadius=30)
# ensure at least some circles were found
if circles is not None:
# convert the (x, y) coordinates and radius of the circles to integers
circles = np.round(circles[0, :]).astype("int")
# get a counter
screw_id = 0
# loop over the (x, y) coordinates and radius of the circles
for (x, y, r) in circles:
# draw the circle in the output image, then draw a rectangle corresponding to the center of the circle
#cv2.circle(resized_img, (x, y), r, (0, 255, 0), 4)
cv2.rectangle(resized_img, (x - r, y - r), (x + r, y + r), (0, 0, 255), 5)
# get the above rectangle as ROI
screw_roi = resized_img[y:y+r, x:x+r]
# feed it into the network
#import IPython; IPython.embed()
predictions = self.session.run(self.softmax_tensor, feed_dict={screw_id: [screw_roi.flatten()]})
# get prediction values in array back
top_k = predictions[0].argsort()[-len(predictions[0]):][::-1]
# output
for node_id in top_k:
human_string = self.class_labels[node_id]
score = predictions[0][node_id]
print('%s (score = %.5f)' % (human_string, score))
# if it is a screw, go on, save its coordinates and append into the network
# remap in the original image
scaled_point = (round(x * (1/RESIZE_FACTOR)), round(y * (1/RESIZE_FACTOR)))
# append to the dict
self.screw_data[scaled_point] = r * RESIZE_FACTOR
# iterate the counter
screw_id += screw_id
# publish the result, which is an image (scaled)
result_image_msg = Image()
try:
result_image_msg = self.bridge.cv2_to_imgmsg(resized_img, "bgr8")
#print(self.screw_data)
except CvBridgeError as e:
print("Could not make it through the cv bridge of death.")
self.result_image_pub.publish(result_image_msg)
else:
print("No detection of circles.")
but I get:
TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a int into a Tensor.
I do know that the variables screw_id and screw_roi are not empty. And I do know that one needs to feed a dictionary in, which is why in the first place I was trying to do that. But I can't get it running for the reason above.
Any thoughts?
EDIT: So normally, this code loads the image and conducts the prediction as follows:
image_data = tf.gfile.FastGFile(image_path, 'rb').read()
softmax_tensor = sess.graph.get_tensor_by_name('final_result:0')
predictions = sess.run(softmax_tensor, {'DecodeJpeg/contents:0': image_data})
All I want is to turn this into a form which operates with the image ROI provided during the operation. It can't be too complicated.

It's not a rocket science, it turns out.
One somehow needs to convert image so that he can pass a string of image bytes, because that's what the function sess.run() expects.
If you don't have a file that you want to load from a file system, then the following is the way:
image_data = cv2.imencode('.jpg', screw_roi)[1].tostring() # pass a string of image bytes
after this, you simply can run:
predictions = self.session.run(self.softmax_tensor, {'DecodeJpeg/contents:0': image_data})
That's it.

feed_dict expect a dictionary with tensors as keys, to populate the placeholders with the specified valued. It's not in your code snippet how does the screw_id is initiated, but I bet it's not a tensor of any kind, hence, your error.