We Keep Coding

Detecting missing components using Opencv and Python

So I have now a running code to detect each components by using template matching, I just used the code that the author has provided in this link (https://www.sicara.fr/blog-technique/object-detection-template-matching).
This code:
import cv2
import numpy as np
DEFAULT_TEMPLATE_MATCHING_THRESHOLD = 0.9
class Template:
"""
A class defining a template
"""
def __init__(self, image_path, label, color, matching_threshold=DEFAULT_TEMPLATE_MATCHING_THRESHOLD):
"""
Args:
image_path (str): path of the template image path
label (str): the label corresponding to the template
color (List[int]): the color associated with the label (to plot detections)
matching_threshold (float): the minimum similarity score to consider an object is detected by template
matching
"""
self.image_path = image_path
self.label = label
self.color = color
self.template = cv2.imread(image_path)
self.template_height, self.template_width = self.template.shape[:2]
self.matching_threshold = matching_threshold
image = cv2.imread("PCB_reference.jpg")
templates = [
Template(image_path="T1.png", label="1", color=(0, 0, 255), matching_threshold=0.88),
Template(image_path="T2.jpg", label="2", color=(0, 255, 0,), matching_threshold=0.8),
Template(image_path="T3.png", label="3", color=(255, 0, 0,), matching_threshold=0.85),
Template(image_path="T4.png", label="4", color=(0, 180, 200,), matching_threshold=0.81),
Template(image_path="T5.png", label="5", color=(110, 180, 200,), matching_threshold=0.91),
Template(image_path="T6.png", label="6", color=(150, 100, 150,), matching_threshold=0.83),
Template(image_path="T7.png", label="7", color=(0, 100, 150,), matching_threshold=0.84),
Template(image_path="T8.png", label="8", color=(110, 100, 200,), matching_threshold=0.96),
]
detections = []
for template in templates:
template_matching = cv2.matchTemplate(template.template, image, cv2.TM_CCOEFF_NORMED)
match_locations = np.where(template_matching >= template.matching_threshold)
for (x, y) in zip(match_locations[1], match_locations[0]):
match = {
"TOP_LEFT_X": x,
"TOP_LEFT_Y": y,
"BOTTOM_RIGHT_X": x + template.template_width,
"BOTTOM_RIGHT_Y": y + template.template_height,
"MATCH_VALUE": template_matching[y, x],
"LABEL": template.label,
"COLOR": template.color
}
detections.append(match)
def compute_iou(boxA, boxB):
xA = max(boxA["TOP_LEFT_X"], boxB["TOP_LEFT_X"])
yA = max(boxA["TOP_LEFT_Y"], boxB["TOP_LEFT_Y"])
xB = min(boxA["BOTTOM_RIGHT_X"], boxB["BOTTOM_RIGHT_X"])
yB = min(boxA["BOTTOM_RIGHT_Y"], boxB["BOTTOM_RIGHT_Y"])
interArea = max(0, xB - xA + 1) * max(0, yB - yA + 1)
boxAArea = (boxA["BOTTOM_RIGHT_X"] - boxA["TOP_LEFT_X"] + 1) * (boxA["BOTTOM_RIGHT_Y"] - boxA["TOP_LEFT_Y"] + 1)
boxBArea = (boxB["BOTTOM_RIGHT_X"] - boxB["TOP_LEFT_X"] + 1) * (boxB["BOTTOM_RIGHT_Y"] - boxB["TOP_LEFT_Y"] + 1)
iou = interArea / float(boxAArea + boxBArea - interArea)
return iou
def non_max_suppression(objects, non_max_suppression_threshold=0.5, score_key="MATCH_VALUE"):
"""
Filter objects overlapping with IoU over threshold by keeping only the one with maximum score.
Args:
objects (List[dict]): a list of objects dictionaries, with:
{score_key} (float): the object score
{top_left_x} (float): the top-left x-axis coordinate of the object bounding box
{top_left_y} (float): the top-left y-axis coordinate of the object bounding box
{bottom_right_x} (float): the bottom-right x-axis coordinate of the object bounding box
{bottom_right_y} (float): the bottom-right y-axis coordinate of the object bounding box
non_max_suppression_threshold (float): the minimum IoU value used to filter overlapping boxes when
conducting non-max suppression.
score_key (str): score key in objects dicts
Returns:
List[dict]: the filtered list of dictionaries.
"""
sorted_objects = sorted(objects, key=lambda obj: obj[score_key], reverse=True)
filtered_objects = []
for object_ in sorted_objects:
overlap_found = False
for filtered_object in filtered_objects:
iou = compute_iou(object_, filtered_object)
if iou > non_max_suppression_threshold:
overlap_found = True
break
if not overlap_found:
filtered_objects.append(object_)
return filtered_objects
NMS_THRESHOLD = 0.2
detections = non_max_suppression(detections, non_max_suppression_threshold=NMS_THRESHOLD)
image_with_detections = image.copy()
for detection in detections:
cv2.rectangle(
image_with_detections,
(detection["TOP_LEFT_X"], detection["TOP_LEFT_Y"]),
(detection["BOTTOM_RIGHT_X"], detection["BOTTOM_RIGHT_Y"]),
detection["COLOR"],
2,)
cv2.imshow("res", image_with_detections)
cv2.waitKey(0)
cv2.destroyAllWindows()
This is now my output where it detects the 8 components that I have as a template:
Current code output
I wanted to make an algorithm where I could detect a missing component. For example, supposedly there are four (4) A3 resistors in the reference image but what if my input reading (real-time camera feed) has only three A3 resistors the same goes to the missing one resistor (3220). I wanted to ask what is the best approach/method to use to make this work.
Example this is my desired output:
Desired output
Should I save the position of each template where they are supposed to be placed and make a database containing the template and their specific location? This is to tell that in this location (e.g x= 20. y = 50) a resistor should be present. I am thinking of having csv file but I don't think it's possible or efficient to store the templates image directory and also the position of each template.
I have read that SSIM could be use for the missing components detection and there are also some that uses deep-learning like YOLO for neural networks but I think that is too complex for me to begin with.

How to split an image into multiple images based on white borders between them

I need to split an image into multiple images, based on the white borders between them.
for example:
output:
using Python, I don't know how to start this mission.

Here is a solution for the "easy" case where we know the grid configuration. I provide this solution even though I doubt this is what you were asked to do.
In your example image of the cat, if we are given the grid configuration, 2x2, we can do:
from PIL import Image
def subdivide(file, nx, ny):
im = Image.open(file)
wid, hgt = im.size # Size of input image
w = int(wid/nx) # Width of each subimage
h = int(hgt/ny) # Height of each subimage
for i in range(nx):
x1 = i*w # Horicontal extent...
x2 = x1+w # of subimate
for j in range(ny):
y1 = j*h # Certical extent...
y2 = y1+h # of subimate
subim = im.crop((x1, y1, x2, y2))
subim.save(f'{i}x{j}.png')
subdivide("cat.png", 2, 2)
The above will create these images:

My previous answer depended on knowing the grid configuration of the input image. This solution does not.
The main challenge is to detect where the borders are and, thus, where the rectangles that contain the images are located.
To detect the borders, we'll look for (vertical and horizontal) image lines where all pixels are "white". Since the borders in the image are not really pure white, we'll use a value less than 255 as the whiteness threshold (WHITE_THRESH in the code.)
The gist of the algorithm is in the following lines of code:
whitespace = [np.all(gray[:,i] > WHITE_THRESH) for i in range(gray.shape[1])]
Here "whitespace" is a list of Booleans that looks like
TTTTTFFFFF...FFFFFFFFTTTTTTTFFFFFFFF...FFFFTTTTT
where "T" indicates the corresponding horizontal location is part of the border (white).
We are interested in the x-locations where there are transitions between T and F. The call to the function slices(whitespace) returns a list of tuples of indices
[(x1, x2), (x1, x2), ...]
where each (x1, x2) pair indicates the xmin and xmax location of images in the x-axis direction.
The slices function finds the "edges" where there are transitions between True and False using the exclusive-or operator and then returns the locations of the transitions as a list of tuples (pairs of indices).
Similar code is used to detect the vertical location of borders and images.
The complete runnable code below takes as input the OP's image "cat.png" and:
Extracts the sub-images into 4 PNG files "fragment-0-0.png", "fragment-0-1.png", "fragment-1-0.png" and "fragment-1-1.png".
Creates a (borderless) version of the original image by pasting together the above fragments.
The runnable code and resulting images follow. The program runs in about 0.25 seconds.
from PIL import Image
import numpy as np
def slices(lst):
""" Finds the indices where lst changes value and returns them in pairs
lst is a list of booleans
"""
edges = [lst[i-1] ^ lst[i] for i in range(len(lst))]
indices = [i for i,v in enumerate(edges) if v]
pairs = [(indices[i], indices[i+1]) for i in range(0, len(indices), 2)]
return pairs
def extract(xx_locs, yy_locs, image, prefix="image"):
""" Locate and save the subimages """
data = np.asarray(image)
for i in range(len(xx_locs)):
x1,x2 = xx_locs[i]
for j in range(len(yy_locs)):
y1,y2 = yy_locs[j]
arr = data[y1:y2, x1:x2, :]
Image.fromarray(arr).save(f'{prefix}-{i}-{j}.png')
def assemble(xx_locs, yy_locs, prefix="image", result='composite'):
""" Paste the subimages into a single image and save """
wid = sum([p[1]-p[0] for p in xx_locs])
hgt = sum([p[1]-p[0] for p in yy_locs])
dst = Image.new('RGB', (wid, hgt))
x = y = 0
for i in range(len(xx_locs)):
for j in range(len(yy_locs)):
img = Image.open(f'{prefix}-{i}-{j}.png')
dst.paste(img, (x,y))
y += img.height
x += img.width
y = 0
dst.save(f'{result}.png')
WHITE_THRESH = 110 # The original image borders are not actually white
image_file = 'cat.png'
image = Image.open(image_file)
# To detect the (almost) white borders, we make a grayscale version of the image
gray = np.asarray(image.convert('L'))
# Detect location of images along the x axis
whitespace = [np.all(gray[:,i] > WHITE_THRESH) for i in range(gray.shape[1])]
xx_locs = slices(whitespace)
# Detect location of images along the y axis
whitespace = [np.all(gray[i,:] > WHITE_THRESH) for i in range(gray.shape[0])]
yy_locs = slices(whitespace)
extract(xx_locs, yy_locs, image, prefix='fragment')
assemble(xx_locs, yy_locs, prefix='fragment', result='composite')
Individual fragments:
The composite image:

how to evaluate if a rectangle fits or not in an image with some "obstacles"

As main data I've an image already converted in a list of lists of tuples(r,g,b) defined start_img,
where each list is a line (y) of pixels and each list inside y contains the actual pixel value (x) as a tuple (r,g,b)
the image is made up as follow:
Some rectangles: each one has a different value for color, width, height.
The others pixels are just rgb(0, 0, 0) pixels.
I CAN'T USE EXTERNAL LIBRARIES
1st goal
I'm looking for a way to find each rectangle already in the image and store it as x,y,h,w,r,g,b.
where (x,y) is the position of the rectangle and referes to the first top left pixel.
example
XXBBBBB
XXBBBXX
XXBBBXX
where X is a non black pixel and B stands for a black pixel
2nd goal
given a list of others rectangles I should check if i could or not fit them (one by one).
FIRST GOAL!!
in the example above there are 2 rectangles and my function should return
output:0,0,3,2,color.
5,5,2,2,color.
where for the first line 0,0 stands for (x,y), (2,2) stans for width, height, and color, if it's red for example there's should be this tuple (255,0,0).
So here I was thinking to use a nested for loop to iterate on each pixel and if it finds a color different then black it should increase the width and store the starting pixel in a dict like this
(x,y) as key and (w,h,r,g,b).
Now since it's a rectangle i can assume that if i find at (0,0),(1,0) red as color, when i'm then iterating on (1,0) if it's red it should assume that (1,1) it must be red and (1,2) must be black (as 0,2... ofc there's the rectangles are always slightly divided by at least a black pixel).
ofc this is just an idea i'm not even sure if it's a good method.
SECOND GOAL!!
For this task i'm just looking for a function that return True or False.

One way to solve this is checking neighbours recursively. This way, one can extract regions with equal color. Afterwards, one can check by the number of elements if the identified region is indeed a rectangle.
For this example I have used "X" and "Y" as color, and "B" as black, but those can be easily replaced by rgb tuples.
img_str = """
XXBBBBB
XXBBBYY
XXBBBYY
"""
# split into list of lists
img = [[letter for letter in line] for line in
img_str.strip().splitlines()]
def get_region(img_list, x, y, color, black):
""" recursive helper function to identify regions of same color """
region = []
try: # to get a pixel of suitable color
test_pixel = img_list[y][x]
if test_pixel != color:
return [] # color not suitable
except IndexError:
return [] # outside of image
img_list[y][x] = black # mask the found pixel (elsewise one would find it again later on)
region.append((x, y)) # add found pixel to region
region.extend(get_region(img_list, x + 1, y, color, black)) # look at left-side neighbour recursively
region.extend(get_region(img_list, x, y + 1, color, black)) # look at down-side neighbour recursively
return region
def get_rectangle_dim(region):
""" check if identified region is a rectangle and return dimensionality"""
min_x = min(region, key=lambda t: t[0])[0]
max_x = max(region, key=lambda t: t[0])[0]
min_y = min(region, key=lambda t: t[1])[1]
max_y = max(region, key=lambda t: t[1])[1]
width = (max_x - min_x + 1)
height = (max_y - min_y + 1)
if width * height == len(region):
return min_x, min_y, width, height
# else this returns None implicitly
potential_rectangles = []
black = "B"
# loop over rows and columns
for y in range(len(img)):
for x in range(len(img[0])):
# identify target color
if (color := img[y][x]) == black:
continue # skip black regions
if region := get_region(img, x, y, color, black):
potential_rectangles.append((region, color)) # add found regions
# filter all regions for rectangles
rectangles = []
for potential_rectangle, color in potential_rectangles:
if rectangle := get_rectangle_dim(potential_rectangle):
rectangles.append(rectangle + (color,))
# show end result
print(rectangles)

Get color from list of tuples python numpy

So I have a numpy array which I convert into a list of tuples like this
orig_img = cv2.imread("plane.jpg")
def cvimg_to_list(img: numpy.ndarray) -> list:
img_lst = []
img_row = img.shape[0] # Width
img_column = img.shape[1] # Height
for x in range(0, img_row): # (Width * Height)
for y in range(0, img_column):
img_lst.append((img[x,y][0], #B value
img[x,y][1], #G value
img[x,y][2])) #R value
return img_lst
orig_list = cvlib.cvimg_to_list(orig_img)
print(orig_list) #hundreds of thousands of values
>>> [(139, 80, 48), (135, 82, 39), ...]
Now I want to write a function generator_from_image which takes a image and returns a function which given an index for a pixel returns the color of that pixel.
The functions that are returned should look like the representation for images as one-dimensional lists. The return value for index 0 is the pixel in the upper left corner, the return value for the width of the image is the upper right corner, and so on.
Here's what I tried:
def generator_from_image(img):
def get_color_from_index(x, y):
color = (x, y) #Need help here...
return color
return get_color_from_index(img)

If what you want is to return the original pixel at (x,y) with generator_from_image taking the list of pixels as input, you will need the original image shape of the image which was (img_row, img_column, 3) for instance:
you can do:
def generator_from_image(img):
def get_color_from_index(x, y):
color = img[x*img_column + y]
return color
return get_color_from_index
What is done here is jumping x times the number of comuns img_column of the original image and adding y to reach the flattened index.
Also, instead of looping through img_row and img_column and append the pixels, you can just do: img.reshape((-1, 3)).tolist() to get a list or img.reshape((-1, 3)) to get a numpy array.

How to compare 2 list for object detection

I have made a function that detects a object on screen using opencv matchTemplate and returns its center locations as (x, y).
I want to compare the results of running the same function on 2 different objects to detect the location of one object in reference of the other. In my case the two objects are a player avatar and some bushes I want to know is the player currently standing near a bush or not.There is only one user avatar on screen therefore it only returns single (x, y) value for it but there are multiple bushes therefore multiple (x, y) values. I want a way to compare those two matrices. Sorry about the code format.
def center(base_img, needle_img, th, color):
result = cv.matchTemplate(BASE_IMG, needle_img, cv.TM_CCOEFF_NORMED)
threshold = th
yloc, xloc = np.where(result >= threshold)
w = needle_img.shape[1]
h = needle_img.shape[0]
rectangles = []
for (x, y) in zip(xloc, yloc):
rectangles.append([int(x), int(y), int(w), int(h)])
rectangles.append([int(x), int(y), int(w), int(h)])
rectangles, weights = cv.groupRectangles(rectangles, 1, 0.4)
points = []
for (x, y, w, h) in rectangles:
certre_x = x + int(w / 2)
certre_y = y + int(h / 2)
cv.drawMarker(base_img, (certre_x, certre_y), color, cv.MARKER_DIAMOND)
points.append((certre_x, certre_y))
# cv.imshow("result", base_img)
# print(points)
# return points

You can loop through the center of the bushes and get the distance to each bush sqrt(( x2-x1)**2 + ( y2-y1)**2) or you could use nearest neighbor and numpy.
Post a code snippet if it worked.