We Keep Coding

Template matching a smaller (cropped) image from a larger whole image

I am attempting to template match a cropped template image from the image it was cropped from.
Here's my attempt:
import cv2
import numpy as np
def main()
img_rgb = cv2.imread('whole_image.jpg')
img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
template = cv2.imread('cropped_image_from_whole_image.jpg', 0)
res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
threshold = 0.8
for i in res:
for x in i:
# Finally check if x >= threshold (means a match?).
if x >= threshold:
print('Match found!')
if __name__ == '__main__':
main()
whole_image.jpg
cropped_image_from_whole_image.jpg
My overarching goal is to accurately check if a given template image is a image cropped from a larger whole image. If there's a match: print to standard output 'Match found!' (No GUI involved, only command line). Is the issue in how I'm handling the res/results? What am I doing wrong?

For matchTemplate the whole image and the template must have the same resolution.
In your case the whole image has 1600x1200px, the cropped image has 640x640px, while the cropped image contains the full height of the original image.
If you know the factor you can align the resolution:
import cv2
img_rgb = cv2.imread('whole_image.jpg', 0)
template = cv2.imread('cropped_image_from_whole_image.jpg', 0)
template = cv2.resize(template, (1200, 1200))
res = cv2.matchTemplate(img_rgb, template, cv2.TM_CCOEFF_NORMED)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
if max_val >= 0.8:
print('Match found!')
If not, you can try this: https://stackoverflow.com/a/5364427/18667225
Or you can search the net for something like "multi scale template matching".

How to improve template matching result?

I'd like to detect juice boxes of a certain type on a store shelf.
Example of image of shelf:
Example of image of box:
My code approach so far:
import numpy as np
import cv2
import json
from skimage.measure import label
import sys
import matplotlib.pyplot as plt
from PIL import Image
def return_boxes(img, template):
w, h = template.shape[::-1]
res_img = np.int16(img.copy())
img_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
match = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
loc = np.where(match >= 0.4)
lbl, n = label(match >= 0.4, connectivity=2, return_num=True)
lbl = np.int16([np.round(np.mean(np.argwhere(lbl == i), axis=0)) for i in range(1, n + 1)])
centers = [[pt[0]+w//2, pt[1]+h//2] for pt in lbl]
for pt in zip(*loc[::-1]):
cv2.rectangle(res_img, pt, (pt[0] + w, pt[1] + h), (0,0,255), 2)
return res_img, centers
def color(img):
return cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
def gray(img):
return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
def predict_image(img, query):
template = gray(query)
res_img, centers = return_boxes(img, template)
return res_img, centers
And the result
is far from accurate. How can I improve it? I can obviously change threshold or resize the image/template, but it does not seem universal and robust. Are there ways to improve such matching for different shelves and boxes? Some preprocessing of image perhaps.

as far as i know, template matching could only get same image with same width and length and

cv2 can't correctly find images. Sometimes cv2 find template in empty place. You cant check or controll it by cv2 methods. I think u must check results using pillow by pixels cpmpare.

Template Matching Returns Zero

I'am simply using template matching with method cv2.TM_CCOEFF_NORMED by using opencv. The difference is that both reference image and warped image are divided into small pieces for instances 128x108 resolution. Generally, it works well but sometimes it returns zero even both pieces are almost same. Below are the sample image pairs that one of them has a line with "1" intensity but all other values are zero. Is there a specific reason why it fails for this example? Maybe because of low resolution of images?
Thanks in advance.
grab_image
ref_image
import cv2
import numpy as np
np.set_printoptions(threshold='nan')
def main():
img_ref = cv2.imread('folderoftheimage')
img_grab = cv2.imread('folderoftheimage')
max_val_array = []
template_matching_array = []
# Apply template Matching
res = cv2.matchTemplate(img_ref,img_grab,cv2.TM_CCOEFF_NORMED)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
max_val_array.append(max_val)
print "min_val", min_val
print "max_val", max_val
print "min_loc", min_loc
print "max_loc", max_loc
template_matching_array.append(np.min(max_val_array))
index_min = max_val_array.index(np.min(max_val_array))
print "template matching", template_matching_array
print "zero element", index_min
main()

Opencv has found image that doesn't exist on screen

I try to use opencv for search button location on screen. If button exist on screen opencv work perfect but it return some !=0 x,y even if image doesn't exist. How to fix it?
import cv2
def buttonlocation(image):
im = ImageGrab.grab()
im.save('screenshot.png')
img = cv2.imread(image,0)
img2 = img.copy()
template = cv2.imread('screenshot.png',0)
w,h = template.shape[::-1]
meth = 'cv2.TM_SQDIFF'
img = img2.copy()
method = eval(meth)
res = cv2.matchTemplate(img,template,method)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
top_left = min_loc
x,y = top_left
return x,y

The documentation of opencv details to two steps of the template matching procedure.
R=cv2.matchTemplate(I,T,method) computes an image R. Each pixel x,y of this image represents a mark depending on the similarity between the template T and the sub-image of I starting at x,y. For instance, if the method cv.TM_SQDIFF is applied, the mark is computed as:
If R[x,y] is null, then the sub-image I[x:x+sxT,y:y+syT] is exactly identical to the template T. The smaller R[x,y] is, the closer to the template the sub-image is.
cv2.minMaxLoc(R) is applied to find the minimum of R. The corresponding subimage of I is expected to closer to the template than any other sub-image of I.
If the image I does not contain the template, the sub-image of I corresponding to the minimum of R can be very different from T. But the value of the minimum reflects this ! Indeed, a threshold on R can be applied as a way to decide whether the template is in the image or not.
Choosing the value for the threshold is a tricky task. It could be a fraction of the maximum value of R or a fraction of the mean value of R. The influence of the size of the template can be discarted by dividing R by the sxT*syT. For instance, the maximum value of R depends on the template size and the type of the image. For instance, for CV_8UC3 (unsigned char, 3 channels) the maximum value of R is 255*3*sxT*syT.
Here is an example:
import cv2
img = cv2.imread('image.jpg',eval('cv2.CV_LOAD_IMAGE_COLOR'))
template = cv2.imread('template.jpg',eval('cv2.CV_LOAD_IMAGE_COLOR'))
cv2.imshow('image',img)
#cv2.waitKey(0)
#cv2.destroyAllWindows()
meth = 'cv2.TM_SQDIFF'
method = eval(meth)
res = cv2.matchTemplate(img,template,method)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
top_left = min_loc
x,y = top_left
h,w,c=template.shape
print 'R='+str( min_val)
if min_val< h*w*3*(20*20):
cv2.rectangle(img,min_loc,(min_loc[0] + w,min_loc[1] + h),(0,255,0),3)
else:
print 'first template not found'
template = cv2.imread('template2.jpg',eval('cv2.CV_LOAD_IMAGE_COLOR'))
res = cv2.matchTemplate(img,template,method)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
top_left = min_loc
x,y = top_left
h,w,c=template.shape
print 'R='+str( min_val)
if min_val< h*w*3*(20*20):
cv2.rectangle(img,min_loc,(min_loc[0] + w,min_loc[1] + h),(0,0,255),3)
else:
print 'second template not found'
cv2.imwrite( "result.jpg", img);
cv2.namedWindow('res',0)
cv2.imshow('res',img)
cv2.waitKey(0)
cv2.destroyAllWindows()
The image:
The first template is to be found:
The second template is not to be found:
The result:

python opencv cv2 matchTemplate with transparency

OpenCV 3.0.0 added the ability to specify a mask while performing templateMatch. When I specify a mask I get this error: error: (-215) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims() <= 2 in function matchTemplateMask
Template image (PNG with transparency):
Source image:
Code
# read the template emoji with the alpha channel
template = cv2.imread(imagePath, cv2.IMREAD_UNCHANGED)
channels = cv2.split(template)
zero_channel = np.zeros_like(channels[0])
mask = np.array(channels[3])
# all elements in alpha_channel that have value 0 are set to 1 in the mask matrix
mask[channels[3] == 0] = 1
# all elements in alpha_channel that have value 100 are set to 0 in the mask matrix
mask[channels[3] == 100] = 0
transparent_mask = cv2.merge([zero_channel, zero_channel, zero_channel, mask])
print image.shape, image.dtype # (72, 232, 3) uint8
print template.shape, template.dtype # (40, 40, 4) uint8
print transparent_mask.shape, transparent_mask.dtype # (40, 40, 4) uint8
# find the matches
res = cv2.matchTemplate(image, template, cv2.TM_CCOEFF_NORMED, mask=transparent_mask)
Is something wrong with the image type? I am unable to find any examples (in Python) using the new mask parameter of the matchTemplate method. Does anyone know how to create the mask?

I was able to get this to work using Python 2.7.13 and opencv-python==3.1.0.4
Here is the code for it.
import cv2
import numpy as np
import sys
if len(sys.argv) < 3:
print 'Usage: python match.py <template.png> <image.png>'
sys.exit()
template_path = sys.argv[1]
template = cv2.imread(template_path, cv2.IMREAD_UNCHANGED)
channels = cv2.split(template)
zero_channel = np.zeros_like(channels[0])
mask = np.array(channels[3])
image_path = sys.argv[2]
image = cv2.imread(image_path, cv2.IMREAD_UNCHANGED)
mask[channels[3] == 0] = 1
mask[channels[3] == 100] = 0
# transparent_mask = None
# According to http://www.devsplanet.com/question/35658323, we can only use
# cv2.TM_SQDIFF or cv2.TM_CCORR_NORMED
# All methods can be seen here:
# http://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html#which-are-the-matching-methods-available-in-opencv
method = cv2.TM_SQDIFF # R(x,y) = \sum _{x',y'} (T(x',y')-I(x+x',y+y'))^2 (essentially, sum of squared differences)
transparent_mask = cv2.merge([zero_channel, zero_channel, zero_channel, mask])
result = cv2.matchTemplate(image, template, method, mask=transparent_mask)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)
print 'Lowest squared difference WITH mask', min_val
# Now we'll try it without the mask (should give a much larger error)
transparent_mask = None
result = cv2.matchTemplate(image, template, method, mask=transparent_mask)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(result)
print 'Lowest squared difference WITHOUT mask', min_val
Here it is as a gist.
Essentially, you need to make sure you're using the right matching method.

My environment is using opencv 3.1.0 and python 2.7.11.
Here is the code that is looking for images in another image where the template is using transparency (alpha channel). I hope this can help you.
def getMultiFullInfo(all_matches,w,h):
#This function will rearrange the data and calculate the tuple
# for the square and the center and the tolerance for each point
result = []
for match in all_matches:
tlx = match[0]
tly = match[1]
top_left = (tlx,tly)
brx = match[0] + w
bry = match[1] + h
bottom_right = (brx,bry)
centerx = match[0] + w/2
centery = match[1] + h/2
center = [centerx,centery]
result.append({'top_left':top_left,'bottom_right':bottom_right,'center':center,'tolerance':match[2]})
return result
def getMulti(res, tolerance,w,h):
#We get an opencv image in the form of a numpy array and we need to
# find all the occurances in there knowing that 2 squares cannot intersect
#This will give us exactly the matches that are unique
#First we need to get all the points where value is >= tolerance
#This wil get sometimes some squares that vary only from some pixels and that are overlapping
all_matches_full = np.where (res >= tolerance)
logging.debug('*************Start of getMulti function')
logging.debug('All >= tolerance')
logging.debug(all_matches_full)
#Now we need to arrange it in x,y coordinates
all_matches_coords = []
for pt in zip(*all_matches_full[::-1]):
all_matches_coords.append([pt[0],pt[1],res[pt[1]][pt[0]]])
logging.debug('In coords form')
logging.debug(all_matches_coords)
#Let's sort the new array
all_matches_coords = sorted(all_matches_coords)
logging.debug('Sorted')
logging.debug(all_matches_coords)
#This function will be called only when there is at least one match so if matchtemplate returns something
#This means we have found at least one record so we can prepare the analysis and loop through each records
all_matches = [[all_matches_coords[0][0],all_matches_coords[0][1],all_matches_coords[0][2]]]
i=1
for pt in all_matches_coords:
found_in_existing = False
logging.debug('%s)',i)
for match in all_matches:
logging.debug(match)
#This is the test to make sure that the square we analyse doesn't overlap with one of the squares already found
if pt[0] >= (match[0]-w) and pt[0] <= (match[0]+w) and pt[1] >= (match[1]-h) and pt[1] <= (match[1]+h):
found_in_existing = True
if pt[2] > match[2]:
match[0] = pt[0]
match[1] = pt[1]
match[2] = res[pt[1]][pt[0]]
if not found_in_existing:
all_matches.append([pt[0],pt[1],res[pt[1]][pt[0]]])
i += 1
logging.debug('Final')
logging.debug(all_matches)
logging.debug('Final with all info')
#Before returning the result, we will arrange it with data easily accessible
all_matches = getMultiFullInfo(all_matches,w,h)
logging.debug(all_matches)
logging.debug('*************End of getMulti function')
return all_matches
def checkPicture(screenshot,templateFile, tolerance, multiple = False):
#This is an intermediary function so that the actual function doesn't include too much specific arguments
#We open the config file
configFile = 'test.cfg'
config = SafeConfigParser()
config.read(configFile)
basepics_dir = config.get('general', 'basepics_dir')
debug_dir = config.get('general', 'debug_dir')
font = cv2.FONT_HERSHEY_PLAIN
#The value -1 means we keep the file as is meaning with color and alpha channel if any
# btw, 0 means grayscale and 1 is color
template = cv2.imread(basepics_dir+templateFile,-1)
#Now we search in the picture
result = findPicture(screenshot,template, tolerance, multiple)
#If it didn't get any result, we log the best value
if not result['res']:
logging.debug('Best value found for %s is: %f',templateFile,result['best_val'])
elif logging.getLogger().getEffectiveLevel() == 10:
screenshot_with_rectangle = screenshot.copy()
for pt in result['points']:
cv2.rectangle(screenshot_with_rectangle, pt['top_left'], pt['bottom_right'], 255, 2)
fileName_top_left = (pt['top_left'][0],pt['top_left'][1]-10)
cv2.putText(screenshot_with_rectangle,str(pt['tolerance'])[:4],fileName_top_left, font, 1,(255,255,255),2)
#Now we save to the file if needed
filename = time.strftime("%Y%m%d-%H%M%S") + '_' + templateFile[:-4] + '.jpg'
cv2.imwrite(debug_dir + filename, screenshot_with_rectangle)
result['name']=templateFile
return result
def extractAlpha(img, hardedge = True):
if img.shape[2]>3:
logging.debug('Mask detected')
channels = cv2.split(img)
mask = np.array(channels[3])
if hardedge:
for idx in xrange(len(mask[0])):
if mask[0][idx] <=128:
mask[0][idx] = 0
else:
mask[0][idx] = 255
mask = cv2.cvtColor(mask, cv2.COLOR_GRAY2BGR)
img = cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)
return {'res':True,'image':img,'mask':mask}
else:
return {'res':False,'image':img}
def findPicture(screenshot,template, tolerance, multiple = False):
#This function will work with color images 3 channels minimum
#The template can have an alpha channel and we will extract it to have the mask
logging.debug('Looking for %s' , template)
logging.debug('Tolerance to check is %f' , tolerance)
logging.debug('*************Start of checkPicture')
h = template.shape[0]
w = template.shape[1]
#We will now extract the alpha channel
tmpl = extractAlpha(template)
logging.debug('Image width: %d - Image heigth: %d',w,h)
# the method used for comparison, can be ['cv2.TM_CCOEFF', 'cv2.TM_CCOEFF_NORMED', 'cv2.TM_CCORR','cv2.TM_CCORR_NORMED', 'cv2.TM_SQDIFF', 'cv2.TM_SQDIFF_NORMED']
meth = 'cv2.TM_CCORR_NORMED'
method = eval(meth)
# Apply template Matching
if tmpl['res']:
res = cv2.matchTemplate(screenshot,tmpl['image'],method, mask = tmpl['mask'])
else:
res = cv2.matchTemplate(screenshot,tmpl['image'],method)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
# If the method is TM_SQDIFF or TM_SQDIFF_NORMED, take minimum
if method in [cv2.TM_SQDIFF, cv2.TM_SQDIFF_NORMED]:
top_left = min_loc
best_val = 1 - min_val
else:
top_left = max_loc
best_val = max_val
#We need to ensure we found at least one match otherwise we return false
if best_val >= tolerance:
if multiple:
#We need to find all the time the image is found
all_matches = getMulti(res, float(tolerance),int(w),int(h))
else:
bottom_right = (top_left[0] + w, top_left[1] + h)
center = (top_left[0] + (w/2), top_left[1] + (h/2))
all_matches = [{'top_left':top_left,'bottom_right':bottom_right,'center':center,'tolerance':best_val}]
#point will be in the form: [{'tolerance': 0.9889718890190125, 'center': (470, 193), 'bottom_right': (597, 215), 'top_left': (343, 172)}]
logging.debug('The points found will be:')
logging.debug(all_matches)
logging.debug('*************End of checkPicture')
return {'res': True,'points':all_matches}
else:
logging.debug('Could not find a value above tolerance')
logging.debug('*************End of checkPicture')
return {'res': False,'best_val':best_val}

In OpenCV 4.2.0 the first two suggested codes result for me in the following error:
cv2.error: OpenCV(4.2.0) C:\projects\opencv-python\opencv\modules\imgproc\src\templmatch.cpp:766: error: (-215:Assertion failed) (depth == CV_8U || depth == CV_32F) && type == _templ.type() && _img.dims() <= 2 in function 'cv::matchTemplateMask'
It looks like things have got much easier in the meantime. Here is my Python code that I tried to reduce at its maximum. The file "crowncap_85x85_mask.png" is a black and white image. All black pixels in the mask will be ignored during matching.
Still only the matching methods TM_SQDIFF and TM_CCORR_NORMED are supported when using a mask.
import cv2 as cv
img = cv.imread("fridge_zoomed.png", cv.IMREAD_COLOR)
templ = cv.imread("crowncap_85x85.png", cv.IMREAD_COLOR)
mask = cv.imread( "crowncap_85x85_mask.png", cv.IMREAD_COLOR )
result = cv.matchTemplate(img, templ, cv.TM_CCORR_NORMED, None, mask)
cv.imshow("Matching with mask", result)
min_val, max_val, min_loc, max_loc = cv.minMaxLoc(result)
print('Highest correlation WITH mask', max_val)
result = cv.matchTemplate(img, templ, cv.TM_CCORR_NORMED)
cv.imshow("Matching without mask", result)
min_val, max_val, min_loc, max_loc = cv.minMaxLoc(result)
print('Highest correlation without mask', max_val)
while True:
if cv.waitKey(10) == 27:
break
cv.destroyAllWindows()
If you want to generate the mask from the alpha channel of your template, you can proceed as follows:
import cv2 as cv
import numpy as np
img = cv.imread("fridge_zoomed.png", cv.IMREAD_COLOR)
templ = cv.imread("crowncap_85x85_transp.png", cv.IMREAD_COLOR)
templ_incl_alpha_ch = cv.imread("crowncap_85x85_transp.png", cv.IMREAD_UNCHANGED)
channels = cv.split(templ_incl_alpha_ch)
#extract "transparency" channel from image
alpha_channel = np.array(channels[3])
#generate mask image, all black dots will be ignored during matching
mask = cv.merge([alpha_channel,alpha_channel,alpha_channel])
cv.imshow("Mask", mask)
result = cv.matchTemplate(img, templ, cv.TM_CCORR_NORMED, None, mask)
cv.imshow("Matching with mask", result)
min_val, max_val, min_loc, max_loc = cv.minMaxLoc(result)
print('Highest correlation WITH mask', max_val)
result = cv.matchTemplate(img, templ, cv.TM_CCORR_NORMED)
cv.imshow("Matching without mask", result)
min_val, max_val, min_loc, max_loc = cv.minMaxLoc(result)
print('Highest correlation without mask', max_val)
while True:
if cv.waitKey(10) == 27:
break
cv.destroyAllWindows()

I have now got the whole thing running in Python 3.9, I have already created a new question here that explicitly refers to Python 3.x using opencv-python:
Template Matching with Python 3.9.1, opencv.python 4.5.1.48 and mask (transparency).
The graphics I have taken from this question, because I have also worked privately with these to test.