We Keep Coding

Python - Find the coordinates of the starting and ending points of the rectangle

I am currently working on a small project, but I have an unresolved problem. That is I want to draw a shape through the desired objects , The first thing is to determine the coordinates of the starting and ending points but I don't have a specific idea yet but I don't know how to do it,I hope you can give me suggestions, Glad to have your help.
i want the result in like this

Here is an example using opencv you can draw rectangle over an object:
By giving the template image of the source image you can draw the shape over the image
# importing needed libraries
import cv2
import numpy as np
img_rgb = cv2.imread(source image) # opening the source image
img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY) # converting to the gray scale
template = cv2.imread(template image,0) # opening the template image
w, h = template.shape[::-1] # giving the sape of template image
res = cv2.matchTemplate(img_gray,template,cv2.TM_CCOEFF_NORMED) # matching both the image using the opencv methods for matching object
threshold = 0.9
loc = np.where( res >= threshold)
for pt in zip(*loc[::-1]):
cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0,0,255), 1)
cv2.imshow('screen',img_rgb)
cv2.waitKey(0)
Source Image
Template Image
Result Image

Cannot use any mouse movement packages with python after using image recognition

ive been trying to use a script with cv2 and numpy to search an image for an icon, th start button for a stress test software, and that all works fine but it doesnt work together with any mouse movement packages, ive tried mouse, pyautogui, autoit and a few more with no results. the mouse movement works if i put it before the cv2 image recognition script but that wont work cause i need the coordinates.
P.S. 100, 500 is not the coordinates i want to move to, i want to move to the "coords" variable which should be equal to (pt[0] + w, pt[1] + h)
import cv2
import numpy as np
#import pyautogui
import time
import win32api
import pydirectinput
def movedamnyou():
pydirectinput.moveTo(100, 500)
#open OCCT
win32api.ShellExecute(0, "open", "D:\AppData\OCCT.exe", None, ".", 0)
#wait for program to open
time.sleep(20)
#classifier
#take screenshot
#im2 = pyautogui.screenshot('scrn.png')
# Read the main image
img_rgb = cv2.imread('scrn.png')
# Convert it to grayscale
img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)
# Read the template
template = cv2.imread('OCCTico.png', 0)
# Store width and height of template in w and h
w, h = template.shape[::-1]
# Perform match operations.
res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
# Specify a threshold
threshold = 0.8
# Store the coordinates of matched area in a numpy array
loc = np.where(res >= threshold)
# Draw a rectangle around the matched region.
for pt in zip(*loc[::-1]):
cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0, 255, 255), 2)
# Show the final image with the matched area.
cv2.imshow('Detected', img_rgb)
#print the output position
coords = (pt[0] + w, pt[1] + h)
pydirectinput.moveTo(100, 500)

import pyautogui
pyautogui.moveTo(coords[0], coords[1])

How to make a shape larger or smaller without changing the resolution of the image using OpenCV or PIL in Python

I would like to be able to make a certain shape in either a PIL image or an OpenCV image 3 times larger and smaller without changing the resolution of the image or changing the shape of the shape I want to make larger. I have tried using OpenCV's dilation method but that is not it's intended use, plus it changed the shape of the image. For an example:
Thanks.

Here's a way of doing it:
find the interesting shape, i.e. non-white ROI area
extract it
scale it up by a factor
clear the original image to white
paste the scaled ROI back into image with same centre
#!/usr/bin/env python3
import cv2
import numpy as np
if __name__ == "__main__":
# Open image
orig = cv2.imread('image.png',cv2.IMREAD_COLOR)
# Get extent of interesting part, i.e. non-white part
y, x, _ = np.nonzero(~orig)
y0, y1 = np.min(y), np.max(y) # top and bottom rows
x0, x1 = np.min(x), np.max(x) # left and right cols
h, w = y1-y0, x1-x0 # height and width
ROI = orig[y0:y1, x0:x1] # extract ROI
cv2.imwrite('ROI.png', ROI) # DEBUG only
# Upscale ROI
factor = 3
scaledROI = cv2.resize(ROI, (w*factor,h*factor), interpolation=cv2.INTER_NEAREST)
newH, newW = scaledROI.shape[:2]
# Clear original image to white
orig[:] = [255,255,255]
# Get centre of original shape, and position of top-left of ROI in output image
cx, cy = (x0 + x1) //2, (y0 + y1)//2
top = cy - newH//2
left = cx - newW//2
# Paste in rescaled ROI
orig[top:top+newH, left:left+newW] = scaledROI
cv2.imwrite('result.png', orig)
That transforms this:
to this:
Puts me in mind of a pantograph:

Group contours with the same y value

I have been following a tutorial about computer vision and doing a little project to read the time from a game. The game time is formatted h:m. So far I got the h and m figured out using findContours, but I'm having trouble isolating the colon as the character shape is not continuous. Because of this when I try to matchTemplate the code freaks out and starts to use the dot to match to all the other digits.
Are there ways to group the contours by X?
Here are simplified code to get the reference digits, the code to get digits from the screen is basically the same.
refCnts = cv2.findContours(ref.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
refCnts = imutils.grab_contours(refCnts)
refCnts = contours.sort_contours(refCnts, method="left-to-right")[0]
digits = {}
# loop over the OCR-A reference contours
for (i, c) in enumerate(refCnts):
# compute the bounding box for the digit, extract it, and resize
# it to a fixed size
(x, y, w, h) = cv2.boundingRect(c)
roi = ref[y:y + h, x:x + w]
roi = cv2.resize(roi, (10, 13))
digits[i] = roi
Im new to python and opencv. Apologies in advance if this is a dumb question.
Here is the reference image I'm using:
Here is the input image I'm trying to read:

Do you have to use findCountours? Because there are better suited methods for such problems. For instance, you can use template matching as shown below:
These are input, template (cut out from your reference image), and output images:
import cv2
import numpy as np
# Read the input image & convert to grayscale
input_rgb = cv2.imread('input.png')
input_gray = cv2.cvtColor(input_rgb, cv2.COLOR_BGR2GRAY)
# Read the template (Using 0 to read image in grayscale mode)
template = cv2.imread('template.png', 0)
# Perform template matching - more on this here: https://docs.opencv.org/4.0.1/df/dfb/group__imgproc__object.html#ga3a7850640f1fe1f58fe91a2d7583695d
res = cv2.matchTemplate(input_gray,template,cv2.TM_CCOEFF_NORMED)
# Store the coordinates of matched area
# found the threshold value of .56 using trial & error using the input image - might be different in your game
lc = np.where( res >= 0.56)
# Draw a rectangle around the matched region
# I used the width and height of the template image but in practice you need to use a better method to accomplish this
w, h = template.shape[::-1]
for pt in zip(*lc[::-1]):
cv2.rectangle(input_rgb, pt, (pt[0] + w, pt[1] + h), (0,255,255), 1)
# display output
cv2.imshow('Detected',input_rgb)
# cv2.imwrite('output.png', input_rgb)
# cv2.waitKey(0)
# cv2.destroyAllWindows()
You may also look into text detection & recognition using openCV.

Get the coordinates of the objects found by a template

I wanted to make a bot for a game that looks for a certain item on the floor, and then clicks on it. I managed to get the first part right (it even draws a rectangle around it) but whats embarrassing is that i cant get the coordinates of that object right. I use cv2.matchTemplate method. This is my code:
import numpy as np
import pyautogui
img_bgr = cv2.imread('gra.png')
img_gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)
template = cv2.imread('bones2.png', 0)
w, h = template.shape[:: -1]
res = cv2.matchTemplate(img_gray, template, cv2.TM_CCOEFF_NORMED)
threshhold = 0.90
loc = np.where( res >= threshhold)
for pt in zip(*loc[:: -1]):
cv2.rectangle(img_bgr, pt, (pt[0] + w, pt[1] + h),(0, 255, 255), 2 )
#here i wanted to move the mouse to the coordinates of a found item, however
#i cant get these two right ↓ ↓
pyautogui.moveTo( ? , ? ,duration=0.5)
cv2.imshow('znalezione', img_bgr)
cv2.waitKey()
cv2.destroyAllWindows()
I tried this:
pyautogui.moveTo( (pt[0] *2 + w)/2 , (pt[1] *2 + h)/2 ,duration=0.5)
but this doesn't work at all. Can someone explain to me what pt is at all and how to get the coordinates?
Also here is a screen shot of what i achieved so far:

From my understanding, both OpenCV and pyautogui uses the same coordinate system as illustrated with an example 1920x1080 resolution.
0,0 X increases -->
+---------------------------+
| | Y increases
| | |
| 1920 x 1080 screen | |
| | V
| |
| |
+---------------------------+ 1919, 1079
OpenCV's cv2.rectangle function takes the top left coordinates and the bottom right coordinates of the rectangle as parameters. Since you were able to draw the bounding box in your image, you have the correct coordinates of the ROI you want to examine. From the docs the moveTo function takes two parameters: x and y. Assuming you want to move the mouse to the center of the bounding box, you can do
pyautogui.moveTo(pt[0] + w/2, pt[1] + h/2, duration=0.5)

first of all. you dont need so complex calculation
x=pt[0]
y=pt[1]
center_x = x + 0.5 * w
center_y = y + 0.5 * h
In terms of points, I don't see any issues. It's not the coordinate issue. I guess high chance, It's the pyautoui function issue. But I could not verify as I cant seems to install it on my PC.
based on example
>>> pyautogui.moveTo(100, 200, 2)
Try to call the same first to rule out the last parameter issue. If can, then its simple format isse.
If can not then it might be an image conversion issue. pyautogui function is using Pillow which is giving a format that must be adapted to work with opencv. So its either datatype RGB, BGR or image coordinate issue(e.g opencv refer to image coordinate, and pyautogui uses desktop coordinate?).