We Keep Coding

How to draw a rectangle around keypoints of a pose?

I have an image of a hand and I passed it through a pre trained hand pose estimation model and got this output. Output
Task
Now I want to draw a rectangle around the hand to carry out some tasks. How do I draw a rectangle around a hand with just using those keypoints(not using another model).
In case if you're interested to know why I need that rectangle.
I want to normalize all the points inside the rectangle to range of (0,1) by dividing each point with width and height of rectangle and top left points to 0s and bottom points to 1s.

I haven't used OpenCV in quite a while, hence the simplest approach I can think of without relying on it's methods would be to use the list of keypoint locations and find the min/max x and y values.
That is, loop through the list of points (which I assume each have a given x and y) and store the minimum x and y, as well as maximum x and y. To do this you will want to initialise the variables according to your image's size, or alternatively store all x and y in their own separate lists and perform min and max functions accordingly.
The rectangle therefore is defined by the 2 corner points of (x_min, y_min) and (x_max, y_max), which you can also extract width and height from using subtraction. Make sure your drawing reference matches up with the xy-reference of the points. To actually draw the rectangle you can refer to the code here: https://docs.opencv.org/master/dc/da5/tutorial_py_drawing_functions.html

openCV perspective transformation doesn't work as expected

I have an image with a text.
This is the image:
What I'm trying to do is to straighten the text, using perspective transformation.
The red dots in the corners are the detected boundaries.
This is more or less my code (hard-coded, for simplicity):
old_pts=np.float32([[2,41],[37,965],[1389,1121],[1389,0]])
bor=cv2.boundingRect(old_pts) #bounding_rect
ul=[bor[0], bor[1]] #upper left
ur=[bor[0], bor[1]+bor[3]] #upper right
br=[bor[0]+bor[2],bor[1]+bor[3]] #bottom right
bl=[bor[0]+bor[2],bor[1]] #bottom left
new_pts=np.float32([ul,ur,br,bl]) #new pts=[[2,0],[2,1122],[1390,1122],[1390,0]]
M = cv2.getPerspectiveTransform(old_pts,new_pts)
transformed_img = cv2.warpPerspective(new_img,M,(bor[3],bor[2])) #bor[3] and bor[4] are the bounding rect height&width.
transforemed_img=transformed_img.astype(int)
cv2.imwrite('transformed.png',transformed_img)
Now, the result I'm getting is this:
Why am I not getting a nice, straightened rectangle??
Any help will be appreciated!

Take a look at your points:
You have :
old_pts=np.float32([[2,41],[37,965],[1389,1121],[1389,0]])
and:
new_pts=np.float32([ul,ur,br,bl]) #new pts=[[2,0],[2,1122],[1390,1122],[1390,0]]
But, OpenCV manages POINTS as (x,y) values, yours are in (y,x)... I know this is confusing, since the matrix manipulation is done with (y,x) notation... The thing is that OpenCV sees the matrix manipulation as the rows and columns like a matrix, but points are seen as Cartesian coordinates...
In conclusion, try flipping the axes for the points and check the results.

Corner points from an image to redraw the outline it in Matlab or Python

I have one image. I want to get the coordinates in order so that the image can be redrawn. The image may be complex. I want to get sampled corner coordinates to redraw the outline of the image.
Attempt 1: corner function
I tried to detect the corner using corner() in matlab but it is not giving exact point. The corners are lying outside. The original image is attached below.
My code for corner detection is:
It=rgb2gray(I);
Itt=im2bw(It);
Itt1=corner(Itt);
imshow(Itt);
hold on
plot(Itt1(:,1), Itt2(:,2), 'r*');
The output of corner detection is attached below:
Problem with corner: If you zoom the image, you will find that the some corners don't lie on boundaries. So, please suggest some efficient and good working method.
Attempt 2: bwtraceboundaries
I also tried using bwtraceboundaries and corners to order the corner in terms of bwtraceboundaries output but the corner is not being exactly detected.
Question: Can you please suggest how can I detect the corner or is there any other method to extract the sampled corner points from the image so that the outline can be redrawn?

You can use bwboundaries to achieve what you want. It returns all the sorted boundary positions as a cell array:
B = bwboundaries(Itt);
imshow(Itt);
hold on
B = B{2}; % select the desired boundary (the second in this case)
plot(B(:,2), B(:,1), 'b-'); % draw a connected line
If you want to obtain only the corner points, you can filter them out manually by checking if the point lies in the middle of the next and previous point as follows:
iBremove = []; % redundant indexes in B
for k=2:length(B)-1 % loop over all the points, except the first and last one
Bk = (B(k-1, :) + B(k+1, :))/2; % interpolate the previous and next point
if all(Bk == B(k, :)) % check if point k is redundant
iBremove = [iBremove k]; % add to the list of indexes to remove
end
end
B(iBremove, :) = []; % remove the redundant points out of B
plot(B(:,2), B(:,1), 'r*-'); % draw a connected line

OpenCV - Estimating Box dimensions in Python

This is the continuation of my previous question. I now have an image like this
Here the corners are detected. Now I am trying to estimate the dimensions of the bigger box while smaller black box dimensions are known.
Can anyone guide me what is the best way to estimate the dimensions of the box? I can do it with simple Euclidean distance but I don't know if it is the correct way or not. Or even if it is the correct way then from a list of tuples (coordinates) how can I find distances like A-B or A-D or G-H but not like A-C or A-F?
The sequence has to be preserved in order to get correct dimensions. Also I have two boxes here so when I create list of corners coordinates then it should contain all coordinates from A-J and I don't know which coordinates belong to which box. So how can I preserve that for two different boxes because I want to run this code for more similar images.
Note: The corners in this image is not a single point but a set of points so I clustered the set of the corner and average them to get a single (x,y) coordinate for each corner.
I have tried my best to explain my questions. Will be extremely glad to have some answers :) Thanks.

For the
How can I find distances like A-B or A-D or G-H but not like A-C or
A-F
part
Here's a quick code, not efficient for images with lots of corners, but for your case it's OK. The idea is to start from the dilated edge image you got in your other question (with only the big box, but the idea is the same for the image where there is also the small box)
then for every possible combination of corners, you look at a few points on an imaginary line between them, and then you check if these points actually fall on a real line in the image.
import cv2
import numpy as np
#getting intermediate points on the line between point1 and point2
#for example, calling this function with (p1,p2,3) will return the point
#on the line between p1 and p2, at 1/3 distance from p2
def get_intermediate_point(p1,p2,ratio):
return [p1[0]+(p2[0]-p1[0])/ratio,p1[1]+(p2[1]-p1[1])/ratio]
#open dilated edge images
img=cv2.imread(dilated_edges,0)
#corners you got from your segmentation and other question
corners=[[29,94],[102,21],[184,52],[183,547],[101,576],[27,509]]
nb_corners=len(corners)
#intermediate points between corners you are going to test
ratios=[2,4,6,8] #in this example, the middle point, the quarter point, etc
nb_ratios=len(ratios)
#list which will contain all connected corners
connected_corners=[]
#double loop for going through all possible corners
for i in range(nb_corners-1):
for j in range(i+1,nb_corners):
cpt=0
c1=corners[i]; c2=corners[j]
#testing every intermediate points between the selected corners
for ratio in ratios:
p=get_intermediate_point(c1,c2,ratio)
#checking if these points fall on a white pixel in the image
if img[p[0],p[1]]==255:
cpt+=1
#if enough of the intermediate points fall on a white pixel
if cpt>=int(nb_ratios*0.75):
#then we assume that the 2 corners are indeed connected by a line
connected_corners.append([i,j])
print(connected_corners)

In general you cannot, since any reconstruction is only up to scale.
Basically, given a calibrated camera and 6 2D-points (6x2=12) you want to find 6 3D points + scale = 6x3+1=19. There aren't enough equations.
In order to do so, you will have to make some assumptions and insert them into the equations.
Form example:
The box edges are perpendicular to each other (which means that every 2 neighboring points share at least one coordinate value).
You need to assume that you know the height of the bottom points, i.e. they are on the same plane as your calibration box (this will give you the Z of the visible bottom points).
Hopefully, these constraints are enough to given you less equations that unknown and you can solve the linear equation set.

Using OpenCV remap function crops image

I am trying to warp an 640x360 image via the OpenCV remap function (in python 2.7). The steps executed are the following
Generate a curve and store its x and y coordinates in two seperate arrays, curve_x and curve_y.I am attaching the generated curve as an image(using pyplot):
Load image via the opencv imread function
original = cv2.imread('C:\\Users\\User\\Desktop\\alaskan-landscaps3.jpg')
Execute a nested for loop so that each pixel is shifted upwards in proportion to the height of the curve at that point.For each pixel I calculate a warping factor by dividing the distance between the curve's y coordinate and the "ceiling" (360) by the height of the image. The factor is then multiplied with the distance between the pixel's y-coordinate and the "ceiling" in order to find the new distance that the pixel must have from the "ceiling" (it will be shorter since we have an upward shift). Finally I subtract this new distance from the "ceiling" to obtain the new y-coordinate for the pixel. I thought of this formula in order to ensure that all entries in the map_y array used in the remap function will be within the area of the original image.
for i in range(0, y_size):
for j in range(0,x_size):
map_y[i][j]= y_size-((y_size - i) * ((y_size - curve_y[j]) / y_size))
map_x[i][j]=j`
Then using the remap function
warped=cv2.remap(original,map_x,map_y,cv2.INTER_LINEAR)
The resulting image appears to be warped somewhat along the curve's path but it is cropped - I am attaching both the original and resulting image
I know I must be missing something but I can't figure out where the mistake is in my code - I don't understand why since all y-coordinates in map_y are between 0-360 the top third of the image has disappeared following the remapping
Any pointers or help will be appreciated. Thanks
[EDIT:] I have edited my function as follows:
#array to store previous y-coordinate, used as a counter during mapping process
floor_y=np.zeros((x_size),np.float32)
#for each row and column of picture
for i in range(0, y_size):
for j in range(0,x_size):
#calculate distance between top of the curve at given x coordinate and top
height_above_curve = (y_size-1) - curve_y_points[j]
#calculated a mapping factor, using total height of picture and distance above curve
mapping_factor = (y_size-1)/height_above_curve
# if there was no curve at given x-coordinate then do not change the pixel coordinate
if(curve_y_points[j]==0):
map_y[i][j]=j
#if this is the first time the column is traversed, save the curve y-coordinate
elif (floor_y[j]==0):
#the pixel is translated upwards according to the height of the curve at that point
floor_y[j]=i+curve_y_points[j]
map_y[i][j]=i+curve_y_points[j] # new coordinate saved
# use a modulo operation to only translate each nth pixel where n is the mapping factor.
# the idea is that in order to fit all pixels from the original picture into a new smaller space
#(because the curve squashes the picture upwards) a number of pixels must be removed
elif ((math.floor(i % mapping_factor))==0):
#increment the "floor" counter so that the next group of pixels from the original image
#are mapped 1 pixel higher up than the previous group in the new picture
floor_y[j]=floor_y[j]+1
map_y[i][j]=floor_y[j]
else:
#for pixels that must be skipped map them all to the last pixel actually translated to the new image
map_y[i][j]=floor_y[j]
#all x-coordinates remain unchanges as we only translate pixels upwards
map_x[i][j] = j
#printout function to test mappings at x=383
for j in range(0, 360):
print('At x=383,y='+str(j)+'for curve_y_points[383]='+str(curve_y_points[383])+' and floor_y[383]='+str(floor_y[383])+' mapping is:'+str(map_y[j][383]))
The bottom line is that now the higher part of the image should not receive mappings from the lowest part so overwriting of pixels should not take place. Yet i am still getting a hugely exaggerated upwards warping effect in the picture which I cannot explain. (see new image below).The top of the curved part is at around y=140 in the original picture yet now is very close to the top i.e y around 300. There is also the question of why I am not getting a blank space at the bottom for the pixels below the curve.
I'm thinking that maybe there is also something going on with the order of rows and columns in the map_y array?

I don't think the image is being cropped. Rather, the values are "crowded" in the top-middle pixels, so that they get overwritten. Consider the following example with a simple function on a checkerboard.
import numpy as np
import cv2
import pickle
y_size=200
x_size=200
x=np.linspace(0,x_size,x_size+1)
y=(-(x-x_size/2)*(x-x_size/2))/x_size+x_size
plt.plot(x,y)
The function looks like this:
Then let's produce an image with a regular pattern.
test=np.zeros((x_size,y_size),dtype=np.float32)
for i in range(0, y_size):
for j in range(0,x_size):
if i%2 and j%2:
test[i][j]=255
cv2.imwrite('checker.png',test)
Now let's apply your shift function to that pattern:
map_y=np.zeros((x_size,y_size),dtype=np.float32)
map_x=np.zeros((x_size,y_size),dtype=np.float32)
for i in range(0, y_size):
for j in range(0,x_size):
map_y[i][j]= y_size-((y_size - i) * ((y_size - y[j]) / y_size))
map_x[i][j]=j
warped=cv2.remap(test,map_x,map_y,cv2.INTER_LINEAR)
cv2.imwrite('warped.png',warped)
If you notice, because of the shift, more than one value corresponds to the top-middle areas, which makes it look like it is cropped. But if you check to the top left and right corners of the image, notice that the values are sparser, thus the "cropping" effect does not occur much. I hope the simple example helps better to understand what is going on.