We Keep Coding

Finding 3d-poses with solvePnP (python, opencv) is off. why?

I calibrated my camera following this tutorial, using 20 images of a point pattern.
The drawn point centers look suitable, however the reprojection error I obtain is 11.5 pixel -which seems large to me? No subpixeling is done yet.
Next, I am using the same images with the calibration data from above to find the poses of the point pattern, using solvePnP-function.
Here, as shown in the following pictures - it seems as if the center is always found correctly, however the drawn tripod is off - its ends should correspond to
(1,0,0), (0,1,0) and (0,0,-1)
respectively.
My question is - why is the tripod off randomly - I would be happy about any hint.
Thanks
unfortunately not having any rep - i cant post pictures here. Thus just links...
img 1
img 2
img 3
img 4
img 5
Update:
It seems to be a problem with using solvePnP:
I reprojected all the objectpoints during calibration at their found position - which looks good:
calibration
However, when using solvePnP different rvecs and tvecs are returned, resulting in wrong projections of the object points.
solvePnP
Any thoughts are welcome ;-)
Here the code how solvePnP is used:
#gray is a grayvalue img of calib plate
#objp is an array of floats containing objpoints
#camera matrix and dist. coeff are imported from previous calibration
axis = np.float32([[1,0,0], [8,0,0], [0,0,-1]])
shape = (4,11)
ret, centers = cv2.findCirclesGrid( \
gray, shape, flags=cv2.CALIB_CB_ASYMMETRIC_GRID)
if ret == True:
# Find the rotation and translation vectors.
ret, rvecs, tvecs = cv2.solvePnP(objp, centers, camera_matrix,
distortion_coefficients)
# project 3D points to image plane
imgpts, jac = cv2.projectPoints(axis, rvecs, tvecs, camera_matrix,
distortion_coefficients)

I think the problem is in distortion coefficients. Probably, these coefficients differ to much for images, which should not be. Try to calibrate your camera with this tutorial :
https://learnopencv.com/camera-calibration-using-opencv/

Fisheye calibration OpenCV Python

I am using a fisheye camera and I would like to calibrate it and correct its barrel distortion using OpenCV or any other library in Python. I have tried different methods using the classic chessboard pattern through different images.
I have been following this approach:
# Prepare object points, like (0,0,0), (1,0,0), (2,0,0), ...,(7,5,0)
object_p = np.zeros((self.__ny*self.__nx,3),np.float32)
object_p[:,:2] = np.mgrid[0:self.__nx,0:self.__ny].T.reshape(-1,2) # s,y coordinates
for image_filename in calibration_filenames:
# Read in each image
image = cv2.imread(image_filename)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # RGB is standard in matlibplot
# Convert to grayscale
gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
# Find the chessboard corners
ret, corners = cv2.findChessboardCorners(gray, (self.__nx, self.__ny), None)
# If found, draw corners
if ret == True:
# Store the corners found in the current image
object_points.append(object_p) # how it should look like
image_points.append(corners) # how it looks like
# Draw and display the corners
cv2.drawChessboardCorners(image, (self.__nx, self.__ny), corners, ret)
plt.figure()
plt.imshow(image)
plt.show()
# Do the calibration
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(object_points, image_points, shape, None, None)
And then this to correct my image:
cv2.undistort(image, mtx, dist, None, mtx)
The main problem is that, though correction performs good with some images, is very susceptible to camera location. A variation of cm in the camera will result in very different corrections (see below).
Any idea of how could I control this zoom effect to show maintain the grid area always in my corrected image?
I am aware that this question may be similar to the one in here.

Your calibration is poorly estimated: as you can see, straight lines in the scene grid are not transformed to straight lines in the (allegedly) undistorted image.
Once you have a better calibration, you can compute (using the inverse mapping) the location of the 4 sides of the undistorted image in the original ones. These define your usable image boundaries, and can guide the placement of the camera with respect to the scene.

Classification on the basis of shape and size

How to classify objects on the basis of shape and size using machine learning?
Say I've an circle and some small dotted squares in an image. The difference between the two is their shape and size. So given an image, how to distinguish between these objects and return the result.
In the actual problem those objects are hot spots in a solar PV folder, which are defected parts of it. I need to classify them.The I/P image is as:
[This is less or more a square type of hot spot:]
https://i.stack.imgur.com/4JL7E.png

This answer doesn't ellaborate machine learning or any approach using classifiers
The circles can be detected by the Hough Circle
Transform
from OpenCV cv2.HoughCircles():
Documentation for Hough Circles in OpenCV
Note: By using the radius you can tune the shape size of the circles you want to detect. And to be honest I didn't really get what dotted squares are, maybe you could show an exemplary image in your question.
If there are only two different kinds of objects in the image you probably don't even need a classifier, because the two classes are already being separated by the subsequent image processing (that though depends highly on your input images).
import cv2
import numpy as np
img = cv2.imread('opencv_logo.png',0)
img = cv2.medianBlur(img,5)
cimg = cv2.cvtColor(img,cv2.COLOR_GRAY2BGR)
circles = cv2.HoughCircles(img,cv2.HOUGH_GRADIENT,1,20,
param1=50,param2=30,minRadius=0,maxRadius=0)
circles = np.uint16(np.around(circles))
for i in circles[0,:]:
# draw the outer circle
cv2.circle(cimg,(i[0],i[1]),i[2],(0,255,0),2)
# draw the center of the circle
cv2.circle(cimg,(i[0],i[1]),2,(0,0,255),3)
cv2.imshow('detected circles',cimg)
cv2.waitKey(0)
cv2.destroyAllWindows()
Result of the code is as follows

Camera Calibration with OpenCV - How to adjust chessboard square size?

I am working on a camera calibration program using the OpenCV/Python example (from: OpenCV Tutorials) as a guidebook.
Question: How do I tailor this example code to account for the size of a square on a particular chessboard pattern? My understanding of the camera calibration process is that this information must somehow be used otherwise the values given by:
cv2.calibrateCamera()
will be incorrect.
Here is the portion of my code that reads in image files and runs through the calibration process to produce the camera matrix and other values.
#import cv2
#import numpy as np
#import glob
"""
Corner Finding
"""
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
# Prepare object points, like (0,0,0), (1,0,0), ....,(6,5,0)
objp = np.zeros((5*5,3), np.float32)
objp[:,:2] = np.mgrid[0:5,0:5].T.reshape(-1,2)
# Arrays to store object points and image points from all images
objpoints = []
imgpoints = []
counting = 0
# Import Images
images = glob.glob('dir/sub dir/Images/*')
for fname in images:
img = cv2.imread(fname) # Read images
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # Convert to grayscale
# Find the chess board corners
ret, corners = cv2.findChessboardCorners(gray, (5,5), None)
# if found, add object points, image points (after refining them)
if ret == True:
objpoints.append(objp)
cv2.cornerSubPix(gray, corners, (11,11), (-1,-1), criteria)
imgpoints.append(corners)
#Draw and display corners
cv2.drawChessboardCorners(img, (5,5), corners, ret)
counting += 1
print str(counting) + ' Viable Image(s)'
cv2.imshow('img', img)
cv2.waitKey(500)
cv2.destroyAllWindows()
# Calibrate Camera
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)

Here, if you have your square size assume 30 mm then multiply this value with objp[:,:2]. Like this
objp[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)*30 # 30 mm size of square
As objp[:,:2] is a set of points of checkboard corners given as (0,0),(0,1), (0,2) ....(8,5). (0,0) point is the left upper most square corner and (8,5) is the right lowest square corner. In this case, these points have no unit but if we multiply these values with square size (for example 30 mm), then these will become (0,0),(0,30), .....(240,150) which are the real world units. Your translation vector will be in mm units in this case.

From here: https://docs.opencv.org/4.5.1/dc/dbb/tutorial_py_calibration.html
What about the 3D points from real world space? Those images are taken from a static camera and chess boards are placed at different locations and orientations. So we need to know (X,Y,Z) values. But for simplicity, we can say chess board was kept stationary at XY plane, (so Z=0 always) and camera was moved accordingly. This consideration helps us to find only X,Y values. Now for X,Y values, we can simply pass the points as (0,0), (1,0), (2,0), ... which denotes the location of points. In this case, the results we get will be in the scale of size of chess board square. But if we know the square size, (say 30 mm), we can pass the values as (0,0), (30,0), (60,0), ... . Thus, we get the results in mm. (In this case, we don't know square size since we didn't take those images, so we pass in terms of square size).

Obtaining the equation of the straight line projecting onto a given pixel in an image with opencv

I am trying to use Opencv with Python 2.7 to obtain world coordinate informations from images. I correctly performed the calibration of the camera (see description below), and obtained the camera pose in world coordinate, including the coordinates of the (virtual) pinhole of the camera. What I now need is a function f(pixel) which takes a pixel [x,y] of the image as an input argument, and returns the equation of the straight line (in world coordinates) that projects onto the given pixel (it is the same as finding the world coordinates of a point on the straight line, since I already have the coordinates of the pinhole). Help will be greatly appreciated. Below are the description of the steps I already performed:
EDIT:
This question may appear to be a duplicate, but it is a fact that I haven't seen a detailed solution of this problem in this forum.
If my understanding is good, the camera axes system is parallel to the axes of the image, and its third axis is pointing toward the view of the camera. So, it is clear that the problem amounts to the following question, interesting by itself:
Where is located the centre of the image with respect to the camera axes system (i.e need one translation vector), and what is the physical length of one pixel ?
STEPS PERFORMED:
Playing with a chess board in front of the camera, I first obtained the intrinsic parameters matrix, and the distortion parameters (see the tutorial in opencv-python):
(ret, mtx, dist, rvecs, tvecs) = cv2.calibrateCamera(objpoints, imgpoints, grayImg.shape[::-1],None,None)
Then I obtained the camera pause (extrinsic parameter matrix) from one image of the chess board, using the following piece of code:
[ret, corners] = cv2.findChessboardCorners(grayImg, [7,7], None)
corners2 = cv2.cornerSubPix(grayImg, corners, (11,11), (-1,-1), criteria)
[ret, rvec, tvec] = cv2.solvePnP(objp, corners2, mtx, dist)
Rmat = cv2.Rodrigues(rvec)[0] #rotation matrix of the camera pause
cameraPinhole = np.dot(-Rmat.T, tvec) #location of the virtual pinhole of the camera with respect to world coordinates
Then I obtained the projection of the world coordinate axes (located at a corner of the chess board) on the image, and use them to plot theses axes on the image:
axes = chess_board_square_sz * np.float32([[3,0,0], [0,3,0], [0,0,-3]]).reshape(-1,3)
[axes_proj, jac] = cv2.projectPoints(axes, rvec, tvec, newcameramtx, dist)
img = draw_axes(img, corners2, axes_coor)
What should be the next steps ?