Related
I made around 40 images with a realsense camera, which gave me rgb and corresponding aligned depth images. With rs.getintrinsic() i got the intrinsic matrix of the camera. But there is still a distortion which can be seen in the pointcloud, which can be easily generated with the depth image. Here you can see it on the right side: PointCloud of a Plane in depth image
The Pointcloud represent a plane.
Now I calculated based on cv.calibrateCamera(..., intrinsic_RS_matrix, flags= cv2.CALIB_USE_INTRINSIC_GUESS|cv2.CALIB_FIX_FOCAL_LENGTH|cv2.CALIB_FIX_PRINCIPAL_POINT) the distortion coefficients of the Camera. For that I use all the 40 rgb images.
Based on the new calculated distortion I calculate with cv2.getOptimalNewCameraMatrix() the new camera matrix and with cv2.undistort(image, cameraMatrix, distCoeffs, None, newCameraMatrix) the undistorted new rgb and depth image.
Now I want to compute the pointcloud of the new undistorted depth image. But which camera Matrix should I use? The newCameraMatrix or the old one which I got from rs.getIntrinsic()?
As well I used alpha=0, so there is no cropping of the image. But if I would use alpha = 1 there would be a cropping. In that case should I use the cropped image or the uncropped one?
Here is the full Code for calculating the distortion and newCameraMatrix:
checkerboard = (6, 10)
criteria = (cv2.TERM_CRITERIA_EPS +
cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
# Vector for 3D points
threedpoints = []
# Vector for 2D points
twodpoints = []
# 3D points real world coordinates
objectp3d = np.zeros((1, checkerboard[0]*checkerboard[1], 3), np.float32)
objectp3d[0, :, :2] = np.mgrid[0:checkerboard[0], 0:checkerboard[1]].T.reshape(-1, 2)* 30
prev_img_shape = None
path = r"..."
resolution= "1280_720"
_,dates,_ = next(os.walk(path))
images = glob.glob(path)
print(len(images))
for filename in images:
image = cv2.imread(filename)
grayColor = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
# Find the chess board corners
ret, corners = cv2.findChessboardCorners(image, checkerboard, flags = cv2.CALIB_CB_ADAPTIVE_THRESH )
if ret == True :
threedpoints.append(objectp3d)
# Refining pixel coordinates for given 2d points.
corners2 = cv2.cornerSubPix(
grayColor, corners,
(11, 11),
(-1, -1), criteria)
twodpoints.append(corners2)
# Draw and display the corners
image = cv2.drawChessboardCorners(image,
checkerboard,
corners2, ret)
print("detected corners: ", len(twodpoints))
K_RS = np.load(r"path to RS intrinsic")
ret, matrix, distortion, r_vecs, t_vecs = cv2.calibrateCamera(
threedpoints, twodpoints, grayColor.shape[::-1], cameraMatrix=K_RS, distCoeffs= None, flags= cv2.CALIB_USE_INTRINSIC_GUESS|cv2.CALIB_FIX_FOCAL_LENGTH|cv2.CALIB_FIX_PRINCIPAL_POINT)# None, None)
def loadUndistortedImage(filename, cameraMatrix, distCoeffs):
image = cv2.imread(filename,-1)
# setup enlargement and offset for new image
imageShape = image.shape #image.size
imageSize = (imageShape[1],imageShape[0])
# # create a new camera matrix with the principal point offest according to the offset above
newCameraMatrix, roi = cv2.getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize,
alpha = 0, imageSize)
# create undistortion maps
R = np.array([[1,0,0],[0,1,0],[0,0,1]])
outputImage = cv2.undistort(image, cameraMatrix, distCoeffs, None, newCameraMatrix)
roi_x, roi_y, roi_w, roi_h = roi
cropped_outputImage = outputImage[roi_y : roi_y + roi_h, roi_x : roi_x + roi_w]
fixed_filename = r"..."
cv2.imwrite(fixed_filename,outputImage)
return newCameraMatrix
#Undistort the images, then save the restored images
newmatrix = loadUndistortedImage(r'...', matrix, distortion)
I would suggest to use uncropped image that has same width and length of the original images that been used for camera calibration. The cropped one will has different image shape/size.
In the image bellow, we see a defined world plane coordinate (X,Y,0) where Z=0. The camera as we can see is heading towards the defined world plane.
World reference point is located on the top left of the Grid (0,0,0). The distance between every two yellow point is 40 cm
I've calibrated my camera using the checkerboard and then used the built-in function cv2.solvePnP in order to estimate the rotation and translation vector of the camera with respect to my defined world coordinates. The results are as follows:
tvec_cam= [[-5.47884374]
[-3.08581371]
[24.15112048]]
rvec_cam= [[-0.02823308]
[ 0.08623225]
[ 0.01563199]]
According to the results, the (tx,ty,tz) seems to be right as the camera is located in the negative quarter of X,Y world-coordinates
However, i'm getting confused by interpreting the rotation vector.!
Does the resulted rotation vector say that the camera coordinates are almost aligned with the world coordinate axis (means almost no rotation!)?,
If yes how could this be true?, since according to OPENCV's camera coordinates, the Z-axis of the camera is pointing towards the scene (which means towards the world plane), the X-axis points towards the image write (which means opposite of X-world axis) and the Y-axis of the camera points towards the image bottom (which also means opposite of the Y-world axis)
Moreover, what is the unit of the tvec?
Note: I've illustrated the orientation of the defined world-coordinate axis according the the result of the translation vector (both tx and ty are negative)
the code i used for computing the rotation and translation vectors is shown below:
import cv2 as cv
import numpy as np
WPoints = np.zeros((9*3,3), np.float64)
WPoints[:,:2] = np.mgrid[0:9,0:3].T.reshape(-1,2)*0.4
imPoints=np.array([[20,143],[90,143],[161,143],[231,144],[303,144],
[374,144],[446,145],[516,146],[587,147],[18,214],[88,214]
,[159,215],[230,215],[302,216],[374,216],[446,216]
,[517,217],[588,217],[16,285],[87,285],[158,286],[229,287],
[301,288]
,[374,289],[446,289],[518,289],[589,289]],dtype=np.float64)
#load the rotation matrix [[4.38073915e+03 0.00000000e+00 1.00593352e+03]
# [0.00000000e+00 4.37829226e+03 6.97020491e+02]
# [0.00000000e+00 0.00000000e+00 1.00000000e+00]]
with np.load('parameters_cam1.npz') as X:
mtx, dist, _, _ = [X[i] for i in ('mtx','dist','rvecs','tvecs')]
ret,rvecs, tvecs = cv.solvePnP(WPoints, imPoints, mtx, dist)
np.savez("extrincic_camera1.npz",rvecs=rvecs,tvecs=tvecs)
print(tvecs)
print(rvecs)
cv.destroyAllWindows()
The code for estimating the intrinsic is show below
import numpy as np
import cv2
import glob
import argparse
import pathlib
ap = argparse.ArgumentParser()
ap.add_argument("-p", "--path", required=True, help="path to images folder")
ap.add_argument("-e", "--file_extension", required=False, default=".jpg",
help="extension of images")
args = vars(ap.parse_args())
path = args["path"] + "*" + args["file_extension"]
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
# prepare object points, like (0,0,0), (0.03,0,0), (0.06,0,0) ....,
#(0.18,0.12,0)
objp = np.zeros((5*7,3), np.float32)
objp[:,:2] = np.mgrid[0:7,0:5].T.reshape(-1,2)*0.03
#print(objp)
# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.
#images = glob.glob('left/*.jpg') #read a series of images
images = glob.glob(path)
path = 'foundContours'
#pathlib.Path(path).mkdir(parents=True, exist_ok=True)
found = 0
for fname in images:
img = cv2.imread(fname) # Capture frame-by-frame
#print(images[im_i])
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# Find the chess board corners
ret, corners = cv2.findChessboardCorners(gray, (7,5), None)
# print(corners)
# If found, add object points, image points (after refining them)
if ret == True:
print('true')
objpoints.append(objp) # Certainly, every loop objp is the same, in 3D.
#print('obj_point',objpoints)
corners2 = cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
# print(corners2)
imgpoints.append(corners2)
print(imgpoints)
print('first_point',imgpoints[0])
#print(imgpoints.shape())
# Draw and display the corners
img = cv2.drawChessboardCorners(img, (7,5), corners2, ret)
found += 1
cv2.imshow('img', img)
cv2.waitKey(1000)
# if you want to save images with detected corners
# uncomment following 2 lines and lines 5, 18 and 19
image_name = path + '/calibresult' + str(found) + '.jpg'
cv2.imwrite(image_name, img)
print("Number of images used for calibration: ", found)
# When everything done, release the capture
# cap.release()
cv2.destroyAllWindows()
#calibration
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints,
gray.shape[::-1],None,None)
#save parameters needed in undistortion
np.savez("parameters_cam1.npz",mtx=mtx,dist=dist,rvecs=rvecs,tvecs=tvecs)
np.savez("points_cam1.npz",objpoints=objpoints,imgpoints=imgpoints)
print ("Camera Matrix = |fx 0 cx|")
print (" | 0 fy cy|")
print (" | 0 0 1|")
print (mtx)
print('distortion coefficients=\n', dist)
print('rotation vector for each image=', *rvecs, sep = "\n")
print('translation vector for each image=', *tvecs, sep= "\n")
Hope you could help me understanding this
Thanks in Advance
First, tvec is a in Axis-angle representation (https://en.wikipedia.org/wiki/Axis%E2%80%93angle_representation).
You can obtain the rotation matrix using cv2.Rodrigues(). For your data, I get almost the identity:
[[ 0.99616253 -0.01682635 0.08588995]
[ 0.01439347 0.99947963 0.02886672]
[-0.08633098 -0.02751969 0.99588635]]
Now, according to the directions of x and y in your picture, the z-axis points downwards (apply carefully the right-hand rule). This explains why the z-axis of the camera is almost aligned with the z-axis of your world reference frame.
Edit: Digging a little bit further, from the code you posted:
WPoints = np.zeros((9*3,3), np.float64)
WPoints[:,:2] = np.mgrid[0:9,0:3].T.reshape(-1,2)*0.4
The values for X and Y are all positive and increment to the right and to the bottom respectively, so you are indeed using the usual convention. You are actually using X and Y incrementing to the right and down respectively and what's wrong is only the arrows you drew in the picture.
Edit Concerning the interpretation of the translation vector: in the OpenCV convention, the points in the local camera reference frame are obtained from the points in the world reference frame like this:
|x_cam| |x_world|
|y_cam| = Rmat * |y_world| + tvec
|z_cam| |z_world|
With this convention, tvec is the position of the world origin in the camera reference frame. What's more easily interpretable is the position of the camera origin in the world reference frame, which can be obtained as:
cam_center = -(tvec * R_inv)
Where R_inv is the inverse of the rotation matrix. Here the rotation matrix is almost the identity, so a quick approximation would be -tvec, which is (5.4, 3.1, -24.1).
I calibrated my camera using, separately, ROS, OpenCv and Matlab. People say that I need extrinsic parameters to calculate real distance between pixels in cm from the image. ROS does not provide extrinsic parameters explicitly, it provides a (4,3) projection matrix which is the output of multiplied intrinsic and extrinsic parameters.
That is why I again calibrated my camera using OpenCv and Matlab to get extrinsic parameters. Although I searched how can I calculate real distance in cm between pixels (i.e from (x1,y1) to (x2,y2)), I could not figure out how to calculate the real distance. Moreover, I did not understand that which parameters to use for distance calculation. I want to use OpenCv to calculate the distance between pixels and write the output as a txt file so that I can use this txt file to move my robot.
For example, here is array of pixel output sample for the path,
array([[ 4.484375 , 799.515625 ],
[ 44.484375 , 487. ],
[255.296875 , 476.68261719],
[267.99707031, 453.578125 ],
[272.484375 , 306. ],
[403.484375 , 300.515625 ],
[539.484375 , 296.515625 ],
[589.421875 , 270.00292969],
[801.109375 , 275.18554688],
[819. , 467.515625 ]])
I want to find the real distance in cm between these pixels in an order.
OpenCv Code that calculating parameters:
import numpy as np
import cv2
import glob
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
cbrow = 6
cbcol = 9
# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
objp = np.zeros((cbrow*cbcol,3), np.float32)
objp[:,:2] = np.mgrid[0:cbcol,0:cbrow].T.reshape(-1,2)
# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.
images = glob.glob('C:\Users\Ender\Desktop\CalibrationPhotos\*.jpg')
for fname in images:
img = cv2.imread(fname)
gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
# Find the chess board corners
ret, corners = cv2.findChessboardCorners(gray, (cbcol,cbrow),None)
# If found, add object points, image points (after refining them)
if ret == True:
print "%s: success" % fname
objpoints.append(objp)
cv2.cornerSubPix(gray,corners,(11,11),(-1,-1),criteria)
imgpoints.append(corners)
# Draw and display the corners
cv2.drawChessboardCorners(img, (cbcol,cbrow), corners,ret)
cv2.imshow('img',img)
cv2.waitKey(150)
else:
print "%s: failed" % fname
cv2.imshow('img',img)
cv2.waitKey(1)
cv2.destroyAllWindows()
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)
print "mtx"
print mtx
print "dist"
print dist
#print "rvecs"
#print rvecs
#print "tvecs"
#print tvecs
np.savez("CalibData.npz" ,mtx=mtx, dist=dist, rvecs=rvecs, tvecs=tvecs)
#UNDISTROTION
img = cv2.imread('C:\Users\Ender\Desktop\Maze\Maze Images\Partial Maze Images-Raw\Raw7.jpg')
h, w = img.shape[:2]
newcameramtx, roi=cv2.getOptimalNewCameraMatrix(mtx,dist,(w,h),1,(w,h))
dst = cv2.undistort(img, mtx, dist, None, newcameramtx)
# crop the image
x,y,w,h = roi
dst = dst[y:y+h, x:x+w]
cv2.imwrite('calibresult.jpg',dst)
#Re-projection Errors
total_error = 0
for i in xrange(len(objpoints)):
imgpoints2, _ = cv2.projectPoints(objpoints[i], rvecs[i], tvecs[i], mtx, dist)
error = cv2.norm(imgpoints[i],imgpoints2, cv2.NORM_L2)/len(imgpoints2)
total_error += error
print "total error: ", total_error/len(objpoints)
What is the way of doing this ?
I'm trying to implement image rectification. I was using a software which is not available anymore. To rectify the image, the software used the height of the camera (h), the distance of two points (d1, d2) from the camera and the correspond lines in the image to the reference points (Line1, Line2).
So the variables are:
h (camera elevation);
Line1, Line2 (row pixel)
d1, d2 (Distance in meters from the camera)
Configuration:
I tried to implement few code using OpenCV (Python) but the final result is not the same of the software. I wrote a code to calibrate the camera and a second to undistort the image and then I want to apply the rectification.
The problem is that I'm using a single camera (take photos of a landscape) that is fixed with a fixed focal length and focus which I can't change anymore.
Can someone tell me a good way to execute the rectification using the same way of the software or an another valid solution?
My code for the calibration is
# Numbers of corners
n_w = 9
n_h = 6
patternSize = (n_w, n_h)
# SIZE OF THE WINDOW TO IMPROVE THE COORDINATES OF CORNERS
windowSize = (11, 11)
# TERMINATION CRITERIA
criteria = (cv2.TERM_CRITERIA_EPS | cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
objp = np.zeros((n_h * n_w, 3), dtype=np.float32)
objp[:, :2] = np.mgrid[0:n_w, 0:n_h].T.reshape(-1, 2)
# LIST OF POINT
objpoints = []
imgpoints = []
# GET ALL IMAGES
images = glob.glob('*.jpg')
for fname in images:
img = cv2.imread(fname)
# IMAGE ON GRAY SACLE
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
# fIND CORNERS
retval, corners = cv2.findChessboardCorners(gray_img, patternSize, None)
if retval == True:
print 'Looping through image %s' % fname
objpoints.append(objp)
cv2.cornerSubPix(gray_img, corners, windowSize, (-1, -1), criteria)
imgpoints.append(corners)
cv2.drawChessboardCorners(img, patternSize, corners, retval)
cv2.imshow('ChessBoard Image %s' % fname, img)
cv2.waitKey(500)
cv2.destroyAllWindows()
print "------START CALIBRATION....."
ret, cameraMatrix, distCoeffs, revcs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray_img.shape[::-1],
None, None)
print ret
print cameraMatrix
print distCoeffs
print '---SAVING CALIBRATION DATA'
np.savez('calibration_data', RMS=ret, distCoeffs=distCoeffs, cameraMatrix=cameraMatrix)
if ret <= 1.0:
print '''-----GOOD CALIBRATION'''
The code to remove the distortion is:
# LOAD CALIBRATION DATA
calibrationData = np.load('calibration_data.npz')
distCoeffs = calibrationData['distCoeffs']
cameraMatrix = calibrationData['cameraMatrix']
calibrationData.close()
# LOAD IMAGES
images = glob.glob('/*.jpg')
for i, fname in enumerate(images):
img = cv2.imread(fname)
# UNDISTORT
undistorted_img = cv2.undistort(img, cameraMatrix, distCoeffs, None)
# SAVE IMAGE
cv2.imwrite(os.path.join(dirname, 'Undistorted_%05d.jpg' % i), undistorted_img)
cv2.imshow('Undistorted Image %s' % fname, undistorted_img)
The first idea to rectify the image was to find the 4 corners inside the real world image of a trapezoid (A4 paper) and compute a transformation matrix given 4 points of a rectangle (real dimension of an A4). But I think that is an wrong approce.
To do this I wrote this code:
#load image
img_Trap = cv2.imread('image.png', cv2.IMREAD_GRAYSCALE)
#points on the image (corners of an A4 paper)
ptsTrap = np.array(((1556, 1050), (1556, 1050), (2189, 1677), (1425, 1723)), dtype=np.float32)
img_Rect = cv2.imread('image2.png', cv2.IMREAD_GRAYSCALE)
# corner of a A4 (saving the aspect ratio)
ptsRect = np.array(((1980, 1381), (2189, 1381), (2189, 1677), (1980, 1677)), dtype=np.float32)
#transformation matrix
T = cv2.getPerspectiveTransform(ptsTrap, ptsRect)
print T
# warp perspective
warp = cv2.warpPerspective(img_Trap, T, img_Rect.shape[:2])
cv2.imwrite('warpimage.png', warp)
I am working on a camera calibration program using the OpenCV/Python example (from: OpenCV Tutorials) as a guidebook.
Question: How do I tailor this example code to account for the size of a square on a particular chessboard pattern? My understanding of the camera calibration process is that this information must somehow be used otherwise the values given by:
cv2.calibrateCamera()
will be incorrect.
Here is the portion of my code that reads in image files and runs through the calibration process to produce the camera matrix and other values.
#import cv2
#import numpy as np
#import glob
"""
Corner Finding
"""
# termination criteria
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
# Prepare object points, like (0,0,0), (1,0,0), ....,(6,5,0)
objp = np.zeros((5*5,3), np.float32)
objp[:,:2] = np.mgrid[0:5,0:5].T.reshape(-1,2)
# Arrays to store object points and image points from all images
objpoints = []
imgpoints = []
counting = 0
# Import Images
images = glob.glob('dir/sub dir/Images/*')
for fname in images:
img = cv2.imread(fname) # Read images
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # Convert to grayscale
# Find the chess board corners
ret, corners = cv2.findChessboardCorners(gray, (5,5), None)
# if found, add object points, image points (after refining them)
if ret == True:
objpoints.append(objp)
cv2.cornerSubPix(gray, corners, (11,11), (-1,-1), criteria)
imgpoints.append(corners)
#Draw and display corners
cv2.drawChessboardCorners(img, (5,5), corners, ret)
counting += 1
print str(counting) + ' Viable Image(s)'
cv2.imshow('img', img)
cv2.waitKey(500)
cv2.destroyAllWindows()
# Calibrate Camera
ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1],None,None)
Here, if you have your square size assume 30 mm then multiply this value with objp[:,:2]. Like this
objp[:,:2] = np.mgrid[0:9,0:6].T.reshape(-1,2)*30 # 30 mm size of square
As objp[:,:2] is a set of points of checkboard corners given as (0,0),(0,1), (0,2) ....(8,5). (0,0) point is the left upper most square corner and (8,5) is the right lowest square corner. In this case, these points have no unit but if we multiply these values with square size (for example 30 mm), then these will become (0,0),(0,30), .....(240,150) which are the real world units. Your translation vector will be in mm units in this case.
From here: https://docs.opencv.org/4.5.1/dc/dbb/tutorial_py_calibration.html
What about the 3D points from real world space? Those images are taken from a static camera and chess boards are placed at different locations and orientations. So we need to know (X,Y,Z) values. But for simplicity, we can say chess board was kept stationary at XY plane, (so Z=0 always) and camera was moved accordingly. This consideration helps us to find only X,Y values. Now for X,Y values, we can simply pass the points as (0,0), (1,0), (2,0), ... which denotes the location of points. In this case, the results we get will be in the scale of size of chess board square. But if we know the square size, (say 30 mm), we can pass the values as (0,0), (30,0), (60,0), ... . Thus, we get the results in mm. (In this case, we don't know square size since we didn't take those images, so we pass in terms of square size).