I am trying to get 3D points using cv2.triangulatePoints but it always returns almost same Z value. My output looks like this: As it seen, all points are in almost same Z value. There is no depth.
Here is my triangulation:
def triangulate(self, proj_mat1, pts0, pts1):
proj_mat0 = np.zeros((3,4))
proj_mat0[:, :3] = np.eye(3)
pts0, pts1 = self.normalize(pts0), self.normalize(pts1)
pts4d = cv2.triangulatePoints(proj_mat0, proj_mat1, pts0.T, pts1.T).T
pts4d /= pts4d[:, 3:]
out = np.delete(pts4d, 3, 1)
print(out)
return out
Here is my projection matrix calculation:
def getP(self, rmat, tvec):
P = np.concatenate([rmat, tvec.reshape(3, 1)], axis = 1)
return P
Here is the part that I get rmat, tvec and call triangulation:
E, mask = cv2.findEssentialMat(np.array(aa), np.array(bb), self.K)
_, R, t, mask = cv2.recoverPose(E, np.array(aa), np.array(bb), self.K)
proj_mat1 = self.getP(R, t)
out = self.triangulate(proj_mat1, np.array(aa, dtype = np.float32), np.array(bb, dtype = np.float32))
My camera matrix:
array([[787.8113353 , 0. , 318.49905794],
[ 0. , 786.9638204 , 245.98673477],
[ 0. , 0. , 1. ]])
My projection matrix 1:
array([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.]])
Explanations:
aa and bb are matched points from 2 frames.
self.K is my camera matrix
rotation and translation matrices are extracted from Essential matrix
Essential matrix calculated from matched keypoints. It changes every frame.
Projection matrix 2 changes every frame.
Output after changing first projection matrix (I switched from matplotlib to pangolin as 3D visualization tool):
Output after using P1 and P2 that I mentioned in comments:
Where is my mistake? Please let me know if any further information needed. I will update my question.
Unfortunately I don't have the possibility to double-check directly but my gut feeling is that the issues you are facing are essentially due to the choice of your first projection matrix
I did some research and I found this great paper with both theory and practice. Despite differing a little bit from your approach, there is a thing that is worth saying
If you check carefully, the first projection matrix is exactly the camera matrix with an additional last column equal to zero. In fact, the rotation matrix for the first camera reduces to the identity matrix and the corresponding translation vector is a null vector, so using this general formula:
P = KT
where P is the projection matrix, K the camera matrix and T the matrix obtained by the rotation matrix R flanked by the translation vector t according to:
T = [R|t]
then you will get:
Coming back to your case, first of all I would suggest to change your first projection matrix as just said
Also, I understand that you are planned to work with something different at every frame but if after the suggested change the things still don't match then in your shoes I'd start working with just 2 images [I think you implicitly did already to create the correspondence between aa and bb], calculating first the matrices with your algorithm and then checking with the ones obtained following the article above
In this way you would be able to understand/debug which matrices are creating you troubles
Thank you so much for all the effort #Antonino. My webcams were pretty bad. After changing every part of my code and making many trials I decided to change my webcams and bought good webcams. It worked :D Here is the result:
Related
I have a 3D image of a house stored in a csv file. I would like to know how a camera with f = 400 pix, 640 x 480, would capture photos in different poses. I can display the image in the csv(first image), but I would like to obtain something like the second image.
import pandas as pd
import numpy as np
import cv2 as cv
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
#load data
data1 = pd.read_csv('house.csv',sep=' ')
x = data1.drop(data1.columns[[0,2,3]], axis=1)
y = data1.drop(data1.columns[[0,1,3]], axis=1)
z = data1.drop(data1.columns[[1,2,3]], axis=1)
fig = plt.figure()
ax1 = fig.add_subplot(111,projection='3d')
# Hide grid lines
plt.grid(b=None)
ax1.scatter(x,y,z)
plt.axis('off')
plt.savefig('house.png')
Heres the images i have:
this is a computer graphics problem. it involves matrix multiplication to transform those points until they are on your image plane. I would suggest that you look up some basic computer graphics math (e.g. OpenGL).
OpenCV has procedures such as cv::projectPoints() to handle some of these steps for you. It's not a computer graphics library however.
I'll give you the broad strokes of the math involved:
first you need translation and rotation matrices to move your camera around the scene. these matrices are all 4x4, and your 3D points will be represented as (x,y,z,1) vectors. the extra coordinate makes translation (and more) possible. you multiply the matrices together into one, then apply this one to your points. this transformation moves all points from world space into camera space (the space around the camera moves).
a simple translation by 5 units in +Z would be:
>>> T = np.eye(4)
>>> T[0:3,3] = (0, 0, +5)
>>> T
array([[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 5.],
[0., 0., 0., 1.]])
homogenization: check that your points still have that 1 in the last place. if not, divide the point/vector by what's in the fourth coordinate. "the point" is all vectors (x,y,z,1)*w for any w, and (x,y,z,1) is the canonical representation.
now the projection matrix. computer graphics uses a 4x4 matrix here too but I'm not experienced with that formulation. I'll instead uses a 3x3 matrix commonly used in OpenCV. it takes your points as (x,y,z) vectors in camera space, and outputs (x,y,1)*w vectors in screen space (yes, homogenization again).
the camera matrix for a 640x480 camera and 60 degrees horizontal field of view would be:
>>> M = np.eye(3)
>>> M[0:2,2] = (640/2, 480/2)
>>> M[0,0] = M[1,1] = (640/2) / atan(60/2 * pi/180)
>>> M
array([[663.42156, 0. , 320. ],
[ 0. , 663.42156, 240. ],
[ 0. , 0. , 1. ]])
now you have your points in screen coordinates. draw the points alone or draw lines between them.
Sorry if the title is confusing, but it is very hard to put what I would like to do in a single sentence. Image you have an image stack stack in the form of N m x n matrices as a numpy array in the shape of (m, n, N). Now, if I want to perform the numpy.median for example along the stack axis N it is very easy: numpy.median(stack, 0). The problem is that for each image of the stack, I also have a mask of pixels that I would not like to include in the operation, in this case numpy.median. Is there any efficient way to do that?
So far, all I could think of is this, but it is increadibly slow and absolutely not feasible:
median = [[]]*images[0].flatten().shape
for i in range(len(images)):
image = images[i].flatten()
mask = mask[i].flatten()
for j in range(len(median)):
if mask[j] == 0:
median[j].append(image[j])
for i in range(len(median)):
median[j] = np.median(median[j]) if median[j] else 0
median = np.array(median).reshape(images[0].shape)
There has to be a better way.
What you can do is build a an array with NaNs in the non-masked values and compute np.nanmedian (which ignores NaNs). You can build such an array "on the fly" using np.where:
x = np.arange(4*3*4).reshape((4,3,4))
m = x%2 == 0
np.nanmedian(np.where(m, x, np.NaN), axis=2)
>>array([[ 1., 5., 9.],
[13., 17., 21.],
[25., 29., 33.],
[37., 41., 45.]])
I have a hard time understanding what you are trying to say, but hopefully this will help:
You can use np.whereto find and replace - or ignore/remove - values that you want to exclude.
Or you can use bin_mask = stack != value_you_want_to_ignore to get a boolean array that you can use to ignore your critical values.
I know how to draw points moved by matrix, like this below
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
x=np.random.randn(2) #2*1 matrix
A=np.random.randn(2,2) #2*2 matrix
print ('the content of x：\n{}\n the content of A：\n{}'.format(x,A))
def action(pt,n):
record=[pt]
for i in range(n):
pt= A#pt
record=np.vstack([record,pt])
plt.scatter(record[:,1],record[:,1])
action(x,100)
the function "action" will draw something like a line, but I want to move points by matrix and then draw it like an orbit
SHORT ANSWER:
plt.scatter(record[:,1],record[:,1]) will feed same values in both x & y dimensions & hence will always return a line. Replace it by:
X,Y = np.hsplit(record,2)
plt.scatter(X,Y)
LONG ANSWER:
The main cause behind plot coming out as a line is that you are generating the plot using 2 constants (although randomly generated). I will illustrate using below example:
>>> c
array([[ 1., 2.],
[ 2., 4.]])
>>> d
array([ 3., 4.])
>>> d#c
array([ 11., 22.])
>>> d#c#c
array([ 55., 110.])
>>> d#c#c#c
array([ 275., 550.])
Notice how all the recursive operation is only multiplying the initial co-ordinate by 5 at each stage.
How to get a non-linear plot??
Utilize the variable 'i' which we are calling for loop operation by giving it a power of 2(parabola) or more.
Use random numbers populated in the 2 matrices greater than 1. Otherwise all the operations either increase the magnitude in -ve or if b/w (-1,1) the magnitude decreases.
Use mathematical functions to introduce non-linearity. Eg:
pt = pt + np.sin(pt)
Reflect if using 2 random matrices & looping over them is the only way to achieve the curve. If this activity is independent from your bigger programme etc, then probably try different approach by using mathematical functions which generate the curve you like.
I have been dealing with linear algebra problems of the form A = Bx in Python and comparing this to a colleague's code in MATLAB and Mathematica. We have noticed differences between Python and the others when B is a singular matrix. When using numpy.linalg.solve() I throw a singular matrix error, so I've instead implemented .pinv() (the Moore Penrose pseudo inverse).
I understand that storing the inverse is computationally inefficient and am first of all curious if there's a better way of dealing with singular matrices in Python. However the thrust of my question lies in how Python chooses an answer from an infinite solution space, and why it chooses a different one than MATLAB and Mathematica do.
Here is my toy problem:
B = np.array([[2,4,6],[1,0,3],[0,7,0]])
A = np.array([[12],[4],[7]])
BI = linalg.pinv(B)
x = BI.dot(A)
The answer that Python outputs to me is:
[[ 0.4]
[ 1. ]
[ 1.2]]
While this is certainly a correct answer, it isn't the one I had intended: (1,1,1). Why does Python generate this particular solution? Is there a way to return the space of solutions rather than one possible solution? My colleague's code returned (1, 1, 1) - is there a reason that Python is different from Mathematica and MATLAB?
In short, your code (and apparently np.linalg.lstsq) uses the Moore-Penrose pseudoinverse, which is implemented in np.linalg.pinv. MATLAB and Mathematica likely use Gaussian elimination to solve the system. We can replicate this latter approach in Python using the LU decomposition:
B = np.array([[2,4,6],[1,0,3],[0,7,0]])
y = np.array([[12],[4],[7]])
P, L, U = scipy.linalg.lu(B)
This decomposes B as B = P L U, where U is now an upper-diagonal matrix, and P L is invertible. In particular, we find:
>>> U
array([[ 2., 4., 6.],
[ 0., 7., 0.],
[ 0., 0., 0.]])
and
>>> np.linalg.inv(P # L) # y
array([[ 12.],
[ 7.],
[ 0.]])
The goal is to solve this under-determined, transformed problem, U x = (P L)^{-1} y. The solution set is the same as our original problem. Let a solution be written as x = (x_1, x_2, x_3). Then we immediately see that any solution must have x_2 = 1. Then we must have 2 x_1 + 4 + 6 x_2 = 12. Solving for x_1, we get x_1 = 4 - 3 x_2. And so any solution is of the form (4 - 3 x_2, 1, x_2).
The easiest way to generate a solution for the above is to simply choose x_2 = 1. Then x_1 = 1, and you recover the solution that MATLAB gives you: (1, 1, 1).
On the other hand, np.linalg.pinv computes the Moore-Penrose pseudoinverse, which is the unique matrix satisfying the pseudionverse properties for B. The emphasis here is on unique. Therefore, when you say:
my question lies in how Python chooses an answer from an infinite solution space
the answer is that all of the choosing is actually done by you when you use the pseudoinverse, because np.linalg.pinv(B) is a unique matrix, and hence np.linalg.pinv(B) # y is unique.
To generate the full set of solutions, see the comment above by #ali_m.
I have got this code:
import numpy as np
from scipy.linalg import eig
transition_mat = np.matrix([
[.95, .05, 0., 0.],\
[0., 0.9, 0.09, 0.01],\
[0., 0.05, 0.9, 0.05],\
[0.8, 0., 0.05, 0.15]])
S, U = eig(transition_mat.T)
stationary = np.array(U[:, np.where(np.abs(S - 1.) < 1e-8)[0][0]].flat)
stationary = stationary / np.sum(stationary)
>>> print stationary
[ 0.34782609 0.32608696 0.30434783 0.02173913]
But I can't understand the line:
stationary = np.array(U[:, np.where(np.abs(S - 1.) < 1e-8)[0][0]].flat)
Can anyone explain the part: U[:, np.where(np.abs(S - 1.) < 1e-8)[0][0]].flat ?
I know that the routine returns S: eigenvalue, U : eigenvector. I need to find the eigenvector corresponding to the eigenvalue 1. I have wrote the code below:
for i in range(len(S)):
if S[i] == 1.0:
j = i
matrix = np.array(U[:, j].flat)
I am getting output:
: [ 0.6144763 0.57607153 0.53766676 0.03840477]
but it does not give the same output. why?!
How to find a stationary distribution.
Ok, I came to this post looking to see if there was a built-in method to find the stationary distribution. It looks like there's not. So, for anyone coming in from Google, this is how I would find the stationary distribution in this circumstance:
import numpy as np
#note: the matrix is row stochastic.
#A markov chain transition will correspond to left multiplying by a row vector.
Q = np.array([
[.95, .05, 0., 0.],
[0., 0.9, 0.09, 0.01],
[0., 0.05, 0.9, 0.05],
[0.8, 0., 0.05, 0.15]])
#We have to transpose so that Markov transitions correspond to right multiplying by a column vector. np.linalg.eig finds right eigenvectors.
evals, evecs = np.linalg.eig(Q.T)
evec1 = evecs[:,np.isclose(evals, 1)]
#Since np.isclose will return an array, we've indexed with an array
#so we still have our 2nd axis. Get rid of it, since it's only size 1.
evec1 = evec1[:,0]
stationary = evec1 / evec1.sum()
#eigs finds complex eigenvalues and eigenvectors, so you'll want the real part.
stationary = stationary.real
What that one weird line is doing.
Let's break that line into parts:
#Find the eigenvalues that are really close to 1.
eval_close_to_1 = np.abs(S-1.) < 1e-8
#Find the indices of the eigenvalues that are close to 1.
indices = np.where(eval_close_to_1)
#np.where acts weirdly. In this case it returns a 1-tuple with an array of size 1 in it.
the_array = indices[0]
index = the_array[0]
#Now we have the index of the eigenvector with eigenvalue 1.
stationary = U[:, index]
#For some really weird reason, the person that wrote the code
#also does this step, which is completely redundant.
#It just flattens the array, but the array is already 1-d.
stationary = np.array(stationary.flat)
If you compress all these lines of code into one line you get stationary = np.array(U[:, np.where(np.abs(S-1.)<1e-8)[0][0]].flat)
If you remove the redundant stuff you get stationary = U[:, np.where(np.abs(S - 1.) < 1e-8)[0][0]]
Why your code gives a different stationary vector.
As #Forzaa pointed out, your vector cannot represent a vector of probabilities because it does not sum to 1. If you divide it by its sum, you'll get the vector the original code snippet has.
Just add this line:
stationary = matrix/matrix.sum()
Your stationary distribution will then match.
stationary = np.array(U[:,np.where(np.abs(S-1.) < 1e-8)[0][0]].flat)
This piece of code is searching for elements in U who's corresponding eigen value - 1 is less than 1e-8
actually, just do a simple while iteration. I'll use a random P as an example
def get_stationary(n):
row = n
pi = np.full((1, row), 1 / row)
T = np.array([[1/4,1/2,1/4],
[1/3,0,2/3],
[1/2,0,1/2]])
while True:
new_pi = np.dot(pi, T)
if np.allclose(pi, new_pi):
return pi
break
pi = new_pi
print(get_stationary(3))