I have an application where I am attempting structure from motion for multiple views from the same camera using OpenCV in Python. (OpenCV isn't a hard requirement but Python is right now). Example: I have 16 camera images to work with with small motions between each frame. In the frames (200x200) I can track ~50 features and I'd like to estimate camera pose and 3D point locations for the features as accurately as possible.
I'm at the point where I'd like to use a stream of frames (from video) to refine the estimates of camera pose and 3D point positions since estimating pose and triangulating 3D points from two frames produces a lot of noise. I believe Bundle Adjustment is the next obvious direction to go in, but I'm not finding any obvious Python implementation of bundle adjustment to use. Many options (such as LevMarqSparse::bundleAdjust()) appear to be only partially completed or not fully adopted.
What is a good place to start? I guess I'm looking for a relatively simple Python bundle adjustment to prototype with and see if that is a direction I want to invest more time into.
Assuming you have a calibrated camera and an initial estimate of intrinsics and extrinsic parameters, you could at first perform a simple bundle adjustment directly in Python.
For instance, you could use stacks of three images, compute 3D points from the feature points via the homogeneous triangulation method. A simple bundle adjustment as a first prototype can be build for instance via scipy.least_squares and the Trusted Region Reflective non-linear optimization technique. Check out this tutorial.
Afterwards you can decide, whether you want to implement or use a Levenberg-Marquardt optimization technique which is able to handle sparse Jacobians or even determine the Jacobian analytically to hypothetically increase the convergence if desired.
So far, I think there is no adequate library in Python which provides a potent and high performant implementation of bundle adjustment.
Related
I'm not sure what this technique called, before going ahead and implement this myself, I wonder does opencv/scipy/pytorch/etc already has such algorithm implemented? This image below visualize what's happening, it's usually used in Struture-from-Motion (SfM). Basically, it calculates how uniform is the distribution of the points in multi-resolution grid cells.
I notice that opencv has SfM module, but I don't see any for such calculations specifically.
I am using win10, python and c#. I want to calculate circumference of human parts (belly, biceps etc) with using point cloud or 3d scans like .stl .obj .ply. Now I can get the point cloud of human body with kinect v2. I have point cloud of human body, scanned 3d human body in .stl .obj .ply formats.
I need some ideas and infos about it. I don't know how to analyse the things I have and how to calculate what I want.
Here I found an example of what I am trying to do but It doesn't need to be perfectly stable like that, Its for a school homework. Maybe you can give me some ideas about how to achieve my goal. Thank you for your help.
https://www.youtube.com/watch?time_continue=48&v=jOvaZGloNRo
I get 3d scanned object with kinect v2 and use PCL to convert it into point cloud.
I don't know about using PCL with Python or C#. In general you are looking at the following steps:
Filtering the points to the interested region
Segmenting the shape
Extracting the parameters
If you're interested in only Python, then OpenCV might be the best option. You can also develop the core logic in C++ and wrap it for Python or C#. C++ also has some nice UI libaries (Qt, nanogui), please see the following details for achieving the objective with PCL
Filtering
CropBox or PassThrough can be used for this. It'll result in similar results as shown in the image assuming that the frame has been chosen properly. If not, the points cloud can be easily transformed
Segmenting the shape
Assuming you want an average circumference, you might need to experiment with Circle 2D, Circle 3D and Cylinder models. More details regarding usage and API are here. The method chosen can be simple SAC (Sample Consensus) like RANSAC (Random SAC) or advanced method like LMEDS (Least Median of Squares) or MLESAC (Max Likelihood Estimation SAC)
Extracting the parameters
All models have a radius field which can be used to find the circumference using standard formula (2*pi*r)
Disclaimer: Please take note that the shape is circular, not ellipse and the cylinder are right angled cylinders. So if the object measured (arm, or bicep) is not circular, the computed value might not be close to ground truth in extreme cases
I have a video of a road/building and I want to create a 3D model out of it. The scene I am looking at is rigid and the drone is moving. I assume not having any extra info like camera pose, accelerations or GPS position. I would love to find a python implementation that I can adapt to my liking.
So far, I have decided to use the OpenCV calcOpticalFlowFarneback() for optical flow, which seems reasonably fast and accurate. With it, I can get the Fundamental Matrix F with findFundamentalMat(). So far so good.
Now, according to the tutorial I am following here, I am supposed to magically have the Calibration Matrix of the camera, which I obviously don't have nor plan to have available in the future app I am developing.
After some long research, I have found a paper (Self-calibration of a moving camera from point correspondences and
fundamental matrices) from 1997 that defines what I am looking for (with a nice summary here). I am looking for the simplest/easiest implementation possible, and I am stuck with these problems:
If the camera I am going to use changes exposure and focus automatically (no zoom), are the intrinsic parameters of the camera going to change?
I am not familiar with the Homotopy Continuation Method for solving equations numerically, plus they seem to be slow.
I intend to use the Extended Kalman Filter, but do not know where to start, knowing that a bad initialization leads to non-convergence.
Digging some more I found a Multi Camera Self Calibration toolbox open-source written for Octave with a Python wrapper. My last resort will be to break down the code and write it in Python directly. Any other options?
Note: I do not want to use the a chess board nor the planarity constraint.
Is there any other way to very accurately self-calibrate my camera? After 20 years of research since 1997, has anyone come up with a more straightforward method??
Is this a one-shot thing, or are you developing an app to process lots videos like these automatically?
If the former, I'd rather use an integrated tool like Blender. Look up one of the motion tracking (or "matchmoving") tutorials on youtube to get an idea of it, for example this one.
Image mosaics use a set of predefined squared images to build a larger image (example here).
There are a lot of solutions and it's quite trivial to achieve this effect. However, it becomes much harder with the following constraints:
The shape of the original mosaics is abstract. Any convex polygon could do.
Each mosaic can only be used once.
There is no need for the mosaics to be absolutely packed (i.e. occupying 100% of the canvas), but they should be as packed as possible without overlapping.
I'm trying to automatize the ancient art of tesselation, specifically the Opus palladianum technique.
My idea is to use simulated annealing or some other heuristic to optimize the position and rotation of each irregular mosaic, swaping two in each iteration, trying to minimize some energy function that reflects the similarity to the target image as well as the "packness" of the tiles.
I'm trying to achieve this in python, any ideas and help would be greatly appreciated.
Example:
I expect that you may probably use GA (Genetic Algorithm) with a "non-overlapping" constraint to do this job.
Parameters for individual (each convex polygon) are:
initial position
rotation
(size ?)
And your fit function will be build to give best note to each individual when polygon are not overlapping (and close to other individual)
You may see this video and this one as example.
Regards
Turtlebot exploration using a pregenerated map
I am working on the exploration strategy of a turtlebot that is supposed to navigate a known environment (with the help of a pre-built map) and searches for an object (e.g. a red ball).
The map has been generated using gmapping and now I am trying to implement a strategy for exploring the known map. Although generating random points on the map and then performing a 360° scan at these points is an option, I am not too enthusiastic about this method, since it does not guarantee that the object will be located if it exists.
What I would like to have is a set of points, such that after visiting all these points and performing a 360° scan at each of those points, the entire accessible/visible area of the map is covered by the robot.
I felt that the art gallery algorithm (described here: http://cs.smith.edu/~orourke/books/ArtGalleryTheorems/Art_Gallery_Full_Book.pdf) also achieves something similar, and hence tried to implement it in my scenario.
The art gallery problem requires polygons and I am finding it quite difficult to generate polygons from maps generated by gmapping on the real robot (although for simpler maps it's possible).
I have tried using the triangle module in Python to create convex hulls of clusters of objects, but that is not too useful in noisy environment like the one linked below. (I converted the .pgm file into a .png by plotting only obstacle points)
obstacle points
Note: My turtlebot does not have a laser scanner. It just has a Asus Xtion Pro Live.
I would really appreciate some help regarding the following:
Is there an easier way of exploring a known map with a turtlebot
(preferably some package in ROS)?
If the art gallery algorithm is a viable option, then how should I obtain accurate polygons from the scan-points of the obstacles?
Thank you in advance.