I'm interested in using a stereo camera for calculating depth in video/images. The camera is a USB 3.0 Stereoscopic camera from Leopard Imaging https://www.leopardimaging.com/LI-USB30-V024STEREO.html. I'm using MAC OS X btw.
I was told by their customer support that it's a "UVC" camera. When connected to an apple computer it gives a greenish image.
My end goal is to use OpenCV to grab the left and right frames from both lenses so that I can calculate depth. I'm familiar with OpenCV, but not familiar with working with stereo cameras. Any help would be much appreciated. So far I have been doing this in Python 3:
import numpy as np
import cv2
import sys
from matplotlib import pyplot as plt
import pdb; pdb.set_trace()
print("Camera 1 capture", file=sys.stderr)
cap = cv2.VideoCapture(1)
print("Entering while", file=sys.stderr)
while(True):
_ = cap.grab()
retVal, frame = cap.retrieve()
#gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
cv2.imshow('frame', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
This works, but it only gives me a green picture/image with no depth. Any advice on how to get both left and right frames from the camera?
The leopard imaging people sent me a clue but I'm not able to make progress because i guess some missing file in my core file. However, I thought it might help somebody. So I'm posting it as an answer.
The message was sent by one of the leopard imaging people I contacted through mail. It goes like this.....
We have a customer who successfully separated the two videos on Linux OS one year ago. I tried to contact him to see if he can share the source code with us. Unfortunately, he already left the former company, but he still found some notes (below). Hope it helps.
The camera combines the image from the two sensors into one 16-bit pixel data (The high 8 bits from one camera, and the low 8 bit from the other camera).
To fix this problem in linux opencv, should skip color transform by opencv: at
modules/videoio/src/cap_v4l.cpp static IplImage* icvRetrieveFrameCAM_V4L( CvCaptureCAM_V4L* capture, int)
case V4L2_PIX_FMT_YUYV:
#if 1
/*
skip color convert. Just copy image buffer to frame.imageData
*/
memcpy(capture->frame.imageData, capture->buffers[capture->bufferIndex].start, capture->buffers[capture->bufferIndex].length);
#else
yuyv_to_rgb24(capture->form.fmt.pix.width, capture->form.fmt.pix.height,(unsigned char*) capture->buffers[capture->bufferIndex].start),
(unsigned char*)capture->frame.imageData);
#endif
Hope this helps.
DISCLAIMER : I went forward with the C++ version(my project was in C++) given in libuvc used the provided routines to obtain the left and right frames separately.
I am chasing the same issue, but with C/C++. I have contacted Leopard and am waiting for an answer. My understanding is that the two grayscale cameras are interlaced into a single image (and I think that OpenCV sees this as a color image, hence the strange colors and being out of focus.) You need to break apart the bytes into two separate frames. I am experimenting, trying to figure out the byte placement, but have not gotten too far. If you figure this out, please let me know!
They have a C# on Windows example here:
https://www.dropbox.com/sh/49cpwx0s70fuich/2e0_mFTJY_
Unfortunately it is using their libraries (which are not source) to do the heavy lifting, so I can't figure out what they are doing.
I met the same issue as you and finally arrived to a solution. But I don't know if OpenCV can handle it directly, especially in Python.
As jordanthompson said, the two images are interlaced into one. The image you receive is in YUYV (Y is the light intensity, UV contains the color information). Each pixel is coded on 16 bits, 8 for Y, 8 for U or V depending on which pixel you are looking at.
Here, the Y bits come from the left image, the UV bits from the right image. But when OpenCV receives this image, it converts it to RGB which then definitely mix the two images.. I could not find a way in Python to tell OpenCV to get the image without converting it... Therefore we need to read the image before OpenCV. I managed to do it with libuvc (https://github.com/ktossell/libuvc) after adding two small functions to perform the proper conversions. I guess you can use this library if you can use openCV in C++ instead of Python. If you really have to stick with Python, then I don't have a complete solution, but at least now you know what to look for: try to read the image directly in YUYV and then separate the bits into left and right image (you will get two grayscale images).
Good luck !
Related
Fairly experienced with OpenCV, Python and Windows/Linux.
When trying to read a high-res (~8 MP) webcam image using CAP_DSHOW (nothing else works), at certain situations I get a black image with the right size. I mean zeros all along.
This happens either when there is too much light, the autofocus camera is too near to the object, or both.
Anyone know what's going on?
Thanks
I'm looking for solution to access thermal data of camera. well i used OpenCV and only could got original image. but there is no more data for process like temperature. I tried available library about HikVision cameras and surfed the net for this. but i could not be succeed. also I tried FLIR library but no success.
second solution that I have is converting RGB to temperature but i don't know what to do for this kind of process. Also I know the range of device temperature which is between-20 to 150 degree
looking for something like this:
# cam model: hikvision DS-2TD2615-10
import cv2
import hikvision api library for example
thermal = cv2.VideoCapture()
thermal.open("rtsp://""user:pass#ip:port/Streaming/channels/202/")
ret, frame = thermal.read()
while True:
ret, frame = thermal.read()
temp_data = api.read_temperature(frame) # -> array or excel file
cv2.imshow('frame', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
thermal.release()
cv2.destroyAllWindows()
and my video input is something similar to this pic and for example i want to find out the nose is how much hot just by click on it:
General answer for thermal images from any camera - you can't just convert grayscale level (or color if you already applied pallete to your image) to temperature values. You need to know some coefficients which relates to IR matrix. Some software may embed that data to image file metadata but there is no standard for that. Also, if you resaved your image file without knowing it, you'll probably lose that metadata.
Also, like plain visible-light camera, IR-camera can adapt it's range to current image. So, if you're shooting a human in a room, minimum temperature in your picture will be like 22°С (cold wall or floor), maximum will be like 37°C (hottest part of human body). In that case you'll get 256 gray levels covering range of 15 degrees, so black color is 22°С, white is 37°С (keep in mind proportion is not linear!). You move your camera to a cup of hot tea with like 60°С and your relation of gray level to temperature changes. So, you need to get coefficients for every frame.
It is possible to "fix" temperature range on some cameras but that depends on specific models.
More than that - most cheap thermal cameras don't deal with temperature values at all.
P.S. Oh, I just noticed exact model of your camera. So answer is even stronger "YOU CAN'T". Don't expect capabilities of science or medical thermal camera from that chinese poorly documented surveilance hardware.
I am new to using python and object detection and need some help for a uni project!
1- I am wandering if it is possible to use the same haar cascade file i used to train the pi to detect a specific object in a still image (using the picamera) for detecting the same image in a live video feed as my uni project requires the robot to search and find the specific object and retrieve it or do is it a completely separate process (i.e. setting up classes etc)? if so, where could I i find the most helpful information on doing this as I haven't been too successful when looking online. (pyimagesearch has a blog post on this but goes about using classes and i'm not sure how to go about even creating a class just for a specific object or if you even need one..)
2- Currently, my pi can kind of detect the object (a specific cube) in the still images but isnt very accurate or consistent, it often detects around the edges of the object as well as incorrectly detecting other things in the background or shadows that are part of the image, as the object as well. It tends to find more than one of the cube (lots of small rectangles mostly around - in close proximity) and in the object so it says its detecting 2+ cubes in an image (sometimes 15-20 or more) rather than just the one cube. I am wandering how I could reduce this error and increase the accuracy and consistency of the pi so it detects just the one, or at least doesn't wrongfully detect background shadows or other things in the image? I understand that lighting affects the result but I am wandering if its due to the quality of the original image i took with the picamera of the cube I used to train the haar cascade (it was quite a dark photo due to insufficient lighting), or maybe the size of the image itself (cropped it down to the edges so it is just the cube and resized it to 50x50), or maybe that I didnt train more than one image of the object against negatives when training the cascade file..? do the images you supply for training the pi have to taken with the picamera or could you take clearer pictures say with your phone and use them to train the cascade for detection via the picamera?I tried upgrading the resolution in the code but that made the data analysis take too long and didnt make much difference. Apologies for the long post as I am new to all this and wandering if theres any way to improve the results or is the only way for higher accuracy to retrain the cascade which I'd rather not do as it took two days to complete due to working with a pi zero W board!
Much Appreciated!
My specs: A raspberry pi Zero W board with a 16gb SD Card on Mac, running openCV 3.2 and Python 2.7.
Code for object detection in an image taken using the pi camera:
import io
import picamera
import cv2
import numpy as np
stream = io.BytesIO()
with picamera.PiCamera() as camera: camera.resolution = (320, 240) camera.capture(stream, format='jpeg')
buff = numpy.fromstring(stream.getvalue(), dtype=numpy.uint8)
image = cv2.imdecode(buff, 1)
cube_cascade = cv2.CascadeClassifier('/home/pi/data/cube1-cascade-10stages.xml')
gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
Cube = cube_cascade.detectMultiScale(gray, 1.1, 5)
print "Found "+str(len(Cube))+" cube(s)"
for (x,y,w,h) in Cube: cv2.rectangle(image, (x,y), (x+w,y+h), (255,255,0), 2)
cv2.imwrite('result.jpg',image)
Thanks in advance.
recently I have been playing with the 360 fly HD camera and wondering if Aruco Marker can be detected during real time. The first thing come to my mind is to convert the fisheye image into perspective image first and then perform the detection on the perspective image(I am gonna try it and will update my result here later).
Converting a fisheye image into a panoramic, spherical or perspective projection
Hugin HowTo: Convert 360 Image to Cropped Flat Panoramic Image
I am not an expert in this field. Has anyone done this before? Is this something can be achieved by calibrating the camera differently such as correcting the camera matrix and distortion coefficient matrix?
If I am heading to the wrong direction, please let me know.
I was able to get a better understanding during the process.
First, I want to say that 360(fisheye, spherical, however you call it) image is NOT distorted. I was so tricked by my intuition and thought that the image was distorted based on what it looks like. NO it is not distorted. Please read enter link description here for more information.
Next, I have tried both 360 fly cameras and neither works. Every time I tried to access the camera with opencv, it automatically powers off and switch to storage mode. I guess the 360 dev team purposely implements this switching function to prevent "hacking" of their products. But, I've seen people successfully hacked the 360 fly, it's definitely workable.
At last, I was able to detect Aruco with Ricoh theta V(theta S should also work). It's so developer friendly and I was able to make it run in my first attempt. You just have to select the right camera and let the code run. The only problem is the range, which is expected(about 6ft) and Ricoh camera is kind of expensive($499).
click here to view succesful detection
I am using OpenCV (2.4) and Python (2.7.3) with a USB camera from Thorlabs (DC1545M).
I am doing some image analysis on a video stream and I would like to be able to change some of the camera parameters from my video stream. The confusing thing is that I am able to change some of the camera properties but not all of them, and I am unsure of what I am doing wrong.
Here is the code, using the cv2 bindings in Python, and I can confirm that it runs:
import cv2
#capture from camera at location 0
cap = cv2.VideoCapture(0)
#set the width and height, and UNSUCCESSFULLY set the exposure time
cap.set(cv2.cv.CV_CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT, 1024)
cap.set(cv2.cv.CV_CAP_PROP_EXPOSURE, 0.1)
while True:
ret, img = cap.read()
cv2.imshow("input", img)
#cv2.imshow("thresholded", imgray*thresh2)
key = cv2.waitKey(10)
if key == 27:
break
cv2.destroyAllWindows()
cv2.VideoCapture(0).release()
For reference, the first argument in the cap.set() command refers to the enumeration of the camera properties, listed below:
0. CV_CAP_PROP_POS_MSEC Current position of the video file in milliseconds.
1. CV_CAP_PROP_POS_FRAMES 0-based index of the frame to be decoded/captured next.
2. CV_CAP_PROP_POS_AVI_RATIO Relative position of the video file
3. CV_CAP_PROP_FRAME_WIDTH Width of the frames in the video stream.
4. CV_CAP_PROP_FRAME_HEIGHT Height of the frames in the video stream.
5. CV_CAP_PROP_FPS Frame rate.
6. CV_CAP_PROP_FOURCC 4-character code of codec.
7. CV_CAP_PROP_FRAME_COUNT Number of frames in the video file.
8. CV_CAP_PROP_FORMAT Format of the Mat objects returned by retrieve() .
9. CV_CAP_PROP_MODE Backend-specific value indicating the current capture mode.
10. CV_CAP_PROP_BRIGHTNESS Brightness of the image (only for cameras).
11. CV_CAP_PROP_CONTRAST Contrast of the image (only for cameras).
12. CV_CAP_PROP_SATURATION Saturation of the image (only for cameras).
13. CV_CAP_PROP_HUE Hue of the image (only for cameras).
14. CV_CAP_PROP_GAIN Gain of the image (only for cameras).
15. CV_CAP_PROP_EXPOSURE Exposure (only for cameras).
16. CV_CAP_PROP_CONVERT_RGB Boolean flags indicating whether images should be converted to RGB.
17. CV_CAP_PROP_WHITE_BALANCE Currently unsupported
18. CV_CAP_PROP_RECTIFICATION Rectification flag for stereo cameras (note: only supported by DC1394 v 2.x backend currently)
(Please note, as commenter Markus Weber pointed out below, in OpenCV 4 you have to remove the "CV" prefix from the property name, eg
cv2.CV_CAP_PROP_FRAME_HEIGHT -> cv2.CAP_PROP_FRAME_HEIGHT)
My questions are:
Is it possible to set camera exposure time (or the other camera parameters) through python/opencv?
If not, how would I go about setting these parameters?
Note: There is C++ code provided by the camera manufacturer showing how to do this, but I'm not an expert (by a long shot) in C++ and would appreciate any python-based solution.
Not all parameters are supported by all cameras - actually, they are one of the most troublesome part of the OpenCV library. Each camera type - from android cameras to USB cameras to professional ones offer a different interface to modify its parameters. There are many branches in OpenCV code to support as many of them, but of course not all possibilities are covered.
What you can do is to investigate your camera driver, write a patch for OpenCV and send it to code.opencv.org. This way others will enjoy your work, the same way you enjoy others'.
There is also a possibility that your camera does not support your request - most USB cams are cheap and simple. Maybe that parameter is just not available for modifications.
If you are sure the camera supports a given param (you say the camera manufacturer provides some code) and do not want to mess with OpenCV, you can wrap that sample code in C++ with boost::python, to make it available in Python. Then, enjoy using it.
I had the same problem with openCV on Raspberry Pi... don't know if this can solve your problem, but what worked for me was
import time
import cv2
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 1280)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 1024)
time.sleep(2)
cap.set(cv2.CAP_PROP_EXPOSURE, -8.0)
the time you have to use can be different
To avoid using integer values to identify the VideoCapture properties, one can use, e.g., cv2.cv.CV_CAP_PROP_FPS in OpenCV 2.4 and cv2.CAP_PROP_FPS in OpenCV 3.0. (See also Stefan's comment below.)
Here a utility function that works for both OpenCV 2.4 and 3.0:
# returns OpenCV VideoCapture property id given, e.g., "FPS"
def capPropId(prop):
return getattr(cv2 if OPCV3 else cv2.cv,
("" if OPCV3 else "CV_") + "CAP_PROP_" + prop)
OPCV3 is set earlier in my utilities code like this:
from pkg_resources import parse_version
OPCV3 = parse_version(cv2.__version__) >= parse_version('3')
I wasn't able to fix the problem OpenCV either, but a video4linux (V4L2) workaround does work with OpenCV when using Linux. At least, it does on my Raspberry Pi with Rasbian and my cheap webcam. This is not as solid, light and portable as you'd like it to be, but for some situations it might be very useful nevertheless.
Make sure you have the v4l2-ctl application installed, e.g. from the Debian v4l-utils package. Than run (before running the python application, or from within) the command:
v4l2-ctl -d /dev/video1 -c exposure_auto=1 -c exposure_auto_priority=0 -c exposure_absolute=10
It overwrites your camera shutter time to manual settings and changes the shutter time (in ms?) with the last parameter to (in this example) 10. The lower this value, the darker the image.
If anyone is still wondering what the value in CV_CAP_PROP_EXPOSURE might be:
Depends. For my cheap webcam I have to enter the desired value directly, e.g. 0.1 for 1/10s. For my expensive industrial camera I have to enter -5 to get an exposure time of 2^-5s = 1/32s.