I get video streams from ip cameras using rtsp in python and want to get absolute timestamp for each frame in the sender reports. If I read a stream the camtime obviously starts from zero:
ret, image_np = cap.read()
camtime = cap.get(cv2.CAP_PROP_POS_MSEC)/1000.
I need to know when the frames are recorded since the cameras are located in different time zones and the frames are received with different delays. Can someone advise what library to use and how the python code should look like?
Thanks!
P.S. I can get camera streaming start time from ffmpeg in SDP (option "o").
I have an Objective-C app thats connects to python server throw Socket.IO,
i currently sends h264 raw frames throw socket and receive them in the Python side, but i dont know how to proceed from there,
i have tried to use numpy and cv2.imdecode to convert the byte frame for process and none of those helped me.
i also tried Ffmpeg but i am new to this library and maybe someone can put a light on it.
the end goal is to process this frames for object-detection, and restream them to the web.
the frame sends as singles each times.
-(void)videoFeed:(DJIVideoFeed *)videoFeed didUpdateVideoData:(NSData *)videoData {
[[DJIVideoPreviewer instance] push:(uint8_t *)videoData.bytes length:(int)videoData.length];
[_videoExtractor parseVideo:(uint8_t *)videoData.bytes length:(int)videoData.length withFrame:^(VideoFrameH264Raw *frame) {
I have a script in python where I need to buffer 5 seconds of a YUV video stream from a camera and do some calculations on the frames before writing to disk. My issue is that I want to be able to use VFR (variable frame rate) with the video, and keep track of the timecode. I can't use resources to do much work to demux the video or decode it.
I need to use a container because I'm using pipes and ffmpeg to stream the video -- if I were to use python it seems like I'm stuck with opencv which doesn't deal with VFR
Long story short, I'm looking for a container format that stores the video in raw YUV and the timecode next to it in a simple and uncompressed way. any help with this would be much appreciated
I've integrated an IP camera with OpenCV in Python to get the video processing done frame by frame from the live stream. I've configured camera FPS as 1 second so that I can get 1 frame per second in the buffer to process, but my algorithm takes 4 seconds to process each frame, causing stagnation of unprocessed frame in the buffer, that keeps growing by time & causing exponentially delay. To sort this out, I have created one more Thread where I'm calling cv2.grab() API to clean the buffer, it moves pointer towards latest frame in each call. In main Thread, I'm calling retrieve() method which gives me the last frame grabbed by the first Thread. By this design, frame stagnation problem is fixed and exponential delay is removed, but still constant delay of 12-13 seconds could not be removed. I suspect when cv2.retrieve() is called its not getting the latest frame, but the 4th or 5th frame from the latest frame. Is there any API in OpenCV or any other design pattern to get this problem fixed so that I can get the latest frame to process.
If you don't mind compromising on speed.
you can create a python generator which opens camera and returns frame.
def ReadCamera(Camera):
while True:
cap = cv2.VideoCapture(Camera)
(grabbed, frame) = cap.read()
if grabbed == True:
yield frame
Now when you want to process the frame.
for frame in ReadCamera(Camera):
.....
This works perfectly fine. except opening and closing camera will add up to time.
Problem Outline
I have an h264 real-time video stream (I'll call this "the stream") being captured in Process1. My goal is to extract each frame from the stream as it comes through and use Process2 to analyze it with OpenCV. (Process1 is nodejs, Process2 is Python)
Things I've tried, and their failure modes:
Send the stream directly from one Process1 to Process2 over a named fifo pipe:
I succeeded in directing the stream from Process1 into the pipe. However, in Process2 (which is Python) I could not (a) extract individual frames from the stream, and (b) convert any extracted data from h264 into an OpenCV format (e.g. JPEG, numpy array).
I had hoped to use OpenCV's VideoCapture() method, but it does not allow you to pass a FIFO pipe as an input. I was able to use VideoCapture by saving the h264 stream to a .h264 file, and then passing that as the file path. This doesn't help me, because I need to do my analysis in real time (i.e. I can't save the stream to a file before reading it in to OpenCV).
Pipe the stream from Process1 to FFMPEG, use FFMPEG to change the stream format from h264 to MJPEG, then pipe the output to Process2:
I attempted this using the command:
cat pipeFromProcess1.fifo | ffmpeg -i pipe:0 -f h264 -f mjpeg pipe:1 | cat > pipeToProcess2.fifo
The biggest issue with this approach is that FFMPEG takes inputs from Process1 until Process1 is killed, and only then does Process2 begin to receive the data.
Additionally, on the Process2 side, I still don't understand how to extract individual frames from the data coming over the pipe. I open the pipe for reading (as "f") and then execute data = f.readline(). The size of data varies drastically (some reads have length on the order of 100, others length on the order of 1,000). When I use f.read() instead of f.readline(), the length is much larger, on the order of 100,000.
If I were to know that I was getting the correct size chunk of data, I would still not know how to transform it into an OpenCV-compatible array because I don't understand the format it's coming over in. It's a string, but when I print it out it looks like this:
��_M~0A0����tQ,\%��e���f/�H�#Y�p�f#�Kus�} F����ʳa�G������+$x�%V�� }[����Wo �1'̶A���c����*�&=Z^�o'��Ͽ� SX-ԁ涶V&H|��$
~��<�E�� ��>�����u���7�����cR� �f�=�9 ��fs�q�ڄߧ�9v�]�Ӷ���& gr]�n�IRܜ�檯����
� ����+ �I��w�}� ��9�o��� �w��M�m���IJ ��� �m�=�Soՙ}S �>j �,�ƙ�'���tad =i ��WY�FeC֓z �2�g�;EXX��S��Ҁ*, ���w� _|�&�y��H��=��)� ���Ɗ3# �h���Ѻ�Ɋ��ZzR`��)�y�� c�ڋ.��v�!u���� �S�I#�$9R�Ԯ0py z ��8 #��A�q�� �͕� ijc �bp=��۹ c SqH
Converting from base64 doesn't seem to help. I also tried:
array = np.fromstring(data, dtype=np.uint8)
which does convert to an array, but not one of a size that makes sense based on the 640x368x3 dimensions of the frames I'm trying to decode.
Using decoders such as Broadway.js to convert the h264 stream:
These seem to be focused on streaming to a website, and I did not have success trying to re-purpose them for my goal.
Clarification about what I'm NOT trying to do:
I've found many related questions about streaming h264 video to a website. This is a solved problem, but none of the solutions help me extract individual frames and put them in an OpenCV-compatible format.
Also, I need to use the extracted frames in real time on a continual basis. So saving each frame as a .jpg is not helpful.
System Specs
Raspberry Pi 3 running Raspian Jessie
Additional Detail
I've tried to generalize the problem I'm having in my question. If it's useful to know, Process1 is using the node-bebop package to pull down the h264 stream (using drone.getVideoStream()) from a Parrot Bebop 2.0. I tried using the other video stream available through node-bebop (getMjpegStream()). This worked, but was not nearly real-time; I was getting very intermittent data streams. I've entered that specific problem as an Issue in the node-bebop repository.
Thanks for reading; I really appreciate any help anyone can give!
I was able to solve opening a Parrot Anafi stream with OpenCV (built with FFMPEG) in Python by setting the following environment variable:
export OPENCV_FFMPEG_CAPTURE_OPTIONS="rtsp_transport;udp"
FFMPEG defaults to TCP transport, but the feed from the drone is UDP so this sets the correct mode for FFMPEG.
Then use:
cv2.VideoCapture(<stream URI>, cv2.CAP_FFMPEG)
ret, frame = cap.read()
while ret:
cv2.imshow('frame', frame)
# do other processing on frame...
ret, frame = cap.read()
if (cv2.waitKey(1) & 0xFF == ord('q')):
break
cap.release()
cv2.destroyAllWindows()
as usual.
This should also work with a Parrot Bebop, but I don't have one to test it.
There are some suggestions online for piping the h264 stream into the opencv program using standard in:
some-h264-stream | ./opencv-program
where opencv-program contains something like:
VideoCapture cap("/dev/stdin");