I am trying to save 10 seconds of buffered video using Python script, in particular '.rgb' format.
In order to do so, I have been using a PiCamera connected to a Raspberry Pi.
Based on the script below, if I choose to save the video using h264 format, I will be able to accomplish the desired goal successfully but if change the format from h264 to .rgb (targeted format), no outputs are generated.
Any thoughts what might be the issue here?
Thanks
Code snap:
import time
import io
import os
import picamera
import datetime as dt
from PIL import Image
import cv2
#obtain current time
def return_currentTime():
return dt.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
#trigger event declaration
def motion_detected():
while True:
print ("Trigger event(y)?")
trigger = input ()
if trigger =="y":
time = return_currentTime()
print ("Buffering...")
camera.wait_recording(5)
stream.copy_to(str(time)+'.rgb')
else:
camera.stop_recording()
break
#countdown timer
def countdown (t):
while t:
mins, secs = divmod (t,60)
timer = '{:02d}:{:02d}'.format(mins, secs)
print(timer, end="\r")
time.sleep(1)
t-=1
print('Buffer available!')
camera = picamera.PiCamera()
camera.resolution = (640, 480)
stream = picamera.PiCameraCircularIO(camera, seconds = 5)
#code will work using h264 as format
camera.start_recording (stream, format = 'rgb')
countdown(5)
motion_detected()
This question has to do with your stream format and how stream.copy_to() works.
According to the docs for the function copy_to(output, size=None, seconds=None, first_frame=2), first_frame is a restriction on the first frame to be copied. This is set to sps_header by default, which is usually the first frame of an H264 stream.
Since your stream format is RGB instead of H264, though, there are no sps_header's, and so copy_to cannot find an sps_header and copies nothing.
To solve this, you have to allow any frame to be the first frame, not just sps_header's. This can be done by setting first_frame=None in your call, like copy_to(file, first_frame=None).
Related
Goal and problem
I'd like to set up an opencv system to process either HLS streams or RMTP streams, however, I am running into a strange issue regarding a reduced frame-rate and an accumulating lag. It's as if the video gets further and further behind from where it is supposed to be in the stream.
I'm looking for a way to keep up to date with a live source even if it means dropping frames.
Current approach
import cv2
cap = cv2.VideoCapture()
cap.open('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8')
while (True):
_, frame = cap.read()
cv2.imshow("camCapture", frame)
cv2.waitKey(1)
I've validated the quality of the stream on VLC and it seems to work fine there.
cv2 speed
.
realistic/expected speed
Questions:
What am I doing wrong here?
Why is it so slow?
How do I sync it to real-time speeds?
My hypothesis is that the jitter is most likely due to network limitations and occurs when a frame packet is dropped. When a frame is dropped, this causes the program to display the last "good" frame which results in the display freezing. This is probably a hardware or bandwidth issue but we can alleviate some of this with software. Here are some possible changes:
1. Set maximum buffer size
We set the cv2.videoCapture() object to have a limited buffer size with the cv2.CAP_PROP_BUFFERSIZE parameter. The idea is that by limiting the buffer, we will always have the latest frame. This can also help to alleviate the problem of frames randomly jumping ahead.
2. Set frame retrieval delay
Currently, I believe the read() is reading too fast even though it is in its own dedicated thread. This may be one reason why all the frames appear to pool up and suddenly burst in the next frame. For instance, say in a one second time interval, it may produce 15 new frames but in the next one second interval, only 3 frames are returned. This may be due to the network packet frame loss so to ensure that we obtain constant frame rates, we simply add a delay in the frame retrieval thread. A delay to obtain roughly ~30 FPS does a good job to "normalize" the frame rate and smooth the transition between frames incase there is packet loss.
Note: We should try to match the frame rate of the stream but I'm not sure what the FPS of the webcam is so I just guessed 30 FPS. Also, there is usually a "direct" stream link instead of going through a intermediate webserver which can greatly improve performance.
If you try using a saved .mp4 video file, you will notice that there is no jitter. This confirms my suspicion that the problem is most likely due to network latency.
from threading import Thread
import cv2, time
class ThreadedCamera(object):
def __init__(self, src=0):
self.capture = cv2.VideoCapture(src)
self.capture.set(cv2.CAP_PROP_BUFFERSIZE, 2)
# FPS = 1/X
# X = desired FPS
self.FPS = 1/30
self.FPS_MS = int(self.FPS * 1000)
# Start frame retrieval thread
self.thread = Thread(target=self.update, args=())
self.thread.daemon = True
self.thread.start()
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
time.sleep(self.FPS)
def show_frame(self):
cv2.imshow('frame', self.frame)
cv2.waitKey(self.FPS_MS)
if __name__ == '__main__':
src = 'https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8'
threaded_camera = ThreadedCamera(src)
while True:
try:
threaded_camera.show_frame()
except AttributeError:
pass
Related camera/IP/RTSP/streaming, FPS, video, threading, and multiprocessing posts
Python OpenCV streaming from camera - multithreading, timestamps
Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture
How to capture multiple camera streams with OpenCV?
OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?
Storing RTSP stream as video file with OpenCV VideoWriter
OpenCV video saving
Python OpenCV multiprocessing cv2.VideoCapture mp4
Attempt at threading
I've attempted this solution from nathancy with minor success.
It involves:
creating a separate thread for image capture from the source
using the main thread exclusively for display.
Code:
import cv2
from threading import Thread
class ThreadedCamera(object):
def __init__(self, source = 0):
self.capture = cv2.VideoCapture(source)
self.thread = Thread(target = self.update, args = ())
self.thread.daemon = True
self.thread.start()
self.status = False
self.frame = None
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
def grab_frame(self):
if self.status:
return self.frame
return None
if __name__ == '__main__':
stream_link = "https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8"
streamer = ThreadedCamera(stream_link)
while True:
frame = streamer.grab_frame()
if frame is not None:
cv2.imshow("Context", frame)
cv2.waitKey(1)
Jittery, but real-time results
.
The streaming works. It maintains real-time. However, it is as if all the frames pool up and suddenly burst into the video. I would like somebody to explain that.
Room for improvement
The real-time stream can be found here.
https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet
This site is scraped for the m3u8 using python's streamlink stream scraper.
import streamlink
streams = streamlink.streams("https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet")
print(streams)
which yeilds:
OrderedDict([
('720p',<HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('live', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet','swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>),
('worst', <HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('best', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet', 'swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>)
])
The possibility that the streams are being read wrong.
I would suggest double checking the compatible video stream codecs with the hardware. I ran into the same issue, frame rate dropped to 5 fps only during streaming, because it was defaulting to a format that is not being streamed so it would convert it then display very lagged (~1s) with lower fps as well.
use Self.capture.set(cv2.CAP_PROP_FOURCC ,cv2.VideoWriter_fourcc('M', 'J', 'P', 'G') ) with the proper codec in place of MJPG and with your cv2.VideoCapture and see if that helps.
Goal and problem
I'd like to set up an opencv system to process either HLS streams or RMTP streams, however, I am running into a strange issue regarding a reduced frame-rate and an accumulating lag. It's as if the video gets further and further behind from where it is supposed to be in the stream.
I'm looking for a way to keep up to date with a live source even if it means dropping frames.
Current approach
import cv2
cap = cv2.VideoCapture()
cap.open('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8')
while (True):
_, frame = cap.read()
cv2.imshow("camCapture", frame)
cv2.waitKey(1)
I've validated the quality of the stream on VLC and it seems to work fine there.
cv2 speed
.
realistic/expected speed
Questions:
What am I doing wrong here?
Why is it so slow?
How do I sync it to real-time speeds?
My hypothesis is that the jitter is most likely due to network limitations and occurs when a frame packet is dropped. When a frame is dropped, this causes the program to display the last "good" frame which results in the display freezing. This is probably a hardware or bandwidth issue but we can alleviate some of this with software. Here are some possible changes:
1. Set maximum buffer size
We set the cv2.videoCapture() object to have a limited buffer size with the cv2.CAP_PROP_BUFFERSIZE parameter. The idea is that by limiting the buffer, we will always have the latest frame. This can also help to alleviate the problem of frames randomly jumping ahead.
2. Set frame retrieval delay
Currently, I believe the read() is reading too fast even though it is in its own dedicated thread. This may be one reason why all the frames appear to pool up and suddenly burst in the next frame. For instance, say in a one second time interval, it may produce 15 new frames but in the next one second interval, only 3 frames are returned. This may be due to the network packet frame loss so to ensure that we obtain constant frame rates, we simply add a delay in the frame retrieval thread. A delay to obtain roughly ~30 FPS does a good job to "normalize" the frame rate and smooth the transition between frames incase there is packet loss.
Note: We should try to match the frame rate of the stream but I'm not sure what the FPS of the webcam is so I just guessed 30 FPS. Also, there is usually a "direct" stream link instead of going through a intermediate webserver which can greatly improve performance.
If you try using a saved .mp4 video file, you will notice that there is no jitter. This confirms my suspicion that the problem is most likely due to network latency.
from threading import Thread
import cv2, time
class ThreadedCamera(object):
def __init__(self, src=0):
self.capture = cv2.VideoCapture(src)
self.capture.set(cv2.CAP_PROP_BUFFERSIZE, 2)
# FPS = 1/X
# X = desired FPS
self.FPS = 1/30
self.FPS_MS = int(self.FPS * 1000)
# Start frame retrieval thread
self.thread = Thread(target=self.update, args=())
self.thread.daemon = True
self.thread.start()
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
time.sleep(self.FPS)
def show_frame(self):
cv2.imshow('frame', self.frame)
cv2.waitKey(self.FPS_MS)
if __name__ == '__main__':
src = 'https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8'
threaded_camera = ThreadedCamera(src)
while True:
try:
threaded_camera.show_frame()
except AttributeError:
pass
Related camera/IP/RTSP/streaming, FPS, video, threading, and multiprocessing posts
Python OpenCV streaming from camera - multithreading, timestamps
Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture
How to capture multiple camera streams with OpenCV?
OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?
Storing RTSP stream as video file with OpenCV VideoWriter
OpenCV video saving
Python OpenCV multiprocessing cv2.VideoCapture mp4
Attempt at threading
I've attempted this solution from nathancy with minor success.
It involves:
creating a separate thread for image capture from the source
using the main thread exclusively for display.
Code:
import cv2
from threading import Thread
class ThreadedCamera(object):
def __init__(self, source = 0):
self.capture = cv2.VideoCapture(source)
self.thread = Thread(target = self.update, args = ())
self.thread.daemon = True
self.thread.start()
self.status = False
self.frame = None
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
def grab_frame(self):
if self.status:
return self.frame
return None
if __name__ == '__main__':
stream_link = "https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8"
streamer = ThreadedCamera(stream_link)
while True:
frame = streamer.grab_frame()
if frame is not None:
cv2.imshow("Context", frame)
cv2.waitKey(1)
Jittery, but real-time results
.
The streaming works. It maintains real-time. However, it is as if all the frames pool up and suddenly burst into the video. I would like somebody to explain that.
Room for improvement
The real-time stream can be found here.
https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet
This site is scraped for the m3u8 using python's streamlink stream scraper.
import streamlink
streams = streamlink.streams("https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet")
print(streams)
which yeilds:
OrderedDict([
('720p',<HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('live', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet','swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>),
('worst', <HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('best', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet', 'swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>)
])
The possibility that the streams are being read wrong.
I would suggest double checking the compatible video stream codecs with the hardware. I ran into the same issue, frame rate dropped to 5 fps only during streaming, because it was defaulting to a format that is not being streamed so it would convert it then display very lagged (~1s) with lower fps as well.
use Self.capture.set(cv2.CAP_PROP_FOURCC ,cv2.VideoWriter_fourcc('M', 'J', 'P', 'G') ) with the proper codec in place of MJPG and with your cv2.VideoCapture and see if that helps.
I'm having problems with using OpenCV, Python, Tkinter and PiCamera in a program.
A Tkinter window is used to display and set the values to be used in OpenCV:
I am trying to continuously read and process the video feed from PiCamera currently I am using:
while True:
for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
root.update_idletasks()
But after some reading on internet I found that using update() is not advisable, so I tried my luck to understand threading but I failed. There are a lot of examples with VideoCapture() which is used with USB cameras but not a lot with PiCamera. Is there any other way than threading?
You can use root.after(...). Below is a sample code:
# define a variable used to stop the image capture
do_image_capture = True
def capture_image():
if do_image_capture:
camera.capture(rawCapture, format='bgr', use_video_port=True)
# do whatever you want on the captured data
...
root.after(100, capture_image) # adjust the first argument to suit your case
capture_image()
Below sample code is using thread:
import threading
stop_image_capture = False
def capture_image():
for frame in camera.capture_continuous(rawCapture, format='bgr', use_video_port=True)
# do whatever you want on the capture image
....
if stop_image_capture:
break
t = threading.Thread(target=capture_image)
t.setDaemon(True)
t.start()
I am working on a project where we are using the Raspicam attached to a Raspberry Pi to capture (and process) images with python using the PiCamera module.
With our current implementation I am experiencing an unexpected behaviour.
The camera is configured to capture images with 15 frames per second. To simulate processing time the program waits 5 seconds.
Minimal example:
#!/usr/bin/env python
import cv2
from picamera import PiCamera
from picamera.array import PiRGBArray
class RaspiCamera:
def __init__(self, width, height, framerate):
self.camera = PiCamera()
self.camera.resolution = (width, height)
self.camera.framerate = framerate
self.rawCapture = PiRGBArray(self.camera, size=self.camera.resolution)
self.capture_continuous = self.camera.capture_continuous(self.rawCapture, format="bgr", use_video_port=True)
def capture(self):
frame = self.capture_continuous.next()
image = self.rawCapture.array
self.rawCapture.truncate(0)
return image
if __name__ == "__main__":
camera = RaspiCamera(640, 480, 15)
while True:
frame = camera.capture()
cv2.imshow("Image", frame)
if cv2.waitKey(5000) & 0xFF == ord('q'):
break
When capture() is called for the first time, self.capture_continuous.next() returns an up to date image. When calling capture() consecutively, it often happens that self.capture_continuous.next() does not return the latest image but one that is already a few seconds old (verified by pointing the camera at a clock). From time to time, it's even older than 10 seconds. On the other hand, sometimes self.capture_continuous.next() actually returns the latest image.
Since capture_continuous is an object of the type generator, my assumption is that it keeps generating camera images in the background that accumulate in a queue while the program waits and on the next call of self.capture_continuous.next() the next element in the queue is returned.
Anyway, I am only interested in the latest, most up to date image the camera has captured.
Some first attempts to get hold of the latest images failed. I tried to call self.capture_continuous.next() repeatedly in a while loop to get to the latest image.
Since a generator is apparently also an iterator I tried some methods mentioned in this post: Cleanest way to get last item from Python iterator.
Simply using the capture() function of the PiCamera class itself is not an option since it takes approx. 0.3 seconds till the image is captured what is too much for our use case.
Does anyone have a clue what might cause the delay described above and how it could be avoided?
I want to create a webcam streaming app that records webcam stream for, say about 30 seconds, and save it as myFile.wmv. Now To get live camera feed I know this code :-
import cv2
import numpy as np
c = cv2.VideoCapture(0)
while(1):
_,f = c.read()
cv2.imshow('e2',f)
if cv2.waitKey(5)==27:
break
cv2.destroyAllWindows()
But I have no idea off how to record for a given number of seconds and save it as a file in its current directory,
Please someone point me to the right direction
Thanks
ABOUT TIME
Why do not use the python time function? In particular time.time() Look at this answer about time in python
NB OpenCV should have (or had) its own timer but I can not tell you for sure if it works in current versions.
ABOUT RECORDING/SAVING
Look at this other answer
OpenCV allows you to record video, but not audio. There is this script I came across from JRodrigoF that uses openCV to record video and pyaudio to record audio. I used it for a while on a similar project; however, I noticed that sometimes the threads would hang and it would cause the program to crash. Another issue is that openCV does not capture video frames at a reliable rate and ffmpeg would distort the video when re-encoding.
https://github.com/JRodrigoF/AVrecordeR
I came up with a new solution that records much more reliably and with much higher quality. It presently only works for Windows because it uses pywinauto and the built-in Windows Camera app. The last bit of the script does some error-checking to confirm the video successfully recorded by checking the timestamp of the name of the video.
https://gist.github.com/mjdargen/956cc968864f38bfc4e20c9798c7d670
import pywinauto
import time
import subprocess
import os
import datetime
def win_record(duration):
subprocess.run('start microsoft.windows.camera:', shell=True) # open camera app
# focus window by getting handle using title and class name
# subprocess call opens camera and gets focus, but this provides alternate way
# t, c = 'Camera', 'ApplicationFrameWindow'
# handle = pywinauto.findwindows.find_windows(title=t, class_name=c)[0]
# # get app and window
# app = pywinauto.application.Application().connect(handle=handle)
# window = app.window(handle=handle)
# window.set_focus() # set focus
time.sleep(2) # have to sleep
# take control of camera window to take video
desktop = pywinauto.Desktop(backend="uia")
cam = desktop['Camera']
# cam.print_control_identifiers()
# make sure in video mode
if cam.child_window(title="Switch to Video mode", auto_id="CaptureButton_1", control_type="Button").exists():
cam.child_window(title="Switch to Video mode", auto_id="CaptureButton_1", control_type="Button").click()
time.sleep(1)
# start then stop video
cam.child_window(title="Take Video", auto_id="CaptureButton_1", control_type="Button").click()
time.sleep(duration+2)
cam.child_window(title="Stop taking Video", auto_id="CaptureButton_1", control_type="Button").click()
# retrieve vids from camera roll and sort
dir = 'C:/Users/michael.dargenio/Pictures/Camera Roll'
all_contents = list(os.listdir(dir))
vids = [f for f in all_contents if "_Pro.mp4" in f]
vids.sort()
vid = vids[-1]
# compute time difference
vid_time = vid.replace('WIN_', '').replace('_Pro.mp4', '')
vid_time = datetime.datetime.strptime(vid_time, '%Y%m%d_%H_%M_%S')
now = datetime.datetime.now()
diff = now - vid_time
# time different greater than 2 minutes, assume something wrong & quit
if diff.seconds > 120:
quit()
subprocess.run('Taskkill /IM WindowsCamera.exe /F', shell=True) # close camera app
print('Recorded successfully!')
win_record(2)