We Keep Coding

Less Than 1 Frame Switching Between Multiple Cameras in Python OpenCV2

Does anyone have better idea on how to quickly switch between an array of web cameras controlled by a python open CV2 program. I would like it to have no noticeable pause between switching cameras. I am willing to go to other languages, maybe C++?
What I have so far:
#%% import the opencv library
import cv2
from multiprocessing import Queue, Process, Manager
import time
import keyboard
import glob
import threading
import signal
import sys
#%%
baseVidList = glob.glob("/dev/video*")
baseVidList = baseVidList[:-2]
#%%
manager = Manager()
selectedCam = manager.Value('i', 0)
endSignal = manager.Value('i', 0)
qqq = Queue()
vidDict = {}
# define a video capture object
for device in baseVidList:
vidDict[device] = lambda: cv2.VideoCapture(f"{device}")
#%%
ejections = []
for device, capDevice in vidDict.items():
vidDict[device] = capDevice()
if vidDict[device].isOpened():
print(f"{device} Sucess")
vidDict[device].set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc('M', 'J', 'P', 'G'))
else:
print(f"{device} Failed, Ejecting")
vidDict[device].release()
ejections.append(device)
for eject in ejections:
vidDict.pop(eject)
#%%
font = cv2.FONT_HERSHEY_SIMPLEX
#### Functions ####
# Thread class
class VideoCaptureThread(threading.Thread):
def __init__(self, device, camID, endSignal):
super().__init__()
self.camID = camID
self.device = device
self.stopped = False
self.fps = 10
self.endSignal = endSignal
def run(self):
print("RUNNNN", self.endSignal != 1)
while self.endSignal != 1:
start_time = time.time()
print("S cam is:", selectedCam.value)
ret, frame = self.device.read()
if ret and selectedCam.value == self.camID:
# print(self.camID, xW, yH)
cv2.putText(frame,f"{self.camID}",(100,100),font,1,(255,0,0),1)
qqq.put(frame)
# Control the frame rate
sleep_time = 1/self.fps - (time.time() - start_time)
if sleep_time > 0:
time.sleep(sleep_time)
def stop(self):
self.stopped = True
self.device.release()
# Multiprocessing thread launch
def runCamera(device, camID):
# assert cap.isOpened()
# Usage
capture_thread = VideoCaptureThread(device, camID, endSignal) # src can be a filepath or a device index
capture_thread.start()
# # To stop the thread
# capture_thread.stop()
# Display Process
def runViewer():
while endSignal != 1:
# print("Qlen: ", qqq.qsize())
if qqq.empty() != True:
frame = qqq.get()
fy, fx, _ = frame.shape
frame = cv2.resize(frame,(int(2*fx),int(2*fy)))
cv2.imshow('frame', frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
while qqq.qsize() > 0: # Empty the Q
qqq.get()
# Signal Handler
def signal_handler(sig, arg2):
print("Exiting...")
endSignal.value = 1
sys.exit(0)
#### MAIN ####
if __name__ == '__main__':
signal.signal(signal.SIGINT, signal_handler)
processes = []
idx = 0
# Setup the cameras
for device, capDevice in vidDict.items():
if capDevice.isOpened():
processes.append(Process(target=runCamera, args=(capDevice,idx)))
idx = idx + 1
else:
print(f"Ejecting: {device}")
numProcesses = len(processes)
print(f"Processes: {numProcesses}")
processes.append(Process(target=runViewer))
# Launch the processes
for process in processes:
process.start()
# Keyboard control
while endSignal != 1:
if keyboard.is_pressed('d'): # if key 'q' is pressed
print('You Pressed d Key!')
selectedCam.value = (selectedCam.value + 1) % numProcesses
time.sleep(250/1000)
if keyboard.is_pressed('a'): # if key 'q' is pressed
print('You Pressed a Key!')
selectedCam.value = (selectedCam.value - 1) % numProcesses
time.sleep(250/1000)
# Graceful shutdown?
print("START END")
for process in processes:
process.join()
for device, capDevice in vidDict.items():
capDevice.release()
# Destroy all the windows
cv2.destroyAllWindows()
This code seems to support up to 5 cameras without issue. Extending to 10 sort of works but some of the cameras seem not to send data. The combination of threading and multiprocessing is an attempt to have the next frame ready for immediate addition to the queue and thereby there is no pause in switching cameras as would happen if we had to go through the whole cap setup and read process.
Any thoughts appreciated.

Run each camera capture in a separate thread by using multi-threading. In this method, while the other threads are capturing the following frame, the main thread can continue to process the currently shown frame.

Open CV RTSP camera buffer lag

I'm struggling to understand why I cant get a "LIVE" feed from my IP camera.
It appears that there is a buffer and it causes the frames to build up if not being read - and as each iteration of my code takes some time there is a backlog and it ends up being almost slow mo to whats actually happening.
I found the below code which triggers a thread to do the reading of the camera on a loop to try and avoid this. But now i'm getting a "LIVE" feed for around 5 frames and then it stalls and shows the same image for another few.
##camera class - this stops the RTSP feed getting caught in the buffer
class Camera:
def __init__(self, rtsp_link):
#init last ready and last frame
self.last_frame = None
self.last_ready = None
self.lock = Lock()
#set capture decive
capture = cv2.VideoCapture(rtsp_link,apiPreference=cv2.CAP_FFMPEG)
#set thread to clear buffer
thread = threading.Thread(target=self.rtsp_cam_buffer, args=(capture,), name="rtsp_read_thread")
thread.daemon = True
thread.start()
#delay start of next step to avoid errors
time.sleep(2)
def rtsp_cam_buffer(self, capture):
#loop forever
while True:
with self.lock:
capture.grab()
self.last_ready, self.last_frame = capture.retrieve()
def getFrame(self):
#get last frame
if (self.last_ready is not None) and (self.last_frame is not None):
return self.last_frame.copy())
else:
return None
Whats the correct thing to do in this situation? Is there a way round this?
OR
Should I use something like gstreamer or ffmpeg to get the camera feed? If so which is better and why? Any advice or pages to give me some python examples of getting it working? I couldn't find loads about that made sense to me.
thanks

After searching online through multiple resources the suggestion for using threads to remove frames from the buffer came up ALOT. And although it seemed to work for a while it caused me issues with duplicate frames being displayed for some reason that I could not work out.
I then tried to build opencv from source with gstreamer support but even once it was compiled correctly it still didn't seem to like interfacing with gstreamer correctly.
Eventually I thought the best bet was to go back down the threading approach but again couldnt get it working. So I gave multiprocessing a shot.
I wrote the below class to handle the camera connection:
import cv2
import time
import multiprocessing as mp
class Camera():
def __init__(self,rtsp_url):
#load pipe for data transmittion to the process
self.parent_conn, child_conn = mp.Pipe()
#load process
self.p = mp.Process(target=self.update, args=(child_conn,rtsp_url))
#start process
self.p.daemon = True
self.p.start()
def end(self):
#send closure request to process
self.parent_conn.send(2)
def update(self,conn,rtsp_url):
#load cam into seperate process
print("Cam Loading...")
cap = cv2.VideoCapture(rtsp_url,cv2.CAP_FFMPEG)
print("Cam Loaded...")
run = True
while run:
#grab frames from the buffer
cap.grab()
#recieve input data
rec_dat = conn.recv()
if rec_dat == 1:
#if frame requested
ret,frame = cap.read()
conn.send(frame)
elif rec_dat ==2:
#if close requested
cap.release()
run = False
print("Camera Connection Closed")
conn.close()
def get_frame(self,resize=None):
###used to grab frames from the cam connection process
##[resize] param : % of size reduction or increase i.e 0.65 for 35% reduction or 1.5 for a 50% increase
#send request
self.parent_conn.send(1)
frame = self.parent_conn.recv()
#reset request
self.parent_conn.send(0)
#resize if needed
if resize == None:
return frame
else:
return self.rescale_frame(frame,resize)
def rescale_frame(self,frame, percent=65):
return cv2.resize(frame,None,fx=percent,fy=percent)
Displaying the frames can be done as below
cam = Camera("rtsp://admin:[somepassword]#192.168.0.40/h264Preview_01_main")
print(f"Camera is alive?: {cam.p.is_alive()}")
while(1):
frame = cam.get_frame(0.65)
cv2.imshow("Feed",frame)
key = cv2.waitKey(1)
if key == 13: #13 is the Enter Key
break
cv2.destroyAllWindows()
cam.end()
This solution has resolved all my issues of buffer lag and also repeated frames. #
Hopefully it will help anyone else in the same situation.

Lewis's solution was helpful to reduce the lag so far but there was still some lag in my case, and I have found this gist, which is a bit faster:
import os
import sys
import time
import threading
import numpy as np
import cv2 as cv
# also acts (partly) like a cv.VideoCapture
class FreshestFrame(threading.Thread):
def __init__(self, capture, name='FreshestFrame'):
self.capture = capture
assert self.capture.isOpened()
# this lets the read() method block until there's a new frame
self.cond = threading.Condition()
# this allows us to stop the thread gracefully
self.running = False
# keeping the newest frame around
self.frame = None
# passing a sequence number allows read() to NOT block
# if the currently available one is exactly the one you ask for
self.latestnum = 0
# this is just for demo purposes
self.callback = None
super().__init__(name=name)
self.start()
def start(self):
self.running = True
super().start()
def release(self, timeout=None):
self.running = False
self.join(timeout=timeout)
self.capture.release()
def run(self):
counter = 0
while self.running:
# block for fresh frame
(rv, img) = self.capture.read()
assert rv
counter += 1
# publish the frame
with self.cond: # lock the condition for this operation
self.frame = img if rv else None
self.latestnum = counter
self.cond.notify_all()
if self.callback:
self.callback(img)
def read(self, wait=True, seqnumber=None, timeout=None):
# with no arguments (wait=True), it always blocks for a fresh frame
# with wait=False it returns the current frame immediately (polling)
# with a seqnumber, it blocks until that frame is available (or no wait at all)
# with timeout argument, may return an earlier frame;
# may even be (0,None) if nothing received yet
with self.cond:
if wait:
if seqnumber is None:
seqnumber = self.latestnum+1
if seqnumber < 1:
seqnumber = 1
rv = self.cond.wait_for(lambda: self.latestnum >= seqnumber, timeout=timeout)
if not rv:
return (self.latestnum, self.frame)
return (self.latestnum, self.frame)
And then you use it like:
# open some camera
cap = cv.VideoCapture('rtsp://URL')
cap.set(cv.CAP_PROP_FPS, 30)
# wrap it
fresh = FreshestFrame(cap)
Use fresh to deal with the open camera

Run dependent threads simultaneously in Python

I have two thread classes extract and detect.
Extract extracts frames from a video and stores it in a folder, Detect takes images from the folder where frames are extracted and detects objects.
But when I run the below code only the extract works:
global q
q = Queue()
class extract(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
print("T1")
cam = cv2.VideoCapture(video_name)
frameNum = 0
# isCaptured = True
frameCount = 0
while True:
isCapture, frame = cam.read()
if not isCapture:
break
if frameCount % 5 == 0:
frameNum = frameNum + 1
fileName = vid + str(frameNum) + '.jpg'
cv2.imwrite('images/extracted/' + fileName, frame)
q.put(fileName)
frameCount += 1
cam.release()
cv2.destroyAllWindows()
class detect(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
print("T2")
#logic to detect objects.
if __name__ == '__main__':
thread1 = extract()
thread1.start()
thread2 = detect()
thread2.start()
This prints only T1 and no T2.
I thought probably detect ran first and queue was empty so nothing happened so I added dummy entries into the queue and it ran how I wanted it to.
But it ran only for the dummy entries, it didn't work for the entries that the extract function added to the queue.
Looked up other questions and none of them seemed to solve the problem, hence posting this here

You probably want to keep your detect logic in infinite loop as well.
class detect(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
while True:
#detect frame
And if it is a single frame detection.
Then consider waiting in detect thread.
from time import sleep
class detect(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
sleep(120)
# Detect logic

Instead of waiting for a hardcoded time you can make use of Event() and make your detect thread wait until the Event() is set before performing the detection
If the event is set that means all tasks are done. Additionally, you also have to keep an eye on the queue if any tasks are there yet not processed.
I have written an example code to demonstrate how it works you can modify the code according to your needs.
Here extract takes 5 seconds to add a task to queue and detect checks for a task every 1 second. So if the extract is slower than also whenever something is available is will be processed by detect. And when all tasks are done detect will break out of the loop.
import threading
import queue
import time
global q
q = queue.Queue()
class extract(threading.Thread):
all_tasks_done = threading.Event()
def __init__(self):
threading.Thread.__init__(self)
def run(self):
counter = 5
while counter:
time.sleep(5)
counter -= 1
q.put(1)
print("added a task to queue")
extract.all_tasks_done.set()
class detect(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
def run(self):
while not extract.all_tasks_done.wait(1) or not q.empty():
print(q.get())
print("detection done")
#logic to detect objects.
if __name__ == '__main__':
thread1 = extract()
thread1.start()
thread2 = detect()
thread2.start()
thread1.join()
thread2.join()
print("work done")

My process finishes its `run` function, but it doesn't die

I'm subclassing multiprocessing.Process to create a class that will asynchronously grab images from a camera and push them to some queues for display and saving to disk.
The problem I'm having is that when I issue a stop command using a multiprocessing.Event object that belongs to the Process-descendant-object, the process successfully completes the last line of the run function, but then it doesn't die. The process just continues to exist and continues to return true from the is_alive function. I don't understand how this could be possible. What would cause a process to complete its run function but not die?
Maddeningly, when I separated this object from the larger context I'm using it in (which includes several other Process subclasses also running simultaneously), I can't reproduce the problem, which tends to make me think it has something to do with the rest of the code, but I don't understand how that could be - if it executed the last line of the run function, shouldn't it die regardless of what else is going on? I must be misunderstanding something about how a Process object works.
Here's the code below. When I run it, I see the message "Video acquires process STOPPED" printed out, but the process doesn't die.
class VideoAcquirer(mp.Process):
def __init__(self, camSerial, imageQueue, monitorImageQueue, acquireSettings={}, monitorFrameRate=15):
mp.Process.__init__(self, daemon=True)
self.camSerial = camSerial
self.acquireSettings = acquireSettings
self.imageQueue = imageQueue
self.monitorImageQueue = monitorImageQueue
self.videoFrequencyEntry.get()Rate = monitorFrameRate
self.stop = mp.Event()
def stopProcess(self):
print('Stopping video acquire process')
self.stop.set()
def run(self):
system = PySpin.System.GetInstance()
camList = system.GetCameras()
cam = camList.GetBySerial(self.camSerial)
cam.Init()
nodemap = cam.GetNodeMap()
setCameraAttributes(nodemap, self.acquireSettings)
cam.BeginAcquisition()
monitorFramePeriod = 1.0/self.monitorFrameRate
print("Video monitor frame period:", monitorFramePeriod)
lastTime = time.time()
k = 0
im = imp = imageResult = None
print("Image acquisition begins now!")
while not self.stop.is_set():
try:
# Retrieve next received image
print(1)
imageResult = cam.GetNextImage(100) # Timeout of 100 ms to allow for stopping process
print(2)
# Ensure image completion
if imageResult.IsIncomplete():
print('Image incomplete with image status %d...' % imageResult.GetImageStatus())
else:
# Print image information; height and width recorded in pixels
width = imageResult.GetWidth()
height = imageResult.GetHeight()
k = k + 1
print('Grabbed Image %d, width = %d, height = %d' % (k, width, height))
im = imageResult.Convert(PySpin.PixelFormat_Mono8, PySpin.HQ_LINEAR)
imp = PickleableImage(im.GetWidth(), im.GetHeight(), 0, 0, im.GetPixelFormat(), im.GetData())
self.imageQueue.put(imp)
# Put the occasional image in the monitor queue for the UI
thisTime = time.time()
if (thisTime - lastTime) >= monitorFramePeriod:
# print("Sent frame for monitoring")
self.monitorImageQueue.put((self.camSerial, imp))
lastTime = thisTime
imageResult.Release()
print(3)
except PySpin.SpinnakerException as ex:
pass # Hopefully this is just because there was no image in camera buffer
# print('Error: %s' % ex)
# traceback.print_exc()
# return False
# Send stop signal to write process
print(4)
self.imageQueue.put(None)
camList.Clear()
cam.EndAcquisition()
cam.DeInit()
print(5)
del cam
system.ReleaseInstance()
del nodemap
del imageResult
del im
del imp
del camList
del system
print("Video acquire process STOPPED")
I start the process from a tkinter GUI thread roughly like this:
import multiprocessing as mp
camSerial = '2318921'
queue = mp.Queue()
videoMonitorQueue = mp.Queue()
acquireSettings = [('AcquisitionMode', 'Continuous'), ('TriggerMode', 'Off'), ('TriggerSource', 'Line0'), ('TriggerMode', 'On')]
v = VideoAcquirer(camSerial, queue, videoMonitorQueue, acquireSettings=acquireSettings, monitorFrameRate=15)
And here's roughly how I stop the process, also from the tkinter GUI thread:
v.stopProcess()
Thanks for your help.

Loop not running with new variables when multi threading

I am currently working on my final year college project and I am stuck on what I think is a threading issue. I want to be able to run my method multiple times but each time it gets ran it should update a variable with new values. I am querying an API to get userID and then passing it into my main method by setting it as a global variable.
def setup():
try:
global gun_cascade, camera, frameRate, property_id, length, firstFrame, framecount,i,increment,start_Time,end_Time,statusCopy,userID
gun_cascade = cv2.CascadeClassifier('cascade.xml')
camera = cv2.VideoCapture('gun.mp4')
if camera.isOpened() == False:
print("Can't open video, isOpened is empty exiting now.")
exit(0)
frameRate = camera.get(5)
property_id = int(cv2.CAP_PROP_FRAME_COUNT)
length = int(cv2.VideoCapture.get(camera, property_id))
firstFrame = None
count = 0
gun_exist = False
increment = 0
start_Time = 0
end_Time = 0
i = 0
except Exception as e:
print(e)
exit(0)
Above I am setting userID to global
def decision():
setup()
user_object = listener.userIdListner()
tokenId = user_object.token
status = user_object.status
if user_object.status == "ON":
#status=statusCopy[:]
#aux = copy.deepcopy(matriz)
global statusCopy
statusCopy = copy.deepcopy(tokenId)
print("About to run mainscrt"+statusCopy)
#print(type(userID))
print(type(user_object))
post_object = listener.mainNullPostMethod()
post_object.null
print(post_object.null)
#opening a a new thread
Thread(target = main).start()
#post_object = listener.mainNullPostMethod()
#post_object.null
#print(post_object.null)
else:
print ("Program failed to run")
Here I am querying my API to get the userId and the status either on or off. At the moment this runs fine. But the problem is If this method is running and want to run it again with a new userID it works right up until the 'while camera.isOpened():' When I get to this point I get no error or anything
def main():
#printing out the userid to see if it's been changed
userID = statusCopy
print("The current userID is "+userID)
while isOpened:
framecount =0
framecount += 1
frameId = camera.get(1) #current frame number
(grabbed, frame) = camera.read()
if not grabbed:
break
# resize the frame, convert it to grayscale, and blur it
frame = imutils.resize(frame, width=500)
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
#gray = cv2.GaussianBlur(gray, (21, 21), 0)
#gray = cv2.dilate(gray, None, iterations=2)
#stuff to try in the future
#scaleFactor=1.1, minNeighbors=5, minSize=(30, 30), flags=cv2.CASCADE_SCALE_IMAGE, outputRejectLevels = True
gun = gun_cascade.detectMultiScale(gray, 3,5)
for (x,y,w,h) in gun:
randID = uuid.uuid4().hex
frame = cv2.rectangle(frame,(x,y),(x+w,y+h),(255,0,0),2)
roi_gray = gray[y:y+h, x:x+w]
roi_color = frame[y:y+h, x:x+w]
rects = gun[0]
neighbours = gun[0]
weights = gun[0]
if (frameId % math.floor(frameRate) == 1):
cv2.putText(frame, datetime.datetime.now().strftime("%A %d %B %Y %I:%M:%S%p"),(10, frame.shape[0] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.35, (255, 165, 0), 1)
cv2.imwrite('bin/' + userID+'-'+randID + '.jpg', frame)
if userID == "NULL":
print("failed due to user null")
break
print("working on pushing images to s3"+userID)
s3.uploadDirectory("bin/", "open-gun-recordings",userID)
picURL = "s3bucket.com/users/screenshots/"+userID+'/'+userID+'-'+randID+'.jpg'
text.fire(picURL)
cv2.imshow("Security Feed", frame)
key = cv2.waitKey(1) & 0xFF
camera.release()
cv2.destroyAllWindows()
Above I want to be able to have multiple instances of this method running at the same time and having a different userId for each instance.
if __name__ == '__main__':
#mainNullPostMethod()
#runDecision()
while True:
time.sleep(5)
decision()
Any help and suggestion would be greatly appreciated. I am not the best at python so apologies if this is a stupid question

First of all don't use global variables they are bad because it makes things difficult to track when changing from multiple functions (and in your case multiple threads) as described in this answer.
The problem I see is with initializing userID in the main function which you use to spawn threads, and the problem is that even though you initialize userID = statusCopy in main, even though you do a deepcopy in decision with statusCopy = copy.deepcopy(tokenId) it will still be overridden globally by any concurrent decision calls.
Let's imagine for a second that you call decision the first time, you initialize the userID and then you spawn a thread for main which makes use of that userID. Now I'm not sure how long it takes for main to execute but let's say for the sake of the argument that you wait for 5 seconds with the sleep and then do the whole thing again (while the first thread is still running). Now you basically change the userID with the second execution of the whole chain of functions and the first thread starts using the modified userID, which by definition is already bad practice since you wanted to use a specific userID information with your first thread. What I recommend is that you pass a deepcopy to the thread and initialize a local userID in the main, so that it does not get altered by concurrent threads.
Additionally, I'm not sure that you want to do a while True and spawn threads every 5 seconds, perhaps you should put a limit there also.