We Keep Coding

How do you run two programs in python simultaneously at different frame rates?

I have been experimenting with OpenCV in python and I want to make a game with it in pygame. But it is way too slow, OpenCV is running at 8 fps but I want the pygame game to be running at 60 fps. I don't need the tracking to be fast but I want pygame to be running at a normal frame rate. I have tried to use the built-in sockets in python but it would make both programs run at 8 fps. Then I tried to stop the socket from pausing at waiting for data but changed nothing. Thanks.
pose.py
import cv2
import mediapipe as mp
import time
import numpy as np
cap = cv2.VideoCapture(0)
mpHands = mp.solutions.hands
hands = mpHands.Hands(static_image_mode=False,
max_num_hands=8,
min_detection_confidence=0.5,
min_tracking_confidence=0.5)
mpDraw = mp.solutions.drawing_utils
pTime = 0
cTime = 0
while True:
success, img = cap.read()
imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
results = hands.process(imgRGB)
#print(results.multi_hand_landmarks)
if results.multi_hand_landmarks:
handId = 0
for handLms in results.multi_hand_landmarks:
avX = avY = l = 0
for id, lm in enumerate(handLms.landmark):
#print(id,lm)
h, w, c = img.shape
l += 1
cx, cy = int(lm.x *w), int(lm.y*h)
avX += cx
avY += cy
#if id ==0:
cv2.circle(img, (cx,cy), 3, (255,0,2555), cv2.FILLED)
avX = avX / l
avY = avY / l
mpDraw.draw_landmarks(img, handLms, mpHands.HAND_CONNECTIONS)
handId += 1
cTime = time.time()
fps = 1/(cTime-pTime)
pTime = cTime
cv2.putText(img,str(int(fps)), (10,70), cv2.FONT_HERSHEY_PLAIN, 3, (255,0,255), 3)
#cv2.imshow("Image", img)
cv2.waitKey(1)
poseGame.py
import pygame
pygame.init()
screen = pygame.display.set_mode((960, 540))
clock = pygame.time.Clock()
while True:
screen.fill((0, 0, 0))
# ...
clock.tick(60)
pygame.display.update()

you have three options.
first option is to have the game or the app running as the main python application and the other one running as a child process using the multiprocessing module and use a manager.queue for communication between the two processes, where the game will check if there is something in the queue on each frame (and pop the message if it finds one).
second (harder) option is to use sockets in non-blocking mode using socket.setblocking, so the game will not block until the other app finishes the transmission, but you will need to incrementally update an internal "message buffer" on each frame of the game as the messages will be received on multiple frames, the advantage of this approach is that the game and the commanding application will be two separate applications with no shared code, so the communication doesn't even have to be within python.
third option is to employ threading with blocking sockets where a child thread will be monitoring the received messages from the other application and putting it on a deque or the main thread to read.
one disadvantage of both approaches is that the application has to constantly send messages to the game, as both applications will be running at different rates, therefore the game needs to save the last message and keep using it until it receives the next message.

You need threads.
Do not mess around with sockets for this, it's needlessly complicated. You also don't necessarily need multiprocessing. Threads are just fine. All the heavy lifting (OpenCV, mediapipe, ...) will release the GIL. If you don't know what that means, it's okay. I'm saying that for those that think threading isn't okay to use.
You need a thread for the mediapipe stuff and another thread for reading the camera. And you can do your game logic in the main thread (the one you didn't start yourself).
The thread for the camera reading is required because if you don't read from the camera as quickly as it can produce frames (say you're processing in the same loop), produced frames will queue up in the driver and you'll get latency. Have this thread read from the camera and keep the most recent frame (in a variable), and do nothing else.
Whenever the mediapipe thread is ready for (more) work, it gets the current frame from the camera thread. Do that by simply accessing the variable. It can then work and send its conclusions to the game, either by setting variables or by posting to Queue objects (python comes with a queue module). When reading from a queue that may not contain an object, but you don't wanna block until there is an element added to it, you need to specify a zero timeout in the "get" call and catch the "empty" exception.
Here's a code skeleton. It's simple. I won't show you how to use classes. Global variables are fine for this demo. However, I'll show you how to use a Condition object, to wait for fresh frames and signal a fresh frame. No, this is not about guarding access (although it does that too), it's about making sure you actually have new data to process.
import threading
import numpy as np
import cv2 as cv
# and pygame? and mediapipe?
running = True # global flag. when False, threads end
latest_frame = None
latest_frame_available = threading.Condition()
def camera_worker_function():
global latest_frame
camera = cv.VideoCapture(0)
assert camera.isOpened()
while running:
(rv, frame) = camera.read()
if not rv: break
with latest_frame_available:
latest_frame = frame
latest_frame_available.notify_all()
camera.release()
def mediapipe_worker_function():
# globals for values you need in the game
while running:
with latest_frame_available:
if latest_frame_available.wait(timeout=1.0):
frame = latest_frame
else: # False means timeout
continue # -> recheck `running`
if frame is None: break # sentinel value to shut down
# do whatever you need here, using `frame`
...
camera_thread = threading.Thread(target=camera_worker_function)
camera_thread.start()
mediapipe_thread = threading.Thread(target=mediapipe_worker_function)
mediapipe_thread.start()
... # game logic
# end threads
running = False
with latest_frame_available:
latest_frame = None # sentinel value
latest_frame_available.notify_all()
mediapipe_thread.join()
camera_thread.join()
I didn't test this code. Take it as guidance.

Python cv2 VideoCapture() in daemon thread turns black randomly

I have a webcam that captures something I want to be able to monitor live. Every 30 seconds an operation needs to be done on the most recent frame of the video. To do this, I've set up a cv2.NamedWindow() in a daemon thread that allows me to see the webcam feed. Then using a scheduler, I can do the operation I need every 30 seconds. The problem is, the camera feed will randomly go black, and save an all black image. Sometimes this happens after 15 minutes of operation, sometimes after 4 hours of operation., so it's very inconsistent and there seems to be no pattern. Here is the code:
import time
import gc
import cv2
import threading
import schedule
import numpy
class operator():
def __init__(self):
print("Start")
def start_cam(self):
is_blurry = True
while is_blurry == True:
print("Start of start_cam")
self.camera = cv2.VideoCapture(0, cv2.CAP_DSHOW) # Make camera object with correct format, size, and autofocus. All of these appear to be necessary to get 4K resolution in OpenCV
self.camera.set(cv2.CAP_PROP_FRAME_WIDTH, 3840) #3840
self.camera.set(cv2.CAP_PROP_FRAME_HEIGHT, 2160) #2160
self.camera.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('m','j','p','g'))
self.camera.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter.fourcc('M','J','P','G'))
self.camera.set(cv2.CAP_PROP_AUTOFOCUS, 1)
self.camera.set(cv2.CAP_PROP_FPS, 60)
i=10
while i>0: # Takes 10 pictures and waits 1 sec in between each. This is done to make sure it has time to autofocus
i=i-1
cv2.waitKey(1000) #1000
ret, self.frame = self.camera.read()
if ret != True:
print("image error") # Check to make sure it actually took a picture and didn't fail
if ret != True and i == 1:
restart_loop = True
print("Image error, restarting loop") #
continue
laplace = cv2.Laplacian(self.frame, cv2.CV_64F).var() # Checks if image is blurry. If it is, it restarts the camera.
print(laplace)
if laplace < 20: # Blurry
print("Blurry image, reinitializing")
self.camera.release()
del(self.camera)
cv2.destroyAllWindows()
gc.collect()
time.sleep(2)
else:
print("image not blurry")
break
def live_loop(self):
loop_bool = True
cv2.namedWindow("Live Feed", cv2.WINDOW_NORMAL) # Creates a normal window so that the 4k image is scaled correctly for the live feed (I have a relatively small monitor)
while loop_bool == True:
ret, self.frame = self.camera.read()
if ret != True:
print("image error")
break
self.frame_rotate = cv2.rotate(self.frame, cv2.ROTATE_180)
cv2.imshow("Live Feed", self.frame_rotate)
k = cv2.waitKey(10)
gc.collect()
def data_operation(self):
print("Start data operation")
imgpath = "path where image is saved"
cv2.imwrite(imgpath, self.frame)
t = time.localtime() # If the image can't be read as a number, saves the image with a timestamp so that it can be examined later
timestamp = time.strftime('%b-%d-%Y_%H;%M', t)
print(timestamp)
if __name__== "__main__":
op = operator()
op.start_cam()
x = threading.Thread(target=op.live_loop, daemon = True)
x.start()
schedule.every(30).seconds.do(op.data_operation)
try:
while True:
schedule.run_pending()
time.sleep(1)
except KeyboardInterrupt:
print("Ended Program")
gc.collect()
This is the smallest amount of code I can reproduce the issue with. There is more going on just this, but this code block has been run and the issue persists. I have found that when I remove the line that saves the image in the data_operation function, which is the cv2.imwrite(imgpath, self.frame), the program seems to work and the camera never goes black and the issue is gone. So I think that is the problem, but I don't know how else to save the frame as an image every 30 seconds while also keeping the live feed up and running. As far as I know, the rest of the code works perfectly except for this, and I do need to be able to save and access the image.
I am not a programmer by trade so I've just been figuring this out as I go. If something is glaringly stupid let me know. If there is a much easier/safer way to display a live webcam feed while also automatically grabbing a picture every 30 seconds and doing stuff with that picture while keeping the live loop going, I would be happy to hear that too.
Any help would be appreciated, and let me know if you have further questions.
EDIT: I have updated the code to simplify and continue shrinking the smallest reproduceable sample. I realized the start_cam function was unnecessary, as all of the cv2 initialization can be done at the start of the live_loop function. The issue persists. I have tried adding a mutex, however that seems to not help (though it's possible I am implementing it wrong). Any info or working examples would be greatly appreciated.
EDIT 2: Another thing to note is that the exact same program running on my laptop (using the built-in webcam rather than the 4k Logitech Brio) doe not seem to have the same issue. Is the frame rate or resolution causing the issue perhaps? Testing is difficult since I need to keep a working program running while working on this one, but I will do more tests when I am able and keep this post updated if anything develops. Still, feel free to toss any ideas you have to me.

OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?

Goal and problem
I'd like to set up an opencv system to process either HLS streams or RMTP streams, however, I am running into a strange issue regarding a reduced frame-rate and an accumulating lag. It's as if the video gets further and further behind from where it is supposed to be in the stream.
I'm looking for a way to keep up to date with a live source even if it means dropping frames.
Current approach
import cv2
cap = cv2.VideoCapture()
cap.open('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8')
while (True):
_, frame = cap.read()
cv2.imshow("camCapture", frame)
cv2.waitKey(1)
I've validated the quality of the stream on VLC and it seems to work fine there.
cv2 speed
.
realistic/expected speed
Questions:
What am I doing wrong here?
Why is it so slow?
How do I sync it to real-time speeds?

My hypothesis is that the jitter is most likely due to network limitations and occurs when a frame packet is dropped. When a frame is dropped, this causes the program to display the last "good" frame which results in the display freezing. This is probably a hardware or bandwidth issue but we can alleviate some of this with software. Here are some possible changes:
1. Set maximum buffer size
We set the cv2.videoCapture() object to have a limited buffer size with the cv2.CAP_PROP_BUFFERSIZE parameter. The idea is that by limiting the buffer, we will always have the latest frame. This can also help to alleviate the problem of frames randomly jumping ahead.
2. Set frame retrieval delay
Currently, I believe the read() is reading too fast even though it is in its own dedicated thread. This may be one reason why all the frames appear to pool up and suddenly burst in the next frame. For instance, say in a one second time interval, it may produce 15 new frames but in the next one second interval, only 3 frames are returned. This may be due to the network packet frame loss so to ensure that we obtain constant frame rates, we simply add a delay in the frame retrieval thread. A delay to obtain roughly ~30 FPS does a good job to "normalize" the frame rate and smooth the transition between frames incase there is packet loss.
Note: We should try to match the frame rate of the stream but I'm not sure what the FPS of the webcam is so I just guessed 30 FPS. Also, there is usually a "direct" stream link instead of going through a intermediate webserver which can greatly improve performance.
If you try using a saved .mp4 video file, you will notice that there is no jitter. This confirms my suspicion that the problem is most likely due to network latency.
from threading import Thread
import cv2, time
class ThreadedCamera(object):
def __init__(self, src=0):
self.capture = cv2.VideoCapture(src)
self.capture.set(cv2.CAP_PROP_BUFFERSIZE, 2)
# FPS = 1/X
# X = desired FPS
self.FPS = 1/30
self.FPS_MS = int(self.FPS * 1000)
# Start frame retrieval thread
self.thread = Thread(target=self.update, args=())
self.thread.daemon = True
self.thread.start()
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
time.sleep(self.FPS)
def show_frame(self):
cv2.imshow('frame', self.frame)
cv2.waitKey(self.FPS_MS)
if __name__ == '__main__':
src = 'https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8'
threaded_camera = ThreadedCamera(src)
while True:
try:
threaded_camera.show_frame()
except AttributeError:
pass
Related camera/IP/RTSP/streaming, FPS, video, threading, and multiprocessing posts
Python OpenCV streaming from camera - multithreading, timestamps
Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture
How to capture multiple camera streams with OpenCV?
OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?
Storing RTSP stream as video file with OpenCV VideoWriter
OpenCV video saving
Python OpenCV multiprocessing cv2.VideoCapture mp4

Attempt at threading
I've attempted this solution from nathancy with minor success.
It involves:
creating a separate thread for image capture from the source
using the main thread exclusively for display.
Code:
import cv2
from threading import Thread
class ThreadedCamera(object):
def __init__(self, source = 0):
self.capture = cv2.VideoCapture(source)
self.thread = Thread(target = self.update, args = ())
self.thread.daemon = True
self.thread.start()
self.status = False
self.frame = None
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
def grab_frame(self):
if self.status:
return self.frame
return None
if __name__ == '__main__':
stream_link = "https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8"
streamer = ThreadedCamera(stream_link)
while True:
frame = streamer.grab_frame()
if frame is not None:
cv2.imshow("Context", frame)
cv2.waitKey(1)
Jittery, but real-time results
.
The streaming works. It maintains real-time. However, it is as if all the frames pool up and suddenly burst into the video. I would like somebody to explain that.
Room for improvement
The real-time stream can be found here.
https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet
This site is scraped for the m3u8 using python's streamlink stream scraper.
import streamlink
streams = streamlink.streams("https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet")
print(streams)
which yeilds:
OrderedDict([
('720p',<HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('live', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet','swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>),
('worst', <HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('best', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet', 'swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>)
])
The possibility that the streams are being read wrong.

I would suggest double checking the compatible video stream codecs with the hardware. I ran into the same issue, frame rate dropped to 5 fps only during streaming, because it was defaulting to a format that is not being streamed so it would convert it then display very lagged (~1s) with lower fps as well.
use Self.capture.set(cv2.CAP_PROP_FOURCC ,cv2.VideoWriter_fourcc('M', 'J', 'P', 'G') ) with the proper codec in place of MJPG and with your cv2.VideoCapture and see if that helps.

Increase Matplotlib graphing and OpenCV video processing performance [duplicate]

Goal and problem
I'd like to set up an opencv system to process either HLS streams or RMTP streams, however, I am running into a strange issue regarding a reduced frame-rate and an accumulating lag. It's as if the video gets further and further behind from where it is supposed to be in the stream.
I'm looking for a way to keep up to date with a live source even if it means dropping frames.
Current approach
import cv2
cap = cv2.VideoCapture()
cap.open('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8')
while (True):
_, frame = cap.read()
cv2.imshow("camCapture", frame)
cv2.waitKey(1)
I've validated the quality of the stream on VLC and it seems to work fine there.
cv2 speed
.
realistic/expected speed
Questions:
What am I doing wrong here?
Why is it so slow?
How do I sync it to real-time speeds?

My hypothesis is that the jitter is most likely due to network limitations and occurs when a frame packet is dropped. When a frame is dropped, this causes the program to display the last "good" frame which results in the display freezing. This is probably a hardware or bandwidth issue but we can alleviate some of this with software. Here are some possible changes:
1. Set maximum buffer size
We set the cv2.videoCapture() object to have a limited buffer size with the cv2.CAP_PROP_BUFFERSIZE parameter. The idea is that by limiting the buffer, we will always have the latest frame. This can also help to alleviate the problem of frames randomly jumping ahead.
2. Set frame retrieval delay
Currently, I believe the read() is reading too fast even though it is in its own dedicated thread. This may be one reason why all the frames appear to pool up and suddenly burst in the next frame. For instance, say in a one second time interval, it may produce 15 new frames but in the next one second interval, only 3 frames are returned. This may be due to the network packet frame loss so to ensure that we obtain constant frame rates, we simply add a delay in the frame retrieval thread. A delay to obtain roughly ~30 FPS does a good job to "normalize" the frame rate and smooth the transition between frames incase there is packet loss.
Note: We should try to match the frame rate of the stream but I'm not sure what the FPS of the webcam is so I just guessed 30 FPS. Also, there is usually a "direct" stream link instead of going through a intermediate webserver which can greatly improve performance.
If you try using a saved .mp4 video file, you will notice that there is no jitter. This confirms my suspicion that the problem is most likely due to network latency.
from threading import Thread
import cv2, time
class ThreadedCamera(object):
def __init__(self, src=0):
self.capture = cv2.VideoCapture(src)
self.capture.set(cv2.CAP_PROP_BUFFERSIZE, 2)
# FPS = 1/X
# X = desired FPS
self.FPS = 1/30
self.FPS_MS = int(self.FPS * 1000)
# Start frame retrieval thread
self.thread = Thread(target=self.update, args=())
self.thread.daemon = True
self.thread.start()
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
time.sleep(self.FPS)
def show_frame(self):
cv2.imshow('frame', self.frame)
cv2.waitKey(self.FPS_MS)
if __name__ == '__main__':
src = 'https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8'
threaded_camera = ThreadedCamera(src)
while True:
try:
threaded_camera.show_frame()
except AttributeError:
pass
Related camera/IP/RTSP/streaming, FPS, video, threading, and multiprocessing posts
Python OpenCV streaming from camera - multithreading, timestamps
Video Streaming from IP Camera in Python Using OpenCV cv2.VideoCapture
How to capture multiple camera streams with OpenCV?
OpenCV real time streaming video capture is slow. How to drop frames or get synced with real time?
Storing RTSP stream as video file with OpenCV VideoWriter
OpenCV video saving
Python OpenCV multiprocessing cv2.VideoCapture mp4

Attempt at threading
I've attempted this solution from nathancy with minor success.
It involves:
creating a separate thread for image capture from the source
using the main thread exclusively for display.
Code:
import cv2
from threading import Thread
class ThreadedCamera(object):
def __init__(self, source = 0):
self.capture = cv2.VideoCapture(source)
self.thread = Thread(target = self.update, args = ())
self.thread.daemon = True
self.thread.start()
self.status = False
self.frame = None
def update(self):
while True:
if self.capture.isOpened():
(self.status, self.frame) = self.capture.read()
def grab_frame(self):
if self.status:
return self.frame
return None
if __name__ == '__main__':
stream_link = "https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w1421640637.m3u8"
streamer = ThreadedCamera(stream_link)
while True:
frame = streamer.grab_frame()
if frame is not None:
cv2.imshow("Context", frame)
cv2.waitKey(1)
Jittery, but real-time results
.
The streaming works. It maintains real-time. However, it is as if all the frames pool up and suddenly burst into the video. I would like somebody to explain that.
Room for improvement
The real-time stream can be found here.
https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet
This site is scraped for the m3u8 using python's streamlink stream scraper.
import streamlink
streams = streamlink.streams("https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet")
print(streams)
which yeilds:
OrderedDict([
('720p',<HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('live', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet','swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>),
('worst', <HLSStream('https://videos3.earthcam.com/fecnetwork/9974.flv/chunklist_w202109066.m3u8')>),
('best', <RTMPStream({'rtmp': 'rtmp://videos3.earthcam.com/fecnetwork/', 'playpath': '9974.flv', 'pageUrl': 'https://www.earthcam.com/usa/newyork/timessquare/?cam=tsstreet', 'swfUrl': 'http://static.earthcam.com/swf/streaming/stream_viewer_v3.swf', 'live': 'true'}, redirect=False>)
])
The possibility that the streams are being read wrong.

I would suggest double checking the compatible video stream codecs with the hardware. I ran into the same issue, frame rate dropped to 5 fps only during streaming, because it was defaulting to a format that is not being streamed so it would convert it then display very lagged (~1s) with lower fps as well.
use Self.capture.set(cv2.CAP_PROP_FOURCC ,cv2.VideoWriter_fourcc('M', 'J', 'P', 'G') ) with the proper codec in place of MJPG and with your cv2.VideoCapture and see if that helps.

Capture Raspicam image via PiCamera module - How to avoid old images?

I am working on a project where we are using the Raspicam attached to a Raspberry Pi to capture (and process) images with python using the PiCamera module.
With our current implementation I am experiencing an unexpected behaviour.
The camera is configured to capture images with 15 frames per second. To simulate processing time the program waits 5 seconds.
Minimal example:
#!/usr/bin/env python
import cv2
from picamera import PiCamera
from picamera.array import PiRGBArray
class RaspiCamera:
def __init__(self, width, height, framerate):
self.camera = PiCamera()
self.camera.resolution = (width, height)
self.camera.framerate = framerate
self.rawCapture = PiRGBArray(self.camera, size=self.camera.resolution)
self.capture_continuous = self.camera.capture_continuous(self.rawCapture, format="bgr", use_video_port=True)
def capture(self):
frame = self.capture_continuous.next()
image = self.rawCapture.array
self.rawCapture.truncate(0)
return image
if __name__ == "__main__":
camera = RaspiCamera(640, 480, 15)
while True:
frame = camera.capture()
cv2.imshow("Image", frame)
if cv2.waitKey(5000) & 0xFF == ord('q'):
break
When capture() is called for the first time, self.capture_continuous.next() returns an up to date image. When calling capture() consecutively, it often happens that self.capture_continuous.next() does not return the latest image but one that is already a few seconds old (verified by pointing the camera at a clock). From time to time, it's even older than 10 seconds. On the other hand, sometimes self.capture_continuous.next() actually returns the latest image.
Since capture_continuous is an object of the type generator, my assumption is that it keeps generating camera images in the background that accumulate in a queue while the program waits and on the next call of self.capture_continuous.next() the next element in the queue is returned.
Anyway, I am only interested in the latest, most up to date image the camera has captured.
Some first attempts to get hold of the latest images failed. I tried to call self.capture_continuous.next() repeatedly in a while loop to get to the latest image.
Since a generator is apparently also an iterator I tried some methods mentioned in this post: Cleanest way to get last item from Python iterator.
Simply using the capture() function of the PiCamera class itself is not an option since it takes approx. 0.3 seconds till the image is captured what is too much for our use case.
Does anyone have a clue what might cause the delay described above and how it could be avoided?