We Keep Coding

How to show a real-time graph while recording video with openCV?

I am developing a pose estimation application using opencv and movenet.
I am recording people's poses using camera and I want to display a graph in real time indicating the x and y co-ordinates of people's joints.
I have tried using FuncAnimation of matplotlib, but I am not getting a real-time graph. Moreover, I keep getting the following error:
UserWarning: Animation was deleted without rendering anything. This is most likely not intended. To prevent deletion, assign the Animation to a variable, e.g. anim, that exists until you have outputted the Animation using plt.show() or anim.save().
For now, I have tried plotting real time graph of coordinates of one of the joints of 1st person with help of below code, but I am only able to get separate graphs and not a real-time graph.
x_points=[]
y_points=[]
def draw_graph(x,y):
x_points.append(x)
y_points.append(y)
plt.cla()
plt.scatter(x,y)
plt.pause(0.5)
cap = cv2.VideoCapture(0)
while cap.isOpened():
ret, frame = cap.read()
# Resize image
img = frame.copy()
img = tf.image.resize_with_pad(tf.expand_dims(img, axis=0), 384,640)
input_img = tf.cast(img, dtype=tf.int32) #input image is represented as int 32
# Detection section
results = movenet(input_img)
keypoints_with_scores = results['output_0'].numpy()[:,:,:51].reshape((6,17,3))
#real time graph
anima = anim.FuncAnimation(plt.gcf(),draw_graph(keypoints_with_scores[0][0][1],keypoints_with_scores[0][0][0]),interval=0.5)
plt.show()
# Render keypoints
loop_through_people(frame, keypoints_with_scores, EDGES, 0.1) #confidence threshold is 0.1
cv2.imshow('MoveNet Lightning',frame) #rendering the img (frame)
if cv2.waitKey(10) & 0xFF==ord('q'): #if we press a key during the frame and the key is q then pose estimation of video/webcam is halted.
break
cap.release() #release webcam
cv2.destroyAllWindows() #close frame
These are the graphs that I am getting:
I want a single real time graph instead of these multiple graphs.

Global variable seems to update on its own

Firstly, please be nice as I'm only getting started with OpenCV in python and my python knowledge isn't spectacular.
I am trying to run OpenCV with numpy. I've made a white canvas where I keep track of the number of left-click events with count, so that I can
start drawing a circle with a mouseclick, or cv.EVENT_LBUTTONDOWN and count%2==1
let the radius grow at mouse movement with cv.EVENT_MOUSEMOVE and count%2==0 (and show the growing radius)
permanently "print" the circle on a second click with cv.EVENT_LBUTTONDOWN and count%2==0
Also, I have made two rectangles blue and green for color selection and I haven't allowed starting the circle within the bounds of these rectangles. Here, canvas is the continuously updated frame variable and frame is a matrix I use to refresh canvas when needed.
Now, in the output, when I drag my mouse, I expect the frames to refresh. So I let canvas=frame. In case of cv.EVENT_LBUTTONDOWN, I instead set frame=canvas so that it saves the final output I want to keep in the output frame.
However, as you can see in the outputs:
1.without clicking rectangles,
2.touching green and
3.touching both blue and green
it leaves a trace of the earlier frames(i.e it saves them to the frame). Not only that, I see the value of global variable frame changing automatically at cv.EVENT_MOUSEMOVE even though there isn't an assignment to frame in the callback function itself for that case! I see just one output with
frame and canvas aren't equivalent
in my termninal,just after the first click.
I am using OpenCV's setMouseCallback() routine for the callback.
Can anyone guide me on what I'm doing wrong here? I know using global variables like this is taboo but I was only guided by some lectures I found.
import cv2 as cv
import numpy as np
point=(0,0)
colour=(0,0,0)
line_width= 5
thickness=-1
count=1
radius=1
value=2**16-1
frame=value*np.ones((600,600,3),'uint16')
canvas=value*np.ones((600,600,3),'uint16')
def func(event,x,y,flags,param):
global point,radius,line_width,thickness,colour,count,frame,canvas
rect1=((x<=150 and x>=50) and (y>=50 and y<=150))
rect2=((x>=200 and x<=300) and (y>=50 and y<=150))
cv.rectangle(canvas,
(50,50),
(150,150),
color=(2**16-1,0,0),
thickness=-1)
cv.putText(canvas,
'Blue',
fontFace=cv.FONT_HERSHEY_SIMPLEX,
org=(70,70),
fontScale=0.5,
thickness=1,
color=(0,0,0))
cv.rectangle(canvas,
(200,50),
(300,150),
color=(0,2**16-1,0),
thickness=-1)
cv.putText(canvas,
'Green',
fontFace=cv.FONT_HERSHEY_SIMPLEX,
fontScale=0.5,
thickness=1,
org=(220,70),
color=(0,0,0))
if event==cv.EVENT_LBUTTONDOWN and rect1:
colour=(2**16-1,0,0)
elif event==cv.EVENT_LBUTTONDOWN and rect2:
colour=(0,2**16-1,0)
elif (event==cv.EVENT_LBUTTONDOWN) and count%2==1:
point=(x,y)
count=count+1
elif event==cv.EVENT_MOUSEMOVE and count%2==0:
if not np.array_equiv(canvas,frame):
print("frame and canvas aren't equivalent")
canvas=frame
radius=round(np.sqrt((x-point[0])**2+(y-point[1])**2))
cv.circle(canvas, point, radius,colour,line_width)
elif event==cv.EVENT_LBUTTONDOWN and count%2==0:
cv.circle(canvas, point, radius,colour,line_width)
count=count+1
frame=canvas
cv.namedWindow("Frame")
cv.setMouseCallback("Frame",func)
while True:
cv.imshow("Frame",canvas)
ch=cv.waitKey(1)
if ch & 0xFF == ord('q'):
break
cv.imwrite("Img.jpg",canvas)
cv.destroyAllWindows()

Add element to OLED display via PIL/Python without erasing rest

I have a SH1106 display connected to my Raspberry Pi that I'm controlling using luma.oled.
I can display all kind of content in different fonts, which is great. However, I can't figure out how to add something to what's currently being displayed without refreshing the whole display. My code is like this:
from os import system
import serial
from time import sleep
from luma.core.interface.serial import i2c
from luma.core.render import canvas
from luma.oled.device import sh1106
from PIL import ImageFont
# config display
device = sh1106(i2c(port=1, address=0x3C), rotate=0)
device.clear()
FA_solid = ImageFont.truetype('/home/pi/Desktop/tests/fa-solid-900.ttf', 16)
FA_regular = ImageFont.truetype('/home/pi/Desktop/tests/fa-regular-400.ttf', 16)
text_large = ImageFont.truetype('/home/pi/Desktop/tests/coolvetica condensed rg.ttf', 48)
text_small = ImageFont.truetype('/home/pi/Desktop/tests/coolvetica condensed rg.ttf', 16)
# display things
def show_icon(code):
with canvas(device) as draw:
draw.text((112, 0), text=code, font=FA_solid, fill="white")
def large_text(content, paddingleft =0, paddingtop =0):
with canvas(device) as draw:
draw.text((0, 0), text=content, font=text_large, fill="white")
def small_text(content, paddingleft =0, paddingtop =0):
with canvas(device) as draw:
draw.text((0, 0), text=content, font=text_small, fill="white")
show_icon("\uf124")
sleep(2)
large_text("Hi ;)")
sleep(10)
device.clear()
This display an icon from fontawesome in the upper right corner, then clears the screen and displays Hi. How can I change this to display the icon + hi? Ideally I'd have "zones" on the screen where I can change the icon zone while keeping the text displayed and vice versa. Thanks!
EDIT --------------------
Here's my code, adapted from Mark's answer below. Better but still not there yet. The Zones 1 and 3 stay the same while 2 is updated but when I redraw the screen, it is blank for half a second and then updates, which I don't want.
def UpdateDisplay(z1,z2,z3):
"""Pass in the three zones and they will be sent to the screen"""
device = sh1106(i2c(port=1, address=0x3C), rotate=0)
# Make a black canvas the size of the entire screen
whole = Image.new("1", (128,64))
# Now paste in the 3 zones to form the whole
whole.paste(z1, (2,2)) # zone1 at top-left
whole.paste(z2, (66,2)) # zone2 at top-right
whole.paste(z3, (2,34)) # zone3 across the bottom
# I save the image here, but you would write it to the screen with "device.display()"
device.display(whole)
return
# Make zone1 dark grey and annotate it
z1 = Image.new("1", (60,30))
z1draw = ImageDraw.Draw(z1)
z1draw.text((10,10),"Zone1", fill="white")
# Make zone2 mid-grey and annotate it
z2 = Image.new("1", (60,30))
z2draw = ImageDraw.Draw(z2)
z2draw.text((10,10),"Zone2", fill="white")
# Make zone3 light grey and annotate it
z3 = Image.new("1", (124,28))
z3draw = ImageDraw.Draw(z3)
z3draw.text((10,10),"Zone3", fill="white")
# Blit all zones to display
UpdateDisplay(z1,z2,z3)
sleep(5)
# Make zone2 mid-grey and annotate it
z2 = Image.new("1", (60,30))
z2draw = ImageDraw.Draw(z2)
z2draw.text((10,10),"Zone2 changed", fill="white")
UpdateDisplay(z1,z2,z3)

I don't have an SH1106 to test with and I have never used the luma library, so there may be a much simpler way of doing what you want. If so, maybe someone will kindly ping me and I'll delete this answer.
I have used PIL quite a lot, so I looked in here around line 28:
background = Image.new("RGB", device.size, "white")
background.paste(frame.resize(size, resample=Image.LANCZOS), posn)
device.display(background.convert(device.mode))
So, it seems you can create a PIL Image and send it to the display like that. The first line creates a blank white canvas the same size as the entire display, the second line pastes another PIL Image onto the canvas at the specified position and the last line sends the image to the display. So, all you need to do, is define your N "zones" separately and draw in them separately (each being a PIL Image), then when you want to update the display, paste your N zones in at the positions you want them and send the completed picture to the display.
Sorry I can't be more precise, but I have nothing to test with. Here's a little example with 3 zones that can be drawn individually and then assembled to a whole before calling device.display()
#!/usr/bin/env python3
from PIL import Image, ImageDraw
def UpdateDisplay(z1,z2,z3):
"""Pass in the three zones and they will be sent to the screen"""
# Make a black canvas the size of the entire screen
whole = Image.new("RGB", (128,64), (0,0,0))
# Now paste in the 3 zones to form the whole
whole.paste(z1, (2,2)) # zone1 at top-left
whole.paste(z2, (66,2)) # zone2 at top-right
whole.paste(z3, (2,34)) # zone3 across the bottom
# I save the image here, but you would write it to the screen with "device.display()"
whole.save('result.png')
return
# Make zone1 dark grey and annotate it
z1 = Image.new("RGB", (60,30), (64,64,64))
z1draw = ImageDraw.Draw(z1)
z1draw.text((10,10),"Zone1")
# Make zone2 mid-grey and annotate it
z2 = Image.new("RGB", (60,30), (128,128,128))
z2draw = ImageDraw.Draw(z2)
z2draw.text((10,10),"Zone2")
# Make zone3 light grey and annotate it
z3 = Image.new("RGB", (124,28), (192,192,192))
z3draw = ImageDraw.Draw(z3)
z3draw.text((10,10),"Zone3")
# Blit all zones to display
UpdateDisplay(z1,z2,z3)
# Now change just zone 2 and update display
z2.paste("red", (0,0,z2.width,z2.height))
UpdateDisplay(z1,z2,z3)
Here is the original display:
And here again after updating just zone2:

ok so I mostly figured it out:
from luma.core.interface.serial import i2c
from luma.core.render import canvas
from luma.oled.device import sh1106
from PIL import ImageFont, Image, ImageDraw
### setting up display using LUMA oled
device = sh1106(i2c(port=1, address=0x3C), rotate=0)
device.clear()
### Initialize drawing zone (aka entire screen)
output = Image.new("1", (128,64))
add_to_image = ImageDraw.Draw(output)
### I have the exterior temp and altitude I want to display. Each has an assigned zone for the icon (FontAwesome) and the data
# temp_ext
temp_zone = [(14,44), (36,64)]
temp_start = (14,44)
temp_icon_zone = [(0,48), (15,64)]
temp_icon_start = (3,48)
add_to_image.text(temp_icon_start, "\uf2c9", font=FA_solid, fill="white")
### every time I have a new reading, I basically draw a black rectangle over what I had and the rewrite the text
add_to_image.rectangle(temp_zone, fill="black", outline = "black")
add_to_image.text(temp_start, str(temp_c), font=text_medium, fill="white")
device.display(output)
This enables me to only update the part of the screen I want, leaving the rest as is and, crucially, not having a blank screen for half a second when rewriting info. Feel free to suggest optimizations!
I still need to look into memory usage, it feels kinda sluggish when the different zones are updating at once. But it works!

How can I merge these two functions into a single one ? [python 3.]

It has been several weeks since I started learning python (via anaconda). I have started to develop my personal code for some applications for my work.
So the question is the next one, I have this two function draw_rectangle and magnification which each one uses a different mouse_handler.
As you can see in the code, both functions are the same, so what I want to do is to merge them into just one function (trying to make a better code structure).
Finally, I call these functions in another script via:
data['image_right'], rectangle['right'] = draw_rectangle(data['image_right'])
magnification = magnification(data['image_left'])
Here are the functions :
def mouse_handler(action, x, y, flags,img) :
# Action to be taken when left mouse button is pressed
if action==cv2.EVENT_LBUTTONDOWN:
rectangle.append([x,y])
# Action to be taken when left mouse button is released
elif action==cv2.EVENT_LBUTTONUP:
rectangle.append([x,y])
def draw_rectangle(img):
global rectangle
rectangle = []
#Create the window to show image
cv2.namedWindow("Image",cv2.WINDOW_NORMAL)
cv2.setMouseCallback("Image", mouse_handler, img)
cv2.imshow("Image", img)
cv2.waitKey(0)
cv2.destroyAllWindows()
img = img[rectangle[0][1]:rectangle[1][1],rectangle[0][0]:rectangle[1][0]]
return img, rectangle
def mouse_handler2(action, x, y, flags,img) :
# Action to be taken when left mouse button is pressed
if action==cv2.EVENT_LBUTTONDOWN:
coordinates.append([x,y])
# Action to be taken when left mouse button is released
elif action==cv2.EVENT_LBUTTONUP:
coordinates.append([x,y])
def magnification(img):
global coordinates
coordinates = []
imgcopy = img.copy()
#Create the window to show image
cv2.namedWindow("Image",cv2.WINDOW_NORMAL)
cv2.setMouseCallback("Image", mouse_handler2, imgcopy)
cv2.putText(imgcopy, "Click/unclick", (int(imgcopy.shape[1]/4), int(imgcopy.shape[0]/2)), cv2.FONT_HERSHEY_SIMPLEX, 3, (0, 0, 0), 3)
cv2.imshow("Image", imgcopy)
cv2.waitKey(0)
cv2.destroyAllWindows()
magnification = 10/np.sqrt( np.power((coordinates[0][1]-coordinates[1][1]),2)+ np.power((coordinates[0][0]-coordinates[1][0]),2))
return magnification

Psychopy: draw all images from a list at once

I have a list with 12 images. All of them have different locations (that is, they do not overlap at all). I want to draw them all at once. In other words, I want to see all 12 pictures at the same time on the screen. So far, I only got this:
lines = [line1,line2,line3,line4,line5,line6
line7,line8,line9,line10,line11,line12]
for i in range(12):
lines[i].draw()
But, of course, this code only draws one pic at a time, after I press any key. Is there then a way to draw the 12 pics at the same time?
Thanks in advance!

Your original code only drew one image at a time because of how the loop was set up -- it was (more or less) saying "For each element in circles, draw several things and flip the front and back buffers". Each time the buffers flip, unless you tell it otherwise via win.flip(clearBuffer = False), the previous things on the screen are removed. To draw the images at the same time, you could just loop through the image list and call the draw() method on each element, e.g.:
for i in imglist:
i.draw()
win.flip()
If you are willing to cede control over properties of individual images, one way would be to use BufferImageStim. This takes longer to initialize, but may be faster than drawing individual images (I haven't timed it properly). Both methods are demonstrated below.
from psychopy import visual, event, core
import urllib
import random
win = visual.Window([400, 400], fullscr = False)
# picture of a cat, save to file
urllib.urlretrieve('https://s-media-cache-ak0.pinimg.com/736x/' +
'07/c3/45/07c345d0eca11d0bc97c894751ba1b46.jpg', 'tmp.jpg')
# create five images with (probably) unique positions
imglist = [visual.ImageStim(win = win, image = 'tmp.jpg',
size = (.2, .2),
pos = ((random.random() - 0.5) * 2,
(random.random() - 0.5) * 2))
for i in xrange(5)]
# draw individual images
for i in imglist:
i.draw()
win.flip()
# wait for key press, then clear window
event.waitKeys()
win.flip()
core.wait(0.5)
# create aggregate stimulus (should look identical)
buffs = visual.BufferImageStim(win, stim = imglist)
buffs.draw()
win.flip()
event.waitKeys()
core.quit()