Optimise Performance - OpenCV and Threads with Python - python

I am having a problem related to performance using OpenCV3.2 with Python. I have just integrated another subsystem to my main program and it slowed down a lot.
This is my initial code without integrating the new subsystem, I am using cv2.getTickCount to measure time, as suggested by OpenCv3.2 Official Website.
# Infinite loop
while True:
# getting tick count
e1 = cv2.getTickCount()
# storing frame
_, img = cap.read()
# define red colours in the screen
findRedColours(img, board)
# getting tick count after the functions
e2 = cv2.getTickCount()
# calculating time
t = (e2 - e1) / cv2.getTickFrequency()
# print time
print(t)
# check if img is none
if img is not None:
# omitted code
k = cv2.waitKey(20) & 0xFF
# start the game, hide info
if (k == ord('s') or k == ord('S')) and start is False:
# create new thread to play game
t = Thread(target=playGame)
t.start()
Basically, I am calling a function in the infinite loop to find red colours and by pressing start I create the Thread and the game starts.
This is the time needed before I press 'S' to create the Thread:
0.019336862
0.016924178
0.022487864
This is the time needed after I press 'S' to create the Thread:
0.091731532
0.125760734
0.098221829
Here everything works fine, there is a light change in the time, but nothing too important. I start to have problems when I add my new subsystem. Here the following code with the integration of the new system, it is the same of the previous one, it is just a function call that changes:
# Infinite loop
while True:
# getting tick count
e1 = cv2.getTickCount()
# storing frame
_, img = cap.read()
# extract grid
gridExtractor.extractGrid(img)
# define red colours in the screen
findRedColours(img, board)
# getting tick count after the functions
e2 = cv2.getTickCount()
# calculating time
t = (e2 - e1) / cv2.getTickFrequency()
# print time
print(t)
# check if img is none
if img is not None:
# omitted code
k = cv2.waitKey(20) & 0xFF
# start the game, hide info
if (k == ord('s') or k == ord('S')) and start is False:
# create new thread to play game
t = Thread(target=playGame)
t.start()
and this is the time before I create the Thread:
0.045629524
0.023788123
0.10517206
It is slightly higher than the one without the integration, but still ok. Here is the time after I create Thread:
1.061517957
0.568310864
0.691701059
There is an enormous difference between this one and the previous one, it reaches even a whole second. It is noticeable even from the camera output, really slow.
My questions are, am I creating my Thread in the wrong way? There is a better and more efficient way to use rather the Threads? Or is there actually a way to optimise performance in this case without having to modify these functions findRedColours(img, board), t = Thread(target=playGame), gridExtractor.extractGrid(img)?
I am new using OpenCV and Python and still having troubles around. Hope someone can address me to the right way. Thanks.

Thanks to user 'deets' who help commenting above it has been possible to optimise performance.
In this case is enough to substitute Thread with Process from multiprocessing module in Python.
from multiprocessing import Process
#omitted code
while True:
# getting tick count
e1 = cv2.getTickCount()
# storing frame
_, img = cap.read()
# extract grid - first subsystem
gridExtractor.extractGrid(img)
# define red colours in the screen - second subsystem
findRedColours(img, board)
# getting tick count after the functions
e2 = cv2.getTickCount()
# calculating time
t = (e2 - e1) / cv2.getTickFrequency()
# print time
print(t)
# check if img is none
if img is not None:
# omitted code
k = cv2.waitKey(20) & 0xFF
# start the game, hide info
if (k == ord('s') or k == ord('S')) and start is False:
# create new thread to play game
p = Process(target=playGame)
p.start()
And the relative time needed is:
0.022570883
0.11354852
0.119643379
Compared to Thread the use of Process is way more efficient in terms of performance.

Related

Kinematic study application. Python and opencv

Im trying to develop a simple code using opencv and python. My idea is the following:
I have a video with a moving object(free falling or parabolic) and I've managed to separate the video in frames. What I need (and I'm a total newby in this aaaand have little time) is to extract coordinates of the object frame by frame. So the idea is to click on the moving object, and get the (x, y) coordinate and the number of frame and open the next frame so I can do the same. So basically something with clicks over the foto and extracting the data in a csv and showing the next frame. Thus I could study its movement through space with its velocity and acelerations and stuff.
Haven't written any code yet.
Thanks in advance.
Look at docs example with using mouse input in opencv w python:
mouse example - opencv
You can define callback reading the click coordinates:
def get_clic_point(event,x,y,flags,param):
if event == cv.EVENT_LBUTTONDBLCLK: # on left double click
print(x,y) # here goes Your sepcific code, using x, y as click coordinates
In main loop, You need to create window and supply it with callback method
cv.namedWindow('main_win')
cv.setMouseCallback('main_win',get_clic_point)
Then using window name (in this case 'main_win') as Your window handle, You can
show image, calling cv.imshow('main_win',img), where img is loaded image.
You can write simple loop like this:
cv.namedWindow('main_win')
cv.setMouseCallback('main_win',get_clic_point)
images = []
# read images to this list
# ...
i = 0
while(1):
cv.imshow('main_win',images[i])
k = cv.waitKey(1) & 0xFF
if k == ord('n'): # n for next
# change image
i = i + 1
i = i % len(images)
elif k == 27:
break
cv.destroyAllWindows()
then just substitiute call back with desired logic

stopping a python program at certain time and resuming temporarily with button

I have a python program running a clock and thermometer (raspberry pi + fourletterphat), see program below. I change between showing time or temperature by clicking a single button (gpio16).
What I need help with:
I want to pause the program during night, 21:00 - 06:00, and clear the display because the light is annoying. With the current code I get the display to clear at time for night to start but it does not start again.
If, during above period, the button is clicked I want the program to run for 10 seconds and then stop/clear display. I simply have no idea how to do this, Not even where start.
Is there an elegant way to do this, preferably by just adding something to the existing program. See below.
I have tried various ways of either clearing the display during night time and/or pausing the program until button is hit (but only during the night time, i want the program running during day to show temperature or time).
I have found many versions on finding it time.now is within my range for night but they seem not to be compatible with starting the program as described in point 2 above. (e.g. if time.now < night_end or time.now >= night_start:)
in code below function bright() sets the brightness AND turns off the display at night start 20:00.
Function night() is my feeble start on the restart display at night time but I have not gotten further.
#! /usr/bin/env python3
#Tempclock working with fourletterphat from pimoroni. One switch to change between temp and clock.
#sets brightness at startup but not continuously.
import glob
import time
import datetime
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
GPIO.setup(16, GPIO.IN, pull_up_down=GPIO.PUD_DOWN )
input_state = GPIO.input(16)
import fourletterphat as flp
# Find 1s temp sensor
base_dir = '/sys/bus/w1/devices/'
device_folder = glob.glob(base_dir + '28*')[0]
device_file = device_folder + '/w1_slave'
# times for setting brightness of display:
# d2 and d3 time dim and brighten display
d2 = 18
d3 = 7
# d4 and d5 time to turn off display for nigh
d4 = 20
d5 = 6
butt = 1
# Set brightness
def bright():
todays_date = datetime.datetime.now()
hn = (todays_date.hour)
if hn < d5 and hn > d4:
flp.clear()
flp.show()
elif hn < d2 and hn > d3:
flp.set_brightness(12)
else:
flp.set_brightness(0)
# Define nighttime display off
def night():
todays_date = datetime.datetime.now()
hn = (todays_date.hour)
if hn < d5 or hn => d4:
flp.set_brightness(5)
# define temp and time reading
def read_temp_raw():
f = open(device_file, 'r')
lines = f.readlines()
f.close()
return lines
def read_temp():
lines = read_temp_raw()
while lines[0].strip()[-3:] != 'YES':
time.sleep(0.2)
lines = read_temp_raw()
equals_pos = lines[1].find('t=')
if equals_pos != -1:
temp_string = lines[1][equals_pos+2:]
temp_c = float(temp_string) / 1000.0
temp_f = temp_c * 9.0 / 5.0 + 32.0
return temp_c, temp_f
def display_temp():
# temp = read_temp()[1] # F
temp = read_temp()[0] # C
flp.print_float(temp, decimal_digits=1, justify_right=True)
flp.show()
def display_time():
now = datetime.datetime.now()
str_time = time.strftime("%H%M")
flp.print_number_str(str_time)
flp.show()
# Display time or temp button on gpio pin 16 push button counter "butt" and set
# brightness bright() according to time of day. Function night() turns on display for 10sec if at night, then turns it off.
while True:
bright()
if GPIO.input(16) == 0:
butt = butt + 1
night()
if butt > 2:
butt = 1
if butt == 1:
display_time()
flp.show()
elif butt == 2:
display_temp()
flp.show()
time.sleep(0.2)
I'll start by assuming that your code to display time and read temperature is working correctly. Also, since I don't have a RaspberryPi with me, I'm focusing on how to organise your logic on the parts that I think you are having difficulty. My code to specifically turn the display on or off may be wrong, but I guess you figured that part out already. Since your question is about an elegant way to do this, the solution is obviously opinion-based. What I am proposing here is a simple but sound way to do it.
The first thing to notice is that you want two modes of operation. I'll call them day_mode and night_mode. The behaviour you expect is quite different for each mode:
day_mode: display is always on, when button 16 is pressed, the display should change from temperature to time. It should be active from 6:00 until 21:00
night_mode: display is normally off. When button 16 is pressed, the display should turn on and display something (you didn't specify, but I'll assume it is the temperature). It should be active when day_mode is not active.
When thinking about modes, it usually helps to separate three things:
What you need to do when you enter that mode of operation (for example, when you enter day_mode you should turn on the display and show the clock, when you enter night_mode you should turn off the display)
What is the behaviour you want during the mode. In your case, it is what we discussed above.
Any clean-up action needed when you exit the mode. Here none is needed, so we will skip this part.
So, let's begin! From your original code, I'll keep everything but the functions night, bright and the final loop. The first thing to do is create a function that tells in which mode we should be:
import datetime as dt
def which_mode():
current_hour = dt.datetime.now().time().hour
# In your code you seemed to need special behaviour between 6 and 7 and 18 and 21,
# but since it was not in your question, I'm not including it here
if 6 <= current_hour <= 21:
return 'day_mode'
else:
return 'night_mode'
To represent the modes, I'll use functions. If your problem were more complex, classes with specific enter, exit and during methods and a full blown state machine implementation would be needed. Since your problem is small, let's use a small solution. Each mode is a function, and the fact that you are entering the mode is informed by a parameter is_entering. The function will also receive a signal indicating that the button was pressed (a rising edge, but change according to your implementation). Before that, we will create a function that displays time or temperature based on a parameter:
def display_what(what):
if what == 0: # Time
display_time()
else: # Temperature
display_temp()
display_info = 0 # 0 means time, 1 means temperature
def day_mode(is_entering=False, button_pressed=False):
global display_info # Not very elegant, but gets the job done
if is_entering:
# Here the display is turned on. I got the values from your `bright` function
flp.set_brightness(12)
flp.clear()
# This is the `during` logic. Note that it will be run right after `entering` the first time
if button_pressed:
display_info = (display_info + 1) % 2 # Cycles between 0 and 1
display_what(display_info) # Update the display accordingly
For the night_mode function, we will need to keep a global variable to record when the button was last pressed. By convention, when it is smaller than 0, it means we are not counting the time (so we do not keep "clearing the display" all the time).
time_of_last_press = -1.0
def night_mode(is_entering=False, button_pressed=False):
if is_entering:
# Setup the dark display
flp.clear()
flp.set_brightness(0)
if button_pressed:
# Record when the button was pressed and display something
time_of_last_press = time.time()
flp.set_brightness(4) # I understood from your code that at night the display is less bright
display_what(1) # Temperature? You can use a similar logic from day_mode to toogle the displays
elif ((time.time() - time_of_last_press) > 10.0) and (time_of_last_press >= 0):
flp.clear()
flp.set_brightness(0)
time_of_last_press = -1.0
Finally, the main loop becomes relatively simple: check what should be the next mode. If it is different from the current one, set is_entering as True. Then call the function for the current mode.
# Setup the GPIO to detect rising edges.
GPIO.add_event_detect(16, GPIO.RISING)
current_mode = None # In the first run, this forces the `entering` code of the correct mode to run
while True:
button_pressed = GPIO.event_detected(16)
next_mode = which_mode()
changed = next_mode != current_mode
current_mode = next_mode
if current_mode == 'day_mode':
day_mode(changed, button_pressed)
else: # There are only two modes, but could be an elif if there were more modes
night_mode(changed, button_pressed)
time.sleep(0.2)
You may have noticed that there is some repetition: for example, in night_mode, the code to clear the display is the same code used when entering it, so you could actually call night_mode(True) to run that code again. Most importantly, there are two ifs that are similar: the one in which_mode and the one in the main loop. This can be fixed (and the code made more generic) if we change which_mode to return the function representing the mode, instead of a string. This may be a little more difficult to read if you are not used to functions returning functions, but it is a very flexible way of doing this sort of state machine:
# Must be defined after the functions `day_mode` and `night_mode` are defined
def which_mode():
current_hour = dt.datetime.now().time().hour
# In your code you seemed to need special behaviour between 6 and 7 and 18 and 21,
# but since it was not in your question, I'm not including it here
if 6 <= current_hour <= 21:
return day_mode # Note that this is not a string. It is a function!
else:
return night_mode
# Setup the GPIO to detect rising edges.
GPIO.add_event_detect(16, GPIO.RISING)
current_mode = None # In the first run, this forces the `entering` code of the correct mode to run
while True:
button_pressed = GPIO.event_detected(16)
next_mode = which_mode()
changed = next_mode != current_mode
current_mode = next_mode
current_mode(changed, button_pressed) # Now current_mode is a reference to one of the two mode functions, so we just need to call it.
time.sleep(0.2)

time.sleep for only a part of the code inside infinite while loop

I'm working with a code that analyzes frames from a live stream with OpenCV and if a condition is met saves the current frame to file.
The infinite loop to analyze the video frame by frame is something like this:
while True:
ret,frame = stream.read()
if conditionisMet :
pil_image = Image.fromarray(frame)
pil_image.save("/path/to/folder/image.jpg")
cv2.imshow("LiveStream", frame)
What I want to add is that if the condition is met again too soon (20-30 sec) the image does not have to be saved and the while loop has to grab another frame and continue its work. I've tried with time.sleep(30.0) inside the if statement but it blocks the while loop waiting for the 30 sec to pass. Is there a way to use time.sleep in this case, or another method suitable for my needs?
Thanks in advance
you could do something like this:
last_grab=time.time()-30 # this to get things started
while True:
if condition and time.time()-last_grab > 30:
last_grab=time.time()
# Do things here
else:
continue
Just add a variable to keep track of your last saving time:
last_save_time = time.time()
while True:
ret,frame = stream.read()
if conditionisMet and time.time() - last_save_time() > 20:
pil_image = Image.fromarray(frame)
pil_image.save("/path/to/folder/image.jpg")
# update last save time
last_save_time = time.time()
cv2.imshow("LiveStream", frame)
Why not just capture the amount of time running then save image if greater than the given amount of time....
a = dt.now()
b = dt.now()
c = b - a
if c < x:
do something

Inadvertently Stacking While Loops

I've got a program right now that is meant to display an image, and allow for various functions on keypress (e.g., load a new frame, crop a region of interest, etc). It's also set up so that if too much time passes with no action, it'll automatically attempt to load a new image.
def triggeredbytimer():
global f, x1i, y1i, x2i, y2i, sid, keepgoing
keepgoing = False
print "Inactivity has lead to an automatic refresh. Loading newest image."
f = 0; t = 0
incoming = imagerequest(sid, f, t, x1i, y1i, x2i, y2i)
imagecallback(incoming)
def imagecallback(data):
print "----------------------- imagecallback"
global sid, f, x1i, y1i, x2i, y2i, img, refPt, cropping, keepgoing
keepgoing = True
########### \\// change this to change wait for autorefresh
tmr = Timer(10.0, triggeredbytimer)
tmr.start()
##################################
b = data.b; t = data.tr; sid = data.sid; error = int(data.error)
t = float(t); sid = int(sid)
expandedt = time.strftime("%m-%d-%Y %H:%M:%S", time.localtime(t))
print str(t) + " ----> " + expandedt
print "----------------------- image incoming"
timestr = time.strftime("%H:%M:%S", time.gmtime(t))
print "Loaded image from %s."%timestr
imagedata = bridge.imgmsg_to_cv2(b, "bgr8")
npimg = cv2.imencode('.jpg', imagedata)
cv2.imwrite('temp_image_np.jpg', imagedata)
img = cv2.imread('temp_image_np.jpg')
clone = img.copy()
cv2.namedWindow('image', cv2.WINDOW_NORMAL)
cv2.setMouseCallback("image", mouseclickcb)
cv2.imshow("image", img)
while keepgoing == True:
key = cv2.waitKey(0) & 0xFF
## (here there are a lot of keypress functions, for brevity I included only one)
elif key == ord("t"):
print "Looking for newest frame..."
tmr.cancel()
t = 0; f = 0
incoming = imagerequest(sid, f, t, x1i, y1i, x2i, y2i)
imagecallback(incoming)
(NOTE: The images are fetched from a ROS program via services, and that is where the "imagerequest" function comes from. All of that works fine: the problem is specifically as prescribed.)
To summarize here- since there's a bit of irrelevant clippings in this snippet - an image is loaded by imagecallback. When that function is called, a timer starts (time limit is arbitrary, set low here for testing). When the timer expires, triggeredbytime() requests a new image. It acquires the image via a ROS service, then takes the return from that and uses it as an input to imagecallback(data), thus loading the image and allowing all the keypress functions to still be valid.
So the problem is that whenever the program refreshes with a new frame, the while keepgoing == True loop does not stop, leading to stacked loops that go on and get worse with every refresh. this leads to a single keypress executing multiple times, which quickly overwhelms the program it's pulling images from. (Originally, this was just a while True: loop, but I added the "keepgoing" variable to try to remedy the situation - unsuccessfully, as it were.) I'm not sure how to resolve this issue, and attempting to turn the loop "off" long enough to kill the first loop but allowing the next one to execute has not worked out so far. Is there any way to exit this while keepgoing == True loop cleanly to prevent things from stacking as such, or is there a better way in general to address the problem of loading a new image after a certain amount of time?
Thanks!
EDIT: to be clear, the problem is with ending the 'while' loop when the timer hits 0. setting keepgoing = False doesn't seem to work (probably because I can't 'return' it), so I need something similar to that if it's possible...

python, opencv: duration of time an object is detected on webcam

i am able to track an object in each frame returned by my webcam. i want to note when the object was detected for the first time and the duration till it was continuously detected thereafter. the webcam is on for an indefinite period i.e. till it is closed by user input.
since the set of code for detecting the object is within a while loop that is needed to read the next frame from cv2.VideoCapture() i am not able to come up with an efficient, pythonic way for doing what i want.
right now i am appending a list with the tuple (timestamp,flag) for each frame. timestamp is the value from python's time.time() and flag is a boolean to indicate if the object is detected. i then sum up all values of timestamp where flag is 'Yes'. but this doesn't quite give me what i want. can you suggest a more appropriate way?
*i wish there was a generic function in opencv like cv2.detectionDuration() :P
--EDIT--
Here is a code for tracking frontal face:
import cv2
import time
faceCascade = cv2.CascadeClassifier('haarcascade_frontalface_alt.xml')
capture = cv2.VideoCapture(0)
keyPressed = -1
faceFound = []
print 'press esc to quit'
while(keyPressed != 27):
ret, camImage = capture.read()
cv2.imshow('camImage', camImage)
try:
faceRegion = faceCascade.detectMultiScale(camImage)
timestamp = time.time()
flag = 1
faceFound.append((timestamp,flag))
except TypeError:
timestamp = time.time()
flag = 0
faceFound.append((timestamp,flag))
print 'check if front face is visible to camera'
pass
keyPressed = cv2.waitKey(1)
cv2.destroyAllWindows()
timeDelta = 0
for tup in faceFound:
if tup[1] == 1:
timeDelta += tup[0]
print timeDelta
also, could you help me obtain a better format for timeDelta so that it can be displayed as day:hour:min:sec:microsec. is there a better alternative to time.time() for my current requirements?
[time_first, time_last, got_first]
if got_first is false and face is detected, you assign time_first to now() and got_first to true.
when the face is not detected, you assign time_last to now() and got_first to false.
import cv2
import time
faceCascade = cv2.CascadeClassifier('haarcascade_frontalface_alt.xml')
capture = cv2.VideoCapture(0)
keyPressed = -1
faceFound = []
ts = [0,0,False]
print 'press esc to quit'
while(keyPressed != 27):
ret, camImage = capture.read()
cv2.imshow('camImage', camImage)
try:
faceRegion = faceCascade.detectMultiScale(camImage)
if ts[2] == False:
ts[0] = time.time()
ts[2] = True
except TypeError:
if ts[2] == True:
ts[1] = time.time()
ts[2] = False
faceFound.append([ts[0], ts[1]])
print 'check if front face is visible to camera'
pass
keyPressed = cv2.waitKey(1)
cv2.destroyAllWindows()
for list in faceFound:
print list[1] - list[0]
although I think there's a problem with your code, cause no face gets detected.
you can print faceRegion and see that, it's an empty tuple.
What you are trying to do is known as dwell time in which we calculate the time of the detected object inside the frame. In order to achieve this you need have some kind of tracking inside you inferencing. Basically the tracker will assign an object id to your detected object and then that object id remains same till the object is detected. Based on that object id you can start a timer and keep on counting it until the object is detected. In opencv you can go with CentroidTracking algorithm
Have a look at this video, this might give you basic idea: https://www.youtube.com/watch?v=JbNeFMKXybw&list=PLWw98q-Xe7iH8UHARl8RGk8MRj1raY4Eh&index=7

Categories