so I have a Python program on my Raspberry Pi running on an infinite while loop that takes an image from the camera every second.
With every iteration, the program creates a thread which uses the image to process. This process consists of: a script extracts a phone screen out of the image using OpenCV and another script to extract a QR code out of that screen.
I used the program to use pre-taken images to process them without problem but it only chokes if I put the program through a continuous loop.
After a few iterations of the loop and trying to process the images, the program unexpectedly breaks and my Raspberry Pi abruptly shuts off. Does anyone know why this is happening?
I have been looking around for an answer but my suspicions are with the threads. Either, I'm overloading the CPU or RAM, there are memory leaks, the Raspberry Pi is using too much power
EDIT:
From the comments below it seems that the power supply to the Pi seems to be the problem. I'm currently running on a phone charger (5.0v, 1.0A) which is (very) below the 5.0v, 2.5A recommended power supply, darn me. I will update this post when I get a new power supply and test the code out.
Also, running the program on my Windows laptop poses no problems at all.
This is my main script:
import picamera
import threading
import time
from processing.qr import get_qr_code
from processing.scan import scan_image
# Thread method
def extract_code(file):
print 'Thread for: ' + file
# Scans image to extract phone screen from image and then gets QR code from it
scan_image(file)
get_qr_code(file)
return
# End methods
camera = picamera.PiCamera()
while True:
time.sleep(1)
# Make epoch time as file name
time_epoch = str(int(time.time()))
image_path = "images/" + time_epoch + ".jpg"
print "Taking photo: " + str(image_path)
camera.capture(image_path)
# Create thread to start processing image
t = threading.Thread(target=extract_code, args=[time_epoch])
t.start()
Below is the script to scan image (scan.py)
In a nutshell, it takes an image, blurs it, finds edges and draws contours, checks to see if there's a rectangle (e.g a phone screen) and transforms and warps it into a new image with only the phone screen.
from transform import four_point_transform
import imutils
import numpy as np
import argparse
import cv2
import os
def scan_image(file):
images_dir = "images/"
scans_dir = "scans/"
input_file = images_dir + file + ".jpg"
print "Scanning image: " + input_file
# load the image and compute the ratio of the old height
# to the new height, clone it, and resize it
image = cv2.imread(input_file)
ratio = image.shape[0] / 500.0
orig = image.copy()
image = imutils.resize(image, height = 500)
# convert the image to grayscale, blur it, and find edges
# in the image
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray, (5, 5), 0)
edged = cv2.Canny(gray, 75, 200)
# show the original image and the edge detected image
print "STEP 1: Edge Detection"
# find the contours in the edged image, keeping only the
# largest ones, and initialize the screen contour
_, cnts, _ = cv2.findContours(edged.copy(), cv2.RETR_LIST, cv2.CHAIN_APPROX_SIMPLE)
cnts = sorted(cnts, key = cv2.contourArea, reverse = True)[:5]
screenCnt = 0
# loop over the contours
for c in cnts:
# approximate the contour
peri = cv2.arcLength(c, True)
approx = cv2.approxPolyDP(c, 0.02 * peri, True)
# if our approximated contour has four points, then we
# can assume that we have found our screen
if len(approx) == 4:
screenCnt = approx
break
# show the contour (outline) of the piece of paper
print "STEP 2: Find contours of paper"
# if screenCnt > 0 :
# cv2.drawContours(image, [screenCnt], -1, (0, 255, 0), 2)
# apply the four point transform to obtain a top-down
# view of the original image
if screenCnt > 0:
warped = four_point_transform(orig, screenCnt.reshape(4, 2) * ratio)
print "STEP 3: Apply perspective transform"
output = scans_dir + file + "-result.png"
if not os.path.exists(scans_dir):
os.makedirs(scans_dir)
cv2.imwrite(output, imutils.resize(warped, height = 650))
else:
print "No screen detected"
This is the code to scan a QR code out of the image:
import os
from time import sleep
def get_qr_code(image_name):
scans_dir = "scans/"
codes_dir = "codes/"
input_scan_path = scans_dir + image_name + "-result.png"
output_qr_path = codes_dir + image_name + "-result.txt"
if not os.path.exists(codes_dir):
os.makedirs(codes_dir)
if os.path.exists(input_scan_path):
os.system("zbarimg -q " + input_scan_path + " > " + output_qr_path)
if os.path.exists(output_qr_path):
strqrcode = open(output_qr_path, 'r').read()
# print strqrcode
print "Results for " + image_name + ": " + strqrcode
else:
print "File does not exist"
After a few iterations of the loop and trying to process the images, the program unexpectedly breaks and my Raspberry Pi abruptly shuts off. Does anyone know why this is happening?
I had a similar experience. My program (C++) realtime processing camera frames unexpectedly crashed. Sometimes with segmentation error inside the OS code, sometimes just shutting off the whole board. Sometimes the program executable file just become zero bytes.
I taken long time looking for memory leaks or CPU peaks or other kind of programming error. Suddenly I realised that my power supply was not so strong. I changed to a new one more robust and I got no more problems.
See in this link the 2.5 A recommendation and check if your UPS actually meets this requirement.
Maybe not the answer you are looking for, but using Python with OpenCV on an embedded system is not really a good idea. I have experienced a lot of problems when using python on embedded systems (even with the powerful hardware that the rasp provides you).
All the performance problems I experienced vanished when I used C++ instead of Python. OpenCV on C++ is pretty solid and I highly recommend you to give it a try.
Related
I'm doing a project in which we need to do segmentation of the window cars from inside the car.
I'm working with OpenCV but this is not mandatory. (Also python or C++ are OK)
Until now, I have some (not so good) results. I have followed this sequence:
1) Apply cv2.grabCut() in ROIs where windows might be.
import cv2
import numpy as np
from matplotlib import pyplot as plt
#Read Image
img = cv2.imread("test1.png",-1)
#Grab Cut iterations
itera = 30
p1 = True
if p1: # This takes quite long so this helps when debugging
gb_mask_w1 = np.zeros(img.shape[:2],np.uint8) #output mask
bgdModel = np.zeros((1,65),np.float64)
fgdModel = np.zeros((1,65),np.float64)
rect = (1,12,390,845) #ROI of window 1
cv2.grabCut(img,gb_mask_w1,rect,bgdModel,fgdModel,itera,cv2.GC_INIT_WITH_RECT)
# 0-pixels and 2-pixels are put to 0 (ie background) and all 1-pixels and
# 3-pixels are put to 1(ie foreground pixels).
mask_w1 = np.where((gb_mask_w1==3),0,1).astype('uint8') #
gb_mask_w2 = np.zeros(img.shape[:2],np.uint8)#output mask
rect_1 = (1500,12,323,820) #ROI of window 2
cv2.grabCut(img,gb_mask_w2,rect_1,bgdModel,fgdModel,itera,cv2.GC_INIT_WITH_RECT)
# 0-pixels and 2-pixels are put to 0 (ie background) and all 1-pixels and
# 3-pixels are put to 1(ie foreground pixels).
mask_w2 = np.where((gb_mask_w2==3),0,1).astype('uint8')
2) Erode on the foreground pixels to have clean(er) edges
# Morphological Operations
kernel = np.ones((10,10),np.uint8)
eroded_w1 = cv2.erode(mask_w1,kernel,iterations = 1)
eroded_w2 = cv2.erode(mask_w2,kernel,iterations = 1)
3) Aplpy cv2.fastNlMeansDenoising() to avoid artifacts
# DeNoise artifacts
mask_w1_porc = cv2.fastNlMeansDenoising(eroded_w1)
mask_w2_porc = cv2.fastNlMeansDenoising(eroded_w2)
4) Just apply resulting mask & plot
img_treated = img.copy()
# Green background
bskg = np.zeros(img.shape[:3],np.uint8)
bskg[:] = (0, 177, 64)
#Apply mask
img_treated[mask_w1_porc==0] = bskg[mask_w1_porc==0]
img_treated[mask_w2_porc==0] = bskg[mask_w2_porc==0]
#Plot
f,ax=plt.subplots(2,2)
ax[0,0].imshow(cv2.cvtColor(img,cv2.COLOR_BGR2RGB ))
ax[0,1].imshow(mask_w1_porc)
ax[1,0].imshow(mask_w2_porc)
ax[1,1].imshow(cv2.cvtColor(img_treated,cv2.COLOR_BGR2RGB ))
Problems:
It happens that only a segment of the window is actually detected.
It may happen that parts of the car are detected as window
Very slow (~20 seconds per window)
Is something working?
At least an important section of the windows is always detected
It's robust to occlusion
What do I want?
I'm wondering what can I add to this pipeline to have a better window detection => a filter to make cv2.grabCut()'s work easier?
Maybe there is there a faster way?
I'm avoiding machine learning or AI aproaches because we work with a not so powerful computer. But I'm open to those ideas
I'm adding the plot so that you can see the results (Original image, both masks and ouput):
[Edit]:
I'm also posting an image with occlusion(which doesn't seem so fail but is the reason to detect windows instead of having fixed masks)
I wrote this script in python to send data over serial communication to my Arduino UNO. It uses the OpenCV library to process a binary image and takes decisions based on the position of the detected object.
Here's the script:-
import numpy as np
import cv2
import serial
ser = serial.Serial('COM3',9600,writeTimeout = 0)
def f(x): return
cv2.namedWindow('Thresholding Control')
# create trackbars for color change
cv2.createTrackbar('High H','Thresholding Control',179,179, f)
cv2.createTrackbar('Low H','Thresholding Control',0,179, f)
cv2.createTrackbar('High S','Thresholding Control',255,255, f)
cv2.createTrackbar('Low S','Thresholding Control',0,255, f)
cv2.createTrackbar('High V','Thresholding Control',255,255, f)
cv2.createTrackbar('Low V','Thresholding Control',0,255, f)
cv2.createTrackbar('Guassian Blur','Thresholding Control',0,99, f)
cap = cv2.VideoCapture(0)
while(True):
ret, image = cap.read()
HSV = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
# Getting trackbar values
highH = cv2.getTrackbarPos('High H','Thresholding Control')
lowH = cv2.getTrackbarPos('Low H','Thresholding Control')
highS = cv2.getTrackbarPos('High S','Thresholding Control')
lowS = cv2.getTrackbarPos('Low S','Thresholding Control')
highV = cv2.getTrackbarPos('High V','Thresholding Control')
lowV = cv2.getTrackbarPos('Low V','Thresholding Control')
# Thresholding the image.
thresh = cv2.inRange( HSV, (lowH, lowS, lowV), (highH, highS, highV))
blurVal = cv2.getTrackbarPos('Guassian Blur','Thresholding Control')
if(blurVal%2==0):
blurVal=blurVal+1
thresh_smooth = cv2.GaussianBlur(thresh, (blurVal, blurVal), 0)
#Defining the kernel to be used for Morphological ops.
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(5,5))
# Applying Opening and Closing.
thresh_smooth = cv2.morphologyEx(thresh_smooth,cv2.MORPH_OPEN,kernel)
thresh_smooth = cv2.morphologyEx(thresh_smooth, cv2.MORPH_CLOSE, kernel)
eleR = np.count_nonzero(thresh_smooth[0:480, 320:550])
eleL = np.count_nonzero(thresh_smooth[0:480, 0:320])
eleO = np.count_nonzero(thresh_smooth[0:480, 550:640])
if (eleL>eleR and eleL>eleO and eleL!= (eleR+eleO)):
cv2.putText(image,"Left Turn", (320,240), cv2.FONT_HERSHEY_SIMPLEX, 2, 255)
ser.write('L')
if (eleO>eleR and eleO>eleL):
cv2.putText(image,"Right Turn", (240,320), cv2.FONT_HERSHEY_SIMPLEX, 2, 255)
ser.write('R')
else:
cv2.putText(image,"Straight", (240,320), cv2.FONT_HERSHEY_SIMPLEX, 2, 255)
ser.write('S')
cv2.imshow("BGR", image)
cv2.imshow("Thresholded", thresh_smooth)
print ser.readline();
if cv2.waitKey(1) & 0xFF == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
The problem here are the if statements. Out of the three conditions, no matter which condition is true, only the value 'S' (in the else statement) is sent over serial, as confirmed by the print ser.readline() command. Then all of a sudden the value of any of the if blocks might be sent and then the same value just goes on, no matter what position in the image the object is. The cv2.putText commands work as expected, it's only the ser.write that's causing the problem.
I tried putting elif in place of the second if, also tried putting the continue statement in each of the conditional blocks, which for some reason crashes the script during runtime. None of them helped. I have no idea how to fix this bug.
Thanks.
UPDATE
Tried increasing the baud rate to 115200, didn't work. Put a capacitor between RESET and GND pins, didn't work.
Okay I figured it out. Putting what I did here, for people who are curious and for those who need help.
It was probably the Arduino, autoresetoverserialcomm, for some reason the Arduino auto-resets whenever you send a value over the serial comm to the Arduino from a computer, not from the Serial port (obviously). I knew of this earlier, and another one of my script which also sends data to the Arduino was working fine with this setup.
What I did? I just put a 10ยต capacitor between the RESET and the GND pins. That didn't work initially, but I tried putting the +ve of the capacitor in the GND pin and -ve in the RESET pin and voila! The script started working just fine.
I guess this wasn't a programming issue at all, but an issue of the Arduino and the script not being able to contact each other.
I still am not sure as to what was causing the problem or whether what I have done is a permanent fix, or even a fix at all, but my problem seems to have been solved. I'll keep updating the post.
Suggestions and discussions are always welcome!
I am currently on a robotics team for my high school and my objective is to use our kinect to detect the balls in this year's game by using simplecv and openkinect. I am currently stuck on an issue that my mentors and I cannot seem to resolve. What the code does is it takes a live-depth map feed from the kinect and binarizes it, once it has done that it looks for blobs and tries to find the circular blobs. Finally it draws a circle over the circular blob. The issue is the code is able to find blobs no problem, but it's not able to find the circular ones. I am at a dead end and am hoping someone in the community will be able to please help me. Thank you, here's the code and an image of what is seen
from SimpleCV import *
cam = Kinect()
display = SimpleCV.Display()
while display.isNotDone():
depth = cam.getDepth().flipHorizontal()
img2 = cam.getImage().flipHorizontal()
filtered = depth.stretch(200,255)
segmented = filtered.binarize(200)
blobs = segmented.findBlobs()
if blobs:
circles = blobs.filter([b.isCircle() for b in blobs])
if circles:
img2.drawCircle((circles[-1].x, circles[-1].y), circles[-1].radius(),SimpleCV.Color.RED,3)
img2.drawText("FOUND A BALL", 50,50,color=Color.RED,fontsize=48)
img2.sideBySide(segmented).show()
I have solved the problem, here is the new version of the code that works. Sorry, no screen cap this time.
import SimpleCV
from SimpleCV import *
display = SimpleCV.Display()
cam = Kinect()
normaldisplay = True
while display.isNotDone():
if display.mouseRight:
normaldisplay = not(normaldisplay)
print "Display Mode:", "Normal" if normaldisplay else "Segmented"
img = cam.getDepth().flipHorizontal()
img2 = cam.getImage().flipHorizontal()
dist = img.colorDistance(SimpleCV.Color.BLACK).dilate(2)
segmented = dist.stretch(200,255)
binar = segmented.binarize(200)
erode = binar.erode(2)
blobs = erode.findBlobs()
if blobs:
circles = blobs.filter([b.isCircle(0.2) for b in blobs])
if circles:
img.drawCircle((circles[-1].x, circles[-1].y), circles[-1].radius(),SimpleCV.Color.BLUE,3)
if normaldisplay:
img.show()
else:
segmented.show()
I am attempting to calibrate my single webcam using opencv cv2 in python. I am using the cv2.findChessboardCorners and cv2.calibrateCamera functions. However, my root mean square returned from the calibrateCamera function seems very high. It is always around 50 no matter how many frames with found boards are used. I read that good values range from 0-1. I am using a 5x8 black and white checker pattern on a piece of paper taped to a wooden board. Can anyone help me out with getting this lower? The weird thing is I used images I rendered from Blender, a 3d modeling software, in which there is no lens distortion and the board's coordinates are known for sure and I was able to get a RMS of 0.22 which is good. Using similar code though I cannot replicate those results with my webcam. Perhaps I am missing something. Thanks so much to everyone who takes a look at this. Here is the complete code:
import sys
import os
import numpy as np
import cv2
import time
'''
This module finds the intrinsic parameters of the camera. These parameters include
the focal length, pixel aspect ratio, image center, and lens distortion (see wiki
entry for "camera resectioning" for more detail). It is important to note that the
parameters found by this class are independent of location and rotation of the camera.
Thus, it only needs to be calculated once assuming the lens and focus of the camera is
unaltered. The location and rotation matrix are defined by the extrinsic parameters.
'''
class Find_Intrinsics:
'''Finds the intrinsic parameters of the camera.'''
def __init__(self):
#Import user input from Blender in the form of argv's
self.rows = int(sys.argv[1])
self.cols = int(sys.argv[2])
self.board_width_pxls = float(sys.argv[3])
self.pxls_per_sq_unit = float(sys.argv[4])
self.printer_scale = float(sys.argv[5])
def find_calib_grid_points(self,cols,rows,board_width_pxls,pxls_per_sq_unit,printer_scale):
'''Defines the distance of the board squares from each other and scale them.
The scaling is to correct for the scaling of the printer. Most printers
cannot print all the way to the end of the page and thus scale images to
fit the entire image. If the user does not desire to maintain real world
scaling, then an arbitrary distance is set. The 3rd value appended to
calib_points signifies the depth of the points and is always zero because
they are planar.
'''
#should be dist for each square
point_dist = (((board_width_pxls)/(pxls_per_sq_unit))*printer_scale)/(cols+2)
calib_points = []
for i in range(0,cols):
for j in range(0,rows):
pointX = 0 + (point_dist*j)
pointY = 0 - (point_dist*i)
calib_points.append((pointX,pointY,0))
np_calib_points = np.array(calib_points,np.float32)
return np_calib_points
def main(self):
print '---------------------------Finding Intrinsics----------------------------------'
np_calib_points = self.find_calib_grid_points(self.cols,self.rows,self.board_width_pxls,
self.pxls_per_sq_unit,self.printer_scale)
pattern_size = (self.cols,self.rows)
obj_points = []
img_points = []
camera = cv2.VideoCapture(0)
found_count = 0
while True:
found_cam,img = camera.read()
h, w = img.shape[:2]
print h,w
gray_img = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
found, corners = cv2.findChessboardCorners(img, pattern_size)
if found:
term = ( cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_COUNT, 30, 0.1 )
cv2.cornerSubPix(gray_img, corners, (5, 5), (-1, -1), term)
cv2.drawChessboardCorners(img, pattern_size, corners, found)
found_count+=1
img_points.append(corners.reshape(-1, 2))
obj_points.append(np_calib_points)
cv2.putText(img,'Boards found: '+str(found_count),(30,30),
cv2.FONT_HERSHEY_DUPLEX, 0.8,(0,0,255,1))
cv2.putText(img,'Press any key when finished',(30,h-30),
cv2.FONT_HERSHEY_DUPLEX, 0.8,(0,0,255,1))
cv2.imshow("webcam",img)
if (cv2.waitKey (1000) != -1):
cv2.destroyAllWindows()
del(camera)
np_obj_points = np.array(obj_points)
print "Calibrating.Please be patient"
start = time.clock()
#OpenCV function to solve for camera matrix
try:
print obj_points[0:10]
rms, camera_matrix, dist_coefs, rvecs, tvecs = cv2.calibrateCamera(obj_points, img_points, (w, h))
print "RMS:", rms
print "camera matrix:\n", camera_matrix
print "distortion coefficients: ", dist_coefs
#Save the camera matrix and the distortion coefficients to the hard drive to use
#to find the extrinsics
#want to use same file directory as this file
#directory = os.path.dirname(os.path.realpath('Find_Intrinsics.py'))
np.save('C:\\Users\\Owner\\Desktop\\3D_Scanning\\Blender_3d_Scanning\\camera_matrix.npy',camera_matrix)
np.save('C:\\Users\\Owner\\Desktop\\3D_Scanning\\Blender_3d_Scanning\\dist_coefs.npy',dist_coefs)
elapsed = (time.clock() - start)
print("Elapsed time: ", elapsed, " seconds")
img_undistort = cv2.undistort(img,camera_matrix,dist_coefs)
cv2.namedWindow('Undistorted Image',cv2.WINDOW_NORMAL)#WINDOW_NORMAL used bc WINDOW_AUTOSIZE does not let you resize
cv2.resizeWindow('Undistorted Image',w,h)
cv2.imshow('Undistorted Image',img_undistort)
cv2.waitKey(0)
cv2.destroyAllWindows()
break
except:
print "\nSorry, an error has occurred. Make sure more than zero boards are found."
break
if __name__ == '__main__' and len(sys.argv)== 6:
Intrinsics = Find_Intrinsics()
Intrinsics_main = Intrinsics.main()
else:
print "Incorrect number of args found. Make sure that the python27 filepath is entered correctly."
print '--------------------------------------------------------------------------------'
Wow, I do not know what it is about having aha moments (or doh! moments if you watch The Simpsons) right after posting on stackoverflow haha. This one was a bone head mistake. I managed to mess up the construction of the obj_points parameter for the calibrateCamera function. OpenCV takes the first point to be the top left and goes through each row, left to right, until it comes to the last point (bottom right). As such, my find_calib_grid_points function was wrong. Here is the correct code just in case someone else gets tripped up on that, though that is probably unlikely:
for j in range(0,rows):
for i in range(0,cols):
pointX = 0 + (point_dist*i)
pointY = 0 - (point_dist*j)
calib_points.append((pointX,pointY,0))
np_calib_points = np.array(calib_points,np.float32)
Thanks to everyone who looked at this and sent me psychic messages so I could figure this one out right haha. They worked! My RMS was 0.33!
I am trying to make a motion detect program using my webcam, but I'm getting this weird results when thresholding the difference of frames:
When Im moving: (seems okay I guess)
![enter image description here][1]
When Im not moving:
![enter image description here][2]
What can this be? I already ran a couple of programs that got exactly the same algorithm and the thresholding is doing fine..
Heres my code:
import cv2
import random
import numpy as np
# Create windows to show the captured images
cv2.namedWindow("window_a", cv2.CV_WINDOW_AUTOSIZE)
cv2.namedWindow("window_b", cv2.CV_WINDOW_AUTOSIZE)
# Structuring element
es = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (9,4))
## Webcam Settings
capture = cv2.VideoCapture(0)
#dimensions
frameWidth = capture.get(cv2.cv.CV_CAP_PROP_FRAME_WIDTH)
frameHeight = capture.get(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT)
while True:
# Capture a frame
flag,frame = capture.read()
current = cv2.blur(frame, (5,5))
difference = cv2.absdiff(current, previous) #difference is taken of the current frame and the previous frame
frame2 = cv2.cvtColor(difference, cv2.cv.CV_RGB2GRAY)
retval,thresh = cv2.threshold(frame2, 10, 0xff, cv2.THRESH_OTSU)
dilated1 = cv2.dilate(thresh, es)
dilated2 = cv2.dilate(dilated1, es)
dilated3 = cv2.dilate(dilated2, es)
dilated4 = cv2.dilate(dilated3, es)
cv2.imshow('window_a', dilated4)
cv2.imshow('window_b', frame)
previous = current
key = cv2.waitKey(10) #20
if key == 27: #exit on ESC
cv2.destroyAllWindows()
break
Thanks in advance!
[1]: http://i.stack.imgur.com/hslOs.png
[2]: http://i.stack.imgur.com/7fB95.png
The first thing that you need is a previous = cv2.blur(frame, (5,5)) to prime your previous sample after a frame grab before your while loop.
This will make the code you posted work, but will not solve your problem.
I think the issue that you are having is due to the type of thresholding algorithm that you are using. Try a binary, cv2.THRESH_BINARY, instead of Otsu's. It seemed to solve the problem for me.