We Keep Coding

Facing problem in opencv background noise subtraction and movement detection

I'm working on a project which requires detection of people and due to the complexity of the system, I decided to use movement detection.
I faced some problems and upon asking on stack overflow, this answer seemed the best.
So I implemented the algorithm in the following steps:
Implement saliency on the input video
Applied K-means clustering
Background Subtraction
Morphological Transformation
Here is the code
import cv2
import time
import numpy as np
cap=cv2.VideoCapture(0)
#i wanted to try different background subtractors to get the best result.
fgbg=cv2.createBackgroundSubtractorMOG2()
fgbg1 = cv2.bgsegm.createBackgroundSubtractorMOG()
h = cap.get(4)
w = cap.get(3)
frameArea = h*w
areaTH = frameArea/150
while(cap.isOpened()):
#time.sleep(0.05)
_,frame=cap.read()
cv2.imshow("frame",frame)
image=frame
################Implementing Saliency########################
saliency = cv2.saliency.StaticSaliencySpectralResidual_create()
(success, saliencyMap) = saliency.computeSaliency(image)
saliencyMap = (saliencyMap * 255).astype("uint8")
#cv2.imshow("Image", image)
#cv2.imshow("Output", saliencyMap)
saliency = cv2.saliency.StaticSaliencyFineGrained_create()
(success, saliencyMap) = saliency.computeSaliency(image)
saliencyMap = (saliencyMap * 255).astype("uint8")
threshMap = cv2.threshold(saliencyMap.astype("uint8"), 0, 255,cv2.THRESH_BINARY | cv2.THRESH_OTSU)[1]
# show the images
#cv2.imshow("Image", image)
#cv2.imshow("saliency", saliencyMap)
#cv2.imshow("Thresh", threshMap)
kouts=saliencyMap
#cv2.imshow("kouts", kouts)
##############implementing k-means clustering#######################
clusters=12
z=kouts.reshape((-1,3))
#covert to np.float32
z=np.float32(z)
#define criteria and accuracy
criteria= (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER,5,1.0)
#apply k-means
ret,label,center=cv2.kmeans(z,clusters,None,criteria,10,cv2.KMEANS_RANDOM_CENTERS)
#converting back the float 32 data to unit 8 and making the image
center=np.uint8(center)
res=center[label.flatten()]
kouts=res.reshape((kouts.shape))
cv2.imshow('clustered image',kouts)
############applying background subtraction#######################
fgmask=fgbg.apply(kouts)
fgmask1=fgbg1.apply(kouts)
cv2.imshow('fg',fgmask)
cv2.imshow('fgmask1',fgmask1)
#as i said earlier, i wanted to get the best background subtractor
#########################morphological transformation#####################
#Below i tried various techniques to get the best possible result
kernel=np.ones((5,5),np.uint8)
erosion=cv2.erode(fgmask1,kernel,iterations=1)
cv2.imshow('erosion',erosion)
dilation=cv2.dilate(fgmask1,kernel,iterations=1)
cv2.imshow('dilation',dilation)
gradient = cv2.morphologyEx(fgmask1, cv2.MORPH_GRADIENT, kernel)
cv2.imshow("gradient",gradient)
opening=cv2.morphologyEx(fgmask1,cv2.MORPH_OPEN,kernel)
closing=cv2.morphologyEx(fgmask1,cv2.MORPH_CLOSE,kernel)
cv2.imshow('opening',opening)
cv2.imshow('closing',closing)
#########for detection of contours##################
contours0, hierarchy = cv2.findContours(erosion,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
for cnt in contours0:
area = cv2.contourArea(cnt)
if area > areaTH and area<frameArea*0.50:
M = cv2.moments(cnt)
x,y,f,g = cv2.boundingRect(cnt)
img = cv2.rectangle(frame,(x,y),(x+f,y+g),(0,255,0),2)
cv2.imshow('Original',frame)
k = cv2.waitKey(1) & 0xff
if k == ord('q'):
break
cap.release()
cv2.destroyAllWindows()
I tried this algorithm on this video but still there was a lot of noise in the output. I previously thought that the problem might be in the quality of the video but when I did cv2.VideoCapture(0), the problem still persist and the code doesn't seem to remove the noise and the situation I'm working in, has sometimes high noise.
Tell me any suggestions or where did I go wrong or a different approach to the problem.
Thanks in advance.

I spent sometime trying to see if something can be done with noise reduction, but I believe you already tried many of the known techniques in OpenCV. My opinion is to approach your problem using neural networks as they will be more accurate detecting the objects.
I created a Colab notebook, to illustrate this:
https://colab.research.google.com/drive/1rBrcu46sfo0F7fsQf4BC9hKoXTk_wNBk?usp=sharing
Even with this simple approach, it's possible to detect objects: persons and clothing. You can set a criteria that can just consider the top 10 items. As a bus entrance has a limit of people that can enter at the same time.
This is not a final solution because I am using a general purpose detector. This can be improved in your application by training the network with your video inputs. Labeling will be required but I believe this will give you the most accurate results.
I also think for the challenge to track the people that are inside the bus and the ones entering. For that you can take track the rectangles. There is an excellent example using dlib: https://www.pyimagesearch.com/2018/10/22/object-tracking-with-dlib/

Difficulty in detected ellipses in image

I am trying to detect ellipses in some images.
After some functions I got this edges map:
I tried using Hough transform to detect ellipses, but this transform has very high complexity, so my computer didn't finish running the transform command even after 5 hours(!).
I also tried doing connected components and got this:
In last case I also tried continue and binarized the image.
In all cases I am stuck in these steps, and have no idea how continue from here.
My mission is detect tomatoes in the image. I am approaching this by trying to detect circles and ellipses and find the radius (or average radius in ellipses case) for each one.
edited:
I add my code for the first method (the result is edge map from above):
img = cv2.imread(r'../images/assorted_tomatoes.jpg')
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
imgAfterLight=lightreduce(img)
imgAfterGamma=gamma_correctiom(imgAfterLight,0.8)
th2 = 255 - cv2.adaptiveThreshold(imgAfterGamma,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C,cv2.THRESH_BINARY,5,3)
median2 = cv2.medianBlur(th2,3)
where median2 is the result of shown above in edge map
and the code for connected components:
import scipy
from scipy import ndimage
import matplotlib.pyplot as plt
import cv2
import numpy as np
fname=r'../images/assorted_tomatoes.jpg'
blur_radius = 1.0
threshold = 50
img = scipy.misc.imread(fname) # gray-scale image
gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
print(img.shape)
# smooth the image (to remove small objects)
imgf = ndimage.gaussian_filter(gray_img, blur_radius)
threshold = 80
# find connected components
labeled, nr_objects = ndimage.label(imgf > threshold)
where labeled is the result above
another edit:
this is the input image:
Input
The problem is that after edge detection, there are a lot of unnecessary edges in sub regions that disturbing for make smooth edge map

To me this looks like a classic problem for the watershed algorithm. It is designed for segmenting out touching objects like the tomatoes. My example is in Matlab (I'm on the wrong computer today) but it should translate to python easily. First convert to greyscale as you do and then invert the images
I=rgb2gray(img)
I2=imcomplement(I)
The image as is will over segment, so we remove minima that are too shallow. This can be done with the h-minima transform
I3=imhmin(I2,50);
You might need to play with the 50 value which is the height threshold for suppressing shallow minima. Now run the watershed algorithm and we get the following result.
L=watershed(I3);
The results are not perfect. It needs additional logic to remove some of the small regions, but it will give a reasonable estimate. The watershed and h-minima are contained in the skimage.morphology package in python.

laser curved line detection using opencv and python_new

Hello, I am new to the opencv world. I would like to detect a bending edge by line laser triangulation. I have a picture with the line of the laser and would now like to edit the image so I only see an averaged lines (drawContour). The image was recorded only with the laser. The line is a little intertwined.
Now my questions / thought process: How can I convert the binarized (threshold) picture so that the line of a color value (everything white) and the rest is black?
Then I need a mean value (min-max, x, y) of all values ​​to characterize a line with the draw function. Is there a function for this?
This new mean value line then has to be in an array that I can do the triangulation. np.array []?
In my previous code I have tried some filters. Unfortunately, I do not come to the result which I imagine. Which filter would be the best? And how do I get the rest over a certain threshold on white? How can I make pixels black which should not be there? What kind of problems do I have? Do I go wrong? Please help me, over every contribution I would be very happy.
Thanks Daniel
My Code:
import cv2
import numpy as np
import matplotlib.pyplot as plt
Img1=cv2.imread('/home/pi/Desktop/6.jpg',0)
Img1=cv2.resize(Img1,(0,0),fx=0.5,fy=0.5)
#Thresh-Binär
retval,th1=cv2.threshold(Img1,30,255,cv2.THRESH_BINARY)
#Thres- Otsu
retval2,th2=cv2.threshold(Img1,30,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
#Thresh-Adaptive+Gauss
th3=cv2.adaptiveThreshold(Img1,255,cv2.ADAPTIVE_THRESH_GAUSSIAN_C, \
cv2.THRESH_BINARY,3,2)
#Gauss
blur=cv2.GaussianBlur(Img1,(5,0),4)
blur2=cv2.GaussianBlur(th1,(5,5),0)
kernel=cv2.getStructuringElement(cv2.MORPH_ELLIPSE,(10,10))
skinmask=cv2.erode(Img1,kernel,iterations=3)
skinmask=cv2.dilate(Img1,kernel,iterations=3)
skinmask=cv2.GaussianBlur(skinmask,(3,3),1)
#Morpholo Filter/noise removal
kernel1 = np.ones((3,2),np.uint8) #np.unit=unsigned interger(0-255)
opening = cv2.morphologyEx(blur,cv2.MORPH_OPEN,kernel, iterations = 2)
#Contour finder
im2,contours,hierarchy=cv2.findContours(skinmask,cv2.RETR_TREE,cv2.CHAIN_APPRO X_SIMPLE)
#kanten = cv2.Canny(opening,100,50)
#draw Contours
cnt=contours[0]
M=cv2.moments(cnt) #??
print(M)
Kontur=cv2.drawContours(Img1,contours,0,(255,255,255),5)
perimeter=cv2.arcLength(cnt,True)
cv2.imshow('Original',Img1)
cv2.imshow('Tresh',th1)
cv2.imshow('Tresh+Otsu',th2)
#cv2.imshow('TreshAdaptiv+Gauss',th3)
cv2.imshow('Kontur',Kontur)
#cv2.imshow('Tresh+Gaussian',blur2)
cv2.imshow('arc',perimeter)
cv2.imshow('Morph',opening)
cv2.imshow('skin',skinmask)
cv2.waitKey(0)
cv2.destroyAllWindows()
print(len(contours))
Edit 28.05.2017
Example

Detecting an approaching object

I read this blog post where he uses a Laser and a Webcam to estimated the distance of the cardboard from the Webcam.
I had another idea about that. I don't want to calculate the distance from the webcam.
I want to check if an object is approaching the webcam. The algorithm, according to me, will be something like:
Detect the object in the webcam feed.
If the object is approaching the webcam it'll grow larger and larger in the video feed.
Use this data for further calculations.
Since I want to detect random objects, I am using the findContours() method to find the contours in the video feed. Using that, I will at least have the outlines of the objects in the video feed. The source code is:
import numpy as np
import cv2
vid=cv2.VideoCapture(0)
ans, instant=vid.read()
average=np.float32(instant)
cv2.accumulateWeighted(instant, average, 0.01)
background=cv2.convertScaleAbs(average)
while(1):
_,f=vid.read()
imgray=cv2.cvtColor(f, cv2.COLOR_BGR2GRAY)
ret, thresh=cv2.threshold(imgray,127,255,0)
diff=cv2.absdiff(f, background)
cv2.imshow("input", f)
cv2.imshow("Difference", diff)
if cv2.waitKey(5)==27:
break
cv2.destroyAllWindows()
The output is:
I am stuck here. I have the contours stored in an array. What do I do with it when the size increases? How do I proceed?

One trouble here is recognising and differentiating the moving objects from other stuff in the video feed. An approach might be to let the camera 'learn' what the background looks like with no object. Then you can constantly compare its input against this background. One way to get the background is to use a running average.
Any difference greater than a small threshold means there is a moving object. If you constantly display this difference, you basically have a motion tracker. The size of the objects is simply the sum of all the non-zero (thresholded) pixels, or their bounding rectangles. You can track this size and use it to guess whether the object is moving closer or further. Morphological operations can help group the contours into one cohesive object.
Since it will be tracking ANY movement, if there are two objects, they will be counted together. Here is where you can use the contours to find and track individual objects, e.g. using the contour bounds or centroids. You could also possibly separate them by colour.
Here are some results using this strategy (the grey blob is my hand):
It actually did a fairly good job of guessing which way my hand was moving.
Code:
import cv2
import numpy as np
AVERAGE_ALPHA = 0.2 # 0-1 where 0 never adapts, and 1 instantly adapts
MOVEMENT_THRESHOLD = 30 # Lower values pick up more movement
REDUCED_SIZE = (400, 600)
MORPH_KERNEL = np.ones((10, 10), np.uint8)
def reduce_image(input_image):
"""Make the image easier to deal with."""
reduced = cv2.resize(input_image, REDUCED_SIZE)
reduced = cv2.cvtColor(reduced, cv2.COLOR_BGR2GRAY)
return reduced
# Initialise
vid = cv2.VideoCapture(0)
average = None
old_sizes = np.zeros(20)
size_update_index = 0
while (True):
got_frame, frame = vid.read()
if got_frame:
# Reduce image
reduced = reduce_image(frame)
if average is None: average = np.float32(reduced)
# Get background
cv2.accumulateWeighted(reduced, average, AVERAGE_ALPHA)
background = cv2.convertScaleAbs(average)
# Get thresholded difference image
movement = cv2.absdiff(reduced, background)
_, threshold = cv2.threshold(movement, MOVEMENT_THRESHOLD, 255, cv2.THRESH_BINARY)
# Apply morphology to help find object
dilated = cv2.dilate(threshold, MORPH_KERNEL, iterations=10)
closed = cv2.morphologyEx(dilated, cv2.MORPH_CLOSE, MORPH_KERNEL)
# Get contours
contours, _ = cv2.findContours(closed, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
cv2.drawContours(closed, contours, -1, (150, 150, 150), -1)
# Find biggest bounding rectangle
areas = [cv2.contourArea(c) for c in contours]
if (areas != list()):
max_index = np.argmax(areas)
max_cont = contours[max_index]
x, y, w, h = cv2.boundingRect(max_cont)
cv2.rectangle(closed, (x, y), (x+w, y+h), (255, 255, 255), 5)
# Guess movement direction
size = w*h
if size > old_sizes.mean():
print "Towards"
else:
print "Away"
# Update object size
old_sizes[size_update_index] = size
size_update_index += 1
if (size_update_index) >= len(old_sizes): size_update_index = 0
# Display image
cv2.imshow('RaptorVision', closed)
Obviously this needs more work in terms of identifying, selecting and tracking the objects etc (at the moment it does horribly if there is something else moving in the background). There are also many parameters to vary and tweak (the ones set are what worked well for my system). I'll leave that up to you though.
Some links:
background extraction
motion tracking
If you want to get a bit more high-tech with the background removal, have a look here:
wallflower

Detect the object in the webcam feed.
If the object is approaching the webcam it'll grow larger and larger in the video feed.
Use this data for further calculations.
Good idea.
If you want to use the contour detection approach, you could do it the following way:
You have a series of Images I1, I2, ... In
Do a contour detection on each one. C1, C2, ..., Cn (Contour is a set of points in OpenCV)
Take a large enough sample on your Image i and i+1: S_i \leq C_i, i \in 1...n
Check for all points in your sample for the nearest point on i+1. Then you trajectorys for all your points.
Check if this trajectorys point mostly outwards (tricky part ;)
If they appear outwards for a suffiecent number of frames your contour got bigger.
Alternative you could try to prune the points that are not part of the correct contour and work with a covering rectangle. It's very easy to check the size that way, but i don't knwo how easy it will be to choose the "correct" points.

OpenCV concave and convex corner points of polygons

Problem
I am working on a project where I need to get the bounding boxes of dumbell like shapes. However, I need the fewest points possible, and the boxes need to fit the shapes at all corners. Here's an Image I made to test: Blurry, cracked, dumbell shape
I don't care about the gaps going into the shape, I just want to clean it up, and straighten the edges so that I can get the contours of a shape like this: Cleaned up
I have been attempting to threshold() it out, getting the contours of it using findContours() and then using approxPolyDP() to simplify the crazy amount of points the contours end up being. So, after fiddling with this for about three days now, how can I simply get either:
Two boxes specifying the ends of the dumbell and a rectangle in the middle, or
One contour with the twelve points for all the corners
The second option would be preferred since that really is my ultimate goal: getting the points that are at those corners.
A few things to note:
I am using OpenCV for Python
There will generally be many of these shapes of all sizes all over the input image
They will have only horizontal or vertical positioning. No strange 27 degree angles...
What I need:
I really don't need someone to write the code for me, I just need some method or algorithm in order to get this done, preferably with some simple examples.
My Code
Here is my overly clean code with functions I don't even use but figure I would use them eventually:
import cv2
import numpy as np
class traceImage():
def __init__(self, imageLocation):
self.threshNum = 127
self.im = cv2.imread(imageLocation)
self.imOrig = self.im
self.imGray = cv2.cvtColor(self.im, cv2.COLOR_BGR2GRAY)
self.ret, self.imThresh = cv2.threshold(self.imGray, self.threshNum, 255, 0)
self.contours, self.hierarchy = cv2.findContours(self.imThresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
def createGray(self):
self.imGray = cv2.cvtColor(self.im, cv2.COLOR_BGR2GRAY)
def adjustThresh(self, threshNum):
self.ret, self.imThresh = cv2.threshold(self.imGray, threshNum, 255, 0)
def getContours(self):
self.contours, self.hierarchy = cv2.findContours(self.imThresh, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
def approximatePoly(self, percent):
i=0
for shape in self.contours:
shape = cv2.approxPolyDP(shape, percent*cv2.arcLength(shape, True), True)
self.contours[i] = shape
i+=1
def drawContours(self, blobWidth, color=(255,255,255)):
cv2.drawContours(self.im, self.contours, -1, color, blobWidth)
def newWindow(self, name):
cv2.namedWindow(name)
def showImage(self, window):
cv2.imshow(window, self.im)
def display(self):
while True:
cv2.waitKey()
def displayUntil(self, key):
while True:
pressed = cv2.waitKey()
if pressed == key:
break
if __name__ == "__main__":
blobWidth = 30
ti = traceImage("dumbell.png")
ti.approximatePoly(0.01)
for thresh in range(127,256):
ti.adjustThresh(thresh)
ti.getContours()
ti.drawContours(blobWidth)
ti.showImage("Image")
ti.displayUntil(10)
ti.createGray()
ti.adjustThresh(127)
ti.getContours()
ti.approximatePoly(0.0099)
ti.drawContours(2, (0,255,0))
ti.showImage("Image")
ti.display()
Code Explanation
I know I might not be doing some things right here, but hey, I'm proud of it :)
So, the idea is that there are very often holes and gaps in these dumbells and so I figured that if I iterated through all the threshold values from 127 to 255 and drew the contours onto the image with large enough thickness, the thickness of drawing the contours would fill in any small enough holes, and I could use the new, blobby image to get the edges and then scale the sides back down to size. That was my thinking. There's got to be another, beter way though...
Summary
I want to end up with 12 points; one for each corner of the shape.
EDIT:
After trying out some erosion and dilation, it seems that the best option would be to slice the contours at certain points and then use bounding boxes around the sliced shapes to get the right boxy corners, and then doing some calculations to rejoin the boxes into one shape. A rather interesting challenge...
EDIT 2:
I discovered something that works well! I made my own line detection system, that only detects horizontal or vertical lines, and then on a detected line/contour edge, the program draws a black line that extends across the whole image, thus effectively slicing the image at the straight lines of the contours. Once it does that, it gets new contours of the sliced up boxes, draws bounding boxes around the pieces and then uses dilation to close the gaps. So far, it works well on large shapes, but when the shapes are small, it tends to lose a bit of the shape.

So, after fiddling with erosion, dilation, closing, opening, and looking at straight contours, I have figured out a solution that works. Thank you #Ante and #a.alsram! Your two ideas combined helped me get to my solution. So here's how it works.
Method
The program iterates over each contour, and over every pair of points in the contour, looking for point pairs that lie on the same axis and calculating the distance between them. If the distance is greater than an adjustable threshold, the program decides that those points are considered an edge on the shape. Then the program uses that edge, and draws a black line along the whole contour, thus cutting the contour at that edge. Then the program redetermines contours and since the shape was cut. These pieces that were cut off are know their own contours, which then are bounded by bounding boxes. and finally, all shapes are dilated and eroded (close) to rejoin the boxes that were cut off.
This method can be done several times, but each time there is a little bit of accuracy loss. But it works for what I need and certainly was a fun challenge! Thanks for your help guys!
natebot13

Maybe simple solution can help. If there is a threshold length to close a gaps,
it is possible to split image in a grid with cell lengths >= threshold, and use
cells that have something inside. With that there will be only horizontal and
vertical lines, and by taking a care about grid to follow original horizontal
and vertical lines it will cover main line features.
Update
Take a look on mathematical morphology. I think closing operation with structuring element (2*k+1)x(2*k+1) pixels can do what you are looking for.
Algorithm should take threshold parameter k, and performs dilation and than erosion. That means change image so that for each white pixel set all neighbours on distance k ((2*k+1)x(2*k+1) box) to the white, and than change image so that for each black pixel set neighbours on distance k to the black.
It is enough to do operations on boundary pixels.