We Keep Coding

corner_fast return from skimage

I just tried corner_fast form skimage and it seems to work pretty well for corner detection:
from skimage.feature import corner_fast
np.random.seed(2018)
img = np.random.normal(size=(20, 20))
img_response = corner_fast(img, n=12, threshold=0.0)
The FAST algorithm is explained in numerous places in the web. However, these explanations suggest FAST should return a boolean value (corner or not-corner).
Yet, img_response is a float array. I guess these numbers correspond to how "cornery" a specific pixel is, but, how are they computed? What do they really represent?

Your guess is right, the response image is a sort of accumulator, the higher the value, the more "cornery" the pixel is.
To extract the corners location, you can call the function corner_peaks on the result returned by corner_fast
About the how to is it computed, the documentation gives these two references:
[1] Edward Rosten and Tom Drummond “Machine Learning for high-speed corner detection”, http://www.edwardrosten.com/work/rosten_2006_machine.pdf
[2] Wikipedia, “Features from accelerated segment test”, https://en.wikipedia.org/wiki/Features_from_accelerated_segment_test
Finally, just know that scikit-image is open source and you can therefore go see the details of the code itself https://github.com/scikit-image/scikit-image/tree/master/skimage

How to find a template in an image using a mask (or transparency) with OpenCV and Python?

Let us assume we are looking for this template:
The corners of our template are transparent, so the background will vary, like so:
Assuming we could use the following mask with our template:
It would be very easy to find it.
What I have tried:
I have tried matchTemplate but it doesn't support masks (as far as I know), and using the alpha channel (transparency) in the template does not achieve this, as it compares the alpha channels instead of ignoring those pixels.
I have also looked into "region of interest", which I thought would be the solution, but with it you can only specify a rectangular area. I'm not even sure if it works on the template or not.
I'm sure this is possible to do by writing my own algorithm, but I was hoping this is possible via. standard OpenCV to avoid reinventing the wheel. Not to mention, it would most likely be more optimised than my own.
So, how could I do something like this with OpenCV + Python?

This could be achieved using only matchTemplate function, but a little workaround is needed.
Lets analyse the default metrics(CV_TM_SQDIFF_NORMED). According to matchTemplate documentation
the default metrics looks like this
R(x, y) = sum (I(x+x', y+y') - T(x', y'))^2
Where I is image matrix, T is template, R is result matrix. Summation is done over template coordinates x' and y',
So, lets alter this metrics by inserting weight matrix W, which has the same dimensions as
T.
Q(x, y) = sum W(x', y')*(I(x+x', y+y') - T(x', y'))^2
In this case, by setting W(x', y') = 0 you can actually make pixel be ignored. So, how to make such metrics? With simple math:
Q(x, y) = sum W(x', y')*(I(x+x', y+y') - T(x', y'))^2
= sum W(x', y')*(I(x+x', y+y')^2 - 2*I(x+x', y+y')*T(x', y') + T(x', y')^2)
= sum {W(x', y')*I(x+x', y+y')^2} - sum{W(x', y')*2*I(x+x', y+y')*T(x', y')} + sum{W(x', y')*T(x', y')^2)}
So, we divided Q metrics into tree separate sums. And all those sums could be calculated
with matchTemplate function (using CV_TM_CCORR method). Namely
sum {W(x', y')*I(x+x', y+y')^2} = matchTemplate(I^2, W, method=2)
sum{W(x', y')*2*I(x+x', y+y')*T(x', y')} = matchTemplate(I, 2*W*T, method=2)
sum{W(x', y')*T(x', y')^2)} = matchTemplate(T^2, W, method=2) = sum(W*T^2)
The last element is a constant, so, for minimisation it does not have any effect. On the other hand, it still might me useful to see if our template have perfect match (if Q is approaching to zero). Nonetheless, for last element we actually do not need matchTemplate function, since it could be calculated directly.
The final pseudocode looks like this:
result = matchTemplate(I^2, W, method=2) - matchTemplate(I, 2*W*T, method=2) + as.scalar(sum(W*T^2))
Does it really do exactly as defined? Mathematically yes.
Practically, there is some small rounding error, because matchTemplate function
works on 32-bit floating-point, but I believe it is not a big problem.
Please note, that you can extent analysis and have weighted equivalents for any metrics offered by matchTemplate.
This actually worked for me. I am sorry I don't give actual code. I am working in R, so
I don't have the code in Python. But idea is quite straightforward.
I hope this will help.

What worked for me the one time I needed this was to fill the "mask" areas with white noise. Then it gets effectively washed out of the correlation when looking for matches. Otherwise I got, as I presume you did, false matches on the masked areas.

One answer to your question is convolution. Use the template as kernel and filter the image.
The destination Mat will have dense bright areas where your template might be. You'll have to cluster the results (e.g. Mean-shift).
In that way, you'll have a very simplistic implementation of the Generalized Hough Transform or a Template-based convolution matching.

Imagemagick 7.0.3.9 now has a masked compare capability so that you can limit the template matching region. See http://www.imagemagick.org/discourse-server/viewtopic.php?f=4&t=31053
Also, I see that OpenCV 3.0 now has masked template matching. See http://docs.opencv.org/3.0.0/df/dfb/group__imgproc__object.html#ga586ebfb0a7fb604b35a23d85391329be
However, it is only for method == CV_TM_SQDIFF and method == CV_TM_CCORR_NORMED. see python opencv matchTemplate is mask feature implemented?

ImageMagick has logic for finding subimages in other images and it works quite well.
compare -verbose -dissimilarity-threshold 0.1 -subimage-search subimage bigimage
I've used it to find and blur watermarks off some products. Don't ask.
(Sometimes you have to do what you have to do..)

2021 Update: I've been trying to find a solution for transparency in templates throughout the day, and I think I finally found a way to do it. matchTemplate() has a mask parameter, which apparently works exactly like OP wants it to: ignore certain pixels from a template when searching for it in another image. And since my templates already contain transparency in them, I decided to use my template as both a template and mask parameter. Surprisingly, it worked.
I'm using JavaScript with opencv4nodejs, so the following python code snippet might be completely off, but the theory is there and I'm fairly positive it should work.
# Import OpenCV
import cv2 as cv
# Read both the image and the template
image = cv.imread("image.png", cv.IMREAD_COLOR)
template = cv.imread("template.png", cv.IMREAD_COLOR)
# Match with template as both template and mask parameter
result = cv.matchTemplate(image, template, cv.TM_CCORR_NORMED, None, template)
Here's a gist for JavaScript with opencv4nodejs if you're interested.
Now that I think about it, it seems really stupid and way too good to be true, but I've been getting good matches (0.98+) on most tests. Hope this helps!

calculating mean of several numpy masked arrays (masked_all)

first of all I'm new to python and programming but you guys already helped me a lot, so thanks a lot! But I've come to a problem I haven't found an answer so far:
I have the data of several plates where the data represents the pressure on each plate at a large number of different spots. The thing is, these plates aren't perfectly round because of the sensors measuring the pressure and sometimes these sensors even produce an error so I don't have any data at a spot within the plate.
When I just have to plot one plate, I'll do it like that:
import numpy.ma as ma
matrix=ma.masked_all((160,65),float)
for x in range(len(plate.X)):
matrix[(plate.Y[x],plate.X[x])]=data.index(plate.measurementname[x])
image.pcolormesh(matrix,min,max)
This works fine. Now that I have several plates I'd like to plot the mean pressure on each spot. Because I don't know any mean function, I thought of adding all plates together and divide by the number of plates...I tried following:
import numpy.ma as ma
meanmatrix=ma.masked_all((160,65),float)
for plate in plateslist:
matrix=ma.masked_all((160,65),float)
for x in range(len(plate.X)):
matrix[(plate.Y[x],plate.X[x])]=data.index(plate.measurementname[x])
meanmatrix+=matrix
meanmatrix=meanmatrix/len(plateslist)
image.pcolormesh(meanmatrix,min,max)
This works pretty good but there's one problem I can't solve. As I said sometimes some plates didn't get all data, therefore there's a "hole" at some spots in the plot. Now my meanmatrix has a whole where ever one of the plates had a whole even if all others had data at that spot.
How can I make sure I won't get these holes or is there even a smoother way of getting my "meanmatrix"?? (I hope my question is clear enough...)
Edit:
The problem is not that I don't get the mean of the data, this actually works (well I don't like how I did it but it works), the problem is that I get these "holes" I described before. That's what bothers me.

EDIT: Sorry, I misinterpreted the question. Try this:
allplates = ma.masked_all((160, 65, numplates))
# fill in allplates
meanplate = allplates.mean(axis=2)
This will compute the mean over the last dimension of the array, i.e., average the plates together. Missing values are ignored.
Earlier answer: You can take the mean of a masked array, and it will ignore the missing values:
>>> X = ma.masked_all((160, 65))
>>> X.mean()
masked
>>> X[0, 0] = 1
>>> X.mean()
1.0
Try to avoid using matrix as a variable name, though, because it also refers to a NumPy data structure.

Ok I got an answer:
import numpy.ma as ma
allplats=ma.masked_all((160,65),float)
for plate in plateslist:
for x in range(len(plate.X)):
allplates[(plate.Y[x],plate.X[x])]+=data.index(plate.measurementname[x])
allplates=allplates/len(plateslist)
image.pcolormesh(meanmatrix,min,max)
This actually works! So i guess there was a mistake when adding two masked_all arrays...("Stupid is as stupid does")
If someone has a better approach to get the mean of all plates at each single spot, it would be nice to read it.

How to find an image within another image using python

I'm trying to use python to determine if one (small) image is within another (large) image.
Any suggestions before I take myself completely down the wrong path?
/edit: Ok, some ideas: I'm using PIL, and I'm converting each image to the 'P' mode so I can compare each pixel as an integer. I'm trying to implement something like a Boyer–Moore string search or the Knuth–Morris–Pratt algorithm, but in 2 dimensions.
Maybe this will help: instead of searching for ABC in XXXABCXXX (answer=4) we are searching for
ABC
DEF
GHI
in
XXXXX
XABCX
XDEFX
XGHIX
XXXXX
(answer=(2,2))

EDIT: Ok, here is the naive way to do this:
import Image, numpy
def subimg(img1,img2):
img1=numpy.asarray(img1)
img2=numpy.asarray(img2)
#img1=numpy.array([[1,2,3],[4,5,6],[7,8,9]])
#img2=numpy.array([[0,0,0,0,0],[0,1,2,3,0],[0,4,5,6,0],[0,7,8,9,0],[0,0,0,0,0]])
img1y=img1.shape[0]
img1x=img1.shape[1]
img2y=img2.shape[0]
img2x=img2.shape[1]
stopy=img2y-img1y+1
stopx=img2x-img1x+1
for x1 in range(0,stopx):
for y1 in range(0,stopy):
x2=x1+img1x
y2=y1+img1y
pic=img2[y1:y2,x1:x2]
test=pic==img1
if test.all():
return x1, y1
return False
small=Image.open('small.tif')
big=Image.open('big.tif')
print subimg(small, big)
It works just fine, but I want to SPEED IT UP. I think the key is in the array 'test' which we might be able to use to skip some positions in the image.
Edit 2: Make sure you use images in a loss-less format to test this.
On Mac, install Pillow and from PIL import Image

Sikuli does it using OpenCV, see here how match_by_template works and then use the Python OpenCV bindings to do the same. Doing it without OpenCV should be hard, take a look at OpenCV documentation, search for template matching, etc...

pyautogui module does the job using pyautogui.locate(small_image, large_image) method which returns 4-integer tuple: (left, top, width, height).

I know it's a little late, but you can use Boyer-Moore to search for the first line of the small image in each of the lines of the large image. The moment you find a match you have the X and Y position and you just have to check if the remainder of the lines of the smaller image match the remainder of the lines of the larger image starting at position X and Y+1,2,3,... At the first mismatch continue with the search of the first line. I don't think you can get faster than this.

Have a look at my answer to a similar question for a code example using OpenCV. The conversion from PIL to numpy is straight forward, e.g. just use np.array(pilimage).

Image Gurus: Optimize my Python PNG transparency function

I need to replace all the white(ish) pixels in a PNG image with alpha transparency.
I'm using Python in AppEngine and so do not have access to libraries like PIL, imagemagick etc. AppEngine does have an image library, but is pitched mainly at image resizing.
I found the excellent little pyPNG module and managed to knock up a little function that does what I need:
make_transparent.py
pseudo-code for the main loop would be something like:
for each pixel:
if pixel looks "quite white":
set pixel values to transparent
otherwise:
keep existing pixel values
and (assuming 8bit values) "quite white" would be:
where each r,g,b value is greater than "240"
AND each r,g,b value is within "20" of each other
This is the first time I've worked with raw pixel data in this way, and although works, it also performs extremely poorly. It seems like there must be a more efficient way of processing the data without iterating over each pixel in this manner? (Matrices?)
I was hoping someone with more experience in dealing with these things might be able to point out some of my more obvious mistakes/improvements in my algorithm.
Thanks!

This still visits every pixel, but may be faster:
new_pixels = []
for row in pixels:
new_row = array('B', row)
i = 0
while i < len(new_row):
r = new_row[i]
g = new_row[i + 1]
b = new_row[i + 2]
if r>threshold and g>threshold and b>threshold:
m = int((r+g+b)/3)
if nearly_eq(r,m,tolerance) and nearly_eq(g,m,tolerance) and nearly_eq(b,m,tolerance):
new_row[i + 3] = 0
i += 4
new_pixels.append(new_row)
It avoids the slicen generator, which will be copying the entire row of pixels for every pixel (less one pixel each time).
It also pre-allocates the output row by directly copying the input row, and then only writes the alpha value of pixels which have changed.
Even faster would be to not allocate a new set of pixels at all, and just write directly over the pixels in the source image (assuming you don't need the source image for anything else).

Honestly, the only heuristic I could conceive is picking a few arbitrary, random points on your image and using a flood fill.
This only works well if your image as large contiguous white portions (if your image is an object with no or little holes in front of a background, then you're in luck -- you actually have a heuristic for which points to flood fill from).
(disclaimer: I am no image guru =/ )

I'm quite sure there is no short cut for this. You have to visit every single pixel.

The issue seems to have more to do with loops in Python than with images.
Python loops are extremely slow, it is best to avoid them and use built-ins loop operators instead.
Here, if you were willing to copy the image, you could use a list comprehension:
def make_transparent(pixel):
if pixel looks "quite white": return transparent
else: return pixel
newImage = [make_transparent(p) for p in oldImage]