Background: I have images I need to compare for differences. The images are large (on the order of 1400x9000 px), machine-generated and highly constrained (screenshots of a particular piece of linear UI), and are expected to be nearly identical, with differences being one of the following three possibilities:
Image 1 has a section image 2 is missing
Image 1 is missing a section image 2 has
Both images have the given section, but its contents differ
I'm trying to build a tool that highlights the differences for a human reviewer, essentially an image version of line-oriented diff. To that end, I'm trying to scan the images line by line and compare them to decide if the lines are identical. My ultimate goal is an actual diff-like output, where it can detect that sections are missing/added/different, and sync the images up as soon as possible for the remaining parts of identical content, but for the first cut, I'm going with a simpler approach where the two images are overlaid (alpha blended), and the lines which were different highlighted with a particular colour (ie. alpha-blended with a third line of solid colour). At first I tried using Python Imaging Library, but that was far several orders of magnitude too slow, so I decided to try using vips, which should be way faster. However, I have absolutely no idea how to express what I'm after using vips operations. The pseudocode for the simpler version would be essentially:
out = []
# image1 and image2 are expected, but not guaranteed to have the same height
# they are likely to have different heights if different
# most lines are entirely white pixels
for line1, line2 in zip(image1, image2):
if line1 == line2:
out.append(line1)
else:
# ALL_RED is a line composed of solid red pixels
out.append(line1.blend(line2, 0.5).blend(ALL_RED, 0.5))
I'm using pyvips in my project, but I'm also interested in code using plain vips or any other bindings, since the operations are shared and easily translated across dialects.
Edit: adding sample images as requested
Edit 2: full size images with missing/added/changed sections:
reference
comparison
How about just using diff? It's pretty quick. All you need to do is turn your PNGs into text a scanline a time, then parse the diff output.
For example:
#!/usr/bin/env python3
import sys
import os
import re
import pyvips
# calculate a checksum for each scanline and write to name_out
def scanline_checksum(name_in, name_out):
a = pyvips.Image.new_from_file(name_in, access="sequential")
# unfold colour channels to make a wider 1-band image
a = a.bandunfold()
# xyz makes an index image, where the value of each pixel is its coordinate
b = pyvips.Image.xyz(a.width, a.height)
# make a pow gradient image ... each pixel is some power of the x coordinate
b = b[0] ** 0.5
# now multiply and sum to make a checksum for each scanline
# "project" returns sum of columns, sum of rows
sum_of_columns, sum_of_rows = (a * b).project()
sum_of_rows.write_to_file(name_out)
to_csv(sys.argv[1], "1.csv")
to_csv(sys.argv[2], "2.csv")
os.system("diff 1.csv 2.csv > diff.csv")
for line in open("diff.csv", "r"):
match = re.match("(\\d+),(\\d+)c(\\d+),(\\d+)", line)
if not match:
continue
print(line)
For your two test images I see:
$ time ./diff.py 1.png 2.png
264,272c264,272
351,359c351,359
real 0m0.346s
user 0m0.445s
sys 0m0.033s
On this elderly laptop. All you need to do is use those "change" commands to mark up your images.
If OpenCV and NumPy are options to you, then there would be a quite simple solution at least for finding and coloring different rows.
In my approach, I just calculate pixel-wise differences using np.abs, and find non-zero row indices with np.nonzero. With these found row indices, I set up an additional black image and draw red lines for each row. The final blending is just some linear mixing:
0.5 * image1 + 0.5 * image2
for all equal rows, or
0.333 * image1 + 0.333 * image2 + 0.333 * red
for all different rows.
Here's the final code:
import cv2
import numpy as np
# Load images
first = cv2.imread('9gOlq.png', cv2.IMREAD_COLOR)
second = cv2.imread('1Hdx4.png', cv2.IMREAD_COLOR)
# Calcluate absolute differences between images
diff = np.abs(np.float32(first) - np.float32(second))
# Find all non-zero rows
nz_rows = np.unique(np.nonzero(diff)[0])
# Set up image with red lines
red = np.zeros(first.shape, np.uint8)
red[nz_rows, :, :] = [0, 0, 255]
# Set up output image
output = np.uint8(0.5 * first + 0.5 * second)
output[nz_rows, :, :] = 0.333 * first[nz_rows, :, :] + 0.333 * second[nz_rows, :, :] + 0.333 * red[nz_rows, :, :]
# Show results
cv2.imshow("diff", np.array(diff, dtype=np.uint8))
cv2.imshow("output", output)
cv2.waitKey()
cv2.destroyAllWindows()
The difference image diff looks like this:
The final output looke like this:
It would be interesting to see two input images with omitted sections as you described in your question. Also, testing this approach using original sized images would be necessary, since you mentioned time is crucial.
Anyway - hope that helps!
Related
The best code for mosaic I've found you can see at this page:
https://github.com/codebox/mosaic
However, the code doesn't work well on my Windows computer, and also I think the code is too advanced for what it should do. Here are my requirements I've posted on reddit:
1) The main photo already has reduced number of colors (8)
2) I have already every image associated with colour needed to be replaced (e.g. number 1 is supposed to replace black pixels, number 2 replaces green pixels...)
3) I need to enlarge the photo by the small photo's size (9 x 9 small photos will produce 81 times bigger image), which should push the pixels "2n" points away from each other, but instead of producing a n x n same-coloured area around every single one of them (this is how I believe enlarging works in general, correct me if I'm wrong), it will just colour the white spaces with unrecognized colour, which is not associated with any small photo (let's call that colour C)
4) Now all it needs is to run through all non-C coloured pixels and put an image centered on that pixel, which would create the mosaic.
Since I'm pretty new to Python (esp. graphics) and need it just for one use, could someone help me with creating that code? I think that code I got inspired with is too complicated. Two things I don't need:
1) "approximation" - if the enlargement is lesser than needed for 100% quality (e.g. the pictures are 9x9, but every side of the original photo can be only 3 times larger, then the program needs to merge some pixels of different colours together, leading to quality loss)
2) association colour - picture: my palette of pictures is small and of colours as well, I can do it manually
For the ones who didn't get what I mean, here is my idea: https://ibb.co/9GNhqBx
I had a quick go using pyvips:
#!/usr/bin/python3
import sys
import os
import pyvips
if len(sys.argv) != 4:
print("usage: tile-directory input-image output-image")
sys.exit(1)
# the size of each tile ... 16x16 for us
tile_size = 16
# load all the tile images, forcing them to the tile size
print(f"loading tiles from {sys.argv[1]} ...")
for root, dirs, files in os.walk(sys.argv[1]):
tiles = [pyvips.Image.thumbnail(os.path.join(root, name), tile_size,
height=tile_size, size="force")
for name in files]
# drop any alpha
tiles = [image.flatten() if image.hasalpha() else image
for image in tiles]
# copy the tiles to memory, since we'll be using them many times
tiles = [image.copy_memory() for image in tiles]
# calculate the average rgb for an image, eg. image -> [12, 13, 128]
def avg_rgb(image):
m = image.stats()
return [m(4,i)[0] for i in range(1,4)]
# find the avg rgb for each tile
tile_colours = [avg_rgb(image) for image in tiles]
# load the main image ... we can do this in streaming mode, since we only
# make a single pass over the image
main = pyvips.Image.new_from_file(sys.argv[2], access="sequential")
# find the abs of an image, treating each pixel as a vector
def pyth(image):
return sum([band ** 2 for band in image.bandsplit()]) ** 0.5
# calculate a distance map from the main image to each tile colour
distance = [pyth(main - colour) for colour in tile_colours]
# make a distance index -- hide the tile index in the bottom 16 bits of the
# distance measure
index = [(distance[i] << 16) + i for i in range(len(distance))]
# find the minimum distance for each pixel and mask out the bottom 16 bits to
# get the tile index for each pixel
index = index[0].bandrank(index[1:], index=0) & 0xffff
# replicate each tile image to make a set of layers, and zoom the index to
# make an index matching the output size
layers = [tile.replicate(main.width, main.height) for tile in tiles]
index = index.zoom(tile_size, tile_size)
# now for each layer, select pixels matching the index
final = pyvips.Image.black(main.width * tile_size, main.height * tile_size)
for i in range(len(layers)):
final = (index == i).ifthenelse(layers[i], final)
print(f"writing {sys.argv[3]} ...")
final.write_to_file(sys.argv[3])
I hope it's easy to read. I can run it like this:
$ ./mosaic3.py smallpic/ mainpic/Use\ this.jpg x.png
loading tiles from smallpic/ ...
writing x.png ...
$
It takes about 5s on this 2015 laptop and makes this image:
I had to shrink it for upload, but here's a detail (bottom left of the first H):
Here's a google drive link to the mosaic, perhaps it'll work: https://drive.google.com/file/d/1J3ofrLUhkuvALKN1xamWqfW4sUksIKQl/view?usp=sharing
And here's this code on github: https://github.com/jcupitt/mosaic
I use OpenCV and Python and I want to remove the small connected object from my image.
I have the following binary image as input:
The image is the result of this code:
dilation = cv2.dilate(dst,kernel,iterations = 2)
erosion = cv2.erode(dilation,kernel,iterations = 3)
I want to remove the objects highlighted in red:
How can I achieve this using OpenCV?
How about with connectedComponentsWithStats (doc):
# find all of the connected components (white blobs in your image).
# im_with_separated_blobs is an image where each detected blob has a different pixel value ranging from 1 to nb_blobs - 1.
nb_blobs, im_with_separated_blobs, stats, _ = cv2.connectedComponentsWithStats(im)
# stats (and the silenced output centroids) gives some information about the blobs. See the docs for more information.
# here, we're interested only in the size of the blobs, contained in the last column of stats.
sizes = stats[:, -1]
# the following lines result in taking out the background which is also considered a component, which I find for most applications to not be the expected output.
# you may also keep the results as they are by commenting out the following lines. You'll have to update the ranges in the for loop below.
sizes = sizes[1:]
nb_blobs -= 1
# minimum size of particles we want to keep (number of pixels).
# here, it's a fixed value, but you can set it as you want, eg the mean of the sizes or whatever.
min_size = 150
# output image with only the kept components
im_result = np.zeros_like(im_with_separated_blobs)
# for every component in the image, keep it only if it's above min_size
for blob in range(nb_blobs):
if sizes[blob] >= min_size:
# see description of im_with_separated_blobs above
im_result[im_with_separated_blobs == blob + 1] = 255
Output :
In order to remove objects automatically you need to locate them in the image.
From the image you provided I see nothing that distinguishes the 7 highlighted items from others.
You have to tell your computer how to recognize objects you don't want. If they look the same, this is not possible.
If you have multiple images where the objects always look like that you could use template matching techniques.
Also the closing operation doesn't make much sense to me.
#For isolated or unconnected blobs: Try this (you can set noise_removal_threshold to whatever you like and make it relative to the largest contour for example or a nominal value like 100 or 25).
mask = np.zeros_like(img)
for contour in contours:
area = cv2.contourArea(contour)
if area > noise_removal_threshold:
cv2.fillPoly(mask, [contour], 255)
Removing small connected components by area is called area opening. OpenCV does not have this as a function, it can be implemented as shown in other answers. But most other image processing packages will have an area opening function.
For example using scikit-image:
import skimage
import imageio.v3 as iio
img = iio.imread('cQMZm.png')[:,:,0]
out = skimage.morphology.area_opening(img, area_threshold=150, connectivity=2)
For example using DIPlib:
import diplib as dip
out = dip.AreaOpening(img, filterSize=150, connectivity=2)
PS: The DIPlib implementation is noticeably faster. Disclaimer: I'm an author of DIPlib.
I am doing some image processing, and I need to check if a binary image is identical to another.
Processing speed isn't an issue, and the simple thing I thought to do was count the white pixels remaining after adding the inverse of image A to image B (these images are very nearly identical, but not quite--some sort of distance metric is the goal).
Note: take the logarithm to linearize the distance
However, in order to create the composite image, I need to include a "mask" that is the same size as the two images.
I am having trouble finding an example of creating the mask online and using it for the Image.composite function.
Here is my code:
compA = ImageOps.invert(imgA)
imgAB = Image.composite(compA,imgB,??? mask)
Right now, I have created a mask of all zeros--however, the composite image does not appear correctly (both A and B are exactly the same images; a mask of all zeros--or all ones for that matter--does not work).
mask = Image.fromarray(np.zeros(imgA.size,dtype=int),mode='L')
imgAB = Image.composite(compA,imgB,mask)
How do I just add these two binary images on top of eachother?
Clearly you're using numpy, so why not just work with numpy arrays and explicitly do whatever arithmetic you want to do in that domain—such as subtracting one image from the other:
arrayA = numpy.asarray( imgA, dtype=int )
arrayB = numpy.asarray( imgB, dtype=int )
arrayDelta = arrayA - arrayB
print( (arrayDelta !=0 ).sum() ) # print the number of non-identical pixels (why count them by hand?)
# NB: this number may be inflated by a factor of 3 if there are 3 identical channels R, G, B
imgDelta = Image.fromarray((numpy.sign(arrayDelta)*127+127).astype('uint8')) # display this image if you want to visualize where the differences are
You could do this even more simply, e.g.
print((numpy.asarray(imgA) != numpy.asarray(imgB)).sum())
but I thought casting to a signed integer type first and then subtracting would allow you to visualize more information (A white and B black -> white pixel in delta; A black and B white -> black pixel in delta)
import cv2
import numpy as np
from PIL import Image
from skimage import morphology
from scipy import signal
img = cv2.imread('thin.jpg',0)
img1 = cv2.imread('thin1.jpg',0)
cv2.imshow('image1',img)
cv2.imshow('image2',img1)
ret,img = cv2.threshold(img,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
ret,img1 = cv2.threshold(img1,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)
size = np.size(img)
size1 = np.size(img1)
skel = np.zeros(img.shape,np.uint8)
skel1 = np.zeros(img1.shape,np.uint8)
element = cv2.getStructuringElement(cv2.MORPH_CROSS,(3,3))
img = 255 - img
img1 = 255 - img1
img = cv2.dilate(img,element,iterations=8)
img1 = cv2.dilate(img1,element,iterations=8)
done = False
while(not done):
eroded = cv2.erode(img,element)
eroded1 = cv2.erode(img1,element)
temp = cv2.dilate(eroded,element)
temp1 = cv2.dilate(eroded1,element)
temp = cv2.subtract(img,temp)
temp1 = cv2.subtract(img1,temp1)
skel = cv2.bitwise_or(skel,temp)
skel1 = cv2.bitwise_or(skel1,temp1)
img = eroded.copy()
img1 = eroded1.copy()
zeros = size - cv2.countNonZero(img)
if zeros==size:
done = True
cv2.imshow('IMAGE',skel)
cv2.imshow('TEMPLATE',skel1)
cv2.imwrite("image.jpg",skel)
if cv2.waitKey(0) & 0xFF == ord('q'):
cv2.destroyAllWindows()
This is the code that i tried to convert two grayscale image to two skeletized image using the method of binarization and thinning and the result is also obtained. Now with these two skeletized image , i want to do a comparison to see whether they match or not. How can i correlate each other? Do we need to convert this skeletized into 2d array? Can anyone suggest any solution. Thanks in advance.
There are a number of ways you can compare the images to see if they match. The simplest is to do a pixelwise subtraction to create a new image and then sum the pixels in the new image. If they sum to zero you have an exact match. The larger the sum the worse the match.
You will however have a problem using most comparison techniques on a skeletonized image. You take the image and reduce it to skinny little lines that are unlikely to overlap for images that only deviate from each other by a little bit.
With skeletonized images you often need to compare features. For example, identify the points of intersection of the skeleton, and use the location of those points for comparing images. In your sample image you might be able to extract the lines (I see three major ones) and then compare images based on the location of the lines.
Binary images are already represented as 2D numpy arrays.
This is a complex problem. You can do this by reshaping the images to two vectors (assuming they are exactly the same size), and then calculating the correlation coefficient:
np.corrcoef(img.reshape(-1), img1.reshape(-1))
One possible solution would be to correlate (or subtract) the blurred version of each skeletonized image with one another.
That way, the unavoidable little offsets between skeleton lines wouldn't have such a negative impact on the outcome as if you subtracted the skeletons directly (since the skeleton lines would most probably not overlay exactly over one another).
I'm assuming here that the original images weren't similar to each other in the first place, otherwise you wouldn't need to skeletonize them, right?
I have a multiband satellite image stored in the band interleaved pixel (BIP) format along with a separate header file. The header file provides the details such as the number of rows and columns in the image, and the number of bands (can be more than the standard 3).
The image itself is stored like this (assume a 5 band image):
[B1][B2][B3][B4][B5][B1][B2][B3][B4][B5] ... and so on (basically 5 bytes - one for each band - for each pixel starting from the top left corner of the image).
I need to separate out each of these bands as PIL images in Python 3.2 (on Windows 7 64 bit), and currently I think I'm approaching the problem incorrectly. My current code is as follows:
def OpenBIPImage(file, width, height, numberOfBands):
"""
Opens a raw image file in the BIP format and returns a list
comprising each band as a separate PIL image.
"""
bandArrays = []
with open(file, 'rb') as imageFile:
data = imageFile.read()
currentPosition = 0
for i in range(height * width):
for j in range(numberOfBands):
if i == 0:
bandArrays.append(bytearray(data[currentPosition : currentPosition + 1]))
else:
bandArrays[j].extend(data[currentPosition : currentPosition + 1])
currentPosition += 1
bands = [Image.frombytes('L', (width, height), bytes(bandArray)) for bandArray in bandArrays]
return bands
This code takes way too long to open a BIP file, surely there must be a better way to do this. I do have the numpy and scipy libraries as well, but I'm not sure how I can use them, or if they'll even help in any way.
Since the number of bands in the image are also variable, I'm finding it hard to figure out a way to read the file quickly and separate the image into its component bands.
And just for the record, I have tried messing with the list methods in the loops (using slices, not using slices, using only append, using only extend etc), it doesn't particularly make a difference as the major time is lost because of the number of iterations involved - (width * height * numberOfBands).
Any suggestions or advice would be really helpful. Thanks.
If you can find a fast function to load the binary data in a big python list (or numpy array), you can de-interleave the data using the slicing notation:
band0 = biglist[::nbands]
band1 = biglist[1::nbands]
....
Does that help?
Standard PIL
To load an image from a file, use the open function in the Image module.
>>> import Image
>>> im = Image.open("lena.ppm")
If successful, this function returns an Image object. You can now use instance attributes to examine the file contents.
>>> print im.format, im.size, im.mode
PPM (512, 512) RGB
The format attribute identifies the source of an image. If the image was not read from a file, it is set to None. The size attribute is a 2-tuple containing width and height (in pixels). The mode attribute defines the number and names of the bands in the image, and also the pixel type and depth. Common modes are "L" (luminance) for greyscale images, "RGB" for true colour images, and "CMYK" for pre-press images.
The Python Imaging Library also allows you to work with the individual bands of an multi-band image, such as an RGB image. The split method creates a set of new images, each containing one band from the original multi-band image. The merge function takes a mode and a tuple of images, and combines them into a new image. The following sample swaps the three bands of an RGB image:
Splitting and merging bands
r, g, b = im.split()
im = Image.merge("RGB", (b, g, r))
So I think you should simply derive the mode and then split accordingly.
PIL with Spectral Python (SPy python module)
However, as you pointed out in your comments below, you are not dealing with a normal RGB image with 3 bands. So to deal with that, SpectralPython (a pure python module which requires PIL) might just be what you are looking for.
Specifically - http://spectralpython.sourceforge.net/class_func_ref.html#spectral.io.bipfile.BipFile
spectral.io.bipfile.BipFile deals with Image files with Band Interleaved Pixel (BIP) format.
Hope this helps.
I suspect that the repetition of extend is not good better allocate all first
def OpenBIPImage(file, width, height, numberOfBands):
"""
Opens a raw image file in the BIP format and returns a list
comprising each band as a separate PIL image.
"""
bandArrays = []
with open(file, 'rb') as imageFile:
data = imageFile.read()
currentPosition = 0
for j in range(numberOfBands):
bandArrays[j]= bytearray(b"\0"*(height * width)):
for i in xrange(height * width):
for j in xrange(numberOfBands):
bandArrays[j][i]=data[currentPosition])
currentPosition += 1
bands = [Image.frombytes('L', (width, height), bytes(bandArray)) for bandArray in bandArrays]
return bands
my measurements doesn't show nsuch a slow down
def x():
height,width,numberOfBands=1401,801,6
before = time.time()
for i in range(height * width):
for j in range(numberOfBands):
pass
print (time.time()-before)
>>> x()
0.937999963760376
EDITED