We Keep Coding

How do I crop multiple images from one picture based on the background color?

So I have an image and I want to cut it up into multiple images to feed into OCR to read.
image example
I only want the messages with the white bubbles and exclude anything with the grey bubbles. I can't figure out how to make a loop to separate each white bubble.
import numpy as np
from PIL import ImageGrab, Image, ImageFilter
img = Image.open('test1.png').convert('RGB')
na = np.array(img)
orig = na.copy()
img = img.filter(ImageFilter.MedianFilter(3))
whiteY, whiteX = np.where(np.all(na==[255,255,255],axis=2))
top, bottom = whiteY[1], whiteY[-1]
left, right = whiteX[1], whiteX[-1]

You could try using the opencv threshold function, followed by the findContours function. This will, if you threshold the image correctly, give you the 'borders' of the bubbles above. Using that, you could then crop out each text bubble.
Here's a simple example of contours being used:
https://www.geeksforgeeks.org/find-and-draw-contours-using-opencv-python/
Otherwise if you'd like to understand better how the opencv functions I mentioned or those that are used in the article above, have a look at the opencv documentation.

How can I display an image to me (as an user) and select a section of that image?

I will explain myself better, the point is that I want to develop a code that displays an image to me, then with the mouse as the image is displayed I can select or crop it as I want.
So, for example, as this code does, it would select any section of the image:
from PIL import Image
im = Image.open("test.jpg")
crop_rectangle = (50, 50, 200, 200)
cropped_im = im.crop(crop_rectangle)
cropped_im.show()
That is basically it, all I want is to crop an image at the points or coordinates that I please, I have been searching but can not find any library that helps me to do so.
NOTE: The code I am showing is an answer on this post, in case you want to check it out: How can i select a part of a image using python?
EDIT: Here is something I foud that may help me in a first instance, but is not entirely what I am looking for, it at least lets me find the coordinates that I want from the image - with some modifications on the code - and then I will keep processing the image. By now it is not solving my issue but it is a beginning. --> Using "cv2.setMouseCallback" method

The comments I received from Mark Setchell and bfris enlightened me, one with a C++ code and another one with a recommendation of OpenCV.
But in addition to their comments, I found this article where they explain exactly what I want to do, so I consider my question answered. Select ROI or Multiple ROIs [Bounding box] in OPENCV python.
import cv2
import numpy as np
#image_path
img_path="image.jpeg"
#read image
img_raw = cv2.imread(img_path)
#select ROI function
roi = cv2.selectROI(img_raw)
#print rectangle points of selected roi
print(roi)
#Crop selected roi from raw image
roi_cropped = img_raw[int(roi[1]):int(roi[1]+roi[3]), int(roi[0]):int(roi[0]+roi[2])]
#show cropped image
cv2.imshow("ROI", roi_cropped)
cv2.imwrite("crop.jpeg",roi_cropped)
#hold window
cv2.waitKey(0)
or
import cv2
import numpy as np
#image_path
img_path="image.jpeg"
#read image
img_raw = cv2.imread(img_path)
#select ROIs function
ROIs = cv2.selectROIs("Select Rois",img_raw)
#print rectangle points of selected roi
print(ROIs)
#Crop selected roi ffrom raw image
#counter to save image with different name
crop_number=0
#loop over every bounding box save in array "ROIs"
for rect in ROIs:
x1=rect[0]
y1=rect[1]
x2=rect[2]
y2=rect[3]
#crop roi from original image
img_crop=img_raw[y1:y1+y2,x1:x1+x2]
#show cropped image
cv2.imshow("crop"+str(crop_number),img_crop)
#save cropped image
cv2.imwrite("crop"+str(crop_number)+".jpeg",img_crop)
crop_number+=1
#hold window
cv2.waitKey(0)

How to slow down image display in Pillow

Probably an unusual question, but I am currently looking for a solution to display image files with PIL slower.
Ideally so that you can see how the image builds up, pixel by pixel from left to right.
Does anyone have an idea how to implement something like this?
It is a purely optical thing, so it is not essential.
Here an example:
from PIL import Image
im = Image.open("sample-image.png")
im.show()
Is there a way to "slow down" im.show()?

AFAIK, you cannot do this directly with PIL's Image.show() because it actually saves your image as a file to /var/tmp/XXX and then passes that file to your OS's standard image viewer to display on the screen and there is no further interaction with the viewer process after that. So, if you draw in another pixel, the viewer will not be aware and if you call Image.show() again, it will save a new copy of your image and invoke another viewer which will give you a second window rather than updating the first!
There are several possibilities to get around it:
use OpenCV's cv2.imshow() which does allow updates
use tkinter to display the changing image
create an animated GIF and start a new process to display that
I chose the first, using OpenCV, as the path of least resistance:
#!/usr/bin/env python3
import cv2
import numpy as np
from PIL import Image
# Open image
im = Image.open('paddington.png')
# Make BGR Numpy version for OpenCV
BGR = np.array(im)[:,:,::-1]
h, w = BGR.shape[:2]
# Make empty image to fill in slowly and display
d = np.zeros_like(BGR)
# Use "x" to avoid drawing and waiting for every single pixel
x=0
for y in range(h):
for x in range(w):
d[y,x] = BGR[y,x]
if x%400==0:
cv2.imshow("SlowLoader",d)
cv2.waitKey(1)
x += 1
# Wait for one final keypress to exit
cv2.waitKey(0)
Increase the 400 near the end to make it faster and update the screen after a greater number of pixels, or decrease it to make it update the screen after a smaller number of pixels meaning you will see them appear more slowly.
As I cannot share a movie on StackOverflow, I made an animated GIF to show how that looks:
I decided to try and do it with tkinter as well. I am no expert on tkinter but the following works just the same as the code above. If anyone knows tkinter better, please feel free to point out my inadequacies - I am happy to learn! Thank you.
#!/usr/bin/env python3
import numpy as np
from tkinter import *
from PIL import Image, ImageTk
# Create Tkinter Window and Label
root = Tk()
video = Label(root)
video.pack()
# Open image
im = Image.open('paddington.png')
# Make Numpy version for simpler pixel access
RGB = np.array(im)
h, w = RGB.shape[:2]
# Make empty image to fill in slowly and display
d = np.zeros_like(RGB)
# Use "x" to avoid drawing and waiting for every single pixel
x=0
for y in range(h):
for x in range(w):
d[y,x] = RGB[y,x]
if x%400==0:
# Convert the video for Tkinter
img = Image.fromarray(d)
imgtk = ImageTk.PhotoImage(image=img)
# Set the image on the label
video.config(image=imgtk)
# Update the window
root.update()
x += 1

Display Animations in Google Colab

I'm trying to display animations in Google Colab. Specifically, I would like to animate a numpy array with cv2, eg drawing lines in a frame-based manner, and show the output in the cell. The closest I got was this, you can try this code in Colab:
from google.colab.patches import cv2_imshow
import IPython
from PIL import Image
import numpy as np
import cv2 as cv
import time
# Create a black image
img = np.zeros((512,512,3), np.uint8)
# Draw a diagonal blue line with thickness of 5 px
cv.line(img,(0,0),(511,511),(255,0,0),5)
cv2_imshow(img)
for i in range(100):
cv.line(img,(i,0),(511,511),(255,0,0),5)
cv2_imshow(img)
IPython.display.clear_output(wait=True)
time.sleep(1/60.0)
At some point of course this should happen without time.sleep, but with repeated callbacks so we don't block any other code execution. However, as you can see, the output flickers and is not smooth at all.
Here are a couple things I've tried:
ipycanvas. This is great in a local Jupyter notebook and is based on HTML5 canvas. It is a bit annoying to get the image data from javascript back to python, but it's possible. However, this does not run in Google Colab.
https://ipycanvas.readthedocs.io/
Matplotlib animations. eg this (not mine):
https://colab.research.google.com/drive/1lnl5UPFWVPrryaZZgEzd0theI6S94c3X#scrollTo=QLRBwgFqdr83
This is alright. However, it renders the whole animation before displaying it, which is not what I want. Especially, I want to be able to add some interactivity to animations, which this limitation rules out (eg clicking in the image or some button to make something happen in the animation).
Some way of explicitly creating an HTML5 canvas in javascript, eg as suggested here:
IPython: Adding Javascript scripts to IPython notebook
However, I'd like all my code to be python, especially my data to be numpy arrays or PIL images.
Any suggestions?

Here's an example using ipywidgets.Image. This approach doesn't flicker like using clear_output, but the updates seem pretty slow. This might be to do with the fact we're running remotely from Colab - it has to send image updates over the net. Looks like I'm getting 2 or 3 per second, and it seems like it "batches up" or discards intervening updates, rather than waiting for each one.
It's pretty smooth running locally on regular Jupyter.
Hope someone can improve on this - it's something we want to do as well :)
import ipywidgets as ipw
from IPython import display
import numpy as np
import PIL
from io import BytesIO
import time
# image size
h,w = 200,300
# Make an Image Widget and display it
wIm = ipw.Image()
display.display(wIm)
# Make an RGBA array for the image
g3 = np.zeros((h,w,4), dtype=np.uint8)
g3[:,:,3] = 255 # opacity
g3[:,:,0:3] = 0 # color black
p = np.array([h//2,w//2], dtype=int)
for i in range(1000):
# Draw a coloured spiral
r = i/10
theta=i/20
p2 = p + r * np.array([ np.cos(theta), np.sin(theta) ])
(y,x) = p2.astype(int)
rgb = np.array([100+r, 100*(1+np.sin(theta)), 100*(1+np.cos(theta))], dtype=np.uint8)
g3[y:y+8, x:x+2, 0:3] = rgb
# convert numpy to PIL to png-format bytes
pilIm = PIL.Image.fromarray(g3, mode="RGBA")
with BytesIO() as fOut:
pilIm.save(fOut, format="png")
byPng = fOut.getvalue()
# set the png bytes as the image value;
# this updates the image in the browser.
wIm.value=byPng
time.sleep(1/60)

Use Python / PIL or similar to shrink whitespace

Any ideas how to use Python with the PIL module to shrink select all? I know this can be achieved with Gimp. I'm trying to package my app as small as possible, a GIMP install is not an option for the EU.
Say you have 2 images, one is 400x500, other is 200x100. They both are white with a 100x100 textblock somewhere within each image's boundaries. What I'm trying to do is automatically strip the whitespace around that text, load that 100x100 image textblock into a variable for further text extraction.
It's obviously not this simple, so just running the text extraction on the whole image won't work! I just wanted to query about the basic process. There is not much available on Google about this topic. If solved, perhaps it could help someone else as well...
Thanks for reading!

If you put the image into a numpy array, it's simple to find the edges which you can use PIL to crop. Here I'm assuming that the whitespace is the color (255,255,255), you can adjust to your needs:
from PIL import Image
import numpy as np
im = Image.open("test.png")
pix = np.asarray(im)
pix = pix[:,:,0:3] # Drop the alpha channel
idx = np.where(pix-255)[0:2] # Drop the color when finding edges
box = map(min,idx)[::-1] + map(max,idx)[::-1]
region = im.crop(box)
region_pix = np.asarray(region)
To show what the results look like, I've left the axis labels on so you can see the size of the box region:
from pylab import *
subplot(121)
imshow(pix)
subplot(122)
imshow(region_pix)
show()

The general algorithmn would be to find the color of the top left pixel, and then do a spiral scan inwards until you find a pixel not of that color. That will define one edge of your bounding box. Keep scanning until you hit one more of each edge.

http://blog.damiles.com/2008/11/basic-ocr-in-opencv/
might be of some help. You can use the simple bounding box method described in that tutorial or #Tyler Eaves spiral suggestion which works equally as well