Overlay a heatmap on an image from streaming FHD video in Python - python

I would like to make a QT based GUI program which overlays a heatmap on an image from 20fps streaming FHD video.
The target image looks like this
(Additionally, a colorbar beside an overlayed image shall also be displayed.)
The size of heatmap source for each image is 100x40, and therefore interpolation for FHD(1920x1080) is needed per frame.
(FYI, The min and max values of heatmap source are around 10 and 100000, respectively.)
First of all, I used cv2.VideoCapture function in Opencv to get images from video. And then, I googled some examples to combine image and heatmap using Matplotlib such as:
https://github.com/durandtibo/heatmap
Heatmap on top of image
Overlay an image segmentation with numpy and matplotlib
The problem that I faced is processing speed to meet 20fps for FHD resolution.
It seemed that Opencv is more adequete rather than Matplotlib for real-time processing.
(I couldn't find the good way to show heatmap and colorbar using Pyqtgraph even though it provides high speed.)
So, I searched another way using cv2.applyColorMap and cv2.resize.
It looks like cv2.applyColorMap function doesn't automatically adjust the range of values unlike imshow function in Matplotlib, and therefore the color of result image is strange.
Moreover, Opencv image needs to be adopted to QtWidget using QtGui.QImage and QtGui.QPixmap which results additional delay.
Finally, the overall processing time of the method I searched can not meet the requirement.
Please show me the way for the solution.
Thanks in advance.

Related

Issue while plotting single channeled images (e.g. grayscale) using imshow

While plotting single channel image (i.e. while plotting grayscale images) when using Python it does not plot in gray-scale.
Example: expected output, after converting a coloured image using COLOR_BGR2GRAY from open cv :
But, the output obtained is:
Can anyone help me find out, what is the exact issue?
Upon researching, I found out that, the issue is actually not with open cv, but it is with matplotlib package. While displaying the image, the matplotlib package uses a colormap and hence it has to be explicitly set to gray, using :
plt.imshow(image, cmap="gray")

How to save grayscale image in Python?

I am trying to save a grayscale image using matplotlib savefig(). I find that the png file which is saved after the use of matplotlib savefig() is a bit different from the output image which is showed when the code runs. The output image which is generated when the code is running contains more details than the saved figure.
How can I save the output plot in such a manner that all details are stored in the output image?
My my code is given below:
import cv2
import matplotlib.pyplot as plt
plt.figure(1)
img_DR = cv2.imread(‘image.tif',0)
edges_DR = cv2.Canny(img_DR,20,40)
plt.imshow(edges_DR,cmap = 'gray')
plt.savefig('DR.png')
plt.show()
The input file (‘image.tif’) can be found from here.
Following is the output image which is generated when the code is running:
Below is the saved image:
Although the two aforementioned images denote the same picture, one can notice that they are slightly different. A keen look at the circular periphery of the two images shows that they are different.
Save the actual image to file, not the figure. The DPI between the figure and the actual created image from your processing will be different. Since you're using OpenCV, use cv2.imwrite. In your case:
cv2.imwrite('DR.png', edges_DR)
Use the PNG format as JPEG is lossy and would thus give you a reduction in quality to promote small file sizes. If accuracy is the key here, use a lossless compression standard and PNG is one example.
If you are somehow opposed to using OpenCV, Matplotlib has an equivalent image writing method called imsave which has the same syntax as cv2.imwrite:
plt.imsave('DR.png', edges_DR, cmap='gray')
Note that I am enforcing the colour map to be grayscale for imsave as it is not automatically inferred like how OpenCV writes images to file.
Since you are using cv2 to load the image, why not using it also to save it.
I think the command you are looking for is :
cv2.imwrite('gray.jpg', gray_image)
Using a DPI that matches the image size seems to make a difference.
The image is of size width=2240 and height=1488 (img_DR.shape). Using fig.get_size_inches() I see that the image size in inches is array([7.24, 5.34]). So an appropriate dpi is about 310 since 2240/7.24=309.4 and 1488/5.34=278.65.
Now I do plt.savefig('DR.png', dpi=310) and get
One experiment to do would be to choose a high enough DPI, calculate height and width of figure in inches, for example width_inch = width_pixel/DPI and set figure size using plt.figure(figsize=(width_inch, height_inch)), and see if the displayed image itself would increase/decrease in quality.
Hope this helps.

How to generate bounding box data from heatmap data of an image?

I have a group of images and some separate heatmap data which (imperfectly) explains where subject of the image is. The heatmap data is in a numpy array with shape (224,224,3). I would like to generate bounding box data from this heatmap data.
The heatmaps are not always perfect, So I guess I'm wondering if anyone can think of an intelligent way to do this.
Here are some examples of what happens when I apply the heatmap data to the image:
I found a solution to this in matlab, but I have no idea how to read this code! I am a python programmer, unfortunately.
https://github.com/metalbubble/CAM/tree/master/bboxgenerator
Anyone have any ideas about how to approach something like this?
I am not quite sure how the heatmap data of your project exactly looks like, but it seems to me that you can use something like Selective Search. You can also have a look on this interesting paper. Maybe you can use this approach on your dataset.
I'm attempting a similar method for automating the creation of bounding boxes (since, lets face it: creating boxes manually takes along time)
this other stackpost covers a similar idea:
EDIT: (i originally had put a link to the current stack post 🤦 - but here is the stack post i was referring to)
Generating bounding boxes from heatmap data
the problem at hand that i recognize is that heatmaps can be fragmented and a bit arbitrary. the solution that comes to mind initially is setting a threshold of the heat map. So in the case of the example heat map images - when applying a bounding box cover all regions that are yellow/orange/red than say green/blue.
It depends on how many bounding boxes you need. You can set a threshold and have multiple bounding boxes for each of the highly activated regions, or try connecting the regions (by a morphological operation maybe) and calculate a single bounding box for connected activated pixels.

Heatmap on top of gray-scale images in python

Simple and straightforward question, I need to plot a heatmap on top of a gray scale image in python, like the one below.
I come from MATLAB. There I would simply generate the probability map, pass it though a colormap and plug the result in the gray scalar picture.
I could do the same in python, though I feel like is not the most "pythonic" way to proceed (with numpy). You know any particular package that handles this kind of visualization?

Custom shape cutouts of a video using Python

I wish to use Python to show a cutout in a certain shape of one video overlaid on top of another video. The invisible parts of the overlaid video should be translucent, so the 'background' video can be seen in those parts. The issue here is that the location of the overlay is dynamic, while the shape remains the same. This means I cannot simply preprocess the videos.
I was thinking to take stills from the overlaid video at runtime, cut out the overlay and superimpose it on the background video in the right location. This would have to be done at a high frequency (30 fps+, probably).
As an example:
I would want the red cutout of this image:
http://i.imgur.com/jEQqvR0.jpg
to appear on top of another image
The Python Image Library (PIL) seems to be able to crop images easily, but only in rectangles and not in a custom shape. I could probably add rectangles together to create a custom shape, but I was hoping there would be an easier way. Maybe I'm overlooking things.
So my question is: What would be the easiest way to do the cut-out? I'm also open to other suggestions for approaches. Ideally, I would use a dynamically positioned translucent video mask partially obscuring the background video with parts of the overlaid video but I'm not sure if this is at all possible.

Categories