We Keep Coding

How do I recognise an image on webpage using Python?

I've created a simple Python application that uses the CV2 computer vision library to recognise a template image on a webpage.
I give the application a template image that it needs to recognise on the source image. In this case, the source image is a screenshot of the website www.google.com and the template image is the Google search button.
Template image
I thought the application worked at first, but it's drawing the rectangle completely in the wrong place on the input (source) image. I've added a picture below of where the application located the template image.
Result
Here's the source code.
Main Application Source
import cv2
import numpy
from io import BytesIO
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np
class Automate:
def __init__(self):
chrome_options = Options()
chrome_options.add_argument("kiosk")
self.driver = webdriver.Chrome(ChromeDriverManager("93.0.4577.63").install(), options=chrome_options)
#self.driver = webdriver.Chrome(executable_path='./chromedriver',options=chrome_options)
self.screenShot = None
self.finalImage = None
def open_webpage(self, url):
print(f"Open webpage {url}")
self.driver.get(url)
def close_webpage(self):
Event().wait(5)
self.driver.close()
print("Closing webpage")
def snap_screen(self):
print("Capturing screen")
self.screenShot = "screenshot.png"
self.driver.save_screenshot(self.screenShot)
print("done.")
def match(self, image, template):
# convert images to greyscale.
src = cv2.cvtColor(cv2.imread(image), cv2.COLOR_BGR2GRAY)
temp = cv2.cvtColor(cv2.imread(template), cv2.COLOR_BGR2GRAY)
cv2.imshow("out", temp)
cv2.waitKey(0)
height, width = src.shape
H, W = temp.shape
result = cv2.matchTemplate(src, temp, cv2.cv2.TM_CCOEFF_NORMED)
minVal, maxVal, minLoc, maxLoc = cv2.minMaxLoc(result)
location = maxLoc
bottomRight = (location[0] + W, location[1] + H)
src2 = cv2.imread(image)
cv2.rectangle(src2, location, bottomRight, (0, 0, 255), 5)
cv2.imshow("output", src2)
cv2.waitKey(0)
cv2.destroyAllWindows()
def main():
url = "http://www.google.com"
auto = Automate()
auto.open_webpage(url)
auto.snap_screen()
auto.close_webpage()
match_image = "images/templates/google-button.png"
# Match screenshot with template image.
auto.check_match(
image=auto.screenShot,
template=match_image
)
I'd appreciate any help or advice on how to solve this issue.
Update
Following the advice given by user zteffi, I resized my template image to the correct image dimensions. After doing this, the match template function works as expected.
You want to make sure that your template image is a close as possible to the actual size of the image you want to be located in the base image. In my case, this was around 150 x 150 or 200 x 200 so that it will be easier to find the button.

Pillow on Mac Book Pro retina displays

I am currently trying to take a screenshot of a specific web element using Selenium and Pillow on Python 2.7. The code I have seems to be able to take the screenshot and then crop the image but it is actually not cropping at the exact location of the element. After researching I found that this could be happening due to the dpi used by the Mac Book Pro retina display but I haven´t found a workaround for this.
What could be the best way to handle this "issue" with the retina display and PIL?
This is the code I am using:
try:
#Wait until finishing loading
waitElement = WebDriverWait(driver,60).until(
EC.presence_of_element_located((By.XPATH, "//*[#id='someElement']/span/input[2]"))
)
try:
#Go to trends report
driver.get("https://somewebsite.com/trends")
waitElement = WebDriverWait(driver,60).until(
EC.presence_of_element_located((By.ID, "Totals"))
)
try:
summary = driver.find_element_by_id("Totals")
driver.save_screenshot("sc.png")
location = summary.location
size = summary.size
x = location['x']
y = location['y']
w = size['width']
h = size['height']
width = x + w
height = y + h
im = Image.open('sc.png')
im = im.crop((int(x), int(y), int(width), int(height)))
im.save('summary.png')
except NoSuchElementException:
print("Something")
except NoSuchElementException:
print("Something")
finally:
driver.close()

I was facing this (broken screenshot issue in case of ratina display on mac) & solved it using
Snippet I used for this
//basically , here I am using 'devicePixelRatio' to calculate proper width/height
WebElement ele = findElement(By.xpath(webElementXpath));
Point point = ele.getLocation();
int devicePixelRatio = Integer.parseInt(returnExecuteJavaScript("return window.devicePixelRatio"));
int eleWidth = ele.getSize().getWidth() * devicePixelRatio;
int eleHeight = ele.getSize().getHeight() * devicePixelRatio;
// ....... for cropping stuff , you can use your own code

Hide phantomjs console

I am doing a project and I want him to be invisible. Therefore, i used this website - http://pytoexe.com/ to create an window based exe file from my python script which means it will not use the windows console.
Unfortuently, since i am using phantomjs driver in my code, he opens a phantomjs console which interrupt me.
In order to slove this problem I need to add a line or script that prevent from the phantomjs console to appear ( changing something in my selenium files/ something like that would not work cause it probably problem in their files and i cannot doing anything with that) .
Someone know what to do?
this is my exe file
and This is my code:
from selenium import webdriver
import time
from PIL import Image
from constants import *
import operator
import os
#Email Constants
DEFAULT_CONTENT = 'example email stuff here'
HOST = 'smtp.gmail.com'
PORT = 587
EMAIL = 'freeadsandspam#gmail.com'
CUSTOMERS = []
SUBJECTS = ['new ad were found', 'ad were found by SMADS', 'best ad for you']
COMMASPACE = ', '
#Getting History
OUTPUT_FILE_PATH = 'C:\search_logger.txt'
COPY_OF_THE_HISTORY_PATH = 'C:\history'
NEW_OUTPUT_FILE_PATH = 'C:\last_search_logger.txt'
#PhantomJs And Ads Finding
PHANTOM_JS_PATH = 'C:\phantomjs-2.1.1-windows\\bin\phantomjs.exe'
OUTPUT_AD_PATH = 'ad.png'
DEFAULT_WINDOW_SIZE = (1024, 768)
AD_DATABASE = 'https://www.findads.com.au/'
KEYWORD_BUTTON_XPATH = '//*[#id="txtSearch"]'
SEARCH_BUTTON_XPATH = '/html/body/div[1]/div/form/button'
CONSTANT_TIME_SLEEPING = 3
AD_XPATH = '/html/body/div[1]/section/div/div[1]/div[4]/div[1]/div[1]/section[24]'
COMPARE_ELEMENT_XPATH = '//*[#id="fSearch"]'
CATAGORY_SORT_XPATH = '/html/body/div[1]/section/div/div[1]/div[5]/div/div[3]/form/div[1]/div[1]'
class PhantomJsDriver:
"""
A class that taking care on the ads finding
in the internet, doing it with PhantomJs -
background driver
"""
def __init__(self, ad_keyword, window_size=DEFAULT_WINDOW_SIZE, panthom_js_path=PHANTOM_JS_PATH, output_ad_path=OUTPUT_AD_PATH):
"""
this function init our object
in order to use the other functions
that the object offer.
Parameters
----------
phantom_js_path : str
path of the PhantomJs ( this essential because
we cannot get the PhantomJs file otherwise)
output_ad_path : str
where you want to save the ad that the system
had found and how you call the name of the ad
file ( eg: ad.png )
ad_keyword : str
the keyword that define what ad the system bring
( eg: dog will bring dog ad )
window_size : double int (int1,int2)
define the window size of the browser ( mainly for the
screenshot )
"""
self.phantom_js_path = panthom_js_path
self.output_ad_path = output_ad_path
self.ad_keyword = ad_keyword
self.window_size = window_size
self.list_of_images = []
self.dict = {}
def get_ad(self):
"""
this function save the ad by searching in the internet
( on specific website ) the keyword that the user chose
and copy it into the output_ad_path.
"""
for i in range(0, 5):
driver = webdriver.PhantomJS(self.phantom_js_path)
driver.set_window_size(self.window_size[0], self.window_size[1])
driver.get(AD_DATABASE)
keyword = driver.find_element_by_xpath(KEYWORD_BUTTON_XPATH)
keyword.send_keys(self.ad_keyword)
search_button = driver.find_element_by_xpath(SEARCH_BUTTON_XPATH)
search_button.click()
driver.save_screenshot("ad" + str(i) + ".png")
element = driver.find_element_by_xpath(AD_XPATH) # find part of the page you want image of
self.crop_image(i, element)
def crop_image(self, i, element):
"""
this function crop the screenshot of the ads website from
the previous function into one single ad.
"""
im = Image.open("ad" + str(i) + ".png") # uses PIL library to open image in memory
location = element.location
size = element.size
left = location['x']
top = location['y']
right = location['x'] + size['width'] + 50
bottom = location['y'] + size['height']
weight, height = im.size
print height
im = im.crop((left, top, right, bottom)) # defines crop points
im.save('test' + str(i) + '.png') # saves new cropped image
self.list_of_images.append('test' + str(i) + '.png')
self.dict['test' + str(i) + '.png'] = 0
def choose_the_best_ad(self):
for img1 in self.list_of_images:
for img2 in self.list_of_images:
im1 = Image.open(img1)
im2 = Image.open(img2)
if list(im1.getdata()) == list(im2.getdata()):
self.dict[img1] += 1
self.dict[img2] += 1
print self.dict
BestImage = max(self.dict.iteritems(), key=operator.itemgetter(1))[0]
print BestImage
if os.path.exists("TheImage.png"):
os.remove("TheImage.png")
os.rename(BestImage, "TheImage.png")
driver = PhantomJsDriver("dog")
driver.get_ad()
driver.choose_the_best_ad()

NAO fails to save captured image to local computer

I'm trying to save a captured 640x480 RGB image with NAO's front camera to my computer. I'm using python and PIL to do so. Unfortunately, the image just won't save on my computer, no matter what image type or path I use for the parameters of the Image.save()- Method. the image created with PIL contains valid RGB-information though. Here's my code sample from choregraphe:
import Image
def onInput_onStart(self):
cam_input = ALProxy("ALVideoDevice")
nameId = cam_input.subscribeCamera("Test_Cam", 1, 2, 13, 20)
image = cam_input.getImageRemote(nameId) #captures an image
w = image[0] #get the image width
h = image[1] #get the image height
pixel_array = image[6] #contains the image data
result = Image.fromstring("RGB", (w, h), pixel_array)
#the following line doesnt work
result.save("C:\Users\Claudia\Desktop\NAO\Bilder\test.png", "PNG")
cam_input.releaseImage(nameId)
cam_input.unsubscribe(nameId)
pass
Thank you so much for your help in advance!
- a frustrated student

In the comment, you say the code is pasted from choregraphe, so I guess you launch it using choregraphe.
If so, then the code is injected into your robot then started.
So your image is saved to the NAO hard drive and I guess your robot doesn't have a folder named: "C:\Users\Claudia\Desktop\NAO\Bilder\test.png".
So change the path to "/home/nao/test.png", start your code, then log into your NAO using putty or browse folder using winscp (as it looks like you're using windows).
And you should see your image-file.

In order for your code to run correctly it needs to be properly indented. Your code should look like this:
import Image
def onInput_onStart(self):
cam_input = ALProxy("ALVideoDevice")
nameId = cam_input.subscribeCamera("Test_Cam", 1, 2, 13, 20)
image = cam_input.getImageRemote(nameId) #captures an image
w = image[0] #get the image width
h = image[1] #get the image height
pixel_array = image[6] #contains the image data
...
Make sure to indent everything that's inside the def onInput_onStart(self): method.

Sorry for the late response, but it maybe helpful for someone. You should try it with naoqi. Here is the documentation for retriving images
http://doc.aldebaran.com/2-4/dev/python/examples/vision/get_image.html
The original code was not working for me so I made some tweeks.
parser = argparse.ArgumentParser()
parser.add_argument("--ip", type=str, default="nao.local.",
help="Robot IP address. On robot or Local Naoqi: use
'nao.local.'.")
parser.add_argument("--port", type=int, default=9559,
help="Naoqi port number")
args = parser.parse_args()
session = qi.Session()
try:
session.connect("tcp://" + args.ip + ":" + str(args.port))
except RuntimeError:
pass
"""
First get an image, then show it on the screen with PIL.
"""
# Get the service ALVideoDevice.
video_service = session.service("ALVideoDevice")
resolution = 2 # VGA
colorSpace = 11 # RGB
videoClient = video_service.subscribe("python_client",0,3,13,1)
t0 = time.time()
# Get a camera image.
# image[6] contains the image data passed as an array of ASCII chars.
naoImage = video_service.getImageRemote(videoClient)
t1 = time.time()
# Time the image transfer.
print ("acquisition delay ", t1 - t0)
#video_service.unsubscribe(videoClient)
# Now we work with the image returned and save it as a PNG using ImageDraw
# package.
# Get the image size and pixel array.
imageWidth = naoImage[0]
imageHeight = naoImage[1]
array = naoImage[6]
image_string = str(bytearray(array))
# Create a PIL Image from our pixel array.
im = Image.fromstring("RGB", (imageWidth, imageHeight), image_string)
# Save the image.
im.save("C:\\Users\\Lenovo\\Desktop\\PROJEKTI\\python2-
connect4\\camImage.png", "PNG")
Be careful to use Python 2.7.
The code runs on your computer not the NAO robot!

python - login to webpage and save as png

I have a website that requires authentication to access that I then want to make an image of. I'm using the following script -
import os
import requests
from subprocess import Popen, PIPE
from selenium import webdriver
abspath = lambda *p: os.path.abspath(os.path.join(*p))
ROOT = abspath(os.path.dirname(__file__))
def execute_command(command):
result = Popen(command, shell=True, stdout=PIPE).stdout.read()
if len(result) > 0 and not result.isspace():
raise Exception(result)
def do_screen_capturing(url, screen_path, width, height):
print "Capturing screen.."
driver = webdriver.PhantomJS()
# it save service log file in same directory
# if you want to have log file stored else where
# initialize the webdriver.PhantomJS() as
# driver = webdriver.PhantomJS(service_log_path='/var/log/phantomjs/ghostdriver.log')
driver.set_script_timeout(30)
if width and height:
driver.set_window_size(width, height)
driver.get(url)
driver.save_screenshot(screen_path)
def do_crop(params):
print "Croping captured image.."
command = [
'convert',
params['screen_path'],
'-crop', '%sx%s+0+0' % (params['width'], params['height']),
params['crop_path']
]
execute_command(' '.join(command))
def do_thumbnail(params):
print "Generating thumbnail from croped captured image.."
command = [
'convert',
params['crop_path'],
'-filter', 'Lanczos',
'-thumbnail', '%sx%s' % (params['width'], params['height']),
params['thumbnail_path']
]
execute_command(' '.join(command))
def get_screen_shot(**kwargs):
url = kwargs['url']
width = int(kwargs.get('width', 1024)) # screen width to capture
height = int(kwargs.get('height', 768)) # screen height to capture
filename = kwargs.get('filename', 'screen.png') # file name e.g. screen.png
path = kwargs.get('path', ROOT) # directory path to store screen
crop = kwargs.get('crop', False) # crop the captured screen
crop_width = int(kwargs.get('crop_width', width)) # the width of crop screen
crop_height = int(kwargs.get('crop_height', height)) # the height of crop screen
crop_replace = kwargs.get('crop_replace', False) # does crop image replace original screen capture?
thumbnail = kwargs.get('thumbnail', False) # generate thumbnail from screen, requires crop=True
thumbnail_width = int(kwargs.get('thumbnail_width', width)) # the width of thumbnail
thumbnail_height = int(kwargs.get('thumbnail_height', height)) # the height of thumbnail
thumbnail_replace = kwargs.get('thumbnail_replace', False) # does thumbnail image replace crop image?
screen_path = abspath(path, filename)
crop_path = thumbnail_path = screen_path
if thumbnail and not crop:
raise Exception, 'Thumnail generation requires crop image, set crop=True'
do_screen_capturing(url, screen_path, width, height)
if crop:
if not crop_replace:
crop_path = abspath(path, 'crop_'+filename)
params = {
'width': crop_width, 'height': crop_height,
'crop_path': crop_path, 'screen_path': screen_path}
do_crop(params)
if thumbnail:
if not thumbnail_replace:
thumbnail_path = abspath(path, 'thumbnail_'+filename)
params = {
'width': thumbnail_width, 'height': thumbnail_height,
'thumbnail_path': thumbnail_path, 'crop_path': crop_path}
do_thumbnail(params)
return screen_path, crop_path, thumbnail_path
if __name__ == '__main__':
'''
Requirements:
Install NodeJS
Using Node's package manager install phantomjs: npm -g install phantomjs
install selenium (in your virtualenv, if you are using that)
install imageMagick
add phantomjs to system path (on windows)
'''
s = requests.Session()
s.auth = ('username', 'password')
r = s.get('https://website.com:8443/path/to/site', verify=False)
url = r.text
screen_path, crop_path, thumbnail_path = get_screen_shot(
url=url, filename='test.png',
crop=True, crop_replace=False,
thumbnail=True, thumbnail_replace=False,
thumbnail_width=200, thumbnail_height=150,
)
I know it's authenticating because I can do print r.status and get 200 and r.headers returns the header. r.text gives encoding errors however. The above code is not failing but returning a blank image.
This is on a Windows machine.
edit - if I remove the requests to just hit the URL without logging in -
url = 'https://website.com:8443/path/to/site'
screen_path, crop_path, thumbnail_path = get_screen_shot(
url=url, filename='test.png',
crop=True, crop_replace=False,
thumbnail=True, thumbnail_replace=False,
thumbnail_width=200, thumbnail_height=150,
)
It takes a screenshot of the login page. What I want to be able to do is get a screenshot of the page after logging in.

It looks like save_screenshot() is called when the page is not completely loaded.
In this case, you need to explicitly wait for a, for example, login form to become visible. Example:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get(url)
WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.ID, "myForm"))
)
driver.save_screenshot(screen_path)