Searching images within region/area and reacting based on them - python

Total beginner here.
In this program, I am trying to "monitor" a specific area on my screen. There are several images entering the region (from the right side) with random frequency. (Basically, a "simple" rhythm" game).
I have tried pyautoguy.locateOnScreen, but it was way too slow even though I set the region into a small square. I don't think I can use pixels because the images entering the region are all the same color but have different symbols on them, and each relates to a keyboard letter.
In this last attempt, I tried this:
Create an "endless" while loop, in which I make a screenshot of my desired region/location(bbox), and within that endless loop, I use IF and several ELIF in which I use pyautogui.locate to search a specific image (tiny screenshot of the symbol) on the bigger screenshot made by ImageGrab. I have several images, and each image has a different keyboard response.
I got to about 75-80% accuracy, but when the symbols are close to each other, it always misses. When I checked at which rate the whole loop runs, I get to like 2.9 runs per second, sometimes 4-5. Could anyone give me some pointers on how to double the speed (if that's even possible?)
Thanks!
import time
import pyautogui
import os
import pydirectinput
from PIL import ImageGrab, Image
import time
start_time = time.time()
x = 1 # frames
counter = 0
bbox=(588,786,635,944) #my region I am searching in
def detect(imgDir, haystackImage, confidence=0.90, cache=True):
if (pyautogui.locate(os.path.join('images', imgDir), haystackImage, confidence=confidence)) is not None:
return True
else:
return False
while True:
ScreenGrab = ImageGrab.grab(bbox)
ScreenName = f"screenshot"
Path = f"./images/{ScreenName}.png"
ScreenGrab.save(Path)
hsImage = Image.open('D:/PY/images/screenshot.png')
if detect('D:/PY/images/a.png', hsImage, cache=True):
pydirectinput.press('a')
continue
elif detect('D:/PY/images/b.png', hsImage, cache=True):
pydirectinput.press('b')
continue
elif detect('D:/PY/images/c.png', hsImage, cache=True):
pydirectinput.press('c')
continue
elif detect('D:/PY/images/d.png', hsImage, cache=True):
pydirectinput.press('d')
continue
elif detect('D:/PY/images/e.png', hsImage, cache=True):
pydirectinput.press('e')
continue
else:
pass
counter+=1 #counter for my runs per second
if (time.time() - start_time) > x :
print("FPS: ", counter / (time.time() - start_time))
counter = 0
start_time = time.time()

Related

How to know the status of monitor is on or off in python?

How to know the status of monitor is on or off in python?
I would like to receive a beep notification every 50 minutes of every hour like this:
import datetime
import time
import winsound
frequency = 2500 # Set Frequency To 2500 Hertz
duration = 200 # Set Duration To 1000 ms == 1 second
while True:
# If the monitor is Black by sleep mode
if ???:
time.sleep(60)
continue
now = datetime.datetime.now()
if now.minute == 50:
print(now)
winsound.Beep(frequency, duration)
time.sleep(60)
However, I don't want to receive a beep notification when the monitor is off because the monitor is on when I'm working, and the monitor is off because it's in power saving mode when I'm not.
But I don't know how to tell if the monitor is on or off.
How can I know the status of my monitor?
best regards!
I Solved like this
import time
from datetime import datetime
import win32api
import winsound
frequency = 1500
duration = 100
def getIdleTime():
return (win32api.GetTickCount() - win32api.GetLastInputInfo()) / 1000.0
while True:
now = datetime.now()
if getIdleTime() > 3600:
continue
elif now.minute == 50:
winsound.Beep(frequency, duration)
time.sleep(60)
``
As stated in this old answer there is no reliable way to check the status of a monitor. But i think you don't have to rely on monitor state. Check inactivity time is a better approach.
This question captivated me, in the same vein as the timeless classic "is the fridge light really on if the door is closed"?
In trying to solve this problem, I wondered what a screenshot would appear as if the display was off. So, I tested it out. The idea is that if the screen is black, you can sum the pixels and detect if the value is 0 to determine whether the monitor is on or off.
Unfortunately, it seems like through testing that the screenshot while the monitor is off is not a black screen, at least on my device. You are welcome to test on yours, and I'll post the "solution" anyways if curious.
The solution will probably need to understand the devices connected and the status of them as known by windows, which I would guess could be tricky in python.
import datetime
import time
import winsound
from mss import mss
from PIL import Image
import numpy
def screen_accumulator():
#take screenshot
mss().shot()
#retrieve png
im = Image.open("monitor-1.png")
pix = numpy.array(im)
return(numpy.sum(pix))
frequency = 2500 # Set Frequency To 2500 Hertz
duration = 200 # Set Duration To 1000 ms == 1 second
while True:
# If the monitor is Black by sleep mode
if screen_accumulator()==0:
time.sleep(60)
continue
now = datetime.datetime.now()
if now.minute == 50:
print(now)
winsound.Beep(frequency, duration)
time.sleep(60)

Why is my function triggering multiple times?

This code:
Looks for an image a
If it finds a it tries to find a match for any image in the array
image_list
If it finds an image in image_list it looks for e
If it finds e it logs it and moves the mouse to x, y checks for pixel colour and then clicks when when it finds a match, clicks.
This is where my problem arises my avoidLog() function is being called 2-3 times per click. I've added print("click") to check if it's actually clicking and it isn't, it's clicking at the right time but for some reason, my log is triggering more than once.
It's logging correctly, just an odd number of times.
I know python is operating in a single thread in my example so I'm not sure why it's looping back round and not clicking. It would make more sense to me if it was clicking multiple times and logging multiple times.
import pyautogui as py
def avoidLog():
avoidLog = open('Avoided.txt', 'a')
avoidLog.write("Found at: " + str(f) + " at: " + str(skipTime))
avoidLog.write("\n")
avoidLog.close()
image_list = []
while True:
if py.locateOnScreen('a.jpg') != None:
for image in image_list:
found = py.locateCenterOnScreen(image)
if found != None:
skipTrigger = py.locateOnScreen('e.jpg')
if skipTrigger != None:
avoidLog()
py.moveTo(x, y)
r = py.pixelMatchesColor(x,y, (r,g,b))
if r == True:
py.sleep(2)
print("click")
py.click()
break
avoidLog() is called whenever e.jpg is located. However, py.click() is only called if pixelMatchesColor is True

Optimise Performance - OpenCV and Threads with Python

I am having a problem related to performance using OpenCV3.2 with Python. I have just integrated another subsystem to my main program and it slowed down a lot.
This is my initial code without integrating the new subsystem, I am using cv2.getTickCount to measure time, as suggested by OpenCv3.2 Official Website.
# Infinite loop
while True:
# getting tick count
e1 = cv2.getTickCount()
# storing frame
_, img = cap.read()
# define red colours in the screen
findRedColours(img, board)
# getting tick count after the functions
e2 = cv2.getTickCount()
# calculating time
t = (e2 - e1) / cv2.getTickFrequency()
# print time
print(t)
# check if img is none
if img is not None:
# omitted code
k = cv2.waitKey(20) & 0xFF
# start the game, hide info
if (k == ord('s') or k == ord('S')) and start is False:
# create new thread to play game
t = Thread(target=playGame)
t.start()
Basically, I am calling a function in the infinite loop to find red colours and by pressing start I create the Thread and the game starts.
This is the time needed before I press 'S' to create the Thread:
0.019336862
0.016924178
0.022487864
This is the time needed after I press 'S' to create the Thread:
0.091731532
0.125760734
0.098221829
Here everything works fine, there is a light change in the time, but nothing too important. I start to have problems when I add my new subsystem. Here the following code with the integration of the new system, it is the same of the previous one, it is just a function call that changes:
# Infinite loop
while True:
# getting tick count
e1 = cv2.getTickCount()
# storing frame
_, img = cap.read()
# extract grid
gridExtractor.extractGrid(img)
# define red colours in the screen
findRedColours(img, board)
# getting tick count after the functions
e2 = cv2.getTickCount()
# calculating time
t = (e2 - e1) / cv2.getTickFrequency()
# print time
print(t)
# check if img is none
if img is not None:
# omitted code
k = cv2.waitKey(20) & 0xFF
# start the game, hide info
if (k == ord('s') or k == ord('S')) and start is False:
# create new thread to play game
t = Thread(target=playGame)
t.start()
and this is the time before I create the Thread:
0.045629524
0.023788123
0.10517206
It is slightly higher than the one without the integration, but still ok. Here is the time after I create Thread:
1.061517957
0.568310864
0.691701059
There is an enormous difference between this one and the previous one, it reaches even a whole second. It is noticeable even from the camera output, really slow.
My questions are, am I creating my Thread in the wrong way? There is a better and more efficient way to use rather the Threads? Or is there actually a way to optimise performance in this case without having to modify these functions findRedColours(img, board), t = Thread(target=playGame), gridExtractor.extractGrid(img)?
I am new using OpenCV and Python and still having troubles around. Hope someone can address me to the right way. Thanks.
Thanks to user 'deets' who help commenting above it has been possible to optimise performance.
In this case is enough to substitute Thread with Process from multiprocessing module in Python.
from multiprocessing import Process
#omitted code
while True:
# getting tick count
e1 = cv2.getTickCount()
# storing frame
_, img = cap.read()
# extract grid - first subsystem
gridExtractor.extractGrid(img)
# define red colours in the screen - second subsystem
findRedColours(img, board)
# getting tick count after the functions
e2 = cv2.getTickCount()
# calculating time
t = (e2 - e1) / cv2.getTickFrequency()
# print time
print(t)
# check if img is none
if img is not None:
# omitted code
k = cv2.waitKey(20) & 0xFF
# start the game, hide info
if (k == ord('s') or k == ord('S')) and start is False:
# create new thread to play game
p = Process(target=playGame)
p.start()
And the relative time needed is:
0.022570883
0.11354852
0.119643379
Compared to Thread the use of Process is way more efficient in terms of performance.

Inadvertently Stacking While Loops

I've got a program right now that is meant to display an image, and allow for various functions on keypress (e.g., load a new frame, crop a region of interest, etc). It's also set up so that if too much time passes with no action, it'll automatically attempt to load a new image.
def triggeredbytimer():
global f, x1i, y1i, x2i, y2i, sid, keepgoing
keepgoing = False
print "Inactivity has lead to an automatic refresh. Loading newest image."
f = 0; t = 0
incoming = imagerequest(sid, f, t, x1i, y1i, x2i, y2i)
imagecallback(incoming)
def imagecallback(data):
print "----------------------- imagecallback"
global sid, f, x1i, y1i, x2i, y2i, img, refPt, cropping, keepgoing
keepgoing = True
########### \\// change this to change wait for autorefresh
tmr = Timer(10.0, triggeredbytimer)
tmr.start()
##################################
b = data.b; t = data.tr; sid = data.sid; error = int(data.error)
t = float(t); sid = int(sid)
expandedt = time.strftime("%m-%d-%Y %H:%M:%S", time.localtime(t))
print str(t) + " ----> " + expandedt
print "----------------------- image incoming"
timestr = time.strftime("%H:%M:%S", time.gmtime(t))
print "Loaded image from %s."%timestr
imagedata = bridge.imgmsg_to_cv2(b, "bgr8")
npimg = cv2.imencode('.jpg', imagedata)
cv2.imwrite('temp_image_np.jpg', imagedata)
img = cv2.imread('temp_image_np.jpg')
clone = img.copy()
cv2.namedWindow('image', cv2.WINDOW_NORMAL)
cv2.setMouseCallback("image", mouseclickcb)
cv2.imshow("image", img)
while keepgoing == True:
key = cv2.waitKey(0) & 0xFF
## (here there are a lot of keypress functions, for brevity I included only one)
elif key == ord("t"):
print "Looking for newest frame..."
tmr.cancel()
t = 0; f = 0
incoming = imagerequest(sid, f, t, x1i, y1i, x2i, y2i)
imagecallback(incoming)
(NOTE: The images are fetched from a ROS program via services, and that is where the "imagerequest" function comes from. All of that works fine: the problem is specifically as prescribed.)
To summarize here- since there's a bit of irrelevant clippings in this snippet - an image is loaded by imagecallback. When that function is called, a timer starts (time limit is arbitrary, set low here for testing). When the timer expires, triggeredbytime() requests a new image. It acquires the image via a ROS service, then takes the return from that and uses it as an input to imagecallback(data), thus loading the image and allowing all the keypress functions to still be valid.
So the problem is that whenever the program refreshes with a new frame, the while keepgoing == True loop does not stop, leading to stacked loops that go on and get worse with every refresh. this leads to a single keypress executing multiple times, which quickly overwhelms the program it's pulling images from. (Originally, this was just a while True: loop, but I added the "keepgoing" variable to try to remedy the situation - unsuccessfully, as it were.) I'm not sure how to resolve this issue, and attempting to turn the loop "off" long enough to kill the first loop but allowing the next one to execute has not worked out so far. Is there any way to exit this while keepgoing == True loop cleanly to prevent things from stacking as such, or is there a better way in general to address the problem of loading a new image after a certain amount of time?
Thanks!
EDIT: to be clear, the problem is with ending the 'while' loop when the timer hits 0. setting keepgoing = False doesn't seem to work (probably because I can't 'return' it), so I need something similar to that if it's possible...

CV2 Error while finding smaller img within img in python

This is a short script using pyscreenshot (https://github.com/ponty/pyscreenshot) to get the screen image then looks for a smaller loaded image (local file 'refresh.png', about 32x32 px) inside of it.
I need to duplicate tabs in Chrome and middle clicking refresh does this wonderfully without any reloads.
I get error C:\slave\WinInstallerMegaPack\src\opencv\modules\core\src\matrix.cpp:113: error: (-215) s >= 0 when it gets to second part of the watch. Getting the whole screen the checking works, but getting just part of the screen then checking is a problem. I have no idea what that error means, even after looking at some source. Checking the whole screen for 'refresh.png' takes a bit over a second. Afterwards it starts failing on the partial screen screengrab check and I can't figure out why or understand this error. Checking just part of the screen would go much faster.
From the only results on Google I'd have to guess that the array has too many variables, but I don't get how. I added some print statements that show everything before and after the problem is 1 variable of len 4.
import Image, numpy
import cv2
import numpy as np
import pyscreenshot as ImageGrab
import os
from pymouse import PyMouse
from time import clock, sleep
def grabScreen(Bbox = None): #unlike rect in pygame, bbox does top left (x,y), bottom right (x,y). So the second x,y isn't width, height, it's width + x, height + y
tempmousexy = m.position() #get current mousexy
m.move(0,0) #move pointer to the corner for screen shot (change this if you put the refresh button in the very top left, weirdo
if Bbox:
screenshot = ImageGrab.grab(bbox=Bbox) #grab part of screen
else:
screenshot = ImageGrab.grab() #grab whole screen
m.move(tempmousexy[0],tempmousexy[1]) #put mouse pointer back after screenshot
return np.array(screenshot)
def saveScreen(savename = 'img.png'): #not used right now, but a cool line
ImageGrab.grab_to_file(savename)
def findMatch(img, template, closenesslimit = .001, method = cv2.TM_CCOEFF_NORMED):
#convert to gray for faster processing - probably still accurate enough
template = cv2.cvtColor(template, cv2.CV_32FC1)
img = cv2.cvtColor(img, cv2.CV_32FC1)
#check for match
m = cv2.matchTemplate(template, img, cv2.cv.CV_TM_SQDIFF_NORMED)
#we want the minimum squared difference - lower means less difference
mn,_,mnLoc,_ = cv2.minMaxLoc(m)
print 'RAW (best match, worst match, bestmXY, worstmXY):', str(cv2.minMaxLoc(m))
#check if match is closer (less than) closenesslimit
print 'Closest match (lower is better):', str(mn)
if mn > closenesslimit: return False
#else
print '~Match found~ (closeness was better - less - than limit)'
#find rectangle
matchx, matchy = mnLoc
matchw, matchh = template.shape[:2]
print (matchx,matchy,matchx+matchw,matchy+matchh) #checking what is about to be returned
return (matchx,matchy,matchx+matchw,matchy+matchh) #x1, y1, x2, y2
def checkScreen():
print 'Checking whole screen'
return findMatch(grabScreen(),templateimg)
def checkArea(checkrect):
print 'Checking area'
return findMatch(grabScreen(checkrect),templateimg)
global m
m = PyMouse() #load PyMouse for mouse moves and position checks
guessxy = 0 #once a good x,y is detected, detection will start out there
startdelay = 3 #delay while waiting for first event
runningdelay = .2 #delay between repeating check events after first event
if os.name == 'nt':
templateimg = cv2.imread('bensmall.png')
else:
templateimg = cv2.imread('alexsmall.png') #changes based on resolution, would be better if this scaled automatically based on screen res
print 'Global mouse set, small template (refresh button image) loaded'
while True:
while not guessxy: #check the whole screen until a location is found
sleep(startdelay)
guessxy = checkScreen()
if not guessxy: print 'Nothing found yet, sleeping', str(startdelay)
print 'GUESSXY:', str(guessxy)
fails = 0
faillimit = 5
while guessxy: #once location is found, guess near this area again
sleep(runningdelay) #sleep at the beginning avoids awkward pauses later when something is found but it has to sleep again before printing about it
runstart = clock()
temppos = m.position()
print 'GUESSXY (limiting screenshot to this area):', str(guessxy)
found = checkArea(guessxy) #increase guessxy area by some small amount on each side later
print found
if found:
guessxy = found
print guessxy
m.move(guessxy[0],guessxy[1]) #remove this line later
print 'found'
else:
fails+=1
print 'Not found,', str(fails)
if fails>= faillimit: #wasn't near the guess point, go back to checking whole screen
print 'Too much fail, resetting...'
guessxy = 0
print clock()-runstart

Categories