This is a short script using pyscreenshot (https://github.com/ponty/pyscreenshot) to get the screen image then looks for a smaller loaded image (local file 'refresh.png', about 32x32 px) inside of it.
I need to duplicate tabs in Chrome and middle clicking refresh does this wonderfully without any reloads.
I get error C:\slave\WinInstallerMegaPack\src\opencv\modules\core\src\matrix.cpp:113: error: (-215) s >= 0 when it gets to second part of the watch. Getting the whole screen the checking works, but getting just part of the screen then checking is a problem. I have no idea what that error means, even after looking at some source. Checking the whole screen for 'refresh.png' takes a bit over a second. Afterwards it starts failing on the partial screen screengrab check and I can't figure out why or understand this error. Checking just part of the screen would go much faster.
From the only results on Google I'd have to guess that the array has too many variables, but I don't get how. I added some print statements that show everything before and after the problem is 1 variable of len 4.
import Image, numpy
import cv2
import numpy as np
import pyscreenshot as ImageGrab
import os
from pymouse import PyMouse
from time import clock, sleep
def grabScreen(Bbox = None): #unlike rect in pygame, bbox does top left (x,y), bottom right (x,y). So the second x,y isn't width, height, it's width + x, height + y
tempmousexy = m.position() #get current mousexy
m.move(0,0) #move pointer to the corner for screen shot (change this if you put the refresh button in the very top left, weirdo
if Bbox:
screenshot = ImageGrab.grab(bbox=Bbox) #grab part of screen
else:
screenshot = ImageGrab.grab() #grab whole screen
m.move(tempmousexy[0],tempmousexy[1]) #put mouse pointer back after screenshot
return np.array(screenshot)
def saveScreen(savename = 'img.png'): #not used right now, but a cool line
ImageGrab.grab_to_file(savename)
def findMatch(img, template, closenesslimit = .001, method = cv2.TM_CCOEFF_NORMED):
#convert to gray for faster processing - probably still accurate enough
template = cv2.cvtColor(template, cv2.CV_32FC1)
img = cv2.cvtColor(img, cv2.CV_32FC1)
#check for match
m = cv2.matchTemplate(template, img, cv2.cv.CV_TM_SQDIFF_NORMED)
#we want the minimum squared difference - lower means less difference
mn,_,mnLoc,_ = cv2.minMaxLoc(m)
print 'RAW (best match, worst match, bestmXY, worstmXY):', str(cv2.minMaxLoc(m))
#check if match is closer (less than) closenesslimit
print 'Closest match (lower is better):', str(mn)
if mn > closenesslimit: return False
#else
print '~Match found~ (closeness was better - less - than limit)'
#find rectangle
matchx, matchy = mnLoc
matchw, matchh = template.shape[:2]
print (matchx,matchy,matchx+matchw,matchy+matchh) #checking what is about to be returned
return (matchx,matchy,matchx+matchw,matchy+matchh) #x1, y1, x2, y2
def checkScreen():
print 'Checking whole screen'
return findMatch(grabScreen(),templateimg)
def checkArea(checkrect):
print 'Checking area'
return findMatch(grabScreen(checkrect),templateimg)
global m
m = PyMouse() #load PyMouse for mouse moves and position checks
guessxy = 0 #once a good x,y is detected, detection will start out there
startdelay = 3 #delay while waiting for first event
runningdelay = .2 #delay between repeating check events after first event
if os.name == 'nt':
templateimg = cv2.imread('bensmall.png')
else:
templateimg = cv2.imread('alexsmall.png') #changes based on resolution, would be better if this scaled automatically based on screen res
print 'Global mouse set, small template (refresh button image) loaded'
while True:
while not guessxy: #check the whole screen until a location is found
sleep(startdelay)
guessxy = checkScreen()
if not guessxy: print 'Nothing found yet, sleeping', str(startdelay)
print 'GUESSXY:', str(guessxy)
fails = 0
faillimit = 5
while guessxy: #once location is found, guess near this area again
sleep(runningdelay) #sleep at the beginning avoids awkward pauses later when something is found but it has to sleep again before printing about it
runstart = clock()
temppos = m.position()
print 'GUESSXY (limiting screenshot to this area):', str(guessxy)
found = checkArea(guessxy) #increase guessxy area by some small amount on each side later
print found
if found:
guessxy = found
print guessxy
m.move(guessxy[0],guessxy[1]) #remove this line later
print 'found'
else:
fails+=1
print 'Not found,', str(fails)
if fails>= faillimit: #wasn't near the guess point, go back to checking whole screen
print 'Too much fail, resetting...'
guessxy = 0
print clock()-runstart
Related
Total beginner here.
In this program, I am trying to "monitor" a specific area on my screen. There are several images entering the region (from the right side) with random frequency. (Basically, a "simple" rhythm" game).
I have tried pyautoguy.locateOnScreen, but it was way too slow even though I set the region into a small square. I don't think I can use pixels because the images entering the region are all the same color but have different symbols on them, and each relates to a keyboard letter.
In this last attempt, I tried this:
Create an "endless" while loop, in which I make a screenshot of my desired region/location(bbox), and within that endless loop, I use IF and several ELIF in which I use pyautogui.locate to search a specific image (tiny screenshot of the symbol) on the bigger screenshot made by ImageGrab. I have several images, and each image has a different keyboard response.
I got to about 75-80% accuracy, but when the symbols are close to each other, it always misses. When I checked at which rate the whole loop runs, I get to like 2.9 runs per second, sometimes 4-5. Could anyone give me some pointers on how to double the speed (if that's even possible?)
Thanks!
import time
import pyautogui
import os
import pydirectinput
from PIL import ImageGrab, Image
import time
start_time = time.time()
x = 1 # frames
counter = 0
bbox=(588,786,635,944) #my region I am searching in
def detect(imgDir, haystackImage, confidence=0.90, cache=True):
if (pyautogui.locate(os.path.join('images', imgDir), haystackImage, confidence=confidence)) is not None:
return True
else:
return False
while True:
ScreenGrab = ImageGrab.grab(bbox)
ScreenName = f"screenshot"
Path = f"./images/{ScreenName}.png"
ScreenGrab.save(Path)
hsImage = Image.open('D:/PY/images/screenshot.png')
if detect('D:/PY/images/a.png', hsImage, cache=True):
pydirectinput.press('a')
continue
elif detect('D:/PY/images/b.png', hsImage, cache=True):
pydirectinput.press('b')
continue
elif detect('D:/PY/images/c.png', hsImage, cache=True):
pydirectinput.press('c')
continue
elif detect('D:/PY/images/d.png', hsImage, cache=True):
pydirectinput.press('d')
continue
elif detect('D:/PY/images/e.png', hsImage, cache=True):
pydirectinput.press('e')
continue
else:
pass
counter+=1 #counter for my runs per second
if (time.time() - start_time) > x :
print("FPS: ", counter / (time.time() - start_time))
counter = 0
start_time = time.time()
This code:
Looks for an image a
If it finds a it tries to find a match for any image in the array
image_list
If it finds an image in image_list it looks for e
If it finds e it logs it and moves the mouse to x, y checks for pixel colour and then clicks when when it finds a match, clicks.
This is where my problem arises my avoidLog() function is being called 2-3 times per click. I've added print("click") to check if it's actually clicking and it isn't, it's clicking at the right time but for some reason, my log is triggering more than once.
It's logging correctly, just an odd number of times.
I know python is operating in a single thread in my example so I'm not sure why it's looping back round and not clicking. It would make more sense to me if it was clicking multiple times and logging multiple times.
import pyautogui as py
def avoidLog():
avoidLog = open('Avoided.txt', 'a')
avoidLog.write("Found at: " + str(f) + " at: " + str(skipTime))
avoidLog.write("\n")
avoidLog.close()
image_list = []
while True:
if py.locateOnScreen('a.jpg') != None:
for image in image_list:
found = py.locateCenterOnScreen(image)
if found != None:
skipTrigger = py.locateOnScreen('e.jpg')
if skipTrigger != None:
avoidLog()
py.moveTo(x, y)
r = py.pixelMatchesColor(x,y, (r,g,b))
if r == True:
py.sleep(2)
print("click")
py.click()
break
avoidLog() is called whenever e.jpg is located. However, py.click() is only called if pixelMatchesColor is True
Context: I'm a low intermediate level programmer trying to make a game with my own code in order to improve my ability. I know pygame etc... exists but I wanted to try and make my own engine. I'm currently trying to create a display system. My thought process was to create a 2D list: SURFACE where SURFACE[y][x] is a 'pixel' at coordinates x,y on the python console. This worked great. But I wanted more, and thought: Why don't I create 3 levels of surface, BACKSURFACE, MIDSURFACE, and FRONTSURFACE. My idea was that I could create a single flattened image by combining whatever is rendered through BACKSURFACE, MIDSURFACE, and FRONTSURFACE. This way I could do things like, create an options menu that overlays on top of the game without clearing the entire display.
Now comes my Issue. I created a function addToSurface() that takes a surface, the x, y positions of where to render the image/text, and a list that contains the render item.
def addToSurface(surface, x_pos, y_pos, renderItem):
global SCREENWIDTH, DISPLAYHEIGHT
returnSurface = []
returnSurface = surface.copy() # Make a copy of the surface list
itemHeight = len(renderItem)
itemWidth = len(renderItem[0])
if itemHeight <= DISPLAYHEIGHT and itemWidth <= SCREENWIDTH:
render_y = 0
for y in range(y_pos, itemHeight+y_pos):
render_x = 0
for x in range(x_pos, itemWidth+x_pos):
if renderItem[render_y][render_x] == " ": # Because the 'image' files are ASCII, I turn every
# space in the image into a [None] pixel. The
# original surface lists are initialised where
# every pixel is [None]
returnSurface[y][x] = [None]
else:
returnSurface[y][x] = renderItem[render_y][render_x] # If a pixel in the image isn't empty
# the equivalent pixel coordinate in the
# surface becomes the pixel.
render_x += 1
render_y += 1
return returnSurface # A new version of the surface is returned.
For example, if I run BACKSURFACE = addToSurface(BACKSURFACE, 0, 3, logoImage) where logoImage is a 2D list containing an ASCII image of my games logo, what I expect is for BACKSURFACE to now contain the logo at starting at BACKSURFACE[3][0] (coordinates 0, 3). Like I mentioned above, this works great. But for some reason, after creating my MIDSURFACE, and FRONTSURFACE lists, everytime I run addToSurface() all three lists are changed.
So now, when I run BACKSURFACE = (addToSurface(BACKSURFACE, 0, 3, logoImage)), not only does BACKSURFACE come with the logo rendered, so do MIDSURFACE and FRONTSURFACE, even though they haven't been referenced in the program at all after initialisation. For reference, here are my other functions in the program so far:
def __init__(height, width):
global SCREENWIDTH, SCREENHEIGHT, TITLE, BACKSURFACE
global MIDSURFACE, FRONTSURFACE, DISPLAYHEIGHT
SCREENWIDTH = width
SCREENHEIGHT = height
DISPLAYHEIGHT = height - 11 #The above 3 lines are just to set display size.
for i in range(DISPLAYHEIGHT): # This loop creates the 2D lists for all surfaces
temp = [] # where each pixel has a default value of [None]
BACKSURFACE.append(temp)
MIDSURFACE.append(temp)
FRONTSURFACE.append(temp)
for j in range(width):
temp.append([None])
def openImage(imageName, centre=False):
temp = []
dir = os.path.join(imageDirectory,imageName)
if not os.path.exists(dir):
return null
elif not centre:
with open(os.path.join(imageDirectory, imageName), "r") as f:
for line in f:
temp.append([char for char in line.rstrip("\n")])
f.close()
elif centre:
with open(os.path.join(imageDirectory, imageName), "r") as f:
for line in f:
centredLine = tech.centre(line.rstrip("\n"))
temp.append([char for char in centredLine])
f.close()
return temp
Like I mentioned above, I would describe myself as just past begginer level programming and this is definitely the most complex thing I've ever attempted. Even so, I have a feeling it's going to be something really simple. If you could point out where the error is and why it's an issue I'd be more than happy. If anyone could suggest a fix I'd appreciate it as well.
The objective is to move alien1, atarts from 0,0 then moves all the way to the right, goes down and then all the way to the left, and then down.
from tkinter import *
import random
def enemigos():
global Enemigos #Enemigos downloads the image for alien1
n = random.randint(1,3)
if n == 1:
def movalien1():
alien1 = CanvasJuego.create_image(0,0, anchor = NW, image = Enemigos[0], tags= ('alien1'))
RIGHT1 = True
CoordsAlien1 = CanvasJuego.coords(alien1)
if (CoordsAlien1[0] < 1000 and RIGHT1==True):
CanvasJuego.coords(alien1, CoordsAlien1[0]+5, CoordsAlien1[1])
if ((CoordsAlien1[0]+5)==1000):
RIGHT1 = False
CanvasJuego.coords(alien1, CoordsAlien1[0], CoordsAlien1[1]+50)
elif (CoordsAlien1[0]>0 and RIGHT1==False):
CanvasJuego.coords(alien1, CoordsAlien1[0]-5, CoordsAlien1[1])
if ((CoordsAlien1[0]-5)==0):
RIGHT1 = True
CanvasJuego.coords(alien1, CoordsAlien1[0], CoordsAlien1[1]+50)
def rec():
movalien1()
root.after(20,rec)
root.after(20,movalien1())
Alien1 does appear at (0,0), but it won't move.
The problem is that you create a new "alien" every 20 milliseconds. You should be creating alien1 exactly once outside of movalien1. What is happening is that you create it at 0,0, then move it to 5.0. The alien is at 5,0. The next time through the loop you create a new alien at 0,0, and then move it to 5,0. You keep creating new aliens over and over and moving the new alien to 5,0.
Also, you can use the move method to move an item instead of adjusting its coordinates.
Finally, even though it doesn't actually matter in this code, you are calling after incorrectly here: root.after(20, movealien1()). It needs to be either root.after(20, movealien1) or just directly call movealien1() without using after.
I've got a program right now that is meant to display an image, and allow for various functions on keypress (e.g., load a new frame, crop a region of interest, etc). It's also set up so that if too much time passes with no action, it'll automatically attempt to load a new image.
def triggeredbytimer():
global f, x1i, y1i, x2i, y2i, sid, keepgoing
keepgoing = False
print "Inactivity has lead to an automatic refresh. Loading newest image."
f = 0; t = 0
incoming = imagerequest(sid, f, t, x1i, y1i, x2i, y2i)
imagecallback(incoming)
def imagecallback(data):
print "----------------------- imagecallback"
global sid, f, x1i, y1i, x2i, y2i, img, refPt, cropping, keepgoing
keepgoing = True
########### \\// change this to change wait for autorefresh
tmr = Timer(10.0, triggeredbytimer)
tmr.start()
##################################
b = data.b; t = data.tr; sid = data.sid; error = int(data.error)
t = float(t); sid = int(sid)
expandedt = time.strftime("%m-%d-%Y %H:%M:%S", time.localtime(t))
print str(t) + " ----> " + expandedt
print "----------------------- image incoming"
timestr = time.strftime("%H:%M:%S", time.gmtime(t))
print "Loaded image from %s."%timestr
imagedata = bridge.imgmsg_to_cv2(b, "bgr8")
npimg = cv2.imencode('.jpg', imagedata)
cv2.imwrite('temp_image_np.jpg', imagedata)
img = cv2.imread('temp_image_np.jpg')
clone = img.copy()
cv2.namedWindow('image', cv2.WINDOW_NORMAL)
cv2.setMouseCallback("image", mouseclickcb)
cv2.imshow("image", img)
while keepgoing == True:
key = cv2.waitKey(0) & 0xFF
## (here there are a lot of keypress functions, for brevity I included only one)
elif key == ord("t"):
print "Looking for newest frame..."
tmr.cancel()
t = 0; f = 0
incoming = imagerequest(sid, f, t, x1i, y1i, x2i, y2i)
imagecallback(incoming)
(NOTE: The images are fetched from a ROS program via services, and that is where the "imagerequest" function comes from. All of that works fine: the problem is specifically as prescribed.)
To summarize here- since there's a bit of irrelevant clippings in this snippet - an image is loaded by imagecallback. When that function is called, a timer starts (time limit is arbitrary, set low here for testing). When the timer expires, triggeredbytime() requests a new image. It acquires the image via a ROS service, then takes the return from that and uses it as an input to imagecallback(data), thus loading the image and allowing all the keypress functions to still be valid.
So the problem is that whenever the program refreshes with a new frame, the while keepgoing == True loop does not stop, leading to stacked loops that go on and get worse with every refresh. this leads to a single keypress executing multiple times, which quickly overwhelms the program it's pulling images from. (Originally, this was just a while True: loop, but I added the "keepgoing" variable to try to remedy the situation - unsuccessfully, as it were.) I'm not sure how to resolve this issue, and attempting to turn the loop "off" long enough to kill the first loop but allowing the next one to execute has not worked out so far. Is there any way to exit this while keepgoing == True loop cleanly to prevent things from stacking as such, or is there a better way in general to address the problem of loading a new image after a certain amount of time?
Thanks!
EDIT: to be clear, the problem is with ending the 'while' loop when the timer hits 0. setting keepgoing = False doesn't seem to work (probably because I can't 'return' it), so I need something similar to that if it's possible...