Run a python code only when a specific object appears in screen - python

In the following image, I want Python (using module Pyautogui) to click the "OK" button.
The problem is, I dont know when the "OK" button appears. So how do I make python to click the "OK" button only when it appears?
References:
If you wonder it looks like a mobile screen, I am using an android emulator.

PyAutoGUI has a function called locateOnScreen() which takes a screenshot of your current screen and then looks for the provided image you pass into the parameter of the function.
First, you need to have a picture of the "OK" button to tell PyAutoGUI what to look for, which you can do using pyautogui.screenshot('ok.png', region=(0, 0, 300, 400)) the region parameter specifies where to look and the size of the picture.
After that, you can run locateOnScreen('ok.png') and if the picture 'ok.png' is currently on your monitor it will return the center coordinates, which you can then pass to pyautogui.click()
import pyautogui
pyautogui.screenshot('ok.png', region=(0, 0, 300, 400))
Run that line of code and play around with the region parameter until you get a good looking 'ok.png' and make sure only the button is visible in the picture as PyAutoGUI looks for an exact match. Once you're done, replace the line with the following:
import pyautogui
location = pyautogui.locateOnScreen('ok.png')
pyautogui.click(location)
Now, whenever the "OK" button is on your screen, PyAutoGUI will click on it

Related

Why isn't win32api.SendMessage able to perfom mouse clicks in some applications

I'm on Windows OS and I'm trying to perform clicks on background (inactive) applications without bringing them to the front.
I've been successful in doing in in Microsoft Paint (painting a dot on a minimised paint instance) but for some reason this doesn't seem to work on a Java based application (RuneLite.exe) which I'm writing this program for.
Some additional information:
Paint has a lot of child windows and performing the click only seems to work when I execute it on a specific child window (class: Afx:00007FF758890000:8) this is the most inner window.
The RuneLite window structure is as follows:
RuneLite (class: SunAwtFrame)
unknown (class: SunAwtCanvas)
unknown (class SunAwtCanvas)
I've tried using the handle of all 3 of the above windows but none seem to work
The code is able to find the handle of the window and it is correct (checked with Spy++)
The window blinks orange in the taskbar after running the code so some event must have been fired
I have determined the coordinates by screenshotting the window and using paint to find coordinates of a specific location. (If there is a better way of doing this, please let me know)
from pywinauto.application import Application
amount = 2
handles = []
def start_apps():
for i in range(amount):
a = Application()
a.start("D:\\Users\\Arno\\AppData\\Local\\RuneLite\\RuneLite.exe")
time.sleep(15)
handle = a.window().child_window(class_name="SunAwtCanvas", found_index=1).handle # select the most inner window
handles.append(handle)
def do_click(h, x, y):
long_position = win32api.MAKELONG(x, y) # simulate the mouse pointer and send it to the specified coordinates
win32api.SendMessage(h, win32con.WM_LBUTTONDOWN, win32con.MK_LBUTTON, long_position) # simulate mouse press
win32api.SendMessage(h, win32con.WM_LBUTTONUP, win32con.MK_LBUTTON, long_position) # Simulate mouse up
start_apps()
print(handles)
time.sleep(20)
for h in handles:
do_click(h, 450, 290)
Output of the code
Does anyone know why this happens and how I could go about fixing it?

pyautogui to detect the button change

pyautogui works well when using it to click on the buttons (of a software ) on the screen
but is there any way to detect the change in button state, because when the first click is completed and the required task is accomplished then the button disappears and a new "yes" button appears and if the task is not accomplished then the "NO" button appears , the problem is , both buttons appear at the same x and y coordinates i mean either yes or no appears at that place , is there any way to find out if yes appeared or no appeared? and then clicking if yes appeared ?
You can use image recognition technique in pyautogui package.
You can use locateOnScreen function of pyautogui. First you can screenshot and save yes and no button image. You crop the image tightly around the button.
And then save them into "yes.png", and "no.png" respectively.
And then,
btnYesButton = None
btnNoButton = None
while btnYesButton == None and btnNoButton == None:
btnYesButton = pyautogui.locateOnScreen("yes.png")
btnNoButton = pyautogui.locateOnScreen("no.png")
if btnYesButton:
tmpCenter = pyautogui.center(btnYesButton)
pyautogui.click(tmpCenter)

How to simulate a mouse click and drag?

I want to simulate an event where I left click on the Windows desktop, I drag the mouse a little to create a Selection box to another point, and then keep holding the left button at that point for some time without the Selection box disappearing.
The problem is, I can't get him to keep the Selection box, whenever he gets to the other point the Selection box disappears indicating that the button has been released.
I tried to implement in Python using PyAutoGUI. I tried several ways to do this but still unsuccessfully. Is there any function I'm missing?
import time
import pyautogui
time.sleep(3)
while True:
pyautogui.moveTo(1080, 380)
pyautogui.mouseDown(button='left')
pyautogui.dragTo(917, 564, 1, button='left')
time.sleep(10)
pyautogui.mouseUp(button='left')
time.sleep(2)
Simply removing 2 lines of code and changing dragTo() to moveTo() seems to do what you are trying to do:
import time
import pyautogui
time.sleep(3)
while True:
pyautogui.moveTo(1080, 380)
pyautogui.mouseDown(button='left')
pyautogui.moveTo(917, 564, 1)
time.sleep(10)
This might help you a bit:
pyautogui.moveTo(1277, 127)
pyautogui.dragTo(1277, 225, button='left', duration=5)
(duration is in seconds)

Why pyautogui cannot find items in OSX Dock?

I started to learn pyautogui for my personal project and almost instantly ran into the problems when trying to open OSX dock icons.
I want to open local Spotify which is under Mac Launchpad.
My code to do this.
import pyautogui
launchpad = pyautogui.locateOnScreen('img/Launchpad.png')
This return None so the image was not found.
image example attached
However, if I open Mac OSX Notes window and paste the same image into it and ran the program again the image is found every time. Similarly, if I just leave image open in my Editor.
Is dock actually part of the OSX screen pyautogui can search from? If not how to interact with it?
Figured that using application hotkeys vs find on the screen is a much less brittle approach. Below how I finally build Spotify bot.
import time
import pyautogui
# use pyauutogui key shortcut to open OSX spotlight search
pyautogui.hotkey('command', 'space')
# type spotify and press enter to open application
pyautogui.typewrite('Spotify')
pyautogui.hotkey('enter')
# use Spotify keyboard shortcuts to select search.
# key docs here: https://support.spotify.com/ie/article/Keyboard-shortcuts/
time.sleep(5)
pyautogui.hotkey('command', 'l')
# typewrite allows passing string arguments using keyboard
pyautogui.typewrite('concentration music')
# move to select the song with tab and press enter to play
pyautogui.hotkey('tab', 'tab', 'tab', 'tab')
time.sleep(2)
pyautogui.hotkey('enter')
pyautogui.hotkey('space')
# sleeps 30 seconds while music is playing
time.sleep(30)
pyautogui.hotkey('command', 'q')
I made a program to pertaining your problem. It is a spotify bot aswell.
import pyautogui as p
from time import sleep as t
p.keyDown("command")
p.press("space")
t(2)
p.keyUp("command")
p.typewrite("Spotify")
p.press("enter")
t(3)
p.moveTo(153,132)
p.click()
t(1)
p.typewrite("Never Gonna Give You Up")
p.press("enter")
p.moveTo(707,324)
t(4)
p.click()
t(10)
p.keyDown("command")
p.press("q")
t(1)
p.keyUp("command")
The coordinates will be different pertaining to the position of your spotify window, so you can do p.mouseInfo/pyautogui.mouseInfo to find the exact coordinates and substitute them in the code I gave above.

Work with menuStrip and its items in python

I have to build an automation ui test for a WinForms application. I am using python 3.4 with python for windows extension and pywinauto.
The test is required to access the menu of the application and click on one of the sub menu items.
I used the code below to try and find the menu.
#arrays to store the controls found
classes = []
objects = []
#recursive method to get all the controls
def getClasses(childHwnd, lparam):
objects.append(childHwnd)
classes.append(win32gui.GetWindowText(childHwnd))
return 1
#find handle of the main window
hwnd = win32gui.FindWindow(None, 'Form1')
#get all controls
win32gui.EnumChildWindows(hwnd, getClasses, "a")
# Result:
# objects [1509794, 3344468]
# classes ['Test', 'menuStrip1']
#trying to get the menu of the form
win32gui.GetMenu(hwnd) #Returns 0
Image of the form on which I tested the code above:
As you can see, the menuStrip1 is discovered, but I have not found a way to get to its children (Meniu 1, Meniu 2).
Any idea on how to find the menu and its children?
In the case of Menustrip item an easy workaround is to use the pywinauto basic mouse input modules.
by using this you can move to a specific position of the screen using (x,y) coordinates.
eg:
pywinauto.mouse.click(button='left', coords=(0, 0)) //Click at the specified coordinates
pywinauto.mouse.double_click(button='left', coords=(0, 0)) //Double click at the specified coordinates
pywinauto.mouse.move(coords=(0, 0)) //Move the mouse
pywinauto.mouse.press(button='left', coords=(0, 0)) //Press the mouse button
pywinauto.mouse.release(button='left', coords=(0, 0)) //Release the mouse button
pywinauto.mouse.right_click(coords=(0, 0)) //Right click at the specified coords
pywinauto.mouse.scroll(coords=(0, 0), wheel_dist=1) //Do mouse wheel
pywinauto.mouse.wheel_click(coords=(0, 0)) //Middle mouse button click at the specified coords
After trying without success to use python 3.4 with python for windows extension and pywinauto to create automated tests for the respective project. I looked into other tools, and finally chose one called Sikuli, it's an automation tool based on image recognition of the elements on the screen. It may not be the perfect tool for the job, but it was enough for me.
Other observations about python for windows extension and pywinauto:
can only be used on controls they are aware of (e.g. not
toolstripmenuitem)
it's impossible to test custom controls (for the
reason above)
Observations regarding Sikuli:
all scripts are written in python
it may not use the latest python syntax
at the moment I used it, it had some small bugs
While an answer has already been posted for an alternative solution, I would like to add a workaround that can be used in pywinauto.
On windows, the ALT key can be used to access a menu. For my specific example, I had a settings option "Settings -> Login Remotely". If I wanted to click this menu option I would first have to send ALT+S, and then type L. To do this using pywinauto, assuming you already have a reference to the window, you can do so like this:
myWindow.SetFocus()
myWindow.TypeKeys("%s") #Clicks Settings
myWindow.TypeKeys("l") #Clicks Login Remotely
The '%' character is used to represent the 'ALT' key on your keyboard. Another thing to note here is that if you have multiple menu options beginning with the same letter, you may have to send the key more than once to hit the one you want, and then send the enter key. If there is no ambiguity (in this case, only one submenu option began with L) it was selected automatically.
This requires you to know your menu well, and in what order, but as long as it doesn't change this should work for you.

Categories