I am writing a python script that automates running a program and performing different tasks within the program. My main problem is figuring out how to click buttons and interact with the GUI of the program to be controlled.
I am currently using the pyautogui library and using pyautogui.click(X,Y) to advance through prompts and click on different menus and menu items. The problem with this approach is that I am relying on a separate script to inform me of the coordinates of interest in my environment by telling me the coordinates of where my cursor is hovering. This probably will not work on other machines and just seems like a one case solution.
My question is how can I automate using a program in windows (clicking around) without having to hard code the exact position of the items I need to click?
For example, If I need to click a "ok" box to accept some setting, how can I make Windows grab the program window, read through the options and click what I need without any prior knowledge of the position of the dialog box and where the "Ok" button is located?
Code:
import pyautogui as gui
gui.click(x,y)
The way you can do this using pyautogui is with their locating methods. You will need a picture (for example of the OK box) and then you can have pyautogui find it on the screen and give you its coordinates. Check out the official documentation on this.
Related
My team is trying to translate an installation automation from WinBatch to Python. There's a step where I need to click a "Next" button. Unfortunately, the screen display size is not always the same so I can't just use ahk.MouseClick() with screen coordinates. I was wondering if there's a way to use something similar to the MouseClickBtn with either ahk or PyAutoGUI.
I am working on a project written in Python that should perform some actions in an opened application window. For example I run application window with 3 buttons at 3 different positions. I write a script using auto-gui lib for python 3 and set x, y coordinates to click on 3 button positions. But my problem is: If I minimize this window, clicker clicks at the given position within another currently open window.
My main question: is there a way (with help of a lib or by any other means) to trigger or bind this script to work only with this opened (or specified) application. And if I minimize this window, clicker continues to work in this minimized window, not in the currently active one. Please suggest.
You should have a look at win32gui module, more specifically to SetActiveWindows, SetForegroundWindow, SetActiveWindows also EnumWindows to find your application.
With these you can detect a certain application by it's title using EnumWindows you can then maximize it and set it as active so it will be fullscreen and the first one you see on the monitor. This way you can now use the (x,y) coordinates for clicks because the windows will always be maximized therefore the buttons in the same position everytime.
This question already has answers here:
Python code to automate desktop activities in windows
(6 answers)
Closed 5 years ago.
I love automating daily tasks using AutoHotKey and Python. I like using my keyboard instead of my mouse, so I tend to make hotkeys with AutoHotKey that do various actions that you'd otherwise need to use the mouse for, or that you'd need to use too many keyboard actions to do.
But there are some actions that I don't succeed in automating. For example, the dropdown menu for changing the number of monitors that a VM uses in VMWare Workstation. I think that one thing that would really help me with this automation task, and possibly with more in the future, is to be able to simulate menu item clicks.
Is this possible? I want to fool a program into thinking that one of its menu items were clicked. I know to program in Python and AutoHotKey. Is it possible at all, and specifically in these two languages?
Presuming that this is Windows, if the app is using standard HMENU items you can do one of two things with (relatively) minimal effort
If there is a hotkey, like Ctrl+S for Save, you could just send Ctrl+S to the window using the SendKeys API (https://msdn.microsoft.com/en-us/library/windows/desktop/ms646310(v=vs.85).aspx)
If there isn't, you can try to find which WM_COMMAND is sent to the window after the item is selected. You can use Spy++ from Microsoft (https://msdn.microsoft.com/en-us/library/dd460756.aspx) to get the code and then do a PostMessage to that window with the code.
Mind you, you'll need to be sure that you do a FindWindow to get the right HWND handle to send it to.
Best of luck :)
Edit Followup: Quick tutorial on how to use Spy++ with Notepad.exe
Start Spy++
NOTE: There are two of them, spyxx.exe and spyxx_amd64.exe. If one doesn't log messages - use the other
NOTE: if you started Spy++ first, hit F5 to refresh the window list AFTER Notepad.exe starts
Spy++ main menu > Search > Find Window
on the "Window search" dialog, click and hold on the "target" icon next to "finder tool"
drag your cursor over the Notepad title bar
Spy++ will show Caption: "Untitled - Notepad" and Class: "Notepad"
Click "OK"
You will now see Notepad in the window list
Right-click on that entry and select "Messages"
To control the flow, Main Menu > Messages > Logging Options, Messages Tab
Clear All, only select WM_COMMAND, click ok
in Notepad, go to the Main Menu > Edit > Go to
in Spy++, you will see a line reading "WM_COMMAND wNotifyCode: 0 (sent from a menu) wID:24"
Result: Now we know that if you were to PostMessage(WM_COMMAND, MAKE_WPARAM(0,24), HWND of any Notepad.exe on the system), the "Go to" window would appear.
There is command In the AutoHotKey that shows tooltip with some message on the top of the screen. I want to do it on python.
In more details I need to make an application, that will work in background, always tracking a keyboard. On the specific hotkey it should show on the top of the screen a message without grabbing focus from other application. And it should remove this message after some conditions (e.g. mouse moving). It should work with full screen apps too.
I couldn't find how to do it on python, or what libraries should I use. Need your help.
Global tooltips are created via WINAPI CreateWindowEx with TOOLTIPS_CLASS for window class. There are some examples which you can adapt. See also the Autohotkey implementation.
Shell tray tooltips are created via Shell_NotifyIcon.
A working example: wontoncc/balloontip.py and the Autohotkey implementation.
I am writing a script to automate repeatedly registering new users for a website (not boosting metrics, not what you think!). I can boil down the process of registering to a series of mouse clicks and typing. I know there are some macro recorders that will let me record how I use the GUI and repeat it, but I need to type something a little different every time - however the mouse clicks are always the same.
Imagining script to look kindof like:
username = "something"
for i in range(0,100):
playback recorded series of mouse clicks A
type username + str(i)
type some other stuff
playback recorded series of mouse clicks B
But I can't find a good tool to record a series of mouse clicks so that I can play them back from either an applescript or a python script.
I found a useful script (http://www.bluem.net/en/mac/cliclick/) I can call that takes in 2 coordinates and clicks. So even if I could find a tool to record the coordinates of mouseclicks that would work!
You can also run JavaScripts in browsers with something like tell application "Safari" to tell document 1 to do JavaScript or tell application "Google Chrome" to tell tab 1 of window 1 to execute javascript.
tell application "Safari" to tell document 1
do JavaScript "document.querySelectorAll('.mainnavs ul li:nth-child(2) a')[0].click()"
delay 1
repeat until do JavaScript "document.readyState" is "complete"
delay 0.1
end repeat
do JavaScript "document.getElementById('tagfilter').value = 'aa'"
end tell
System Events has commands for simulating clicks and keystrokes:
delay 1
tell application "System Events" to tell process "Safari"
click at {76, 117}
repeat with i from 1 to 3
keystroke "aa" & i & return
end repeat
keystroke "a" using command down
end tell
You can see the positions of UI elements from Accessibility Inspector.app. It is part of Xcode, but it can also be downloaded from developer.apple.com/downloads.
If you want to get the screen coordinates of where you want to perform mouse clicks, you can use a free tool I made called MouseTools found here. I made it for just this purpose. You'll want to do something like I show in the example #1 applescript near the bottom of the page. Just move your mouse to a position on the screen and run the applescript to get the coordinates. Note: you'll want to move your mouse to a location and press command-r (while the applescript is frontmost) to run the script in AppleScript Editor.
In addition, I found that clicking at screen coordinates, as Lauri Ranta showed in her applescript using System Events, often does not reliably work. System Events is good at clicking in the Finder however it often has trouble clicking in the window of an application like Safari. If you find that problem too then you can also use the MouseTools to perform the mouse clicks. Again, this is why I created the program and there are examples of how to do this on the web page.
I hope it helps. Good luck.
How about Carsten Blum's Clicclick?
His link
It is a shell app which will emulate mouse clicks at arbitrary screen coordinates. This will work with applescript.
Hopefully you are doing test automation!
Have you looked into using something like Selenium WebDriver instead?