We Keep Coding

How can I click buttons in a windows GUI program using python?

I am writing a python script that automates running a program and performing different tasks within the program. My main problem is figuring out how to click buttons and interact with the GUI of the program to be controlled.
I am currently using the pyautogui library and using pyautogui.click(X,Y) to advance through prompts and click on different menus and menu items. The problem with this approach is that I am relying on a separate script to inform me of the coordinates of interest in my environment by telling me the coordinates of where my cursor is hovering. This probably will not work on other machines and just seems like a one case solution.
My question is how can I automate using a program in windows (clicking around) without having to hard code the exact position of the items I need to click?
For example, If I need to click a "ok" box to accept some setting, how can I make Windows grab the program window, read through the options and click what I need without any prior knowledge of the position of the dialog box and where the "Ok" button is located?
Code:
import pyautogui as gui
gui.click(x,y)

The way you can do this using pyautogui is with their locating methods. You will need a picture (for example of the OK box) and then you can have pyautogui find it on the screen and give you its coordinates. Check out the official documentation on this.

How do I use python to auto-answer a pop-up prompt in Windows?

I use a ticketing program at work all day every day. Every 5 minutes it produces a little pop-up window asking "are you still using the program?" and if you don't click yes in 60 seconds it logs you out and closes the program. Very annoying.
I want to use Python (or whatever else works, I don't really care about the language) to make a script that detects when this window pops up and automatically presses the "yes I am still here" button on that window.
I believe that I can locate the pop-up window using Python's subprocess library and win32gui library. That is still a work in progress but if you think you have a better method I'm open to suggestions.
Once I do identify the pop-up window, what would be the best way to "click" the "yes I am still here" button? It would be nice if I could do this in the background milliseconds after the window is generated so that it's almost like the window never existed at all, but it's fine if I need to bring the window to the foreground and use something like x,y coordinates to simulate a click.
Thank you for any help!

as #barmar said, https://pyautogui.readthedocs.io/en/latest/ will help you.
You can take a screen shot and scale it to get the image file of the popup. Then, you can use some python code to locate the image and press the button.
ButtonLocation = pyautogui.locateOnScreen('bttnScshot.png')
ButtonPoint = pyautogui.center(ButtonLocation)
button7x, button7y = ButtonPoint
pyautogui.click(button7x, button7y)
Thank you for your time, I am new to
stackoverflow. :)

Using PIL (Python Image Library) to detect image on screen

I am trying to understand how I can use PIL in Python 2.7 to search the whole screen for a certain image and click on it. I've been searching around and haven't been able to find a solution. I want to create a small GUI with one button in the middle of it that when clicked will search the entire screen for a predefined image. Once the image is found the program will then click in the centre of it and end. In short the program will detect if an image is present on the users screen and click it.
I did find an interesting bit on Sikuli, but that doesn't help me because it's unable to export to an .exe.
The image that the program will look for will most likely be in the same place each time it searches, but I didn't want to hard-code the location as it has the potential to move and I don't want that being an issue later on.
What I need is the code method I would use to search for the image on screen and send back the cords to a variable.
Image explanation/example:
Reference image of rifle:

PIL is the wrong tool for this job. Instead you should look into openCV (open source computer vision), which has fantastic python bindings. Here is a link to an example (in C but should be easy to redo with the python bindings) that does what you are looking for, but even allows the image to be rotated, scaled, etc.
http://docs.opencv.org/doc/tutorials/features2d/feature_homography/feature_homography.html
http://docs.opencv.org/doc/tutorials/features2d/detection_of_planar_objects/detection_of_planar_objects.html
Edit:
I assume you are using windows, as your example image looks like window. In this case you can use:
from PIL import ImageGrab
pil_img = ImageGrab.grab()
opencv_img = numpy.array(pil_img)
then use opencv to process the image to find sub image you are looking for.
If you want to do this cross platform, then you will need to use wxWidgets to do the screengrab: https://stackoverflow.com/a/10089645/455532

Even I wanted to do the same but using different module - pyautogui. I finally found the solution for my problem and I am sure this solution will also help you.
You have to just go to this webpage and read the locate function topic completely
and you'll be able to solve your problem.

I recommend you give a look on PyAutoGUI, a well documented library to control mouse and keyboard, also can locate imagens on screen, find the position, move the mouse to any location and clicks on location, also can simulate drag and drop, type on input fields, give double clicks and much more.

python pygtk how to put a window on a specific display monitor

I'm using Python PyGTK library on OS Linux based (suse, ubuntu)
Working on devices with multiple Display Monitors.
I'm trying to put a Full Screen Window (in python, gtk.Window) on a Specific Display monitor connected to the Device.
I'm using the following code:
n_monitors = gtk.gdk.Screen.get_n_monitors() # Get number of monitors.
gtk.gdk.Screen.get_monitor_geometry(*monitor_number*) # Get the size of a specific monitor.
The second api returns the monitor size and offset information.
gtk.Window.move(offset_x, offse_y) can be used for moving a window on a specific monitor.
Though this doesn't seem to work all the time. It looks like the window has an affinity to the mouse location or if the monitor resolutions are not the same this does not work.
Is there any property of the gtk.Window which would help mitigate this issue.
I tried playing with following which didn't help :
gtk.Window.set_position(gtk.WIN_POS_NONE)
gtk.Window.set_gravity(gtk.gdk.GRAVITY_NORTH_WEST)
Any ideas.

Your monitors collectively form one screen. You can query a screen to find out about its monitors. You can then position your window on a given monitor by ofsetting the window's (x,y) with that of the monitor.
Assuming your window is window, you can get the screen like this
screen = window.get_screen()
and then get the monitor data like this
monitors = []
for m in range(screen.get_n_monitors())
monitors.append(screen.get_monitor_geometry(m)
monitors are numbers 0,1,2,... You can get the current monitor, if you need to, like this:
screen.get_monitor_at_window(screen.get_active_window())
To position a window on a monitor, you add the monitor's (x,y) to the window's (x,y). Calculate those x,y values and then move the window:
window.move(x,y)
See a similar answer I provided here.

Is there a way to prevent display turning off?

I'm writing a presentation software. I control the presentation on my primary screen, and I display it on the second screen. If the image on the second screen doesn't change after a while, my computer turns off the second screen. Off course, I could simply disable powersaving features on my computer, but what I'd like would be to have my software tell the OS not to turn off the second display when it displays fullscreen on it. How can I achieve that? I'm on Linux, and using Python and GTK3.