Clicking on pixel for certain process with python - python

Hello I am programming a program to automate mouse presses on a program for certain pixels but I don't want a second program to come in the way with that click, my program is going to look for a green pixel and click it on a certain part of the screen, but if there is another program/image in the way that is green I don't want it to click on that
I just want it to click on the process/program I want it to click on, and not click on the screen
If anyone could give me some tips on this, that would be helpful

To get the focused window (do not click if this is focused) use:
from win32gui import GetWindowText, GetForegroundWindow
print(GetWindowText(GetForegroundWindow()))
Do this on your windows and then do if statements to stop clicking on pixels.
But to click on the pixels you can use win32con and win32api:
import win32api, win32con
def click(x,y):
win32api.SetCursorPos((x,y))
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,0,0)
sleep(0.01)
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,0,0)
To get the pixels and click on them use PyAutoGui
import pyautogui
from pyautogui import *
width = 1920
hight = 1080
while WindowIsFocused:
pic = pyautogui.screenshot()
for x in range(0, width, 1):
for y in range(0, hight, 1):
r, g, b = pic.getpixel((x, y))
if r == 252:
if g == 200:
if b == 118:
click(x,y)
print("Clicked")
Set width and hight to your screen resolution and WindowIsFocused to True if the window you want is focused. Use an extra function for that (the function should run constantly).
I hope that I could help. For any questions ask me. :)

Im sorry im not familiar with keys interacting with programs, but i made a little research and found a library called PyWin32 that should statisfy your need. You can search for its documentation or try your luck by finding videos on this particullar library on youtube.
Anyways, hopefully this helped you getting set in the right direction, and feel free to ask any question

Related

How can I keep it open even when I interact with the background? [duplicate]

I'm making a pseudo transparent window in pygame with the intent of displaying varied info like a "HUD"
The script uses PIL to grab an image of the desktop and use it as the background of the window.
A simple version:
import pygame as py
from ctypes import windll
import ImageGrab, Image
SetWindowPos = windll.user32.SetWindowPos
py.init()
def get_image():
im = ImageGrab.grab((0,0,window_x,window_y))
mode = im.mode
size = im.size
data = im.tobytes()
im = py.image.fromstring(data,size,mode)
return im
window_x = 1920
window_y = 100
background = py.Surface((window_x,window_y))
background.blit(get_image(),(0,0))
window_pos = (0,0)
screen = py.display.set_mode((window_x,window_y),py.HWSURFACE|py.NOFRAME)
SetWindowPos(py.display.get_wm_info()['window'],-1,0,0,0,0,0x0001)
clock = py.time.Clock()
done = False
while not done:
for event in py.event.get():
if event.type == py.QUIT:
done = True
screen.blit(background,(0,0))
py.display.flip()
clock.tick(30)
py.quit()
This creates a Pygame window at the top of the screen.
My problem is that the Pygame window blocks any mouse interaction with anything beneath it.
Is there a way to allow mouse events to be ignored and go 'through' the window, like for example clicking on a desktop icon, underneath a Pygame window.
You will need to do a bit of an extra hacking which is outside what PyGame gives you. It should be possible to render the PyGame canvas into another windowing framework in Python and try to use advanced features of that library to achieve this.
In Windows
One example is wxWidgets. As described in this thread, which sounds quite similar to what you are trying to achieve, the user has setup a window which can be clicked through and is transparent.
Also see this another Stackoverflow Post which mentions how to handle the Hit Testing in C#. Posting code from that post here:
protected override void WndProc(ref Message m)
{
if (m.Msg == (int)WM_NCHITTEST)
m.Result = (IntPtr)HTTRANSPARENT;
else
base.WndProc(ref m);
}
It is possible to do similar testing in Python using win32 APIs. This is a piece of Python code that does exactly this. Locate the part where the programmer sets up the callback for the event (something like win32con.WM_NCHITTEST: self.onChi, where self.onChi is the callback).
I hope this gives you a starting point. I doubt there is anything readymade that you will find out of the box but these should give you some pointers on what to look for.
This is an old question, but I have ran into it quite a few times and only now got it to work right. Here is the relevant bit.
import win32api
import win32con
import win32gui
fuchsia = (255, 0, 128) # Transparency color
hwnd = pygame.display.get_wm_info()["window"] # Handle
styles = win32gui.GetWindowLong(hwnd, win32con.GWL_EXSTYLE)
styles = win32con.WS_EX_LAYERED | win32con.WS_EX_TRANSPARENT
win32gui.SetWindowLong(hwnd, win32con.GWL_EXSTYLE, styles)
win32gui.SetLayeredWindowAttributes(hwnd, win32api.RGB(*fuchsia), 0, win32con.LWA_COLORKEY)

Pyautogui macro background

I have been learning python recently and decided to learn more about pyautogui, what you see down below is a macro of mine. My question is simple; is there a way to let this macro run in a specific window. For example: I want this macro to run in google chrome while I am in discord chatting with my friends (text channel so I'm not in the google chrome window). (Ignore my sloppy method of writing code)
import pyautogui
import random
import time
import mouse
#############################
tijd = 0
actief = 0
float (actief)
#############################
while not mouse.is_pressed('right'):
time.sleep(0.01)
bank_x1, bank_y1 = pyautogui.position()
time.sleep (0.5)
while not mouse.is_pressed('right'):
time.sleep(0.01)
bank_x2, bank_y2 = pyautogui.position()
print ("{} {} {} {}".format(bank_x1,bank_x2,bank_y1,bank_y2))
#############################
lijst = [[bank_x1,bank_x2,bank_y1,bank_y2,200,243],[1203,1236,721,749,23,49],[390,422,112,140,22,46]]
while not mouse.is_pressed('middle') or actief > tijd:
for i in range(0, 4):
x = random.randint(lijst[i][0], lijst[i][1])
y = random.randint(lijst[i][2], lijst[i][3])
pyautogui.moveTo(x, y)
wacht = random.randint(lijst[i][4], lijst[i][5]) / 100
time.sleep(wacht)
str (actief_str)
pyautogui.click()
pyautogui.press('esc')
No. Not with pyautogui.
Pyautogui simulates actual keyboard and mouse input to the system, not to a specific window. Thus, it will always act exactly the same as if you were pressing the keys and clicking the mouse yourself. So, no. You cannot use it to send keystrokes to background applications. The keystrokes will always effect the window that is focused, just as they do when you physically input them with a keyboard or mouse.
This question shows how you could achieve it with winapi, but it is much more complicated and less user-friendly than pyautogui.

Prevent MouseClicks from focussing pygame/SDL window [duplicate]

I'm making a pseudo transparent window in pygame with the intent of displaying varied info like a "HUD"
The script uses PIL to grab an image of the desktop and use it as the background of the window.
A simple version:
import pygame as py
from ctypes import windll
import ImageGrab, Image
SetWindowPos = windll.user32.SetWindowPos
py.init()
def get_image():
im = ImageGrab.grab((0,0,window_x,window_y))
mode = im.mode
size = im.size
data = im.tobytes()
im = py.image.fromstring(data,size,mode)
return im
window_x = 1920
window_y = 100
background = py.Surface((window_x,window_y))
background.blit(get_image(),(0,0))
window_pos = (0,0)
screen = py.display.set_mode((window_x,window_y),py.HWSURFACE|py.NOFRAME)
SetWindowPos(py.display.get_wm_info()['window'],-1,0,0,0,0,0x0001)
clock = py.time.Clock()
done = False
while not done:
for event in py.event.get():
if event.type == py.QUIT:
done = True
screen.blit(background,(0,0))
py.display.flip()
clock.tick(30)
py.quit()
This creates a Pygame window at the top of the screen.
My problem is that the Pygame window blocks any mouse interaction with anything beneath it.
Is there a way to allow mouse events to be ignored and go 'through' the window, like for example clicking on a desktop icon, underneath a Pygame window.
You will need to do a bit of an extra hacking which is outside what PyGame gives you. It should be possible to render the PyGame canvas into another windowing framework in Python and try to use advanced features of that library to achieve this.
In Windows
One example is wxWidgets. As described in this thread, which sounds quite similar to what you are trying to achieve, the user has setup a window which can be clicked through and is transparent.
Also see this another Stackoverflow Post which mentions how to handle the Hit Testing in C#. Posting code from that post here:
protected override void WndProc(ref Message m)
{
if (m.Msg == (int)WM_NCHITTEST)
m.Result = (IntPtr)HTTRANSPARENT;
else
base.WndProc(ref m);
}
It is possible to do similar testing in Python using win32 APIs. This is a piece of Python code that does exactly this. Locate the part where the programmer sets up the callback for the event (something like win32con.WM_NCHITTEST: self.onChi, where self.onChi is the callback).
I hope this gives you a starting point. I doubt there is anything readymade that you will find out of the box but these should give you some pointers on what to look for.
This is an old question, but I have ran into it quite a few times and only now got it to work right. Here is the relevant bit.
import win32api
import win32con
import win32gui
fuchsia = (255, 0, 128) # Transparency color
hwnd = pygame.display.get_wm_info()["window"] # Handle
styles = win32gui.GetWindowLong(hwnd, win32con.GWL_EXSTYLE)
styles = win32con.WS_EX_LAYERED | win32con.WS_EX_TRANSPARENT
win32gui.SetWindowLong(hwnd, win32con.GWL_EXSTYLE, styles)
win32gui.SetLayeredWindowAttributes(hwnd, win32api.RGB(*fuchsia), 0, win32con.LWA_COLORKEY)

Detecting a smaller image (.png) within a larger image retrieved with ImageGrab?

I've spent the past hour researching this simple topic, but all of the answers I've come across have been very complex and, as a noob to Python, I've been unable to incorporate any of them into my program.
I am trying to make an AI play a browser version of the Piano Tiles Game. As of now, I'm simply trying to take a capture of the games window (a small portion of my computer screen), and then check that games window with a .png of the game's start button. From there I will go on to CLICK that start button, but that's a problem for another time.
How can I check to see if a Image contains a .png file?
Here is my current code:
from PIL import ImageGrab as ig, ImageOps as io, Image
import pyautogui
import bbox
def grabStart(window):
#The start button
start = Image.open("res/StartButton.PNG")
start = io.grayscale(start)
#This is the part I need to figure out. The following is just pseudocode
if window.contains(start): #I know that this doesn't actually work. Just pseudocode
#I'd like to return the location of 'start' in one of the following forms
return either: (x1, y1, x2, y2), (x1, y1, width, height), (a coordinate within 'start'))
def grabGame():
#The coordinates of the entire game window
x1 = 2222
y1 = 320
x2 = 2850
y2 = 1105
#The entire screen of the game
window = ig.grab(bbox = (x1, y1, x2, y2))
window = io.grayscale(window)
return window
grabStart(grabGame())
Try using pyautogui.locate(). Function takes in input two parameter's, the first is the image which needs to be found, the second one is the image in which the smaller image needs to be found. This method only works for images, so if you want to run this for a live window, you might consider another option. Secondly pyautogui is just a wrapper over PIL so if you run into efficiency issues, you might wanna translate the locate() into its PIL equivalent for performance.
Here's a way of doing it. I just leave the program running and open/close and move a preview of the button around the screen seeing if it spots the button and reports the coordinates correctly.
#!/usr/bin/env python3
from PIL import ImageGrab as ig, Image
import pyautogui as ag
def checkButton(button, window):
try:
location = ag.locate(button, window, confidence=0.8)
print(f'location: {location[0]},{location[1]},{location[2]},{location[3]}')
except:
print('Not found')
# Load button just once at startup
button = Image.open("button.png")
# Loop, looking for button
while True:
window = ig.grab()
checkButton(button, window)

Pygame: allow clicks to go through the window

I'm making a pseudo transparent window in pygame with the intent of displaying varied info like a "HUD"
The script uses PIL to grab an image of the desktop and use it as the background of the window.
A simple version:
import pygame as py
from ctypes import windll
import ImageGrab, Image
SetWindowPos = windll.user32.SetWindowPos
py.init()
def get_image():
im = ImageGrab.grab((0,0,window_x,window_y))
mode = im.mode
size = im.size
data = im.tobytes()
im = py.image.fromstring(data,size,mode)
return im
window_x = 1920
window_y = 100
background = py.Surface((window_x,window_y))
background.blit(get_image(),(0,0))
window_pos = (0,0)
screen = py.display.set_mode((window_x,window_y),py.HWSURFACE|py.NOFRAME)
SetWindowPos(py.display.get_wm_info()['window'],-1,0,0,0,0,0x0001)
clock = py.time.Clock()
done = False
while not done:
for event in py.event.get():
if event.type == py.QUIT:
done = True
screen.blit(background,(0,0))
py.display.flip()
clock.tick(30)
py.quit()
This creates a Pygame window at the top of the screen.
My problem is that the Pygame window blocks any mouse interaction with anything beneath it.
Is there a way to allow mouse events to be ignored and go 'through' the window, like for example clicking on a desktop icon, underneath a Pygame window.
You will need to do a bit of an extra hacking which is outside what PyGame gives you. It should be possible to render the PyGame canvas into another windowing framework in Python and try to use advanced features of that library to achieve this.
In Windows
One example is wxWidgets. As described in this thread, which sounds quite similar to what you are trying to achieve, the user has setup a window which can be clicked through and is transparent.
Also see this another Stackoverflow Post which mentions how to handle the Hit Testing in C#. Posting code from that post here:
protected override void WndProc(ref Message m)
{
if (m.Msg == (int)WM_NCHITTEST)
m.Result = (IntPtr)HTTRANSPARENT;
else
base.WndProc(ref m);
}
It is possible to do similar testing in Python using win32 APIs. This is a piece of Python code that does exactly this. Locate the part where the programmer sets up the callback for the event (something like win32con.WM_NCHITTEST: self.onChi, where self.onChi is the callback).
I hope this gives you a starting point. I doubt there is anything readymade that you will find out of the box but these should give you some pointers on what to look for.
This is an old question, but I have ran into it quite a few times and only now got it to work right. Here is the relevant bit.
import win32api
import win32con
import win32gui
fuchsia = (255, 0, 128) # Transparency color
hwnd = pygame.display.get_wm_info()["window"] # Handle
styles = win32gui.GetWindowLong(hwnd, win32con.GWL_EXSTYLE)
styles = win32con.WS_EX_LAYERED | win32con.WS_EX_TRANSPARENT
win32gui.SetWindowLong(hwnd, win32con.GWL_EXSTYLE, styles)
win32gui.SetLayeredWindowAttributes(hwnd, win32api.RGB(*fuchsia), 0, win32con.LWA_COLORKEY)

Categories