I need to run GUI application on remote windows host and then do some actions with mouse and keyboard. This should be done from local pc without opening any GUI application (for example "Remote Desktop Connection").
So I have a python script on remote server which does all the actions I need (tested on local pc) and I run the script via psexec which successfully opens the GUI application on server.
The problem is that when python tries to programatically move and click mouse it throws an exception because there is no screen.
Actions with keyboard (Ctrl-A, Ctrl-C, Ctrl-V) can be done successfully.
Here is the code that I use to simulate mouse click but as I mentioned it doesn't work on server because there is no actual screen.
win32api.SetCursorPos((x,y))
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN,x,y,0,0)
win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP,x,y,0,0)
Can anyone suggest me a solution?
Try connecting directly to user32.dll through ctypes and use mouse controlling functions from there. Perhaps win32types problem is that it is mostly statically linked to MS API. So it is programmed for desktop environment.
Also, you can try pymouse module that you can find on pypi.python.org, of course, which does the same as I suggested above.
Related
We have hosted our Jenkins server on an RHEL 7.9. We want to connect to the remote windows server from the RHEL box and execute a python program that uses pywinauto package to manipulate a GUI application that runs on the remote windows server for functional tests. Our python script that uses pywinauto is ready and working fine when executed from cmdline on the remote windows server and want to trigger the script from a Jenkins job. Now, Can someone here please let me know the options I should think of to connect to the Remote windows server from RHEL and execute a python script that does GUI automation?
Holla, I have done this recently from centOS 8, This was quite challenging though. Since pywinauto requires active desktop to perform actions on the UI. For this purpose we can not do SSH or similar ways.
Here is how to...step by step.
You can use to do the RDP connection to the windows server using xFreeRDP from RHEL.
xfreerdp /u:username /p:password /v:hostname
Ref.- FreeRDP
Up to this all works smooth here onwards you can not execute the commands on RDP session to trigger the execution.
But windows server is your savior. It does provide you the EVENTCREATE command which allows you to create event. It will create event on your behalf in your windows server. This command will allow you flexibility for scheduling your job. The events created by this command will be listed in the event-viewer utility of windows. To see the window event created by you, give meaningful message. To create event user should be administrator.
Example -
eventcreate /t INFORMATION /id 1000 /d "Create event in WinMgmt source"
In this step you will have to setup/schedule the job based on event occurred on windows. By right clicking on event and then select option 'Attach task to this event...' or by directly going in to the Task Schedular (another windows utility tool).
a. Open Task Schedular.
b. Right click on root tree node - "Task Schedular Library"
c. Select the option "Create Task..."
d. Popup will open to create Task Fill Name and description of task.
e. Select the checkbox "Run with highest privileges".
f. Go to trigger tab and New button.
g. Select the first dropdown value for "Begin the task" as "On an Event".
i. In the setting section of popup select log type of event.
ii. Source and Event ID values should be similar that you will use in your custom event(step 2 event).
iii. Select other scheduling setting as per your need.
i. Go to Actions tab and click on New button.
j. On the New Action popup, select Action as "Start a program"
k. Select your program (python script) to be executed and parameters if any in setting section.
l. Click OK and test this connection of event and task by firing event from cmd (step 2)
This is almost done now except the one critical part. And that is how to fire eventcreate command from RHEL. You can do that by writing small python code using SSH or paramiko packages.
NOTE's
Opening multiple RDP terminals is possible using XfreeRDP.
Multiple xfreeRDP windows will execute the scripts in parallel without any hassle.
I made a python script that runs a graphical automation using pyautogui (mouse movements) over a huge number of PDFs.
The automation appears to need an active display, for the mouse movements and the PDF to be opened.
If I connect to the Azure VM (with Windows OS) with SSH and start the python script, I get an error from pyautogui as below:
pyautogui.FailSafeException:
PyAutoGUI fail-safe triggered from mouse moving to a corner of the screen.
To disable this fail-safe, set pyautogui.FAILSAFE to False.
DISABLING FAIL-SAFE IS NOT RECOMMENDED.
I have tried with the failsafe disable and still it doesn't work.
As I have read, this happens because there is no active display opened.
If I connect to the VM using RDP, the automation starts and works as expect until I minimize or close the window. When I do that I get the same failsafe error from pyautogui.
But I cannot keep the window open, because I would need to start the same automation on 16 more VMs.
Is there a way to run such graphical automations in Azure VMs or any other similar solution? Docker maybe?
Is there any solution to run or host VM machines with permanent active display opened? Is something like this possible?
I ended up using Virtual Box.
I first created one VM with the needed software and files.
You can start the VM in headless mode. Then show the VM in order to log in to Windows and start the automation script. Then you can close the VM window and let it run in the background. The graphical automation will still run ok in that case. PyAutoGUI won't crash as if there were no active display.
I cloned the original VM 16 times and split the work evenly between the VMs.
Using ssh I am able to connect to each VM and monitor the status of the automations.
It would be great if there were a solution similar to kubernetes that could help with the deployment of such automations and with the monitoring of the status, but until then this is ok.
The automation using pywinauto is working fine in the Remote Desktop Applications with mouse/keyboard and send_keys events.
But I need to track the events generated from the application running on remote desktop to build conditional use case.
For Example, Let's say, I have an excel file opened in the remote desktop and using the pywinauto and send_keys function I can able to write some text in some predefined cell (using mouse co-ordinates) of that file.
But when I'm clicking the exit button in excel it will pop up an window asking if I want to save the excel before closing. I need to catch that event in pywinauto.
pywinauto is running on my local and all the tasks are going to happen on remote desktop (win32).
Note: I don't want to tackle this situation with mouse click on predefined co-ordinates because in-real, based the popup window of my application, I need to do some other tasks.
Remote desktop doesn't pass any GUI elements info to local machine. So you have to run pywinauto script on a remote machine. All necessary recipes are here: Remote Execution Guide.
I have been facing issue with automation execution of my script on one of the VM. I have automated the functionality of Saving a Document which is ideally a Windows Designed UI. I have tried using various technologies/tools like AutoIT, Python, Sikuli but the script halts if VM is minimized. It works perfectly fine is VM is open via RDP and I can see runtime execution. But If I minimize the RDP, the script halts at 'Save As' dialog box, none of the send keys (Cntrl+s) or (Enter) work via AutoIt script. Please help with some solution so as to have successfully execution of script even in minimized mode.
The reason why your script fails when it gets executed over a minimized RDP session is quite simple. GUI automation/testing tools need to have an unlocked, active desktop - otherwise the operation system thinks that it doesn't need to actually render GUI operations (which is time consuming) since there no user can that can see the rendered graphical user interface anyway. And programs don't communicate via GUIs normally ...
This is why QF-Test and other GUI automation/testing tools often have a note in their FAQs describing this kind of problem. For example FAQ 14 in the case of QF-Test, see https://www.qfs.de/qf-test-handbuch/lc/manual-en-faq.html
As described in the FAQ 14 on Windows 10 or Windows Server 2016 and in case of an RDP connection you need to modify the Registry. Go to
HKEY_CURRENT_USER\Software\Microsoft\Terminal Server Client
and add a new value
RemoteDesktop_SuppressWhenMinimized as DWORD having the value 2
After restarting you will then be able to minimize the RDP connections. However disconnecing or closing the RDP connection will probably still result in a failure.
You could try running tscon.exe RDP-Tcp#0 /dest:console as admin as mentioned here. This will disconnect your RDP connection but should leave all GUI programs running normally on the VM. I have personally used this with autoit on a VM and it worked OK. Of course you will not be able to monitor your script as it runs so this may or may not work for you.
I have very similar problem to this one : SetForegroundWindow in Remote Desktop Connection
Everything works when I am connected and watching the RDC, but when I'm not.. Nothing happens.
I am using python and pywinauto, trying to use SendKeys method : SetForegroundWindow returns 0, the same as GetLastError after that, so I have no idea what might cause the trouble.
Edit: I also tried other methods like BringWindowToTop, or SetActiveWindow, also I tried to sent alt key before changing windows - nothing worked.
If your pywinauto script works on remote machine, it cannot manage RDP window at all because RDP window is on your local machine.
To prevent lost of GUI context in RDP you do NOT need to minimize RDP window locally. RDP can lose focus safely, but minimizing leads to stop of any GUI related activity.
It's correct for any GUI automation, not pywinauto only. If you have a lot of test machines, the best way is having 1 master and many slaves. Master host can initiate and keep non-minimized remote sessions, slaves are running GUI automation scripts.