I have written a web scraper for a mate to save him time at work. It is written in Python, using Selenium and opening a Firefox browser.
I have written this code myself on a Linux machine I use PyVirtualDisplay so Firefox doesn't actually open and disturb my work.
How can I make it run within a virtual display on a Windows PC?
The reason you can not run PyVirtualDisplay on Windows is that PyVirtualDisplay uses Xvfb as it's display and Xvfb is a headless display server for the X Window System, Windows does not use the X Window System.
not recommended
So... what you can do if you insist on working with PyVirtualDisplay is to change the Display(visible=True) Or you can set the backend as is shown in the API here.
My recommendation
Don't use PyVirtualDisplay you can use any webdriver such as Chrome driver and just add ChromeOptions with --headless.
Or in your case you use firefox so it would look something like:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(firefox_options=options, executable_path="C:\\Utility\\BrowserDrivers\\geckodriver.exe")
print("Firefox Headless Browser Invoked")
driver.get('http://google.com/')
driver.quit()
For more updated info just have a look here.
Hope this helps you!
Related
I am trying to run selenium based scraping service on ubuntu but chrome driver does not load page. It is stuck on loading, however when I open new tab manually, it loads the website? I cannot figure out the exact issue because it does not throw any error.
Following is the output I get,
Following is the code I am running.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
ch_ser=Service(executable_path='/usr/bin/chromedriver')
chrome_options = Options()
chrome_options.add_argument('--no-sandbox')
##chrome_options.add_argument('--headless')
chrome_options.add_argument('--disable-extensions')
driver = webdriver.Chrome(service=ch_ser,options=chrome_options)
#print("Opening page."
driver.get('https://google.com')
I had the same problem on my Raspberry pi 4 with XFCE (xubuntu 64bit) linux.
The problem is probably in Chromedriver.
I solved this problem by replacing Chromium with Firefox.
If you are on Linux too, try to install geckodriver for Firefox and use it instead of Chrome.
sudo apt install firefox-geckodriver
I want to write a Python script using Selenium and Chrome where Selenium won't close the Chrome browser when the script finishes. From doing a bunch of googling, it looks like the standard solution is to use the detach option. But when I run the following script:
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://www.google.com/")
It opens up Chrome, goes to Google's homepage, and then closes the browser. It's not throwing any errors.
Any idea why it's not working? I'm using the latest version of Google Chrome on Windows 10, and I've got the latest version of the selenium module installed. I couldn't find anything online that said the experimental detach option no longer existed. And I double checked the API, and it looks like it's the right syntax.
I discovered another way to go: start Chrome in remote debugging mode, then connect to it. That way, not only does the browser stay open, but you can also use your existing Chrome profile so you can take advantage of any sites your cookies allow you to access without having to log in every time you run the script.
Here's what you need to do if you're on Windows 10:
Start Google Chrome up remotely, pointed towards your existing user profile and the port you want to use:
cd "C:\Program Files (x86)\Google\Chrome\Application"
chrome.exe -remote-debugging-port=9014 --user-data-dir="%LOCALAPPDATA%\Google\Chrome\User Data"
In your python script, connect to the local port that this version of Chrome is running on:
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("debuggerAddress", "localhost:9014")
driver = webdriver.Chrome(options=chrome_options)
driver.get("https://github.com/")
This code worked perfectly for me using : selenium-wire , hope it works for you
from seleniumwire import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
driver = webdriver.Chrome(ChromeDriverManager().install(),options=chrome_options)
driver.get("https://www.google.com/")
Does anyone know how to force pyautogui or python in general to recognize a set screen size and take in display data, even if no display is connected?
I have a task that uses pyautogui, and selenium chrome driver, which both require a display, or they fail.
It runs on a server, so the start of the program requires my laptop to remote desktop into the server, allowing it to have a display, which allows launching a page with chromedriver, and pyautogui click components / screen search to work.
The issue arises that should my home network ever be down, it cannot kick off the remote desktop, and therefore my automation would fail.
My solution would be to emulate or force the program to behave as if a display existed, so it can just be run server side.
All of my servers are windows, so XVFB does not seem to be an option based on
Xvfb on Windows
Well I am using something similar on regular basis,
I am using Windows server to run my automated python script which uses selenium webdriver.
so, first your answer is you need to use code for screen size in the script and you can run that script with Windows task scheduler so, you don't have to touch your laptop or desktop to run remote desktop.
you can use chrome options if you are using chrome as headless browser.
I also, suggest if you are using server which is controlled by some third party who provides regular updates then, you can use chromedriver_autoinstaller package to take updated or supported version chrome driver according to your current version of chrome.
code:
import chromedriver_autoinstaller
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep
from selenium.webdriver.chrome.options import Options
chromedriver_autoinstaller.install()
#maximize the chrome
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('website address')
For automating task Windows task scheduler is the best option. you can refer this documentation or find something according to your need.
https://towardsdatascience.com/automate-your-python-scripts-with-task-scheduler-661d0a40b279
Note: If you just need to set screen size then your answer start here #maximize the chrome
You should emulate your display driver. Run:
xvfb-run --server-args="-screen 0 1024x768x24" python my_script.py
to launch your script instead of just python my_script.py.
I know the question has been a little while...
I just found out this using PyAutoGUI on remote machines or headless machines
at here https://github.com/asweigart/pyautogui/issues/133
Can look at this guys workaround: http://fredtantini.free.fr/blog/index.php?article58/automatiser-des-actions-avec-selenium-pyautogui-et-xvfb
Thank you
I'm attempting to run a Headless selenium session via Chrome Web Driver in python.
The code I have here should be able to run it headless mode, but it runs headed and I don't get any error codes. I'm running Windows 10, Python 3.7.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
driver = webdriver.Chrome(options=options)
options.add_argument("--headless")
options.addArguments("headless");
Try without -- before headless and see if it works.
You may use this as well:
options.headless = True
im trying to load a specific chrome profile using Python selenium webdriver, but i cant interact with the driver after assigning the chrome profile. it opens the chrome profile that i wanted, but from there - nothing. i cant do any action. for example - im trying to open Microsoft.com:
This works:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('http://www.microsoft.com')
But this doesnt work at all:
from selenium import webdriver
import getpass
username = getpass.getuser()
options = webdriver.ChromeOptions()
options.add_argument('user-data-dir=C:/Users/'+username+'/AppData/Local/Google/Chrome/User Data/')
driver = webdriver.Chrome(executable_path='C:/Users/'+username+'/Documents/selProject/chromedriver.exe', chrome_options=options)
driver.get('http://www.microsoft.com')
The above code opens chrome, but doesnt go to microsoft.com or any other action.
thanks for reading!
This is likely because you have Chrome open already with that user signed in.
In order to use chrome with that profile, while also running the script you'll need to separate the directories where the profiles are pulled from. That is, move (or copy) the Default profile to another directory that you call to within the user-data-dir argument.
You do following things and check whether it works.
Upgrade python bindings using
pip install -U selenium
for chrome browser download the latest chrome driver "ChromeDriver 2.45"
form http://chromedriver.chromium.org/downloads and write the code
from selenium import webdriver
driver=webdriver.Chrome("Path of the Chromedriver" + "chromedriver.exe" )
driver.get('http://www.microsoft.com')