I followed setting up chrome requirements for selenium.webdriver.Chrome, and I used the following code Running webdriver chrome with Selenium:
import os
from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
driver = webdriver.Chrome()
driver.get("http://www.google.com")
print driver.page_source.encode('utf-8')
I was very surprised to see that the code works, but no browser pops up. I would think this is almost impossible to develop, as I can't see what I'm doing right/wrong. How do I use selenium to actually see a functional Chrome driver (python)? Thank you
ANSWER:
set visible to 1, install emulator:
sudo apt-get install xvfb xserver-xephyr
display = Display(visible=1, size=(800, 600))
In java we add these two lines for executing code in chrome
System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
WebDriver driver = new ChromeDriver();
not sure if there something similar for python as well!
Related
There are about 100 posts about the same issue but none of them seem to work for me, hence asking again. I'm trying to launch a Firefox browser using Python and Selenium and I get the following error:
WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.
I tried each and every answer on the web but nothing seems to work.
This is my code:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
caps = DesiredCapabilities.FIREFOX
caps["marionette"] = False
binary = FirefoxBinary('d:\\Desktop\\IEDriver\\geckodriver.exe')
options = Options()
options.set_headless(headless=True)
driver = webdriver.Firefox(firefox_binary=binary, firefox_options=options, executable_path=r'd:\\Desktop\\IEDriver\\geckodriver.exe')
driver.get("http://google.com/")
print ("Headless Firefox Initialized")
driver.quit()
If I set caps["marionette"] = True then the error I get is
SessionNotCreatedException: Message: Unable to find a matching set of capabilities
Versions of software I'm running:
Firefox: 62.0 (64 bit)
Selenium: 3.14.0
Gecko: 0.21.0
Python: 3
OS: Windows 8.1 64 bit
Any help would be highly appreciated.
EDIT: I've uninstalled and re-installed Firefox but didn't work. Also tried installing Firefox 61.0.2, still no luck.
This error message...
WebDriverException: Message: The browser appears to have exited before we could connect.
If you specified a log_file in the FirefoxBinary constructor, check it for details.
...implies that the GeckoDriver was unable to initiate/spawn a new WebBrowser i.e. Firefox Browser session.
You need to take care of a couple of things as follows:
To set the FirefoxBinary you need to use the FirefoxOptions() and instead of passing the absolute path of geckodriver binary, you have to pass the absolute path of the desired firefox binary.
As you are using GeckoDriver v0.21.0 you have to mandatorily use marionette so either keep it unchanged (by default true) or set marionette to true.
Your own code with incorporating the minor changes will be:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
binary = r'C:\Program Files\Mozilla Firefox\firefox.exe'
options = Options()
options.set_headless(headless=True)
options.binary = binary
cap = DesiredCapabilities().FIREFOX
cap["marionette"] = True #optional
driver = webdriver.Firefox(firefox_options=options, capabilities=cap, executable_path="C:\\Utility\\BrowserDrivers\\geckodriver.exe")
driver.get("http://google.com/")
print ("Headless Firefox Initialized")
driver.quit()
Console Output:
Headless Firefox Initialized
Here you can find a detailed discussion on Unable to find a matching set of capabilities with selenium 3.4.3, firefox 54.0 and gecko driver 0.17
Make sure (especially on Windows (Win 10)) that your browser and controller (python/C/java/perl/etc) is all either x64 or win32, Microsoft will not thunk between them anymore.
So, if your trying to control a 64 bit browser (what will be downloaded by default from firefox) from a x32 bit python, it will exit before you can connect.. go and install a win32 version of firefox for the magic to happen
After trying almost all of the answers on different forums, a simple self trial resolved the problem and i.e. you need to have python, firefox browser and geckodriver in either 62 bit or 32 bit. Mismatch in this caused the problem in my case.
After ensuring that you are using the same bit version for all the 3 components, just use following lines to run firefox:
ffPath = "C:\\Drivers\\geckodriver.exe"
os.environ["webdriver.firefox.driver"] = ffPath
driver = webdriver.Firefox(executable_path=ffPath)
driver.get(url)
The issue for me was the mis-match version between python and gekodriver . once all the involved party were 64 bit it worked like a charm
This error has troubled me a lot.
Install this:
pip install -U selenium
binary = r'C:\Program Files\Mozilla Firefox\firefox.exe'
options = Options()
options.binary = binary
browser = webdriver.Firefox(firefox_options=options, executable_path=r"C:\Drivers\geckodriver.exe")
i use selenium for instagram appicatin.
i wonder there is any way to run selenium without browser opening in windows?
there is some solution in linux like install Xvfb but i don't see any solution for windows!
Do you mean that --headless?
You can do some operations without opening Chrome if you have chrome-driver.
Here is how to set headless:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chromeoption = Options()
chromeoption.add_argument('--headless')
browser = webdriver.Chrome(options=chromeoption)
# Then it is your work.
browser.get(url="xxx")
So I'm trying some stuff out with selenium and I really want it to be quick.
So my thought is that running it with headless chrome would make my script faster.
First is that assumption correct, or does it not matter if i run my script with a headless driver?
Anyways I still want to get it to work to run headless, but I somehow can't, I tried different things and most suggested that it would work as said here in the October update
How to configure ChromeDriver to initiate Chrome browser in Headless mode through Selenium?
But when I try that, I get weird console output and it still doesn't seem to work.
Any tipps appreciated.
To run chrome-headless just add --headless via chrome_options.add_argument, i.e.:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
#chrome_options.add_argument("--disable-extensions")
#chrome_options.add_argument("--disable-gpu")
#chrome_options.add_argument("--no-sandbox") # linux only
chrome_options.add_argument("--headless")
# chrome_options.headless = True # also works
driver = webdriver.Chrome(options=chrome_options)
start_url = "https://duckgo.com"
driver.get(start_url)
print(driver.page_source.encode("utf-8"))
# b'<!DOCTYPE html><html xmlns="http://www....
driver.quit()
So my thought is that running it with headless chrome would make my
script faster.
Try using chrome options like --disable-extensions or --disable-gpu and benchmark it, but I wouldn't count with much improvement.
References: headless-chrome
Install & run containerized Chrome:
docker pull selenium/standalone-chrome
docker run --rm -d -p 4444:4444 --shm-size=2g selenium/standalone-chrome
Connect using webdriver.Remote:
driver = webdriver.Remote('http://localhost:4444/wd/hub', webdriver.DesiredCapabilities.CHROME)
driver.set_window_size(1280, 1024)
driver.get('https://www.google.com')
from time import sleep
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path="./chromedriver", options=chrome_options)
url = "https://stackoverflow.com/questions/53657215/running-selenium-with-headless-chrome-webdriver"
driver.get(url)
sleep(5)
h1 = driver.find_element_by_xpath("//h1[#itemprop='name']").text
print(h1)
Then I run script on our local machine
➜ python script.py
Running Selenium with Headless Chrome Webdriver
It is working and it is with headless Chrome.
If you are using Linux environment, may be you have to add --no-sandbox as well and also specific window size settings. The --no-sandbox flag is no needed on Windows if you set user container properly.
Use --disable-gpu only on Windows. Other platforms no longer require it. The --disable-gpu flag is a temporary work around for a few bugs.
//Headless chrome browser and configure
WebDriverManager.chromedriver().setup();
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.addArguments("--no-sandbox");
chromeOptions.addArguments("--headless");
chromeOptions.addArguments("disable-gpu");
// chromeOptions.addArguments("window-size=1400,2100"); // Linux should be activate
driver = new ChromeDriver(chromeOptions);
Once you have selenium and web driver installed. Below worked for me with headless Chrome on linux cluster :
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("--disable-extensions")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--no-sandbox")
options.add_experimental_option("prefs",{"download.default_directory":"/databricks/driver"})
driver = webdriver.Chrome(chrome_options=options)
Todo (tested on headless server Debian Linux 9.4):
Do this:
# install chrome
curl -sS -o - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list
apt-get -y update
apt-get -y install google-chrome-stable
# install chrome driver
wget https://chromedriver.storage.googleapis.com/77.0.3865.40/chromedriver_linux64.zip
unzip chromedriver_linux64.zip
mv chromedriver /usr/bin/chromedriver
chown root:root /usr/bin/chromedriver
chmod +x /usr/bin/chromedriver
Install selenium:
pip install selenium
and run this Python code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("no-sandbox")
options.add_argument("headless")
options.add_argument("start-maximized")
options.add_argument("window-size=1900,1080");
driver = webdriver.Chrome(chrome_options=options, executable_path="/usr/bin/chromedriver")
driver.get("https://www.example.com")
html = driver.page_source
print(html)
As stated by the accepted answer:
options.add_argument("--headless")
These tips might help to speed things up especially for headless:
There are quite a few things you can do in headless that you cant do in non headless
Since you will be using Chrome Headless, I've found adding this reduces the CPU usage by about 20% for me (I found this to be a CPU and memory hog when looking at htop)
--disable-crash-reporter
This will only disable when you are running in headless This might speed things up for you!!!
My settings are currently as follows and I reduce the CPU (but only a marginal time saving) by about 20%:
options.add_argument("--no-sandbox");
options.add_argument("--disable-dev-shm-usage");
options.add_argument("--disable-renderer-backgrounding");
options.add_argument("--disable-background-timer-throttling");
options.add_argument("--disable-backgrounding-occluded-windows");
options.add_argument("--disable-client-side-phishing-detection");
options.add_argument("--disable-crash-reporter");
options.add_argument("--disable-oopr-debug-crash-dump");
options.add_argument("--no-crash-upload");
options.add_argument("--disable-gpu");
options.add_argument("--disable-extensions");
options.add_argument("--disable-low-res-tiling");
options.add_argument("--log-level=3");
options.add_argument("--silent");
I found this to be a pretty good list (full list I think) of command line switches with explanations: https://peter.sh/experiments/chromium-command-line-switches/
Some additional things you can turn off are also mentioned here: https://github.com/GoogleChrome/chrome-launcher/blob/main/docs/chrome-flags-for-tools.md
I hope this helps someone
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=r"C:\Program
Files\Google\Chrome\Application\chromedriver.exe", options=chrome_options)
This is ok for me.
Okay, so first I learned that Xvfb wasn't included with my OS X version, so I installed it from http://www.xquartz.org/.
and that seemed to have worked:
which xvfb
/opt/X11/bin/xvfb
But when I try using it with either pyvirtualdisplay and xvfbwrapper, following advice I found on this question How do I run Selenium in Xvfb? My script runs without errors but just opens in a Chrome browser window:
from selenium import webdriver
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
browser = webdriver.Chrome()
browser.get('google.com')
Am I doing something wrong here?
I believe that Chrome is built for Quartz ui framework, so it ignores the X11 windowing engine. You will need to install an X11 version of a browser and then execute that.
For me, this code works fine on OSX 10.13. You don't need pyvirtualdisplay, because you can run chrome in headless mode. Just download chromedriver that fits to your chrome version and put it to usr/local/bin
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--mute-audio')
options.add_argument('--lang=de')
options.add_argument('--window-size=800,600')
options.add_argument('--disable-notifications')
options.add_argument('--enable-popup-blocking')
browser = webdriver.Chrome(chrome_options=options, executable_path='/usr/local/bin/chromedriver')
browser.get('some url')
I've used Firefox with Webdriver on this machine before, but suddenly when I use any code that utilizes Firefox as the driver, Firefox will open and just sit there blankly. Absolutely nothing loads, my program hangs until I close the window at which point it raises the "browser not open" error.
Tried reinstalling. Chrome/IE Drivers work but I don't really want to use them.
Included a pic of exactly what happens when I use any code, for instance
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
url = "http://google.com"
driver.get(url)
Even this gets me nothing.
upgrade to selenium version 2.29.0
pip install -U selenium
there was a bug fix in the previous version