How to setup Selenium with headless Firefox on Raspbian Buster - python

I have gotten this to work in Jessie/Stretch with previous Firefox versions but can't seem to get it to work with Raspberry Pi 3B or 4 running Raspbian Buster and the latest Selenium, 3.141.0 in Python.
Since iceweasel is no longer available I install firefox-esr via apt and get the latest ARM geckodriver, which is currently 0.23.0, from https://github.com/mozilla/geckodriver/releases then run the code below:
from selenium import webdriver
from pyvirtualdisplay import Display
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
capabilities = webdriver.DesiredCapabilities().FIREFOX
capabilities["marionette"] = False
driver = webdriver.Firefox(options=options, executable_path='/usr/bin/geckodriver', capabilities=capabilities)
driver.get('https://www.google.com')
print(driver.page_source)
driver.close()
display.stop()
This returns the error:
Message: Can't load the profile. Possible firefox version mismatch. You must use GeckoDriver instead for Firefox 48+. Profile Dir: /tmp/tmpxMUUcF If you specified a log_file in the FirefoxBinary constructor, check it for details.
Traceback (most recent call last):
File "SeleniumDebugExample.py", line 50, in <module>
driver = webdriver.Firefox(options=options, executable_path='/srv/main/geckodriver', capabilities=capabilities)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 191, in __init__
self.binary, timeout)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 52, in __init__
self.binary.launch_browser(self.profile, timeout=timeout)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 73, in launch_browser
self._wait_until_connectable(timeout=timeout)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 114, in _wait_until_connectable
% (self.profile.path))
WebDriverException: Message: Can't load the profile. Possible firefox version mismatch. You must use GeckoDriver instead for Firefox 48+. Profile Dir: /tmp/tmpxMUUcF If you specified a log_file in the FirefoxBinary constructor, check it for details.
I am specifying geckodriver though so I am not sure what the issue is...
If anyone has gotten headless selenium to work on Raspbian Buster and can share some could it would be greatly appreciated!

After a bunch of testing it seems the issue was with adding the marrionette=False capability... Back in Jessie with iceweasel I had needed to add that for it to work but it seems with firefox-esr that is no longer necessary... I was able to get this working with just the below:
from selenium import webdriver
from pyvirtualdisplay import Display
driver = webdriver.Firefox(executable_path='/usr/bin/geckodriver')
driver.get('https://www.google.com')
print(driver.page_source)
driver.close()
display.stop()

Related

This version of ChromeDriver only supports Chrome version 79 - Python

This error drives me crazy. My code is:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import chromedriver_binary
from selenium.webdriver.common.by import By
import time
from influxdb import InfluxDBClient
chrome_options = Options()
chrome_options.add_argument("--headless")
chromedriver_binary =
"/home/dario/scripts/cron_run/web_app_login_checker/chromedriver/chromedriver"
driver = webdriver.Chrome(chromedriver_binary, options=chrome_options)
If I do:
./chromedriver -v
ChromeDriver 79.0.3945.36 (3582db32b33893869b8c1339e8f4d9ed1816f143-refs/branch-
heads/3945#{#614})
It's the right version.
Full traceback is:
Traceback (most recent call last):
File "grafana.py", line 12, in <module>
driver = webdriver.Chrome(chromedriver_binary, options=chrome_options)
File
"/home/dario/scripts/cron_run/web_app_login_checker/lib/python3.6/site-
packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File
"/home/dario/scripts/cron_run/web_app_login_checker/lib/python3.6/site-
packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File
"/home/dario/scripts/cron_run/web_app_login_checker/lib/python3.6/site-
packages/selenium/webdriver/remote/webdriver.py", line 252, in
start_session
response = self.execute(Command.NEW_SESSION, parameters)
File
"/home/dario/scripts/cron_run/web_app_login_checker/lib/python3.6/site-
packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File
"/home/dario/scripts/cron_run/web_app_login_checker/lib/python3.6/site-
packages/selenium/webdriver/remote/errorhandler.py", line 242, in
check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session
not created: This version of ChromeDriver only supports Chrome version
79
None of the already discussed topics here on Stack Overflow helped me.
The problem is the Chrome browser version, not the ChromeDriver version. You need to update it to version 79, or downgrade the ChromeDriver. You can find here the matching versions.
If you are getting this error when you run stuffs on automated cluster and you are downloading the stable version of the google chrome every time then you can use the below shell script to download the compatible version of the chrome driver dynamically every time even if the stable version of the chrome gets updated. You could do something similar to this python to make it work.
%sh
#downloading compatible chrome driver version
#getting the current chrome browser version
**chromeVersion=$(google-chrome --product-version)**
#getting the major version value from the full version
**chromeMajorVersion=${chromeVersion%%.*}**
# setting the base url for getting the release url for the chrome driver
**baseDriverLatestReleaseURL=https://chromedriver.storage.googleapis.com/LATEST_RELEASE_**
#creating the latest release driver url based on the major version of the chrome
**latestDriverReleaseURL=$baseDriverLatestReleaseURL$chromeMajorVersion**
**echo $latestDriverReleaseURL**
#file name of the file that gets downloaded which would contain the full version of the chrome driver to download
**latestDriverVersionFileName="LATEST_RELEASE_"$chromeMajorVersion**
#downloading the file that would contain the full release version compatible with the major release of the chrome browser version
**wget $latestDriverReleaseURL**
#reading the file to get the version of the chrome driver that we should download
**latestFullDriverVersion=$(cat $latestDriverVersionFileName)**
**echo $latestFullDriverVersion**
#creating the final URL by passing the compatible version of the chrome driver that we should download
**finalURL="https://chromedriver.storage.googleapis.com/"$latestFullDriverVersion"/chromedriver_linux64.zip"**
**echo $finalURL**
**wget $finalURL**
I was able to get the compatible version of chrome browser and chrome driver using the above approach when running scheduled job on the databricks environment and it worked like a charm without any issues.
Hope it helps others in one way or other.
Check your chrome version
Download the same version of the chrome driver
Check the latest driver here
Update your driver in the reference folder.
In my case, I am using the windows 10 and python I've updated with the latest chromedriver.exe file (for Chrome v83)
This error message...
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 79
...implies that the ChromeDriver was unable to initiate/spawn a new Browsing Context i.e. Chrome Browser v79.x session.
Your main issue is the incompatibility between the version of the binaries you are using as follows:
You are using chromedriver=79.0.3945.36
Release Notes of chromedriver=79.0.3945.36 clearly mentions the following :
Supports Chrome version 79
Presumably you are using the latest Version 78.0.3904.108 (Official Build)
So there is a clear mismatch between the ChromeDriver v79.0.3945.36 and Chrome Browser v78.0.3904.108
Solution
There are two possible solutions:
Either you need to downgrade ChromeDriver to ChromeDriver 78.0.3904.105
Download, install and use Chrome from the Chrome Release Channels
Linux Stable channel
Linux Beta channel
Linux Development channel
latest build

Selenium webdriver crashing Firefox during unittests

How do you determine why Firefox is crashing when run from inside some Django unittests via Selenium?
My testcase is:
from django.contrib.staticfiles.testing import StaticLiveServerTestCase
from pyvirtualdisplay import Display
from selenium import webdriver
class Tests(StaticLiveServerTestCase):
def setUp(self):
super(Tests, self).setUp()
self.vdisplay = Display(visible=0, size=(1920, 1080), backend='xvfb')
self.vdisplay.start()
profile = webdriver.FirefoxProfile()
log_path = '/tmp/tests.log'
self.driver = webdriver.Firefox(profile, log_path=log_path)
def test_abc(self):
blah
When I run this on a headless server with:
python manage.py test functional_tests --nomigrations --failfast
it almost immediately errors with:
ERROR: test_abc (myproject.functional_tests.tests.Tests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/myproject/src/buildbot/worker3/myproject_runtests/build/src/myproject/functional_tests/tests.py", line 15, in setUp
self.driver = webdriver.Firefox(profile, log_path=log_path)
File "/usr/local/myproject/src/buildbot/worker3/myproject_runtests/build/.env/local/lib/python2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 158, in __init__
keep_alive=True)
File "/usr/local/myproject/src/buildbot/worker3/myproject_runtests/build/.env/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/myproject/src/buildbot/worker3/myproject_runtests/build/.env/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/myproject/src/buildbot/worker3/myproject_runtests/build/.env/local/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/usr/local/myproject/src/buildbot/worker3/myproject_runtests/build/.env/local/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: Process unexpectedly closed with status: 1
So webdriver is having trouble initializing a Firefox instance. However, the log file only shows:
1528498122788 geckodriver INFO geckodriver 0.19.0
1528498122799 geckodriver INFO Listening on 127.0.0.1:39255
1528498123950 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "-profile" "/tmp/rust_mozprofile.LrTWF7H6fk2y"
My versions are:
Ubuntu Version is 16.04
Selenium Version is 3.8.1
Geckodriver Version is 0.19.0
Firefox Version is 58.0.1
I tried upgrading to the most recent Selenium 3.12, Geckodriver 0.20.1, and Firefox 60, but I get the exact same error.
I also tried running all system package upgrades, and rebooting, but that had no effect.
How do I fix this, or at least get a better error message about why Firefox is crashing?
Edit: This is not the same as this question, which returns a very different error message. It's solution does not resolve my error.
First, maybe you should enable trace log by:
capabilities = DesiredCapabilities.FIREFOX.copy()
capabilities["moz:firefoxOptions"] = {
"log": {
"level": "trace",
},
}
driver = webdriver.Firefox(capabilities=capabilities)
Or
opts = Options()
opts.log.level = "trace"
driver = webdriver.Firefox(firefox_options=opts)
Second, as far as I think, this issue is usually casued by Graphic environment requirement. It seems if you are not running a firefox instance in a session with valid DISPLAY, it will exit with code 1.
So maybe your question can be resolved by:
options = Options()
profile = webdriver.FirefoxProfile()
options.add_argument("--headless")
driver = webdriver.Firefox(profile, firefox_options=options)
Finally, I must say I'm actually not sure about your problem as there is just limited information.

selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities with Firefox 46 through Selenium

I must have some versions here that don't match up since I can't get Selenium with Python to fire up a Firefox web browser. I'm using an older version of Firefox because other people in here have the same old version of Python and for them the old version of Firefox works best.
Code:
from selenium import webdriver
from selenium import common
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
driver=webdriver.Firefox(capabilities=DesiredCapabilities.FIREFOX)
Error:
Traceback (most recent call last):
File "scrapeCommunitySelenium.py", line 13, in <module>
driver=webdriver.Firefox(capabilities=DesiredCapabilities.FIREFOX)
File "/Library/Python/2.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 158, in __init__
keep_alive=True)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 311, in execute
self.error_handler.check_response(response)
File "/Library/Python/2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities
Version info:
Python 2.7.10
Selenium 3.8.0
Firefox 46.0
GeckoDriver 0.19.1 (It's in a folder which is in my PATH environment variable)
MacOS 10.12.6
As you are using Selenium 3.8.0 you have to use GeckoDriver as a mandatory. But again as you are using Firefox v46.0 you have to set the capability marionette as False through DesiredCapabilities() as follows :
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
cap = DesiredCapabilities().FIREFOX
cap["marionette"] = False
browser = webdriver.Firefox(capabilities=cap, executable_path="C:\\path\\to\\geckodriver.exe")
browser.get('http://google.com/')
browser.quit()
If you're going to use Geckodriver, you definitely need to use a newer version of Firefox. Frex: https://github.com/mozilla/geckodriver/releases/tag/v0.19.0 lists FF55 or greater.
If you plan on using FF46, don't use geckodriver. Update your capabilities to have marionette set to False:
caps = DesiredCapabilities.FIREFOX.copy()
caps['marionette'] = False
driver=webdriver.Firefox(capabilities=caps)
I had this issue on my MacOS 10.5 Catalina.
What I did:
1. Installed the geckodriver using brew install geckodriver
2. Deleted/uninstalled my existing(OLD) Firefox browser (v.46) and installed v70.
3. tried:
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://google.com')
The above worked fine with no errors, by launching Firefox and loading google.com
I got this error because the Firefox browser was not installed on my machine. You can download Firefox or download the Chrome driver here. If you use the Chrome drive, make sure you add it to the path (just like the geckodriver).
And the you can use it like this:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("http://www.python.org")
You can see similar error on Chrome as well. If you are seeing it on Ubuntu, the reason is probably you have a pre-installed version of Chrome and Firefox which is older. And you have downloaded the latest version of Chrome/Firefox driver.
Simple solution is:
Uninstall the existing Chrome/Firefox browser provided from Ubuntu : Go to Applications(top left corner)->Ubuntu software center-> search Chrome and uninstall it.
Install latest browser.
For Chrome, steps are as follows:
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo dpkg -i google-chrome-stable_current_amd64.deb
Done!
There are some possible reasons for that error like:
Firefox is installed in your system
Firefox access is admin only
Firefox is not installed with same name
Firefox version is not updated
This error can also come from the version 32bits, choose a x64 version to fix it.

How to launch Chrome in Selenium correctly

I can not launch Chrome in Selenium.
driver=webdriver.Chrome()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/anaconda/lib/python2.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 67, in __init__
desired_capabilities=desired_capabilities)
File "/usr/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 87, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 141, in start_session
'desiredCapabilities': desired_capabilities,
File "/usr/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 201, in execute
self.error_handler.check_response(response)
File "/usr/anaconda/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 181, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: crashed
(Driver info: chromedriver=2.33.506092 (733a02544d189eeb751fe0d7ddca79a0ee28cce4),platform=Linux 3.19.8-100.fc20.x86_64 x86_64)
I use "chromedriver_linux64.zip 2017-10-03 21:09:52 3.90MB" from
url.
You are using the current latest chromedriver 2.33 with Google Chrome 38.0.2125.104.
From the release notes, the support for this version is:
----------ChromeDriver v2.33 (2017-10-03)----------
Supports Chrome v60-62
Make sure also that you are using the latest stable version of selenium.
Furthermore, from Help WebDriver find the downloaded ChromeDriver executable, you should do one of these
include the ChromeDriver location in your PATH environment variable
(Python only) include the path to ChromeDriver when instantiating webdriver.Chrome (see sample below)
If you're going to use an old version of Chrome - you will need to match the version of chromedriver to it. In your case you are using Chrome 38, which was last officially supported by ChromeDriver 2.13, which you can find here
If your users are using newer versions of Chrome, I would recommend updating your distro and installed chrome to match them.
The error says it all :
File "<stdin>", line 1, in <module>
Seems the error occurs on the very first line which is as :
driver=webdriver.Chrome()
This is because here in this code block unless you import webdriver, the driver object is unable to properly initiate/handle the instance of WebBrowser i.e. Chrome Browser.
Solution :
While working with Selenium 3.x, ChromeDriver 2.33.x you need Chrome Browser v60-62 with Python 3.x bindings, and you have to do the following:
Download the chromedriver binary from this link.
Import the necessary modules
Provide the absolute path of the chromedriver binary in your system.
Below is the minimal code :
//The Linux Example
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'/usr/bin/chromedriver')
driver.get('https://www.google.co.in')
print("Page Title is : %s" %driver.title)
driver.quit()
//The Windows Example
from selenium import webdriver
driver = webdriver.Chrome(executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://www.google.co.in')
print("Page Title is : %s" %driver.title)
driver.quit()
Update :
On a separate note as you are seeing WebDriverException: Message: unknown error: Chrome failed to start: crashed perform the following additional steps :
Uninstall Google Chrome from your system through Revo Uninstaller.
Run CCleaner to wipe out all the unwanted OS chores.
Take a system Reboot
Install latest official build of Google Chrome
Execute your Test.
import the selenium and chrome.option to let chrom run in headlessmode
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
if you use chrome or chromiun as root user on linux you should add "--no-sandbox" option and set windows size to avoid some item no show because the window size is too small
chrome_options = Options()
chrome_options.add_argument("--headless") # headless mode
chrome_options.add_argument("--no-sandbox") # run as root user should add --no-sandbox option
chrome_options.add_argument("--window-size=1920x1080")
you can download the chrome driver from http://chromedriver.storage.googleapis.com the last version is 2.38
chrome_driver = "path to chromedriver"
# start the driver
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=chrome_driver)
driver.get("https://www.google.com")

chrome driver is not working in windows

I want to run my python selenium scripts in chrome browser on windows. I downloaded chrome driver and placed the exe file in "C:\Python27\Scripts" but still I'm getting error as
Traceback (most recent call last):
File "C:\Users\INNOVATE\Desktop\workspace\Sample\sample.py", line 3, in <module>
driver = webdriver.Chrome()
File "C:\Python27\lib\site-packages\selenium-2.44.0-py2.7.egg\selenium\webdriver\chrome\webdriver.py", line 59, in __init__
self.service.start()
File "C:\Python27\lib\site-packages\selenium-2.44.0-py2.7.egg\selenium\webdriver\chrome\service.py", line 66, in start
"ChromeDriver executable needs to be available in the path. "
selenium.common.exceptions.WebDriverException: Message: ChromeDriver executable needs to be available in the path. Please download from http://chromedriver.storage.googleapis.com/index.html and read up at http://code.google.com/p/selenium/wiki/ChromeDriver
what should I do?
try this package.
from selenium import webdriver
import chromedriver_autoinstaller
chromedriver_autoinstaller.install()
#maximize the chrome
chrome_options = Options()
chrome_options.add_argument("--start-maximized")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("https://www.google.com")
you can defiantly remove chrome_options or add more arguments.
Please see the documentation, https://pypi.org/project/chromedriver-autoinstaller/
Please make sure you have performed all the required steps to open Chrome using Selenium.
Step1: Download Chrome Driver
Latest Release of Chrome Driver you can found is ChromeDriver 2.13.
Please use Link Chrome Driver to get the latest one.
Step2: Save the Chrome Driver to your Project Location and give the path of that folder in your code.
In my case i have saved the Chrome Driver in 'S' drive.
System.setProperty("webdriver.chrome.driver","S://Chrome Driver//chromedriver.exe");
Please check the path again where i believe is the problem in your code.

Categories