We Keep Coding

Selenium Chromedriver not working with Google Colab Anymore on Python 3.8.16?

I have been using selenium chromedriver in google colab for the past year and it seems to be working perfectly.
But last week, the script doesnt seem to be working anymore. I looked at the python version of google colab and it's now on python 3.8.16 which I think is the culprit of this code breaking.
I use the code:
!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install -y chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver= webdriver.Chrome('chromedriver',options=chrome_options)`
And now in this line:
driver= webdriver.Chrome('chromedriver',options=chrome_options)
I get an error saying:
WebDriverException: Message: Service chromedriver unexpectedly exited. Status code was: 1
Anyone already found a fix for this?

Seems to be a problem with chromedriver itsself, and not python.
Also, I couldn't reproduce the error.
The execution seems to be fine for me.
You might check, if there's something else in your code which might break it and you didn't provide here.

Run This Codes. For me it Worked
%%shell
# Ubuntu no longer distributes chromium-browser outside of snap
#
# Proposed solution: https://askubuntu.com/questions/1204571/how-to-install-chromium-without-snap
# Add debian buster
cat > /etc/apt/sources.list.d/debian.list <<'EOF'
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster.gpg] http://deb.debian.org/debian buster main
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster-updates.gpg] http://deb.debian.org/debian buster-updates main
deb [arch=amd64 signed-by=/usr/share/keyrings/debian-security-buster.gpg] http://deb.debian.org/debian-security buster/updates main
EOF
# Add keys
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys DCC9EFBF77E11517
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 112695A0E562B32A
apt-key export 77E11517 | gpg --dearmour -o /usr/share/keyrings/debian-buster.gpg
apt-key export 22F3D138 | gpg --dearmour -o /usr/share/keyrings/debian-buster-updates.gpg
apt-key export E562B32A | gpg --dearmour -o /usr/share/keyrings/debian-security-buster.gpg
# Prefer debian repo for chromium* packages only
# Note the double-blank lines between entries
cat > /etc/apt/preferences.d/chromium.pref << 'EOF'
Package: *
Pin: release a=eoan
Pin-Priority: 500
Package: *
Pin: origin "deb.debian.org"
Pin-Priority: 300
Package: chromium*
Pin: origin "deb.debian.org"
Pin-Priority: 700
EOF
Then Run This
!apt-get update
!apt-get install chromium chromium-driver
!pip3 install selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url = "http://example.com/"
options = Options()
options.add_argument("--headless")
options.add_argument("--no-sandbox")
driver = webdriver.Chrome("chromedriver", options=options)
driver.get(url)
print(driver.title)
driver.quit()

How to install or read Firefox binary for use with Selenium on Databricks?

I'm trying to use Selenium with Firefox on Databricks but can't get it to work, and I haven't come across any others with quite the same issues as me. The problem seems to lie with the Firefox installation.
My first approach was to install Selenium, geckodriver and Firefox within the notebook. The first two installed fine, but the Firefox installation showed some network connection issues. Here's what I ran:
Install Selenium
%pip install selenium
Install geckodriver
%sh
wget https://github.com/mozilla/geckodriver/releases/download/v0.31.0/geckodriver-v0.31.0-linux64.tar.gz -O /tmp/geckodriver.tar.gz
%sh
tar -xvzf /tmp/geckodriver.tar.gz -C /tmp
Install Firefox
%sh
/usr/bin/yes | sudo apt-get update --fix-missing
which gives the error
Ign:1 https://apt.datadoghq.com stable InRelease
Err:2 https://apt.datadoghq.com stable Release Could not handshake: Error in the pull function. [IP: xx.xxx.xx.xx xxx]
Err:3 http://security.ubuntu.com/ubuntu focal-security InRelease Connection failed [IP: xxx.xxx.xxx.xx xx]
Hit:4 https://repos.azul.com/zulu/deb stable InRelease
Err:5 http://archive.ubuntu.com/ubuntu focal InRelease Connection failed [IP: xx.xxx.xx.xx xxx]
Err:6 http://archive.ubuntu.com/ubuntu focal-updates InRelease Connection failed [IP: xxx.xxx.xxx.xx xx]
Err:7 http://archive.ubuntu.com/ubuntu focal-backports InRelease Connection failed [IP: xx.xxx.xx.xx xxx]
I tried skipping the apt-get update command and running
%sh
sudo apt-get --yes --force-yes install firefox
but it resulted in similar errors.
I later discovered that our databricks environment is set up in such a way that we can't send anything outside of our network, so this approach of installing Firefox will not work.
My second approach was to download and install Firefox to my local machine and upload the firefox.exe file to the dbfs. I installed selenium and the geckodriver as above and ran the following code.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
options.binary_location = r'/dbfs/path/to/firefox/firefox.exe'
driver = webdriver.Firefox(options=options, executable_path = '/path/to/driver/geckodriver')
which gave the error
driver = webdriver.Firefox(options=options, executable_path = '/path/to/driver/geckodriver')
InvalidArgumentException: Message: binary is not a Firefox executable
It seems that for some reason, it can't read the firefox.exe file that I've uploaded to dbfs.
Any suggestions as to possible fixes for this? Or any alternative ways to get Selenium working with Firefox (taking the network restrictions in to account)?

The error message from apt package manager shows it couldnt connect to pkg repository. Which indicates that you dont have access grated to that site. Might be some whitelisting issue, or firewall conn request item...
Look up "headless firefox setup" you might also need that to able to run firefox withough X windows system...
hope this helps, Zoltan

WebDriverException: Message: Service geckodriver unexpectedly exited. Status code was: 64 error using Selenium Geckodriver Firefox in FreeBSD jail

For some tests, I've set up a plain new TrueNAS 12.3 FreeBSD Jail and started it, then installed python3, firefox, geckodriver and pip using the following commands:
pkg install python3 firefox geckodriver py38-pip
pip install --upgrade pip
setenv CRYPTOGRAPHY_DONT_BUILD_RUST 1
pip install cryptography==3.4.7
pip install selenium
Afterwards, when I want to use Selenium with Firefox in my Python code, it does not work:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
it brings
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
self.service.start()
File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 98, in start
self.assert_process_still_running()
File "/usr/local/lib/python3.8/site-packages/selenium/webdriver/common/service.py", line 110, in assert_process_still_running
raise WebDriverException(
selenium.common.exceptions.WebDriverException: Message: Service geckodriver unexpectedly exited. Status code was: 64
Funnily, on another Jail that I've set up approximately a year ago (approximately in the mentioned way as well), it just works and does not throw the error (so different versions maybe?)!
This is the only content of geckodriver.log:
geckodriver: error: Found argument '--websocket-port' which wasn't expected, orisn't valid in this context
USAGE:
geckodriver [FLAGS] [OPTIONS]
For more information try --help
Is there anything I could try to get it working? I've already seen this question, but it seems fairly outdated.
Firefox 95.0.2, geckodriver 0.26.0, Python 3.8.12, Selenium 4.1.0

This error message...
selenium.common.exceptions.WebDriverException: Message: Service geckodriver unexpectedly exited. Status code was: 64
and the GeckoDriver log...
geckodriver: error: Found argument '--websocket-port' which wasn't expected, orisn't valid in this context
...implies that the GeckoDriver was unable to initiate/spawn a new Browsing Context i.e. firefox session.
Your main issue is the incompatibility between the version of the binaries you are using as follows:
Your Selenium Client version is 4.1.0.
But your GeckoDriver version is 0.26.0.
As #ernstki mentions in their comment:
You are running a geckodriver older than 0.30.0, and it is missing the --websocket-port option, which newer/new-ish versions of Selenium seem to depend on.
To put it in simple words, till the previous GeckoDriver release of v0.29.0 the --websocket-port option wasn't in use, which is now mandatory with Selenium v4.0.1.
Further #whimboo also confirmed in his comment:
As it has been manifested the problem here is not geckodriver but Selenium. As such you should create an issue on the Selenium repository instead, so that an option could be added to not always pass the --websocket-port argument. If that request gets denied you will have to use older releases of Selenium if testing with older geckodriver releases is really needed.
Solution
Ensure that:
Selenium is upgraded to current levels Version 4.1.0.
GeckoDriver is upgraded to GeckoDriver v0.30.0 level.
Firefox is upgraded to current Firefox v96.0.2 levels.
FreeBSD versions
Incase you are using FreeBSD versions where the GeckoDriver versions are older, in those cases you have to downgrade Selenium to v3.x levels.
Commands (courtesy: Kurtibert):
Uninstall Selenium:
pip3 uninstall selenium;
Install Selenium:
pip3 install 'selenium<4.0.0'

I added the following codes to my dockerfile and my problem was solved. My problems were with the version.
My problem is solved with Selenium 4.1.0 and geckodriver v30.
RUN pip install -U selenium==4.1.0
RUN wget https://github.com/mozilla/geckodriver/releases/download/v0.30.0/geckodriver-v0.30.0-linux64.tar.gz
RUN tar -x geckodriver -zf geckodriver-v0.30.0-linux64.tar.gz -O > /usr/local/bin/geckodriver
RUN chmod +x /usr/local/bin/geckodriver
RUN rm geckodriver-v0.30.0-linux64.tar.gz

None permission for chromedriver.exe in colab

I am trying to run the webdriver resource in the selenium module (python) in Chrome for the google colab. Firstval I have problems to parse the chromedriver.exe file in the command (selenium.webdriver.Chrome('/chromedriver.exe')), overcome that I found the continuos failure of none permission to run the chromedriver.exe, and the version is ok, who knows what possibly is wrong?
WebDriverException: Message: 'chromedriver.exe' executable may have wrong permissions.

You can do it by installing the chromium webdriver and adjusting some options such that it does not crash in google colab:
!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://www.webite-url.com")

Gitlab CI - Django functional tests - splinter

I want to run some automation tests on github on a project of mine with a Django framework. Therefore I am using a Django functional test. While executing the test on my local pc works fine, my pipeline is always failing with those tests.
I assumed, that chromedriver wasn't working correctly and after some research on the internet, I found out, that I need to install chrome as browser, so I modified my requirements.txt for pip like this:
applescript==2018.11.19
astroid==2.1.0
autopep8==1.4.3
chromedriver==2.24.1
decorator==4.3.0
detect==2018.11.19
Django==2.1.3
flake8==3.6.0
google-chrome==2018.11.19
google-chrome-cli==2018.11.19
isort==4.3.4
lazy-object-proxy==1.3.1
mccabe==0.6.1
only==2018.11.20
psutil==5.4.8
public==2018.11.20
pycodestyle==2.4.0
pyflakes==2.0.0
pylint==2.2.1
pytz==2018.7
runcmd==2018.11.20
selenium==3.141.0
six==1.11.0
splinter==0.10.0
temp==2018.11.20
urllib3==1.24.1
wrapt==1.10.11
.gitlab-ci.yml
image: python:latest
before_script:
- pip install virtualenv
- virtualenv --python=python3 venv/
- source venv/bin/activate
- pip install -r requirements.txt
- cd src/
- python manage.py migrate
stages:
- quality
- tests
flake8:
stage: quality
script:
- flake8 ./
test:
stage: tests
script:
- python manage.py test
test_functional.py
def setUp(self):
# LINUX x64
executable_path = {'executable_path': settings.CHROMEDRIVER_PATH_LINUX64}
# chrome
self.browser_chrome = Browser('chrome', **executable_path)
[..]
With this, a chrome browser has been installed, but now I get this error:
selenium.common.exceptions.WebDriverException:
Message: Service /builds/mitfahrzentrale/mitfahrzentrale/venv/chromedriver unexpectedly exited.
Status code was: 127
What do I need to modify in order to use chromedriver for gitlab?

I don't think the google-chrome package does what you think it does. Looking at its source code, it's a Python wrapper for a set of AppleScript commands around the Chrome browser on MacOS and will certainly not install the browser on Linux.
For reference, here is the (stripped) Gitlab CI pipeline we're using with Django and Selenium to run tests with Firefox and Chrome:
stages:
- test
.test:
coverage: '/TOTAL.*\s+(\d+%)$/'
test-linux_x86_64:
extends: .test
image: python:3.7.1-stretch
stage: test
tags:
- linux_x86_64
script:
- apt -qq update
- DEBIAN_FRONTEND=noninteractive apt -qq -y install xvfb firefox-esr chromium chromedriver
# download geckodriver as no distro offers a package
- apt install -qq -y jq # I don't want to parse JSON with regexes
- curl -s https://api.github.com/repos/mozilla/geckodriver/releases/latest | jq -r '.assets[].browser_download_url | select(contains("linux64"))' | xargs -n1 curl -sL | tar -xz -C /usr/local/bin
- chmod +x /usr/local/bin/geckodriver
# prepare Django installation
- python -m venv /opt/testing
# bundled pip and setuptools are outdated
- /opt/testing/bin/pip install --quiet --upgrade pip setuptools
- /opt/testing/bin/pip install --quiet -r requirements.txt
- xvfb-run /opt/testing/bin/python manage.py test
Some notes:
taking a closer look at the job, all the steps besides the last two are preparation steps; moving them to a custom Docker image will reduce the test running time and the amount of boilerplate in your pipeline.
here, xvfb is used to run the browser in a virtual display; the modern browsers are able to run in headless mode (add --headless to chromedriver options), making the virtual display unnecessary. If you don't need to support old browser versions, you can omit the xvfb installation and xvfb-run usage.
The tests will run as root in container; at first, we got the error
E selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
E (unknown error: DevToolsActivePort file doesn't exist)
E (The process started from chrome location /usr/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
E (Driver info: chromedriver=2.41,platform=Linux 4.15.10-300.fc27.x86_64 x86_64)
If you face this, you need to pass the additional flag --no-sandbox to Chrome because it refuses to run as root without it:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
ds = DesiredCapabilities.CHROME
ds['loggingPrefs'] = {'browser': 'ALL'}
driver = webdriver.Chrome(desired_capabilities=ds, options=chrome_options)