We Keep Coding

Start Selenium in Raspberry Pi from crontab exit abnormally

I tried to launch a simple Python script using selenium library from the crontab but it doesn't work.
I'm on a RaspberryPi 3B+. A screen is connected by HDMI port.
The Python script is located in the directory /home/pi/Documents/
The code works fine when launched from terminal.
Here is the code I tried to launch (test2.py):
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
chrome_options = Options()
chrome_options.BinaryLocation = "/usr/lib/chromium-browser"
driver_path = "/usr/lib/chromium-browser/chromedriver"
chrome_options.add_argument('disable-infobars')
chrome_options.add_argument("--kiosk")
chrome_options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(service=Service(driver_path), options=chrome_options)
driver.get("https://www.google.com")
Here is my crontab file :
* * * * * /usr/bin/python3 /home/pi/Documents/test2.py > /home/pi/Documents/logs.txt 2>&1
Here is my logs.txt file :
> Traceback (most recent call last): File
> "/home/pi/Documents/test2.py", line 11, in <module>
> driver = webdriver.Chrome(service=Service(driver_path), options=chrome_options) File
> "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/chrome/webdriver.py",
> line 80, in __init__
> super().__init__( File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/chromium/webdriver.py",
> line 104, in __init__
> super().__init__( File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 286, in __init__
> self.start_session(capabilities, browser_profile) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 378, in start_session
> response = self.execute(Command.NEW_SESSION, parameters) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 440, in execute
> self.error_handler.check_response(response) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/errorhandler.py",
> line 245, in check_response
> raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: unknown error:
> Chrome failed to start: exited abnormally. (unknown error:
> DevToolsActivePort file doesn't exist) (The process started from
> chrome location /usr/bin/chromium-browser is no longer running, so
> ChromeDriver is assuming that Chrome has crashed.)
I tried for days to debug this but with no success:
I replaced in my code the driver path by /usr/bin/chromiumdriver
I replaced in my code the BinaryLocation by
/usr/bin/chromium-browser
I add in crontab file DISPLAY=:0
I add in crontab file
PATH=/usr/local/sbin:/usr/loal/bin:usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/usr/lib/chromium-b$
I add in crontab file the
PATH=<return_of_command_echo_$PATH_launched_from_terminal_in_/home/pi/Documents>
I re-installed selenium
I Installed webdriver-manager
I saw these kind of issues many times in the web but I didn't find a solution which solved mine. Could you provide some help please ?

selenium.common.exceptions.WebDriverException:unknown error:cannot connect to chrome at 127.0.0.1:9222 from unknownerror:unable to discover open pages

I managed to connect selenium driver to an already opened chrome session with the following method in python
def iniciar_google_previa():
os.system('google-chrome --no-sandbox --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default" &')
time.sleep(2)
options1 = Options()
options1.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
chrome_driver = "/usr/local/bin/chromedriver"
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options1)
return driver
And it worked really good, then I can open any site with driver.get("site_url") to start scraping.
I wanted to try the same with docker containers. Installed all the requierements for the dockerfile, built it and then tried to run it but I get the following error
File "final1.py", line 104, in <module>
iniciar_ciclo()
File "final1.py", line 21, in iniciar_ciclo
driver = iniciar_google_previa()
File "final1.py", line 67, in iniciar_google_previa
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options1)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:9222
from unknown error: unable to discover open pages
I thought it was because it was trying to run a GUI so I added
--headless
google-chrome --no-sandbox --headless --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default"
but got the same error.
already tried installing
xvfb
and then
xvfb-run google-chrome --no-sandbox --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default"
which gave the output
DevTools listening on ws://127.0.0.1:9222/devtools/browser/52bbc92b-e096-4897-8661-233ee573edaf
apparentley it executes the chrome session but is still unable to connect to it and I don't know what else to try

Unable to run Chrome on Amazon Linux

I have a python script that launches a headless webdriver. The exact same code works with no issues on CentOS7. I'm trying to run it on an Amazon Linux
google-chrome --version output is Google Chrome 83.0.4103.61
chromedriver --version output is ChromeDriver 83.0.4103.39
Here's the function
def LoadWebDrivers():
options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--ignore-ssl-errors')
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_argument('--window-size=800,600')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--remote-debugging-port=443')
global WebDriver
WebDriver = webdriver.Chrome(executable_path=driverPath,options=options)
WebDriver.get('site')
return WebDriver
If I run chrome directly from the CLI, I get this (there are also a bunch of Fontconfig warnings which I read can be ignored)
[3779:3779:0520/015819.096591:ERROR:browser_main_loop.cc(1473)] Unable to open X display.
[0520/015819.108227:ERROR:nacl_helper_linux.cc(308)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
Segmentation fault
The script fails with this
File "/home/ec2-user/.local/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 76, in __init__
RemoteWebDriver.__init__(
File "/home/ec2-user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/home/ec2-user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/ec2-user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/ec2-user/.local/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
After enabling logging, here's what the log file spit out (multiple entries of the devtools error)
[0520/021834.142267:ERROR:socket_posix.cc(148)] bind() failed: Permission denied (13)
[0520/021834.143153:ERROR:socket_posix.cc(148)] bind() failed: Permission denied (13)
[0520/021834.143191:ERROR:devtools_http_handler.cc(298)] Cannot start http server for devtools.
[1589941174.014][DEBUG]: DevTools HTTP Request: http://localhost:443/json/version
[1589941174.015][DEBUG]: DevTools HTTP Request failed
[1589941174.076][INFO]: [1d7ba4dc7a13a3e6bce467b1e3c51393] RESPONSE InitSession ERROR chrome not reachable
[1589941174.076][DEBUG]: Log type 'driver' lost 0 entries on destruction
[1589941174.076][DEBUG]: Log type 'browser' lost 0 entries on destruction

Try with below settings in additional to the above. Also add some logging at verbose level.
chromedriver_path = "<chromedriver_binary_path>"
outputdir = "<log_dir_path>"
service_log_path = "{}/chromedriver.log".format(outputdir)
service_args = ['--verbose']
chrome_options = Options()
chrome_options.add_argument("--no-sandbox")
chrome_options.add_argument("--disable-gpu")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--headless")
driver = webdriver.Chrome(executable_path=chromedriver_path, service_log_path=service_log_path, service_args=service_args, options=chrome_options)
driver.get(url)

After looking at the logs error404 asked me to add and seeing
[0520/021834.142267:ERROR:socket_posix.cc(148)] bind() failed: Permission denied (13)
[0520/021834.143153:ERROR:socket_posix.cc(148)] bind() failed: Permission denied (13)
[0520/021834.143191:ERROR:devtools_http_handler.cc(298)] Cannot start http server for devtools.
I ran the script as sudo and it successfully loads now.

Selenium Python selenium.common.exceptions.WebDriverException: Message: connection refused using geckodriver and firefox

I'm having issues running my automation test scripts. When I run my script, a browser will appear but it will not type the URL and waits for 10 seconds until it throws an exception. Is there any solutions I can use so then I can get my automation test scripts to work?
Geckodriver.log:
1523997052492 geckodriver INFO geckodriver 0.20.1
1523997052531 geckodriver INFO Listening on 127.0.0.1:37807
1523997052592 mozrunner::runner INFO Running command: "/usr/bin/firefox/firefox" "-marionette" "--headless" "-profile" "/tmp/rust_mozprofile.PU1cngaAJ5Tg"
1523997054831 Marionette INFO Listening on port 2828
Stack Traces:
Error
Traceback (most recent call last):
File
"/home/kavin/PycharmProjects/untitled/Testing/purchaseAmazonItems.py", line 13, in setUp
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/firefox/webdriver.py", line 162, in __init__
keep_alive=True)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Code:
def setUp(self):
binary = FirefoxBinary('/usr/bin/firefox/firefox')
opts = FirefoxOptions()
opts.add_argument("--headless")
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts)
driver = self.driver
driver.get('https://www.amazon.com/')
Specs:
Ubuntu 16.04 geckodriver 0.20.1 firefox
59.0.2+build1-0ubuntu0.16.04.3 Python 3.6 Pycharm 2016.3 Selenium 3.11.0

In absence of the error stack trace configuration issues are pretty hard to debug. Having said that I don't see any major issues in your code block. You may require to perform some additional steps as follows :
Pass the Key executable_path along with the Value referring to the absolute path of the GeckoDriver as follows :
def setUp(self):
binary = FirefoxBinary('/usr/bin/firefox/firefox')
opts = FirefoxOptions()
opts.add_argument("--headless")
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts, executable_path='/path/to/geckodriver')
driver = self.driver
driver.get('https://www.amazon.com/')
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
Use CCleaner tool to wipe off all the OS chores before and after the execution of your Test Suite.
If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
Take a System Reboot.
Execute your #Test.
Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
Alternative
As an alternative you can also try to use the set_headless(headless=boolean_value) as follows :
def setUp(self):
binary = FirefoxBinary('/usr/bin/firefox/firefox')
opts = FirefoxOptions()
opts.set_headless(headless=True)
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts, executable_path='/path/to/geckodriver')
driver = self.driver
driver.get('https://www.amazon.com/')
Here you can find a detailed discussion on How to make firefox headless programatically in Selenium with python?

These two commands both start a webdriver on the same port. The second one causes the error because the port is already in use:
self.driver = webdriver.Firefox(firefox_binary=binary)
browser = webdriver.Firefox(firefox_options=opts)
To correct this, set the options before initializing the driver (in the first command).
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts)

Selenium chromedriver hangs if I specify user-data-dir in Chrome options

If I specify chrome_options, then it hangs:
params = {'executable_path': path_to_driver}
if self._chrome_options is not None:
params['chrome_options'] = self._chrome_options
print "# before construct"
self._driver = webdriver.Chrome(**params)
print "# after construct"
So, the message after_construct is not shown. In chrome_options I passed the string:
user-data-dir=/home/sergzach/.config/google-chrome
So, the Chrome is starting and entering into my normal profile. But the Python script hangs on construction the self._driver and I can't continue to work with the Python script.
If I do not pass self._chrome_options (None) then all is OK: Chrome is starting and execution is going farther (both before_construct and after_construct are printing).
If I pass empty chrome_options:
webdriver.ChromeOptions()
then it doesn't hang.
Installed Chrome version: 55.0.2883.75 (64-bit)
webdriver version: 2.25.426924
OS: Ubuntu.
Update
There is a traceback (it raises in about 20 seconds after script hangs):
File "test.py", line 6, in <module>
w.start()
File "/usr/local/lib/python2.7/dist-packages/walker/walker.py", line 164, in start
self._driver = webdriver.Chrome(**params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 92, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 179, in start_session
response = self.execute(Command.NEW_SESSION, capabilities)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(Driver info: chromedriver=2.27.440175 (9bc1d90b8bfa4dd181fbbf769a5eb5e575574320),platform=Linux 4.2.0-42-generic x86_64
Update
It causes because Chrome can't connect to remote debugger. I output the log:
...
params.update(service_args=['--verbose'])
params.update(service_log_path='/tmp/selenium.log')
...
self._chrome_options.add_argument("--verbose")
...
And I see the reason. But I do not realise how to switch off the option --remote-debugging-port=xxxx which is passing to chrome driver. OK, let's analyze the sources further.

The only one client could be connected to a debugger in one time. So, to fix the issue, when we want to enter to a user profile using a debugger - to avoid chromedriver hangs trying to connect to the debugger, we must close an existing Chrome session (I share this conversation one more time).

I get round this by killing all the chrome processes first:
import psutil
for proc in psutil.process_iter():
print(proc)
# Check whether the process name matches or not
if proc.name() == 'chrome' or proc.name() == 'chromedriver':
proc.kill()