I tried to launch a simple Python script using selenium library from the crontab but it doesn't work.
I'm on a RaspberryPi 3B+. A screen is connected by HDMI port.
The Python script is located in the directory /home/pi/Documents/
The code works fine when launched from terminal.
Here is the code I tried to launch (test2.py):
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
chrome_options = Options()
chrome_options.BinaryLocation = "/usr/lib/chromium-browser"
driver_path = "/usr/lib/chromium-browser/chromedriver"
chrome_options.add_argument('disable-infobars')
chrome_options.add_argument("--kiosk")
chrome_options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(service=Service(driver_path), options=chrome_options)
driver.get("https://www.google.com")
Here is my crontab file :
* * * * * /usr/bin/python3 /home/pi/Documents/test2.py > /home/pi/Documents/logs.txt 2>&1
Here is my logs.txt file :
> Traceback (most recent call last): File
> "/home/pi/Documents/test2.py", line 11, in <module>
> driver = webdriver.Chrome(service=Service(driver_path), options=chrome_options) File
> "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/chrome/webdriver.py",
> line 80, in __init__
> super().__init__( File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/chromium/webdriver.py",
> line 104, in __init__
> super().__init__( File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 286, in __init__
> self.start_session(capabilities, browser_profile) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 378, in start_session
> response = self.execute(Command.NEW_SESSION, parameters) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/webdriver.py",
> line 440, in execute
> self.error_handler.check_response(response) File "/usr/local/lib/python3.9/dist-packages/selenium/webdriver/remote/errorhandler.py",
> line 245, in check_response
> raise exception_class(message, screen, stacktrace) selenium.common.exceptions.WebDriverException: Message: unknown error:
> Chrome failed to start: exited abnormally. (unknown error:
> DevToolsActivePort file doesn't exist) (The process started from
> chrome location /usr/bin/chromium-browser is no longer running, so
> ChromeDriver is assuming that Chrome has crashed.)
I tried for days to debug this but with no success:
I replaced in my code the driver path by /usr/bin/chromiumdriver
I replaced in my code the BinaryLocation by
/usr/bin/chromium-browser
I add in crontab file DISPLAY=:0
I add in crontab file
PATH=/usr/local/sbin:/usr/loal/bin:usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/usr/lib/chromium-b$
I add in crontab file the
PATH=<return_of_command_echo_$PATH_launched_from_terminal_in_/home/pi/Documents>
I re-installed selenium
I Installed webdriver-manager
I saw these kind of issues many times in the web but I didn't find a solution which solved mine. Could you provide some help please ?
Related
I tried to make a bot which looks up if the Bob Marley TShirt from Ajax Amsterdamn is available.
https://www.adidas.ch/de/ajax-3-jsy/GT9559.html <- there is the link
I was able to get it running (also added an telegram bot to report the succses:
#!/usr/bin/python3
from selenium import webdriver
import requests
token="" # expunged for obvious reasons
chat="" # expunged for obvious reasons
message="Available"
fireFoxOptions = webdriver.FirefoxOptions()
fireFoxOptions.set_headless()
browser = webdriver.Firefox(firefox_options=fireFoxOptions)
browser.get("https://www.adidas.ch/de/ajax-3-jsy/GT9559.html")
if ("Dieses Produkt ist leider ausverkauft." not in browser.page_source):
send_text = 'https://api.telegram.org/bot' + token + '/sendMessage?chat_id=' + chat + '&parse_mode=Markdown&text=' + message
response = requests.get(send_text)
print(response.json())
browser.close()
Working on:
selenium 3.141.0
python 3
Firefox 91.0.2
geckodriver 0.29.1
OS: Manjaro Linux
So after that I tried to deploy it on my Debian 10 Server but here is where the struggle began. I had to install Firefox 78.14.0esr and according to the github release page of the Geckodriver version 0.27.0 of it. Selenium stayed the same with 3.141.0. From what I know and what I researched the versions should be alright but when executed throw this nervewrecking error:
Traceback (most recent call last):
File "./ajax.py", line 18, in <module>
browser = webdriver.Firefox(options=options) #, capabilities=cap, executable_path="/usr/local/bin/geckodriver")
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
keep_alive=True)
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Unable to find a matching set of capabilities
I searched the error up and apparently you have to define the binary paths and the capability "marionette" now I did this and now the code looks like this (including some debugging stuff):
#!/usr/bin/python3
from selenium import webdriver
import requests
from selenium.webdriver.firefox.options import Options as FirefoxOptions
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
#token="1995953311:AAH-D7S-MISkCa0yQxUc84Gf978fz0vtoqY"
#chat="1917512203"
message="just a test"
options = FirefoxOptions()
options.add_argument("--headless")
cap = DesiredCapabilities().FIREFOX
cap["marionette"] = False
binary = "/usr/bin/firefox"
options.binary = binary
browser = webdriver.Firefox(options=options, capabilities=cap, executable_path="/usr/local/bin/geckodriver")
browser.get("https://www.adidas.ch/de/ajax-3-jsy/GT9559.html")
if ("Dieses Produkt ist leider ausverkauft." not in browser.page_source):
send_text = 'https://api.telegram.org/bot' + token + '/sendMessage?chat_id=' + chat + '&parse_mode=Markdown&text=' + message
response = requests.get(send_text)
print(response.json())
else:
print("succ")
browser.close()
But now I get the following error:
Traceback (most recent call last):
File "./ajax.py", line 18, in <module>
browser = webdriver.Firefox(options=options, capabilities=cap, executable_path="/usr/local/bin/geckodriver")
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 191, in __init__
self.binary, timeout)
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/firefox/extension_connection.py", line 52, in __init__
self.binary.launch_browser(self.profile, timeout=timeout)
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 73, in launch_browser
self._wait_until_connectable(timeout=timeout)
File "/home/webadmin/.local/lib/python3.7/site-packages/selenium/webdriver/firefox/firefox_binary.py", line 104, in _wait_until_connectable
"The browser appears to have exited "
selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.
Also changing the cap["marionette"] = False to True just gives me the older error message.
Thank You!
I just reverted all the changes, made an Docker container and put that on the server.
I am new to python and trying to build a selenium code to open a website via firefox.
I am using a debian stretch machine for the tests. The versions of the tools are given below :
geckodriver 0.24.0 ( 2019-01-28) , Python 2.7.13 , Mozilla Firefox 52.7.3 , selenium (3.141.0)
I see that the firefox window opens( even though in headless). But it doesnt proceed further with opening the website. The firefox instance waits for sometime after opening and then stops, the script gives error like
WebHandle.open_application("Firefox" , "http://www.google.com")
File "/home/yyyyy/yyyyy/yyyyy/yyyyy/script.py", line 49, in open_application
driver = webdriver.Firefox(firefox_options = options, executable_path='/usr/local/bin/geckodriver')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 174, in init
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in init
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Code :
from selenium import webdriver
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options as FirefoxOptions
options = FirefoxOptions()
options.add_argument("--headless")
print (" * Opening firefox session")
driver = webdriver.Firefox(firefox_options = options, executable_path='/usr/local/bin/geckodriver')
driver.get("https://www.google.com")
driver.maximize_window()
print(driver.title)
While looking into the geckodriver.log , I see the following errors
1624193615192 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.Wwi0327B9dAL"
1624193616996 Marionette INFO Listening on port 2828
[Child 28357] ###!!! ABORT: Aborting on channel error.: file /build/firefox-esr-52.7.3esr/ipc/glue/MessageChannel.cpp, line 2152
[Child 28357] ###!!! ABORT: Aborting on channel error.: file /build/firefox-esr-52.7.3esr/ipc/glue/MessageChannel.cpp, line 2152
###!!! [Child][MessageChannel] Error: (msgtype=0x3E0003,name=PCompositable::Msg_Destroy) Channel error: cannot send/recv
###!!! [Child][MessageChannel] Error: (msgtype=0x3E0003,name=PCompositable::Msg_Destroy) Channel error: cannot send/recv
Thanks already for any help in fixing the issue
Try placing the geckodriver.exe file in a separate folder & drive and updating the path in executable_path.
eg:
driver = webdriver.Firefox(firefox_options = options, executable_path=r"D:/Python/drivers/geckodriver.exe")
I managed to connect selenium driver to an already opened chrome session with the following method in python
def iniciar_google_previa():
os.system('google-chrome --no-sandbox --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default" &')
time.sleep(2)
options1 = Options()
options1.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
chrome_driver = "/usr/local/bin/chromedriver"
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options1)
return driver
And it worked really good, then I can open any site with driver.get("site_url") to start scraping.
I wanted to try the same with docker containers. Installed all the requierements for the dockerfile, built it and then tried to run it but I get the following error
File "final1.py", line 104, in <module>
iniciar_ciclo()
File "final1.py", line 21, in iniciar_ciclo
driver = iniciar_google_previa()
File "final1.py", line 67, in iniciar_google_previa
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options1)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:9222
from unknown error: unable to discover open pages
I thought it was because it was trying to run a GUI so I added
--headless
google-chrome --no-sandbox --headless --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default"
but got the same error.
already tried installing
xvfb
and then
xvfb-run google-chrome --no-sandbox --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default"
which gave the output
DevTools listening on ws://127.0.0.1:9222/devtools/browser/52bbc92b-e096-4897-8661-233ee573edaf
apparentley it executes the chrome session but is still unable to connect to it and I don't know what else to try
I'm trying to test my code using selenium with chrome driver. But I always got this error message in my terminal.
======================================================================
ERROR: test_input_status (myweb.tests.StorySixFunctionalTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/fredypasaud/Documents/PPW/story_six/myweb/tests.py", line 58, in setUp
self.selenium = webdriver.Chrome('./chromedriver',chrome_options = chrome_options)
File "/home/fredypasaud/Documents/PPW/django01/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/home/fredypasaud/Documents/PPW/django01/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/home/fredypasaud/Documents/PPW/django01/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/home/fredypasaud/Documents/PPW/django01/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/fredypasaud/Documents/PPW/django01/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited normally
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /snap/bin/chromium is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
And i put my chromedriver in the same directory as the manage.py of my project. And here's some code about from my test.py
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
class StorySixFunctionalTest(TestCase):
def setUp(self):
chrome_options = Options()
self.selenium = webdriver.Chrome('./chromedriver',chrome_options = chrome_options)
chrome_options.add_argument('--dns-prefetch-disable')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless')
chrome_options.add_argument('disable-gpu')
chrome_options.addArguments("--disable-extensions");
chrome_options.addArguments("--disable-dev-shm-usage");
super(StorySixFunctionalTest, self).setUp()
def tearDown(self):
self.selenium.quit()
super(StorySixFunctionalTest, self).tearDown()
def test_input_status(self):
selenium = self.selenium
selenium.get('http://127.0.0.1:8000/index/')
status = selenium.find_element_by_id('id_status')
submit = selenium.find_element_by_id('submit')
status.send_keys("Coba Coba")
submit.send_keys(Keys.RETURN)
Update :
I'm trying to debug my program by showing the log of chromedriver, and i got this error instead (which is weird because i have canbera module installed)
Gtk-Message: 14:40:58.177: Failed to load module "canberra-gtk-module"
Gtk-Message: 14:40:58.196: Failed to load module "canberra-gtk-module"
I'm trying to use the HTMLUnit WebDriver from Python with the following code:
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.webdriver.remote.webdriver import WebDriver
if __name__ == '__main__':
webdriver = WebDriver('http://127.0.0.1:4444/wd/hub', DesiredCapabilities.HTMLUNIT)
webdriver.get('http://www.google.com')
... and get the following error:
Traceback (most recent call last):
File "bcc_mon_webdriver.py", line 8, in <module>
webdriver = WebDriver('http://127.0.0.1:4444/wd/hub', DesiredCapabilities.HTMLUNIT)
File "c:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 63, in __init__
self.start_session(desired_capabilities, browser_profile)
File "c:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 105, in start_session
'desiredCapabilities': desired_capabilities,
File "c:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 156, in execute
self.error_handler.check_response(response)
File "c:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 147, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: u'Error forwarding the new session cannot find : {platform=ANY, browserName=htmlunit, version=}' ; Stacktrace: Method process threw an error in RequestHandler.java
I use selenium-server-standalone-2.25.0.jar with the Python selenium module also in version 2.25. The Selenium server is running on localhost and it works fine with e.g. DesiredCapabilities.FIREFOX.
Do I have to install htmlunit manually? The selenium websites says that the standalone-jar contains all dependencies.
The problem is you don't have a node that matches the {platform=ANY, browserName=htmlunit, version=} pattern. To fix it you need to start a selenium node with those browser settings, like this:
java -jar selenium-server-standalone-2.25.0.jar -role node -hub http://localhost:4444/grid/register -browser browserName=htmlunit
On the Selenium wiki ( http://code.google.com/p/selenium/wiki/Grid2 ) it says:
"By default, this starts 11 browsers : 5 Firefox, 5 Chrome, 1 Internet
Explorer."
So to be able to use different browsers - like htmlunit - you'll have to start nodes with -browser parameter, check desired_capabilities.py file (located in your selenium egg under selenium/webdriver/common/) for a reference of which parameters are required for each browser.