We Keep Coding

Firefox fails to open url when using webdriver python API

I am new to python and trying to build a selenium code to open a website via firefox.
I am using a debian stretch machine for the tests. The versions of the tools are given below :
geckodriver 0.24.0 ( 2019-01-28) , Python 2.7.13 , Mozilla Firefox 52.7.3 , selenium (3.141.0)
I see that the firefox window opens( even though in headless). But it doesnt proceed further with opening the website. The firefox instance waits for sometime after opening and then stops, the script gives error like
WebHandle.open_application("Firefox" , "http://www.google.com")
File "/home/yyyyy/yyyyy/yyyyy/yyyyy/script.py", line 49, in open_application
driver = webdriver.Firefox(firefox_options = options, executable_path='/usr/local/bin/geckodriver')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 174, in init
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in init
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Code :
from selenium import webdriver
from selenium.webdriver import Firefox
from selenium.webdriver.firefox.options import Options as FirefoxOptions
options = FirefoxOptions()
options.add_argument("--headless")
print (" * Opening firefox session")
driver = webdriver.Firefox(firefox_options = options, executable_path='/usr/local/bin/geckodriver')
driver.get("https://www.google.com")
driver.maximize_window()
print(driver.title)
While looking into the geckodriver.log , I see the following errors
1624193615192 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "--headless" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile.Wwi0327B9dAL"
1624193616996 Marionette INFO Listening on port 2828
[Child 28357] ###!!! ABORT: Aborting on channel error.: file /build/firefox-esr-52.7.3esr/ipc/glue/MessageChannel.cpp, line 2152
[Child 28357] ###!!! ABORT: Aborting on channel error.: file /build/firefox-esr-52.7.3esr/ipc/glue/MessageChannel.cpp, line 2152
###!!! [Child][MessageChannel] Error: (msgtype=0x3E0003,name=PCompositable::Msg_Destroy) Channel error: cannot send/recv
###!!! [Child][MessageChannel] Error: (msgtype=0x3E0003,name=PCompositable::Msg_Destroy) Channel error: cannot send/recv
Thanks already for any help in fixing the issue

Try placing the geckodriver.exe file in a separate folder & drive and updating the path in executable_path.
eg:
driver = webdriver.Firefox(firefox_options = options, executable_path=r"D:/Python/drivers/geckodriver.exe")

selenium.common.exceptions.WebDriverException:unknown error:cannot connect to chrome at 127.0.0.1:9222 from unknownerror:unable to discover open pages

I managed to connect selenium driver to an already opened chrome session with the following method in python
def iniciar_google_previa():
os.system('google-chrome --no-sandbox --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default" &')
time.sleep(2)
options1 = Options()
options1.add_experimental_option("debuggerAddress", "127.0.0.1:9222")
chrome_driver = "/usr/local/bin/chromedriver"
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options1)
return driver
And it worked really good, then I can open any site with driver.get("site_url") to start scraping.
I wanted to try the same with docker containers. Installed all the requierements for the dockerfile, built it and then tried to run it but I get the following error
File "final1.py", line 104, in <module>
iniciar_ciclo()
File "final1.py", line 21, in iniciar_ciclo
driver = iniciar_google_previa()
File "final1.py", line 67, in iniciar_google_previa
driver = webdriver.Chrome(executable_path=ChromeDriverManager().install(), options=options1)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/opt/app-root/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:9222
from unknown error: unable to discover open pages
I thought it was because it was trying to run a GUI so I added
--headless
google-chrome --no-sandbox --headless --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default"
but got the same error.
already tried installing
xvfb
and then
xvfb-run google-chrome --no-sandbox --remote-debugging-port=9222 --user-data-dir="/home/natanael/.config/google-chrome/Default"
which gave the output
DevTools listening on ws://127.0.0.1:9222/devtools/browser/52bbc92b-e096-4897-8661-233ee573edaf
apparentley it executes the chrome session but is still unable to connect to it and I don't know what else to try

Why does this python code work on my XUbuntu (Ubuntu 20.04) machine but not my Ubuntu 18.04 Server

So I have written this python code which is supposed to update my advertisment on a german roomshare website automatically:
from time import sleep
from selenium import webdriver
import sched, time
from selenium.webdriver import FirefoxOptions
import datetime
s = sched.scheduler(time.time, time.sleep)
def click_button_by_id(button_s, browser):
button = browser.find_element_by_id(button_s)
button.click()
def refresh_ads(sc):
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
browser = webdriver.Chrome(options = opts)
browser.get('https://www.wg-gesucht.de')
try:
click_button_by_id("cmpbntyestxt", browser)
except:
pass
browser.implicitly_wait(5)
login_link = browser.find_element_by_xpath('//*[#id="headbar_wrapper"]/div[2]/a[2]')
login_link.click()
sleep(2)
username_input = browser.find_element_by_id("login_email_username")
password_input = browser.find_element_by_id("login_password")
username_input.send_keys("MY_USERNAME")
password_input.send_keys("MY_PASSWORD")
login_button = browser.find_element_by_id("login_submit")
login_button.click()
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
print("Refreshed adds at:",datetime.datetime.now().strftime("%c"))
sleep(2)
browser.close()
s.enter(1800, 1, refresh_ads, (sc,))
s.enter(1, 1, refresh_ads, (s,))
s.run()
It works completely fine on my homemachine using this:
Ubuntu 20.04
Python 3.8.5
Chromium 88.0.4324.96
Chromedriver 1:85.0.4183.83-0ubuntu0.20.04.2
While crashing on my Server using this:
Ubuntu 18.04
Python 3.8.7 (was 3.6.9 before. I updated)
Chromium 87.0.4280.66
Chromedriver 87.0.4280.66-0ubuntu0.18.04.1
and this error message:
File "wg_gesucht_bot.py", line 66, in <module>
s.run()
File "/usr/lib/python3.6/sched.py", line 154, in run
action(*argument, **kwargs)
File "wg_gesucht_bot.py", line 23, in refresh_ads
browser = webdriver.Chrome(options = opts)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from tab crashed
(Session info: headless chrome=87.0.4280.66)
I have already tried using Firefox, it was actually what I used in the first place, and is also the reason why I still import FirefoxOptions. Because otherwise it won't work properly (it starts but doesnt get far on the webpage itself) but that is a problem I can solve afterwards.
There was this: https://stackoverflow.com/a/40379664/10811865 but updating to a 5 year old version didn't make a whole lot of sense to me
And this: https://stackoverflow.com/a/59191325/10811865
Which I also tried.
Any help is appreciated

So I found a solution for the problem:
First of all I had to update python to 3.8.5 which I did using this :
https://linuxize.com/post/how-to-install-python-3-8-on-ubuntu-18-04/
Afterwards I would encounter a no module named selenium found problem so I followed this:
https://shashanksrivastava.medium.com/how-to-fix-no-module-named-selenium-error-in-python-3-da3fd7b61485
to install in manually
Then I still encountered an error described here:
WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser
Those fixes I had already implemented so they weren't a whole lot of help but killing all chromium and chromedriver processes on my machine fixed the problem for me.
And long last I had to add this line of code:
opts.add_argument('window-size=1920x1080');
so that my code would run properly even in headless mode.

Selenium Python selenium.common.exceptions.WebDriverException: Message: connection refused using geckodriver and firefox

I'm having issues running my automation test scripts. When I run my script, a browser will appear but it will not type the URL and waits for 10 seconds until it throws an exception. Is there any solutions I can use so then I can get my automation test scripts to work?
Geckodriver.log:
1523997052492 geckodriver INFO geckodriver 0.20.1
1523997052531 geckodriver INFO Listening on 127.0.0.1:37807
1523997052592 mozrunner::runner INFO Running command: "/usr/bin/firefox/firefox" "-marionette" "--headless" "-profile" "/tmp/rust_mozprofile.PU1cngaAJ5Tg"
1523997054831 Marionette INFO Listening on port 2828
Stack Traces:
Error
Traceback (most recent call last):
File
"/home/kavin/PycharmProjects/untitled/Testing/purchaseAmazonItems.py", line 13, in setUp
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/firefox/webdriver.py", line 162, in __init__
keep_alive=True)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Code:
def setUp(self):
binary = FirefoxBinary('/usr/bin/firefox/firefox')
opts = FirefoxOptions()
opts.add_argument("--headless")
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts)
driver = self.driver
driver.get('https://www.amazon.com/')
Specs:
Ubuntu 16.04 geckodriver 0.20.1 firefox
59.0.2+build1-0ubuntu0.16.04.3 Python 3.6 Pycharm 2016.3 Selenium 3.11.0

In absence of the error stack trace configuration issues are pretty hard to debug. Having said that I don't see any major issues in your code block. You may require to perform some additional steps as follows :
Pass the Key executable_path along with the Value referring to the absolute path of the GeckoDriver as follows :
def setUp(self):
binary = FirefoxBinary('/usr/bin/firefox/firefox')
opts = FirefoxOptions()
opts.add_argument("--headless")
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts, executable_path='/path/to/geckodriver')
driver = self.driver
driver.get('https://www.amazon.com/')
Clean your Project Workspace through your IDE and Rebuild your project with required dependencies only.
Use CCleaner tool to wipe off all the OS chores before and after the execution of your Test Suite.
If your base Web Client version is too old, then uninstall it through Revo Uninstaller and install a recent GA and released version of Web Client.
Take a System Reboot.
Execute your #Test.
Always invoke driver.quit() within tearDown(){} method to close & destroy the WebDriver and Web Client instances gracefully.
Alternative
As an alternative you can also try to use the set_headless(headless=boolean_value) as follows :
def setUp(self):
binary = FirefoxBinary('/usr/bin/firefox/firefox')
opts = FirefoxOptions()
opts.set_headless(headless=True)
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts, executable_path='/path/to/geckodriver')
driver = self.driver
driver.get('https://www.amazon.com/')
Here you can find a detailed discussion on How to make firefox headless programatically in Selenium with python?

These two commands both start a webdriver on the same port. The second one causes the error because the port is already in use:
self.driver = webdriver.Firefox(firefox_binary=binary)
browser = webdriver.Firefox(firefox_options=opts)
To correct this, set the options before initializing the driver (in the first command).
self.driver = webdriver.Firefox(firefox_binary=binary, firefox_options=opts)

How to avoid a socket timeout for selenium webdriver?

I am having a complex python-selenium test suite to test a non-public webpage. In that setup I need to get the webdriver like follows:
self.driver = webdriver.Firefox(firefox_profile = profile, log_path = logfile)
with some profile and log path. In most cases this line of code just works fine, but sometimes (5% or the cases) I get a socket timeout error:
File "/root/tests/usecase_tests/tools/basicsuite.py", line 213, in set_driver_firefox
self.driver = webdriver.Firefox(firefox_profile = profile, log_path = logfile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 158, in __init__
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 309, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 460, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 484, in _request
resp = self._conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
What could be a possible cause of this error? How to debug this error? How to fix it?

The error says it all :
File "/usr/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
Sequence of events
Here is the sequence of events that happened :
Initially the error occurs at the following line in basicsuite.py file :
self.driver = webdriver.Firefox(firefox_profile = profile, log_path = logfile)
After a series of checks finally the following method is called which fails :
def readinto(self, b):
"""Read up to len(b) bytes into the writable buffer *b* and return
the number of bytes read. If the socket is non-blocking and no bytes
are available, None is returned.
If *b* is non-empty, a 0 return value indicates that the connection
was shutdown at the other end.
"""
self._checkClosed()
self._checkReadable()
if self._timeout_occurred:
raise OSError("cannot read from timed out object")
while True:
try:
return self._sock.recv_into(b)
except timeout:
self._timeout_occurred = True
raise
except error as e:
if e.args[0] in _blocking_errnos:
return None
raise
The program errors out at :
self._sock.recv(b) # where If *b* is non-empty, a 0 return value indicates that the connection was shutdown at the other end.
Conclusion
The conclusion is that the attempt to make the Connection wasn't successfull which means the initialization of the webdriver instance and further spawning of a new Mozilla Firefox Browser Session was unsuccessful.
Analysis
It won't be possible to conclude the real reason why timeout: timed out occured. But you can follow some of the Best Practices as follows :
Provide the complete name of the logfile along with the logical location of the logfile (from Project Level) as follows :
self.driver = webdriver.Firefox(firefox_profile=profile, log_path='./Log/geckodriver.log')
Always use quit() in the tearDown() method so that the webdriver and the webclient both are properly destroyed.
Before starting your Test Execution, through Task Manager ensure that there are no dangling instances of GeckoDriver or Firefox process within your system.
Ensure that the binary versions you are using JDK, Selenium, GeckoDriver, Mozilla Firefox Browser are compatible. You can find a detailed discussion in this QA Selenium WebDriver 3.4.0 + geckodriver 0.18.0 + Firefox ?? - which combination works?
Clean the Project Workspace from your IDE before and after executing your Test Suite.
Use CCleaner tool regularly to wipe off the OS chores.
If the base version of Firefox Browser is too ancient, uninstall the Firefox Browser through Revo Uninstaller with Moderate Scan and install a recent GA-Released version of Firefox Browser.

This happened to me in my Django app. The problem went away when I set DEBUG = False in my settings.py file.

it's probably not related but
if you are using vs-code on windows and you got this error just try to run from cmd, not from vs-code