I am having a complex python-selenium test suite to test a non-public webpage. In that setup I need to get the webdriver like follows:
self.driver = webdriver.Firefox(firefox_profile = profile, log_path = logfile)
with some profile and log path. In most cases this line of code just works fine, but sometimes (5% or the cases) I get a socket timeout error:
File "/root/tests/usecase_tests/tools/basicsuite.py", line 213, in set_driver_firefox
self.driver = webdriver.Firefox(firefox_profile = profile, log_path = logfile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 158, in __init__
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 309, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 460, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 484, in _request
resp = self._conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 453, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
line = self.fp.readline(_MAXLINE + 1)
File "/usr/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
What could be a possible cause of this error? How to debug this error? How to fix it?
The error says it all :
File "/usr/lib/python2.7/socket.py", line 480, in readline
data = self._sock.recv(self._rbufsize)
timeout: timed out
Sequence of events
Here is the sequence of events that happened :
Initially the error occurs at the following line in basicsuite.py file :
self.driver = webdriver.Firefox(firefox_profile = profile, log_path = logfile)
After a series of checks finally the following method is called which fails :
def readinto(self, b):
"""Read up to len(b) bytes into the writable buffer *b* and return
the number of bytes read. If the socket is non-blocking and no bytes
are available, None is returned.
If *b* is non-empty, a 0 return value indicates that the connection
was shutdown at the other end.
"""
self._checkClosed()
self._checkReadable()
if self._timeout_occurred:
raise OSError("cannot read from timed out object")
while True:
try:
return self._sock.recv_into(b)
except timeout:
self._timeout_occurred = True
raise
except error as e:
if e.args[0] in _blocking_errnos:
return None
raise
The program errors out at :
self._sock.recv(b) # where If *b* is non-empty, a 0 return value indicates that the connection was shutdown at the other end.
Conclusion
The conclusion is that the attempt to make the Connection wasn't successfull which means the initialization of the webdriver instance and further spawning of a new Mozilla Firefox Browser Session was unsuccessful.
Analysis
It won't be possible to conclude the real reason why timeout: timed out occured. But you can follow some of the Best Practices as follows :
Provide the complete name of the logfile along with the logical location of the logfile (from Project Level) as follows :
self.driver = webdriver.Firefox(firefox_profile=profile, log_path='./Log/geckodriver.log')
Always use quit() in the tearDown() method so that the webdriver and the webclient both are properly destroyed.
Before starting your Test Execution, through Task Manager ensure that there are no dangling instances of GeckoDriver or Firefox process within your system.
Ensure that the binary versions you are using JDK, Selenium, GeckoDriver, Mozilla Firefox Browser are compatible. You can find a detailed discussion in this QA Selenium WebDriver 3.4.0 + geckodriver 0.18.0 + Firefox ?? - which combination works?
Clean the Project Workspace from your IDE before and after executing your Test Suite.
Use CCleaner tool regularly to wipe off the OS chores.
If the base version of Firefox Browser is too ancient, uninstall the Firefox Browser through Revo Uninstaller with Moderate Scan and install a recent GA-Released version of Firefox Browser.
This happened to me in my Django app. The problem went away when I set DEBUG = False in my settings.py file.
it's probably not related but
if you are using vs-code on windows and you got this error just try to run from cmd, not from vs-code
Related
So I have written this python code which is supposed to update my advertisment on a german roomshare website automatically:
from time import sleep
from selenium import webdriver
import sched, time
from selenium.webdriver import FirefoxOptions
import datetime
s = sched.scheduler(time.time, time.sleep)
def click_button_by_id(button_s, browser):
button = browser.find_element_by_id(button_s)
button.click()
def refresh_ads(sc):
opts = webdriver.ChromeOptions()
opts.add_argument("--headless")
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
browser = webdriver.Chrome(options = opts)
browser.get('https://www.wg-gesucht.de')
try:
click_button_by_id("cmpbntyestxt", browser)
except:
pass
browser.implicitly_wait(5)
login_link = browser.find_element_by_xpath('//*[#id="headbar_wrapper"]/div[2]/a[2]')
login_link.click()
sleep(2)
username_input = browser.find_element_by_id("login_email_username")
password_input = browser.find_element_by_id("login_password")
username_input.send_keys("MY_USERNAME")
password_input.send_keys("MY_PASSWORD")
login_button = browser.find_element_by_id("login_submit")
login_button.click()
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
sleep(2)
browser.get('https://www.wg-gesucht.de/angebot-bearbeiten.html?action=update_offer&offer_id=xxxxxxxx')
click_button_by_id("set_today_date",browser)
click_button_by_id("update_offer",browser)
print("Refreshed adds at:",datetime.datetime.now().strftime("%c"))
sleep(2)
browser.close()
s.enter(1800, 1, refresh_ads, (sc,))
s.enter(1, 1, refresh_ads, (s,))
s.run()
It works completely fine on my homemachine using this:
Ubuntu 20.04
Python 3.8.5
Chromium 88.0.4324.96
Chromedriver 1:85.0.4183.83-0ubuntu0.20.04.2
While crashing on my Server using this:
Ubuntu 18.04
Python 3.8.7 (was 3.6.9 before. I updated)
Chromium 87.0.4280.66
Chromedriver 87.0.4280.66-0ubuntu0.18.04.1
and this error message:
File "wg_gesucht_bot.py", line 66, in <module>
s.run()
File "/usr/lib/python3.6/sched.py", line 154, in run
action(*argument, **kwargs)
File "wg_gesucht_bot.py", line 23, in refresh_ads
browser = webdriver.Chrome(options = opts)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created
from tab crashed
(Session info: headless chrome=87.0.4280.66)
I have already tried using Firefox, it was actually what I used in the first place, and is also the reason why I still import FirefoxOptions. Because otherwise it won't work properly (it starts but doesnt get far on the webpage itself) but that is a problem I can solve afterwards.
There was this: https://stackoverflow.com/a/40379664/10811865 but updating to a 5 year old version didn't make a whole lot of sense to me
And this: https://stackoverflow.com/a/59191325/10811865
Which I also tried.
Any help is appreciated
So I found a solution for the problem:
First of all I had to update python to 3.8.5 which I did using this :
https://linuxize.com/post/how-to-install-python-3-8-on-ubuntu-18-04/
Afterwards I would encounter a no module named selenium found problem so I followed this:
https://shashanksrivastava.medium.com/how-to-fix-no-module-named-selenium-error-in-python-3-da3fd7b61485
to install in manually
Then I still encountered an error described here:
WebDriverException: unknown error: DevToolsActivePort file doesn't exist while trying to initiate Chrome Browser
Those fixes I had already implemented so they weren't a whole lot of help but killing all chromium and chromedriver processes on my machine fixed the problem for me.
And long last I had to add this line of code:
opts.add_argument('window-size=1920x1080');
so that my code would run properly even in headless mode.
If I specify chrome_options, then it hangs:
params = {'executable_path': path_to_driver}
if self._chrome_options is not None:
params['chrome_options'] = self._chrome_options
print "# before construct"
self._driver = webdriver.Chrome(**params)
print "# after construct"
So, the message after_construct is not shown. In chrome_options I passed the string:
user-data-dir=/home/sergzach/.config/google-chrome
So, the Chrome is starting and entering into my normal profile. But the Python script hangs on construction the self._driver and I can't continue to work with the Python script.
If I do not pass self._chrome_options (None) then all is OK: Chrome is starting and execution is going farther (both before_construct and after_construct are printing).
If I pass empty chrome_options:
webdriver.ChromeOptions()
then it doesn't hang.
Installed Chrome version: 55.0.2883.75 (64-bit)
webdriver version: 2.25.426924
OS: Ubuntu.
Update
There is a traceback (it raises in about 20 seconds after script hangs):
File "test.py", line 6, in <module>
w.start()
File "/usr/local/lib/python2.7/dist-packages/walker/walker.py", line 164, in start
self._driver = webdriver.Chrome(**params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/chrome/webdriver.py", line 70, in __init__
desired_capabilities=desired_capabilities)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 92, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 179, in start_session
response = self.execute(Command.NEW_SESSION, capabilities)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
(Driver info: chromedriver=2.27.440175 (9bc1d90b8bfa4dd181fbbf769a5eb5e575574320),platform=Linux 4.2.0-42-generic x86_64
Update
It causes because Chrome can't connect to remote debugger. I output the log:
...
params.update(service_args=['--verbose'])
params.update(service_log_path='/tmp/selenium.log')
...
self._chrome_options.add_argument("--verbose")
...
And I see the reason. But I do not realise how to switch off the option --remote-debugging-port=xxxx which is passing to chrome driver. OK, let's analyze the sources further.
The only one client could be connected to a debugger in one time. So, to fix the issue, when we want to enter to a user profile using a debugger - to avoid chromedriver hangs trying to connect to the debugger, we must close an existing Chrome session (I share this conversation one more time).
I get round this by killing all the chrome processes first:
import psutil
for proc in psutil.process_iter():
print(proc)
# Check whether the process name matches or not
if proc.name() == 'chrome' or proc.name() == 'chromedriver':
proc.kill()
Hi I am new in Selenium Webdriver and I am currently creating a test case for a specific function.
I have a current test case that is already working for another function, but when I include a new test case and run it. I receive an error: BadStatusLine.
Also when I run the individual test case it work perfectly fine, but when I run the whole test I will receive an error: BadStatusLine.
ERROR: test_task_xml (__main__.TestActuserLayouts)
Test if the task xml is hidden
----------------------------------------------------------------------
Traceback (most recent call last):
File "acttemplate_layouts.py", line 25, in setUp
driver.find_element_by_name("password").submit()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 84, in submit
self._execute(Command.SUBMIT_ELEMENT)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webelement.py", line 457, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 231, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 395, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 426, in _request
resp = self._conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1051, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 415, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 379, in _read_status
raise BadStatusLine(line)
BadStatusLine: ''
This is the whole test:
import unittest
from selenium import webdriver
class TestActuserLayouts(unittest.TestCase):
"""Test user functions """
driver = webdriver.Firefox()
driver.implicitly_wait(30)
base_url = "http://samplepage.com"
def setUp(self):
"""Base setting before test
"""
driver = self.driver
# Login
driver.get(self.base_url + "/")
driver.find_element_by_name("user_name").clear()
driver.find_element_by_name("user_name").send_keys("userme")
driver.find_element_by_name("password").clear()
driver.find_element_by_name("password").send_keys("123")
driver.find_element_by_name("password").submit()
self.assertEqual(self.base_url + "/userme/inbox/", driver.current_url)
def test_task_xml(self):
"""Test if the task xml is hidden
"""
driver = self.driver
# Get page by URL
driver.get(self.base_url + "/userme/mission/")
# Get page by URL
driver.get(self.base_url + "/mission/update/0000000a-0000-0000-0000-000000000000/")
self.assertEqual(self.base_url + "/mission/update/0000000a-0000-0000-0000-000000000000/", driver.current_url)
driver.find_element_by_id("discussion-btn").click()
# Scan browser if Task xml is not present
self.assertFalse('Task xml' in self.driver.page_source)
def test_task_list(self):
"""Test if the number of task for each mission is null
"""
driver = self.driver
#Get page url
driver.get(self.base_url + "/userme/mission/")
#Get page url
driver.get(self.base_url + "/acttemplate/list-all/")
#Scan browser if the number of task is null
self.assertEqual(driver.find_element_by_xpath("//tr[3]/td[4]").text != "", True)
def tearDown(self):
"""Clear setting after test
"""
self.driver.close()
# Run unit test directly
if __name__ == "__main__":
unittest.main()
While the other test case is a failure. So, I am not really familiar with selenium and I need help to locate this error. Can someone please help me in this issue. Thanks a lot.
I'm not sure, but I'll make a guess. Somehow, browser dies while you perform such actions like pressing buttons, clicking.
It's an error raised by httplib (low-level python http library), means that server inside browser didn't responded any of known http status codes (200, 300, 500, etc.), and instead, the response was '' (empty string).
Since we started running selenium UI tests in jenkins, we noticed a small but annoying frequency of errors during tests. We get BadStatusLine and CannotSendRequest errors on seemingly random selenium actions (click, quit, visit, etc.).
They would usually look something like:
File "/usr/lib/python2.7/unittest/case.py", line 327, in run
testMethod()
File "/home/jenkins/workspace/Create and Upload Functional Testing/shapeways/test_suite/Portal/CreateAndUpload/TestUploadWhenNotLoggedIn_ExpectLoginModal.py", line 22, in runTest
self.dw.visit(ShapewaysUrlBuilder.build_model_upload_url())
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 212, in visit
return self.execute_and_handle_webdriver_exceptions(lambda: _visit(url))
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 887, in execute_and_handle_webdriver_exceptions
return function_to_execute()
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 212, in <lambda>
return self.execute_and_handle_webdriver_exceptions(lambda: _visit(url))
File "/home/jenkins/workspace/Create and Upload Functional Testing/webdriver/webdriverwrapper/WebDriverWrapper.py", line 205, in _visit
return self.driver.get(url)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 185, in get
self.execute(Command.GET, {'url': url})
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 171, in execute
response = self.command_executor.execute(driver_command, params)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 349, in execute
return self._request(command_info[0], url, body=data)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/remote_connection.py", line 380, in _request
resp = self._conn.getresponse()
File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
response.begin()
File "/usr/lib/python2.7/httplib.py", line 407, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
raise BadStatusLine(line)
This particular case came from the following stack:
selenium==2.44.0
python==2.7.3
firefox==34.0
jenkins
xvfb (using the jenkins plugin for headless displays)
though we've seen these errors popping up all the time across many different version permutations of firefox/selenium.
I ran a tcpdump to capture the actual request sent right before the BadStatusLine error came up and got the following.
POST /hub/session/ab64574a-4a17-447a-b2e8-5b0f5ed5e923/url HTTP/1.1
Host: 127.0.0.1:41246
Accept-Encoding: identity Content-Length: 102
Connection: keep-alive
Content-type: application/json;charset="UTF-8"
POST: /hub/session/ab64574a-4a17-447a-b2e8-5b0f5ed5e923/url
Accept: application/json
User-Agent: Python http auth
{"url": "http://example.com/login", "sessionId": "ab64574a-4a17-447a-b2e8-5b0f5ed5e923"}
Response comes back with 0 bytes. So the BadStatusLine was caused by an empty response, which makes sense.
The question is, why would selenium's server return an empty response. If the server died, wouldn't we get a ConnectionError or something along those lines?
For a while, I had no repro and no idea what the cause was. I was finally able to repro by running:
import requests
import json
while True:
requests.post('http://127.0.0.1/hub/session/', data=json.dumps({"url": "http://example.com/login", "sessionId": "ab64574a-4a17-447a-b2e8-5b0f5ed5e923"}))
While this was running, I quit the browser and got a BadStatusLine error! When I tried making that request again, that's when I got the expected "ConnectionError" that you would see from any dead server.
SO, what I suspect happens is that when the browser is sent the kill signal, there is a short window during its shutdown where any response will still be returned but with 0 bytes. That's why you get different types of exceptions for essentially the same problem (browser dies). Turns out we had a cron which was killing our browsers in the background.
I'm having trouble running SeleniumRC from Django. I can run the sample code provided by the Selenium docs and the python selenium client docs just fine from in a Python shell without running Django (so no manage.py), but when I actually try to run Selenium from a django TestCase or from the Django shell, I get a timeout error.
Here is the code I'm trying to run:
from selenium import selenium
from django.test import TestCase
class TestSelenium(TestCase):
def setUp(self):
self.verificationErrors = []
self.selenium = selenium("localhost", 4444, "*firefox", "http://127.0.0.1:8000/")
self.selenium.start()
def test_foo(self):
sel = self.selenium
sel.open("/")
Running with manage.py test registration.TestSelenium produces the following error:
======================================================================
ERROR: test_testformmaintainsdata (registration.tests.TestSelenium)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/sam/Documents/dev/app/CustomMade/registration/tests.py", line 158, in setUp
self.selenium.start()
File "/usr/local/lib/python2.6/dist-packages/selenium/selenium.py", line 189, in start
result = self.get_string("getNewBrowserSession", start_args)
File "/usr/local/lib/python2.6/dist-packages/selenium/selenium.py", line 223, in get_string
result = self.do_command(verb, args)
File "/usr/local/lib/python2.6/dist-packages/selenium/selenium.py", line 214, in do_command
response = conn.getresponse()
File "/usr/lib/python2.6/httplib.py", line 990, in getresponse
response.begin()
File "/usr/lib/python2.6/httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "/usr/lib/python2.6/httplib.py", line 349, in _read_status
line = self.fp.readline()
File "/usr/lib/python2.6/socket.py", line 427, in readline
data = recv(1)
timeout: timed out
----------------------------------------------------------------------
Ran 1 test in 12.475s
FAILED (errors=1)
Destroying test database 'default'...
The strange thing is that even though the error is thrown, and Python stops, the SeleniumRC server does actually launch Firefox, but then I can't run further Selenium commands since Django has stopped. This is the output from SeleniumServer:
14:21:48.362 INFO - Checking Resource aliases
14:21:48.369 INFO - Command request: getNewBrowserSession[*firefox, http://127.0.0.1:8000/, ] on session null
14:21:48.372 INFO - creating new remote session
14:21:48.443 INFO - Allocated session a3ea05a3d0eb4956ba69a67583ea49ba for http://127.0.0.1:8000/, launching...
14:21:48.533 INFO - Preparing Firefox profile...
14:21:51.473 INFO - Launching Firefox...
14:21:55.904 INFO - Got result: OK,a3ea05a3d0eb4956ba69a67583ea49ba on session a3ea05a3d0eb4956ba69a67583ea49ba
Anyone have any ideas?
In case anyone else has this problem, I was able to solve it by increasing my socket timeout in the setUp method.
This will do the trick with the timeout value in float seconds:
import socket
from selenium import selenium
from django.test import TestCase
class TestSelenium(TestCase):
def setUp(self):
socket.settimeout(30)
# ...
self.selenium.start()
Refer to python stdlib docs