This question has been asked over and over again - and in-spite of trying all the hacks I still can't seem to figure out what's wrong.
I tried increasing the implicitly_wait to 30 (and even increased it upto 100) - yet it did not work.
Use case -: I am trying to create a list that wil populate all the items in the page here, as a base case - and I intend to bind this to a mini-module that I already have with scrapy which has all (pages with similar web elements) crawled links - so essentially will be building the whole pipeline, post I am done with this.
###My source code - generated via Selenium IDE, exported to a Python webdriver and manipulated a little later ###
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.wait import WebDriverWait
import unittest, time, re
class Einstein(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.base_url = "http://shopap.lenovo.com/in/en/laptops/"
self.verificationErrors = []
self.accept_next_alert = True
def test_einstein(self):
driver = self.driver
driver.get(self.base_url)
print driver.title
driver.find_element_by_link_text("T430").click()
print driver.title
# driver.find_element_by_xpath("id('facetedBrowseWrapper')/div/div/div[1]/div[2]/ul[1]/li[1]/a").click()
driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
print driver.title
# driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
try: self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
except AssertionError as e: self.verificationErrors.append(str(e))
# Everything ok till here
#**THE CODE FAILS HERE**#
laptop1 = driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text
print laptop1
price1 = driver.find_element_by_css_selector("span.price").text
print price1
detail1 = self.is_element_present(By.CSS_SELECTOR, "div.desc.std")
print detail1
def is_element_present(self, how, what):
try: self.driver.find_element(by=how, value=what)
except NoSuchElementException, e: return False
return True
def is_alert_present(self):
try: self.driver.switch_to_alert()
except NoAlertPresentException, e: return False
return True
def close_alert_and_get_its_text(self):
try:
alert = self.driver.switch_to_alert()
alert_text = alert.text
if self.accept_next_alert:
alert.accept()
else:
alert.dismiss()
return alert_text
finally: self.accept_next_alert = True
def tearDown(self):
self.driver.quit()
self.assertEqual([], self.verificationErrors)
if __name__ == "__main__":
unittest.main()
Errors & output :
ekta#ekta-VirtualBox:~$ python einstein.py
Laptops & Ultrabooks | Lenovo (IN)
ThinkPad T430 Laptop PC for Business Computing | Lenovo (IN)
Buy Lenovo Thinkpad Laptops | Lenovo Thinkpad Laptops Price India
E
======================================================================
ERROR: test_einstein (__main__.Einstein)
----------------------------------------------------------------------
Traceback (most recent call last):
File "einstein.py", line 27, in test_einstein
try: self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 246, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 680, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 165, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 158, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: u'Unable to locate element: {"method":"link text","selector":"Thinkpad Edge E530 (Black)"}' ; Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmphli5Jg/extensions/fxdriver#googlecode.com/components/driver_component.js:8444)
at fxdriver.Timer.prototype.setTimeout/<.notify (file:///tmp/tmphli5Jg/extensions/fxdriver#googlecode.com/components/driver_component.js:386)
----------------------------------------------------------------------
Ran 1 test in 79.348s
FAILED (errors=1)
Questions & comments:
If you are answering this question - please mention why this specific "find_element_by_link_text" does not work.
(Very Basic) In the GUI of my selenium IDE -> Show all available commands - why dont I see the css (find_element_by_css_selector) for all the web elements - is there a way to force feed an element to be read as a CSS selector ?
In case you suggest using some other locator - please mention if that will be consistent way to fetch elements, given (1)
Does assert work to capture the exceptions and "move on" - since even after trying "verify" , "assert" loops, I still cant fetch this "find_element_by_link_text"
I tried using Xpath to build this "element" , but in the view Xpath (in firefox) - I see nothing, to clue why that happens (Of course I removed the namespace ":x" )
Other things I tried apart from implicity_wait(30):
find_element_by_partial_link(“Thinkpad”) and appending Unicode to this (wasn’t sure if it was reading the brackets ( , driver.find_element_by_link_text(u"Thinkpad Edge E530 (Black)").text, still did not work.
Related questions:
How to use find_element_by_link_text() properly to not raise NoSuchElementException?
NoSuchElement Exception using find_element_by_link_text when implicitly_wait doesn't work?
It happened to me before that the find_element_by_link_text method sometimes works and sometimes doesn't work; even in a single case. I think it's not a reliable way to access elements; the best way is to use find_element_by_id.
But in your case, as I visit the page, there is no id to help you. Still you can try find_elements_by_xpath in 3 ways:
1- Accessing title: find_element_by_xpath["//a[contains(#title = 'T430')]"]
2- Accessing text: find_element_by_xpath["//a[contains(text(), 'T430')]"]
3- Accessing href: find_element_by_xpath["//a[contains(#href = 'http://www.thedostore.com/laptops/thinkpad-laptops/thinkpad-t430-u-black-627326q.html')]"].
Hope it helps.
NoSuchElementException is thrown when the element could not be found.
If you encounter this exception, please check the followings:
Check your selector used in your find_by...
Element may not yet be on the screen at the time of the find operation.
If webpage is still loading, check for selenium.webdriver.support.wait.WebDriverWait() and write a wait wrapper to wait for an element to appear.
Troubleshooting and code samples
You can add breakpoint just before your failing line pdb.set_trace() (don't forget to import pdb), then run your test and once your debugger stops, then do the following tests.
You could try:
driver.find_element_by_xpath(u'//a[text()="Foo text"]')
instead. This is more reliable test, so if this would work, use it instead.
If above won't help, please check if your page has been loaded properly via:
(Pdb) driver.execute_script("return document.readyState")
'complete'
Sometimes when the page is not loaded, you're actually fetching the elements from the old page. But even though, readyState could still indicate the state of the old page (especially when using click()). Here is how this is explained in this blog:
Since Selenium webdriver has become more advanced, clicks are much more like "real" clicks, which has the benefit of making our tests more realistic, but it also means it's hard for Selenium to be able to track the impact that a click has on the browsers' internals -- it might try to poll the browser for its page-loaded status immediately after clicking, but that's open to a race condition where the browser was multitasking, hasn't quite got round to dealing with the click yet, and it gives you the .readyState of the old page.
If you think this is happening because the page wasn't loaded properly, the "recommended" (however still ugly) solution is an explicit wait:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
old_value = browser.find_element_by_id('thing-on-old-page').text
browser.find_element_by_link_text('my link').click()
WebDriverWait(browser, 3).until(
expected_conditions.text_to_be_present_in_element(
(By.ID, 'thing-on-new-page'),
'expected new text'
)
)
The naive attempt would be something like this:
def wait_for(condition_function):
start_time = time.time()
while time.time() < start_time + 3:
if condition_function():
return True
else:
time.sleep(0.1)
raise Exception(
'Timeout waiting for {}'.format(condition_function.__name__)
)
def click_through_to_new_page(link_text):
browser.find_element_by_link_text('my link').click()
def page_has_loaded():
page_state = browser.execute_script(
'return document.readyState;'
)
return page_state == 'complete'
wait_for(page_has_loaded)
Another, better one would be (credits to #ThomasMarks):
def click_through_to_new_page(link_text):
link = browser.find_element_by_link_text('my link')
link.click()
def link_has_gone_stale():
try:
# poll the link with an arbitrary call
link.find_elements_by_id('doesnt-matter')
return False
except StaleElementReferenceException:
return True
wait_for(link_has_gone_stale)
And the final example includes comparing page ids as below (which could be bulletproof):
class wait_for_page_load(object):
def __init__(self, browser):
self.browser = browser
def __enter__(self):
self.old_page = self.browser.find_element_by_tag_name('html')
def page_has_loaded(self):
new_page = self.browser.find_element_by_tag_name('html')
return new_page.id != self.old_page.id
def __exit__(self, *_):
wait_for(self.page_has_loaded)
And now we can do:
with wait_for_page_load(browser):
browser.find_element_by_link_text('my link').click()
Above code samples are from Harry's blog.
Here is the version proposed by Tommy Beadle (by using staleness approach):
import contextlib
from selenium.webdriver import Remote
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import staleness_of
class MyRemote(Remote):
#contextlib.contextmanager
def wait_for_page_load(self, timeout=30):
old_page = self.find_element_by_tag_name('html')
yield
WebDriverWait(self, timeout).until(staleness_of(old_page))
If you think it isn't about page load, double check if your element isn't in iframe or different window. If so, you've to switch to it first. To check list of available windows, run: driver.window_handles.
From viewing the source of the page that you provided a link to, it seems you are using an incorrect selector.
You should use instead find_elements_by_link_text(u'text here')[0] to select the first occurrence instead as there seems to be the potential for multiple links with the same link text.
So instead of:
self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
You should use:
self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_elements_by_link_text("Thinkpad Edge E530 (Black)")[0].text)
Solution posted by OP:
Hack 1: Instead of identifying the element as a text-link, I identified the "bigger frame" in which this element was present.
itemlist_1 = driver.find_element_by_css_selector("li.item.first").text
This will give the whole item along with the name, price and detail (and the unwanted add to cart and compare"
See the attached image for more .
Hack 2: I found that the "Buy Now" which was an image element with xPath (driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
, in the code above) , could be made to click/identified faster if I added the following line, before finding this by xpath. I think this sort of narrows down where the Webdriver is looking for an element. This is what I added " driver.find_element_by_css_selector("#subseries").text"
This must have decreased my wait by at least 20 seconds, on that page .Hope that helps.
Related
My code works, but not in all cases
Basically the functionality is to click a load_more button until it no longer appears.
As of right now, I simply have a loop which finds the loadmore button and clicks it twice, but there are cases that it will click on something else when the load more button disappears.
I was planning on making a while loop, which would constantly find the click the load_more option until the loadmore disappeared then break the loop.
Here is the code: (This simply finds and clicks it twice)
load_more = browser.find_element_by_css_selector("#mainContent > div.left-panel > div > div.result-list > div > div.content")
WebDriverWait(browser, timeout).until(EC.visibility_of(load_more))
#Need bugfix,
for i in range(2):
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
ActionChains(browser).move_to_element(load_more).click().perform()
I noticed when playing around with the load more button that.
<div class="progressbtnwrap" data-search-type="search" style="display: block;">
When the load more button is present on the site, the element is set to "display: block;"
But once the load more button disappears,
<div class="progressbtnwrap" data-search-type="search" style="display: none;">
the element changes to none, notice "display: none;"
Any suggestions how I can search for this?
When looking through the selenium documentations I couldn't find any way of searching for this element and specifically checking if the style is triggered to none,
https://selenium-python.readthedocs.io/locating-elements.html
My goal here is to create something like this
while(True):
if browser.find_element_by_notsurewhat == "block":
ActionChains(browser).move_to_element(load_more).click().perform()
if browser.find_element_by_notsurewhat == "none":
break
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
I'm sure the logic must be much more complicated than that, or even if what I want to achieve is even possible, Any suggestions would be amazing!
Thank you all!
UPDATE:
def load_more(browser):
print("I'm in the function LOAD MORE")
try:
if browser.find_element_by_xpath('//*[#id="mainContent"]/div[1]/div/div[5]/div'):
print("I HAVE ENTERED THE TRY BLOCK WITHIN THE LOAD MORE FUNCTION")
return True
except Exception as e:
print(e)
return False
return False
while load_more(browser):
print("I'm in the while loop!")
ActionChains(browser).move_to_element(load_more).click().perform()
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
When placing my locating and clicking commands, I began to receive the following error:
Traceback (most recent call last):
File "C:\Users\David\eclipse-workspace\Web_Scrap\setup.py", line 81, in <module>
ActionChains(browser).move_to_element(load_more).click().perform()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\action_chains.py", line 83, in perform
action()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\action_chains.py", line 293, in <lambda>
Command.MOVE_TO, {'element': to_element.id}))
AttributeError: 'function' object has no attribute 'id'
I noticed from trying to figure out exactly where the program crashes that, once the code below is run, the program crashes, but this works prior to placing this inside the while loop, or the function. (I tried to place the scrollIntoView, line inside the function right before the try, and I receive a similar error).
ActionChains(browser).move_to_element(load_more).click().perform()
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
The idiomatic way to do this is to use "explicit waits" (AKA WebDriverWait with ExpectedConditions).
The following will wait until the element is no longer visible. If it doesn't disappear in 10 secs, a TimeOutError is raised:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(EC.invisibility_of_element_located(By.CLASS_NAME, 'progressbtnwrap'))
If the style attribute for an html element is set to display: none, selenium won't be able to find the element using the built-in DOM selector functions like find_element_by_id/find_elements_by_class etc.
You could simply wrap the find operation in a try except block and add a delay to allow the browser some time for the Ajax call.
def load_more(browser):
time.sleep(1)
try:
display = browser.execute_script("return document.getElementsByClassName('progressbtnwrap')[0].style.display")
if display == 'none':
return False
elem = browser.find_element_by_xpath('//div[contains(#class, "progressbtnwrap")]/div[contains(#class, "content")]')
browser.execute_script("arguments[0].click();", elem)
return True
except Exception as e:
print("Error")
print(e)
return False
while load_more(browser):
print("scrolling further")
Assuming you are currently just trying to find a way on how you can check the current style of your element you can use this code.
driver.execute_script("return arguments[0].style.display;", load_more)
And you can use to check that when the return value is 'none' for a few seconds which means no more data will be loaded, you can exit your loop.
Working from Test Driven Development with Python, and I'm currently encountering a 'StaleElementReferenceException' when running the functional test immediately after migration. Here's the full text of the error:
ERROR: test_start_and_retrieve_list (__main__.NewVisitorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "functional_tests.py", line 53, in test_start_and_retrieve_list
rows = table.find_elements_by_tag_name('tr')
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 237, in find_elements_by_tag_name
return self.find_elements(by=By.TAG_NAME, value=name)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 527, in find_elements
{"using": by, "value": value})['value']
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <table id="id_list_table"> stale: either the element is no longer attached to the DOM or the page has been refreshed
----------------------------------------------------------------------
Ran 1 test in 8.735s
FAILED (errors=1)
Here's the test:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import unittest
class NewVisitorTest(unittest.TestCase):
def setUp(self):
self.browser = webdriver.Firefox()
self.browser.implicitly_wait(3)
def tearDown(self):
self.browser.close()
def check_for_row(self, row_text):
table = self.browser.find_element_by_id('id_list_table')
rows = table.find_elements_by_tag_name('tr')
self.assertIn(row_text, [row.text for row in rows])
def test_start_and_retrieve_list(self):
self.browser.get('http://localhost:8000')
self.assertIn('To-Do', self.browser.title)
header_text = self.browser.find_element_by_tag_name('h1').text
self.assertIn('To-Do', header_text)
inputbox = self.browser.find_element_by_id('id_new_item')
self.assertEqual(
inputbox.get_attribute('placeholder'),
'Enter a to-do item'
)
inputbox.send_keys('Buy peacock feathers')
inputbox.send_keys(Keys.ENTER)
self.check_for_row('1: Buy peacock feathers')
inputbox = self.browser.find_element_by_id('id_new_item')
inputbox.send_keys('Use peacock feathers to make a fly')
inputbox.send_keys(Keys.ENTER)
table = self.browser.find_element_by_id('id_list_table')
rows = table.find_elements_by_tag_name('tr')
self.check_for_row('1: Buy peacock feathers')
self.check_for_row('2: Use peacock feathers to make a fly')
self.fail('Finish the test!')
if __name__ == '__main__':
unittest.main(warnings='ignore')
How do I configure the test to prevent this? Selenium's own page says this issue can occur when the page refreshes, but this is a necessary part of the application logic as it's configured so far.
Add these imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
Change these lines
inputbox.send_keys(Keys.ENTER)
self.check_for_row('1: Buy peacock feathers')
to:
inputbox.send_keys(Keys.ENTER)
WebDriverWait(self.browser, 10).until(
expected_conditions.text_to_be_present_in_element(
(By.ID, 'id_list_table'), 'Buy peacock feathers'))
self.check_for_row('1: Buy peacock feathers')
This replaces the time.sleep(1) with something more "reasonable"
I have been using selenium for a while now so I understand the struggles of the Stale Element Exception. While not perfect, selenium provides a series of "wait" commands to allow for the website to load complete. Unfortunately, its not perfect as loading can take different time on each run, but these are the tools provided by selenium.
I haven't worked in python but have worked on java/selenium. But,I can give you the idea to overcome staleness.
Generally we will be getting the Stale Exception if the element attributes or something is changed after initiating the webelement. For example, in some cases if user tries to click on the same element on the same page but after page refresh, gets staleelement exception.
To overcome this, we can create the fresh webelement in case if the page is changed or refreshed. Below code can give you some idea.(It's in java but the concept will be same)
Example:
webElement element = driver.findElement(by.xpath("//*[#id='StackOverflow']"));
element.click();
//page is refreshed
element.click();//This will obviously throw stale exception
To overcome this, we can store the xpath in some string and use it create a fresh webelement as we go.
String xpath = "//*[#id='StackOverflow']";
driver.findElement(by.xpath(xpath)).click();
//page has been refreshed. Now create a new element and work on it
driver.fineElement(by.xpath(xpath)).click(); //This works
Another example:
for(int i = 0; i<5; i++)
{
String value = driver.findElement(by.xpath("//.....["+i+"]")).getText);
System.out.println(value);
}
Hope this helps you. Thanks
To prevent an element to become stale, place a new element on the current page, hit the link and wait until the element is not available anymore. Then wait for an element on the new page to appear
script_to_execute = """
var new_element = document.createElement('span');
new_element.setAttribute('id', 'wait4me');
document.body.appendChild(new_element);"""
self.driver.execute_script(script_to_execute)
self.driver.find_element_by_xpath("{}".format(locator)).click()
WebDriverWait(self.driver, self.time_out).until (
lambda x: not x.find_elements_by_id("wait4me"))
This issue happens when the loop starts before an updated page has fully loaded. Especially when you update a page in an application or a form.
One workaround is to place an element on the current page, then update and use the WebDriverWait statement until the element is not found anymore.
Then start your loop. (Otherwise the reload happens during the loop...)
I read the same book as you do and encountered the same problem (solutions from this page didn't work for me).
Here's how I resolved it.
Problem
Exception is thrown whenever you try to access a stale object. So we have to wait for situation when this exception is NOT thrown anymore.
My solution
I created method that waits for my actions until they pass
from selenium.common.exceptions import StaleElementReferenceException
[...]
def stale_aware_for_action(self, action):
while(True):
try:
action()
break
except StaleElementReferenceException:
continue
And in test method I defined actions that I want to wait to finish:
def test_can_start_a_list_and_retrieve_it_later(self):
[...]
def insert_second_item_to_inputbox():
inputbox = self.browser.find_element_by_id('id_new_item')
inputbox.send_keys('Use peacock feathers to make a fly')
inputbox.send_keys(Keys.ENTER)
self.stale_aware_for_action(insert_second_item_to_inputbox)
def check_for_first_item():
self.check_for_row_in_list_table('1: Buy peacock feathers')
def check_for_second_item():
self.check_for_row_in_list_table('2: Use peacock feathers to make a fly')
self.stale_aware_for_action(check_for_first_item)
self.stale_aware_for_action(check_for_second_item)
I try to understand where is the problem in code:
class WebTest(unittest.TestCase):
#classmethod
def setUpClass(cls):
binary = FirefoxBinary('/home/andrew/Downloads/firefox 45/firefox')
cls.browser = webdriver.Firefox(firefox_binary=binary)
cls.wait = WebDriverWait(cls.browser, 10)
cls.browser.maximize_window()
cls.browser.get('http://www.test.com/')
def test_login_menu_elements(self):
self.wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#id='menu_min']"))).click()
check_icons(self)
self.wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#id='menu_min']"))).click()
check_fields(self)
def test_add_news(self):
self.wait.until(EC.element_to_be_clickable((By.XPATH, "//span[contains(.,'News')]"))).click()
self.wait.until(EC.element_to_be_clickable((By.XPATH, "//a[#href='/manager/news']"))).click()
#classmethod
def tearDownClass(cls):
cls.browser.quit()
if __name__=='__main__':
unittest.main()
Every time I receive TimeoutException, and I really don't understand why, and where is the problem in the code
A TimeoutException can be received without having any logical or syntantic errors with your code.
TimeoutExceptions will be raised when the wait.until expected conditions aren't found.
Some things I have found to help:
Isolate the xpath by using chrome/firefox dev tools and right clicking on the element, and show xpath
Using the xpath from the step above, make sure that the condition chose is correct
ime having front end experience, using css selectors is usually more intuative and more understandable than relative xpaths.
check the selector you are using by opening up dev tools console and using $x({{ XPATH_HERE }}) to make sure it is valid
for dynamic HTML use python debugger and make sure that html is in the expected state between each expected condition
I'm trying to take look at several pages on one web with Selenium - PhantomJS().
The problem is that it started freezing and I can't figure out why. It is probably something with Timeout.
Here is the__init__ method of a class.
self.driver = webdriver.PhantomJS(service_args=["--load-images=false"])
self.wait = WebDriverWait(self.driver, 2)
And here is the method:
def click_next_page(self):
log('click_next_page : '+self.driver.current_url) # THIS LINE RUNS
rep = 0
while 1:
try:
self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a'))) # IT MAY FREEZE HERE
self.driver.find_element_by_css_selector('li.arr-rgt.active a').click()# IT MAY FREEZE HERE
print 'NEXT' # DOESNT PRINT ANY TEXT SO THIS LINE NOT EXECUTED
log('NEXT PAGE')
return True
except Exception as e:
log('click next page EXCEPTION') # DONT HAVE THIS TEXT IN MY LOG SO IT DOES NOT RAISES ANY EXCEPTION
self.driver.save_screenshot('click_next_page_exception.png')
self.driver.back()
self.driver.forward()
rep += 1
log('REPEAT '+str(rep))
if rep>4:
break
sleep(4)
return False
The problem is that it does not raises any exception or any message.
The line log('click_next_page : '+self.driver.current_url) is working and then it freezes, I know it because I have click_next_page : http://.... in my log as a last line.
The problem is definitely somewhere here:
self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a')))
self.driver.find_element_by_css_selector('li.arr-rgt.active a').click()
But I can't realize where because it does not raise any Exception.
Could you give me an advice?
I don't have any idea about how Selenium works in PhantomJS. But, I am not seeing any issues within your code. To help you in knowing the exact problem, I would suggest you to debug it in smaller chunks and using one line at a time in console (not by running the python file).
So check with this :-
>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS(service_args=["--load-images=false"])
>>> wait = WebDriverWait(driver, 2)
>>> code for clicking next page
>>> time.sleep(5)
>>> driver.find_element_by_css_selector('li.arr-rgt.active a')
So, this should return you the selenium webdriver instance for the object you are searching using the css selector. If, the element itself is not found then it will throw error.
If the above code runs then re-run the above code with following modifications :-
>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS(service_args=["--load-images=false"])
>>> wait = WebDriverWait(driver, 2)
>>> code for clicking next page
>>> wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a')))
>>> driver.find_element_by_css_selector('li.arr-rgt.active a').click()
Here you will be able to check whether there is actually problem with wait_until(). If there is any error, you can point it out by running it one by one. Hope this helps...
for i in driver.find_elements_by_class_name("endorse-count"):
try:
i.click()
except:
continue
elem = WebDriverWait(driver, 100).until(EC.presence_of_element_located((By.CLASS_NAME, "dialog-window")))
src = elem.get_attribute("innerHTML")
add_skill(name, src)
WebDriverWait(driver, timeout=10)
I'm getting the following error while running the above code -
selenium.common.exceptions.StaleElementReferenceException: Message: u'Element is no longer attached to the DOM' ; Stacktrace:
at fxdriver.cache.getElementAt (resource://fxdriver/modules/web_element_cache.js:7646)
for line -
src = elem.get_attribute("innerHTML")
I'm running this code on LinkedIn user profile page, after logging in.
I tried putting the following line of code after "i.click()" -
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
But then I see that function "add_skill(name, src)" is not called and none of the code after driver.manage() is called, though for loop and further i.click() work fine.
Selenium is trying to complete actions (such as clicking a button or link) before verifying that the target element has rendered on the page. Selenium can be more patient, but you have to explicitly ask him to be.
For example, if you are testing something that makes an AJAX request, you can try something like this (in Ruby):
# timeout is in seconds
def wait_for_ajax(timeout=x)
time_limit, interval = (Time.now + timeout), 0.5
loop do
break if #driver.execute_script "return jQuery.active == 0"
sleep interval
raise "Wait for AJAX timed out after waiting for #{timeout} seconds" if Time.now > time_limit
end
end
To ensure your tests are fully comprehensive, always make Selenium waits for elements to load before running a task.
I had faced a similar issue and tried refreshing the page before finding that element, and it worked...
driver.navigate().refresh();
Though I couldnt reason out how this worked.
If this works for you as well, please let me know. I just want to learn more about this exception.
you can refer this page to learn about a similar issue
I had a similar problem when trying to execute some javascript (IJavaScripExecutor). I created an IWebElement and passed that to the JSE and that failed for me. When I moved the driver.FindElement(BySelector) into my JSE call, then it worked. (C# code ahead.)
Instead of:
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
IWebElement tableEl = driver.FindElement(selector);
js.ExecuteScript(script, tableEl);
I had to do:
IJavaScriptExecutor js = (IJavaScriptExecutor)driver;
js.ExecuteScript(script, driver.FindElement(selector));
You may have to do something similar: move your selector or element creation onto the same line as what you are trying to do. Or, maybe, in your case:
src = driver.find_element_by_class_name("dialog-window").get_attribute("innerHTML")
Upon closer inspection, that's what looks to be your problem, there's a stale web element object when you try to use the get_attribute method.