We Keep Coding

Understanding selenium move_to_element behaviour + StaleElementReference exception

I am using Python3.9+Selenium to write a small script that fills an online form for me.
A bit of context: the webpage contains a field (locationField) expecting a street address as input, and located on top of some sort of "google maps wrapper".
When typing in the field, it loads a drop-down list of one element (locationField_sugg) with the compatible complete address, and when this option is selected the map zooms-in on the chosen part of the city.
Other than this, there is a descriptionField to be filled with some random text, and then a submitButton to be clicked in order to send the form.
I noticed that if I use actionChains.move_to_element(locationField_sugg).click().perform() to click on the address in the drop-down list, then submitButton throws a StaleElementReference exception, while if I just use locationField_sugg.click() this is not the case, and the code proceeds as it should.
I've been reading through many Q/A about this notorious exception handling, but none of them seemed to explain the reason why this happens in my code.
For me it seems to be related to the behaviour of move_to_element() in combination with the "map wrapper" (?) but I do not understand why, since this function is just supposed to move the mouse in the middle of a given element.
No reload nor other changes in the webpage seem to happen (I verified that if I query the button at the beginning and re-query at the end of the script, I get the same exact instance representation string).
Besides, I query the submitButton right before performing an action on it, and I assume it is properly found since it can be printed.
Below there is a snippet of my code and of the output I get (Note: the code works properly if I use the alternative commented option, but I am curious to understand what I am missing)
CODE SNIPPET
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
websiteUrl = "https://mywebsite"
option = webdriver.ChromeOptions()
option.add_argument("-incognito")
browser = webdriver.Chrome(executable_path="/Applications/chromedriver", options=option)
browser.get(websiteUrl)
actionChains = ActionChains(browser)
# Write and select complete address
locationField = browser.find_element_by_id("location")
print("locationField = ", locationField)
locationField.send_keys("my location")
locationField_sugg = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.ID, "as-listbox")))
print("locationField_sugg = ", locationField_sugg)
#
# this throws stale element reference exception:
actionChains.move_to_element(locationField_sugg).click().perform()
#
# this does not:
# locationField_sugg.click()
# Write description
descriptionField = browser.find_element_by_id("description")
print("descriptionField = ", descriptionField)
descriptionField.send_keys("my description")
# Submit form
submitButton = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.value")))
print("submitButton = ", submitButton)
actionChains.move_to_element(submitButton).click().perform()
OUTPUT
locationField = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="e75539a7-ebd0-4090-9c04-0c4994afe03f")>
locationField_sugg = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="53ef0269-aceb-451a-b559-d9e5e7aa7851")>
descriptionField = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="dce478de-a23d-4ca2-817c-81ee5ce0c232")>
nowButton = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="f2036f1c-d164-4211-a37c-2ee50e5c55c1")>
Traceback (most recent call last):
File "/Users/alice/Desktop/wasteComplaints_selenium.py", line 44, in <module>
actionChains.move_to_element(nowButton).click().perform()
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/common/action_chains.py", line 80, in perform
self.w3c_actions.perform()
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/common/actions/action_builder.py", line 76, in perform
self.driver.execute(Command.W3C_ACTIONS, enc)
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=92.0.4515.131)

I'm not sure I know why this occurs without debugging it. Maybe the suggested address is disappearing at the moment when focus is removed from the input field? If so, when you insert the address string to the input field and then instantly clicking on the suggested address it works correct, but if after inserting the input address string and the suggested address appears you are applying actionChains.move_to_element this moves the mouse from it's initial position to the suggested address element. So the focus is moved from input field to the mouse cursor. This causes the suggested address to disappear and this is why StaleElementReference exception is thrown.

Selenium python, Click only under certain conditions

My code works, but not in all cases
Basically the functionality is to click a load_more button until it no longer appears.
As of right now, I simply have a loop which finds the loadmore button and clicks it twice, but there are cases that it will click on something else when the load more button disappears.
I was planning on making a while loop, which would constantly find the click the load_more option until the loadmore disappeared then break the loop.
Here is the code: (This simply finds and clicks it twice)
load_more = browser.find_element_by_css_selector("#mainContent > div.left-panel > div > div.result-list > div > div.content")
WebDriverWait(browser, timeout).until(EC.visibility_of(load_more))
#Need bugfix,
for i in range(2):
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
ActionChains(browser).move_to_element(load_more).click().perform()
I noticed when playing around with the load more button that.
<div class="progressbtnwrap" data-search-type="search" style="display: block;">
When the load more button is present on the site, the element is set to "display: block;"
But once the load more button disappears,
<div class="progressbtnwrap" data-search-type="search" style="display: none;">
the element changes to none, notice "display: none;"
Any suggestions how I can search for this?
When looking through the selenium documentations I couldn't find any way of searching for this element and specifically checking if the style is triggered to none,
https://selenium-python.readthedocs.io/locating-elements.html
My goal here is to create something like this
while(True):
if browser.find_element_by_notsurewhat == "block":
ActionChains(browser).move_to_element(load_more).click().perform()
if browser.find_element_by_notsurewhat == "none":
break
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
I'm sure the logic must be much more complicated than that, or even if what I want to achieve is even possible, Any suggestions would be amazing!
Thank you all!
UPDATE:
def load_more(browser):
print("I'm in the function LOAD MORE")
try:
if browser.find_element_by_xpath('//*[#id="mainContent"]/div[1]/div/div[5]/div'):
print("I HAVE ENTERED THE TRY BLOCK WITHIN THE LOAD MORE FUNCTION")
return True
except Exception as e:
print(e)
return False
return False
while load_more(browser):
print("I'm in the while loop!")
ActionChains(browser).move_to_element(load_more).click().perform()
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)
When placing my locating and clicking commands, I began to receive the following error:
Traceback (most recent call last):
File "C:\Users\David\eclipse-workspace\Web_Scrap\setup.py", line 81, in <module>
ActionChains(browser).move_to_element(load_more).click().perform()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\action_chains.py", line 83, in perform
action()
File "C:\Users\David\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\common\action_chains.py", line 293, in <lambda>
Command.MOVE_TO, {'element': to_element.id}))
AttributeError: 'function' object has no attribute 'id'
I noticed from trying to figure out exactly where the program crashes that, once the code below is run, the program crashes, but this works prior to placing this inside the while loop, or the function. (I tried to place the scrollIntoView, line inside the function right before the try, and I receive a similar error).
ActionChains(browser).move_to_element(load_more).click().perform()
browser.execute_script("return arguments[0].scrollIntoView(true);", load_more)

The idiomatic way to do this is to use "explicit waits" (AKA WebDriverWait with ExpectedConditions).
The following will wait until the element is no longer visible. If it doesn't disappear in 10 secs, a TimeOutError is raised:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = WebDriverWait(driver, 10)
element = wait.until(EC.invisibility_of_element_located(By.CLASS_NAME, 'progressbtnwrap'))

If the style attribute for an html element is set to display: none, selenium won't be able to find the element using the built-in DOM selector functions like find_element_by_id/find_elements_by_class etc.
You could simply wrap the find operation in a try except block and add a delay to allow the browser some time for the Ajax call.
def load_more(browser):
time.sleep(1)
try:
display = browser.execute_script("return document.getElementsByClassName('progressbtnwrap')[0].style.display")
if display == 'none':
return False
elem = browser.find_element_by_xpath('//div[contains(#class, "progressbtnwrap")]/div[contains(#class, "content")]')
browser.execute_script("arguments[0].click();", elem)
return True
except Exception as e:
print("Error")
print(e)
return False
while load_more(browser):
print("scrolling further")

Assuming you are currently just trying to find a way on how you can check the current style of your element you can use this code.
driver.execute_script("return arguments[0].style.display;", load_more)
And you can use to check that when the return value is 'none' for a few seconds which means no more data will be loaded, you can exit your loop.

"Element is not clickable" error in selenium python

I have been using the same scripts for over a year, but since yesterday i am getting this error when click on links having images. I am getting the element by xpath and then clicking it.
test_101_HomePage_links (__main__.SprintTests) ... ERROR
======================================================================
ERROR: test_101_HomePage_links (__main__.SprintTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "D:\Zaakpay\website\sanity results\debug\tests.py", line 17, in test_101_HomePage_links
a.click()
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py", line 73, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py", line 456, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
WebDriverException: Message: Element is not clickable at point (281.25, 61). Other element would receive the click: <span></span>
----------------------------------------------------------------------
Ran 1 test in 23.062s
FAILED (errors=1)
Other info:
using windows,
using python,
using firefox,
same script was working fine till yesterday
My code:
import unittest
from selenium import webdriver
import datetime
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import time
class SprintTests(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.driver.maximize_window()
self.driver.get("https://www.zaakpay.com")
def test_101_HomePage_links(self):
a= self.driver.find_element_by_xpath("/html/body/div[5]/div[1]/div[3]/ul/li[1]/a/i")
a.click()
time.sleep(5)
a = self.driver.find_element_by_xpath('//*[#id="view1"]/p')
b=a.text
self.assertEqual('-Payment Gateway Services.\n-More than you want payment options with.\n-major credit cards, debit cards and 52 netbanking banks.\n-Fastest Merchant Approval.\n-Smooth integration across 22 platforms.\n-Start in minutes.\n-Multi-Currency Processing Service with 13 currencies.\n\nSIGN UP',b )
def tearDown(self):
self.driver.quit()
if __name__ == '__main__':
unittest.main(verbosity=2)
link i am trying to click is circular image above text- WEBSITE PAYMENT GATEWAY

Using XPATHs like this /html/body/div[5]/div[1]/div[3]/ul/li[1]/a/i are problematic for a number of reasons.
First and foremost, they are extremely brittle. All it takes is a minor update to the site, one that might night even be visually perceptible, and the hard coded div indexes break. The site devs might have added a div to slightly alter a font style and then your XPATH is completely broken.
Second, it makes it next to impossible to debug once it does break. I have absolutely no idea what you were intending to click on, and if this were production code that someone else was attempting to fix, they would also have no idea what you were intending to click on.
My best guess is that zaakpay made some small, imperceptible change that slightly changed the location of your target element in the DOM. If you update the XPATH to something like //div[#class=\"prdtBlck\"]/ul/li/a, as I've done below, your script works:
import unittest
from selenium import webdriver
import datetime
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import time
class SprintTests(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.driver.maximize_window()
self.driver.get("https://www.zaakpay.com")
def test_101_HomePage_links(self):
a=self.driver.find_element_by_xpath("//div[#class=\"prdtBlck\"]/ul/li/a")
a.click()
time.sleep(5)
a = self.driver.find_element_by_xpath('//*[#id="view1"]/p')
b=a.text
self.assertEqual('-Payment Gateway Services.\n-More than you want payment options with.\n-major credit cards, debit cards and 52 netbanking banks.\n-Fastest Merchant Approval.\n-Smooth integration across 22 platforms.\n-Start in minutes.\n-Multi-Currency Processing Service with 13 currencies.\n\nSIGN UP',b )
def tearDown(self):
self.driver.quit()
if __name__ == '__main__':
unittest.main(verbosity=2)

How to find_element_by_link_text while having: NoSuchElement Exception?

This question has been asked over and over again - and in-spite of trying all the hacks I still can't seem to figure out what's wrong.
I tried increasing the implicitly_wait to 30 (and even increased it upto 100) - yet it did not work.
Use case -: I am trying to create a list that wil populate all the items in the page here, as a base case - and I intend to bind this to a mini-module that I already have with scrapy which has all (pages with similar web elements) crawled links - so essentially will be building the whole pipeline, post I am done with this.
###My source code - generated via Selenium IDE, exported to a Python webdriver and manipulated a little later ###
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.wait import WebDriverWait
import unittest, time, re
class Einstein(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.base_url = "http://shopap.lenovo.com/in/en/laptops/"
self.verificationErrors = []
self.accept_next_alert = True
def test_einstein(self):
driver = self.driver
driver.get(self.base_url)
print driver.title
driver.find_element_by_link_text("T430").click()
print driver.title
# driver.find_element_by_xpath("id('facetedBrowseWrapper')/div/div/div[1]/div[2]/ul[1]/li[1]/a").click()
driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
print driver.title
# driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
try: self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
except AssertionError as e: self.verificationErrors.append(str(e))
# Everything ok till here
#**THE CODE FAILS HERE**#
laptop1 = driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text
print laptop1
price1 = driver.find_element_by_css_selector("span.price").text
print price1
detail1 = self.is_element_present(By.CSS_SELECTOR, "div.desc.std")
print detail1
def is_element_present(self, how, what):
try: self.driver.find_element(by=how, value=what)
except NoSuchElementException, e: return False
return True
def is_alert_present(self):
try: self.driver.switch_to_alert()
except NoAlertPresentException, e: return False
return True
def close_alert_and_get_its_text(self):
try:
alert = self.driver.switch_to_alert()
alert_text = alert.text
if self.accept_next_alert:
alert.accept()
else:
alert.dismiss()
return alert_text
finally: self.accept_next_alert = True
def tearDown(self):
self.driver.quit()
self.assertEqual([], self.verificationErrors)
if __name__ == "__main__":
unittest.main()
Errors & output :
ekta#ekta-VirtualBox:~$ python einstein.py
Laptops & Ultrabooks | Lenovo (IN)
ThinkPad T430 Laptop PC for Business Computing | Lenovo (IN)
Buy Lenovo Thinkpad Laptops | Lenovo Thinkpad Laptops Price India
E
======================================================================
ERROR: test_einstein (__main__.Einstein)
----------------------------------------------------------------------
Traceback (most recent call last):
File "einstein.py", line 27, in test_einstein
try: self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 246, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 680, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 165, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 158, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: u'Unable to locate element: {"method":"link text","selector":"Thinkpad Edge E530 (Black)"}' ; Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmphli5Jg/extensions/fxdriver#googlecode.com/components/driver_component.js:8444)
at fxdriver.Timer.prototype.setTimeout/<.notify (file:///tmp/tmphli5Jg/extensions/fxdriver#googlecode.com/components/driver_component.js:386)
----------------------------------------------------------------------
Ran 1 test in 79.348s
FAILED (errors=1)
Questions & comments:
If you are answering this question - please mention why this specific "find_element_by_link_text" does not work.
(Very Basic) In the GUI of my selenium IDE -> Show all available commands - why dont I see the css (find_element_by_css_selector) for all the web elements - is there a way to force feed an element to be read as a CSS selector ?
In case you suggest using some other locator - please mention if that will be consistent way to fetch elements, given (1)
Does assert work to capture the exceptions and "move on" - since even after trying "verify" , "assert" loops, I still cant fetch this "find_element_by_link_text"
I tried using Xpath to build this "element" , but in the view Xpath (in firefox) - I see nothing, to clue why that happens (Of course I removed the namespace ":x" )
Other things I tried apart from implicity_wait(30):
find_element_by_partial_link(“Thinkpad”) and appending Unicode to this (wasn’t sure if it was reading the brackets ( , driver.find_element_by_link_text(u"Thinkpad Edge E530 (Black)").text, still did not work.
Related questions:
How to use find_element_by_link_text() properly to not raise NoSuchElementException?
NoSuchElement Exception using find_element_by_link_text when implicitly_wait doesn't work?

It happened to me before that the find_element_by_link_text method sometimes works and sometimes doesn't work; even in a single case. I think it's not a reliable way to access elements; the best way is to use find_element_by_id.
But in your case, as I visit the page, there is no id to help you. Still you can try find_elements_by_xpath in 3 ways:
1- Accessing title: find_element_by_xpath["//a[contains(#title = 'T430')]"]
2- Accessing text: find_element_by_xpath["//a[contains(text(), 'T430')]"]
3- Accessing href: find_element_by_xpath["//a[contains(#href = 'http://www.thedostore.com/laptops/thinkpad-laptops/thinkpad-t430-u-black-627326q.html')]"].
Hope it helps.

NoSuchElementException is thrown when the element could not be found.
If you encounter this exception, please check the followings:
Check your selector used in your find_by...
Element may not yet be on the screen at the time of the find operation.
If webpage is still loading, check for selenium.webdriver.support.wait.WebDriverWait() and write a wait wrapper to wait for an element to appear.
Troubleshooting and code samples
You can add breakpoint just before your failing line pdb.set_trace() (don't forget to import pdb), then run your test and once your debugger stops, then do the following tests.
You could try:
driver.find_element_by_xpath(u'//a[text()="Foo text"]')
instead. This is more reliable test, so if this would work, use it instead.
If above won't help, please check if your page has been loaded properly via:
(Pdb) driver.execute_script("return document.readyState")
'complete'
Sometimes when the page is not loaded, you're actually fetching the elements from the old page. But even though, readyState could still indicate the state of the old page (especially when using click()). Here is how this is explained in this blog:
Since Selenium webdriver has become more advanced, clicks are much more like "real" clicks, which has the benefit of making our tests more realistic, but it also means it's hard for Selenium to be able to track the impact that a click has on the browsers' internals -- it might try to poll the browser for its page-loaded status immediately after clicking, but that's open to a race condition where the browser was multitasking, hasn't quite got round to dealing with the click yet, and it gives you the .readyState of the old page.
If you think this is happening because the page wasn't loaded properly, the "recommended" (however still ugly) solution is an explicit wait:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
old_value = browser.find_element_by_id('thing-on-old-page').text
browser.find_element_by_link_text('my link').click()
WebDriverWait(browser, 3).until(
expected_conditions.text_to_be_present_in_element(
(By.ID, 'thing-on-new-page'),
'expected new text'
)
)
The naive attempt would be something like this:
def wait_for(condition_function):
start_time = time.time()
while time.time() < start_time + 3:
if condition_function():
return True
else:
time.sleep(0.1)
raise Exception(
'Timeout waiting for {}'.format(condition_function.__name__)
)
def click_through_to_new_page(link_text):
browser.find_element_by_link_text('my link').click()
def page_has_loaded():
page_state = browser.execute_script(
'return document.readyState;'
)
return page_state == 'complete'
wait_for(page_has_loaded)
Another, better one would be (credits to #ThomasMarks):
def click_through_to_new_page(link_text):
link = browser.find_element_by_link_text('my link')
link.click()
def link_has_gone_stale():
try:
# poll the link with an arbitrary call
link.find_elements_by_id('doesnt-matter')
return False
except StaleElementReferenceException:
return True
wait_for(link_has_gone_stale)
And the final example includes comparing page ids as below (which could be bulletproof):
class wait_for_page_load(object):
def __init__(self, browser):
self.browser = browser
def __enter__(self):
self.old_page = self.browser.find_element_by_tag_name('html')
def page_has_loaded(self):
new_page = self.browser.find_element_by_tag_name('html')
return new_page.id != self.old_page.id
def __exit__(self, *_):
wait_for(self.page_has_loaded)
And now we can do:
with wait_for_page_load(browser):
browser.find_element_by_link_text('my link').click()
Above code samples are from Harry's blog.
Here is the version proposed by Tommy Beadle (by using staleness approach):
import contextlib
from selenium.webdriver import Remote
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import staleness_of
class MyRemote(Remote):
#contextlib.contextmanager
def wait_for_page_load(self, timeout=30):
old_page = self.find_element_by_tag_name('html')
yield
WebDriverWait(self, timeout).until(staleness_of(old_page))
If you think it isn't about page load, double check if your element isn't in iframe or different window. If so, you've to switch to it first. To check list of available windows, run: driver.window_handles.

From viewing the source of the page that you provided a link to, it seems you are using an incorrect selector.
You should use instead find_elements_by_link_text(u'text here')[0] to select the first occurrence instead as there seems to be the potential for multiple links with the same link text.
So instead of:
self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
You should use:
self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_elements_by_link_text("Thinkpad Edge E530 (Black)")[0].text)

Solution posted by OP:
Hack 1: Instead of identifying the element as a text-link, I identified the "bigger frame" in which this element was present.
itemlist_1 = driver.find_element_by_css_selector("li.item.first").text
This will give the whole item along with the name, price and detail (and the unwanted add to cart and compare"
See the attached image for more .
Hack 2: I found that the "Buy Now" which was an image element with xPath (driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
, in the code above) , could be made to click/identified faster if I added the following line, before finding this by xpath. I think this sort of narrows down where the Webdriver is looking for an element. This is what I added " driver.find_element_by_css_selector("#subseries").text"
This must have decreased my wait by at least 20 seconds, on that page .Hope that helps.

Click on a dropdown element menu with Selenium Webdriver

I'm trying to automate an administration task, so far I have made selenium to click on an element to show a dropdown menu.
When it comes the time to click on one of those menu elements I've got an error saying that the element must be displayed.
Code:
driver = webdriver.Chrome()
driver.implicitly_wait(10)
driver.get(url)
doc = driver.find_element_by_css_selector('td.ms-vb-title > table')
try:
doc.click()
time.sleep(4)
menu = driver.find_element_by_xpath('//menu/span[5]')
time.sleep(4)
print dir(menu)
menu.click()
except:
traceback.print_exc()
driver.quit()
Error:
Traceback (most recent call last):
File "aprobar_docs.py", line 22, in main
menu.click()
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 52, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 205, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 156, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py"
, line 147, in check_response
raise exception_class(message, screen, stacktrace)
ElementNotVisibleException: Message: u'Element must be displayed to click'
As you can see the code waits a lot to get the element loaded. I've also tried to set the element's is_displayed property to True but didn't work neither.
Note: the element that's not displayed is the one on the xpath search, it is present because I've logged it with dir(menu)
Edit:
The menu variable is not the menu itself it's one of the spans that are elements of the menu, doc is the Perfil html element getting clicked to show the dropdown.
Edit 2:
Inspecting the DOM on chrome tools, when you click a doc a new menu gets created on the tree, I don't know if it's because of an ajax call or vanilla js, I don't think it's really that important how it's created. I can't retrieve it from the page and make a python object from it, it's just not being displayed at least on code.
Final Edit:
I ended up executing some JavaScript to make it work. Apparently when Selenium finds the menu item the first element that triggers the menu drop down loses the focus and it makes the menu invisible again, if you don't select a menu item and wait for some time the menu dropdown still is shown, if you try to select one element from the menu the menu disappears.

Why don't you select an option like this
el = driver.find_element_by_id('id_of_select')
for option in el.find_elements_by_tag_name('option'):
if option.text == 'The Options I Am Looking For':
option.click() # select() in earlier versions of webdriver
If your click is not firing an ajax call to populate your list, you don't actually need to execute the click.

You need to find the link of the target. You don't really click elements, you click links... (or rather, you click elements with links inside them). That being said, the most sure fire way to click a link is to isolate the link element.
frame = driver.find_element_by_id('this_is_your_frame_name')
links = frame.find_elements_by_xpath('.//a')
links[1].click()
or alternatively,
for link in links:
if link.text() == "Administratar Permisos":
link.click()