We Keep Coding

ElementNotInteractableException: Message: Element <button class="btn btn-outline-light btn-icons" type="button"> could not be scrolled into view

I am trying to press a the download on this page
https://data.unwomen.org/data-portal/sdg?annex=All&finic[]=SUP_1_1_IPL_P&flocat[]=478&flocat[]=174&flocat[]=818&flocat[]=504&flocat[]=729&flocat[]=788&flocat[]=368&flocat[]=400&flocat[]=275&flocat[]=760&fys[]=2015&fyr[]=2030&fca[ALLAGE]=ALLAGE&fca[<15Y]=<15Y&fca[15%2B]=15%2B&fca[15-24]=15-24&fca[25-34]=25-34&fca[35-54]=35-54&fca[55%2B]=55%2B&tab=table
i am using python selenium with firefox and this is what i tried:
driver.set_page_load_timeout(30)
driver.get(url)
time.sleep(1)
WebDriverWait(driver, timeout=20).until(EC.presence_of_element_located((By.ID, 'SDG-Indicator-Dashboard')))
time.sleep(1)
download_div = driver.find_element(By.CLASS_NAME, 'float-buttons-wrap')
buttons = download_div.find_elements(By.TAG_NAME, 'button')
buttons_attributes = [i.get_attribute('title') for i in buttons]
download_button_index = buttons_attributes.index('Download')
buttons[download_button_index].location_once_scrolled_into_view
buttons[download_button_index].click()```
i keep getting the same error:
ElementNotInteractableException: Message: Element <button class="btn btn-outline-light btn-icons" type="button"> could not be scrolled into view
eventho i am getting the correct element and i tried using js like this:
```driver.execute_script("return arguments[0].scrollIntoView(true);", element)```
also did not work.

You have to modify the XPath, try the below code:
driver.get("https://data.unwomen.org/data-portal/sdg?annex=All&finic[]=SUP_1_1_IPL_P&flocat[]=478&flocat[]=174&flocat[]=818&flocat[]=504&flocat[]=729&flocat[]=788&flocat[]=368&flocat[]=400&flocat[]=275&flocat[]=760&fys[]=2015&fyr[]=2030&fca[ALLAGE]=ALLAGE&fca[<15Y]=<15Y&fca[15%2B]=15%2B&fca[15-24]=15-24&fca[25-34]=25-34&fca[35-54]=35-54&fca[55%2B]=55%2B&tab=table")
driver.implicitly_wait(15)
time.sleep(2)
download_btn = driver.find_element(By.XPATH, "(.//button[#type='button' and #title='Download'])[2]")
download_btn.location_once_scrolled_into_view
time.sleep(1)
download_btn.click()

1.You need to make sure you are giving enough time to complete page loading
2.Once you started to find button element , it's better to use try/catch blocks multiple times and put sleep function when exception occures to provide appropriate time to load scripts and elements.
3.Try finding by xpath instead of finding elements by indexes and be aware that you need to use the index [1] instead of [0] in a xpath string

Blocking login overlay window when scraping web page using Selenium

I am trying to scrape a long list of books in 10 web pages. When the loop clicks on next > button for the first time the website displays a login overlay so selenium can not find the target elements.
I have tried all the possible solutions:
Use some chrome options.
Use try-except to click X button on the overlay. But it appears only one time (when clicking next > for the first time). The problem is that when I put this try-except block at the end of while True: loop, it became infinite as I use continue in except as I do not want to break the loop.
Add some popup blocker extensions to Chrome but they do not work when I run the code although I add the extension using options.add_argument('load-extension=' + ExtensionPath).
This is my code:
options = Options()
options.add_argument('start-maximized')
options.add_argument('disable-infobars')
options.add_argument('disable-avfoundation-overlays')
options.add_argument('disable-internal-flash')
options.add_argument('no-proxy-server')
options.add_argument("disable-notifications")
options.add_argument("disable-popup")
Extension = (r'C:\Users\DELL\AppData\Local\Google\Chrome\User Data\Profile 1\Extensions\ifnkdbpmgkdbfklnbfidaackdenlmhgh\1.1.9_0')
options.add_argument('load-extension=' + Extension)
options.add_argument('--disable-overlay-scrollbar')
driver = webdriver.Chrome(options=options)
driver.get('https://www.goodreads.com/list/show/32339._50_?page=')
wait = WebDriverWait(driver, 2)
review_dict = {'title':[], 'author':[],'rating':[]}
html_soup = BeautifulSoup(driver.page_source, 'html.parser')
prod_containers = html_soup.find_all('table', class_ = 'tableList js-dataTooltip')
while True:
table = driver.find_element_by_xpath('//*[#id="all_votes"]/table')
for product in table.find_elements_by_xpath(".//tr"):
for td in product.find_elements_by_xpath('.//td[3]/a'):
title = td.text
review_dict['title'].append(title)
for td in product.find_elements_by_xpath('.//td[3]/span[2]'):
author = td.text
review_dict['author'].append(author)
for td in product.find_elements_by_xpath('.//td[3]/div[1]'):
rating = td.text[0:4]
review_dict['rating'].append(rating)
try:
close = wait.until(EC.element_to_be_clickable((By.XPATH, '/html/body/div[3]/div/div/div[1]/button')))
close.click()
except NoSuchElementException:
continue
try:
element = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, 'next_page')))
element.click()
except TimeoutException:
break
df = pd.DataFrame.from_dict(review_dict)
df
Any help like if I can change the loop to for loop clicks next > button until the end rather than while loop or where should I put try-except block to close the overlay or if there is Chromeoption can disable overlay.
Thanks in advance

Thank you for sharing your code and the website that you are having trouble with. I was able to close the Login Modal by using xpath. I took this challenge and broke up the code using class objects. 1 object is for the selenium.webdriver.chrome.webdriver and the other object is for the page that you wanted to scrape the data against ( https://www.goodreads.com/list/show/32339 ). In the following methods, I used the Javascript return arguments[0].scrollIntoView(); method and was able to scroll to the last book that displayed on the page. After I did that, I was able to click the next button
def scroll_to_element(self, xpath : str):
element = self.chrome_driver.find_element(By.XPATH, xpath)
self.chrome_driver.execute_script("return arguments[0].scrollIntoView();", element)
def get_book_count(self):
return self.chrome_driver.find_elements(By.XPATH, "//div[#id='all_votes']//table[contains(#class, 'tableList')]//tbody//tr").__len__()
def click_next_page(self):
# Scroll to last record and click "next page"
xpath = "//div[#id='all_votes']//table[contains(#class, 'tableList')]//tbody//tr[{0}]".format(self.get_book_count())
self.scroll_to_element(xpath)
self.chrome_driver.find_element(By.XPATH, "//div[#id='all_votes']//div[#class='pagination']//a[#class='next_page']").click()
Once I clicked on the "Next" button, I saw the modal display. I was able to find the xpath for the modal and was able to close the modal.
def is_displayed(self, xpath: str, int = 5):
try:
webElement = DriverWait(self.chrome_driver, int).until(
DriverConditions.presence_of_element_located(locator = (By.XPATH, xpath))
)
return True if webElement != None else False
except:
return False
def is_modal_displayed(self):
return self.is_displayed("//body[#class='modalOpened']")
def close_modal(self):
self.chrome_driver.find_element(By.XPATH, "//div[#class='modal__content']//div[#class='modal__close']").click()
if(self.is_modal_displayed()):
raise Exception("Modal Failed To Close")
I hope this helps you to solve your problem.

selenium: wait for element to be clickable not working

I'm attempting to use expected_conditions.element_to_be_clickable but it doesn't appear to be working. I'm still seeing "Element...is not clickable at point" errors in about 30% of the runs.
Here's the full error message:
selenium.common.exceptions.WebDriverException: Message: unknown
error: Element ... is not clickable at point (621, 337). Other
element would receive the click: ...
(Session info: chrome=60.0.3112.90)
(Driver info: chromedriver=2.26.436421 (6c1a3ab469ad86fd49c8d97ede4a6b96a49ca5f6),platform=Mac OS X 10.12.6
x86_64)
Here's the code I'm working with:
def wait_for_element_to_be_clickable(selector, timeout=10):
global driver
wd_wait = WebDriverWait(driver, timeout)
wd_wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, selector)),
'waiting for element to be clickable ' + selector)
print ('WAITING')
return driver.find_element_by_css_selector(selector)
Update:
So now this is really odd. Even if I add a couple of fixed wait periods, it still throws the error message occasionally. Here's the code where the call is being made:
sleep(5)
elem = utils.wait_for_element_to_be_clickable('button.ant-btn-primary')
sleep(5)
elem.click()

Ended up creating my own custom function to handle the exception and perform retries:
""" custom clickable wait function that relies on exceptions. """
def custom_wait_clickable_and_click(selector, attempts=5):
count = 0
while count < attempts:
try:
wait(1)
# This will throw an exception if it times out, which is what we want.
# We only want to start trying to click it once we've confirmed that
# selenium thinks it's visible and clickable.
elem = wait_for_element_to_be_clickable(selector)
elem.click()
return elem
except WebDriverException as e:
if ('is not clickable at point' in str(e)):
print('Retrying clicking on button.')
count = count + 1
else:
raise e
raise TimeoutException('custom_wait_clickable timed out')

The problem is stated in the error message.
Element ... is not clickable at point (621, 337). Other element would receive the click: ...
The problem is that some element, the details of which you removed from the error message, is in the way... on top of the element you are trying to click. In many cases, this is some dialog or some other UI element that is in the way. How to deal with this depends on the situation. If it's a dialog that is open, close it. If it's a dialog that you closed but the code is running fast, wait for some UI element of the dialog to be invisible (dialog is closed and no longer visible) then attempt the click. Generally it's just about reading the HTML of the element that is blocking, find it in the DOM, and figure out how to wait for it to disappear, etc.

Selenium WebDriverWait does not work properly?

I'm trying to take look at several pages on one web with Selenium - PhantomJS().
The problem is that it started freezing and I can't figure out why. It is probably something with Timeout.
Here is the__init__ method of a class.
self.driver = webdriver.PhantomJS(service_args=["--load-images=false"])
self.wait = WebDriverWait(self.driver, 2)
And here is the method:
def click_next_page(self):
log('click_next_page : '+self.driver.current_url) # THIS LINE RUNS
rep = 0
while 1:
try:
self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a'))) # IT MAY FREEZE HERE
self.driver.find_element_by_css_selector('li.arr-rgt.active a').click()# IT MAY FREEZE HERE
print 'NEXT' # DOESNT PRINT ANY TEXT SO THIS LINE NOT EXECUTED
log('NEXT PAGE')
return True
except Exception as e:
log('click next page EXCEPTION') # DONT HAVE THIS TEXT IN MY LOG SO IT DOES NOT RAISES ANY EXCEPTION
self.driver.save_screenshot('click_next_page_exception.png')
self.driver.back()
self.driver.forward()
rep += 1
log('REPEAT '+str(rep))
if rep>4:
break
sleep(4)
return False
The problem is that it does not raises any exception or any message.
The line log('click_next_page : '+self.driver.current_url) is working and then it freezes, I know it because I have click_next_page : http://.... in my log as a last line.
The problem is definitely somewhere here:
self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a')))
self.driver.find_element_by_css_selector('li.arr-rgt.active a').click()
But I can't realize where because it does not raise any Exception.
Could you give me an advice?

I don't have any idea about how Selenium works in PhantomJS. But, I am not seeing any issues within your code. To help you in knowing the exact problem, I would suggest you to debug it in smaller chunks and using one line at a time in console (not by running the python file).
So check with this :-
>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS(service_args=["--load-images=false"])
>>> wait = WebDriverWait(driver, 2)
>>> code for clicking next page
>>> time.sleep(5)
>>> driver.find_element_by_css_selector('li.arr-rgt.active a')
So, this should return you the selenium webdriver instance for the object you are searching using the css selector. If, the element itself is not found then it will throw error.
If the above code runs then re-run the above code with following modifications :-
>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS(service_args=["--load-images=false"])
>>> wait = WebDriverWait(driver, 2)
>>> code for clicking next page
>>> wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a')))
>>> driver.find_element_by_css_selector('li.arr-rgt.active a').click()
Here you will be able to check whether there is actually problem with wait_until(). If there is any error, you can point it out by running it one by one. Hope this helps...

How to find_element_by_link_text while having: NoSuchElement Exception?

This question has been asked over and over again - and in-spite of trying all the hacks I still can't seem to figure out what's wrong.
I tried increasing the implicitly_wait to 30 (and even increased it upto 100) - yet it did not work.
Use case -: I am trying to create a list that wil populate all the items in the page here, as a base case - and I intend to bind this to a mini-module that I already have with scrapy which has all (pages with similar web elements) crawled links - so essentially will be building the whole pipeline, post I am done with this.
###My source code - generated via Selenium IDE, exported to a Python webdriver and manipulated a little later ###
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support.wait import WebDriverWait
import unittest, time, re
class Einstein(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.base_url = "http://shopap.lenovo.com/in/en/laptops/"
self.verificationErrors = []
self.accept_next_alert = True
def test_einstein(self):
driver = self.driver
driver.get(self.base_url)
print driver.title
driver.find_element_by_link_text("T430").click()
print driver.title
# driver.find_element_by_xpath("id('facetedBrowseWrapper')/div/div/div[1]/div[2]/ul[1]/li[1]/a").click()
driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
print driver.title
# driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
try: self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
except AssertionError as e: self.verificationErrors.append(str(e))
# Everything ok till here
#**THE CODE FAILS HERE**#
laptop1 = driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text
print laptop1
price1 = driver.find_element_by_css_selector("span.price").text
print price1
detail1 = self.is_element_present(By.CSS_SELECTOR, "div.desc.std")
print detail1
def is_element_present(self, how, what):
try: self.driver.find_element(by=how, value=what)
except NoSuchElementException, e: return False
return True
def is_alert_present(self):
try: self.driver.switch_to_alert()
except NoAlertPresentException, e: return False
return True
def close_alert_and_get_its_text(self):
try:
alert = self.driver.switch_to_alert()
alert_text = alert.text
if self.accept_next_alert:
alert.accept()
else:
alert.dismiss()
return alert_text
finally: self.accept_next_alert = True
def tearDown(self):
self.driver.quit()
self.assertEqual([], self.verificationErrors)
if __name__ == "__main__":
unittest.main()
Errors & output :
ekta#ekta-VirtualBox:~$ python einstein.py
Laptops & Ultrabooks | Lenovo (IN)
ThinkPad T430 Laptop PC for Business Computing | Lenovo (IN)
Buy Lenovo Thinkpad Laptops | Lenovo Thinkpad Laptops Price India
E
======================================================================
ERROR: test_einstein (__main__.Einstein)
----------------------------------------------------------------------
Traceback (most recent call last):
File "einstein.py", line 27, in test_einstein
try: self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 246, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 680, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 165, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 158, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: u'Unable to locate element: {"method":"link text","selector":"Thinkpad Edge E530 (Black)"}' ; Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmphli5Jg/extensions/fxdriver#googlecode.com/components/driver_component.js:8444)
at fxdriver.Timer.prototype.setTimeout/<.notify (file:///tmp/tmphli5Jg/extensions/fxdriver#googlecode.com/components/driver_component.js:386)
----------------------------------------------------------------------
Ran 1 test in 79.348s
FAILED (errors=1)
Questions & comments:
If you are answering this question - please mention why this specific "find_element_by_link_text" does not work.
(Very Basic) In the GUI of my selenium IDE -> Show all available commands - why dont I see the css (find_element_by_css_selector) for all the web elements - is there a way to force feed an element to be read as a CSS selector ?
In case you suggest using some other locator - please mention if that will be consistent way to fetch elements, given (1)
Does assert work to capture the exceptions and "move on" - since even after trying "verify" , "assert" loops, I still cant fetch this "find_element_by_link_text"
I tried using Xpath to build this "element" , but in the view Xpath (in firefox) - I see nothing, to clue why that happens (Of course I removed the namespace ":x" )
Other things I tried apart from implicity_wait(30):
find_element_by_partial_link(“Thinkpad”) and appending Unicode to this (wasn’t sure if it was reading the brackets ( , driver.find_element_by_link_text(u"Thinkpad Edge E530 (Black)").text, still did not work.
Related questions:
How to use find_element_by_link_text() properly to not raise NoSuchElementException?
NoSuchElement Exception using find_element_by_link_text when implicitly_wait doesn't work?

It happened to me before that the find_element_by_link_text method sometimes works and sometimes doesn't work; even in a single case. I think it's not a reliable way to access elements; the best way is to use find_element_by_id.
But in your case, as I visit the page, there is no id to help you. Still you can try find_elements_by_xpath in 3 ways:
1- Accessing title: find_element_by_xpath["//a[contains(#title = 'T430')]"]
2- Accessing text: find_element_by_xpath["//a[contains(text(), 'T430')]"]
3- Accessing href: find_element_by_xpath["//a[contains(#href = 'http://www.thedostore.com/laptops/thinkpad-laptops/thinkpad-t430-u-black-627326q.html')]"].
Hope it helps.

NoSuchElementException is thrown when the element could not be found.
If you encounter this exception, please check the followings:
Check your selector used in your find_by...
Element may not yet be on the screen at the time of the find operation.
If webpage is still loading, check for selenium.webdriver.support.wait.WebDriverWait() and write a wait wrapper to wait for an element to appear.
Troubleshooting and code samples
You can add breakpoint just before your failing line pdb.set_trace() (don't forget to import pdb), then run your test and once your debugger stops, then do the following tests.
You could try:
driver.find_element_by_xpath(u'//a[text()="Foo text"]')
instead. This is more reliable test, so if this would work, use it instead.
If above won't help, please check if your page has been loaded properly via:
(Pdb) driver.execute_script("return document.readyState")
'complete'
Sometimes when the page is not loaded, you're actually fetching the elements from the old page. But even though, readyState could still indicate the state of the old page (especially when using click()). Here is how this is explained in this blog:
Since Selenium webdriver has become more advanced, clicks are much more like "real" clicks, which has the benefit of making our tests more realistic, but it also means it's hard for Selenium to be able to track the impact that a click has on the browsers' internals -- it might try to poll the browser for its page-loaded status immediately after clicking, but that's open to a race condition where the browser was multitasking, hasn't quite got round to dealing with the click yet, and it gives you the .readyState of the old page.
If you think this is happening because the page wasn't loaded properly, the "recommended" (however still ugly) solution is an explicit wait:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
old_value = browser.find_element_by_id('thing-on-old-page').text
browser.find_element_by_link_text('my link').click()
WebDriverWait(browser, 3).until(
expected_conditions.text_to_be_present_in_element(
(By.ID, 'thing-on-new-page'),
'expected new text'
)
)
The naive attempt would be something like this:
def wait_for(condition_function):
start_time = time.time()
while time.time() < start_time + 3:
if condition_function():
return True
else:
time.sleep(0.1)
raise Exception(
'Timeout waiting for {}'.format(condition_function.__name__)
)
def click_through_to_new_page(link_text):
browser.find_element_by_link_text('my link').click()
def page_has_loaded():
page_state = browser.execute_script(
'return document.readyState;'
)
return page_state == 'complete'
wait_for(page_has_loaded)
Another, better one would be (credits to #ThomasMarks):
def click_through_to_new_page(link_text):
link = browser.find_element_by_link_text('my link')
link.click()
def link_has_gone_stale():
try:
# poll the link with an arbitrary call
link.find_elements_by_id('doesnt-matter')
return False
except StaleElementReferenceException:
return True
wait_for(link_has_gone_stale)
And the final example includes comparing page ids as below (which could be bulletproof):
class wait_for_page_load(object):
def __init__(self, browser):
self.browser = browser
def __enter__(self):
self.old_page = self.browser.find_element_by_tag_name('html')
def page_has_loaded(self):
new_page = self.browser.find_element_by_tag_name('html')
return new_page.id != self.old_page.id
def __exit__(self, *_):
wait_for(self.page_has_loaded)
And now we can do:
with wait_for_page_load(browser):
browser.find_element_by_link_text('my link').click()
Above code samples are from Harry's blog.
Here is the version proposed by Tommy Beadle (by using staleness approach):
import contextlib
from selenium.webdriver import Remote
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import staleness_of
class MyRemote(Remote):
#contextlib.contextmanager
def wait_for_page_load(self, timeout=30):
old_page = self.find_element_by_tag_name('html')
yield
WebDriverWait(self, timeout).until(staleness_of(old_page))
If you think it isn't about page load, double check if your element isn't in iframe or different window. If so, you've to switch to it first. To check list of available windows, run: driver.window_handles.

From viewing the source of the page that you provided a link to, it seems you are using an incorrect selector.
You should use instead find_elements_by_link_text(u'text here')[0] to select the first occurrence instead as there seems to be the potential for multiple links with the same link text.
So instead of:
self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_element_by_link_text("Thinkpad Edge E530 (Black)").text)
You should use:
self.assertEqual("Thinkpad Edge E530 (Black)", driver.find_elements_by_link_text("Thinkpad Edge E530 (Black)")[0].text)

Solution posted by OP:
Hack 1: Instead of identifying the element as a text-link, I identified the "bigger frame" in which this element was present.
itemlist_1 = driver.find_element_by_css_selector("li.item.first").text
This will give the whole item along with the name, price and detail (and the unwanted add to cart and compare"
See the attached image for more .
Hack 2: I found that the "Buy Now" which was an image element with xPath (driver.find_element_by_xpath("//div[#id='subseries']/div[2]/div/p[3]/a").click()
, in the code above) , could be made to click/identified faster if I added the following line, before finding this by xpath. I think this sort of narrows down where the Webdriver is looking for an element. This is what I added " driver.find_element_by_css_selector("#subseries").text"
This must have decreased my wait by at least 20 seconds, on that page .Hope that helps.