We Keep Coding

selenium python issue - invalid element state

Getting error: "invalid element state" when using Chrome driver for selenium.
What I'm trying to do:
Pass in some data to http://www.dhl.de/onlinefrankierung
My first issue is that when I try to use the .click() method on the checkbox named "Nachnahme" nothing happens, no check is made.
When you make a check manually, the page refreshes and additional fields open up, which is what I'm trying to access.
The second issue, which throws the invalid element state, happens when trying to pass in data using the .send_keys() method.
Here is my code so far:
from selenium import webdriver
driver = webdriver.Chrome('C:\\Users\\Owner\\AppData\\Local\\Programs\\Python\\Python36-32\\Lib\\site-packages\\chromedriver.exe')
driver.get('http://www.dhl.de/onlinefrankierung')
product_element = driver.find_element(by='id', value='bpc_PAK02')
product_element.click()
services_element = driver.find_element(by='id', value='sc_NNAHME')
services_element.click()
address_element_name = driver.find_element(by='name', value='formModel.sender.name')
address_element_name.send_keys("JackBlack")
ERROR:
C:\Users\Owner\AppData\Local\Programs\Python\Python36-32\python.exe
"C:/Users/Owner/Desktop/UpWork/Marvin Sessner/script.py" Traceback
(most recent call last): File "C:/Users/Owner/Desktop/UpWork/Marvin
Sessner/script.py", line 23, in
address_element_name.send_keys("tester") File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webelement.py",
line 352, in send_keys
'value': keys_to_typing(value)}) File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webelement.py",
line 501, in _execute
return self._parent.execute(command, params) File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webdriver.py",
line 308, in execute
self.error_handler.check_response(response) File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\errorhandler.py",
line 194, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidElementStateException: Message:
invalid element state (Session info: chrome=HIDDEN) (Driver info:
chromedriver=HIDDEN
(5b82f2d2aae0ca24b877009200ced9065a772e73),platform=Windows NT
10.0.15063 x86_64)

Putting just a small sleep between two actions solve the issue. Following code working perfectly fine.
Now before someone downvotes or put a comment about sleep. let me clarify, is this the best solution? No, it's not
But now you know why it was not working, Your action is generating some AJAX request and before it completes you are trying to do another action which is creating the issue.
The good solution would be to write the condition, which waits until that action is complete but meanwhile you have a working temporary solution.
import time
from selenium import webdriver
driver = webdriver.Chrome('h:\\bin\\chromedriver.exe')
driver.get('http://www.dhl.de/onlinefrankierung')
product_element = driver.find_element(by='id', value='bpc_PAK02')
product_element.click()
time.sleep(5)
services_element = driver.find_element(by='id', value='sc_NNAHME')
services_element.click()
time.sleep(5)
address_element_name = driver.find_element(by='name', value='formModel.sender.name')
address_element_name.send_keys("JackBlack")

If you use explicit waits, you can often avoid this error. In particular, if an element can be clicked (an input, button, select, etc.), you can wait for it to be clickable. Here is what will work in your case.
from selenium import webdriver
from selenium.webdriver.common import utils
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def waitForElementClickable(timeout=5, method='id', locator=None):
try:
element = WebDriverWait(driver, timeout).until(
EC.element_to_be_clickable((method, locator))
)
print("Element", method + '=' + locator, "can be clicked.")
return element
except Exception:
print("Element", method + '=' + locator, "CANNOT be clicked.")
raise
options = Options()
options.add_argument('--disable-infobars')
driver = webdriver.Chrome(chrome_options=options)
driver.get('http://www.dhl.de/onlinefrankierung')
product_element = waitForElementClickable(method='id', locator='bpc_PAK02')
product_element.click()
services_element = waitForElementClickable(method='id', locator='sc_NNAHME')
services_element.click()
address_element_name = waitForElementClickable(method='name', locator='formModel.sender.name')
address_element_name.send_keys("JackBlack")

Selenium working only for few pages -- Element is not clickable at point [duplicate]

This question already has answers here:
Selenium Web Driver & Java. Element is not clickable at point (x, y). Other element would receive the click
(9 answers)
Closed 5 years ago.
I tried to automate Amazon/novels list page using selenium. It is working sometimes and not working sometimes. I am unable to understand the mistake in the code. It worked fine for some time and scrolled 13 pages out of 20. But from next time, it is not working properly. Till now it didn't scroll complete 20 pages.
from selenium import webdriver
from time import sleep
from bs4 import BeautifulSoup
class App:
def __init__(self,path='F:\Imaging'):
self.path=path
self.driver = webdriver.Chrome('F:\chromedriver')
self.driver.get('https://www.amazon.in/s/ref=sr_pg_1?rh=i%3Aaps%2Ck%3Anovels&keywords=novels&ie=UTF8&qid=1510727563')
sleep(1)
self.scroll_down()
self.driver.close()
def scroll_down(self):
self.driver.execute_script("window.scrollTo(0,5500);")
sleep(1)
load_more = self.driver.find_element_by_xpath('//span[#class="pagnRA"]/a[#title="Next Page"]')
load_more.click()
sleep(2)
for value in range(2,19):
print(self.driver.current_url)
sleep(3)
self.driver.execute_script("window.scrollTo(0,5500);")
sleep(2)
load_more = self.driver.find_element_by_xpath('//span[#class="pagnRA"]/a[#title="Next Page"]')
load_more.click()
sleep(3)
if __name__=='__main__':
app=App()
The output for this code which i am getting is:
C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\python.exe C:/Users/Akhil/Scrape/amazon.py
https://www.amazon.in/s/ref=sr_pg_2/257-8503487-3570721?rh=i%3Aaps%2Ck%3Anovels&page=2&keywords=novels&ie=UTF8&qid=1510744188
https://www.amazon.in/s/ref=sr_pg_3?rh=i%3Aaps%2Ck%3Anovels&page=3&keywords=novels&ie=UTF8&qid=1510744197
https://www.amazon.in/s/ref=sr_pg_4?rh=i%3Aaps%2Ck%3Anovels&page=4&keywords=novels&ie=UTF8&qid=1510744204
Traceback (most recent call last):
File "C:/Users/Akhil/Scrape/amazon.py", line 31, in <module>
app=App()
File "C:/Users/Akhil/Scrape/amazon.py", line 11, in __init__
self.scroll_down()
File "C:/Users/Akhil/Scrape/amazon.py", line 26, in scroll_down
load_more.click()
File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webelement.py", line 501, in _execute
return self._parent.execute(command, params)
File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 308, in execute
self.error_handler.check_response(response)
File "C:\Users\Akhil\AppData\Local\Programs\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element <a title="Next Page" id="pagnNextLink" class="pagnNext" href="/gp/search/ref=sr_pg_5?rh=i%3Aaps%2Ck%3Anovels&page=5&keywords=novels&ie=UTF8&qid=1510744210">...</a> is not clickable at point (809, 8). Other element would receive the click: ...
(Session info: chrome=62.0.3202.94)
(Driver info: chromedriver=2.33.506120 (e3e53437346286c0bc2d2dc9aa4915ba81d9023f),platform=Windows NT 10.0.15063 x86_64)
Process finished with exit code 1
How to solve this problem?

Try the following code:
load_more = ui.WebDriverWait(driver, timeout).until(EC.element_to_be_clickable((By.XPATH, "//span[#class="pagnRA"]/a[#title="Next Page"]")))
driver.execute_script("arguments[0].scrollIntoView(true);", load_more)
load_more.click()
where timeout -- time in seconds for waiting for the element to be clickable.
Also, import the following at the beginning of the script:
from selenium.webdriver.support import ui
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

The error is that there is no Next Page element visible or clickable. You can either wait for the presence of this element like this, or put the .click() in a try / exception block to detect when it fails.
It is probably that either your target has legitimately run out of next pages (you have seen them all), or the page is still loading, or the format of the next link has changed.

I finally got the correct result using small modifications to the answer given by #RatmirAsanov.
Please see this code. This will scroll all pages without fail.
from selenium import webdriver
from time import sleep
from bs4 import BeautifulSoup
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
class App:
def __init__(self,path='F:\Imaging'):
self.path=path
self.driver = webdriver.Chrome('F:\chromedriver')
self.driver.get('https://www.amazon.in/s/ref=sr_pg_1?rh=i%3Aaps%2Ck%3Anovels&keywords=novels&ie=UTF8&qid=1510727563')
sleep(1)
self.scroll_down()
self.driver.close()
def scroll_down(self):
sleep(3)
self.driver.execute_script("window.scrollTo(0,5450);")
sleep(3)
load_more = self.driver.find_element_by_xpath('//span[#class="pagnRA"]/a[#title="Next Page"]')
load_more.click()
sleep(3)
for value in range(2,19):
print(self.driver.current_url)
sleep(5)
self.driver.execute_script("window.scrollTo(0,5500);")
sleep(3)
load_more = WebDriverWait(self.driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//span[#class='pagnRA']/a[#title='Next Page']")))
self.driver.execute_script("arguments[0].click();", load_more)
#load_more.click()
sleep(3)
sleep(3)
if __name__=='__main__':
APP=App()

Python selenium refresh if wait more than 10s

from selenium import webdriver
driver=webdriver.Firefox()
driver.get(url)
Sometimes the webdriver is stuck on a file or response and the page is never full-loaded so the line
driver.get(url)
is never finished. But I already got enough source code to run the rest of my code. I am wondering how can I bypass or refresh the page if the page is not full-loaded in 10 seconds.
I have tried
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
driver=webdriver.Firefox()
driver.set_page_load_timeout(10)
while True:
try:
driver.get(url)
except TimeoutException:
print("Timeout, retrying...")
continue
else:
break
but the line
driver.set_page_load_timeout(10)
always gives me following error
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 727, in set_page_load_timeout
'pageLoad': int(float(time_to_wait) * 1000)})
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 238, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 193, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message:
This is nothing after Message:. I can't identify the type of error. It's weird that my laptop can't run
driver.set_page_load_timeout(10)
My next step is to click a button on the page, but that button doesn't always exist even after full-loaded. Thus I can't use explicit wait.
Thanks

(In your code snippet you don't define URL, but I'll assume URL is defined somewhere in your actual code.)
You could combine the retry and timeout-decorator packages for this:
from retry import retry
from timeout_decorator import timeout, TimeoutError
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
#retry(TimeoutError, tries=3)
#timeout(10)
def get_with_retry(driver, url):
driver.get(url)
def main():
url = "http://something.foo"
driver=webdriver.Firefox()
try:
get_with_retry(driver, url)
foo(driver) # do whatever it is you need to do
finally:
driver.quit()
if __name__ == "__main__":
main()
Note that you would need to either not set driver.set_page_load_timeout to anything, or set it to something higher than 10 seconds.

Selenium Code works locally but not on PythonAnywhere

I have a webscraper that is running on my system and I wanted to migrate it over to PythonAnywhere, but when I moved it now it doesn't work.
Exactly the sendkeys does not seem to work - after the following code is executed I never move on to the next webpage so an attribute error gets tripped.
My code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import csv
import time
# Lists for functions
parcel_link =[]
token = []
csv_output = [ ]
# main scraping function
def getLinks(link):
# Open web browser and get url - 3 second time delay.
#Open web browser and get url - 3 second time delay.
driver.get(link)
time.sleep(3)
inputElement = driver.find_element_by_id("mSearchControl_mParcelID")
inputElement.send_keys(parcel_code+"*/n")
print("ENTER hit")
pageSource = driver.page_source
bsObj = BeautifulSoup(pageSource)
parcel_link.clear()
print(bsObj)
#pause = WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID, "mResultscontrol_mGrid_RealDataGrid")))
for link in bsObj.find(id="mResultscontrol_mGrid_RealDataGrid").findAll('a'):
parcel_link.append(link.text)
print(parcel_link)
for test in parcel_link:
clickable = driver.find_element_by_link_text(test)
clickable.click()
time.sleep(2)
The link I am trying to operate is:
https://ascendweb.jacksongov.org/ascend/%280yzb2gusuzb0kyvjwniv3255%29/search.aspx
and I am trying to send: 15-100*
TraceBack:
03:12 ~/Tax_Scrape $ xvfb-run python3.4 Jackson_Parcel_script.py
Traceback (most recent call last):
File "Jackson_Parcel_script.py", line 377, in <module>
getLinks("https://ascendweb.jacksongov.org/ascend/%28biohwjq5iibvvkisd1kjmm45%29/result.aspx")
File "Jackson_Parcel_script.py", line 29, in getLinks
inputElement = driver.find_element_by_id("mSearchControl_mParcelID")
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 206, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 662, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.4/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: 'Unable to locate element: {"method":"id","selector":"mSearchControl_mParcelID"}' ; Stac
ktrace:
at FirefoxDriver.findElementInternal_ (file:///tmp/tmpiuuqg3m7/extensions/fxdriver#googlecode.com/components/driver_component.js:9470)
at FirefoxDriver.findElement (file:///tmp/tmpiuuqg3m7/extensions/fxdriver#googlecode.com/components/driver_component.js:9479)
at DelayedCommand.executeInternal_/h (file:///tmp/tmpiuuqg3m7/extensions/fxdriver#googlecode.com/components/command_processor.js:11455)
at DelayedCommand.executeInternal_ (file:///tmp/tmpiuuqg3m7/extensions/fxdriver#googlecode.com/components/command_processor.js:11460)
at DelayedCommand.execute/< (file:///tmp/tmpiuuqg3m7/extensions/fxdriver#googlecode.com/components/command_processor.js:11402)
03:13 ~/Tax_Scrape $
Selenium Innitation:
for retry in range(3):
try:
driver = webdriver.Firefox()
break
except:
time.sleep(3)
for parcel_code in token:
getLinks("https://ascendweb.jacksongov.org/ascend/%28biohwjq5iibvvkisd1kjmm4 5%29/result.aspx")
PythonAnywhere uses a virtual instance of FireFox that is suppose to be headless like JSPhantom so I do not have a version number.
Any help would be great
RS

Well, maybe the browser used on PythonAnywhere does not load the site fast enough. So instead of time.sleep(3) try implicitly waiting for the element.
An implicit wait is to tell WebDriver to poll the DOM for a certain
amount of time when trying to find an element or elements if they are
not immediately available. The default setting is 0. Once set, the
implicit wait is set for the life of the WebDriver object instance.
Using time.sleep() with Selenium is not a good idea in general.
And give it more than just 3 second, with implicitly_wait() you specify the maximum time spent waiting for an element.
So if you set implicitly_wait(10) and the page loads, for example, in 5 seconds then Selenium will wait only 5 seconds.
driver.implicitly_wait(10)

Selenium webdriver finding element in next link

open main(home) page and then go to login page using click() function , now i want to find element in this page how could I?
here is my code ...
import unittest,time,re
from selenium import webdriver
from selenium import selenium
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
#self.selenium = selenium("localhost", 4444, "*firefox","http://www.google.com/")
self.driver = webdriver.Firefox()
def test_search_in_python_org(self):
driver = self.driver
driver.get("https://bitbucket.org/")
elem = driver.find_element_by_id("user-options")
elem = elem.find_element_by_class_name("login-link")
elem.click()
print "check"
#elem = WebDriverWait(driver, 30).until(EC.elementToBeVisible(By.name("username")));
#elem.send_keys("my_username#bitbucket.org")
user_name_field = driver.find_element_by_id('id_username')
password_field = driver.find_element_by_id('id_password')
user_name_field.send_keys('your_user_name')
password_field.send_keys('your_password')
def tearDown(self):
pass
#self.driver.close()
if __name__ == "__main__":
unittest.main()
I got this error (file name python_org_search.py)
E
======================================================================
ERROR: test_search_in_python_org (__main__.PythonOrgSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "python_org_search.py", line 25, in test_search_in_python_org
user_name_field = driver.find_element_by_id('id_username')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 197, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 681, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: u'Unable to locate element: {"method":"id","selector":"id_username"}' ; Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/driver_component.js:8860)
at FirefoxDriver.prototype.findElement (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/driver_component.js:8869)
at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10831)
at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10836)
at DelayedCommand.prototype.execute/< (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10778)
----------------------------------------------------------------------
Ran 1 test in 30.994s
FAILED (errors=1)
I tried other solutions as well but still same error .

You better should try to retrieve unique elements like the username field and password field by it's id. The name might not be unique and misleading.
Try the following find_element_by_id('id_username') and find_element_by_id('id_password').
Sometimes you need to wait some time that your browser renders the page
and retrieves all contents - so it can be useful to wait some time,
e.g. three seconds before you look deeper for elements.
Code
import time
time.sleep(3)
user_name_field = driver.find_element_by_id('id_username')
password_field = driver.find_element_by_id('id_password')
user_name_field.send_keys('your_user_name')
password_field.send_keys('your_password')
password_field.send_keys(Keys.RETURN)
...
Moreover, I recommend you to use different variables names for the different fields. The variable elem could lead to difficult bugs.
Three seconds of waiting time like mentioned above can be unreliable - a more structured way of assuring some waiting time is it to tell it your driver through implicitly_wait:
driver = webdriver.Firefox()
driver.implicitly_wait(30) # maximally wait 30 seconds before raising an exception.
# ...
# go to the page etc.
There are two ways of waiting time a driver can do:
Wait explicitly: The driver waits this time before executing every next step.
Wait implicitly: If you look for an element the driver searches for this time before he raises an exception.
For more details see the documentation: http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp#implicit-waits

The following code use elem of previous page, which does not exist.
elem = elem.find_element_by_name("username")
Instead, you should use driver:
elem = driver.find_element_by_name("username")

Try
elem = new WebDriverWait(Driver, 30).until(ExpectedConditions.visibilityOfElementLocated(By.name("username")));
elem.send_keys("my_username#bitbucket.org");

To fill any type of form...the best method is go through its name in any case where ever available.
driver=self.webdriver
driver.find_element_by_name("username").click()
driver.find_element_by_name("username").send_keys("XXXXX")
time.sleep(5)
driver.find_element_by_name("password").click()
driver.find_element_by_name("password").send_keys("XXXXX")