We Keep Coding

Locating a website element using selenium webdriver for automation task

I am working on a project using selenium webdriver which requires me to locate an element and click on it. The program starts by entering a website , clicking the searchbar , typing in a preentered string and clicking enter , up to this point everything is successful. The next thing I want it to do is find the first result of the search and click on it. This part I am having trouble with. I have successfully located all elements up to this point but i cant locate this one as an error pops up. Here is my code:
from selenium import webdriver
import time
trackname = input("Track Name: ")
driver = webdriver.Chrome('D:\WebDrivers\chromedriver.exe')
driver.get('https://music.apple.com/us/artist/search/166949667')
time.sleep(2)
searchbox = driver.find_element_by_tag_name('input')
searchbox.send_keys(trackname)
from keyboard import press
press('enter')
time.sleep(2)
result = driver.find_element_by_id('search-list-lockup__description')
result.click()
I have tried locating the element other ways but it wont work , I am guessing that the issue is that after searching I have to tell it to search on that page but I am not sure. Here is the error:
Traceback (most recent call last):
File "D:\Python Projekti\iTunesDataFiller\iTunesDataFiller.py", line 18, in <module>
result = driver.find_element_by_id('search-list-lockup__description')
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="search-list-lockup__description"]"}
(Session info: chrome=89.0.4389.114)
Process finished with exit code 1
What do I do?

This could be due to the element you want to access not being available.
For example say you load the page, the element could not be visible to selenium. So basically you're trying to click on an invisible element.
I suggest using this template to make sure elements are loaded before accessing them.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
waitshort = WebDriverWait(driver,.5)
wait = WebDriverWait(driver, 20)
waitLonger = WebDriverWait(driver, 100)
visible = EC.visibility_of_element_located
driver = webdriver.Chrome(executable_path='path')
driver.get('link')
element_you_want_to_access = wait.until(visible((By.XPATH,'xpath')))

Automate login with selenium and python, how to correctly extract element for user and password

I am trying to automate the login of a bunch of users to the fitbit site to download their data using the python's module selenium. Unfortunately I do not fully understand by looking at the source html code of the page, how to select the right elements.
My code looking something like this:
driver = webdriver.Firefox()
driver.get("https://accounts.fitbit.com/login")
driver.find_element_by_id("email").send_keys('name.surname#gmail.com')
driver.find_element_by_id("pass").send_keys("myfakepass")
driver.find_element_by_id("loginbutton").click()
But I do not know how to extract the element since driver.find_element_by_id("email") returns the error:
>>> driver.find_element_by_id("email")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="email"]
This makes totally sense to me since if I check the source page of https://accounts.fitbit.com/login I cannot detect any field called "email". But, probably due to my lack of experience, I cannot detect in the html source any of the elements that I need for the login.
Anyone could help me?
Thanks

Your locator seems wrong. Try with below xpath.
Induce WebDriverWait() and wait for element_to_be_clickable()
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[#type='email']"))).send_keys('name.surname#gmail.com')
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[#type='password']"))).send_keys("myfakepass")
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Login']"))).click()
You need to import below libraries.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

There is a code that works.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome('your path')
url = "https://accounts.fitbit.com/login"
driver.get(url)
time.sleep(2)
email = driver.find_element_by_id("ember660")
email.send_keys("Edward#elric.com")
password = driver.find_element_by_id("ember661")
password.send_keys("Password")
login_button = driver.find_element_by_id("ember701").click()
Your program didn't work because of the id which are wrong

I came up with a simple code that works.
It has been very useful to use the "Inspect" command integrated in chrome for finding the id or the xpath relative to each field. Also the package time was needed to avoid errors.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome('path to chromedriver')
url = "https://accounts.fitbit.com/login"
driver.get(url)
time.sleep(2)
email = driver.find_element_by_id("ember659")
email.send_keys("mail#gmail.com")
password = driver.find_element_by_id("ember660")
password.send_keys('actualpassword')
driver.find_element_by_xpath('//*[#id="loginForm"]/div[4]/div').click()

selenium python issue - invalid element state

Getting error: "invalid element state" when using Chrome driver for selenium.
What I'm trying to do:
Pass in some data to http://www.dhl.de/onlinefrankierung
My first issue is that when I try to use the .click() method on the checkbox named "Nachnahme" nothing happens, no check is made.
When you make a check manually, the page refreshes and additional fields open up, which is what I'm trying to access.
The second issue, which throws the invalid element state, happens when trying to pass in data using the .send_keys() method.
Here is my code so far:
from selenium import webdriver
driver = webdriver.Chrome('C:\\Users\\Owner\\AppData\\Local\\Programs\\Python\\Python36-32\\Lib\\site-packages\\chromedriver.exe')
driver.get('http://www.dhl.de/onlinefrankierung')
product_element = driver.find_element(by='id', value='bpc_PAK02')
product_element.click()
services_element = driver.find_element(by='id', value='sc_NNAHME')
services_element.click()
address_element_name = driver.find_element(by='name', value='formModel.sender.name')
address_element_name.send_keys("JackBlack")
ERROR:
C:\Users\Owner\AppData\Local\Programs\Python\Python36-32\python.exe
"C:/Users/Owner/Desktop/UpWork/Marvin Sessner/script.py" Traceback
(most recent call last): File "C:/Users/Owner/Desktop/UpWork/Marvin
Sessner/script.py", line 23, in
address_element_name.send_keys("tester") File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webelement.py",
line 352, in send_keys
'value': keys_to_typing(value)}) File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webelement.py",
line 501, in _execute
return self._parent.execute(command, params) File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webdriver.py",
line 308, in execute
self.error_handler.check_response(response) File "C:\Users\Owner\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\errorhandler.py",
line 194, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.InvalidElementStateException: Message:
invalid element state (Session info: chrome=HIDDEN) (Driver info:
chromedriver=HIDDEN
(5b82f2d2aae0ca24b877009200ced9065a772e73),platform=Windows NT
10.0.15063 x86_64)

Putting just a small sleep between two actions solve the issue. Following code working perfectly fine.
Now before someone downvotes or put a comment about sleep. let me clarify, is this the best solution? No, it's not
But now you know why it was not working, Your action is generating some AJAX request and before it completes you are trying to do another action which is creating the issue.
The good solution would be to write the condition, which waits until that action is complete but meanwhile you have a working temporary solution.
import time
from selenium import webdriver
driver = webdriver.Chrome('h:\\bin\\chromedriver.exe')
driver.get('http://www.dhl.de/onlinefrankierung')
product_element = driver.find_element(by='id', value='bpc_PAK02')
product_element.click()
time.sleep(5)
services_element = driver.find_element(by='id', value='sc_NNAHME')
services_element.click()
time.sleep(5)
address_element_name = driver.find_element(by='name', value='formModel.sender.name')
address_element_name.send_keys("JackBlack")

If you use explicit waits, you can often avoid this error. In particular, if an element can be clicked (an input, button, select, etc.), you can wait for it to be clickable. Here is what will work in your case.
from selenium import webdriver
from selenium.webdriver.common import utils
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def waitForElementClickable(timeout=5, method='id', locator=None):
try:
element = WebDriverWait(driver, timeout).until(
EC.element_to_be_clickable((method, locator))
)
print("Element", method + '=' + locator, "can be clicked.")
return element
except Exception:
print("Element", method + '=' + locator, "CANNOT be clicked.")
raise
options = Options()
options.add_argument('--disable-infobars')
driver = webdriver.Chrome(chrome_options=options)
driver.get('http://www.dhl.de/onlinefrankierung')
product_element = waitForElementClickable(method='id', locator='bpc_PAK02')
product_element.click()
services_element = waitForElementClickable(method='id', locator='sc_NNAHME')
services_element.click()
address_element_name = waitForElementClickable(method='name', locator='formModel.sender.name')
address_element_name.send_keys("JackBlack")

Python selenium refresh if wait more than 10s

from selenium import webdriver
driver=webdriver.Firefox()
driver.get(url)
Sometimes the webdriver is stuck on a file or response and the page is never full-loaded so the line
driver.get(url)
is never finished. But I already got enough source code to run the rest of my code. I am wondering how can I bypass or refresh the page if the page is not full-loaded in 10 seconds.
I have tried
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
driver=webdriver.Firefox()
driver.set_page_load_timeout(10)
while True:
try:
driver.get(url)
except TimeoutException:
print("Timeout, retrying...")
continue
else:
break
but the line
driver.set_page_load_timeout(10)
always gives me following error
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 727, in set_page_load_timeout
'pageLoad': int(float(time_to_wait) * 1000)})
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 238, in execute
self.error_handler.check_response(response)
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 193, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message:
This is nothing after Message:. I can't identify the type of error. It's weird that my laptop can't run
driver.set_page_load_timeout(10)
My next step is to click a button on the page, but that button doesn't always exist even after full-loaded. Thus I can't use explicit wait.
Thanks

(In your code snippet you don't define URL, but I'll assume URL is defined somewhere in your actual code.)
You could combine the retry and timeout-decorator packages for this:
from retry import retry
from timeout_decorator import timeout, TimeoutError
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
#retry(TimeoutError, tries=3)
#timeout(10)
def get_with_retry(driver, url):
driver.get(url)
def main():
url = "http://something.foo"
driver=webdriver.Firefox()
try:
get_with_retry(driver, url)
foo(driver) # do whatever it is you need to do
finally:
driver.quit()
if __name__ == "__main__":
main()
Note that you would need to either not set driver.set_page_load_timeout to anything, or set it to something higher than 10 seconds.

Selenium webdriver finding element in next link

open main(home) page and then go to login page using click() function , now i want to find element in this page how could I?
here is my code ...
import unittest,time,re
from selenium import webdriver
from selenium import selenium
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
#self.selenium = selenium("localhost", 4444, "*firefox","http://www.google.com/")
self.driver = webdriver.Firefox()
def test_search_in_python_org(self):
driver = self.driver
driver.get("https://bitbucket.org/")
elem = driver.find_element_by_id("user-options")
elem = elem.find_element_by_class_name("login-link")
elem.click()
print "check"
#elem = WebDriverWait(driver, 30).until(EC.elementToBeVisible(By.name("username")));
#elem.send_keys("my_username#bitbucket.org")
user_name_field = driver.find_element_by_id('id_username')
password_field = driver.find_element_by_id('id_password')
user_name_field.send_keys('your_user_name')
password_field.send_keys('your_password')
def tearDown(self):
pass
#self.driver.close()
if __name__ == "__main__":
unittest.main()
I got this error (file name python_org_search.py)
E
======================================================================
ERROR: test_search_in_python_org (__main__.PythonOrgSearch)
----------------------------------------------------------------------
Traceback (most recent call last):
File "python_org_search.py", line 25, in test_search_in_python_org
user_name_field = driver.find_element_by_id('id_username')
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 197, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 681, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 164, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 164, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: Message: u'Unable to locate element: {"method":"id","selector":"id_username"}' ; Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/driver_component.js:8860)
at FirefoxDriver.prototype.findElement (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/driver_component.js:8869)
at DelayedCommand.prototype.executeInternal_/h (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10831)
at DelayedCommand.prototype.executeInternal_ (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10836)
at DelayedCommand.prototype.execute/< (file:///tmp/tmpKo1TXx/extensions/fxdriver#googlecode.com/components/command_processor.js:10778)
----------------------------------------------------------------------
Ran 1 test in 30.994s
FAILED (errors=1)
I tried other solutions as well but still same error .

You better should try to retrieve unique elements like the username field and password field by it's id. The name might not be unique and misleading.
Try the following find_element_by_id('id_username') and find_element_by_id('id_password').
Sometimes you need to wait some time that your browser renders the page
and retrieves all contents - so it can be useful to wait some time,
e.g. three seconds before you look deeper for elements.
Code
import time
time.sleep(3)
user_name_field = driver.find_element_by_id('id_username')
password_field = driver.find_element_by_id('id_password')
user_name_field.send_keys('your_user_name')
password_field.send_keys('your_password')
password_field.send_keys(Keys.RETURN)
...
Moreover, I recommend you to use different variables names for the different fields. The variable elem could lead to difficult bugs.
Three seconds of waiting time like mentioned above can be unreliable - a more structured way of assuring some waiting time is it to tell it your driver through implicitly_wait:
driver = webdriver.Firefox()
driver.implicitly_wait(30) # maximally wait 30 seconds before raising an exception.
# ...
# go to the page etc.
There are two ways of waiting time a driver can do:
Wait explicitly: The driver waits this time before executing every next step.
Wait implicitly: If you look for an element the driver searches for this time before he raises an exception.
For more details see the documentation: http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp#implicit-waits

The following code use elem of previous page, which does not exist.
elem = elem.find_element_by_name("username")
Instead, you should use driver:
elem = driver.find_element_by_name("username")

Try
elem = new WebDriverWait(Driver, 30).until(ExpectedConditions.visibilityOfElementLocated(By.name("username")));
elem.send_keys("my_username#bitbucket.org");

To fill any type of form...the best method is go through its name in any case where ever available.
driver=self.webdriver
driver.find_element_by_name("username").click()
driver.find_element_by_name("username").send_keys("XXXXX")
time.sleep(5)
driver.find_element_by_name("password").click()
driver.find_element_by_name("password").send_keys("XXXXX")