I am just getting started with Selenium and I am having some issues trying to locate different elements on a particular website. The website I want to crawl is https://connect.garmin.com/signin since they unfortunately do not provide API access to individuals.
In any case, the element I am trying to "find" is or the login box. I am having much issues with that however, and I do not know what I am doing wrong at this point.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = browser=webdriver.Firefox()
driver.get("https://connect.garmin.com/signin")
element = driver.find_element("name", "username")
element.clear()
element.send_keys("testing#gmail.com")
This is my current code.
Traceback (most recent call last):
File "/Users/will/PycharmProjects/GarminCrawlerTwitter/Crawler.py", line 7, in <module>
element = driver.find_element("name", "username")
File "/Users/will/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 855, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/Users/will/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 428, in execute
self.error_handler.check_response(response)
File "/Users/will/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 243, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [name="username"]
Stacktrace:
WebDriverError#chrome://remote/content/shared/webdriver/Errors.jsm:188:5
NoSuchElementError#chrome://remote/content/shared/webdriver/Errors.jsm:400:5
element.find/</<#chrome://remote/content/marionette/element.js:292:16
And that is the error message. It seems to run into issues trying to find the element "username". Tried using XPATH //[#id="username"] and //[#name="username"] but with same results.
I have also used driver.implicitly_wait(3) thinking that it might have been that the site wasn't fully loaded before trying to locate the username element, and that was why it was failing but that seems to have been incorrect as well.
#Zeno, the login pop-up is inside an iframe so first, you need to switch to that frame. The following should work.
driver.get("https://connect.garmin.com/signin")
iframe = driver.find_element(By.ID, "gauth-widget-frame-gauth-widget")
driver.switch_to.frame(iframe)
element = driver.find_element("name", "username")
element.clear()
element.send_keys("testing#gmail.com")
you will need the following import as I have used By:
from selenium.webdriver.common.by import By
Related
I am trying to use selenium with chrome driver to connect to a website. But it couldn't be reached.
Here is my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
CHROME_EXECUTABLE_PATH = "C://Program Files (x86)//Chrome Driver//chromedriver.exe"
CHROME_OPTIONS = webdriver.ChromeOptions()
CHROME_OPTIONS.add_argument("--disable-notifications")
BASE_URL = "https://www.nordstrom.com/"
driver = webdriver.Chrome(executable_path=CHROME_EXECUTABLE_PATH, options=CHROME_OPTIONS)
# locators
search_button_locator = "//a[#id='controls-keyword-search-popover']"
search_box_locator = "//*[#id='keyword-search-input']"
driver.get(BASE_URL)
driver.find_element(By.XPATH, search_button_locator)
driver.find_element(By.XPATH, search_box_locator).send_keys("Fave Slipper")
This code gives me some error:
E:\Python\Nordstrom.com\venv\Scripts\python.exe E:/Python/Nordstrom.com/pages/simple.py
Traceback (most recent call last):
File "E:\Python\Nordstrom.com\pages\simple.py", line 14, in <module>
driver.find_element(By.XPATH, search_button_locator)
File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[#id='controls-keyword-search-popover']"}
(Session info: chrome=94.0.4606.61)
Process finished with exit code 1
The page looks like this:
But the expected page should be looks like this:
How to access this website?
The error points out that it was unable to find the XPATH element, which is why it errored out.
The main causes for this can be either:
the XPATH is wrong
the element has not loaded yet on the page
the site has detected your scraping attempt and blocked you
In this case it's a combination of the 2nd and 3rd options. Whenever you use a webdriver, it exposes javascript hooks that websites can detect. To hide your activity you should learn more on how device fingerprinting and either customize your script to hide itself or use a pre-made solution for it (such as PhantomJS).
Most likely you should also look into hiding your IP by using a proxy.
There is a problem with your 'BASE_URL' so try another Browser to debug the issue and also try to use explicit wait before click or locate any element
I'm trying to scrape Amazon product prices. And I want to scrape prices text without opening the Chrome browser. I searched this on the Internet but it didn't help me.
This is my code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
# Driver and link
driver = webdriver.Chrome('C:/Users/musta/Desktop/chromedriver.exe')
driver.get("https://www.amazon.com/dp/b07h9fldcd")
getText = driver.find_element_by_class_name("a-section a-spacing-micro").get_attribute("textContent")
print(getText)
driver.close()
But this didn't work. It keeps giving me this error message:
Traceback (most recent call last):
File "C:\Users\musta\Desktop\asd.py", line 8, in <module>
getText = driver.find_element_by_class_name("a-section a-spacing-micro").get_attribute("textContent")
File "C:\Users\musta\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 564, in find_element_by_class_name
return self.find_element(by=By.CLASS_NAME, value=name)
File "C:\Users\musta\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\musta\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\musta\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".a-section a-spacing-micro"}
(Session info: chrome=91.0.4472.114)
What should I do? I'm stuck at here. I want to scrape price from given div without opening Chrome browser. Hope you understand what I mean.
You need headless broswer I think :
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("start-maximized")
driver = webdriver.Chrome(r'C:/Users/musta/Desktop/chromedriver.exe', options = options)
driver.get("https://www.amazon.com/dp/b07h9fldcd")
and regarding your error, in Selenium class name do not work with spaces :
so instead of this,
.a-section a-spacing-micro
do this with css :
.a-section.a-spacing-micro
so code should look like this :
getText = driver.find_element_by_css_selector(".a-section.a-spacing-micro").get_attribute("innerHTML")
print(getText.strip())
if it just the price you wanna grab, try with this css :
span#price_inside_buybox
in code it would look like this :
getText = driver.find_element_by_css_selector("span#price_inside_buybox
").text
print(getText.strip())
There are two questions in your query:
You can open the chrome browser in headless mode so it wont render in frontend
Try to inspect element and confirm the id/class name you have used to get_attribute
I am working on a project using selenium webdriver which requires me to locate an element and click on it. The program starts by entering a website , clicking the searchbar , typing in a preentered string and clicking enter , up to this point everything is successful. The next thing I want it to do is find the first result of the search and click on it. This part I am having trouble with. I have successfully located all elements up to this point but i cant locate this one as an error pops up. Here is my code:
from selenium import webdriver
import time
trackname = input("Track Name: ")
driver = webdriver.Chrome('D:\WebDrivers\chromedriver.exe')
driver.get('https://music.apple.com/us/artist/search/166949667')
time.sleep(2)
searchbox = driver.find_element_by_tag_name('input')
searchbox.send_keys(trackname)
from keyboard import press
press('enter')
time.sleep(2)
result = driver.find_element_by_id('search-list-lockup__description')
result.click()
I have tried locating the element other ways but it wont work , I am guessing that the issue is that after searching I have to tell it to search on that page but I am not sure. Here is the error:
Traceback (most recent call last):
File "D:\Python Projekti\iTunesDataFiller\iTunesDataFiller.py", line 18, in <module>
result = driver.find_element_by_id('search-list-lockup__description')
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "D:\Python App\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="search-list-lockup__description"]"}
(Session info: chrome=89.0.4389.114)
Process finished with exit code 1
What do I do?
This could be due to the element you want to access not being available.
For example say you load the page, the element could not be visible to selenium. So basically you're trying to click on an invisible element.
I suggest using this template to make sure elements are loaded before accessing them.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
waitshort = WebDriverWait(driver,.5)
wait = WebDriverWait(driver, 20)
waitLonger = WebDriverWait(driver, 100)
visible = EC.visibility_of_element_located
driver = webdriver.Chrome(executable_path='path')
driver.get('link')
element_you_want_to_access = wait.until(visible((By.XPATH,'xpath')))
Python, Selenium, XPath.
I want to open this page https://www.tesla.com/en_gb/models/design#battery and click the performance button programmatically with python.
Here is what I want to click on:
image showing what I want to click on
My problem is properly describing the button. Maybe I don't understand xpath properly or there's a better method to point to the desired element.
Here's what I tried
from selenium import webdriver
browser = webdriver.Chrome('../Downloads/chromedriver.exe')
browser.get('https://www.tesla.com/en_gb/models/design#battery')
A = browser.find_element_by_xpath('/html/body/div/div/main/div/div/div[2]/div[5]/div/div[1]/div/div[2]/div[2]/div[1]')
A.click();
and I get this error
Traceback (most recent call last):
File "C:\Users\User\Desktop\666.py", line 4, in <module>
A = browser.find_element_by_xpath('/html/body/div/div/main/div/div/div[2]/div[5]/div/div[1]/div/div[2]/div[2]/div[1]')
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\User\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div/main/div/div/div[2]/div[5]/div/div[1]/div/div[2]/div[2]/div[1]"}
(Session info: chrome=80.0.3987.149)
full xpath {/html/body/div/div/main/div/div/div[2]/div[5]/div/div1/div/div[2]/div[2]/div1}
html of element i want to click on
<div role="button" tabindex="0" class="group--options_block m3-animate--all" aria-label="Performance"><div class="group--options_block_title"><span><p class="group--options_block--name text-loader--content" tabindex="-1">Performance</p></span><p class="group--options_block-container_price text-loader--content price-not-included">£95,800</p></div></div>
im copying and pasting the full xpath of the element i want. is that not the correct way to do this?
edit:
if it works the range should be 367 not 379
Use following xpath
//div[#class='group--options_block_title']/span/p
OR
//p[contains(text(),'Performance')]
OR CSS selector
div[aria-label='Performance']
Don't forget to introduce Implicit or Explicit wait to avoid synchronization issue in your scripts. reference
Please refer below code sometime sites are taking too long while loading so would be great if you use WebDriverWait in your solution. Also its not good practice to use Abs XPath.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.Chrome(executable_path=r"C:\New folder\chromedriver.exe")
driver.get('https://www.tesla.com/en_gb/models/design#battery')
wait = WebDriverWait(driver,30)
element = wait.until(EC.element_to_be_clickable((By.XPATH, "//p[contains(text(),'Performance')]")))
print element.text
element.click()
element1 = wait.until(EC.presence_of_element_located((By.XPATH, "//span[contains(text(),'367')]")))
print element1.text
Output:
I'm trying to log onto a webpage with python selenium. I've found an element and it is enabled, but when I try to send_keys() to it I get an error. The main thing (I think) in the error output is
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
My code is
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import contextlib
with contextlib.closing(webdriver.Firefox()) as driver:
driver.get('http://www.etoro.com/au')
elem = driver.find_element_by_class_name('inputUsername')
print 'enabled:', elem.is_enabled()
print 'selected:', elem.is_selected()
elem.send_keys('myusername')
And the output is
enabled: True
selected: False
Traceback (most recent call last):
File "3_trying_again.py", line 10, in <module>
elem.send_keys('ianafterglow')
File "/Users/ian/miniconda/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 303, in send_keys
self._execute(Command.SEND_KEYS_TO_ELEMENT, {'value': typing})
File "/Users/ian/miniconda/lib/python2.7/site-packages/selenium/webdriver/remote/webelement.py", line 385, in _execute
return self._parent.execute(command, params)
File "/Users/ian/miniconda/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 173, in execute
self.error_handler.check_response(response)
File "/Users/ian/miniconda/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: Element is not currently visible and so may not be interacted with
Stacktrace:
at fxdriver.preconditions.visible (file:///var/folders/5b/ym07nh6d74gcn_773ynwqkth0000gn/T/tmpN1MV8l/extensions/fxdriver#googlecode.com/components/command-processor.js:8959:12)
at DelayedCommand.prototype.checkPreconditions_ (file:///var/folders/5b/ym07nh6d74gcn_773ynwqkth0000gn/T/tmpN1MV8l/extensions/fxdriver#googlecode.com/components/command-processor.js:11618:15)
at DelayedCommand.prototype.executeInternal_/h (file:///var/folders/5b/ym07nh6d74gcn_773ynwqkth0000gn/T/tmpN1MV8l/extensions/fxdriver#googlecode.com/components/command-processor.js:11635:11)
at DelayedCommand.prototype.executeInternal_ (file:///var/folders/5b/ym07nh6d74gcn_773ynwqkth0000gn/T/tmpN1MV8l/extensions/fxdriver#googlecode.com/components/command-processor.js:11640:7)
at DelayedCommand.prototype.execute/< (file:///var/folders/5b/ym07nh6d74gcn_773ynwqkth0000gn/T/tmpN1MV8l/extensions/fxdriver#googlecode.com/components/command-processor.js:11582:5)
So, what do I need to do?
To make the username field to be visible, you need to move cursor to the login link:
....
driver.get('http://www.etoro.com/au')
action = webdriver.ActionChains(driver)
action.move_to_element(driver.find_element_by_xpath(
'.//a[#class="top-link"]/span[text()="Login"]'
))
action.perform()
# TODO Need to wait until the `inputUsername` field is visible
elem = driver.find_element_by_class_name('inputUsername')
...
You can use explicit waits:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CLASSNAME, "inputUsername"))
)
...
I know that this problem is solved ,
I got stuck in similar problem and same error
I have fixed it by just make my script sleep for 2 seconds then resume it was just Connection speed problem
...
time.sleep(2)
...
don't forget to import time module
import time
wish that help anyone in future :D
I had similar issue, selenium was not able to focus and open the login modal. Instead it was focusing on the next element. This was the locator I was using:
elem = browser.find_element_by_xpath("//nav[2]/ul/li[3]/a").click()
I just changed [3] with [2] and it was able to locate the element and open the modal:
elem = browser.find_element_by_xpath("//nav[2]/ul/li[2]/a").click()