So I'm a little stuck! I'm trying to select an item from the 'All reviews' drop down however it doesn't interact like a where each item I could select the element of and then click it.
Instead the acts like an element where upon its label changing different results are displayed. Does anyone know how I could select an element from this menu?
For example, making the menu select the "Google" tab from the drop down.
for reference:
https://www.google.com/maps/place/Hilton+London+Bankside/#51.5056536,-0.1033145,17z/data=!3m1!4b1!4m10!3m9!1s0x487604af6af74cc7:0x6c4cb3cbe03e95bf!5m2!4m1!1i2!8m2!3d51.5056536!4d-0.1011258!9m1!1b1
Induce WebDriverWait() and element_to_be_clickable() and click on the All reviews div element to open up the dropdown menu and then select the items based on text.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver=webdriver.Chrome()
driver.get("https://www.google.com/maps/place/Hilton+London+Bankside/#51.5056536,-0.1033145,17z/data=!3m1!4b1!4m10!3m9!1s0x487604af6af74cc7:0x6c4cb3cbe03e95bf!5m2!4m1!1i2!8m2!3d51.5056536!4d-0.1011258!9m1!1b1")
#Dropdown text provide here
selectItem='Agoda'
#First click on the All reviews element to open up the dorpdown element
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"div[aria-label='All reviews']"))).click()
#Select item from menu dropdown by text
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.XPATH,"//div[#role='menuitem']//div[text()='"+ selectItem +"']"))).click()
Browser snapshot: after execution
At this particular webpage elements are appearing,many of them don't just exist in the DOM content, you should use WebDriverWait method to wait until the specific element gets located.
For example, let's select "Google" as you asked:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get("the google link here, it's too big to paste it")
#Waiting until dropdown is visible , there are two dropdowns, taking the first one
menu = WebDriverWait(driver,5).until(EC.visibility_of_element_located((By.XPATH, "
(//div[#class='cYrDcjyGO77__container'])[1]")))
menu.click()
#Waiting untill menu items is visible then selecting the second element - Google
item = WebDriverWait(driver,5).until(EC.visibility_of_element_located((By.XPATH, "
(//div[#role='menuitem'])[2]")))
item.click()
What you can do is to find your dropdown and then list thru all the options and select your one.
Here is how I'm doing it
el = driver.find_element_by_id("dropdown_id")
for option in el.find_elements_by_tag_name('option'):
if "GB" in option.text:
option.click() # select() in earlier versions of webdriver
break
I'm selecting the dropdown that have Value of the state "GB".
Related
Hi i'm new at selenium and webscraping and i need some help.
i try to scrape one site and i need and i dont know how to get span class.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
PATCH = "/Users/bobo/Downloads/chromedriver"
driver = webdriver.Chrome(PATCH)
driver.get("https://neonet.pl")
print(driver.title)
search = driver.find_element_by_class_name("inputCss-input__label-263")
search.send_keys(Keys.RETURN)
time.sleep(5)
i try to extract this span
<span class="inputCss-input__label-263">Szukaj produktu</span>
I can see that you are trying to search something in the search bar.
First I recommend you to use the xpath instead of the class name, here is a simple technique to get the xpath of every element on a webpage:
right-click/ inspect element/ select the mouse in a box element on the upper left/ click on the element on the webpage/ it will directly show you the corresponding html/ then right click on the selected html/ copy options and then xpath.
Here is a code example that searches an element on the webpage, I also included the 'Webdriver-wait' option because sometimes the code runs to fast and can't find the next element so this function make the code wait till the element is visible:
from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome(executable_path="/Users/bobo/Downloads/chromedriver")
driver.get("https://neonet.pl") #loading page
wait = WebDriverWait(driver, 20) #defining webdriver wait
search_word = 'iphone\n' # \n is going to act as an enter key
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/div[1]/div[4]/div/div/div[2]/div/button[1]'))).click() #clicking on cookies popup
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/main/header/div[2]/button'))).click() #clicking on search button
wait.until(EC.visibility_of_element_located((By.XPATH, '//*[#id="root"]/aside[2]/section/form/label/input'))).send_keys(search_word) #searching on input button
print('done!')
sleep(10)
Hope this helped you!
wait=WebDriverWait(driver,10)
driver.get('https://neonet.pl')
elem=wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(#class,'inputCss-input')]"))).text
print(elem)
To output the value of the search bar you use .text on the Selenium Webelement.
Imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I'm trying to select a specific option in a dropdown menu but haven't been able to do so. When inspecting the web page, I haven't been able to open the menu to see if there's any xpath to a specific path I could use. The web in question is https://megatlon.com/clases , I'm trying to select another gym other than Megatlon Alcorta. Let's suppose Megatlon Barracas. The following line helped me open the menu but that's as far as I could get:
driver.find_element_by_xpath('//*[#id="root"]/div/div[2]/div[1]/div[1]/div/div/div[2]/div').click()
How can I select another option and leave the dropdown menu? Thank you!
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.find_element_by_xpath('//*[#id="root"]/div/div[2]/div[1]/div[1]/div/div/div[2]/div').click()
WebDriverWait(browser, 15).until(EC.visibility_of_element_located(
(By.XPATH, '//*[contains(text(),"Megatlon Barrio Jardín (Córdoba)")]'))).click()
Click the dropdown, wait for visibility and then click on the text you want
I'm navigating a JS-heavy webpage with Selenium and I need to be able to interact with a dojo component on the page. The page I'm looking at has a dojo dijit form with a combobox that has subject names for my university. I want to expose and iteratively click on every item in the list in order to scrape the course names for that subject when it redirects. The list items are exposed when the dropdown arrow button is clicked.
the url I'm automating: http://sis.rutgers.edu/soc/#subjects?semester=12020&campus=NB,NK,CM&level=U,G
I'm inspecting the element for the dropdown button and copying the XPath.
dropdownButton = driver.find_element_by_xpath('//*[#id="widget_dijit_form_FilteringSelect_0"]/div[1]/input')
Running this yields:
NoSuchElementException: Message: no such element: Unable to locate
element:
{"method":"xpath","selector":"//*[#id="widget_dijit_form_FilteringSelect_0"]/div[1]/input"}
EDIT: I've made some progress, turn out the element wasn't rendered by the time find_by_xpath was called. I added a wait in my program, and now Selenium is able to locate and click the drowdown button.
Use WebDriverWait to wait require element conditions. Dropdown disappears on any action on the page, that's why to get option locator you can do one of the following:
all options loaded after first expand, that's why you can search option element by text in chrome dev tools and get locator
pause and inspect the element.
You can google best practice for locators,here and here.
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
with driver:
driver.get("http://sis.rutgers.edu/soc/#subjects?semester=12020&campus=NB,NK,CM&level=U,G")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#filteringSelectDiv .dijitArrowButtonInner"))).click()
options = driver.execute_script('return [...arguments[0]].map(e=>e.textContent)',
wait.until(EC.presence_of_all_elements_located(
(By.CSS_SELECTOR, ".dijitComboBoxMenuPopup .dijitMenuItem[item]"))))
for option in options:
driver.find_element_by_css_selector(".dijitInputInner").clear()
driver.find_element_by_css_selector(".dijitInputInner").send_keys(option, Keys.TAB)
wait.until(lambda d: d.execute_script("return document.readyState === 'complete'"))
# collect data
So I'm scraping using selenium and I want to click 'next' button in 'Defensive' section but the code I wrote clicks 'next' on 'Summary'.
Here's the url for you to try :
https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019
So it's selecting 'Defensive' and I can see it selected in the window but the next page doesnt appear. On clicking 'Summary' I found out next function is actually happening there.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser= webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
browser.find_element_by_xpath("""//*[#id="stage-top-player-stats-options"]/li[2]/a""").click()
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, """//*[#id="next"]""")))
browser.execute_script("arguments[0].click();", element)
The xpath for next button is not unique for this page. try this,
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
browser.execute_script("arguments[0].click();", element)
or
element = WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//*[#id='stage-top-player-stats-defensive']//a[#id='next']")))
element.click()
For each tab (Summary, Defensive, ..) new next button with same id=next added to the DOM.
Select Defensive and you will see there will be two next buttons with same id=next, select Offensive and there will be three next buttons.
With basic id=next selector you always click to the first next button from Summary tab. Because you're using JavaScript and nothing happen, try to click with Selenium click method and you will get an error.
To solve the problem adjust your selector to be more specific to the dom - #statistics-paging-defensive #next.
Also when you first time open the page there's cookies acceptance screen appears and block the page, you can use method like below to skip it.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import selenium.common.exceptions as EX
def accept_cookies():
try:
WebDriverWait(browser, 20)\
.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.qc-cmp-button")))\
.click()
except EX.NoSuchElementException or EX.TimeoutException:
pass
#...
browser = webdriver.Chrome(executable_path ="C:\Program Files (x86)\Google\Chrome\chromedriver.exe")
browser.get('https://www.whoscored.com/Regions/252/Tournaments/2/Seasons/7361/Stages/16368/PlayerStatistics/England-Premier-League-2018-2019')
wait = WebDriverWait(browser, 20)
browser.get(baseUrl)
accept_cookies()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "[href='#stage-top-player-stats-defensive']"))).click()
next_button = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "#statistics-paging-defensive #next")))
next_button.click()
Your elements locators must be unique
Avoid using XPath wildcards - * as it will cause performance degradation and prolonged elements lookup timings
Avoid using JavaScriptExecutor for clicking, well-behaved Selenium test must do what real user does and I doubt that real user will be opening browser console and typing something like document.getElementById('next').click(), he will use the mouse
Assuming all above you should come up with a selector which uniquely identifies next button on Defensive tab which would be something like:
//div[#id='statistics-paging-defensive']/descendant::a[#id='next']
References:
XPath Tutorial
XPath Axes
XPath Operators & Functions
I tried to select 2002 in dropdown menu.
It doesn't work at any late.
I used xpath
driver.find_element_by_xpath("html/body/main/div/form/div[3]/div[1]/section/div[3]/fieldset/div[7]/dl[1]/dd/ul/li[1]/a").click()
but it doesn't work..I tried all the solutions I got...
How can I select this?
If you're able to open dropdown item but unable to click on item, you should try using Explicit Waits with WebDriverWait to wait until this element is visible and enable to click as below :-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "ul#ulBirthYear a[data-value='2002']")))
element.click()
Or
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "2002")))
element.click()
First of all, try to avoid using absolute XPATH.
Use something like this:
'//ul[#id="uiBirthYear"]/li/a[#data-value="2002"]'
Also ensure, that the DOM is fully built, before you trying to get/click on this element.
Try to set an implicit wait
driver.implicitly_wait(10)
or an explicit wait (read more: http://selenium-python.readthedocs.io/waits.html)