I wanted to scrape data from a website which is "https://fhsaa.com/sports/2020/1/28/member_directory.aspx". The idea is to search school names using id or school name and then scrape the information. but when I used xpath or css_selecotr there is an exception "o such element: Unable to locate element: {"method":"xpath","selector":"//input[#name='school_name']"}
" I have used both xpath, css_selector and waits too but no success. Here is the code...
<fieldset class="border">
<legend class="border">School Filter</legend>
<form method="GET" action="https://fhsaahome.org/widget/school-directory/submit" accept-charset="UTF-8">
<div class="row">
<div class="col-md-4">
<select class="form-control" name="school_id"><option selected="selected" value="">--Select--</option><option value="714">Abundant Life Christian (Margate)</option><option value="665">Academy at the Lakes (Land O'Lakes)</option><option value="639">Academy Prep (St. Petersburg)</option><option value="4">Admiral Farragut (St. Petersburg)</option><option value="1191">AIE Charter (Miami Springs)</option>value="542">Zephyrhills</option><option value="1062">Zephyrhills Christian</option></select>
</div>
<div class="col-md-4">
<input class="form-control" placeholder="Enter School Name" name="school_name" type="text" value="">
</div>
<br><br>
<div class="col-md-2">
<input class="btn btn-secondary btn-md" type="submit" value="Search">
</div>
</div>
This is an iframe game, I see the website has called an API to continuously shuffle the xpath you are looking. I add a piece of code to switch between iframes and they worked:
iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(iframe)
element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')
This is the full program:
import os
from selenium import webdriver
from time import sleep
from random import randint
chromedriver = "C:\Program Files\Python39\Scripts\chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
url = 'https://fhsaa.com/sports/2020/1/28/member_directory.aspx'
driver.get(url)
sleep(randint(2,3))
iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(iframe)
element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')
element.send_keys('yourschool')
sleep(randint(2,3))
I have tested your xpath in BaseX software and your xpath is correct.
However your HTML 5 is not correct
You have this
ter (Miami Springs)</option>value="542">Zephyrhills</option>
and you should have
ter (Miami Springs)</option><option value="542">Zephyrhills</option>
Related
I am trying to navigate to a search box and send_keys with selenium python but completely stuck.
And here is the source code snippet:
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<div id="LeftTreeFrame" class="leftNavBackground" >
<div class="ui-widget searchPanelContainer">
<div id="searchPanel" class="search-field-container search-field-container-margin">
<input type="text" doesntDirty id="Search" name="Search" class="search-text-field-left-tree-frame" NoHighlight="nohighlight"/>
<div class="search-field-icon-container">
<a id="searchlbl" href="#"><img src="../images/normal_search_u39.svg" title="Go To Page" /></a>
</div>
</div>
</div>
<div id='pageNavigation'>
<div id='ootbNavigationPage'></div>
<div id='favoriteNavigationPage'></div>
<div id='adminNavigationPage'></div>
<div id='navigationEmptyState' class="treeEmptyState">
<div class="message"></div>
</div>
</div>
<div class="navigation-view-mode-container">
<div class="box" onclick="renderModel(0)">
<button type="button">
<span class="svg-load ootb-icon" data-src="~/images/Reskin/ootb-icon.svg"></span>
</button>
</div>
<div class="star" onclick="renderModel(1)">
<button type="button">
<span class="svg-load star-icon" data-src="~/images/Reskin/star.svg"></span>
</button>
</div>
<div class="person" onclick="renderModel(2)">
<button type="button">
<span class="svg-load person-icon" data-src="~/images/Reskin/person-nav.svg"></span>
</button>
</div>
</div>
</div>
When I try to do
element = driver.find_element(By.XPATH, '//input[#name="Search"]')
element.send_keys('test')
I get error "selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable"
I have tried everything I can imagine, but cannot click the element or send keys.
Also, this page is a new page that opens after the last successful click. I first tried switching to this page by
#printing handles
handles = driver.window_handles
i=0
for handle in handles:
print(f"Handle {i}: {handle}\n")
i +=1
#after confirming new page is second handle via:
driver.switch_to.window(handles[1])
print(f" Title: {driver.title}")
print(f" Current url: {driver.current_url}")
print('\n')
#I can even find the tag I am looking for after switching to new window:
all_div_tags = driver.find_elements(By.TAG_NAME, "input")
for tag in all_div_tags:
print(f"Attribute name: {tag.get_attribute('name')}\n")
#but i cannot get to the search box. Thank you in advance!
Look at the html code, notice that //input[#name="Search"] is contained in an <iframe>. In order to select an element inside an iframe with find_element() you have first to switch to the iframe, as shown in the code
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "frmCode")))
element = driver.find_element(By.XPATH, '//input[#name="Search"]')
...
Could not click the button English.
It says no such element exists.
Tried the following and Select method.
element = WebDriverWait(driver, 40).until(expected_conditions.element_to_be_clickable((By.XPATH, "/html/body/content/main/div/form")))
select = Select(driver.find_element_by_css_selector("label[for='language-MCQ'] span[class='me-2']"))
Code block:
<input class="form-check-input d-none lang-list" type="checkbox" id="language-MCQ" name="language_ids[]" value="1" wire:model="language_ids.0" wire:change="addLanguageQuestions(1)">
<label class="form-check-label d-flex flex-nowrap align-items-center rounded-pill btn btn-lg btn-lang-select text-white cursor-pointer" for="language-MCQ">
<span class="me-2">
English
</span>
</label>
<div class="form-check ps-0 me-3">
<input class="form-check-input d-none lang-list" type="checkbox" id="language-Kannada" name="language_ids[]" value="2" wire:model="language_ids.1" wire:change="addLanguageQuestions(2)">
<label class="form-check-label d-flex flex-nowrap align-items-center rounded-pill btn btn-lg btn-lang-select text-white cursor-pointer" for="language-Kannada">
<span class="me-2">
Kannada
</span>
</label>
</div>
Did you try this xpath :
//span[text()='English']
if you do not wish to use text()
//label[#for='language-MCQ']/span
also this looks wrong
select = Select(driver.find_element_by_css_selector("label[for='language-MCQ'] span[class='me-2']"))
Note that Select is only applicable for html select tag.
To minimize the possibility of any kind of exception it is recommended that your xpath should be pointed to the interactive element, In this case span or label is not interactive element but the input tag.
and your possible xpath solutions can be -
.//span[contains(text(),'English')]/ancestor::label/preceding-sibling::input
Or
.//input[#id='language-MCQ']
Or
.//label[contains(#class,'btn-lang-select')]/preceding-sibling::input[#value='1']
something like this should work -
element = WebDriverWait(driver, 40).until(expected_conditions.element_to_be_clickable((By.XPATH, ".//span[contains(text(),'English')]/ancestor::label/preceding-sibling::input")))
element.click()
If the above xpaths are not working with the explicit wait and expected conditions then check element should be in iframe.
I am trying to do web scraping to weather data from:
https://www.wunderground.com/history/daily/us/dc/washington/KDCA
Here is HTML:
<div _ngcontent-app-root-c136="" id="wuForm" class="ui-front wu-form">
<div _ngcontent-app-root-c136="" id="wuSearch-contain" class="wu-search-contain ng-star-inserted"><label _ngcontent-app-root-c136="" for="wuSearch" class="visuallyHidden">Search</label><input _ngcontent-app-root-c136="" type="search" name="query" value="" id="wuSearch" placeholder="Search Locations" aria-label="Search" autocomplete="off" class="wu-search ng-untouched ng-pristine ng-valid"><span _ngcontent-app-root-c136="" class="close-search"><i _ngcontent-app-root-c136="" class="material-icons">close</i></span><span _ngcontent-app-root-c136="" class="geolocate-wrap"><i _ngcontent-app-root-c136="" class="material-icons">gps_fixed</i></span></div>
<!----><!---->
<search-autocomplete _ngcontent-app-root-c136="" _nghost-app-root-c135="">
<ul _ngcontent-app-root-c135="" tabindex="0" class="ui-autocomplete ui-front ui-menu ui-widget ui-widget-content ui-corner-all hide">
<li _ngcontent-app-root-c135="" class="ui-autocomplete-geolocate ng-star-inserted">
<div _ngcontent-app-root-c135="" class="mimic-a menu-geolocate"><i _ngcontent-app-root-c135="" class="material-icons">gps_fixed</i>Find Nearest Station </div>
</li>
<!----><!----><!----><!----><!----><!---->
<li _ngcontent-app-root-c135="" class="ui-autocomplete-last ui-menu-item manage-favorites"><a _ngcontent-app-root-c135="" tabindex="-1" href="/member/favorites" class="ui-corner-all">Manage Favorite Cities</a></li>
</ul>
</search-autocomplete>
</div>
I started with
driver = webdriver.Chrome(r'C:\Users\bno00\Downloads\chromedriver_win32\chromedriver.exe')
driver.get('https://www.wunderground.com/history/daily/us/dc/washington/KDCA')
and it's working fine
then when I trying to send zipcode to search field by using .submit():
zipcode='22202'
xpath='//*[#id="wuSearch"]'
search=driver.find_element_by_xpath(xpath)
search.clear()
search.send_keys(zipcode)
search.submit() #error occurs here
The error is:
NoSuchElementException: Message: no such element: Unable to locate
element:{"method":"xpath","selector":"./ancestor-or-self::form"}
(Session info: chrome=89.0.4389.114)
That <input> tag is not part of a form. You can't submit it. You either need to send a newline, or click the gps_fixed icon that triggers the search.
Try using the following:
input_field = driver.find_element_by_css_selector("input[id=wuSearch]")
input_field.click()
input_field.send_keys("some zip code")
Or just:
input_field = driver.find_element_by_id("wuSearch")
input_field.send_keys("some zip code")
Try to avoid click(). I am not sure it is really required.
After you input a zip code, you should wait for your search result to appear.
wait = WebDriverWait(driver, 10)
driver.get('https://www.wunderground.com/history/daily/us/dc/washington/KDCA')
zipcode='22202'
xpath='//*[#id="wuSearch"]'
search=wait.until(EC.element_to_be_clickable((By.XPATH,xpath)))
search.clear()
search.send_keys(zipcode)
time.sleep(1)
search.send_keys(Keys.ENTER)
Just delay it a bit. Use a webdriver wait to make sure you send the zipcode to the input and then send Keys.Enter.
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from time import sleep
I am trying to click into a radio button inside a span. Trying to execute:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chromeDriver = webdriver.Chrome()
chromeDriver.get('https://www.flytap.com/pt-br/')
#Works Fine
chromeDriver.find_element_by_id('origin')
origin.click()
origin.clear()
origin.send_keys('(RIO) Rio De Janeiro - todos os aeroportos , Brasil')
origin.click()
#Where I got the error
milesBox = chromeDriver.find_element_by_id("booking-type-2").click()
#Also tried:
milesBox = chromeDriver.find_element_by_id("booking-type-2").send_keys(Keys.ENTER)
#And, finally:
milesBox = chromeDriver.find_element_by_id("booking-type-2").send_keys(Keys.SPACE)
Here is the error I got:
ElementNotVisibleException: Message: element not visible
HTML code:
<!-- Payment Methods START -->
<fieldset id="booking-payment-drag" class="booking-payment">
<div class="toolbar-radio-wrapper">
<legend class="ipt-label">Reservar com:</legend>
</div>
<div class="toolbar-radio-wrapper">
<div class="radio">
<span class="checked"><input type="radio" id="booking-type-1" name="booking-type" value="1" checked></span>
</div>
<label for="booking-type-1" class="ipt-label">Dinheiro</label>
</div>
<div class="toolbar-radio-wrapper js-country-not-eligible">
<div class="radio">
<span class=""><input type="radio" id="booking-type-2" name="booking-type" value="2"></span>
</div>
<label for="booking-type-2" class="ipt-label">Milhas</label>
</div>
<div class="toolbar-radio-wrapper">
<div class="radio is-disabled js-country-eligible">
<span class=""><input type="radio" id="booking-type-3" name="booking-type" value="3" disabled></span>
</div>
<label for="booking-type-3" class="ipt-label">Miles&Cash</label>
</div>
</fieldset>
As per the HTML you have shared to click on the radio button beside the <label> with text as Milhas you can use the following line of code :
chromeDriver.find_element_by_xpath("//input[#id='booking-type-2' and #name='booking-type']").click()
You can use this xpath
//div[#class='radio']//span//input[#id='booking-type-1']
//div[#class='radio']//span//input[#id='booking-type-2']
//div[#class='radio is-disabled js-country-eligible']//span//input[#id='booking-type-3']
It seems like the origin element's dropdown covers the radio button you are trying to click. Maybe you could change the order of operations though? Clicking the destination element also opens up the section where the radio button is but it doesn't seem to cover it. This is a quick example of clicking that radio button and then you could go fillout the origin input after.
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.flytap.com/pt-br/')
destination = driver.find_element_by_id('destination')
destination.click()
miles_box = driver.find_element_by_id('booking-type-2')
miles_box.click()
div class="row-fluid">
<div id="sfx" class="span12">
<ui-view>
<div class="row-fluid fa-rep-nav">
<div id="test"></div>
<div class="row-fluid">
<div id="sfx" class="span12">
<ui-view>
<div class="row-fluid">
<div class="span12">
<div class="ledger-btn-block clearfix">
<form class="ng-pristine ng-valid" name="reportGenForm">
<span class="input-append ledger-date-block fa-block-pull-left">
<label class="ledger-label">Select·Report:· </label>
<select class="select_report_type ng-pristine ng-untouched ng-valid" ng-model="selectedReport" ng-options="str.name for str in reportType">
<label class="ledger-label"> ·Date·Range:· </label>
<input class="date-picker report-date-input ng-pristine ng-untouched ng-valid" date-range-picker="" readonly="" ng-model="sfxReportModel.date" options="selectedReport.opts" min="selectedReport.opts.minDate" max="selectedReport.opts.maxDate" type="text">
<i class="icon-calendar add-on no-border-radius"></i>
</span>
I have used all the method of locating an element which are applicable here. I have also used time.sleep() function so that the page on which I am locating element can load successfully before finding, but even then also location is not appearing.Here is the pic of html code. Highlighted is the one I want to locate.
I have used methods like find_element_by_xpath and find_by_class_name where i get xpath using Firebug but getting the same errorenter image description here. I need help in locating "select" tag.
As far as I understood you want to select the dropdown. Use the following code in python to select a dropdown value
from selenium.webdriver.support.ui import Select
select = Select(driver.find_element_by_xpath("//select[#class='select_report_type ng-pristine ng-untouched ng-valid']"))
select.select_by_index(1)
// OR
select.select_by_visible_text("text")
// OR
select.select_by_value(value)
Let me know if still facing same issue or any further things