I am trying to do web scraping to weather data from:
https://www.wunderground.com/history/daily/us/dc/washington/KDCA
Here is HTML:
<div _ngcontent-app-root-c136="" id="wuForm" class="ui-front wu-form">
<div _ngcontent-app-root-c136="" id="wuSearch-contain" class="wu-search-contain ng-star-inserted"><label _ngcontent-app-root-c136="" for="wuSearch" class="visuallyHidden">Search</label><input _ngcontent-app-root-c136="" type="search" name="query" value="" id="wuSearch" placeholder="Search Locations" aria-label="Search" autocomplete="off" class="wu-search ng-untouched ng-pristine ng-valid"><span _ngcontent-app-root-c136="" class="close-search"><i _ngcontent-app-root-c136="" class="material-icons">close</i></span><span _ngcontent-app-root-c136="" class="geolocate-wrap"><i _ngcontent-app-root-c136="" class="material-icons">gps_fixed</i></span></div>
<!----><!---->
<search-autocomplete _ngcontent-app-root-c136="" _nghost-app-root-c135="">
<ul _ngcontent-app-root-c135="" tabindex="0" class="ui-autocomplete ui-front ui-menu ui-widget ui-widget-content ui-corner-all hide">
<li _ngcontent-app-root-c135="" class="ui-autocomplete-geolocate ng-star-inserted">
<div _ngcontent-app-root-c135="" class="mimic-a menu-geolocate"><i _ngcontent-app-root-c135="" class="material-icons">gps_fixed</i>Find Nearest Station </div>
</li>
<!----><!----><!----><!----><!----><!---->
<li _ngcontent-app-root-c135="" class="ui-autocomplete-last ui-menu-item manage-favorites"><a _ngcontent-app-root-c135="" tabindex="-1" href="/member/favorites" class="ui-corner-all">Manage Favorite Cities</a></li>
</ul>
</search-autocomplete>
</div>
I started with
driver = webdriver.Chrome(r'C:\Users\bno00\Downloads\chromedriver_win32\chromedriver.exe')
driver.get('https://www.wunderground.com/history/daily/us/dc/washington/KDCA')
and it's working fine
then when I trying to send zipcode to search field by using .submit():
zipcode='22202'
xpath='//*[#id="wuSearch"]'
search=driver.find_element_by_xpath(xpath)
search.clear()
search.send_keys(zipcode)
search.submit() #error occurs here
The error is:
NoSuchElementException: Message: no such element: Unable to locate
element:{"method":"xpath","selector":"./ancestor-or-self::form"}
(Session info: chrome=89.0.4389.114)
That <input> tag is not part of a form. You can't submit it. You either need to send a newline, or click the gps_fixed icon that triggers the search.
Try using the following:
input_field = driver.find_element_by_css_selector("input[id=wuSearch]")
input_field.click()
input_field.send_keys("some zip code")
Or just:
input_field = driver.find_element_by_id("wuSearch")
input_field.send_keys("some zip code")
Try to avoid click(). I am not sure it is really required.
After you input a zip code, you should wait for your search result to appear.
wait = WebDriverWait(driver, 10)
driver.get('https://www.wunderground.com/history/daily/us/dc/washington/KDCA')
zipcode='22202'
xpath='//*[#id="wuSearch"]'
search=wait.until(EC.element_to_be_clickable((By.XPATH,xpath)))
search.clear()
search.send_keys(zipcode)
time.sleep(1)
search.send_keys(Keys.ENTER)
Just delay it a bit. Use a webdriver wait to make sure you send the zipcode to the input and then send Keys.Enter.
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from time import sleep
Related
I wanted to scrape data from a website which is "https://fhsaa.com/sports/2020/1/28/member_directory.aspx". The idea is to search school names using id or school name and then scrape the information. but when I used xpath or css_selecotr there is an exception "o such element: Unable to locate element: {"method":"xpath","selector":"//input[#name='school_name']"}
" I have used both xpath, css_selector and waits too but no success. Here is the code...
<fieldset class="border">
<legend class="border">School Filter</legend>
<form method="GET" action="https://fhsaahome.org/widget/school-directory/submit" accept-charset="UTF-8">
<div class="row">
<div class="col-md-4">
<select class="form-control" name="school_id"><option selected="selected" value="">--Select--</option><option value="714">Abundant Life Christian (Margate)</option><option value="665">Academy at the Lakes (Land O'Lakes)</option><option value="639">Academy Prep (St. Petersburg)</option><option value="4">Admiral Farragut (St. Petersburg)</option><option value="1191">AIE Charter (Miami Springs)</option>value="542">Zephyrhills</option><option value="1062">Zephyrhills Christian</option></select>
</div>
<div class="col-md-4">
<input class="form-control" placeholder="Enter School Name" name="school_name" type="text" value="">
</div>
<br><br>
<div class="col-md-2">
<input class="btn btn-secondary btn-md" type="submit" value="Search">
</div>
</div>
This is an iframe game, I see the website has called an API to continuously shuffle the xpath you are looking. I add a piece of code to switch between iframes and they worked:
iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(iframe)
element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')
This is the full program:
import os
from selenium import webdriver
from time import sleep
from random import randint
chromedriver = "C:\Program Files\Python39\Scripts\chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
url = 'https://fhsaa.com/sports/2020/1/28/member_directory.aspx'
driver.get(url)
sleep(randint(2,3))
iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(iframe)
element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')
element.send_keys('yourschool')
sleep(randint(2,3))
I have tested your xpath in BaseX software and your xpath is correct.
However your HTML 5 is not correct
You have this
ter (Miami Springs)</option>value="542">Zephyrhills</option>
and you should have
ter (Miami Springs)</option><option value="542">Zephyrhills</option>
I want to click open the options in the menu on the left but they are blocked by the pop-up. How can I do that?
here is what I have tried:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
chromedriver = "/usr/share/chromedriver/chromedriver"
driver = webdriver.Chrome(chromedriver)
driver.get("https://student.amizone.net")
driver.find_element(By.NAME, "_UserName").send_keys("username")
driver.find_element(By.NAME, "_Password").send_keys("password")
driver.find_element(By.CSS_SELECTOR, "#loginform .login100-form-btn").click()
driver.implicitly_wait(10)
#11 | click | id=ModalPopAmityHostel |
driver.find_element(By.ID, "ModalPopAmityHostel").click()
# 11 | click | id=StudentSatisfactionPop | |
driver.find_element(By.ID, "StudentSatisfactionPop").click()
this code closes the 1st pop-up but doesn't close the 2nd pop-up. In the code first I login to the website https://student.amizone.net i have not shown my username and password(obviously). after that driver.find_element(By.ID, "ModalPopAmityHostel").click() is supposed to click outside the pop-up which closes the pop-up. similarly driver.find_element(By.ID, "StudentSatisfactionPop").click() is supposed to close the 2nd pop-up.
this is the snippet of html code of the pop up elements:
<form action="/PopUp/Home/SANGATHANQuizpopupSave" data-ajax="true" data-ajax-loading="#lodingDiv" data-ajax-method="POST" data-ajax-mode="replace" data-ajax-success=" $('#SANGATHANQuizpop').modal('hide');$('.modal-backdrop').remove();$(document.body).removeClass('modal-open');alertify.alert('Successfully Saved.'); " data-ajax-update="#Div_Partial" id="form0" method="post"></form>
<div id="StudentSatisfactionPop" class="modal fade in" role="dialog" aria-hidden="false" style="display: block; padding-right: 15px;">
<div class="modal-dialog">
<!-- Modal content-->
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal">×</button>
<h4 class="modal-title">STUDENT SATISFACTION SURVEY </h4>
</div>
<div class="modal-body">
<div>
<p>
<b>Dear Mr MANIK RAINA </b>
</p>
<p>
Please tell us about you! Office of Research, Planning & Statistical Services, Amity University Uttar Pradesh is conducting a survey of its students. This survey asks your opinion on many items relevant to examining the impact of college. It asks about your transition to university, your academic habits and experiences, your interaction with peers and faculty, your involvement in campus activities and programs, and how you spend your time.
</p>
<p>Results from this survey will be used by us to understand and improve your experiences. Data received from your responses will be analyzed and interpreted batch wise only, and at no time the responses be linked with an individual. Data will be used for revision and improvement of resource planning for better learning experiences of students of future batches. Be assured that your responses will remain confidential.</p>
<p>We would very much appreciate you taking a few moments to fill out the Survey Questionnaire.</p>
<p><b>
To Start the survey :
<a data-ajax="true" data-ajax-begin="$('#sidebar').removeClass('display');" data-ajax-loading="#lodingDiv" data-ajax-mode="replace" data-ajax-success=" $('#StudentSatisfactionPop').modal('hide');$('.modal-backdrop').remove();$(document.body).removeClass('modal-open');" data-ajax-update="#Div_Partial" href="/Survey/StudentSatisfaction" id="12" rel="0">Click Here</a>.
</b></p>
</div>
</div>
<div class="modal-footer">
<button type="button" id="btnClosevoter" class="btn btn-default" data-dismiss="modal">Close</button>
</div>
</div>
</div>
</div> <script>
$(document).ready(function () {
$('#StudentSatisfactionPop').modal('show');
});
</script>
<script>
$(document).ready(function () {
$('#ModalPopAmityHostel').modal('show');
});
</script>
<div id="ModalPopAmityHostel" class="modal fade in" role="dialog" aria-hidden="false" style="display: block;">
<div class="modal-dialog " style="z-index:104546464; ">
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal">×</button>
<h4 class="modal-title">Amity Hostel</h4>
</div>
<div class="modal-body">
<p class="text-center">
“A few hostel seats (A/C and Non A/C) are available for allocation. Students desirous of availing the seats may apply on Amizone and also contact hostel office. Allocation of seats will be done on ‘first come, first served’ basis.”
</p>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-primary" data-dismiss="modal">Close</button>
<h4 class="modal-title"></h4>
</div>
</div>
</div>
</div>
You can close popups by clicking on close buttons. In your code you click to the div instead of close buttons, that's why popups do not close (see correct locators for the close buttons).
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chromedriver = "/usr/share/chromedriver/chromedriver"
driver = webdriver.Chrome(chromedriver)
wait = WebDriverWait(driver, 10)
driver.get("https://student.amizone.net")
driver.find_element(By.NAME, "_UserName").send_keys("username")
driver.find_element(By.NAME, "_Password").send_keys("password")
driver.find_element(By.CSS_SELECTOR, "#loginform .login100-form-btn").click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ModalPopAmityHostel button.btn"))).click()
driver.execute_script("arguments[0].click()", driver.find_element_by_css_selector("#StudentSatisfactionPop button.btn"))
# wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#StudentSatisfactionPop button.btn"))).click()
As the <button> tags with text as Close are with in Modal Dialog Box so to locate and click() on the desired elements you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://student.amizone.net')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "form#loginform input[name='_UserName']"))).send_keys("7071804")
driver.find_element_by_css_selector("form#loginform input[name='_Password']").send_keys("62ae6f")
driver.find_element_by_css_selector("form#loginform button[type='submit']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#ModalPopAmityHostel div.modal-footer button.btn.btn-primary[data-dismiss='modal']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#StudentSatisfactionPop div.modal-footer button.btn.btn-default[data-dismiss='modal']"))).click()
Using XPATH:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://student.amizone.net')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//form[#id='loginform']//input[#name='_UserName']"))).send_keys("7071804")
driver.find_element_by_xpath("//form[#id='loginform']//input[name='_Password']").send_keys("62ae6f")
driver.find_element_by_xpath("//form[#id='loginform']//button[type='submit']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='ModalPopAmityHostel']//div[#class='modal-footer']//button[#class='btn btn-primary' and #data-dismiss='modal']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='StudentSatisfactionPop']//div[#class='modal-footer']//button[#class='btn btn-default' and #data-dismiss='modal']"))).click()
Here's the angular source
Trying to create a simple script that checks/unchecks some boxes with python/selenium.
I've thrown what little book I have at it and can't get it to correctly identify the elements. I've tried (likely incorrectly) css, xpath, label, text.
Any help would be greatly appreciated.
Code trial:
driver.find_elements_by_class_name('list-group-items')[1].click()
HTML:
<div class="col-sm-12">
<fieldset>
<legend>
Step 2: Choose Data fields
</legend>
<div class="col-sm-5 col-lg-4">
<div class="panel minimal minimal-gray">
<div class="panel-title">
<h4>
Standard Fields:
</h4>
</div>
<ul class="list-group scrollable ng-scope" id="standard-fields" ng-class="{error: ctrl.export.errors.columns}">
<!-- ngRepeat: item in ctrl.exportFormats.columns track by item.code --><li ng-repeat="item in ctrl.exportFormats.columns track by item.code" class="list-group-item ng-binding ng-scope list-group-item-success" ng-class="{'list-group-item-success' : item.checked}">
<input type="checkbox" ng-model="item.checked" class="ng-pristine ng-untouched ng-valid">
Date
</li><!-- end ngRepeat: item in ctrl.exportFormats.columns track by item.code --><li ng-repeat="item in ctrl.exportFormats.columns track by item.code" class="list-group-item ng-binding ng-scope list-group-item-success" ng-class="{'list-group-item-success' : item.checked}">
<input type="checkbox" ng-model="item.checked" class="ng-pristine ng-untouched ng-valid">
Time
</li><!-- end ngRepeat: item in ctrl.exportFormats.columns track by item.code --><li ng-repeat="item in ctrl.exportFormats.columns track by item.code" class="list-group-item ng-binding ng-scope list-group-item-success" ng-class="{'list-group-item-success' : item.checked}">
Seems you were pretty close. As per your code trials instead of find_elements_by_class_name('list-group-items')[1], using find_element_by_class_name('list-group-item') would have taken you more closer.
However, as the desired elements are Angular elements so to click the checkboxes associated with the texts e.g. Date, Time etc, you have to induce WebDriverWait for the element to be clickable and you can use either of the following solutions:
Date:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[#class='list-group-item ng-binding ng-scope list-group-item-success' and contains(#ng-repeat,'exportFormats')][contains(.,'Date')]/input[#class='ng-pristine ng-untouched ng-valid']"))).click()
Time:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[#class='list-group-item ng-binding ng-scope list-group-item-success' and contains(#ng-repeat,'exportFormats')][contains(.,'Time')]/input[#class='ng-pristine ng-untouched ng-valid']"))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I am trying to click on element with following html code:
<div class="delivery-pseudo-table">
<div class=" i-popup-lk user-address" id="userAddress">
<div class="address-container">
<div class="titlebar">
<h1 class="title"></h1>
</div>
<div id="userAddressContent"></div>
</div>
<div class="hide j-map map" id="addressMap"></div>
<div class="hide j-map-pickpoint postamat-map" id="pickpointMap"></div>
<div class="added-addres-loader"></div>
</div>
<ul class="delivery-method" data-jsv-df=""><li class="selfDelivery selectDeliveryWay active" data-jsv="#34_#35_" title="">
<script type="jsv#37_"></script>
<label><input autocomplete="off" checked="checked" data-id="2" name="orderDetails.DeliveryWay_q" type="radio" value="Self"/>Самовывоз</label>
<script type="jsv/37_"></script>
</li><li class="courierDelivery selectDeliveryWay" data-jsv="/35_#38_" title="">
<script type="jsv#39_"></script>
<label><input autocomplete="off" data-id="1" name="orderDetails.DeliveryWay_q" type="radio" value="Courier"/>Доставка курьером</label>
<script type="jsv/39_"></script>
</li><li class="wbpostamatDelivery selectDeliveryWay" data-jsv="/38_#40_" title="">
<script type="jsv#41_"></script>
<label><input autocomplete="off" data-id="16" name="orderDetails.DeliveryWay_q" type="radio" value="WbPostamat"/>Постамат</label>
<script type="jsv/41_"></script>
</li></ul>
I need to click on button with value "Courier"
And I am trying to do it with following Python code:
elem = browser.find_element_by_css_selector("input[type='radio'][#value='Courier']")
elem.click()
(I've also tried to find by xpath)
But it says that the element is invisible. Are the any ways to click on such element?
Try to use this xPath:
//input[#type='radio' and #value='Courier']
also try to use WebDriverWait:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[#type='radio' and #value='Courier']"))).click()
Note: you have to add some imports:
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
WebDriverWait statement is for waiting at least 10 seconds until element will be clickable and only then clicks on it.
PS: more information about WebDriverWait can be found here.
EDIT: try to execute JavaScript:
radio_btn = driver.find_element_by_xpath("//input[#type='radio' and #value='Courier']")
driver.execute_script("arguments[0].click();", radio_btn)
As per the HTML you have shared the <ul> tag contains a lot of JavaScripts so to invoke click() on the desired element you have to induce WebDriverWait for the element to be clickable and you can use either of the following solutions:
CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "ul.delivery-method li.courierDelivery.selectDeliveryWay input[value='Courier']"))).click()
XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//ul[#class='delivery-method']//li[#class='courierDelivery selectDeliveryWay']//input[#value='Courier']"))).click()
div class="row-fluid">
<div id="sfx" class="span12">
<ui-view>
<div class="row-fluid fa-rep-nav">
<div id="test"></div>
<div class="row-fluid">
<div id="sfx" class="span12">
<ui-view>
<div class="row-fluid">
<div class="span12">
<div class="ledger-btn-block clearfix">
<form class="ng-pristine ng-valid" name="reportGenForm">
<span class="input-append ledger-date-block fa-block-pull-left">
<label class="ledger-label">Select·Report:· </label>
<select class="select_report_type ng-pristine ng-untouched ng-valid" ng-model="selectedReport" ng-options="str.name for str in reportType">
<label class="ledger-label"> ·Date·Range:· </label>
<input class="date-picker report-date-input ng-pristine ng-untouched ng-valid" date-range-picker="" readonly="" ng-model="sfxReportModel.date" options="selectedReport.opts" min="selectedReport.opts.minDate" max="selectedReport.opts.maxDate" type="text">
<i class="icon-calendar add-on no-border-radius"></i>
</span>
I have used all the method of locating an element which are applicable here. I have also used time.sleep() function so that the page on which I am locating element can load successfully before finding, but even then also location is not appearing.Here is the pic of html code. Highlighted is the one I want to locate.
I have used methods like find_element_by_xpath and find_by_class_name where i get xpath using Firebug but getting the same errorenter image description here. I need help in locating "select" tag.
As far as I understood you want to select the dropdown. Use the following code in python to select a dropdown value
from selenium.webdriver.support.ui import Select
select = Select(driver.find_element_by_xpath("//select[#class='select_report_type ng-pristine ng-untouched ng-valid']"))
select.select_by_index(1)
// OR
select.select_by_visible_text("text")
// OR
select.select_by_value(value)
Let me know if still facing same issue or any further things