We Keep Coding

How to handle dropdown without select in selenium python

Hello I would like to be able to change the value of "50 Profiles / Page" to "500 Profiles / Page", but the problem is that in the HTML there is no "Select" tag.
I tried doing this but it didn't work
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://www.personality-database.com/profile?pid=1&sort=hot'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.maximize_window()
driver.implicitly_wait(30)
driver.get(url)
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="root"]/div/section/main/div[1]/div[2]/div/div[5]/ul/li[10]/div/div[1]/span[2][text()="500 Profiles / Page"]'))).click()
Here is the code The HTML code
<li class="rc-pagination-options">
<div class="rc-select rc-pagination-options-size-changer rc-select-single rc-select-show-arrow">
<span class="rc-select-arrow" unselectable="on" aria-hidden="true">
<span class="rc-select-arrow-icon"></span></span>
<div class="rc-select-dropdown rc-select-dropdown-placement-topLeft rc-select-dropdown-hidden">
<div role="listbox" id="rc_select_0_list">
<div aria-label="20 Profiles / Page" role="option" id="rc_select_0_list_0"
aria-selected="false">20</div>
</div>
<div class="rc-virtual-list" style="position: relative;">
<div class="rc-virtual-list-holder">
<div class="rc-virtual-list-holder-inner"
style="display: flex; flex-direction: column;">
<div aria-selected="false" class="rc-select-item rc-select-item-option"
title="20 Profiles / Page">
<div class="rc-select-item-option-content">20 Profiles / Page</div><span
class="rc-select-item-option-state" unselectable="on" aria-hidden="true"
style="user-select: none;"><span
class="rc-select-item-option-state-icon"></span></span>
</div>
<div aria-selected="false" class="rc-select-item rc-select-item-option"
title="500 Profiles / Page">
<div class="rc-select-item-option-content">500 Profiles / Page</div><span
class="rc-select-item-option-state" unselectable="on" aria-hidden="true"
style="user-select: none;"><span
class="rc-select-item-option-state-icon"></span></span>
</div>
...
</li>

First we need to close the pop-ups and then try to click on pagination options.
And using both Implicit wait and Explicit wait is not Recommended.
Try the following solution:
driver.get("https://www.personality-database.com/profile?pid=1&sort=hot")
wait = WebDriverWait(driver,30)
try:
# Close the footer add
wait.until(EC.element_to_be_clickable((By.XPATH,"//span[#id='ezmob-wrapper']/div/center/span/div/div/span"))).click()
# Scroll a distance so that the Cookie pop up appears and Close it
driver.execute_script("window.scrollBy(0,50);")
wait.until(EC.element_to_be_clickable((By.XPATH,"//button[#id='rcc-confirm-button']"))).click()
except:
print("no adds")
# click on the drop down option
pagination = wait.until(EC.element_to_be_clickable((By.XPATH,"//li[#class='rc-pagination-options']")))
pagination.click()
# Click on the 500 profiles
option = wait.until(EC.element_to_be_clickable((By.XPATH,"//div[#class='rc-virtual-list-holder-inner']//div[text()='500 Profiles / Page']")))
option.click()

First xpath to click dropdown:
//div[#class='rc-select rc-pagination-options-size-changer rc-select-single rc-select-show-arrow']
Second xpath to click the option for 500 pages:
//div[#class='rc-select-item-option-content']/self::div[text()='500 Profiles / Page']
Here is a cheatsheet for relative xpaths https://devhints.io/xpath
Please be aware that browsers use xpath 1.0 and selenium also only supports 1.0,
So some things like 'ends-with' won't work.

Selenium unable to locate locate element within a fieldset

I wanted to scrape data from a website which is "https://fhsaa.com/sports/2020/1/28/member_directory.aspx". The idea is to search school names using id or school name and then scrape the information. but when I used xpath or css_selecotr there is an exception "o such element: Unable to locate element: {"method":"xpath","selector":"//input[#name='school_name']"}
" I have used both xpath, css_selector and waits too but no success. Here is the code...
<fieldset class="border">
<legend class="border">School Filter</legend>
<form method="GET" action="https://fhsaahome.org/widget/school-directory/submit" accept-charset="UTF-8">
<div class="row">
<div class="col-md-4">
<select class="form-control" name="school_id"><option selected="selected" value="">--Select--</option><option value="714">Abundant Life Christian (Margate)</option><option value="665">Academy at the Lakes (Land O'Lakes)</option><option value="639">Academy Prep (St. Petersburg)</option><option value="4">Admiral Farragut (St. Petersburg)</option><option value="1191">AIE Charter (Miami Springs)</option>value="542">Zephyrhills</option><option value="1062">Zephyrhills Christian</option></select>
</div>
<div class="col-md-4">
<input class="form-control" placeholder="Enter School Name" name="school_name" type="text" value="">
</div>
<br><br>
<div class="col-md-2">
<input class="btn btn-secondary btn-md" type="submit" value="Search">
</div>
</div>

This is an iframe game, I see the website has called an API to continuously shuffle the xpath you are looking. I add a piece of code to switch between iframes and they worked:
iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(iframe)
element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')
This is the full program:
import os
from selenium import webdriver
from time import sleep
from random import randint
chromedriver = "C:\Program Files\Python39\Scripts\chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)
url = 'https://fhsaa.com/sports/2020/1/28/member_directory.aspx'
driver.get(url)
sleep(randint(2,3))
iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(iframe)
element = driver.find_element_by_xpath('//html/body/div[1]/fieldset/form/div/div[2]/input')
element.send_keys('yourschool')
sleep(randint(2,3))

I have tested your xpath in BaseX software and your xpath is correct.
However your HTML 5 is not correct
You have this
ter (Miami Springs)</option>value="542">Zephyrhills</option>
and you should have
ter (Miami Springs)</option><option value="542">Zephyrhills</option>

Web Scraping error in while using Search.submit()

I am trying to do web scraping to weather data from:
https://www.wunderground.com/history/daily/us/dc/washington/KDCA
Here is HTML:
<div _ngcontent-app-root-c136="" id="wuForm" class="ui-front wu-form">
<div _ngcontent-app-root-c136="" id="wuSearch-contain" class="wu-search-contain ng-star-inserted"><label _ngcontent-app-root-c136="" for="wuSearch" class="visuallyHidden">Search</label><input _ngcontent-app-root-c136="" type="search" name="query" value="" id="wuSearch" placeholder="Search Locations" aria-label="Search" autocomplete="off" class="wu-search ng-untouched ng-pristine ng-valid"><span _ngcontent-app-root-c136="" class="close-search"><i _ngcontent-app-root-c136="" class="material-icons">close</i></span><span _ngcontent-app-root-c136="" class="geolocate-wrap"><i _ngcontent-app-root-c136="" class="material-icons">gps_fixed</i></span></div>
<!----><!---->
<search-autocomplete _ngcontent-app-root-c136="" _nghost-app-root-c135="">
<ul _ngcontent-app-root-c135="" tabindex="0" class="ui-autocomplete ui-front ui-menu ui-widget ui-widget-content ui-corner-all hide">
<li _ngcontent-app-root-c135="" class="ui-autocomplete-geolocate ng-star-inserted">
<div _ngcontent-app-root-c135="" class="mimic-a menu-geolocate"><i _ngcontent-app-root-c135="" class="material-icons">gps_fixed</i>Find Nearest Station </div>
</li>
<!----><!----><!----><!----><!----><!---->
<li _ngcontent-app-root-c135="" class="ui-autocomplete-last ui-menu-item manage-favorites"><a _ngcontent-app-root-c135="" tabindex="-1" href="/member/favorites" class="ui-corner-all">Manage Favorite Cities</a></li>
</ul>
</search-autocomplete>
</div>
I started with
driver = webdriver.Chrome(r'C:\Users\bno00\Downloads\chromedriver_win32\chromedriver.exe')
driver.get('https://www.wunderground.com/history/daily/us/dc/washington/KDCA')
and it's working fine
then when I trying to send zipcode to search field by using .submit():
zipcode='22202'
xpath='//*[#id="wuSearch"]'
search=driver.find_element_by_xpath(xpath)
search.clear()
search.send_keys(zipcode)
search.submit() #error occurs here
The error is:
NoSuchElementException: Message: no such element: Unable to locate
element:{"method":"xpath","selector":"./ancestor-or-self::form"}
(Session info: chrome=89.0.4389.114)

That <input> tag is not part of a form. You can't submit it. You either need to send a newline, or click the gps_fixed icon that triggers the search.

Try using the following:
input_field = driver.find_element_by_css_selector("input[id=wuSearch]")
input_field.click()
input_field.send_keys("some zip code")
Or just:
input_field = driver.find_element_by_id("wuSearch")
input_field.send_keys("some zip code")
Try to avoid click(). I am not sure it is really required.
After you input a zip code, you should wait for your search result to appear.

wait = WebDriverWait(driver, 10)
driver.get('https://www.wunderground.com/history/daily/us/dc/washington/KDCA')
zipcode='22202'
xpath='//*[#id="wuSearch"]'
search=wait.until(EC.element_to_be_clickable((By.XPATH,xpath)))
search.clear()
search.send_keys(zipcode)
time.sleep(1)
search.send_keys(Keys.ENTER)
Just delay it a bit. Use a webdriver wait to make sure you send the zipcode to the input and then send Keys.Enter.
Import
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from time import sleep

How to click on elements blocked by pop up using selenium python?

I want to click open the options in the menu on the left but they are blocked by the pop-up. How can I do that?
here is what I have tried:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
chromedriver = "/usr/share/chromedriver/chromedriver"
driver = webdriver.Chrome(chromedriver)
driver.get("https://student.amizone.net")
driver.find_element(By.NAME, "_UserName").send_keys("username")
driver.find_element(By.NAME, "_Password").send_keys("password")
driver.find_element(By.CSS_SELECTOR, "#loginform .login100-form-btn").click()
driver.implicitly_wait(10)
#11 | click | id=ModalPopAmityHostel |
driver.find_element(By.ID, "ModalPopAmityHostel").click()
# 11 | click | id=StudentSatisfactionPop | |
driver.find_element(By.ID, "StudentSatisfactionPop").click()
this code closes the 1st pop-up but doesn't close the 2nd pop-up. In the code first I login to the website https://student.amizone.net i have not shown my username and password(obviously). after that driver.find_element(By.ID, "ModalPopAmityHostel").click() is supposed to click outside the pop-up which closes the pop-up. similarly driver.find_element(By.ID, "StudentSatisfactionPop").click() is supposed to close the 2nd pop-up.
this is the snippet of html code of the pop up elements:
<form action="/PopUp/Home/SANGATHANQuizpopupSave" data-ajax="true" data-ajax-loading="#lodingDiv" data-ajax-method="POST" data-ajax-mode="replace" data-ajax-success=" $('#SANGATHANQuizpop').modal('hide');$('.modal-backdrop').remove();$(document.body).removeClass('modal-open');alertify.alert('Successfully Saved.'); " data-ajax-update="#Div_Partial" id="form0" method="post"></form>
<div id="StudentSatisfactionPop" class="modal fade in" role="dialog" aria-hidden="false" style="display: block; padding-right: 15px;">
<div class="modal-dialog">
<!-- Modal content-->
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal">×</button>
<h4 class="modal-title">STUDENT SATISFACTION SURVEY </h4>
</div>
<div class="modal-body">
<div>
<p>
<b>Dear Mr MANIK RAINA </b>
</p>
<p>
Please tell us about you! Office of Research, Planning & Statistical Services, Amity University Uttar Pradesh is conducting a survey of its students. This survey asks your opinion on many items relevant to examining the impact of college. It asks about your transition to university, your academic habits and experiences, your interaction with peers and faculty, your involvement in campus activities and programs, and how you spend your time.
</p>
<p>Results from this survey will be used by us to understand and improve your experiences. Data received from your responses will be analyzed and interpreted batch wise only, and at no time the responses be linked with an individual. Data will be used for revision and improvement of resource planning for better learning experiences of students of future batches. Be assured that your responses will remain confidential.</p>
<p>We would very much appreciate you taking a few moments to fill out the Survey Questionnaire.</p>
<p><b>
To Start the survey :
<a data-ajax="true" data-ajax-begin="$('#sidebar').removeClass('display');" data-ajax-loading="#lodingDiv" data-ajax-mode="replace" data-ajax-success=" $('#StudentSatisfactionPop').modal('hide');$('.modal-backdrop').remove();$(document.body).removeClass('modal-open');" data-ajax-update="#Div_Partial" href="/Survey/StudentSatisfaction" id="12" rel="0">Click Here</a>.
</b></p>
</div>
</div>
<div class="modal-footer">
<button type="button" id="btnClosevoter" class="btn btn-default" data-dismiss="modal">Close</button>
</div>
</div>
</div>
</div> <script>
$(document).ready(function () {
$('#StudentSatisfactionPop').modal('show');
});
</script>
<script>
$(document).ready(function () {
$('#ModalPopAmityHostel').modal('show');
});
</script>
<div id="ModalPopAmityHostel" class="modal fade in" role="dialog" aria-hidden="false" style="display: block;">
<div class="modal-dialog " style="z-index:104546464; ">
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal">×</button>
<h4 class="modal-title">Amity Hostel</h4>
</div>
<div class="modal-body">
<p class="text-center">
“A few hostel seats (A/C and Non A/C) are available for allocation. Students desirous of availing the seats may apply on Amizone and also contact hostel office. Allocation of seats will be done on ‘first come, first served’ basis.”
</p>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-primary" data-dismiss="modal">Close</button>
<h4 class="modal-title"></h4>
</div>
</div>
</div>
</div>

You can close popups by clicking on close buttons. In your code you click to the div instead of close buttons, that's why popups do not close (see correct locators for the close buttons).
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
chromedriver = "/usr/share/chromedriver/chromedriver"
driver = webdriver.Chrome(chromedriver)
wait = WebDriverWait(driver, 10)
driver.get("https://student.amizone.net")
driver.find_element(By.NAME, "_UserName").send_keys("username")
driver.find_element(By.NAME, "_Password").send_keys("password")
driver.find_element(By.CSS_SELECTOR, "#loginform .login100-form-btn").click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#ModalPopAmityHostel button.btn"))).click()
driver.execute_script("arguments[0].click()", driver.find_element_by_css_selector("#StudentSatisfactionPop button.btn"))
# wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#StudentSatisfactionPop button.btn"))).click()

As the <button> tags with text as Close are with in Modal Dialog Box so to locate and click() on the desired elements you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://student.amizone.net')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "form#loginform input[name='_UserName']"))).send_keys("7071804")
driver.find_element_by_css_selector("form#loginform input[name='_Password']").send_keys("62ae6f")
driver.find_element_by_css_selector("form#loginform button[type='submit']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#ModalPopAmityHostel div.modal-footer button.btn.btn-primary[data-dismiss='modal']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#StudentSatisfactionPop div.modal-footer button.btn.btn-default[data-dismiss='modal']"))).click()
Using XPATH:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://student.amizone.net')
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//form[#id='loginform']//input[#name='_UserName']"))).send_keys("7071804")
driver.find_element_by_xpath("//form[#id='loginform']//input[name='_Password']").send_keys("62ae6f")
driver.find_element_by_xpath("//form[#id='loginform']//button[type='submit']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='ModalPopAmityHostel']//div[#class='modal-footer']//button[#class='btn btn-primary' and #data-dismiss='modal']"))).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#id='StudentSatisfactionPop']//div[#class='modal-footer']//button[#class='btn btn-default' and #data-dismiss='modal']"))).click()

How to set the xpath for a element in html using python selenium webdriver

<div id="div_1" class="info-container">
<div class="name" data-toggle="dropdown"><div>
<div class="name" data-toggle="dropdown"><div>
<div id="div_2" class="btn-group user-helper-dropdown">
<i class="material-icons" data-toggle="dropdown"
aria-haspopup="true" aria-expanded="true"></i>
<ul id="div_3" class="dropdown-menu pull-right">
<li><a href="/profile/user/view"><i class="material
icons">person</i>Profile</a></li>
<li role="seperator" class="divider"></li>
<li id="sgn_out"><a id="sign_out_button" href="/sign-out"><i
class="material-icons">input</i>Sign Out</a></li>
</ul>
</div>
<div>
Hello I am having trouble in find the XPath for the tag which has id='sign_out_button' in selenium webdriver in Python.
This is the what I have tried:
driver.find_element_by_xpath("//div[#id='div_1']/div[#id='div_2']/ul[#id='div_3']/li[#id='sgn_out']/a")
But it gives me an error like this:
selenium.common.exceptions.NoSuchElementException: Message: no such element:
Unable to locate element:
{"method":"xpath","selector":"//div[#id='div_1']/div[#id='div_2']/ul[#id='div_3']/li[#id='sgn_out']/a"}

As per the HTML you have shared to invoke click() on the element with id='sign_out_button' as it is a dropdown item you have to induce WebDriverWait for the element to be clickable as follows:
CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "ul.dropdown-menu#div_3 li#sgn_out>a#sign_out_button"))).click()
XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//ul[#class='dropdown-menu pull-right' and #id='div_3']//li[#id='sgn_out']/a[#id='sign_out_button']"))).click()