Selenium version 3.141.0
I'm writing a web scraping script that should select a certain option from a dropdown menu with Selenium webdriver. The problem is, I cannot seem to detect this dropdown menu element. I tried detecting it with Class and by CSS selector, but it's still undetectable.
the dropdown menu is a status menu, it contains:
Draft
Submitted
Reviewed
Released
Rejected
Obsolete
This is the HTML code of the part of the page where the dropdown menu is:
<div class="controls col-md-5 angular2-multiselect" id="status-field">
<ctf-angular2-multiselect class="defaultSettings ng-valid ng-touched ng-dirty">
<div class="cuppa-dropdown" qa-name="dropdown-multiselect">
<div class="selected-list" tabindex="0">
<div class="c-btn" qa-name="toggle-dropdown-statusField">
<!----><!----><!---->
<span>
<!----><span qa-name="item-0">Draft</span>
</span>
<!----><!----><!---->
<div class="dropdown-caret"></div>
</div>
</div>
<div class="dropdown-container" qa-name="dropdown" hidden="">
<div class="dropdown-list">
<div class="list-area" qa-name="list-area">
<!----><!----><!----><!----><!---->
<ul class="lazyContainer">
<!----><!---->
<span>
<!---->
<li class="pure-checkbox single-select-label-selected">
<!----><label qa-name="item-0" title="Draft" class="single-select-label">Draft</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-1" title="Submitted" class="single-select-label">Submitted</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-2" title="Reviewed" class="single-select-label">Reviewed</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-3" title="Released" class="single-select-label">Released</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-4" title="Rejected" class="single-select-label">Rejected</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-5" title="Obsolete" class="single-select-label">Obsolete</label>
</li>
</span>
<!---->
</ul>
<!----><!----><!----><!---->
</div>
</div>
</div>
</div>
</ctf-angular2-multiselect>
</div>
Apparently I'm not that good with HTML, so I was depending on IDs to detect elements in the previous codes I wrote. This code doesn't have any.
This is how the GUI looks like:
Picture of GUI
I tried using classes to detect the dropdown menu like this:
Select(driver.find_element(By.CSS_SELECTOR, 'ctf-angular2-multiselect')).select_by_value("Released")
But it doesn't work.
Trying to detect with ID like this:
Select(driver.find_element_by_id('status-field')).select_by_value("Released")
doesn't work either
You have at least two challenges to overcome:
This isn't a normal html select. You need to click on things through Selenium.
The 'dropdown-container' is hidden. You need click on whatever opens it first.
Should we assume the div with the 'c-btn' class opens the drop down? I'd normally check in chrome
dev tools if that element has event listeners or use chrome to
save element as a variable and run a js click on it to verify it's
the right element to click before adding it in selenium
Once .dropdown-container is not longer hidden. I think your life would be easier with using xpath to select the list elements. to select the 'Released' option, the xpath would just be //label[#title='Released']
You can also use chrome dev tools to verify xpath selections before adding to your selenium script.
With python selenium, you can then click on your xpath selection like this:
driver.find_element(
By.XPATH, "//label[#title='Released']").click()
Related
I can't select values in a list. I tried using find_element_by_class_name() to open the menu but when I need to select a <li> returns that element doesn't have a function click().
Here the code:
click_menu = driver.find_element_by_class_name("periodSelector")
click_menu[1].click()
Here is the HTML that I am trying to parse:
<div data-period-selector="" data-period="periodFilter">
<div class="periodSelectorContainer">
<div class="btn-group periodSelector">
<button class="flat-btn dropdown-toggle periodToggle ng-binding" data-toggle="dropdown"> 20/02/2021 - 22/03/2021 <span class="dropdown-arrow"></span> </button>
<ul class="dropdown-menu">
<li>
<a href="javascript:void(0);" class="new-financ" ng-click="selectToday()"><i></i>
<span class="pull-left">Hoje</span>
<span class="pull-right"></span>
</a>
</li>
<li>
<a href="javascript:void(0);" class="new-financ" ng-click="selectThisWeek()"><i>
</li>
There are multiple class names you have to use a css selector.
click_menu = driver.find_element_by_css_selector("button.flat-btn.dropdown-toggle.periodToggle.ng-binding")
click_menu.click()
Clicks 1st li tag.
driver.find_element_by_xpath("ul[#class='dropdown-menu']/li[1]").click()
periodSelector is a class on a DIV
<div class="btn-group periodSelector">
I'm assuming that you need to click on the BUTTON
<button class="flat-btn dropdown-toggle periodToggle ng-binding" data-toggle="dropdown">
Most of those classes seem generic (probably not unique) but I'm guessing that periodToggle might be unique given the date range. Try
driver.find_element_by_css_selector("button.periodToggle").click()
NOTE:
You have an error in your code. You are using .find_element_by_class_name() (singular) but have array notation on the next line, click_menu[1]. In this case, you can just use click_menu.click(). You'd only need the array notation if you were using .find_elements_by_*() (note the plural, elements).
I am trying to scrape data from a web page. However, some of the data I need is in div elements with class "ReactModalPortal". The child elements of these divs are not available in the DOM until a button is clicked and the modal overlay appears. The text I need is in these child elements i.e. the text that appears in said modal overlay.
What is the quickest way of scraping this data in Python? I need to get this data off of thousands of pages and hence I want to avoid using a web driver/ browser automator (e.g. Selenium) to click the portal elements.
What the element looks like in the DOM prior to button click:
<div class="ReactModalPortal">
<div data-reactid=".1"></div>
</div>
What the element looks like after button click, containing the text I want to scrape:
<div class="ReactModalPortal">
<div data-reactid=".c" class="ReactModal__Overlay ReactModal__Overlay--after-open modalify-overlay">
<div class="ReactModal__Content ReactModal__Content--after-open modalify modalify-container" tabindex="-1" data-reactid=".c.0">
<div class="modalify-content" data-reactid=".c.0.0">
<div class="object-modifier" data-reactid=".c.0.0.0">
<div class="object-modifier__group" data-reactid=".c.0.0.0.0">
<div class="modifier-group__description" data-reactid=".c.0.0.0.0.2">TEXT I WANT TO SCRAPE.</div>
</div>
</div>
</div>
</div>
</div>
</div>
UPDATED
In my Django settings, when I set DEBUG to false, Selenium is able to interact with the elements. It still does not work when DEBUG is set to true
So I am trying to select one option in a dropdown menu, but when I try to directly access the <select> tag I get the error ElementNotVisibleException: Message: element not interactable.
In the console whenever I click on the dropdown in only refers to an input tag that dynamically changes based on what I click on. Ultimately, I want the test to click on the dropdown and then select 'Tango' as shown in the link below.
My Selenium code is:
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import Select
from selenium import webdriver
driver = webdriver.Chrome()
driver.get("localhost:8000/")
select = Select(driver.find_element_by_xpath('//*[#id="select-dance"]').click())
select.select_by_value('1')
And HTML:
<form onsubmit="return false;">
<div class="row border-light bg-white rounded-left rounded-right no-gutters">
<div class="col-12 col-lg px-3">
<div class="select-wrapper mdb-select">
<span class="caret"></span>
<input type="text" class="select-dropdown" readonly="true" data-activates="select-options-20f378f1-9560-4598-b8e8-
3ffe496cd688" value="Choose your dance event">
<ul id="select-options-20f378f1-9560-4598-b8e8-3ffe496cd688" class="dropdown-content select-dropdown" style="width: 658px; position:absolute;top: 0px; left: 0px;opacity: 1; display: none;">
<li class="">
<span class="filtrable">Choose your dance event</span></li>
<li class="">
<span class="filtrable">Tango</span></li>
<li class="active selected">
<span class="filtrable">Swing</span>
</li>
<li class="">
<span class="filtrable">Latin/Salsa</span>
</li>
<li class="">
<span class="filtrable">Ballroom</span>
</li>
<li class="">
<span class="filtrable">Bachata</span>
</li>
</ul>
<select class="mdb-select initialized" id="select-dance">
<option value="0">Choose your dance event</option>
<option value="1">Tango</option>
<option value="2">Swing</option>
<option value="3">Latin/Salsa</option>
<option value="4">Ballroom</option>
<option value="5">Bachata</option>
</select>
</div>
</div>
</div>
</div>
</form>
I do not know how to deal with the input tag interfering with the select (it only shows up in the elements console in Chrome).
Thank you!
You can handle by,
driver = webdriver.Chrome()
driver.get("localhost:8000/")
time.sleep(10)
select = Select(driver.find_element_by_id('select-dance'))
select.select_by_value('1')
You can try this,
driver = webdriver.Chrome()
driver.get("localhost:8000/")
time.sleep(10)
driver.find_element_by_id('select-dance').click()
when you click then new list of s will get populated under , then call select tag
select = Select(driver.find_element_by_id('select-dance'))
select.select_by_value('1')
or
select = Select(driver.find_element_by_xpath('//*[#id="select-dance"]'))
select.select_by_value('1')
Your page uses custom Select Component which is not the default html select box. In your case, they have used MDBootstrap Select Component which cannot be interacted using selenium Select class.
Please read this answer for your another question which related to the same issue.
I have my script to login to a site. i then need to click on another link which is contained in an
<a href> </a>
I have tried multiple methods without success. The link I need "Available Deployments" only appears after clicking a dropdown box called "Job Board".
The site code looks like this:
<li class="">
<a aria-expanded="false" class="dropdown-toggle" data-toggle="dropdown" href="portalPost?s=a1W390000045MxAEAU&p=a1V39000003y7e1EAA" role="button">Job Board <span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="portalPage?s=a1W390000045MxAEAU&p=a1V39000003y7dbEAA">Available Deployments
</a>
</li>
i've tried a couple of versions, without success:
-SNIP-
driver.find_element_by_name("logmein").click()
driver.find_element_by_linkText("Job Board").click()
driver.find_element_by_linkText("Available Deployments").click()
and
-SNIP-
driver.find_element_by_name("logmein").click()
driver.find_element_by_xpath(u'//a[text()="Job Board"]').click()
driver.find_element_by_xpath(u'//a[text()="Available Deployments"]').click()
The errors I get typically look like:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[text()="Available Deployments"]"}
It looks like there is a whitespace in text of the element.
You have to use normalize-space to trim text. See an example below.
driver.find_element_by_xpath(u'//a[text()="Job Board"]').click()
waitForPresence = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.XPATH,'//a[normalize-space(text())="Available Deployments"]')))
driver.find_element_by_xpath('//a[normalize-space(text())="Available Deployments"]').click()
I solved this with the following:
driver.find_element_by_link_text("Job Board").click()
driver.find_element_by_link_text("Available Deployments").click()
I clicked on a link which opens a small window/popup/iframe and the popup window has Xpath:
//html/body/div[2]
ie. it has no window_id but in firebug the identifier shows the window as a <div class="some_name"> so I'm assuming it's a popup window; correct me if I'm wrong.
When I use the selectPopup of selenium IDE it works fine in switching from the main window to the popup/new_window but when exporting to webdriver-python it doesn't have that option. So I tried using driver.switch_to_window and switch_to_frame along with the xpath mentioned above but no luck ie. driver.switch_to_frame(driver.find_element_by_xpath("//html/body/div[2]"))
Error thrown : NoSuchElementException and that's because it's not able to select the iframe.
Since it's working fine in Selenium IDE I exported it to python-webdriver which converted the wait_for_element clause to
for i in range(60):
try:
if self.is_element_present(By.XPATH, "//*[#id='heading']/div[2]/div/div/ul/li[2]/a"): break
except: pass
time.sleep(1) else: self.fail("time out")
which returns the above error
Detailed html:
//*[#id='heading']/div[2]/div/div/ul/li[2]/a is the xpath of the element and as html this is what it is Home and in detail:
`
<div class="help">
<div class="page-header">
<div id="heading">
<div id="search">
<div class="navbar navbar-fixed-top">
<div class="navbar-inner">
<div class="container">
<ul class="nav">
<li class="active">
<li>
Home
</li>
<li>
<li>
<li>
</ul>
<form class="navbar-form pull-right">
</div>
</div>
</div>
</div>`
try out this :
new Actions(driver).click(driver.findElement(By.xpath("//div[#id='pop-up-window']"))).perform();