We Keep Coding

Web Scraping with Selenium - Detecting a dropdown menu

Selenium version 3.141.0
I'm writing a web scraping script that should select a certain option from a dropdown menu with Selenium webdriver. The problem is, I cannot seem to detect this dropdown menu element. I tried detecting it with Class and by CSS selector, but it's still undetectable.
the dropdown menu is a status menu, it contains:
Draft
Submitted
Reviewed
Released
Rejected
Obsolete
This is the HTML code of the part of the page where the dropdown menu is:
<div class="controls col-md-5 angular2-multiselect" id="status-field">
<ctf-angular2-multiselect class="defaultSettings ng-valid ng-touched ng-dirty">
<div class="cuppa-dropdown" qa-name="dropdown-multiselect">
<div class="selected-list" tabindex="0">
<div class="c-btn" qa-name="toggle-dropdown-statusField">
<!----><!----><!---->
<span>
<!----><span qa-name="item-0">Draft</span>
</span>
<!----><!----><!---->
<div class="dropdown-caret"></div>
</div>
</div>
<div class="dropdown-container" qa-name="dropdown" hidden="">
<div class="dropdown-list">
<div class="list-area" qa-name="list-area">
<!----><!----><!----><!----><!---->
<ul class="lazyContainer">
<!----><!---->
<span>
<!---->
<li class="pure-checkbox single-select-label-selected">
<!----><label qa-name="item-0" title="Draft" class="single-select-label">Draft</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-1" title="Submitted" class="single-select-label">Submitted</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-2" title="Reviewed" class="single-select-label">Reviewed</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-3" title="Released" class="single-select-label">Released</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-4" title="Rejected" class="single-select-label">Rejected</label>
</li>
<li class="pure-checkbox">
<!----><label qa-name="item-5" title="Obsolete" class="single-select-label">Obsolete</label>
</li>
</span>
<!---->
</ul>
<!----><!----><!----><!---->
</div>
</div>
</div>
</div>
</ctf-angular2-multiselect>
</div>
Apparently I'm not that good with HTML, so I was depending on IDs to detect elements in the previous codes I wrote. This code doesn't have any.
This is how the GUI looks like:
Picture of GUI
I tried using classes to detect the dropdown menu like this:
Select(driver.find_element(By.CSS_SELECTOR, 'ctf-angular2-multiselect')).select_by_value("Released")
But it doesn't work.
Trying to detect with ID like this:
Select(driver.find_element_by_id('status-field')).select_by_value("Released")
doesn't work either

You have at least two challenges to overcome:
This isn't a normal html select. You need to click on things through Selenium.
The 'dropdown-container' is hidden. You need click on whatever opens it first.
Should we assume the div with the 'c-btn' class opens the drop down? I'd normally check in chrome
dev tools if that element has event listeners or use chrome to
save element as a variable and run a js click on it to verify it's
the right element to click before adding it in selenium
Once .dropdown-container is not longer hidden. I think your life would be easier with using xpath to select the list elements. to select the 'Released' option, the xpath would just be //label[#title='Released']
You can also use chrome dev tools to verify xpath selections before adding to your selenium script.
With python selenium, you can then click on your xpath selection like this:
driver.find_element(
By.XPATH, "//label[#title='Released']").click()

Python Selenium Click button inside span tag using button text

The html text below exits on a page. I am trying to have selenium click the button that this tag is referring to. I can not search this href because the id for given app is unknown and link will always have a different id. I have to click on this button by locating it by text inside button which is "New Database/Server"
<a class="btn btn-primary" data-turbo-frame="modal" href="/repo/servers/new?repo_application_id=280"><svg class="bi flex-shrink-0 me-2" height="24" role="img" width="24"><use href="#icon_database-plus"></use></svg><span class="pe-3">New Database/Server</span></a>
How do I have the driver go to this href link?

Use following xpath to click on the button.
driver.find_element(By.XPATH, "//a[.//span[text()='New Database/Server']]").click()
or use link_text
driver.find_element(By.LINK_TEXT, "New Database/Server").click()

selenium python find link by href text and click it

I have my script to login to a site. i then need to click on another link which is contained in an
<a href> </a>
I have tried multiple methods without success. The link I need "Available Deployments" only appears after clicking a dropdown box called "Job Board".
The site code looks like this:
<li class="">
<a aria-expanded="false" class="dropdown-toggle" data-toggle="dropdown" href="portalPost?s=a1W390000045MxAEAU&p=a1V39000003y7e1EAA" role="button">Job Board <span class="caret"></span>
</a>
<ul class="dropdown-menu" role="menu">
<li>
<a href="portalPage?s=a1W390000045MxAEAU&p=a1V39000003y7dbEAA">Available Deployments
</a>
</li>
i've tried a couple of versions, without success:
-SNIP-
driver.find_element_by_name("logmein").click()
driver.find_element_by_linkText("Job Board").click()
driver.find_element_by_linkText("Available Deployments").click()
and
-SNIP-
driver.find_element_by_name("logmein").click()
driver.find_element_by_xpath(u'//a[text()="Job Board"]').click()
driver.find_element_by_xpath(u'//a[text()="Available Deployments"]').click()
The errors I get typically look like:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[text()="Available Deployments"]"}

It looks like there is a whitespace in text of the element.
You have to use normalize-space to trim text. See an example below.
driver.find_element_by_xpath(u'//a[text()="Job Board"]').click()
waitForPresence = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.XPATH,'//a[normalize-space(text())="Available Deployments"]')))
driver.find_element_by_xpath('//a[normalize-space(text())="Available Deployments"]').click()

I solved this with the following:
driver.find_element_by_link_text("Job Board").click()
driver.find_element_by_link_text("Available Deployments").click()

Navigating website using selenium

I am trying to scrape information from a website but am having trouble navigating it using Selenium. The site uses ng-click to update a table so I must activate different tabs on the page to get the information I want. This is the html that generates the tabs:
<ul class="tabs swiper-wrapper" ng-class="{'swiper-wrapper' : swiperActive }">
<li ng-repeat="category in Report.Winners track by $index" ng-click="updateCategory(category.key)" ng-class="{'active' : category.key == activeCategory, 'swiper-slide' : swiperActive }" class="ng-scope active">
<p class="category text-small ng-binding">Category 1</p>
<p class="winner">
</p>
</li><li ng-repeat="category in Report.Winners track by $index" ng-click="updateCategory(category.key)" ng-class="{'active' : category.key == activeCategory, 'swiper-slide' : swiperActive }" class="ng-scope">
<p class="category text-small ng-binding">Category 2</p>
<p class="winner">
</p>
</li><li ng-repeat="category in Report.Winners track by $index" ng-click="updateCategory(category.key)" ng-class="{'active' : category.key == activeCategory, 'swiper-slide' : swiperActive }" class="ng-scope">
<p class="category text-small ng-binding">Category 3</p>
<p class="winner">
</p>
</li>
</ul>
I have figured out how to scrape the information from "Category 1" since it loads by default. How do I navigate to "Category 2" and "Category 3" so I can scrape those as well? Thanks!
Update:
I ended up using this to find the links for each category:
available_categories_links = browser.find_elements_by_css_selector("ul > [ng-click*=updateCategory]")
And then I loop through them like this:
for x in range(len(available_categories_links)):
available_categories_links[x].click()
Doing it this way doesn't let me access different tabs by name like I had originally hoped to do, and it's probably not the most efficient or very robust, but it gets the job done in my particular case.

I assume your tabs load on runtime.
So, to activate the tab, you need to use FindElement(By) to locate the tab and click it before you could access the information from that tab.
According to your code, no Id found to use FindElement(By.Id). So I suggest you use either FindElement(By.CSSSelector) or FindElement(By.Xpath) which you could copy the Locator string from browser's development tool, i.e. in Google Chrome right click==> inspect==>Copy==>Copy Selector or Copy Xpath.
After you got the Tab Element, i.e.
IWebElement tab = driver.FindElement(By.CSSSelector);
tab.Click();//Tab activated
//...Do your thing afterwards.
Hope this helps.

want to find link/text and Click on it

I am using fake mail generator tool to send a mail and click on a link in the mail..
enter link description here
<iframe id="emailFrame" frameborder="0" width="100%" height="360px" src="http://www.fakemailgenerator.com/email/dayrep.com/testlk/message-108204614/" scrolling="no" onload="autoResize('emailFrame')">
<html>
<head>
<body>
<div>
<p>Good day - </p>
<p>You have been assigned an Action from the motion A Name iwhirxpppk: s.
</p>
<p>Kindly follow - up on the Touchpoint Action listed below.
</p>
<ul>
Please click the below link to complete your Action.
<p>
<a target="_blank" href="http://cfn- svr001.cloudapp.net:7100/Home/ActionResponse?eid=ygfWFB5a99mtAUQBxjNUDHjpC9AdFz/9&tpid=14iwlvior8ak6FGifOI3MSBNxnNvHiT9">Click here
</a>
</p>
This email has been generated from CFN Insight by Auto man, auto#
</div>
</body>
</html>
</iframe>
want to find below part of the code
<p>
<a target="_blank" href="http://cfn-svr001.cloudapp.net:7100/Home/ActionResponse?eid=ygfWFB5a99mtAUQBxjNUDHjpC9AdFz/9&tpid=14iwlvior8ak6FGifOI3MSBNxnNvHiT9">Click here</a>
</p>
I tried all possible combinations what ever I know but nothing helped me..
Here are the scripts which I tried
browser = webdriver.Firefox() # for b in
browser.find_element_by_id('emailFrame').find_elements_by_xp‌​ath(".//*"):
print b # browser.find_element_by_xpath("html/body/div[1]/p[4]/a") #
browser.find_element_by_xpath(".//*[text()='Click here']") #
browser.find_element_by_xpath[".//a[contains(., 'Click here')]"] #
browser.find_element_by_xpath(".//div[1]//p[4]/a")
browser.find_element_by_id('emailFrame').find_elements_by_ta‌​g_name("a"):

First, you should switch to the frame and then find the element inside it.
try as follows:
from selenium import webdriver
from selenium.webdriver.common.by import By
browser = webdriver.Firefox()
browser.maximize_window()
browser.get("http://www.fakemailgenerator.com/inbox/armyspy.com/morly1985/")
frame = browser.find_element(By.ID, 'emailFrame')
browser.switch_to_frame(frame) # switch to frame by finding the iframe webelement
click_here = browser.find_element_by_xpath(".//*[text()='Click here']") # try different xpaths.
click_here.location_once_scrolled_into_view # As the element needs scroll down
click_here.click()
References:
https://stackoverflow.com/a/40759300/2575259
http://selenium-python.readthedocs.io/navigating.html#moving-between-windows-and-frames
http://selenium-python.readthedocs.io/api.html