Trouble finding element in Selenium with Python - python

I have been trying to collect a list of live channels/viewers on Youtube Gaming. I am using selenium with Python to force the website to scroll down the page so it loads more that 11 channels. For reference, this is the webpage I am working on.
I have found the location of the data I want, but I am struggling with getting selenium to go there. The part I am having trouble with looks like this:
<div class="style-scope ytg-gaming-video-renderer" id="video-metadata"><span class="title ellipsis-2 style-scope ytg-gaming-video-renderer"><ytg-nav-endpoint class="style-scope ytg-gaming-video-renderer x-scope ytg-nav-endpoint-2"><a href="/watch?v=FFKSD1HHrdA" tabindex="0" class="style-scope ytg-nav-endpoint" target="_blank">
Live met Bo3
</a></ytg-nav-endpoint></span>
<div class="channel-info small layout horizontal center style-scope ytg-gaming-video-renderer">
<ytg-owner-badges class="style-scope ytg-gaming-video-renderer x-scope ytg-owner-badges-0">
<template class="style-scope ytg-owner-badges" is="dom-repeat"></template>
</ytg-owner-badges>
<ytg-formatted-string class="style-scope ytg-gaming-video-renderer">
<ytg-nav-endpoint class="style-scope ytg-formatted-string x-scope ytg-nav-endpoint-2">Rico Eeman
</ytg-nav-endpoint>
</ytg-formatted-string>
</div><span class="ellipsis-1 small style-scope ytg-gaming-video-renderer" id="video-viewership-info" hidden=""></span>
<div id="metadata-badges" class="small style-scope ytg-gaming-video-renderer">
<ytg-live-badge-renderer class="style-scope ytg-gaming-video-renderer x-scope ytg-live-badge-renderer-1">
<template class="style-scope ytg-live-badge-renderer" is="dom-if"></template>
<span aria-label="" class="text layout horizontal center style-scope ytg-live-badge-renderer">4 watching</span>
<template class="style-scope ytg-live-badge-renderer" is="dom-if"></template>
</ytg-live-badge-renderer>
</div>
</div>
Currently, I am trying:
#This part works fine. I can use the unique ID
meta_data = driver.find_element_by_id('video-metadata')
#This part is also fine. Once again, it has an ID.
viewers = meta_data.find_element_by_id('metadata-badges')
print(viewers.text)
However, I am have been having trouble getting to the channel name (in this example 'Rico Eeman', and it is under the first nested div tag). Because its a compound class name, I cannot find the element by class name, and trying the following xpaths doesnt work:
name = meta_data.find_element_by_xpath('/div[#class="channel-info small layout horizontal center style-scope ytg-gaming-video-renderer"]/ytg-formatted-string'
name = meta_data.find_element_by_xpath('/div[1])
They both raise the element not found error. I am not really sure what to do here. Does anyone have a working solution?

The name id not in the <ytg-formatted-string> tag, its in one of it descendants. Try
meta_data.find_element_by_css_selector('.style-scope.ytg-formatted-string.x-scope.ytg-nav-endpoint-2 > a')
Or with xpath
meta_data.find_element_by_xpath('//ytg-nav-endpoint[#class="style-scope ytg-formatted-string x-scope ytg-nav-endpoint-2"]/a')

This will get all the names, even if your xpath worked using video-metadata would not get all the names, the id is repeated per div for each user so you would need find_elements and to iterate over the returned elements:
names = dr.find_elements_by_css_selector("a.style-scope.ytg-nav-endpoint[href^='/channel/']")
print([name.get_attribute("text") for name in names])
Which gives you:
['NinjaNation Gaming', 'DURX DANIEL', 'DEMON', 'Perfection', 'The one and only jd', 'Violator Games', 'KingLuii718', 'NinjaNation Gaming', 'DURX DANIEL', 'DEMON', 'Perfection']

Related

Finding/locating a clickable text by xpath in div/span format

I used the following line to click the Availability Grid button, but it failed to locate the element.
Class Sarsa-button-content is used everywhere so, I added text together to make it unique. However, it couldn't find it. What am I missing?
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//a[#class='sarsa-button-content']/span[text()='Availability Grid']"))).click()
<div class="sticky-top-wrapper" style="top: 80px;">
<div class="site-filter-container" id="site-filter-container">
<a data-component="Button" class="sarsa-button view-by-availability-grid--button-tracker sarsa-button-primary sarsa-button-sm" href="/site/123456/availability">
<span class="sarsa-button-inner-wrapper">
<span class="sarsa-button-content">Availability Grid</span>
your xpath is wrong, try this:
//span[contains(#class, 'sarsa-button-content') and text() = 'Availability Grid']

Error Message: Unable to locate element. What may be the problem? Python 3.7, Selenium 3.141.0

When using selenium to find an element tag using Xpath, it works using the xpath copied from chrome for that element, this was to test if it was working. Since the id element changes every time I open the webpage I am aware that this will cause an error in my web automation program, now from that same tag I am using the class element as it does not change every time I open the webpage but Selenium fails to locate that element at all. When I change the element in xpath to class with its relative name from the xpath I initially copied and tested from the webbroswer. What may be the problem?
Here is a example of the html source code I am wanting to go to and the Xpath:
<div class="popover fade right in" role="tooltip" id="popover784483" style="top: -9px; left: 328.5px; display: block;">
<div class="arrow" style="top: 88.9423%;"></div>
<h3 class="popover-title">
<font style="vertical-align: inherit;">
<font style="vertical-align: inherit;">Member details</font>
</font>
</h3>
<div class="popover-content">
<img class="center-block" src="http://api.vdarts.net:8080/picture/portrait/default.png" style="max-width:200px;height:auto;">
<hr>
<font style="vertical-align: inherit;">
<font style="vertical-align: inherit;">Account: CapitainJack </font>
Xpath intitially used but id changes:
//*[#id="popover784483"]/div[2]/font[1]/font
Xpath to be used:
//*[#class="popover fade right in"]/div[2]/font[1]/font
There are many ways to approach this issue.
You can match partial Text content Xpath like
//*[contains(text(), 'Account')]
You can use Img tag and following font like
(//img//following::font)[2]
I would recommend you use CSS selectors instead.
Compound class names are when an element has multiple classes separated by spaces. E.g. <div class="class1 class2 class3"></div>
if you want to select element with all 3 classes (in any order), you can use:
element = driver.find_element_by_css_selector(".class1.class2.class3")
If you want to use XPath, I have found that the following method works (using the same example as above):
browser.find_elements_by_xpath("//*[contains(concat(' ',#class,' '),' class1 ') and contains(concat(' ',#class,' '),' class2 ') and contains(concat(' ',#class,' '),' class3 ')]")
Note: you can use different boolean operators to filter out certain class names (e.g. 'and' 'and not' etc)

Trying to use Python + XPath to click the highlighted text?

Fairly new to coding and Python, I'm trying to use find_element_by_xpath to click the text highlighted text "Snoring Chin Strap by TheFamilyMarket".
time.sleep(2)
#btn = br.find_element_by_name("#Anti Snoring Chin Strap Kit")
# btn = br.find_element_by_link_text('Snoring Chin Strap')
The HTML code:
<div class="tableD">
<div class="productDiv" id="productDiv69507">
<h2 class="productTitle" id="productTitle69507" onclick="goToProduct(7)">Snoring Chin Strap by TheFamilyMarket</h2>
<img class="productImage" src="https://images-na.ssl-images-amazon.com/images/I/516fC3JruqL.jpg" onclick="goToProduct(7)">
<hr>
<h4 class="normalPrice" id="normalPrice7" onclick="goToProduct(7)">Normally: <span class="currency">$ </span>19.99</h4>
<h4 class="promoPrice" style="margin:2.5px auto;" id="promoPrice69507" onclick="goToProduct(7)">Your Amazon Price: <span class="currency">$ </span>1.99</h4>
<h3>Your Total: <span class="currency">$ </span>1.99</h3>
<p class="clickToViewP" id="cToVP69507" onclick="goToProduct(7)">Click to view and purchase!</p>
</div>
</div>
br.find_element_by_xpath("//h2[text()='Snoring Chin Strap by TheFamilyMarket']");
XPath is sometimes fast to get because you can get it from the browser, and that's why so many people use it, but in my opinion for long term, learning JavaScript and CSS selectors can help you in many instances in the future.
The above can be done also by selecting all the h2 elements and looking for text using plain JavaScript and passing the result to python:
link_you_search = br.execute_script('''
links= document.querySelectorAll("h2");
for (link of links) if (link.textContent.includes("Chin Strap")) return link;
''')
link_you_search.click()
or alternatively you can select by class:
link_you_search = br.execute_script('''
links= document.querySelectorAll(".productDiv");
for (link of links) if (link.textContent.includes("Chin Strap")) return link;
''')
link_you_search.click()
given that your element has an id attribute usually selecting by id it is best practice since it is the fastest search and you should only have only one element with that id and usually ids don't change so often in case of translation etc, so in your case it would be:
link_you_search = br.find_element_by_id('productTitle69507')
link_you_search.click()

Robot Framework anchor tag value is not retrieved

<div class="m-page-nav m-bottom-2" data-hbui="mobile-nav" role="navigation">
<span aria-`enter code here`label="Collapse page navigation" class="a-icon m-page-nav-icon m-page-nav-icon-collapse">expand_more</span>
<span aria-label="Expand page navigation" class="a-icon m-page-nav-icon m-page-nav-icon-expand">expand_more</span>
<button data-scroll-header class="m-page-nav-button"></button>
<div data-gumshoe class="m-page-nav-list">
<a data-scroll href="#intro" tabindex="0">Intro</a>
<a data-scroll href="#courseHeadingDescription" tabindex="0">Enrolment Disclaimer</a>
<a data-scroll href="#admissionRequirements" tabindex="0">Admission Requirements</a>
<a data-scroll href="#programRequirements" tabindex="0">Program Requirements</a>
<a data-scroll href="#professionalOutcomes" tabindex="0">Professional Outcomes</a>
<a data-scroll href="#recognitionMasters" tabindex="0">Recognition of Achievement</a>
<a data-scroll href="#fees" tabindex="0">Program Fees</a>
</div>
</div>
I am kind of new to robot framework. I am trying to retrieve the value from the href anchor tag value that is Intro using the following xpath
//*[#id="pageNavContainer"]/div/div/a[2]/#href
But when I run the script to get the text with the above xpath I am getting the below error:
InvalidSelectorException: Message: invalid selector: The result of the xpath express
I tried using getattribute also but it didn't work.
Can anyone please help me out what is the correct xpath for this?
Thanks in advance
The error you received says it all - the selector you provided is not a valid one; the problem comes from the /#href construct at the end.
In XPath one can for sure address a particular node's attribute with this syntax. Yet in Selenium (which Robotframework wraps), a locator has to point to an element, not its attribute - thus when it evaluates your expression, it throws the exception.
If you change the locator to
//*[#id="pageNavContainer"]/div/div/a[2]
, and the use Get Element Attribute passing href as the target, you'll get the desired value.
To extract the text Intro you can use the following xpath :
//*[#id="pageNavContainer"]//div[#class='m-page-nav m-bottom-2']//div[#class='m-page-nav-list']//following-sibling::a[1]
Next you can use the Get Element Attribute to extract the href :
${HREF}= Get Element Attribute ${element_xpath}#href

Webdriver - Locate Input via Label (Python)

How do I locate an input field via its label using webdriver?
I like to test a certain web form which unfortunately uses dynamically generated
ids, so they're unsuitable as identifiers.
Yet, the labels associated with each web element strike me as suitable.
Unfortunately I was not able to do it with the few suggestions
offered on the web. There is one thread here at SO, but which did not
yield an accepted answer:
Selenium WebDriver Java - Clicking on element by label not working on certain labels
To solve this problem in Java, it is commonly suggested to locate the label as an anchor via its text content and then specifying the xpath to the input element:
//label[contains(text(), 'TEXT_TO_FIND')]
I am not sure how to do this in python though.
My web element:
<div class="InputText">
<label for="INPUT">
<span>
LABEL TEXT
</span>
<span id="idd" class="Required" title="required">
*
</span>
</label>
<span class="Text">
<input id="INPUT" class="Text ColouredFocus" type="text" onchange="var wcall=wicketAjaxPost(';jsessionid= ... ;" maxlength="30" name="z1013400259" value=""></input>
</span>
<div class="RequiredLabel"> … </div>
<span> … </span>
</div>
Unfortunately I was not able to use CSS or XPATH expressions
on the site. IDs and names always changed.
The only solution to my problem I found was a dirty one - parsing
the source code of the page and extract the ids by string operations.
Certainly this is not the way webdriver was intended to be used, but
it works robustly.
Code:
lines = []
for line in driver.page_source.splitlines():
lines.append(line)
if 'LABEL TEXT 1' in line:
id_l1 = lines[-2].split('"', 2)[1]
You should start with a div and check that there is a label with an appropriate span inside, then get the input element from the span tag with class Text:
//div[#class='InputText' and contains(label/span, 'TEXT_TO_FIND')]/span[#class='Text']/input

Categories