Web Scraping with Selenium questions, no results with any method - python

When hitting inspect the element on a paragraph, I get directed to a tag. If it is empty, Where is the text that I see on the screen coming from?
I have attempted:
driver.find_elements_by_css_selector('the class of the textarea')
driver.find_elements_by_id("the id of the textarea")
driver.find_elements_by_class("the class of the textarea")
I've also tried the ladder methods with the parent divs of the text area. nothing.
Then I did the following:
I copied the text and cntrl+F to find it in the HTML, I was directed to a tag. I tried the following:
everything = driver.find_elements_by_tag_name("script")
for item in everything:
print(driver.execute_script(item.text))
I get a list of Nones printed out.
I have been stuck at this for days.
This is the tag:
<textarea style="width: 658px; height: 128px; overflow: auto;" autocomplete="off" id="ContentPH_description" name="ContentPH_description" role="textbox" aria-readonly="false" aria-required="false" aria-multiline="fals" class="x-form-textarea x-form-field vms-viewmode-view-set" readonly="" aria-labelledby="ContentPH_description_label" title="" aria-invalid="false" maxlength="10000" oldindex="0" tabindex="-1"></textarea>
I, unfortunately, cannot share a link or the full HTML of the page since it is work-related and you'd need a login to access the data I am trying to scrape.
What can I do to solve this?

I don't know if it will help you, but when I try to access visual hidden text elements (i.e. text from a collapsed tab), I always replace element.text to element.get_attribute('textContent') or element.get_attribute("innerText").

Related

How do you write in Robot Framework a dynamic element with multiple identifiers?

I am new to Robot Framework as well as writing automation using Pycharm and have run into an issue I cannot find an answer to. I have text boxes that the uid changes every time a change is made to the page. I have tried using an xpath but all of the text boxes appear to have the same starting ID of 56$551056-- so i cannot use xpath=//input[contains(#id='56$551056--')] without trying to add something additional to it.
Example TextBox1
<input type="text" class="gwt-TextBox WNN2" data-automation-id="textInputBox" tabindex="0" role="textbox" id="56$551056--uid74-input" dir="ltr" aria-invalid="false" xpath="1">
Example TextBox2
<input type="text" class="gwt-TextBox WNN2" data-automation-id="textInputBox" tabindex="0" role="textbox" id="56$551056--uid75-input" dir="ltr" aria-invalid="false" xpath="1">
There are 3-5 boxes in each section I am automating and they all seem to have the exact same code except for the uid. Removing ID and Class name (as these both throw a warning in ChroPath of being dynamic just gives me a string of //div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//div//li[2]//div[2]//div[1]//div[1]
Any help or direction on this greatly appreciated.
Update
So I have these paths connecting the ID and the Label but I would have to remove uid100 as this is the part that continues to change on the page. I have tried several different combinations to get this to work in RF but I cannot seem to get it exactly right. //label[#id='56$551056--uid100-formLabel'] //div[contains(text(),'First Name')] //*[#id="56$551056--uid100-input"]
<li class="WDSO WOTO" role="presentation" data-automation-id="formLabelRequired"><div class="WFSO WIRO"><label id="56$551056--uid430-formLabel" data-automation-id="formLabel" for="56$551056--uid430-input">First Name</label><div class="WISO wd-74ef43c8-48bb-445c-9b49-1a788d7c4b10" aria-hidden="true">First Name</div></div><div data-automation-id="decorationWrapper" id="56$551056" class="WJSO"><div class="WOAJ"><div class="WHN2 textInput WGN2 WJ5" data-automation-id="textInput" id="56$551056--uid430" data-metadata-id="56$551056" style="visibility: visible;"><input type="text" class="gwt-TextBox WON2" data-automation-id="textInputBox" tabindex="0" role="textbox" id="56$551056--uid430-input" dir="ltr" aria-invalid="false" aria-required="true"></div></div></div></li>
The best solution is to work with the developers to provide unique attributes for each input element.
If you can't do that, and if you can't easily uniquely identify an input element, then find the nearest element that you can uniquely identify, and use that as a base for finding the input element.
For example, if the input element is associate with a label -- and most are -- you can use the label element as part of the xpath. For example, //label[#text='First Name']//input. Exactly how you do that depends on the structure of the page - the input element might be the first sibling, or the first child, or some element elsewhere in the DOM but pointed to by an attribute in the label (eg: <label for= "56$551056--uid74-input">)
This question isn't unique to robot framework, it's common to just about every test automation framework.
Try to insert item label into your xpath instead of id. For example, if your textbox having the label Customer Name, then something will be like:
//label[text()='Customer Name']/../../input
(The dark part is my assumption)
So, you could get rid of the annoying dynamic ID
Please provide your full HTML, so I could give a better answer
Find the element by using xpath locators using the reference to parent element of the following element
<input type="text" class="gwt-TextBox WNN2" data-automation-id="textInputBox" tabindex="0" role="textbox" id="56$551056--uid74-input" dir="ltr" aria-invalid="false" xpath="1">
xpath example would be as follows:
xpath of parent//input[contains(#class,'TextBox')]

Selecting dynamic element using selenium and python

Could someone please tell me how can I select a dynamic element using selenium?
I would like to select the "limit-order" element.
<div class="tab-control" id="uniqName_0_85" widgetid="uniqName_0_85">
<span data-tab="market-order" class="tab-item tab-active">Market</span>
<span data-tab="limit-order" class="tab-item">Limit</span>
<span data-tab="stop-order" class="tab-item">Stop</span>
<span data-tab="stop_limit-order" class="tab-item">Stop Limit</span>
</div>
I tried this but no luck:
btn_limit_name_xpath = '//div[contains(#class,"tab-control")]/span[2]'
btn_limit = browser.find_element_by_xpath(btn_limit_name_xpath)
btn_limit.click()
What sometimes does the job for me is copy the full xpath instead of the shorter one.
If that doesn't work either, you could try and check this out.
They show you how you can use an xpath to find a specific piece of text and select the object in that way. So in your case you could try and find it by searching for 'limit'.

How to find element in selenium python

I have been using selenium for a little bit now and have been stuck on this issue for the past few hours. It seems really simple but I just can't seem to figure out a proper execution. I basically am trying to select a shoe size based on the user's input. Here are a few of the buttons I am trying to sort through as well as the surrounding HTML.
<p class="checkbox-size">
<input type="radio" value="500" id="super_attribute[150]_500" name="super_attribute[150]" class="product_attribute">
<label for="super_attribute[150]_500">
<span id="label_eu0" class="label_hidden"> 38.5</span>
<span id="label_us0" class="label_show"> 6</span>
<span id="label_uk0" class="label_hidden"> 5.5</span>
<span id="label_cm0" class="label_hidden"> 24</span>
</label>
</p>
The specific button I am trying to press is this one:
<label for="super_attribute[150]_137">
<span id="label_eu10" class="label_hidden"> 45</span>
<span id="label_us10" class="label_show"> 11</span>
<span id="label_uk10" class="label_hidden"> 10</span>
<span id="label_cm10" class="label_hidden"> 29</span>
</label>
Now I tried many different methods of searching then clicking one of the buttons but nothing has worked yet. Any suggestions? Here is what I am currently using to try and find and click the button:
driver.find_element_by_css_selector("input[type='radio'][value='11']").click()
Looking back I might have not provided the correct code for the buttons, so here is a snapshot of the inspect element as well as the actual page if you want to go check it out for yourself. I am trying to click the size buttons. Button inspect element
Solved!
Here is the code I ended up using.
sizes = driver.find_elements_by_class_name('checkbox-size')
for size in sizes:
if size.text in [usersize]:
size.click()
print colored('Carted size %s'%(size.text), 'green')
break
continue
There are a few problems with your approach.
You are trying to click the INPUT when you need to click a SPAN
The sizes on the page contain some amount of whitespace which makes your strict string comparison, value='11', not work. Also the contained text is not considered a value.
Another issue you will run into is that all of the sizes aren't displayed. The sizes displayed is controlled by the links above the sizes, EU, US, UK, and CM. The ones that are displayed have class="label_show" so you will want to specify that in your answer or you will attempt to click elements that are not visible which will throw an exception. (Selenium only interacts with visible elements by design).
With all this, we can build the following XPath
//span[#class='label_show'][normalize-space(.)='10']
If it were me, I would throw this in a function that pass in the desired size as a string and insert that parameter into the above XPath to make this much more reusable.
Try the below locator(xpath):
.//span[contains(#id,'label_eu') and text()=' 40']
Ok so here is what I ended up using which solved the issue. I made a list of all the web elements with the class 'checkbox-size' then traversed that list to to find the correct size, and then click on that element.
Here is the code:
sizes = driver.find_elements_by_class_name('checkbox-size')
for size in sizes:
if size.text in [usersize]:
size.click()
print colored('Carted size %s'%(size.text), 'green')
break
continue

Python xpath - Exclude content with style="display:none"

I'm using xpath to get some information from a website and I came across a block of code that contains style="display:none or block and I want to only include the code that has display:block; I watched some examples but I couldn't get it working on my code. I want to use an if statement to run the code if it has display:block but I don't know if that is possible. This is what I have:
if guide_page.xpath(".//div[#class='build-box']/#style/text()") == "display: block;":
for build_names in guide_page.xpath(".//div[#class='build-gradient']"):
for title in build_names.xpath("div/h2/text()"):
print("\n")
print(title)
And this is the div that has it:
<div class="build-box" style="display: block;">
I'm not sure if I should paste more of the html or if that's enough, otherwise, please tell me and thanks for any help :)
You can do this without using if statement. Just add a not(...condition...) in predicate to exclude elements matching certain condition. For example, the following XPath returns div elements with certain class attribute value, that don't have attribute style="display: block;" :
.//div[#class='build-box' and not(#style='display: block;')]

What is the name of the page element that reads "+ Add telephone" on the Google Hangouts start page?

I'm looking for the name of the page element that reads "+ Add telephone" on Google Hangouts start page, for the purpose of plugging it into Selenium Python API's webdriver.Firefox().find_element_by_name() command.
Alternatively, the element ID would also work, as I know there is also a find_element_by_id() command.
That element does not have any name. Go for xpath contains function and use find_element_by_xpath
//div[contains(text(),'Add telephone')]
However, it's always good idea to not to use Google for any kind of testing. That's kind of trap. Specially for Selenium
Inspecting the element (using Google Chrome), it looks like this:
<div class="c-N-K c-za-b ZG Kn-Mo" role="button" tabindex="0" style="-webkit-user-select: none;">+ Add telephone</div>
There's no name or id on that element. But it is wrapped within a span that has a unique id!
<span id=":sb.og">
<div class="c-N-K c-za-b ZG Kn-Mo" role="button" tabindex="0" style="-webkit-user-select: none;">+ Add telephone</div>
</span>
Alternatively, as #Saifur points out, you can also find by text.

Categories