I am trying to scrape dynamic content from a webpage. The page shows the 10 first elements and then I have to click a 'next' button to access the next 10 and so on. The button clicking works fine. However, afterwards, it returns only empty lists after the button is clicked.
I would like to get the content of the following element:
//*[#id="bt-collapse-442112-slider"]/div[1]/div/div/div[1]/table/tbody/tr[1]/td[3]/div/h3
I noticed that the first div has the attribute "data-currentslidecount" which is "10" in the beginning but is added 10 each time the button is clicked.
My initial code looked like this:
> for a in range(100):
> WebDriverWait(driver, mywaits).until(EC.presence_of_element_located((By.XPATH,
> '//*[#id="bt-collapse-442112-slider"]/div[1]')))
> print(driver.find_element_by_xpath('//*[#id="bt-collapse-442112-slider"]/div[1]/div/div/div[1]/table/tbody/tr[1]/td[3]/div/h3').text)
> WebDriverWait(driver, mywaits).until(EC.element_to_be_clickable((By.XPATH,
> '//*[#id="bt-collapse-442112-slider"]/div[1]/button[2]')))
> button = driver.find_element_by_xpath('//*[#id="bt-collapse-442112-slider"]/div[1]/button[2]')
> button.click()
> button_count += 1
> print(button_count) driver.close()
Trying to find a solution I found out that often, this problem occurs when content is embedded in iframes, and that one needs to switch frames to get dynamic content. However, in, my case, there seems to be no iframe.
Also, I tried using find_element_by_css_selector instead of by_xpath because I thought I could access the div like this or maybe even loop over the slide count. However, `
find_element_by_css_selector(div[data-currentslidecount="10")
produced a syntax error. I didn't find a solution online for this specific css selector and also, I am not very hopeful that this will be the solution to my problem.
Edit:
Here is the HTML:
<div class="bt-slider row slick-initialized slick-slider" data-allitemcount="4184" data-currentslidecount="10"><button type="button" class="slick-prev slick-arrow slick-disabled" aria-disabled="true" style="display: block;"><span class="bab-slick-arrow-text">Zurück</span></button>
<div class="slick-list draggable">
<div class="slick-track" style="opacity: 1; width: 2264px; left: 0px;">
<div class="bt-slide col-xs-12 bt-standard-content slick-slide slick-current slick-active" data-slick-index="0" aria-hidden="false" style="width: 1132px;" tabindex="-1">
<table class="table bt-table-data">
<thead>
<tr>
<th>
<p>Veröffentlichung</p>
</th>
<th>
<p>Sitzung</p>
</th>
<th>
<p>Beschreibung</p>
</th>
</tr>
</thead>
<tbody>
<tr>
<td data-th="04.02.2019">
<p>
04.02.2019<br>Montag</p>
</td>
<td data-th="Sitzung">
<p>19/78</p>
</td>
<td data-th="Beschreibung">
<div class="bt-documents-description">
<h3>
78. Sitzung, 19. Wahlperiode, 01.02.2019</h3>
Related
Anyone know how to get the text color for both "office" and "room"?
office_color = "#FD7C04"
room_color = "green"
HTML code:
<tr style="background-color:#999" id="tagging">
<td><b style="white-space: nowrap;"> tag_table</b></td>
<td style="text-align:center;">
<span class="big" style="font-size:24px; font-weight:700; color: #FD7C04">office</span></td>
<td style="text-align:center;">
<span class="big" style="font-size:24px; font-weight:700; color: green" id="rigan">room</span></td>
</tr>
First, get the element you want to extract your atribute from, in this case the span. You choose your prefered method for that. One of them is:
selector="body > span:nth-child(2)"
element1=driver.find_element_by_css_selector(selector)
element2=driver.find_element_by_id("#rigan")
One way to get this is opening the html on your browser, right click on the element and "copy selector" to get a css selector.
Then get the atribute from the elements:
color1=element1.get_attribute("color")
color2=element2.get_attribute("color")
I need to select the radio button below with value="QMBT-0029104.xlsx;SYPFJPC2MQHC-5-688478#QHBTW"
HTML:
<form name="ViewQueryForm" method="post" action="/equery/getAttachments.do">
<div class="txt_align_left innerdvcont" id="tabmenu1" style="display:">
<div class="clear"></div>
<div class="txt_align_left innerdvcont" id="tabmenu011" name="tabmenu011">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<thead>
<tr>
<th width="10%" style="text-align: left"></th>
<th width="60%" style="text-align: left">Attachment </th>
<th width="30%" style="text-align: left">Date </th>
</tr>
</thead>
<tbody>
<tr>
<td align="center" valign="top">
<input type="radio" name="getAttachmentValue" id="getAttachmentValue" value="Dispute_1466718.xlsx;SYPFJPC2MQHC-5-687433#QHBTW"> </td>
<td style="padding:5px 4px">Dispute_1466718.xlsx</td>
<td style="padding:5px 4px">2021-02-16T10:34:08.617</td>
</tr>
</tbody><tbody>
<tr>
<td align="center" valign="top">
<input type="radio" name="getAttachmentValue" id="getAttachmentValue" value="QMBT-0029104.xlsx;SYPFJPC2MQHC-5-688478#QHBTW"> </td>
<td style="padding:5px 4px">QMBT-0029104.xlsx</td>
<td style="padding:5px 4px">2021-03-27T08:08:46.09</td>
</tr>
</tbody>
</table>
</div>
So far I have been able to click on it using the code below:
radiobutton2 = driver.find_element_by_xpath("//input[#value='QMBT-0029104.xlsx;SYPFJPC2MQHC-5-688478#QHBTW']");
radiobutton2.click()
However, the value changes every time which means that it's not something that I can use when running the code. Is there any way to select the second radio button by default for example.
Alternatively, I will know the QMBT-00000 reference, so is there a way to select the radio button by searching for that text?
I have tried:
radiobutton2 = driver.find_element_by_xpath('//*[contains(text(), "QMBT-0029104") and #id="getAttachmentValue"]');
radiobutton2.click()
However, that gives me an error:
Unable to locate element
Great first question.
If the radio button you need to select will always be the second option, you can select it by the index (below is in C#, but should be similar for Python):
// get all elements where id = "getAttachmentValue"
var radioButtons = driver.FindElements(By.Id("getAttachmentValue"));
// click second element
radioButtons[1].Click();
Edit (Python):
from selenium.webdriver.common.by import By
radioButtons = driver.find_elements(By.ID, 'getAttachmentValue')
radioButtons[1].click()
I added your HTML to a simple HTML file and was able to select the second radio button using the above example (using C#):
You can select the second radio button by xpath using array index notation. Instead of targeting the value attribute, use the name attribute instead. The value of the name attribute seems to be consistent:
xpath = "(//input[#type = 'radio' and #name = 'getAttachmentValue'])[2]"
radio_button = driver.find_element_by_xpath(xpath)
radio_button.click()
You probably should not use the id attribute in your locator. The Id attribute value must be unique for the entire web page, but the Id attribute value is repeated for each radio button. This is invalid HTML. It is best to avoid using attributes in locators where the values of those attributes violate the HTML spec. You never fully know how the browser is going to treat it.
For your use case, the name attribute would work best. It is perfectly valid HTML to have repeated name attribute values, and the value of the name attribute appears to be stable for each page view.
If you have only two same ids, the answer is simple. Try this.
driver.find_element_by_css_selector("#getAttachmentValue:nth-of-type(2)")
If there are more - the solution may be more complicated.
How may IDs are there?
So I want to take the text of td class.
The html page
<table class="table table-striped">
<tbody>
<tr>
<td class="text-center">
<img .....>
</td>
<td>text</td>
<td>text</td>
<td class="text-center">
<a ....></a>
</td>
<td class="text-center">
TEXT I WANT TO TAKE HERE
</td>
<td class="text-center">
<a ....><i class="fa fa-times"></i></a>
</td>
</tr>
</tbody>
</table>
The text I want to take is "TEXT I WANT TO TAKE HERE".
I tried using the xpath like below but it didnt work
table = browser.find_element_by_xpath(("//div[#class='table table-striped']/tbody/tr/td[5]"));
I got an error saying:
no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='table table-striped']/tbody/tr/td[5]"}
Is it because I have multiple classes in the selector and I have to use dot?
(I tried: 'table.table-striped' but it still didnt work)
Your xpath is incorrect. You have a table tag but, you are looking for a div tag. So, you just need to replace div with table.
table = browser.find_element_by_xpath(("//table[#class='table table-striped']/tbody/tr/td[5]"));
Use below xpath to get the text
browser.find_element_by_xpath("//td[#class='text-center']").text
And use the index as well to better find your row e.g.
browser.find_element_by_xpath("//td[#class='text-center'][3]").text
Use Below xpath to get the text TEXT I WANT TO TAKE HERE
//table//tr/td[contains(text(), 'TEXT I WANT TO TAKE HERE')]
Updated Answer: You can refer any of these below mentioned xpath to get your webelement.
//td[5]
OR
//table[#class='table table-striped']//td[5]
OR
//table[#class='table table-striped']/..//following-sibling::td[5]
OR
//td[#class='text-center'][3]
In your XPath expression you are looking for a div tag, but your HTML does not have that. Perhaps you are looking to the table tag:
table = browser.find_element_by_xpath(("//table[#class='table table-striped']/tbody/tr/td[5]"));
I have the following structure, where I am trying to click the second trash icon which is a button next to Test1.
<tr class=“ng-scope”>
<td class=“ng-scope”></td>
<td class=“ng-scope”></td>
<td class=“ng-scope”>
<span class=“ng-binding”>Test0</span>
</td>
<td class=“ng-scope”>
<button class=“bin btn-xs”>
<i class="glyphicon glyphicon-trash"></i>
</button>
</td>
</tr>
<tr class=“ng-scope”>
<td class=“ng-scope”></td>
<td class=“ng-scope”></td>
<td class=“ng-scope”>
<span class=“ng-binding”>Test1</span>
</td>
<td class=“ng-scope”>
<button class=“bin btn-xs”>
<i class="glyphicon glyphicon-trash"></i>
</button>
</td>
</tr>
Currently how I am implementing is by doing find_element_by_xpath where xpath is //i#class="glyphicon glyphicon-trash" and do an index searching with the given results.
This however I find quit inefficient, especially the given results can be theoretically many and I have to loop through the result list.
I tried also the following lines:
myxpath = "//*[contains(text(), 'Test1')]/following-sibling::tr/button[#class='glyphicon glyphicon-trash']"
driver.find_by_xpath(myxpath)
which does not work (because the trash icon is not actually the sibling of Test1.
How can I implement this in a better way (i.e. I want to use Test1 as anchor and click the trash button next to it and not next to Test0)?
To select the button in the row having the text:
//tr[.//text()='Test1']//button
To select the button in cell 4 in the row having the text in cell 3:
//tr[td[3]//text()='Test1']/td[4]//button
To select the cell having the text and then the button in the following cell:
//td[.//text()='Test1']/following-sibling::td[1]//button"
I'm not very clear whether you want the button or the icon...
here is for the i tag
try
//i[#class='glyphicon glyphicon-trash' and ../../../td/span/text() = "Test1"]
ps note also that:
<span class="ng-binding">Test1</span>
and
<span class="ng-binding"> Test1 </span>
are different.
I am trying to get an link from another frame on our webpage. After using
Select Frame css=frame[name'submenu']
Click Link css=#navigation_user > tr:nth-child(2) > td:nth-child(1) > a:nth-child(1) > span:nth-child(1)
I would have directly used the link=users BUT there is another link with the same link and it is a parent element of it so I can't use it.
Any ideas on how can I access this link?
Excerpt of the html:
<frame src="sample.asp" name="submenu">
<tbody>
<tr>
<td class="navigation">
<a class="navigationheadline">
<span id="user" class="navigationheadline">Users</span>
</a>
</td>
</tr>
</tbody>
<tbody id="navigation_user">
<tr>
<td class="navigation">
<a href="UserSearch.asp?null=" target="main">
<span class="navigation" onclick="hideNv()">Users</span>
</a>
</td>
</tr>
</tbody>
</frame>
Thank you in advance!
have you tried a different locator method? like xpath or jquery? I think the route of your problem is the locator you're using. How did you determine the one you're using is correct? Just plucked out of dev tools?
xpath=//*[#id="navigation_user"]/tr/td/a/span