We Keep Coding

Element finding with repeated tags in python selenium

This is the html I have on a website:
<table class="table table-fixed table-header-right text-medium">
<tbody><tr><th class="no-border">Certification Number</th><td class="no-border">48487270</td></tr>
<tr>
<th>Label Type</th>
<td>
<img width="69" height="38" class="margin-right-min" alt="" aria-hidden="true" src="https://i.psacard.com/psacard/images/cert/table-image-ink.png" style="">
<span class="inline-block padding-top-min">with fugitive ink technology</span>
</td>
</tr>
<tr><th>Reverse Cert Number/Barcode</th><td>Yes</td></tr>
<tr><th>Year</th><td>2020</td></tr>
<tr><th>Brand</th><td>TOPPS</td></tr>
<tr><th>Sport</th><td>BASEBALL CARDS</td></tr>
<tr><th>Card Number</th><td>20</td></tr>
<tr><th>Player</th><td>ARISTIDES AQUINO</td></tr>
<tr><th>Variety/Pedigree</th><td></td></tr>
<tr><th>Grade</th><td>NM-MT 8</td></tr>
</tbody></table>
I am trying to figure out a way to get and set the year to a variable, the normal way I find elements is with XPath but since these tags are repeated so many times with no other indicators I am unsure how to go about this. The year will change so I cant search by text. Any help would be appreciated.

Use BeautifulSoup to find the <th> tag with the text 'Year'. Then find the next <td> tag and extract the text from that:
from bs4 import BeautifulSoup
html = '''<table class="table table-fixed table-header-right text-medium">
<tbody><tr><th class="no-border">Certification Number</th><td class="no-border">48487270</td></tr>
<tr>
<th>Label Type</th>
<td>
<img width="69" height="38" class="margin-right-min" alt="" aria-hidden="true" src="https://i.psacard.com/psacard/images/cert/table-image-ink.png" style="">
<span class="inline-block padding-top-min">with fugitive ink technology</span>
</td>
</tr>
<tr><th>Reverse Cert Number/Barcode</th><td>Yes</td></tr>
<tr><th>Year</th><td>2020</td></tr>
<tr><th>Brand</th><td>TOPPS</td></tr>
<tr><th>Sport</th><td>BASEBALL CARDS</td></tr>
<tr><th>Card Number</th><td>20</td></tr>
<tr><th>Player</th><td>ARISTIDES AQUINO</td></tr>
<tr><th>Variety/Pedigree</th><td></td></tr>
<tr><th>Grade</th><td>NM-MT 8</td></tr>
</tbody></table>'''
soup = BeautifulSoup(html, 'html.parser')
year = soup.find('th', text='Year').find_next('td').text
print(year)
Output:
'2020'

Firstly we need to find out webelements using driver.findelements function using that classname
And then we can get elements from that list
By list.get(index)
Or,
You can store all the td/th elements in a list and than search the list for year you are looking for.

How to extract HTML table following a specific heading?

I am using BeautifulSoup to parse HTML files. I have a HTML file similar to this:
<h3>Unimportant heading</h3>
<table class="foo">
<tr>
<td>Key A</td>
</tr>
<tr>
<td>A value I don't want</td>
</tr>
</table>
<h3>Unimportant heading</h3>
<table class="foo">
<tr>
<td>Key B</td>
</tr>
<tr>
<td>A value I don't want</td>
</tr>
</table>
<h3>THE GOOD STUFF</h3>
<table class="foo">
<tr>
<td>Key C</td>
</tr>
<tr>
<td>I WANT THIS STRING</td>
</tr>
</table>
<h3>Unimportant heading</h3>
<table class="foo">
<tr>
<td>Key A</td>
</tr>
<tr>
<td>A value I don't want</td>
</tr>
</table>
I want to extract the string "I WANT THIS STRING". The perfect solution would be to get the first table following the h3 heading called "THE GOOD STUFF". I have no idea how to do this with BeautifulSoup - I only know how to extract a table with a specific class, or a table nested within some particular tag, but not following a particular tag.
I think a fallback solution could make use of the string "Key C", assuming it's unique (it almost certainly is) and appears in only that one table, but I'd feel better with going for the specific h3 heading.

Following the logic of #Zroq's answer on another question, this code will give you the table following your defined header ("THE GOOD STUFF"). Please note I just put all your html in the variable called "html".
from bs4 import BeautifulSoup, NavigableString, Tag
soup=BeautifulSoup(html, "lxml")
for header in soup.find_all('h3', text=re.compile('THE GOOD STUFF')):
nextNode = header
while True:
nextNode = nextNode.nextSibling
if nextNode is None:
break
if isinstance(nextNode, Tag):
if nextNode.name == "h3":
break
print(nextNode)
Output:
<table class="foo">
<tr>
<td>Key C</td>
</tr>
<tr>
<td>I WANT THIS STRING</td>
</tr>
</table>
Cheers!

The docs explain that if you don't want to use find_all, you can do this:
for sibling in soup.a.next_siblings:
print(repr(sibling))

I am sure there are many ways to this more efficiently, but here is what I can think about right now:
from bs4 import BeautifulSoup
import os
os.chdir('/Users/Downloads/')
html_data = open("/Users/Downloads/train.html",'r').read()
soup = BeautifulSoup(html_data, 'html.parser')
all_td = soup.find_all("td")
flag = 'no_print'
for td in all_td:
if flag == 'print':
print(td.text)
break
if td.text == 'Key C':
flag = 'print'
Output:
I WANT THIS STRING

Python BeautifulSoup find next_sibling

I have some html scraping code issues with beautiful soup. I cannot figure out how to go through the whole html document to find the rest of the things I am looking for.
I have this code that will find and print the word "Totem" in the below html. I want to be able to cycle through the html and find the remaining "One, Two, Three", and "Rent"
Code that works to find the first tag and text:
print(html.find('td', {'class': 'play'}).next_sibling.next_sibling.text)
Let the below be the sample html to scrape:
<tr>
<td class="play">
<span class="play-button as_audio-button"></span>
<audio class="as_audio_preview" src="https://shopify.audiosalad.com/" >foo</audio>
</td>
**<td>Totem</td>**
<!--<td>$0.99</td>-->
<td class="buy">
<tr>
<td class="play">
<span class="play-button as_audio-button"></span>
<audio class="as_audio_preview" src="https://shopify.audiosalad.com/" >foo</audio>
</td>
**<td>One, Two, Three</td>**
<!--<td>$0.99</td>-->
<td class="buy">
<tr>
<td class="play">
<span class="play-button as_audio-button"></span>
<audio class="as_audio_preview" src="https://shopify.audiosalad.com/" >foo</audio>
</td>
**<td>Rent</td>**
<!--<td>$0.99</td>-->
<td class="buy">

Try this. It should fetch you the content you are after:
from bs4 import BeautifulSoup
soup = BeautifulSoup(content,"lxml")
for items in soup.find_all(class_="play"):
data = items.find_next_sibling().text
print(data)
Or, you can try like this as well:
for items in soup.find_all(class_="play"):
data = items.find_next("td").text
print(data)
Output:
Totem
One, Two, Three
Rent

you have to iterate over elements, like this:
for td in html.find_all('td', {'class': 'play'}):
print(td.next_sibling.next_sibling.text)

Can't scrape HTML table using BeautifulSoup

I'm trying to scrape data off a table on a web page using Python, BeautifulSoup, Requests, as well as Selenium to log into the site.
Here's the table I'm looking to get data for...
<div class="sastrupp-class">
<table>
<tbody>
<tr>
<td class="key">Thing I dont want 1</td>
<td class="value money">$1.23</td>
<td class="key">Thing I dont want 2</td>
<td class="value">99,999,999</td>
<td class="key">Target</td>
<td class="money value">$1.23</td>
<td class="key">Thing I dont want 3</td>
<td class="money value">$1.23</td>
<td class="key">Thing I dont want 4</td>
<td class="value percentage">1.23%</td>
<td class="key">Thing I dont want 5</td>
<td class="money value">$1.23</td>
</tr>
</tbody>
</table>
</div>
I can find the "sastrupp-class" fine, but I don't know how to look through it and get to the part of the table I want.
I figured I could just look for the class that I'm searching for like this...
output = soup.find('td', {'class':'key'})
print(output)
but that doesn't return anything.
Important to note:
< td>s inside the table have the same class name as the one that I want. If I can't separate them out, I'm ok with that although I'd rather just return the one I want.
2.There are other < div>s with class="sastrupp-class" on the site.
I'm obviously a beginner at this so let me know if I can help you help me.
Any help/pointers would be appreciated.

1) First of, to get your 'Target' you need find_all, not find. Then, considering you know exactly in which position your target will be (in the example you gave it is index=2) the solution could be reached like this:
from bs4 import BeautifulSoup
html = """(YOUR HTML)"""
soup = BeautifulSoup(html, 'html.parser')
table = soup.find('div', {'class': 'sastrupp-class'})
all_keys = table.find_all('td', {'class': 'key'})
my_key = all_keys[2]
print my_key.text # prints 'Target'
2)
There are other < div>s with class="sastrupp-class" on the site
Again, you need to select the one you need using find_all and then selecting the correct index.
Example HTML:
<body>
<div class="sastrupp-class"> Don't need this</div>
<div class="sastrupp-class"> Don't need this</div>
<div class="sastrupp-class"> Don't need this</div>
<div class="sastrupp-class"> Target</div>
</body>
To extract the target, you can just:
all_divs = soup.find_all('div', {'class':'sastrupp-class'})
target = all_divs[3] # assuming you know exactly which index to look for

python BeautifulSoup4 break for loop when tag found

I have a problem breaking a for loop when going trough a html with bs4.
I want to save a list separated with headings.
The HTML code can look something like below, however it contains more information between the desired tags:
<h2>List One</h2>
<td class="title">
<a title="Title One">This is Title One</a>
</td>
<td class="title">
<a title="Title Two">This is Title Two</a>
</td>
<h2>List Two</h2>
<td class="title">
<a title="Title Three">This is Title Three</a>
</td>
<td class="title">
<a title="Title Four">This is Title Four</a>
</td>
I would like to have the results printed like this:
List One
This is Title One
This is Title Two
List Two
This is Title Three
This is Title Four
I have come this far with my script:
import urllib2
from bs4 import BeautifulSoup
html = urllib2.urlopen('some webiste')
soup = BeautifulSoup(html, "lxml")
quote1 = soup.h2
print quote1.text
quote2 = quote1.find_next_sibling('h2')
print quote2.text
for quotes in soup.findAll('h2'):
if quotes.find(text=True) == quote2.text:
break
if quotes.find(text=True) == quote1.text:
for anchor in soup.findAll('td', {'class':'title'}):
print anchor.text
print quotes.text
I have tried to break the loop when "quote2" (List Two) is found. But the script gets all the td-content and ignoring the next h2-tags.
So how do I break the for loop with next h2-tag?

In my opinion the problem lies in your HTML syntax. According to https://validator.w3.org it's not legal to mix "td" and "h3" (or generally any header tag). Also, implementing list with tables is most likely not a good practice.
If you can manipulate your input files, the list you seem to need could be implemented with "ul" and "li" tags (first 'li' in 'ul' containing the header) or, if you need to use tables, just put your header inside of "td" tag, or even more cleanly with "th"s:
<table>
<tr>
<th>Your title</th>
</tr>
<tr>
<td>Your data</td>
</tr>
</table>
If the input is not under your control, your script could perform search and replace on the input text anyway, putting the headers into table cells or list items.