We Keep Coding

How to get all attributes of section of text from a html string in Python?

I have a html string:
Normal<span style="font-weight: bold;">Bold <span style="font-style: italic;">BoldAndItalic</span></span><span style="font-style: italic;">Italic</spa
Which renders out to
NormalBold BoldAndItalicItalic
What I want to get is a python dictionary, that lists all the attributes given, to all the pieces of text, kind of like this:
[
{"text":"Normal","styles":{"color":None,"font-style":None,"font-weight":None}},
{"text":"Bold","styles":{"color":None,"font-style":None,"font-weight":"bold"}},
{"text":" ","styles":{"color":None,"font-style":None,"font-weight":None}},
{"text":"BoldAndItalic","styles":{"color":None,"font-style":"italic","font-weight":"bold"}},
{"text":"Italic","styles":{"color":None,"font-style":"italic","font-weight":None}}
]
Where None could be considered default/not given.
However, when I parse the html string via the BeautifulSoup Library, I cannot find a way to access each section of text individually, I need to access the span tags first, and since there are span tags within other span tags, making a parser myself becomes very difficult, which I cannot seem to figure out.
What I attempted was this:
def stylesandtext(obj):
if obj.decomposed:
return
text = obj.text
styles={"color":None,"font-weight":None}
stylestr = obj.attrs['style'].split(": ")
styles[stylestr[0]] = stylestr[1].replace(";","")
if obj.find('span') !=None:
getsecstyle = stylesandtext(obj.find('span'))['styles']
if getsecstyle['color'] !=None:
styles['color'] = getsecstyle['color']
if getsecstyle['font-weight'] !=None:
styles['font-weight'] = getsecstyle['font-weight']
obj.find('span').decompose()
return {"text":text,"styles":styles}
Where, using BeautifulSoup, for every span tag, I tried to check for inner span tags, got the attributes, and combined it to the Dict storing all the stuff. It kind of worked, but it did not pick any words not in the span tags.
How would I go about getting the attributes to a text section?
Few more details:
There are no other tags than span
I only need to count for a few style attributes: color, font-weight, font-style, text-decoration

There is a attribute called children, for every tag, which separates all the elements inside it(like soup.children orsoup.span.children).
Then I can recursively, in a function, get all attributes and text, which I store in a list.
This is the code I figured out:
import bs4
def get_as_list(obj,extstyle=None):
alldata = []
style = {"color":None,"font-weight":None,"font-style":None,"text-decoration":None}
if extstyle != None:
style=extstyle
if 'style' in obj.attrs:
spanstyleaslist = obj.attrs['style'].split(": ")
#obj.attrs is like {'style': 'color: #55FF55'}
style[spanstyleaslist[0]] = spanstyleaslist[1]
stuffaslist = list(obj.children)
for x in stuffaslist:
if type(x) == bs4.element.NavigableString:
alldata.append({'text':str(x),'styles':style})
else:
alldata.extend(get_as_list(x,style))
return alldata
which I call in a external function, for every element in the soup.children, excluding NavigableStrings.

How to remove empty <p> tags with Beautiful Soup 4

I have a generated html and in some places it has <p><br/></p> or <p>\n\t</p>, <p><strong></strong><strong></strong></p> like empty tags. I would like to remove them.
for tag in soup("strong"):
if len(tag.get_text(strip=True)) == 0:
print(tag)
for tag in soup("p"):
if len(tag.get_text(strip=True)) == 0:
print(tag)
But this finds img tags wrapped inside p tags as it is not considered as inner text of the tag.
Any help appreciated.

Here is two ways to remove empty tags;
for x in soup.find_all():
if len(x.get_text(strip=True)) == 0:
x.extract()
The second way;
[x.decompose() for x in soup.findAll(lambda tag: not tag.contents and not tag.name == 'br' )]

beautifulsoup how to recombine words

Some of the words outputted are split when running this code. Like the word "tolerances" is split into "tole rances". I looked at the html source and it seems that's how the page was created.
There are also many other places where the word is split. How do I recombine them before writing to text?
import requests, codecs
from bs4 import BeautifulSoup
from bs4.element import Comment
path='C:\\Users\\jason\\Google Drive\\python\\'
def tag_visible(element):
if element.parent.name in ['sup']:
return False
if isinstance(element, Comment):
return False
return True
ticker = 'TSLA'
quarter = '18Q2'
mark1= 'ITEM 1A'
mark2= 'UNREGISTERED SALES'
url_new='https://www.sec.gov/Archives/edgar/data/1318605/000156459018019254/tsla-10q_20180630.htm'
def get_text(url,mark1,mark2):
html = requests.get(url)
soup = BeautifulSoup(html.text, 'html.parser')
for hr in soup.select('hr'):
hr.find_previous('p').extract()
texts = soup.findAll(text=True)
visible_texts = filter(tag_visible, texts)
text=u" ".join(t.strip() for t in visible_texts)
return text[text.find(mark1): text.find(mark2)]
text = get_text(url_new,mark1,mark2)
file=codecs.open(path + "test.txt", 'w', encoding='utf8')
file.write (text)
file.close()

You are dealing with HTML formatted with Microsoft Word. Don't extract text and try to process it without that context.
The section you want to process is clearly delineated with <a name="..."> tags, lets start with selecting all elements with the <a name="ITEM_1A_RISK_FACTORS"> marker, all the way up to but not including the <a name="ITEM2_UNREGISTERED_SALES"> marker:
def sec_section(soup, item_name):
"""iterate over SEC document paragraphs for the section named item_name
Item name must be a link target, starting with ITEM
"""
# ask BS4 to find the section
elem = soup.select_one('a[name={}]'.format(item_name))
# scan up to the parent text element
# html.parser does not support <text> but lxml does
while elem.parent is not soup and elem.parent.name != 'text':
elem = elem.parent
yield elem
# now we can yield all next siblings until we find one that
# is also contains a[name^=ITEM] element:
for elem in elem.next_siblings:
if not isinstance(elem, str) and elem.select_one('a[name^=ITEM]'):
return
yield elem
This function gives us all child nodes from the <text> node in the HTML document that start at a paragraph containing a specific link target, all the way through to the next link target that names an ITEM.
Next, the usual Word cleanup task is to remove <font> tags, style attributes:
def clean_word(elem):
if isinstance(elem, str):
return elem
# remove last-rendered break markers, non-rendering but messy
for lastbrk in elem.select('a[name^=_AEIOULastRenderedPageBreakAEIOU]'):
lastbrk.decompose()
# remove font tags and styling from the document, leaving only the contents
if 'style' in elem.attrs:
del elem.attrs['style']
for e in elem: # recursively do the same for all child nodes
clean_word(e)
if elem.name == 'font':
elem = elem.unwrap()
return elem
The Tag.unwrap() method is what'll most help your case, as the text is divided up almost arbitrarily by <font> tags.
Now it's suddenly trivial to extract the text cleanly:
for elem in sec_section(soup, 'ITEM_1A_RISK_FACTORS'):
clean_word(elem)
if not isinstance(elem, str):
elem = elem.get_text(strip=True)
print(elem)
This outputs, among the rest of the text:
•that the equipment and processes which we have selected for Model 3 production will be able to accurately manufacture high volumes of Model 3 vehicles within specified design tolerances and with high quality;
The text is now properly joined up, no re-combining required any more.
The whole section is still in a table but clean_word() cleaned this up now to the much more reasonable:
<div align="left">
<table border="0" cellpadding="0" cellspacing="0">
<tr>
<td valign="top">
<p> </p></td>
<td valign="top">
<p>•</p></td>
<td valign="top">
<p>that the equipment and processes which we have selected for Model 3 production will be able to accurately manufacture high volumes of Model 3 vehicles within specified design tolerances and with high quality;</p></td></tr></table></div>
so you can use smarter text extraction techniques to further ensure a clean text conversion here; you could convert such bullet tables to a * prefix, for example:
def convert_word_bullets(soup, text_bullet="*"):
for table in soup.select('div[align=left] > table'):
div = table.parent
bullet = div.find(string='\u2022')
if bullet is None:
# not a bullet table, skip
continue
text_cell = bullet.find_next('td')
div.clear()
div.append(text_bullet + ' ')
for i, elem in enumerate(text_cell.contents[:]):
if i == 0 and elem == '\n':
continue # no need to include the first linebreak
div.append(elem.extract())
In addition, you probably want to skip the page breaks too (a combination of <p>[page number]</p> and <hr/> elements), if you run
for pagebrk in soup.select('p ~ hr[style^=page-break-after]'):
pagebrk.find_previous_sibling('p').decompose()
pagebrk.decompose()
This is more explicit than your own version, where you remove all <hr/> elements and preceding <p> element regardless of whether they are actually siblings.
Execute both before cleaning up your Word HTML. Combined with your function that together becomes:
def get_text(url, item_name):
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
for pagebrk in soup.select('p ~ hr[style^=page-break-after]'):
pagebrk.find_previous_sibling('p').decompose()
pagebrk.decompose()
convert_word_bullets(soup)
cleaned_section = map(clean_word, sec_section(soup, item_name))
return ''.join([
elem.get_text(strip=True) if elem.name else elem
for elem in cleaned_section])
text = get_text(url, 'ITEM_1A_RISK_FACTORS')
with open(os.path.join(path, 'test.txt'), 'w', encoding='utf8') as f:
f.write(text)

This page markup is really bad. You will need to remove excess tags to fix your issue. Luckily for you, beautifulsoup can do the heavy-lifting. The code below will remove all font tags.
soup = BeautifulSoup(html.text, 'html.parser')
for font in soup.find_all('font'):
font.unwrap()

BeautifulSoup Remove all html tags except for those in whitelist such as "img" and "a" tags with python

Given some html code, how can I remove all the tags, keep the text and img and a tags?
For example, I have
<div><script bla bla></script><p>Hello all <img rscr="xx"></img></p></div>
I want to keep
Hello to <img rscr="xx"></img>
Is there something implemented in BeautifulSoup or Python?
Thanks

You could select all of the descendant nodes by accessing the .descendants property.
From there, you could iterate over all of the descendants and filter them based on the name property. If the node doesn't have a name property, then it is likely a text node, which you want to keep. If the name property is a or img, then you keep it as well.
# This should be the wrapper that you are targeting
container = soup.find('div')
keep = []
for node in container.descendants:
if not node.name or node.name == 'a' or node.name == 'img':
keep.append(node)
Here is an alternative where all the filtered elements are used to create the list directly:
# This should be the wrapper that you are targeting
container = soup.find('div')
keep = [node for node in container.descendants
if not node.name or node.name == 'a' or node.name == 'img']
Also, if you don't want strings that are empty to be returned, you can trim the whitespace and check for that as well:
keep = [node for node in container.descendants
if (not node.name and len(node.strip())) or
(node.name == 'a' or node.name == 'img')]
Based on the HTML that you provided, the following would be returned:
> ['Hello all ', , <img rscr="xx"/>]

import bs4
html = '''<div><script bla bla></script><p>Hello all <img rscr="xx"></img></p></div>'''
soup = bs4.BeautifulSoup(html, 'lxml')
soup.div.text, soup.div.find_next('a'), soup.div.find_next('img')
out:
('Hello all ', , <img rscr="xx"/>)
When the next element is tag's descendent, there is a shortcut:
soup.div.text, soup.div.a, soup.div.img
out:
('Hello all ', , <img rscr="xx"/>)
when you use bs4's parser, 'img' tag will be self-closed tag
You can aways use find_next to get next element in the DOM

Combine multiple tags with lxml

I have an html file which looks like:
...
<p>
<strong>This is </strong>
<strong>a lin</strong>
<strong>e which I want to </strong>
<strong>join.</strong>
</p>
<p>
2.
<strong>But do not </strong>
<strong>touch this</strong>
<em>Maybe some other tags as well.</em>
bla bla blah...
</p>
...
What I need is, if all the tags in a 'p' block are 'strong', then combine them into one line, i.e.
<p>
<strong>This is a line which I want to join.</strong>
</p>
Without touching the other block since it contains something else.
Any suggestions? I am using lxml.
UPDATE:
So far I tried:
for p in self.tree.xpath('//body/p'):
if p.tail is None: #no text before first element
children = p.getchildren()
for child in children:
if len(children)==1 or child.tag!='strong' or child.tail is not None:
break
else:
etree.strip_tags(p,'strong')
With these code I was able to strip off the strong tag in the part desired, giving:
<p>
This is a line which I want to join.
</p>
So now I just need a way to put the tag back in...

I was able to do this with bs4 (BeautifulSoup):
from bs4 import BeautifulSoup as bs
html = """<p>
<strong>This is </strong>
<strong>a lin</strong>
<strong>e which I want to </strong>
<strong>join.</strong>
</p>
<p>
<strong>But do not </strong>
<strong>touch this</strong>
</p>"""
soup = bs(html)
s = ''
# note that I use the 0th <p> block ...[0],
# so make the appropriate change in your code
for t in soup.find_all('p')[0].text:
s = s+t.strip('\n')
s = '<p><strong>'+s+'</strong></p>'
print s # prints: <p><strong>This is a line which I want to join.</strong></p>
Then use replace_with():
p_tag = soup.p
p_tag.replace_with(bs(s, 'html.parser'))
print soup
prints:
<html><body><p><strong>This is a line which I want to join.</strong></p>
<p>
<strong>But do not </strong>
<strong>touch this</strong>
</p></body></html>

I have managed to solve my own problem.
for p in self.tree.xpath('//body/p'):
if p.tail is None: # some conditions specifically for my doc
children = p.getchildren()
if len(children)>1:
for child in children:
#if other stuffs present, break
if child.tag!='strong' or child.tail is not None:
break
else:
# If not break, we find a p block to fix
# Get rid of stuffs inside p, and put a SubElement in
etree.strip_tags(p,'strong')
tmp_text = p.text_content()
p.clear()
subtext = etree.SubElement(p, "strong")
subtext.text = tmp_text
Special thanks to #Scott who helps me come down to this solution. Although I cannot mark his answer correct, I have no less appreciation to his guidance.

Alternatively, you can use more specific xpath to get the targeted p elements directly :
p_target = """
//p[strong]
[not(*[not(self::strong)])]
[not(text()[normalize-space()])]
"""
for p in self.tree.xpath(p_target):
#logic inside the loop can also be the same as your `else` block
content = p.xpath("normalize-space()")
p.clear()
strong = etree.SubElement(p, "strong")
strong.text = content
brief explanation about xpath being used :
//p[strong] : find p element, anywhere in the XML/HTML document, having child element strong...
[not(*[not(self::strong)])] : ..and not having child element other than strong...
[not(text()[normalize-space()])] : ..and not having non-empty text node child.
normalize-space() : get all text nodes from current context element, concatenated with consecutive whitespaces normalized to single space