We Keep Coding

python beautifulsoup: how to find all before certain stop tag?

I need to find all tags of a certain kind (class "nice") but excluding those after a certain other tag (class "stop").
<div class="nice"></div>
<div class="nice"></div>
<div class="stop">here should be the end of found items</div>
<div class="nice"></div>
<div class="nice"></div>
How do I accomplish this using bs4?
I found this as a similar question but it appears a bit fuzzy.

You can use for example .find_previous to filter out unwanted tags:
from bs4 import BeautifulSoup
html_doc = """\
<div class="nice">want 1</div>
<div class="nice">want 2</div>
<div class="stop">here should be the end of found items</div>
<div class="nice">do not want 1</div>
<div class="nice">do not want 2</div>"""
soup = BeautifulSoup(html_doc, "html.parser")
for div in soup.find_all("div", class_="nice"):
if div.find_previous("div", class_="stop"):
break
print(div)
Prints:
<div class="nice">want 1</div>
<div class="nice">want 2</div>

how to get specific links with BeautifulSoup?

I am trying to crawl HTML source with Python using BeautifulSoup.
I need to get the href of specific link <a> tags.
This is my test code. I want to get links <a href="/example/test/link/activity1~10"target="_blank">
<div class="listArea">
<div class="activity_sticky" id="activity">
.
.
</div>
<div class="activity_content activity_loaded">
<div class="activity-list-item activity_item__1fhpg">
<div class="activity-list-item_activity__3FmEX">
<div>...</div>
<a href="/example/test/link/activity1" target="_blank">
<div class="activity-list-item_addr">
<span> 0x1292311</span>
</div>
</a>
</div>
</div>
<div class="activity-list-item activity_item__1fhpg">
<div class="activity-list-item_activity__3FmEX">
<div>...</div>
<a href="/example/test/link/activity2" target="_blank">
<div class="activity-list-item_addr">
<span> 0x1292312</span>
</div>
</a>
</div>
</div>
.
.
.
</div>
</div>

Check the main page of the bs4 documentation:
for link in soup.find_all('a'):
print(link.get('href'))

This is a code for the problem. You should find the all <a></a>, then to getting the value of href.
soup = BeautifulSoup(html, 'html.parser')
for i in soup.find_all('a'):
if i['target'] == "_blank":
print(i['href'])
Hope my answer could help you.

Select the <a> specific - lternative to #Mason Ma answer you can also use css selectors:
soup.select('.activity_content a')]
or by its attribute target -
soup.select('.activity_content a[target="_blank"]')
Example
Will give you a list of links, matching your condition:
import requests
from bs4 import BeautifulSoup
html = '''
<div class="activity_content activity_loaded">
<div class="activity-list-item activity_item__1fhpg">
<div class="activity-list-item_activity__3FmEX">
<div>...</div>
<a href="/example/test/link/activity1" target="_blank">
<div class="activity-list-item_addr">
<span> 0x1292311</span>
</div>
</a>
</div>
</div>
<div class="activity-list-item activity_item__1fhpg">
<div class="activity-list-item_activity__3FmEX">
<div>...</div>
<a href="/example/test/link/activity2" target="_blank">
<div class="activity-list-item_addr">
<span> 0x1292312</span>
</div>
</a>
</div>
</div>
'''
soup = BeautifulSoup(html)
[x['href'] for x in soup.select('.activity_content a[target="_blank"]')]
Output
['/example/test/link/activity1', '/example/test/link/activity2']

Based on my understanding of your question, you're trying to extract the links (href) from anchor tags where the target value is _blank. You can do this by searching for all anchor tags then narrowing down to those whose target == '_blank'
links = soup.findAll('a', attrs = {'target' : '_blank'})
for link in links:
print(link.get('href'))

How to extract the data from encoded HTML class using python

How can I retrieve the page encoded div class of a webpage (title html tag) using Python?
Here my sample html code.

You need to use requests to make a request (it will automatically decode the page, in most cases), and beautifulsoup to extract the data from the HTML.
Update after OP clarifications. CSS classes are not dynamically updating, they're the same (that's what I noticed). Since they're the same, you can:
grab a container with all needed data (a container (CSS selector) that wraps needed data)
for result in soup.select(".pSzOP-AhqUyc-qWD73c.GNzUNc span"):
# ...
use regex to filter (find) all needed data via re.findall() and capture group (.*): only this match will be captured and returned. .*: means to capture everything.
if re.findall(r"^Telephone\s?:\s?(.*)", result.text):
# ...
Have a look at the SelectorGadget Chrome extension to grab CSS selectors by clicking on the desired element in your browser. On that note, there's a dedicated web scraping with CSS selectors blog post of mine.
Code and example in the online IDE:
import requests, re
from bs4 import BeautifulSoup
html = requests.get("https://sites.google.com/a/arden.solihull.sch.uk/futures/home")
soup = BeautifulSoup(html.text, "html.parser")
# all regular expressions for this task
# https://regex101.com/r/cxdxgq/1
for result in soup.select(".pSzOP-AhqUyc-qWD73c.GNzUNc span"):
if re.findall(r"^Careers\s?.*\s?:\s?(.*)", result.text):
name = "".join(re.findall(r"^Careers\s?.*\s?:\s?(.*)", result.text.strip()))
print(name)
if re.findall(r"^Telephone\s?:\s?(.*)", result.text):
telephone = "".join(re.findall(r"^Telephone\s?:\s?(.*)", result.text.strip()))
print(telephone)
if re.findall(r"^Email\s?:\s?(.*)", result.text):
email = "".join(re.findall(r"^Email\s?:\s?(.*)", result.text.strip()))
print(email)
# to scrape the role you can do the same thing with regex. Test on regex101.com
'''
Mrs A. Fallis
01564 773348
afallis#arden.solihull.sch.uk
Mr S. Brady
01564 7733478
sbrady#arden.solihull.sch.uk
'''
First solutions without OP clarifications (shows only extraction part since you haven't provided a website URL):
from bs4 import BeautifulSoup
html = """
<div class="L581yb VICjCf" hjdwnd-ahquyc-r6poud="" jndksc="" l6ctce-pszop"="" l6ctce-purzt="" tabindex=" == $0
<div class=">
</div>
<div class="hJDwNd-AhqUyc-WNfPc purZT-AhqUyC-I15mzb PSzOP-AhqUyc-qWD73c JNdks <div class=" jndksc-smkayb"="">
<div class="" f570id"="" jsaction="zXBUYD: ZTPCnb; 2QF9Uc: Qxe3nd;
jsname=" jscontroller="SGWD4d">
>
<div class="oKdM2C KzvoMe">
<div class="hJDwNd-AhqUyc-WNFPC PSzOP-AhqUyc- qWD73c jXK9ad D2fZ2 Oj CsFc whaque GNzUNC" id="h.7f5e93de0cf8a767_49">
<div class="]XK9ad-SmkAyb">
<div class="ty]Ctd mGzaTb baZpAe">
<div class="GV3q8e aP9Z7e" id="h.p_9livxd801krd">
</div>
<h3 class="CDt4ke zfr3Q OmQG5e" dir="ltr" id="h.p_9livxd801krd" tabindex="-1">
.
</h3>
<div class="GV3q8e aP9z7e" id="h.p JrEgQYpyORCF">
</div>
<h3 class="CDt 4Ke zfr3Q OmQG5e" dir="ltr" id="h.p_JrEgQYPYORCF" tabindex="-1">
<div class="CjVfdc" jsaction="touchstart:UrsOsc; click:Kjs
qPd; focusout:QZoaz; mouseover:yOpDld; mouseout:dq0hvd;fvlRjc:jbFSO
d;CrflRd:SzACGe;" jscontroller="Ae65rd">
<div class="PPHIP rviiZ" jsname="haAclf">
.
</div>
<span style="font-family: 'Oswald'; font-weight: 500;">
Telephone : 01564 773348
</span>
</div>
</h3>
<div class="GV3q8e aP9z7e" id="h.p_sylefz-BOSBX">
</div>
><h3 id="h.p_sylefz-BOSBX" dir="ltr" class="CDt 4Ke zfr3Q OmQG5e"
</div>
</div>
</div>
</div>
</div>
</div>
"""
# pass HTML to BeautifulSoup object and assign a html.parser as a HTML parser
soup = BeautifulSoup(html, "html.parser")
# grab a phone number (only first occurrence will be extracted)
# https://www.crummy.com/software/BeautifulSoup/bs4/doc/#css-selectors
print(soup.select_one('.CjVfdc span').text.strip())
# Telephone : 01564 773348
# extract <div> element with .L581yb class. returns a list()
print(soup.select('.L581yb'))
'''
[<div class="L581yb VICjCf" hjdwnd-ahquyc-r6poud="" jndksc="" l6ctce-pszop"="" l6ctce-purzt="" tabindex=" == $0
<div class=">
</div>]
'''
# extract <div> element with .hJDwNd-AhqUyc-WNfPc class. returns a list()
print(soup.select('.hJDwNd-AhqUyc-WNfPc'))
'''
[<div class="hJDwNd-AhqUyc-WNfPc purZT-AhqUyC-I15mzb PSzOP-AhqUyc-qWD73c JNdks <div class=" jndksc-smkayb"="">
<div class="" f570id"="" jsaction="zXBUYD: ZTPCnb; 2QF9Uc: Qxe3nd;
jsname=" jscontroller="SGWD4d">
>
<div class="oKdM2C KzvoMe">
<div class="hJDwNd-AhqUyc-WNFPC PSzOP-AhqUyc- qWD73c jXK9ad D2fZ2 Oj CsFc whaque GNzUNC" id="h.7f5e93de0cf8a767_49">
<div class="]XK9ad-SmkAyb">
<div class="ty]Ctd mGzaTb baZpAe">
<div class="GV3q8e aP9Z7e" id="h.p_9livxd801krd">
</div>
<h3 class="CDt4ke zfr3Q OmQG5e" dir="ltr" id="h.p_9livxd801krd" tabindex="-1">
.
</h3>
<div class="GV3q8e aP9z7e" id="h.p JrEgQYpyORCF">
</div>
<h3 class="CDt 4Ke zfr3Q OmQG5e" dir="ltr" id="h.p_JrEgQYPYORCF" tabindex="-1">
<div class="CjVfdc" jsaction="touchstart:UrsOsc; click:Kjs
qPd; focusout:QZoaz; mouseover:yOpDld; mouseout:dq0hvd;fvlRjc:jbFSO
d;CrflRd:SzACGe;" jscontroller="Ae65rd">
<div class="PPHIP rviiZ" jsname="haAclf">
.
</div>
<span style="font-family: 'Oswald'; font-weight: 500;">
Telephone : 01564 773348
</span>
</div>
</h3>
<div class="GV3q8e aP9z7e" id="h.p_sylefz-BOSBX">
</div>
><h3 id="h.p_sylefz-BOSBX" dir="ltr" class="CDt 4Ke zfr3Q OmQG5e"
</div>
</div>
</div>
</div>
</div>
</div>]
'''

Beautifulsoup: Replace all <div> with aria-level attributes with <h> tags of the same level

I have a HTML source where <div> elements serve as headings. Using Beautifulsoup and the attribute aria-level I would like to replace all <div> elements with <h> tags of the same level. My code kind of works for my purpose but it seems inelegant and ideally, the attributes of the former <div> elements would be removed.
import bs4
html = '''<div id="container">
<div role="heading" aria-level="1">The main page heading</div>
<p>This article is about showing a page structure.</p>
<div role="heading" aria-level="2">Introduction</div>
<p>An introductory text.</p>
<div role="heading" aria-level="2">Chapter 1</div>
<p>Text</p>
<div role="heading" aria-level="3">Chapter 1.1</div>
<p>More text in a sub section.</p>
</div>'''
soup = bs4.BeautifulSoup(html, "html.parser")
for divheader in soup.find_all("div", {"aria-level": "1"}):
divheader.name = "h1"
for divheader in soup.find_all("div", {"aria-level": "2"}):
divheader.name = "h2"
for divheader in soup.find_all("div", {"aria-level": "3"}):
divheader.name = "h3"
print(soup)
Output:
<div id="container">
<h1 aria-level="1" role="heading">The main page heading</h1>
<p>This article is about showing a page structure.</p>
<h2 aria-level="2" role="heading">Introduction</h2>
<p>An introductory text.</p>
<h2 aria-level="2" role="heading">Chapter 1</h2>
<p>Text</p>
<h3 aria-level="3" role="heading">Chapter 1.1</h3>
<p>More text in a sub section.</p>
</div>
What it should look like:
<div id="container">
<h1>The main page heading</h1>
<p>This article is about showing a page structure.</p>
<h2>Introduction</h2>
<p>An introductory text.</p>
<h2>Chapter 1</h2>
<p>Text</p>
<h3>Chapter 1.1</h3>
<p>More text in a sub section.</p>
</div>

You can use del.attrs to delete all attributes from tag:
for div in soup.select("div[aria-level]"):
div.name = f'h{div["aria-level"]}'
del div.attrs
print(soup)
Prints:
<div id="container">
<h1>The main page heading</h1>
<p>This article is about showing a page structure.</p>
<h2>Introduction</h2>
<p>An introductory text.</p>
<h2>Chapter 1</h2>
<p>Text</p>
<h3>Chapter 1.1</h3>
<p>More text in a sub section.</p>
</div>

Python: extract certain class separately by using bs4

<div class="michelinKeyBenefitsComp">
<section id="benefit-one-content">
<div class="inner">
<div class="col">
<h4 class="h-keybenefits">Banana is yellow.</h4>
<div class="content">
<p>Yellow is my favorite color.</p>
<p> </p>
<p>I love Banana.</p>
</div>
</div>
</div>
</section>
<section id="benefit-two-content">
<div class="inner">
<div class="col">
<h4 class="h-keybenefits">Apple is red.</h4>
<div class="content"><p>Red is not my favorite color.</p>
<p> </p>
<p>I don't like apple.</p>
</div>
</div>
</div>
</section>
</div>
I know how to extract all the text I want from this HTML. Here is my code:
for item in soup.find('div', {'class' : 'michelinKeyBenefitsComp'}):
try:
for tex in item.find_all('div', {'class' : 'col'}):
print(tex.text)
except:
pass
But what i would like to do is extract the content separately, so I can save them separately. The result is expected like this:
Banana is yellow.
Yellow is my favorite color.
I love Banana.
#save first
Apple is red.
Red is not my favorite color.
I don't like apple.
#save next
By the way, in this case, there are only 2 paragraph, but in other cases, there are probably three or more paragraphs. How can I extract them without knowing how many paragraphs they have? TIA

May be you should try this way for extracting text, you have div with unique_id, but for selecting section text inside it you can use classes for properly select text from particular div,
from bs4 import BeautifulSoup
text = """
<div class="michelinKeyBenefitsComp">
<section id="benefit-one-content">
<div class="inner">
<div class="col">
<h4 class="h-keybenefits">Banana is yellow.</h4>
<div class="content">
<p>Yellow is my favorite color.</p>
<p> </p>
<p>I love Banana.</p>
</div>
</div>
</div>
</section>
<section id="benefit-two-content">
<div class="inner">
<div class="col">
<h4 class="h-keybenefits">Apple is red.</h4>
<div class="content"><p>Red is not my favorite color.</p>
<p> </p>
<p>I don't like apple.</p>
</div>
</div>
</div>
</section>
</div>
"""
soup = BeautifulSoup(text, 'html.parser')
main_div = soup.find('div', class_='michelinKeyBenefitsComp')
for idx, div in enumerate(main_div.select('section > div.inner > div.col')):
with open('file_'+str(idx)+'.txt', 'w', encoding='utf-8') as f:
f.write(div.get_text())
#Output in separate file: file_1.txt> Banana is yellow.
# Yellow is my favorite color.
# I love Banana.

This should help.
from bs4 import BeautifulSoup
import re
soup = BeautifulSoup(html, "html.parser")
for i in soup.find_all("section", {"id": re.compile("benefit-[a-z]+-content")}):
with open(i["id"]+".txt", "a") as outfile: #Create filename based on section ID and write.
outfile.write("\n".join([i for i in i.text.strip().split("\n") if i.strip()]) + "\n\n")