>>> soup_brand
<a data-role="BRAND" href="/URL/somename">
Some Name
</a>
>>> type(soup_brand)
<class 'bs4.BeautifulSoup'>
>>> print(soup_brand.get('href'))
None
Documentation followed: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
Hi people from all over the world,
does someone now whats going wrong or am I targeting the object wrong ?
Need to get the href.
Have you tried:
soup.find_all(name="a")
or
soup.select_one(selector="a")
it should also be possible to catch with
all_anchor_tags = soup.find_all(name="a")
for tag in all_anchor_tags:
print(tag.get("href")) #prints the href element of each a tag, thus each link
Although the all bs4 looks for multiple elemnts (the reason why we have a loop here) I encountered, that bs4 sometime is better in catching things, if you give it a search for all approach and then iterate over the elements
in order to apply ['href'] the object must be <bs4.Element.Tag>.
so, try this:
string = \
"""
<a data-role="BRAND" href="/URL/somename">
Some Name
</a>
"""
s = BeautifulSoup(string)
a_tag = s.find('a')
print(a_tag["href"])
out
/URL/somename
or if you have multiple a tags you can try this:
a_tags = s.findAll('a')
for a in a_tags:
print(a.get("href"))
out
/URL/somename
Related
The output is part of a list. When I try to figure out the output's type using type() it returns : <class 'bs4.element.Tag'>.
I am trying to remove everything to the left of "href" and everything to the right of "<img". I have tried lstrip and rstrip but they do not work because each output in my list is unique. Even though each output in the list is unqiue they all have the same format with "href" and "<img".
Here is an example of what one of the outputs in my list:
<a class="BlogList-item-image-link" href="/new-blog/nova-approval">
<img alt="Nova Approval" data-image="https://static1.squarespace.com/static/54ceeff4e4b0d9096117315a/5a3ff7e48165f5d70b78414a/5a504ba90d9297f9a55e4ab6/1516062801655/7P1A5814+cropped.jpg" data-image-dimensions="2432x2688" data-image-focal-point="0.5,0.5" data-load="false" data-src="https://static1.squarespace.com/static/54ceeff4e4b0d9096117315a/5a3ff7e48165f5d70b78414a/5a504ba90d9297f9a55e4ab6/1516062801655/7P1A5814+cropped.jpg"/>
</a>
Using lstrip and rstrip won't be the answer.
Have you tried looking at the bs4 docs?
Because the type of your output is a bs4 object. You can just find the attribute of the object to get the href.
<a class="BlogList-item-image-link" href="/new-blog/nova-approval">
<img alt="Nova Approval" data-image="https://static1.squarespace.com/static/54ceeff4e4b0d9096117315a/5a3ff7e48165f5d70b78414a/5a504ba90d9297f9a55e4ab6/1516062801655/7P1A5814+cropped.jpg" data-image-dimensions="2432x2688" data-image-focal-point="0.5,0.5" data-load="false" data-src="https://static1.squarespace.com/static/54ceeff4e4b0d9096117315a/5a3ff7e48165f5d70b78414a/5a504ba90d9297f9a55e4ab6/1516062801655/7P1A5814+cropped.jpg"/>
</a>
from bs4 import BeautifulSoup
soup = BeautifulSoup('html') #put the link there
links = soup.find_all('a') # All of the anchor tags in a list
for link in links:
print(link.get('href'))
This will print all of the href values in the HTML file.
You are probably trying to extract the link in href. For that you don't need to strip the string. You could do it in the following way -
string = '''<a class="BlogList-item-image-link" href="/new-blog/nova-approval">
<img alt="Nova Approval" data-image="https://static1.squarespace.com/static/54ceeff4e4b0d9096117315a/5a3ff7e48165f5d70b78414a/5a504ba90d9297f9a55e4ab6/1516062801655/7P1A5814+cropped.jpg" data-image-dimensions="2432x2688" data-image-focal-point="0.5,0.5" data-load="false" data-src="https://static1.squarespace.com/static/54ceeff4e4b0d9096117315a/5a3ff7e48165f5d70b78414a/5a504ba90d9297f9a55e4ab6/1516062801655/7P1A5814+cropped.jpg"/>
</a>'''
print( string[string.find('href="')+6:string.find('>')-1] )
Output :
/new-blog/nova-approval
In the above print() statement, string.find('href="') will return the index of that string and we are then looping from that index + 6 to the end of href tag. This is assuming > follows right after href.
Hope this helps !
I'm having value like
<a href="/for-sale/property/abu-dhabi/page-3/" title="Next" class="b7880daf"><div title="Next" class="ea747e34 ">
I need to pull out only ""Next" from title="Next" for the one i used
soup.find('a',attrs={"title": "Next"}).get('title')
is there any method to get the tittle value with out using .get("title")
My code
next_page_text = soup.find('a',attrs={"title": "Next"}).get('title')
Output:
Next
I need:
next_page_text = soup.find('a',attrs={"title": "Next"})
Output:
Next
Please let me know if there is any method to find.
You should get Next.Try this. Using find() or select_one() and Use If to check if element is present on a page.
from bs4 import BeautifulSoup
import requests
res=requests.get("https://www.bayut.com/for-sale/property/abu-dhabi/page-182/")
soup=BeautifulSoup(res.text,"html.parser")
if soup.find("a", attrs={"title": "Next"}):
print(soup.find("a", attrs={"title": "Next"})['title'])
If you want to use css selector.
if soup.select_one("a[title='Next']"):
print(soup.select_one("a[title='Next']")['title'])
I'm re-writing my answer as there was confusion in your original post.
If you'd like to take the URL associated with the Next tag:
soup.find('a', title='Next')['href']
['href'] can be replaced with any other attribute in the element, so title, itemprop etc.
If you'd like to select the element with Next in the title:
soup.find('a', title='Next')
I am scraping items from a webpage (there are multiple of these):
<a class="iusc" style="height:160px;width:233px" m="{"cid":"T0QMbGSZ","purl":"http://www.tti.library.tcu.edu.tw/DERMATOLOGY/mm/mmsa04.htm","murl":"http://www.tti.lcu.edu.tw/mm/img0035.jpg","turl":"https://tse2.mm.bing.net/th?id=OIP.T0QMbGSZbOpkyXU4ms5SFwEsDI&pid=15.1","md5":"4f440c6c64996cea64c975389ace5217"}" mad="{"turl":"https://tse3.mm.bing.net/th?id=OIP.T0QMbGSZbOpkyXU4ms5EsDI&w=300&h=200&pid=1.1","maw":"300","mah":"200","mid":"C303D7F4BB661CA67E2CED4DB11E9154A0DD330B"}" href="/images/search?view=detailV2&ccid=T0QMbGSZ&id=C303D7F4BB661E2CED4DB11E9154A0DD330B&thid=OIP.T0QMbGSZbOpkyXU4ms5SFwEsDI&q=searchtearm;amp;simid=6080204499593&selectedIndex=162" h="ID=images.5978_5,5125.1" data-focevt="1"><div class="img_cont hoff"><img class="mimg" style="color: rgb(169, 88, 34);" height="160" width="233" src="https://tse3.mm.bing.net/th?id=OIP.T0QMbGSZ4ms5SFwEsDI&w=233&h=160&c=7&qlt=90&o=4&dpr=2&pid=1.7" alt="Image result fsdata-bm="169" /></div></a>
What I want to do is download the image and information associated with it in the m attribute.
To accomplish that, I tried something like this to get the attributes:
links = soup.find_all("a", class_="iusc")
And then, to get the m attribute, I tried something like this:
for a in soup.find_all("m"):
test = a.text.replace(""" '"')
metadata = json.loads(test)["murl"]
print(str(metadata))
However, that doesn't quite work as expected, and nothing is printed out (with no errors either).
You are not iterating through the links list. Try this.
links = soup.find_all("a", class_="iusc")
for link in links:
print(link.get('m'))
I have the following html part which repeates itself several times with other href links:
<div class="product-list-item margin-bottom">
<a title="titleexample" href="http://www.urlexample.com/example_1" data-style-id="sp_2866">
Now I want to get all the href links in this document that are directly after the div tag with the class "product-list-item".
Pretty new to beautifulsoup and nothing that I came up with worked.
Thanks for your ideas.
EDIT: Does not really have to be beautifulsoup; when it can be done with regex and the python html parser this is also ok.
EDIT2: What I tried (I'm pretty new to python, so what I did might be totaly stupid from an advanced viewpoint):
soup = bs4.BeautifulSoup(htmlsource)
x = soup.find_all("div")
for i in range(len(x)):
if x[i].get("class") and "product-list-item" in x[i].get("class"):
print(x[i].get("class"))
This will give me a list of all the "product-list-item" but then I tried something like
print(x[i].get("class").next_element)
Because I thought next_element or next_sibling should give me the next tag but it just leads to AttributeError: 'list' object has no attribute 'next_element'. So I tried with only the first list element:
print(x[i][0].get("class").next_element)
Which led to this error: return self.attrs[key] KeyError: 0.
Also tried with .find_all("href") and .get("href") but this all leads to the same errors.
EDIT3: Ok seems I found out how to solve it, now I did:
x = soup.find_all("div")
for i in range(len(x)):
if x[i].get("class") and "product-list-item" in x[i].get("class"):
print(x[i].next_element.next_element.get("href"))
This can also be shortened by using another attribute to the find_all function:
x = soup.find_all("div", "product-list-item")
for i in x:
print(i.next_element.next_element.get("href"))
greetings
I want to get all the href links in this document that are directly after the div tag with the class "product-list-item"
To find the first <a href> element in the <div>:
links = []
for div in soup.find_all('div', 'product-list-item'):
a = div.find('a', href=True) # find <a> anywhere in <div>
if a is not None:
links.append(a['href'])
It assumes that the link is inside <div>. Any elements in <div> before the first <a href> are ignored.
If you'd like; you can be more strict about it e.g., taking the link only if it is the first child in <div>:
a = div.contents[0] # take the very first child even if it is not a Tag
if a.name == 'a' and a.has_attr('href'):
links.append(a['href'])
Or if <a> is not inside <div>:
a = div.find_next('a', href=True) # find <a> that appears after <div>
if a is not None:
links.append(a['href'])
There are many ways to search and navigate in BeautifulSoup.
If you search with lxml.html, you could also use xpath and css expressions if you are familiar with them.
I'm trying to grab the string immediately after the opening <td> tag. The following code works:
webpage = urlopen(i).read()
soup = BeautifulSoup(webpage)
for elem in soup('td', text=re.compile(".\.doc")):
print elem.parent
when the html looks like this:
<td>plan_49913.doc</td>
but not when the html looks like this:
<td>plan_49913.doc<br />
<font color="#990000">Document superseded by: </font>January 2012</td>
I've tried playing with attrs but can't get it to work. Basically I just want to grab 'plan_49913.doc' in either instance of html.
Any advice would be greatly appreciated.
Thank you in advance.
~chrisK
This works for me:
>>> html = '<td>plan_49913.doc<br /> <font color="#990000">Document superseded by: </font>January 2012</td>'
>>> soup = BeautifulSoup(html)
>>> soup.find(text=re.compile('.\.doc'))
u'plan_49913.doc
Is there something I'm missing?
Also, note that according to the documentation:
If you use text, then any values you give for name and the keyword arguments are ignored.
So you don't need to pass 'td' since it's already being ignored, that is, any text that matches under any other tag will be returned.
Just use the next property, it contains the next node, and that's a textual node.
>>> html = '<td>plan_49913.doc<br /> <font color="#990000">Document superseded by: </font>January 2012</td>'
>>> bs = BeautifulSoup(html)
>>> texts = [ node.next for node in bs.findAll('td') if node.next.endswith('.doc') ]
>>> texts
[u'plan_49913.doc']
you can change the if clause to use a regex if you prefer.