Is it possible to set up in beautifulSoup that I can print only links that has <img> inside its content?
Currently my code looks like this:
import urllib
import re
import mechanize
from bs4 import BeautifulSoup
import urlparse
url = "http://www.nytimes.com"
htmlcontent = urllib.urlopen(url)
soup = BeautifulSoup(htmlcontent)
for link in soup.find_all('a'):
print link.contents
which print outs all content inside links. But my true need is to print links that has <img> tags inside it content and I don't know how to do that...
any help is welcome
Just try to find img tag inside the link:
for link in soup.find_all('a'):
if link.find('img'):
print link
Related
I am working with beautiful soup to extract the URL. I get all the attributes of the href but i want to get only specific URL.
Here is my code:
import requests
from bs4 import BeautifulSoup
page=requests.get("https://www.youtube.com/results?search_query=cooking")
soup = BeautifulSoup(page.content ,'html.parser')
for a_tag in soup.findAll("a"):
if a_tag.has_attr("href"):
print(a_tag['href'])
enter image description here
but i want only this
watch?v=nTe_44ao82w
/watch?v=nTe_44ao82w
More Minimization to the first answer:
import requests
from bs4 import BeautifulSoup
page=requests.get("https://www.youtube.com/results?search_query=cooking")
soup = BeautifulSoup(page.content ,'html.parser')
for a_tag in soup.findAll("a"):
if 'watch' in a_tag['href']:
print(a_tag['href'])
This will check if the href tag has string watch in it.
Hope this helps!
There doesn't really seem to be a good way to differentiate those a tags other than by the URL itself (they don't have any unique classes or anything) so I would probably just check if the URL contains "watch":
...
for a_tag in soup.findAll("a"):
if a_tag.has_attr("href") and "watch" in a_tag["href"]:
print(a_tag['href'])
Outputs
/watch?v=cbxe1ANrfDo
/watch?v=nTe_44ao82w
/watch?v=v1wIThmCams
/watch?v=FTociictyyE
/watch?v=dw2QHkAtB_Y
/watch?v=ej9UHVwlQqk
/watch?v=KGAj8IhnR3c
/watch?v=G8A73R_gZdM
/watch?v=XPQW_2YOmjY
/watch?v=J0pS2lhH0Vc
/watch?v=5aU5qrbCsF4
/watch?v=kvAJ_mc9NXs
/watch?v=kKiYVLIk_9s
/watch?v=G2jYIGdmC6I
/watch?v=jMW5ZDQviOA
/watch?v=iTmcGy9CWhE
/watch?v=66Ck_5SePZg
/watch?v=lyD9t3uhHio
HTML noob here, so I could be misunderstanding something about the HTML document, so bear with me.
I'm using Beautiful Soup to parse web data in Python. Here is my code:
import urllib
import BeautifulSoup
url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url).read()
soup = BeautifulSoup.BeautifulSoup(page)
indicateGameDone = str(soup.find("div", {"class": "nbaModTopStatus"}))
print indicateGameDone
now, if you look at the website, the HTML code has the line <p class="nbaLiveStatTxSm"> FINAL </p>, (inspect the 'Final' text on the left side of the container on the first ATL-WAS game on the page to see it for youself.) But when I run the code above, my code doesn't return the 'FINAL' that is seen on the webpage, and instead the nbaLiveStatTxSm class is empty.
On my machine, this is the output when I print indicateGameDone:
<div class="nbaModTopStatus"><p class="nbaLiveStatTx">Live</p><p class="nbaLiveStatTxSm"></p><p class="nbaFnlStatTx">Final</p><p class="nbaFnlStatTxSm"></p></div>
Does anyone know why this is happening?
EDIT: clarification: the problem isn't retrieving the text within the tag, the problem is that when I take the html code from the website and print it out in python, something that I saw when I inspected the element on the web is not there in the print statement in Python.
You can use this logic to extract any text.
This code allows you to extract any data between any tags.
Output - FINAL
import urllib
from bs4 import BeautifulSoup
url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url)
soup = BeautifulSoup(page)
indicateGameDone = soup.find("div", {"class": "nbaFnlStatTx"})
for p in indicateGameDone:
p_text = soup.find("p", {"class": "nbaFnlStatTxSm"})
print(p_text.getText())
break;
It looks like your problem is not with BeautifulSoup but instead with urllib.
Try running the following commands
>>> import urllib
>>> url = "http://www.nba.com/gameline/20160323/"
>>> page = urllib.urlopen(url).read()
>>> page.find('<div class="nbaModTopStatus">')
44230
Which is no surprise considering that Beautiful Soup was able to find the div itself. However when we look a little deeper into what urllib is actually collecting we can see that the <p class="nbaFnlStatTxSm"> is empty by running
>>> page[44230:45000]
'<div class="nbaModTopStatus"><p class="nbaLiveStatTx">Live</p><p class="nbaLiveStatTxSm"></p><p class="nbaFnlStatTx">Final</p><p class="nbaFnlStatTxSm"></p></div><div id="nbaGLBroadcast"><img src="/.element/img/3.0/sect/gameline/broadcasters/lp.png"></div><div class="nbaTeamsRow"><div class="nbaModTopTeamScr nbaModTopTeamAw"><h5 class="nbaModTopTeamName awayteam">ATL</h5><img src="http://i.cdn.turner.com/nba/nba/.element/img/2.0/sect/gameline/teams/ATL.gif" width="34" height="22" title="Atlanta Hawks"><h4 class="nbaModTopTeamNum win"></h4></div><div class="nbaModTopTeamScr nbaModTopTeamHm"><h5 class="nbaModTopTeamName hometeam">WAS</h5><img src="http://i.cdn.turner.com/nba/nba/.element/img/2.0/sect/gameline/teams/WAS.gif" width="34" '
You can see that the tag is empty, so your problem is the data that's being passed to Beautiful Soup, not the package itself.
changed the import of beautifulsoup to the proper syntax for the current version of BeautifulSoup
corrected the way you were constructing the BeautifulSoup object
fixed your find statement, then used the .text command to get the string representation of the text in the HTML you're after.
With some minor modifications to your code as listed above, your code runs for me.
import urllib
from bs4 import BeautifulSoup
url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url).read()
soup = BeautifulSoup(page)
indicateGameDone = soup.find("div", {"class": "nbaModTopStatus"})
print indicateGameDone.text ## "LiveFinal "
to address comments:
import urllib
from bs4 import BeautifulSoup
url = "http://www.nba.com/gameline/20160323/"
page = urllib.urlopen(url).read()
soup = BeautifulSoup(page)
indicateGameDone = soup.find("p", {"class": "nbaFnlStatTx"})
print indicateGameDone.text
I want to extract specific URLs from an HTML page.
from urllib2 import urlopen
import re
from bs4 import BeautifulSoup
url = http://bassrx.tumblr.com/tagged/tt # nsfw link
page = urlopen(url)
html = page.read() # get the html from the url
# this works without BeautifulSoup, but it is slow:
image_links = re.findall("src.\"(\S*?media.tumblr\S*?tumblr_\S*?jpg)", html)
print image_links
The output of the above is exactly the URL, nothing else: http://38.media.tumblr.com/tumblr_ln5gwxHYei1qi02clo1_500.jpg
The only downside is it is very slow.
BeautifulSoup is extremely fast at parsing HTML, so that's why I want to use it.
The urls that I want are actually the img src. Here's a snippet from the HMTL that contains that information that I want.
<div class="media"><a href="http://bassrx.tumblr.com/image/85635265422">
<img src="http://38.media.tumblr.com/tumblr_ln5gwxHYei1qi02clo1_500.jpg"/>
</a></div>
So, my question is, how can I get BeautifulSoup to extract all of those 'img src' urls cleanly without any other cruft?
I just want a list of matching urls. I've been trying to use soup.findall() function, but cannot get any useful results.
from urllib2 import urlopen
from bs4 import BeautifulSoup
url = 'http://bassrx.tumblr.com/tagged/tt'
soup = BeautifulSoup(urlopen(url).read())
for element in soup.findAll('img'):
print(element.get('src'))
You can use div.media > a > img CSS selector to find img tags inside a which is inside a div tag with media class:
from urllib2 import urlopen
from bs4 import BeautifulSoup
url = "<url_here>"
soup = BeautifulSoup(urlopen(url))
images = soup.select('div.media > a > img')
print [image.get('src') for image in images]
In order to make the parsing faster you can use lxml parser:
soup = BeautifulSoup(urlopen(url), "lxml")
You need to install lxml module first, of course.
Also, you can make use of a SoupStrainer class for parsing only relevant part of the document.
Hope that helps.
Have a look a BeautifulSoup.find_all with re.compile mix
from urllib2 import urlopen
import re
from bs4 import BeautifulSoup
url = "http://bassrx.tumblr.com/tagged/tt" # nsfw link
page = urlopen(url)
html = page.read()
bs = BeautifulSoup(html)
a_tumblr = [a_element for a_element in bs.find_all(href=re.compile("media\.tumblr"))]
##[<link href="http://37.media.tumblr.com/avatar_df3a9e37c757_128.png" rel="shortcut icon"/>, <link href="http://37.media.tumblr.com/avatar_df3a9e37c757_128.png" rel="apple-touch-icon"/>]
I'm having a problem in Python with BeautifulSoup. I need to extract all files on the page that end in ".php", but they also have to be local files. They can't be from another website. This is what I have so far:
from bs4 import BeautifulSoup
import mechanize
import sys
url = sys.argv[1]
br = mechanize.Browser()
code = br.open(url)
html = code.read()
soup = BeautifulSoup(html)
This is where I get stuck on what to do. I imagine using soup.findall to get all the "a href" tags.
Try like this,
page=urllib2.urlopen(url)
soup=BeautifulSoup(page.read())
for a in soup.findAll('a'):
if a['href'].endswith('.php'):
print a['href']
import glob,os
path=input("Enter Your Path in "" =")+"//"
print path
for i in glob.glob(os.path.join(str(path),"*.php")):
print i
I am doing some python program to extract lyrics
the code i use:
import urllib
from bs4 import BeautifulSoup
url = urllib.urlopen("http://www.lyricsnmusic.com/david-bowie/slip-away-lyrics/22143075")
soup = BeautifulSoup(url.read())
print soup.find('pre', itemprop='description')
the result gets me what i need but with the extra of the tag
for example : <pre item="description> then the lyrics
anyone know how to get only the lyrics?
the structure puts the lyrics between the pre tag
thanks in advance
Use the text attribute of the node that you've found
import urllib
from BeautifulSoup import BeautifulSoup
url = urllib.urlopen("http://www.lyricsnmusic.com/david-bowie/slip-away-lyrics/2
2143075")
soup = BeautifulSoup(url.read())
desc=soup.find('pre', itemprop='description')
print desc.text