We Keep Coding

How do I filter tags with class in Python and BeautifulSoup?

I'm trying to scrape images from a site using beautifulsoup HTML parser.
There are 2 kinds of image tags for each image on the site. One is for the thumbnail and the other is the bigger size image that only appears after I click on the thumbnail and expand. The bigger size tag contains a class="expanded-image" attribute.
I'm trying to parse through the HTML and get the "src" attribute of the expanded image which contains the source for the image.
When I try to execute my code, nothing happens. It just says the process finished without scraping any image. But when I don't try to filter the code and just give tag as an argument, it downloads all the thumbnails.
Here's my code:
import webbrowser, requests, os
from bs4 import BeautifulSoup
def getdata(url):
r = requests.get(url)
return r.text
htmldata = getdata('https://boards.4chan.org/a/thread/30814')
soup = BeautifulSoup(htmldata, 'html.parser')
list = []
for i in soup.find_all("img",{"class":"expanded-thumb"}):
list.append(i['src'].replace("//","https://"))
def download(url, pathname):
if not os.path.isdir(pathname):
os.makedirs(pathname)
filename = os.path.join(pathname, url.split("/")[-1])
response = requests.get(url, stream=True)
with open(filename, "wb") as f:
f.write(response.content)
for a in list:
download(a,"file")

You might be running into a problem using "list" as a variable name. It's a type in python. Start with this (replacing TEST_4CHAN_URL with whatever thread you want), incorporating my suggestion from the comment above.
import requests
from bs4 import BeautifulSoup
TEST_4CHAN_URL = "https://boards.4chan.org/a/thread/<INSERT_THREAD_ID_HERE>"
def getdata(url):
r = requests.get(url)
return r.text
htmldata = getdata(TEST_4CHAN_URL)
soup = BeautifulSoup(htmldata, "html.parser")
src_list = []
for i in soup.find_all("a", {"class":"fileThumb"}):
src_list.append(i['href'].replace("//", "https://"))
print(src_list)

Downloading images with BeautifulSoup when complete image link does not appear unless hovering over the src tag

I'm trying to download images from this page. I wrote the following Python script:
import requests
import subprocess
from bs4 import BeautifulSoup
request = requests.get("http://ottofrello.dk/malerierstor.htm")
content = request.content
soup = BeautifulSoup(content, "html.parser")
element = soup.find_all("img")
for img in element:
print (img.get('src'))
However, I get only the image names and not the complete path. On the site, I can hover over the image name when I inspect the html and the link appears. Is there any way I can parse this link using BeautifulSoup?
Image

The image URIs in your page are marked up relative to the hostname.
You can build an absolute url for each image using urljoin function in urllib.parse module.
from urllib.parse import urljoin
page_url = "http://ottofrello.dk/malerierstor.htm"
request = requests.get(page_url)
...
for img in element:
image_url = urljoin(
page_url,
img.get('src')
)
print(image_url)

From what I understood, you are interested in absolute image path and not the relative path which you are getting right now. The only change I made is in your print statement.
import requests
import subprocess
from bs4 import BeautifulSoup
request = requests.get("http://ottofrello.dk/malerierstor.htm")
content = request.content
soup = BeautifulSoup(content, "html.parser")
element = soup.find_all("img")
for img in element:
print ('http://ottofrello.dk/' + img.get('src'))

Trying to use bs4 to skip an attribute if there is no data

I'm trying to write a program that will pull data from a URL and format it so that I can copy into another program. I've got everything working except I can't get it to skip an item if there is no img src in the imagelink tag.
import requests, sys, webbrowser, bs4
res = requests.get('http://hzws.selco.info/prototype.php?type=new-arrivals&lib=nor&collect=Bnewnf,Bnewmys,Bnewf,Bnewsf&days=14&key=7a8adfa9aydfa999997af')
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, "lxml")
img = soup.select('imagelink') #why won't this pull anything?!?!?!?!
link = soup.select('cataloglink')
length = min([14, len(img)])
for i in range(length):
img1 = img[i].getText()
link1 = link[i].getText()
print('<div>' + link1 + img1 + '</a></div>')
Right now this prints all of the URLs regardless of whether or not there is an imagelink attached to it. I've tried numerous different things to get it to skip an item if there is no img src. Any ideas?

Looking at the BS4 docs, It looks like "lxml" is actually a HTML parser. You should replace it with "lxml-xml", since you're trying to scrape an XML page. That should be working.

How to get website images src using BeautifulSoup

I'm trying to get all the images src/hyperlink form a webpage
import requests
from bs4 import BeautifulSoup
image_list = []
r = requests.get('https://example.com')
soup = BeautifulSoup(r.content)
for link in soup.find_all('img'):
image_list.append(link)

find the attributes of an html tag using get function. Pass the name of the attribute you want to extract from html tag to get function
for link in soup.find_all('img'):
image_list.append(link.get('src'))

Python Beautifulsoup img tag parsing

I am using beautifulsoup to parse all img tags which is present in 'www.youtube.com'
The code is
import urllib2
from BeautifulSoup import BeautifulSoup
page = urllib2.urlopen('http://www.youtube.com/')
soup = BeautifulSoup(page)
tags=soup.findAll('img')
But am not getting all img tags.The getting img tags are invalid also.
The img tags which i got after parsing is different from the page source img tags. Some attributes are missing.
I need to get all video img tags in youtube.com
Please help

Seems to work when I try it here
import urllib2
from BeautifulSoup import BeautifulSoup
page = urllib2.urlopen('http://www.youtube.com/')
soup = BeautifulSoup(page)
tags=soup.findAll('img')
print "\n".join(set(tag['src'] for tag in tags))
Produces this which looks OK to me
http://i1.ytimg.com/vi/D9Zg67r9q9g/market_thumb.jpg?v=723c8e
http://s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif
//s.ytimg.com/yt/img/pixel-vfl3z5WfW.gif
/gen_204?a=fvhr&v=mha7pAOfqt4&nocache=1337083207.97
http://i3.ytimg.com/vi/fNs8mf2OdkU/market_thumb.jpg?v=4f85544b
http://i4.ytimg.com/vi/CkQFjyZCq4M/market_thumb.jpg?v=4f95762c
http://i3.ytimg.com/vi/fzD5gAecqdM/market_thumb.jpg?v=b0cabf
http://i3.ytimg.com/vi/2M3pb2_R2Ng/market_thumb.jpg?v=4f0d95fa
//i2.ytimg.com/vi/mha7pAOfqt4/hqdefault.jpg

I had the similar problem. I couldn't find all images. So here is the piece of code that will give you any attribute value of an image tag.
from BeautifulSoup import BeautifulSoup as BSHTML
import urllib2
page = urllib2.urlopen('http://www.youtube.com/')
soup = BSHTML(page)
images = soup.findAll('img')
for image in images:
#print image source
print image['src']
#print alternate text
print image['alt']

Explicitly using soup.findAll(name='img') worked for me, and I don't appear to be missing anything from the page.

def grabimagetags():
import urllib2
from BeautifulSoup import BeautifulSoup
page = urllib2.urlopen('http://www.youtube.com/')
soup = BeautifulSoup(page)
tags = soup.findAll('img')
list.extend(set(tag['src'] for tag in tags))
return list
grabimagetags()
i would only make this change so that you can pass the list of img tags

in my case some images didn't contain src.
so i did this to avoid keyError exception:
art_imgs = set(img['src'] for img in article.find_all('img') if img.has_attr('src'))

Try this.
from simplified_scrapy import SimplifiedDoc, req
url = 'https://www.youtube.com'
html = req.get(url)
doc = SimplifiedDoc(html)
imgs = doc.listImg(url = url)
print([img.url for img in imgs])
imgs = doc.selects('img')
for img in imgs:
print (img)
print (doc.absoluteUrl(url,img.src))