I have a website where I'd like to get all the images from the website.
The website is kind of a dynamic in nature, I tried using google's Agenty Chrome extension and followed the steps:
I Choose one image that I want to extract using CSS selector, this will make the extension select the same other images automatically.
Viewed the Show button and select ATTR(attribute).
Changed src as an ATTR field.
Gave a name field name option.
Saved it & ran it in using Agenty platform/API.
This should yield me the result but it's not, it is returning an empty output.
Is there any better option? Will BS4 a better option for this? Any help is appreciated.
I am assuming you want to download all images in the website. It is actually very easy to do this effectively using beautiful soup 4 (BS4).
#code to find all images in a given webpage
from bs4 import BeautifulSoup
import urllib.request
import requests
import shutil
url=('https://www.mcmaster.com/')
html_page = urllib.request.urlopen(url)
soup = BeautifulSoup(html_page, features="lxml")
for img in soup.findAll('img'):
assa=(img.get('src'))
new_image=(url+assa)
You can also download the image with this tacked-on to the end:
response = requests.get(my_url, stream=True)
with open('Mypic.bmp', 'wb') as file:
shutil.copyfileobj(response.raw, file)
Everything in two lines:
from bs4 import BeautifulSoup; import urllib.request; from urllib.request import urlretrieve
for img in (BeautifulSoup((urllib.request.urlopen("https://apod.nasa.gov/apod/astropix.html")), features="lxml")).findAll('img'): assa=(img.get('src')); urlretrieve(("https://apod.nasa.gov/apod/"+assa), "Mypic.bmp")
The new image should be in the same directory as the python file, but can be moved with:
os.rename()
In the case of the McMaster website, the images are linked differently, so the above methods won't work. The following code should get most of the images on the website:
from bs4 import BeautifulSoup
from urllib.request import Request, urlopen
import re
import urllib.request
import shutil
import requests
req = Request("https://www.mcmaster.com/")
html_page = urlopen(req)
soup = BeautifulSoup(html_page, "lxml")
links = []
for link in soup.findAll('link'):
links.append(link.get('href'))
print(links)
UPDATE: I found from some github post the below code that is MUCH more accurate:
import requests
import re
image_link_home=("https://images1.mcmaster.com/init/gfx/home/.*[0-9]")
html_page = requests.get(('https://www.mcmaster.com/'),headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'}).text
for item in re.findall(image_link_home,html_page):
if str(item).startswith('http') and len(item) < 150:
print(item.strip())
else:
for elements in item.split('background-image:url('):
for item in re.findall(image_link_home,elements):
print((str(item).split('")')[0]).strip())
Hope this helps!
You should use scrapy, it makes the crawling seamless, by selecting the content you wish to download with css tags You can automate the crawling easily.
You can use Agenty Web Scraping Tool.
Setup your scraper using Chrome extension to extract src attribute from images
Save the agent to run on cloud.
Here is similar question answered on Agenty forum - https://forum.agenty.com/t/can-i-extract-images-from-website/24
Full Disclosure - I am working at Agenty
This site using CSS embedding to store images. If you check the source code you can find links which has https://images1.mcmaster.com/init/gfx/home/ those are the actual images but its actually stitched together (row of images)
Example : https://images1.mcmaster.com/init/gfx/home/Fastening-and-Joining-Fasteners-sprite-60.png?ver=1539608820
import requests
import re
url=('https://www.mcmaster.com/')
image_urls = []
html_page = requests.get(url,headers={'User-Agent':'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'}).text
for values in re.findall('https://images1.mcmaster.com/init/gfx/home/.*[0-9]',html_page):
if str(values).startswith('http') and len(values) < 150:
image_urls.append(values.strip())
else:
for elements in values.split('background-image:url('):
for urls in re.findall('https://images1.mcmaster.com/init/gfx/home/.*[0-9]',elements):
urls = str(urls).split('")')[0]
image_urls.append(urls.strip())
print(len(image_urls))
print(image_urls)
Note: Scraping website is subject to copyrights
Related
I've created a script using requests and BeautifulSoup library to parse the links of some images from a webpage. The image links are visible when you use this selector [class^='cylindo-viewer-frame'] > img[src*='/frames/'] within the search bar (Ctrl + F) after inspecting element. This how they look like in the dom.
I know I can grab those image links using selenium but I would like to stick with requests module. I've noticed several times that there are always possibilities to grab such dynamic content using requests module. I've tried finding those links within script tag and in dev tools but no luck.
Two of the expected links out of 32 are:
https://content.cylindo.com/api/v2/4616/products/657285/frames/5/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/7/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
This is how I've tried:
import requests
from bs4 import BeautifulSoup
link = 'https://www.ethanallen.com/on/demandware.store/Sites-ethanallen-us-Site/en_US/Product-Variation?pid=emersonQS&dwvar_emersonQS_Fabric=Q1031&dwvar_emersonQS_seatingSize=90sofa&step=2'
with requests.Session() as s:
s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36'
r = s.get(link)
soup = BeautifulSoup(r.text,"lxml")
for item in soup.select(".cylindo-viewer-container li[class^='cylindo-viewer-frame'] > img[src*='/frames/']"):
print(item.get("src"))
How can I grab those image links using requests?
Why you should use selenium?
Website serves content dynamically, what is not to handle with requests, cause the information you try to match is not in the response.
Take a look, it is not that hard ;)
Example
from selenium import webdriver
from bs4 import BeautifulSoup
from time import sleep
driver = webdriver.Chrome(executable_path='C:\Program Files\ChromeDriver\chromedriver.exe')
url = "https://www.ethanallen.com/on/demandware.store/Sites-ethanallen-us-Site/en_US/Product-Variation?pid=emersonQS&dwvar_emersonQS_Fabric=Q1031&dwvar_emersonQS_seatingSize=90sofa&step=2"
driver.get(url)
sleep(2)
soup = BeautifulSoup(driver.page_source, 'lxml')
for item in soup.select(".cylindo-viewer-container li[class^='cylindo-viewer-frame'] > img[src*='/frames/']"):
print(item.get("src"))
driver.close()
Output
https://content.cylindo.com/api/v2/4616/products/657285/frames/3/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/27/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/29/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/11/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/31/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
https://content.cylindo.com/api/v2/4616/products/657285/frames/5/657285.JPG?background=FFFFFF&feature=FABRIC:Q1031&size=1268
...
Link:https://www.exam-mate.com/topicalpastpapers/?cat=3&subject=22&years=&seasons=&paper=&zone=&chapter=&order=asc0
This website has questions in image form that I need to scrape. However I cannot even get a link to their source and it outputs links to some loading gifs. When I saw the source code, there weren't even any "src" to the images. You can see how the website works on the link provided above. How can I download all these images?
from bs4 import BeautifulSoup
import requests
import os
url = "https://www.exam-mate.com/topicalpastpapers/?cat=3&subject=22&years=&seasons=&paper=&zone=&chapter=&order=asc0"
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
images = soup.find_all('img')
for image in images:
link = image['src']
print (link)
The question id's are embedded as part of the page, try extracting the id using the re(regex) module.
import re
import requests
from bs4 import BeautifulSoup
headers = {
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36"
}
URL = "https://www.exam-mate.com/topicalpastpapers/?cat=3&subject=22&years=&seasons=&paper=&zone=&chapter=&order=asc0"
BASE_URL = "https://www.exam-mate.com"
soup = BeautifulSoup(requests.get(URL).content, "html.parser")
for tag in soup.select("td:nth-of-type(1) a"):
# Find the question id within the page
question_link = re.search(r"/questions.*\.png", tag["onclick"]).group()
print(BASE_URL + question_link)
Output:
https://www.exam-mate.com/questions/1240/1362/1240_q_1362_1_1.png
https://www.exam-mate.com/questions/1240/1363/1240_q_1363_2_1.png
https://www.exam-mate.com/questions/1240/1364/1240_q_1364_3_1.png
https://www.exam-mate.com/questions/1240/1365/1240_q_1365_4_1.png
https://www.exam-mate.com/questions/1240/1366/1240_q_1366_5_1.png
...And on
As the page is dynamic BeautifulSoup doesn't work here. Have to use selenium
Navigate to the site
Get all questions using xpath: //div/div[3]/center/table/tbody/tr/td[1]/center/a and loop and click on them.
Get the image source using xpath: //*[#id="question_prev"]/div[2]/img/#src then get and save the image.
I have looked through the previous answers but none seemed to be applicable. I am building an open source quizlet scraper to extract all links from a class (e.g. https://quizlet.com/class/3675834/). In this case, the tag is a and class is "UILink". But when I use the following code, the list returned does not contain the element that I am looking for. Is it because of the JavaScript issue described here?
I tried to use the previous method of importing folder as written here but it does not contain the urls.
How can I scrape these urls?
import requests
from bs4 import BeautifulSoup
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36"
}
url = 'https://quizlet.com/class/8536895/'
response = requests.get(url, verify=False, headers=headers)
soup = BeautifulSoup(response.text,'html.parser')
b = soup.find_all("a", class_="UILink")
You wouldn't be able to directly scrape dynamic webpages using just requests. What you see browser is fully rendered page taken care by browser.
Inorder to scrape data from these kind of webpages, you following any of below approaches.
Use requests-html instead of requests
pip install requests-html
scraper.py
from requests_html import HTMLSession
from bs4 import BeautifulSoup
session = HTMLSession()
url = 'https://quizlet.com/class/8536895/'
response = session.get(url)
response.html.render() # render the webpage
# access html page source with html.html
soup = BeautifulSoup(response.html.html, 'html.parser')
b = soup.find_all("a", class_="UILink")
print(len(b))
Note: this uses headless browser(chromium) under the hood to render the page. So it can timeout or be a little slow at times.
Use selenium webdriver
Use driver.get(url) to get the page and pass the page source to beautiful Soup with driver.page_source
Note: run this in headless mode as well and there might be some latency at times.
Ok, I have been scratching my head on this for way too long. I am trying to retrieve the url for an embedded video on a web page using Beautiful Soup and requests modules in Python 2.7.6. I inspect the html in chrome and I can see the url to the video but when I get the page using requests and use Beautiful Soup I can't find the "video" node. From looking at the source it looks like the video window is a nested html document. I have searched all over and can't find out why I can't retrieve this. If anyone could point me in the right direction I would greatly appreciate it. Thanks.
here is the url to one of the videos:
http://www.growingagreenerworld.com/episode125/
The problem is that there is an iframe with the video tag inside which is loaded asynchronously in the browser.
Good news is that you can simulate that behavior by making an additional request to the iframe URL passing the current page URL as a Referer.
Implementation:
import re
from bs4 import BeautifulSoup
import requests
url = 'http://www.growingagreenerworld.com/episode125/'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36'}
with requests.Session() as session:
session.headers = headers
response = session.get(url)
soup = BeautifulSoup(response.content)
# follow the iframe url
response = session.get('http:' + soup.iframe['src'], headers={'Referer': url})
soup = BeautifulSoup(response.content)
# extract the video URL from the script tag
print re.search(r'"url":"(.*?)"', soup.script.text).group(1)
Prints:
http://pdl.vimeocdn.com/43109/378/290982236.mp4?token2=1424891659_69f846779e96814be83194ac3fc8fbae&aksessionid=678424d1f375137f
This is my first attempt at web scraping. I am trying to use Beautiful Soup to scrape phone numbers from Raymond James' website. An example would be http://www.raymondjames.com/office_locator_display.asp?addressline=90210
Whenever I use BeautifulSoup, I am unable to find the appropriate information in the HTML.
import urllib2
from bs4 import BeautifulSoup
url='http://www.raymondjames.com/office_locator_display.asp?addressline=90210'
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36')]
page_to_scrape=opener.open(url).read()
soup=BeautifulSoup(page_to_scrape.decode('utf-8','ignore'))
The output produced does not contain the information I need. It seems the URL I provide does not point to the location frame.
I don't use Python for a whole lot of work with web data so I am ignorant on how to direct Beautiful Soup into the 'frame' in order to get contact information.
As Martijn said, dig in the network requests, and the source data is there. In this case it's an xml response to a GET request made in the iframe. Armed with that url, the solution is pretty simple:
import urllib2
from bs4 import BeautifulSoup
soup = BeautifulSoup(urllib2.urlopen('http://hosted.where2getit.com/raymondjames/ajax?&xml_request=%3Crequest%3E%3Cappkey%3E7BD67064-FC36-11E0-B80D-3AEEDDB2B31E%3C%2Fappkey%3E%3Cformdata+id%3D%22locatorsearch%22%3E%3Cdataview%3Estore_default%3C%2Fdataview%3E%3Climit%3E30%3C%2Flimit%3E%3Cgeolocs%3E%3Cgeoloc%3E%3Caddressline%3E90210%3C%2Faddressline%3E%3Clongitude%3E%3C%2Flongitude%3E%3Clatitude%3E%3C%2Flatitude%3E%3Ccountry%3E%3C%2Fcountry%3E%3C%2Fgeoloc%3E%3C%2Fgeolocs%3E%3Csearchradius%3E25%7C50%7C100%3C%2Fsearchradius%3E%3C%2Fformdata%3E%3C%2Frequest%3E'), 'lxml')
# parse the points of interest into a list
pois = soup.find_all('poi')
# now have your way with them!