for context I am pretty new to Python. I am trying to use bs4 to parse some data out of https://bigfuture.collegeboard.org/college-university-search/university-of-california-los-angeles
To be exact, I want to obtain the 57% number in the "paying" section of the webpage.
My problem is that bs4 will only return the first layer of the HTML, while the data I want is deeply nested in the code. I think it's under 17 divs.
Here is my python code:
import requests
import bs4
url = 'https://bigfuture.collegeboard.org/college-university-search/university-of-california-los-angeles'
res = requests.get(url)
soup = bs4.BeautifulSoup(res.text, "html.parser")
print(soup.find_all("div", {"id": "gwtDiv"}))
(This returns [<div class="clearfix margin60 marginBottomOnly" id="gwtDiv" style="min-height: 300px;height: 300px;height: auto;"></div>] None of the elements inside it are shown.)
If the page is using js to render content inside the element then requests will not be able to get that content since that content is rendered on the client side in a browser. I'd recommend using ChromeDriver and Selenium along with BeautifulSoup.
You can download the chrome driver from here:https://chromedriver.chromium.org/
Put this in the same folder in which you're running your program.
Try something like this
from selenium import webdriver
from bs4 import BeautifulSoup
url = 'https://bigfuture.collegeboard.org/college-university-search/university-of-california-los-angeles'
driver = webdriver.Chrome()
driver.get(url)
html = driver.execute_script("return document.documentElement.outerHTML")
sel_soup = BeautifulSoup(html, 'html.parser')
print(soup.find_all("div", {"id": "gwtDiv"}))
Related
I'm trying to scrape the link to the image on this reddit website for practice, but BS4 seems to be returning none whenever I use find() to find the class of the object. Any help?
from bs4 import BeautifulSoup as soup
page = requests.get("https://www.reddit.com/r/wallpaper/comments/qswblq/the_frontier_by_me_5120x2880/")
soup = soup(page.content, "html.parser")
print(soup.find(class_="ImageBox-image")['src'])
As mentioned in the comments there is an alternativ, you can use selenium.
Instead of requests it will render the site like a browser and will give page_source you could inspect and find your element.
Example:
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Chrome('YOUR PATH TO CHROMEDIVER')
driver.get('https://www.reddit.com/r/wallpaper/comments/qswblq/the_frontier_by_me_5120x2880/')
content = driver.page_source
soup = BeautifulSoup(content,'html.parser')
soup.find(class_="ImageBox-image")['src']
Im want to use bs4 in my Flask-App for searching a specific span.
I never used bs4 before so I'm a little bit confused why I don't get any results for my search.
from bs4 import BeautifulSoup
url = "https://www.mcfit.com/de/fitnessstudios/studiosuche/studiodetails/studio/berlin-lichtenberg/"
html_content = requests.get(url).text
soup = BeautifulSoup(html_content, "lxml")
spans = soup.find_all('span', {'class': 'sc-fzoXWK hnKkAN'})
print(spans)
The class 'sc-fzoXWK hnKkAN' only contains 1 span.
When I execute I only get a []as result.
Those contents are dynamically generated using javascript so using requests to retrieve the HTML will just retrieve the static contents, you can combine BeautifulSoup with something like Selenium to achieve what you want:
Install selenium:
pip install selenium
And then retrieve the contents using the Firefox engine or any other that supports javascript:
from bs4 import BeautifulSoup
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://www.mcfit.com/de/fitnessstudios/studiosuche/studiodetails/studio/berlin-lichtenberg/')
html_content = driver.page_source
soup = BeautifulSoup(html_content, "lxml")
elems = soup.find_all('div', {'class': 'sc-fzoXWK hnKkAN'})
print(elems)
If you use Firefox, the geckodriver needs to be accessible by your script, you can download it from https://github.com/mozilla/geckodriver/releases and put it in your PATH (or c:/windows if you are using this OS) so it is available from everywhere.
This is the website I'm trying to scrape with Python:
https://www.ebay.de/sch/i.html?_from=R40&_nkw=iphone+8&_sacat=0&LH_Sold=1&LH_Complete=1&rt=nc&LH_ItemCondition=3000
I want to access the 'ul' element with the class of 'srp-results srp-list clearfix'. This is what I tried with requests and BeautifulSoup:
from bs4 import BeautifulSoup
import requests
url = 'https://www.ebay.de/sch/i.html?_from=R40&_nkw=iphone+8&_sacat=0&LH_Sold=1&LH_Complete=1&rt=nc&LH_ItemCondition=3000'
r = requests.get(url)
soup = BeautifulSoup(r.text, 'html.parser')
uls = soup.find_all('ul', attrs = {'class': 'srp-results srp-list clearfix'})
And the output is always an empty string.
I also tried scraping the website with Selenium Webdriver and I got the same result.
First I was a little bit confused about your error but after a bit of debugging I figured out that: eBay dynamically generates that ul with JavaScript
So since you can't execute JavaScript with BeautifulSoup you have to use selenium and wait until the JavaScript loads that ul
It is probably because the content you are looking for is rendered by JavaScript After the page loads on a web browser this means that the web browser load that content after running javascript which you cannot get with requests.get request from python.
I would suggest to learn Selenium to Scrape the data you want
I have been digging on the site for some time and im unable to find the solution to my issue. Im fairly new to web scraping and trying to simply extract some links from a web page using beautiful soup.
url = "https://www.sofascore.com/pt/futebol/2018-09-18"
page = urlopen(url).read()
soup = BeautifulSoup(page, "lxml")
print(soup)
At the most basic level, all im trying to do is access a specific tag within the website. I can work out the rest for myself, but the part im struggling with is the fact that a tag that I am looking for is not in the output.
For example: using the built in find() I can grab the following div class tag:
class="l__grid js-page-layout"
However what i'm actually looking for are the contents of a tag that is embedded at a lower level in the tree.
js-event-list-tournament-events
When I perform the same find operation on the lower-level tag, I get no results.
Using Azure-based Jupyter Notebook, i have tried a number of the solutions to similar problems on stackoverflow and no luck.
Thanks!
Kenny
The page use JS to load the data dynamically so you have to use selenium. Check below code.
Note you have to install selenium and chromedrive (unzip the file and copy into python folder)
import time
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url = "https://www.sofascore.com/pt/futebol/2018-09-18"
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome(chrome_options=options)
driver.get(url)
time.sleep(3)
page = driver.page_source
driver.quit()
soup = BeautifulSoup(page, 'html.parser')
container = soup.find_all('div', attrs={
'class':'js-event-list-tournament-events'})
print(container)
or you can use their json api
import requests
url = 'https://www.sofascore.com/football//2018-09-18/json'
r = requests.get(url)
print(r.json())
I had the same problem and the following code worked for me. Chromedriver must be installed!
import time
from bs4 import BeautifulSoup
from selenium import webdriver
chromedriver_path= "/Users/.../chromedriver"
driver = webdriver.Chrome(chromedriver_path)
url = "https://yourURL.com"
driver.get(url)
time.sleep(3) #if you want to wait 3 seconds for the page to load
page_source = driver.page_source
soup = bs4.BeautifulSoup(page_source, 'lxml')
This soup you can use as usual.
I want to load the list of images in this page in Python. However, when I opened the page in my browser (Chrome or Safari) and opened the dev tools, the inspector returned the list of images as <img class="grid-item--image">....
However, when I tried to parse it in Python, the result seemed different. Specifically, I got the list of images as <img class="carousel--image"...>, whereas the soup.findAll("img", "grid-item--image") did return an empty list. Also, I tried saving those images using its srcset tag, most of the saved images are NOT those that were listed on the web.
I think the web page used some sort of technics when rendering. How can I parse the web pages successfully?
I used BeautifulSoup 4 on Python 3.5. I loaded the page as follows:
import requests
from bs4 import BeautifulSoup
html = requests.get(url).text
soup = BeautifulSoup(html, "html.parser", from_encoding="utf-8")
return soup
You would do better to use something like selenium for this as follows:
from bs4 import BeautifulSoup
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("http://www.vogue.com/fashion-shows/fall-2016-menswear/fendi#collection")
html_source = browser.page_source
soup = BeautifulSoup(html_source, "html.parser")
for item in soup.find_all("img", {"class":"grid-item--image"}):
print(item.get('srcset'))
This would display the following kind of output:
http://assets.vogue.com/photos/569d37e434324c316bd70f04/master/w_195/_FEN0016.jpg
http://assets.vogue.com/photos/569d37e5d928983d20a78e4f/master/w_195/_FEN0027.jpg
http://assets.vogue.com/photos/569d37e834324c316bd70f0a/master/w_195/_FEN0041.jpg
http://assets.vogue.com/photos/569d37e334324c316bd70efe/master/w_195/_FEN0049.jpg
http://assets.vogue.com/photos/569d37e702e08d8957a11e32/master/w_195/_FEN0059.jpg
...
...
...
http://assets.vogue.com/photos/569d3836486d6d3e20ae9625/master/w_195/_FEN0616.jpg
http://assets.vogue.com/photos/569d381834324c316bd70f3b/master/w_195/_FEN0634.jpg
http://assets.vogue.com/photos/569d3829fa6d6c9057f91d2a/master/w_195/_FEN0649.jpg
http://assets.vogue.com/photos/569d382234324c316bd70f41/master/w_195/_FEN0663.jpg
http://assets.vogue.com/photos/569d382b7dcd2a8a57748d05/master/w_195/_FEN0678.jpg
http://assets.vogue.com/photos/569d381334324c316bd70f2f/master/w_195/_FEN0690.jpg
http://assets.vogue.com/photos/569d382dd928983d20a78eb1/master/w_195/_FEN0846.jpg
This allows the full rendering of the page to take place inside the browser, and the resulting HTML can then be obtained.