In my HTML code, I have 8 tables. Here is how I am trying to get them:
url ="http://www.uefa.com/worldcup/season=2014/standings/"
r = requests.get(url)
soup = BeautifulSoup(r.content)
table = soup.find("table")
This piece of code gives me only the first one table. Now the action takes place:
rows = table.findAll('tr')
data = [[td.text.strip().encode("utf-8") for td in tr.findAll("td")] for tr in rows]
head = [[th.text.strip().encode("utf-8") for th in tr.findAll("th")] for tr in rows]
for i in data:
if i:
flag = i[1][:3] + ".png"
i.insert(1, Datas(i, "http://img.uefa.com/imgml/flags/18x18/" + flag))
return render(request, 'Titles.html', {"data": data})
After action ends, I'm wondering how to start extracting second table with the same variables? Of course I could add more variables like data1, data2, head1, head2 and so on. However I don't want to repeat myself.
So maybe you can help me and find the better way to do it?
You probably want something like this:
import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get(url).text)
tables = soup.find_all('table') # this returns 8 tables
print len(tables)
for table in tables:
for tr in table.find_all('tr'):
print tr.text
print
If you try that code, you can see you get the content out of each table,
Related
I am new to data scraping with Beautiful Soup. I would like to get data from pro-football-reference on these stats: https://www.pro-football-reference.com/boxscores/201009090nor.htm#all_pbp
I would like to iterate through every row under the 'Detail Column' under the Full Play-By-Play Table so that if the Detail contains the word "Penalty" I can save that. Any chance anyone knows how I could possibly do this? This table seems different than others.
# Any example of how I extracted another element (Referee Name)
# from the same page but different table
table = soup.select_one('#all_officials').find_next(text=lambda t: isinstance(t, Comment))
table = BeautifulSoup(table, 'html.parser')
for tr in table.select('tr'):
tds = [td.get_text(strip=True) for td in tr.select('td')]
if str(*tds) != "Officials":
referee = str(*tds)
break
The table is commented out. A common and reliable way is to import Comment and handle with for comment in soup.find_all(text=lambda text: isinstance(text, Comment)) as shown here: https://stackoverflow.com/a/60381103.
For this particular instance, I am just removing the comments strings through substitution.
Then I use :-soup-contains to target the appropriate rows, filtering on only those rows within the table where the text Penalty appears in the elements with data-stat attribute having value = detail i.e. the detail column.
I then use pandas to reconstitute the table from the filtered trs html joined and then book-ended by table tags
from bs4 import BeautifulSoup as bs
import pandas as pd
import requests
import re
r = requests.get('https://www.pro-football-reference.com/boxscores/201009090nor.htm#all_pbp',
headers={'User-Agent': 'Mozilla/5.0'})
s = re.sub(r'<!--|-->', '', r.text)
soup = bs(s, 'lxml')
s2 = '<table>' + ''.join([str(i) for i in soup.select(
'#pbp tr:has([data-stat=detail]:-soup-contains("Penalty"))')]) + '</table>'
df = pd.read_html(s2)[0]
df.columns = [i.text for i in soup.select('#pbp thead > tr > th')]
df
I am new to web scraping and I'm trying to scrape the "statistics" page of yahoo finance for AAPL. Here's the link: https://finance.yahoo.com/quote/AAPL/key-statistics?p=AAPL
Here is the code I have so far...
from bs4 import BeautifulSoup
from requests import get
url = 'https://finance.yahoo.com/quote/AAPL/key-statistics?p=AAPL'
response = get(url)
soup = BeautifulSoup(response.text, 'html.parser')
stock_data = soup.find_all("table")
for stock in stock_data:
print(stock.text)
When I run that, I return all of the table data on the page. However, I only want specific data from each table (e.g. "Market Cap", "Revenue", "Beta").
I tried messing around with the code by doing print(stock[1].text) to see if I could limit the amount of data returned to just the second value in each table but that returned an error message. Am I on the right track by using BeautifulSoup or do I need to use a completely different library? What would I have to do in order to only return particular data and not all of the table data on the page?
Examining the HTML-code gives you the best idea of how BeautifulSoup will handle what it sees.
The web page seems to contain several tables, which in turn contain the information you are after. The tables follow a certain logic.
First scrape all the tables on the web page, then find all the table rows (<tr>) and the table data (<td>) that those rows contain.
Below is one way of achieving this. I even threw in a function to print only a specific measurement.
from bs4 import BeautifulSoup
from requests import get
url = 'https://finance.yahoo.com/quote/AAPL/key-statistics?p=AAPL'
response = get(url)
soup = BeautifulSoup(response.text, 'html.parser')
stock_data = soup.find_all("table")
# stock_data will contain multiple tables, next we examine each table one by one
for table in stock_data:
# Scrape all table rows into variable trs
trs = table.find_all('tr')
for tr in trs:
# Scrape all table data tags into variable tds
tds = tr.find_all('td')
# Index 0 of tds will contain the measurement
print("Measure: {}".format(tds[0].get_text()))
# Index 1 of tds will contain the value
print("Value: {}".format(tds[1].get_text()))
print("")
def get_measurement(table_array, measurement):
for table in table_array:
trs = table.find_all('tr')
for tr in trs:
tds = tr.find_all('td')
if measurement.lower() in tds[0].get_text().lower():
return(tds[1].get_text())
# print only one measurement, e.g. operating cash flow
print(get_measurement(stock_data, "operating cash flow"))
Although this isn't Yahoo Finance, you can do something very similar like this...
import requests
from bs4 import BeautifulSoup
base_url = 'https://finviz.com/screener.ashx?v=152&o=price&t=MSFT,AAPL,SBUX,S,GOOG&o=price&c=0,1,2,3,4,5,6,7,8,9,25,63,64,65,66,67'
html = requests.get(base_url)
soup = BeautifulSoup(html.content, "html.parser")
main_div = soup.find('div', attrs = {'id':'screener-content'})
light_rows = main_div.find_all('tr', class_="table-light-row-cp")
dark_rows = main_div.find_all('tr', class_="table-dark-row-cp")
data = []
for rows_set in (light_rows, dark_rows):
for row in rows_set:
row_data = []
for cell in row.find_all('td'):
val = cell.a.get_text()
row_data.append(val)
data.append(row_data)
# sort rows to maintain original order
data.sort(key=lambda x: int(x[0]))
import pandas
pandas.DataFrame(data).to_csv("C:\\your_path\\AAA.csv", header=False)
This is a nice substitute in case Yahoo decided to depreciate more of the functionality of their API. I know they cut out a lot of things (mostly historical quotes) a couple years ago. It was sad to see that go away.
Hi I am trying to use Python and Beautiful Soup to scrape a webpage. There are various tables in the webpage with results that I want out of them, but I am struggling to:
1) find the right table
2) find the right two cells
3) write the cells 1 and 2 into a dictionary key and value, respectively.
So far, after making a request, and parsing the HTML, I use:
URL='someurl.com'
def datascrape(url):
page=requests.get(url)
print ("requesting page")
soup = BeautifulSoup(page.content, "html.parser")
return(soup)
soup=datascrape(URL)
results = {}
for row in soup.findAll('tr'):
aux = row.findAll('td')
try:
if "Status" in (aux.stripped_strings):
key=(aux[0].strings)
value=(aux[1].string)
results[key] = value
except:
pass
print (results)
Unfortunately "results" is always empty. I am really not sure where I am going wrong. Could anyone enlighten me please?
I'm not sure why you're using findAll() instead of find_all() as I'm fairly new to web-scraping, but nevertheless I think this gives you the output you're looking for.
URL='http://sitem.herts.ac.uk/aeru/bpdb/Reports/2070.html'
def datascrape(url):
page=requests.get(url)
print ("requesting page")
soup = BeautifulSoup(page.content,
"html.parser")
return(soup)
soup=datascrape(URL)
results = {}
table_rows = soup.find_all('tr')
for tr in table_rows:
td = tr.find_all('td')
row = [i.text for i in td]
try:
for i in row:
if "Status" in i:
key=(row[0].strip())
value=(row[1].strip())
results[key] = value
else:
pass
print(results)
Hope this helps!
If just after the Status and Not Applicable you can use positional nth-of-type css selectors. This does depend on position being the same across pages.
import requests
from bs4 import BeautifulSoup
url ='https://sitem.herts.ac.uk/aeru/bpdb/Reports/2070.htm'
page=requests.get(url)
soup = BeautifulSoup(page.content, "lxml")
tdCells = [item.text.strip() for item in soup.select('table:nth-of-type(2) tr:nth-of-type(1) td')]
results = {tdCells[0] : tdCells[1]}
print(results)
I am currently running the following python script:
import requests
from bs4 import BeautifulSoup
origin= ["USD","GBP","EUR"]
i=0
while i < len(origin):
page = requests.get("https://www.x-rates.com/table/?from="+origin[i]+"&amount=1")
soup = BeautifulSoup(page.content, "html.parser")
tables = soup.findChildren('table')
my_table = tables[0]
rows = my_table.findChildren(['td'])
i = i +1
for rows in rows:
cells = rows.findChildren('a')
for cell in cells:
value = cell.string
print(value)
To scrape data from this HTML:
https://i.stack.imgur.com/DkX83.png
The problem I have is that I'm struggling to only scrape the first column without scraping the second one as well because they are both under tags and in the same table row as each other. The href is the only thing which differentiates between the two tags and I have tried filtering using this but it doesn't seem to work and returns a blank value. Also when i try to sort the data manually the output is amended vertically and not horizontally, I am new to coding so any help would be appreciated :)
There is another way you might wanna try as well to achieve the same:
import requests
from bs4 import BeautifulSoup
keywords = ["USD","GBP","EUR"]
for keyword in keywords:
page = requests.get("https://www.x-rates.com/table/?from={}&amount=1".format(keyword))
soup = BeautifulSoup(page.content, "html.parser")
for items in soup.select_one(".ratesTable tbody").find_all("tr"):
data = [item.text for item in items.find_all("td")[1:2]]
print(data)
It is easier to follow what happens when you print every item you got from the top e.g. in this case from table item. The idea is to go one by one so you can follow.
import requests
from bs4 import BeautifulSoup
origin= ["USD","GBP","EUR"]
i=0
while i < len(origin):
page = requests.get("https://www.x-rates.com/table/?from="+origin[i]+"&amount=1")
soup = BeautifulSoup(page.content, "html.parser")
tables = soup.findChildren('table')
my_table = tables[0]
i = i +1
rows = my_table.findChildren('tr')
for row in rows:
cells = row.findAll('td',class_='rtRates')
if len(cells) > 0:
first_item = cells[0].find('a')
value = first_item.string
print(value)
I was trying to crawl the link : "http://codeforces.com/contest/554/standings" .
I used the given two lines to read all contestant names :
table1 = soup.find("table", {'class':'standings'})
table2 = table1.find_all("tr")
However table2 doesn't print all the table rows.
I found " <--suppress HtmlUnknownAttribute --> " written before all the rows I wasn't able to crawl.
Is there any particular reason for it.
I am just a beginner to web crawling
You may need to share the code in entirety. I get the expected 100 contestant names based on your initial "tr" find_all:
import urllib2
from bs4 import BeautifulSoup
response = urllib2.urlopen('http://codeforces.com/contest/554/standings')
html = response.read()
soup = BeautifulSoup(html, 'html.parser')
table = soup.find('table', {'class': 'standings'})
rows = table.find_all('tr')
for row in rows:
contestant = row.find_all('td', {'class': 'contestant-cell'})
if len(contestant) > 0:
# Quick'n dirty dig. Makes un-safe assumptions about the HTML structure.
print contestant[0].a.string
You'll note that some additional digging is required after you get the table rows since not every row contains contestant info.