We Keep Coding

Web scrape to obtain table data from guru focus site

I want to scrape specific data from guru focus website.
https://www.gurufocus.com/stock/AAHTF/summary?search=AAPICO
Currently i am fetching number value. For example:financial strength value is "4" out of 10. Now i want to fetch sub components data as well.
code to fetch only number value:
for name in names:
start_time = time.time()
# getting the symbol
URL = f'https://www.gurufocus.com/search?s={name}'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(URL)
html_source = driver.page_source
driver.close()
soup = BeautifulSoup(html_source, 'html.parser')
headers = soup.find_all("span")
# saving only the first link
for i, head in enumerate(headers):
try:
h = head.find("a").get("href")
link = "https://www.gurufocus.com" + h
break
except:
pass
try:
# loading the link page
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(link)
html_source = driver.page_source
driver.close()
soup = BeautifulSoup(html_source, 'html.parser')
headers = soup.find_all("span", class_="t-default bold")
ratings = []
for head in headers:
ratings.append(int(head.get_text()))
if len(ratings) == 0:
continue
ratings_dict = {"Financial Strength": ratings[0],
"Growth Rank" : ratings[1],
"Momentum Rank" : ratings[2],
"Profitability Rank": ratings[3],
"GF Value Rank" : ratings[4],
}
print(ratings_dict)
# ratings_dict = json.loads(ratings_dict)
with open(f"output/gurufocus/{name}.json", 'w') as f:
json.dump(str(ratings_dict), f)
end_time = time.time()
print("time taken for %s is: %.2f" %(name, (end_time-start_time)))
except:
print("no data found")
output:
"{'Financial Strength': 6, 'Growth Rank': 4, 'Momentum Rank': 4, 'Profitability Rank': 7, 'GF Value Rank': 5}"
Expection:
I want to fetch full table data( below image) along with rank into data frame.
How do I need to change my code to obtain the other specific data?

You can use Pandas to write a clean solution for this problem:
import pandas as pd
import requests
import json
from bs4 import BeautifulSoup
from collections import ChainMap
tables = pd.read_html(
requests.get(
'https://www.gurufocus.com/stock/AAHTF/summary?search=AAPICO'
).text,
header=0
)
sub_table_values = [[{record["Name"]: record["Current"]} for record in json.loads(e)] for e in [i.to_json(orient="records") for i in tables]]
sub_formatted = [dict(ChainMap(*a)) for a in sub_table_values]
print(json.dumps(sub_formatted, indent=4))
Description:
First, I obtain all the tables and convert those to DataFrames(using pandas).
Then, I convert the dataframes to json and only extract the desired result(Name and Curent)
Format the data.
It would return:
[
{
"WACC vs ROIC": null,
"Beneish M-Score": "-2.28",
"Altman Z-Score": "1.98",
"Piotroski F-Score": "7/9",
"Interest Coverage": "4.68",
"Debt-to-EBITDA": "2.55",
"Debt-to-Equity": "0.85",
"Equity-to-Asset": "0.37",
"Cash-To-Debt": "0.1"
},
{
"Future 3-5Y Total Revenue Growth Rate": 13.71,
"3-Year Book Growth Rate": 2.8,
"3-Year FCF Growth Rate": 49.9,
"3-Year EPS without NRI Growth Rate": -5.2,
"3-Year EBITDA Growth Rate": 9.0,
"3-Year Revenue Growth Rate": 9.6
}...
]
However, this is solution works because the web is structured with tables. For complex/irregular websites I prefer to use scrapy as we use in my job.

Web Scraping with Python BS

Trying to scrape some weather data off of Weather Underground. I haven't had any difficulty getting the data of interest until I came to getting the day/date, hi/lo temps, and forecast (ie. "Partly Cloudy"). Each is in a div without a class. The parent, of each, is a div with a class="obs-date" (see image below)
[WxUn HTML image][1]
Attempted code below with other options commented out. Each returns an empty list.
def get_wx(city, state):
city=city.lower()
state=state.lower()
# get current conditions; 'weather' in url
current_dict = get_current(city, state)
# get forecast; 'forecast' in url
f_url = f'https://www.wunderground.com/forecast/us/{state}/{city}'
f_response = req.get(f_url)
f_soup = BeautifulSoup(f_response.text, 'html.parser')
cast_dates = f_soup.find_all('div', class_="obs-date")
# cast_dates = f_soup.find_all('div', attrs={"class":"obs-date"})
# cast_dates = f_soup.select('div.obs-date')
print(cast_dates)
get_wx("Portland", "ME")
Any help with what I'm missing is appreciated.

As far as I can see the whole block you're trying to parse is driven by javascript, that's why you're getting empty results using beautifulsoup
The ADDITIONAL CONDITIONS part could be parsed completely using bs4 as well as everything below. Table at the end could be parsed using pandas.
To scrape javascript content, you can use requests-html or selenium libraries.
from requests_html import HTMLSession
import json
session = HTMLSession()
url = "https://www.wunderground.com/weather/us/me/portland"
response = session.get(url)
response.html.render(sleep=1)
data = []
current_date = response.html.find('.timestamp strong', first = True).text
weather_conditions = response.html.find('.condition-icon p', first = True).text
gusts = response.html.find('.medium-uncentered span', first = True).text
current_temp = response.html.find('.current-temp .is-degree-visible', first = True).text
data.append({
"Last update": current_date,
"Current weather": weather_conditions,
"Temperature": current_temp,
"Gusts": gusts,
})
print(json.dumps(data, indent = 2, ensure_ascii = False))
Output:
[
{
"Last update": "1:27 PM EDT on April 14, 2021",
"Current weather": "Fair",
"Temperature": "49 F",
"Gusts": "13 mph"
}
]

Scrape Text After Specific Text and Before Specific Text

<script type="text/javascript">
'sku': 'T3246B5',
'Name': 'TAS BLACKY',
'Price': '111930',
'categories': 'Tas,Wanita,Sling Bags,Di bawah Rp 200.000',
'brand': '',
'visibility': '4',
'instock': "1",
'stock': "73.0000"
</script>
I want to scrape the text between : 'stock': " and .0000" so the desireable result is 73
What I used to know is to do something like this:
for url2 in urls2:
req2 = Request(url2, headers={'User-Agent': 'Chrome/39.0.2171.95'})
html2 = uReq(req2).read()
page_soup2 = soup(html2, "html.parser")
# Grab text
stock = page_soup2.findAll("p", {"class": "stock"})
stocks = stock[0].text
I used something like this in my previous code, It works before the web change their code.
But now there is more than 1 ("script", {"type": "text/javascript"}) in the entire page I want to scrape. So I dont know how to find the right ("script", {"type": "text/javascript"})
I also don't know hot to get the specific text before and after the text.
I have googled it all this day but can't find the solution. Please help.
I found that strings = 'stock': " and .0000" is unique in the entire page, only 1 'stock': and only 1 .0000"
So I think it could be the sign of location where I want to scrape the text.
Please help, thank you for your kindness.
I also apologize for my lack of English, and I am actually unfamiliar with programming. I'm just trying to learn from Google, but I can't find the answer. Thank you for your understanding.
the url = view-source:sophieparis.com/blacky-bag.html

Since you are sure 'stock' only shows up in the script tag you want, you can pull out that text that contains 'stock. Once you have that, it's a matter of trimming off the excess, and change to double quotes to get it into a valid json format and then simply read that in using json.loads()
import requests
from bs4 import BeautifulSoup
import json
url2 = 'https://www.sophieparis.com/blacky-bag.html'
req2 = requests.get(url2, headers={'User-Agent': 'Chrome/39.0.2171.95'})
page_soup2 = BeautifulSoup(req2.text, "html.parser")
scripts = page_soup2.find_all('script')
for script in scripts:
if 'stock' in script.text:
jsonStr = script.text
jsonStr = jsonStr.split('productObject = ')[-1].strip()
jsonStr = jsonStr.rsplit('}',1)[0].strip() + '}'
jsonData = json.loads(jsonStr.replace("'",'"'))
print (jsonData['stock'].split('.')[0])
Output:
print (jsonData['stock'].split('.')[0])
71
You could also do this without the loop and just grab the script that has the string stock in it using 1 line:
jsonStr = page_soup2.find('script', text=re.compile(r'stock')).text
Full code would look something like:
import requests
from bs4 import BeautifulSoup
import json
import re
url2 = 'https://www.sophieparis.com/blacky-bag.html'
req2 = requests.get(url2, headers={'User-Agent': 'Chrome/39.0.2171.95'})
page_soup2 = BeautifulSoup(req2.text, "html.parser")
jsonStr = page_soup2.find('script', text=re.compile(r'stock')).text
jsonStr = jsonStr.split('productObject = ')[-1].strip()
jsonStr = jsonStr.rsplit('}',1)[0].strip() + '}'
jsonData = json.loads(jsonStr.replace("'",'"'))
print (jsonData['stock'].split('.')[0])

I would write a regex that targets the javascript dictionary variable that houses the values of interest. You can apply this direct to response.text with no need for bs4.
The dictionary variable is called productObject, and you want the non-empty dictionary which is the second occurrence of productObject = {..} i.e. not the one which has 'var ' preceeding it. You can use negative lookbehind to specify this requirement.
Use hjson to handle property names enclosed in single quotes.
Py
import requests, re, hjson
r = requests.get('https://www.sophieparis.com/blacky-bag.html')
p = re.compile(r'(?<!var\s)productObject = ([\s\S]*?})')
data = hjson.loads(p.findall(r.text)[0])
print(data)
Regex: try

If you want to provide me with the webpage you wish to scrape the data from, I'll see if I can fix the code to pull the information.

python loop requests.get() only returns first loop

Trying to scrape a table from multiple webpages and store in a list. The list prints out the results from the first webpage 3 times.
import pandas as pd
import requests
from bs4 import BeautifulSoup
dflist = []
for i in range(1,4):
s = requests.Session()
res = requests.get(r'http://www.ironman.com/triathlon/events/americas/ironman/world-championship/results.aspx?p=' + str(i) + 'race=worldchampionship&rd=20181013&agegroup=Pro&sex=M&y=2018&ps=20#axzz5VRWzxmt3')
soup = BeautifulSoup(res.content,'lxml')
table = soup.find_all('table')
dfs = pd.read_html(str(table))
dflist.append(dfs)
s.close()
print(dflist)

You left out the & after '?p=' + str(i), so your requests all have p set to ${NUMBER}race=worldchampionship, which ironman.com presumably can't make sense of and just ignores. Insert a & at the beginning of 'race=worldchampionship'.
To prevent this sort of mistake in the future, you can pass the URL's query parameters as a dict to the params keyword argument like so:
params = {
"p": i,
"race": "worldchampionship",
"rd": "20181013",
"agegroup": "Pro",
"sex": "M",
"y": "2018",
"ps": "20",
}
res = requests.get(r'http://www.ironman.com/triathlon/events/americas/ironman/world-championship/results.aspx#axzz5VRWzxmt3', params=params)

extract yahoo finance balance sheet with python

I am learning to use beautifulsoup and python to extract an html table. I tried using the following code to extract the balance sheet for Google. However, I can't seem to get all the rows scraped correctly.
I can't manage to omit rows that is just a spacer and I don't manage to extract rows of the Totals (eg. Total Asset).
Any advice? Advice on simplifying the code also valuable.
from bs4 import BeautifulSoup
import requests
def bs_extract(stock_ticker):
url= 'https://finance.yahoo.com/q/bs?s='+str(stock_ticker)+'&annual'
source_code = requests.get(url)
plain_text=source_code.text
soup = BeautifulSoup(plain_text)
c1= ""
c2= ""
c3= ""
c4= ""
c5= ""
table = soup.find("table", { "class" : "yfnc_tabledata1" })
# print (table)
for row in table.findAll("tr"):
cells = row.findAll("td")
if len(cells)==5:
c1=cells[0].find(text=True)
c2=cells[1].find(text=True)
c3=cells[2].find(text=True)
c4=cells[3].find(text=True)
c5=cells[4].find(text=True)
elif len(cells)==6:
c1=cells[1].find(text=True)
c2=cells[2].find(text=True)
c3=cells[3].find(text=True)
c4=cells[4].find(text=True)
c5=cells[5].find(text=True)
elif len(cells)==1:
c1=cells[0].find(text=True)
c2=""
c3=""
c4=""
c5=""
else:
pass
print(c1,c2,c3,c4,c5)
bs_extract('goog')

You might find it easier to get this data structured, through YQL. See http://goo.gl/qKeWXw