I would like to scrape news from yahoo's finance, for a pair.
How does bs4's find() or find_all() work?
for this example:
with this link:
I'm traying to extract the data ... but no data is scraped. why? what's wrong?
I'm using this, but nothing is printed (except the tickers)
html = BeautifulSoup(source_s, "html.parser") # "html")
news_table_s = html.find_all("div",{"class":"Py(14px) Pos(r)"})
news_tables_s[ticker_s] = news_table_s
print("news_tables", news_tables_s)
I would like to extract the headers from a yahoo finance web page.
You have to iterate your ResultSet to get anything out.
for e in html.find_all("div",{"class":"Py(14px) Pos(r)"}):
print(e.h3.text)
Recommendation - Do not use dynamic classes to select elements use more static ids or HTML structure, here selected via css selector
for e in html.select(' div:has(>h3>a)'):
print(e.h3.text)
Example
from bs4 import BeautifulSoup
import requests
url='https://finance.yahoo.com/quote/EURUSD%3DX?p=EURUSD%3DX'
html = BeautifulSoup(requests.get(url).text)
for e in html.select(' div:has(>h3>a)'):
print(e.h3.text)
Output
EUR/USD steadies, but bears sharpen claws as dollar feasts on Fed bets
EUR/USD Weekly Forecast – Euro Gives Up Early Gains for the Week
EUR/USD Forecast – Euro Plunges Yet Again on Friday
EUR/USD Forecast – Euro Gives Up Early Gains
EUR/USD Forecast – Euro Continues to Test the Same Indicator
Dollar gains as inflation remains sticky; sterling retreats
Siemens Issues Blockchain Based Euro-Denominated Bond on Polygon Blockchain
EUR/USD Forecast – Euro Rallies
FOREX-Dollar slips with inflation in focus; euro, sterling up on jobs data
FOREX-Jobs figures send euro, sterling higher; dollar slips before CPI
Question #1: (use while loop only)
An auto repair shop in Miami estimates that, your vehicle will cost $1689 to repair. Assuming that you could not afford to come up with such amount up front, the repair shop offers to let you pay the total repair cost ($1689) by installment, but at an annual interest rate of 3.00% compounded monthly.
Using a while loop,
Display the remaining balance after each month until pay off
Example/Hint: Balance will be $1,493.22 after 1 month.
Using the output from item 1 above, determine how many months will the vehicle be three quarter
(¾) paid off? Use a print statement to display the message.
Example: "Repair cost total will be 3/4 paid off in 2 months"
I am struggling to figure out how to attack this problem. I'm not sure how to calculate how much each monthly payment will be. There is a hint within the question that in one month, your repair cost goes from $1689 to $1493.22, implying that for the first month, you paid off $195.78 of the repair cost. Any help on how to even start this problem would be much appreciated, thank you.
Your first step is to calculate how much the client is paying per month and with that information apply the "compound interest with monthly withdrawals" formula.
To calculate that, you first calculate how much the balance will be after one month with interest:
monthly_interest_rate = annual_interest_rate / 12
balance += (balance * monthly_interest_rate)
Now subtract the balance after one month gave as a tip from the balance calculated above:
balance_after_one_month = 1493.22
paid_monthly_by_customer = balance - balance_after_one_month
I have this title, where the variable option indicates the regional level.
option=3
year=2017
ax.set_title('Household Gas Consumption, Water Heating,NUTS %.i'%option,fontdict={'fontsize':'10','fontweight':'3'})
How could I also add the variable year so the title looks like this?
Household Gas Consumption, Water Heating, NUTS 3, Year: 2017
I tried different ways but I could't make it work. Thanks in advance.
'Household Gas Consumption, Water Heating,NUTS %i Year: %i' %(option , year)
I have the following line of code
print("Income $ {:>8,d} pays Flat tax ${:>9.2f} and Current tax ${:>9.2f}".format(income,tax,curr_tax))
It gives me the output
Income $ 10,000 pays Flat tax $ 0.00 and Current tax $ 0.00
But I am trying to join the $ sign with the numbers without any spaces in between. How do we do this with string.format()
Edit:
This is the output I am trying to achieve
Income $50,000 pays Flat tax $780.00 and Current tax $0.00
Income $10,00 pays Flat tax $0.00 and Current tax $0.00
You could format within your format
print("Income {:>8} pays Flat tax {:>9} and Current tax {:>9}".format('${:,d}'.format(income), '${:.2f}'.format(tax), '${:.2f}'.format(curr_tax)))
Which gives
Income $10,000 pays Flat tax $0.00 and Current tax $0.00
Income $50,000 pays Flat tax $780.00 and Current tax $0.00
#aydow has a solid answer.
Alternatively, use #student comment by changing > to <:
print("Income ${:<8,d} pays Flat tax ${:<9.2f} and Current tax ${:<9.2f}".format(10000,0.00,0.00))
Results in:
Income $10,000 pays Flat tax $0.00 and Current tax $0.00
Still leaves blanks after values but not sure if that bugs you.
I tried to download the dataset from http://www.msmarco.org/dataset.aspx. Glad to see that the dataset is in JSON.
Now, if you open the dev dataset, you can see that its constituent elements are like:
{"passages": [{"is_selected": 1, "url": "http://www.indeed.com/cmp/Walgreens/salaries", "passage_text": "The average Walgreens salary ranges from approximately $15,000 per year for Customer Service Associate / Cashier to $179,900 per year for District Manager. Average Walgreens hourly pay ranges from approximately $7.35 per hour for Laboratory Technician to $68.90 per hour for Pharmacy Manager. Salary information comes from 7,810 data points collected directly from employees, users, and jobs on Indeed."}, {"is_selected": 0, "url": "http://www.answers.com/Q/What_is_the_average_gross_sales_volume_of_a_single_Walgreen's_Store", "passage_text": "The average revenue in 2011 of a Starbuck Store was $1,078,000, up from $1,011,000 in 2010. The average ticket (total purchase) at domestic Starbuck stores in No … vember 2007 was reported at $6.36. In 2008, the average ticket was flat (0.0% change)."}, {"is_selected": 0, "url": "http://news.walgreens.com/fact-sheets/frequently-asked-questions.htm", "passage_text": "In fiscal 2014, Walgreens opened a total of 184 new locations and acquired 84 locations, for a net decrease of 273 after relocations and closings. How big are your stores? The average size for a typical Walgreens is about 14,500 square feet and the sales floor averages about 11,000 square feet. How do we select locations for new stores? There are several factors that Walgreens takes into account, such as major intersections, traffic patterns, demographics and locations near hospitals."}, {"is_selected": 0, "url": "http://www.babson.edu/executive-education/thought-leadership/retailing/Documents/walgreens-strategic-evolution.pdf", "passage_text": "th store in 1984, reaching $4 billion in sales in 1987, and $5 billion two years later. Walgreens ended the 1980s with 1,484 stores, $5.3 billion in revenues and $154 million in profits. However, profit margins remained just below 3 percent of sales, and returns on assets of less than 10 percent."}, {"is_selected": 0, "url": "http://www.trefis.com/stock/wag/articles/199532/key-trends-driving-walgreens-business/2013-08-07", "passage_text": "The number of Walgreen stores has risen from 5,000 in 2005 to more than 8,000 at present. The average square footage per store stood at approximately 10,200 and we forecast the figure to remain constant over our review period. Walgreen earned $303 as average front-end revenue per store square foot in 2012."}, {"is_selected": 0, "url": "http://www.walgreens.com/storelocator/find.jsp?requestType=locator", "passage_text": "Your Walgreens Store. Select a store from the search results to make it Your Walgreens Store and save time getting what you need. Your Walgreens Store will be the default location for picking up prescriptions, photos, in store orders and finding deals in the Weekly Ad."}], "query_id": 9652, "answers": ["Approximately $15,000 per year."], "query_type": "numeric", "query": "walgreens store sales average"}
{"passages": [{"is_selected": 0, "url": "http://www.breakintobartending.com/how-much-do-bartenders-make/", "passage_text": "A bartender’s income is comprised mostly of tips– 55% to be exact. In some states, employers aren’t even required to pay their bartenders the minimum wage and can pay as low as $2.13 per hour, and they depend on their tips almost entirely. Bartending can be a lot of things. For some it is exciting, for others exhausting. At times there is a lot of fun to be had, at others it is rather dull. But for the most part, bartending is almost always rewarding in the financial sense, as long as you stick with it."}, {"is_selected": 1, "url": "http://www.breakintobartending.com/how-much-do-bartenders-make/", "passage_text": "According to the Bureau of Labor Statistics, the average hourly wage for a bartender is $10.36, and the average yearly take-home is $21,550. Bartending can be a lot of things. For some it is exciting, for others exhausting. At times there is a lot of fun to be had, at others it is rather dull. But for the most part, bartending is almost always rewarding in the financial sense, as long as you stick with it."}, {"is_selected": 0, "url": "http://careerswiki.com/how-much-do-bartenders-make/", "passage_text": "About 551,100 individuals are employed as bartenders, with half of this number working part-time. The average annual salary for bartenders is $19,050 or an equivalent of $9.16 per hour, including tips. No formal training is needed for one to get a job as a bartender as all it takes are good customer service skills and a comprehensive knowledge about beverages and recipes. "}, {"is_selected": 0, "url": "http://www.answers.com/Q/How_much_do_bartenders_in_Las_Vegas_make", "passage_text": "Confidence votes 11. Bartenders in Vegas can make up to $7-$15 wage plus $10-$50 in tips per hour. But that totally depends on personality, gender and outlet. A hot young female can make $1000 a day tending bar in a strip club, but that is very exceptional. A bartender makes as much as a bartender wants to make. I wouldn't say there is a cap on how much a bartender make in one year. If the service is good and the coversation is i … nteresting a bartender can make alot of money. Location is also a big factor."}, {"is_selected": 0, "url": "https://answers.yahoo.com/question/index?qid=20100102233548AAZEutT", "passage_text": "Best Answer: An average bartender makes about...2 to 3 dollars an hour. but all the money is made off tips depending on the popularity of the bar. I used to make $700 to $1000 a night. but that is in Atlanta. If the bar is busy and you are a good bartender you will make quite a bit. I dont know how much, because I live in a town with a population of 2000 so there is not alot going on around here. Im sure the bartenders make a hundred to two hundred a night total (on a good night)...just depends."}, {"is_selected": 0, "url": "https://answers.yahoo.com/question/index?qid=20100102233548AAZEutT", "passage_text": "Report Abuse. no way to tell you how much bartenders make. in wages anything from minimum to $15 an hour. tips, anywhere from $20 to $300 or more a night. depends on a lot of things. those top dollar jobs only come after a lot of experience. If the bar is busy and you are a good bartender you will make quite a bit. I dont know how much, because I live in a town with a population of 2000 so there is not alot going on around here. Im sure the bartenders make a hundred to two hundred a night total (on a good night)...just depends."}, {"is_selected": 0, "url": "http://work.chron.com/much-bartender-make-annually-7503.html", "passage_text": "Pay by Employment Setting. Bartenders who worked in hotels and hotel restaurants generally reported the highest incomes in 2011, an average of $26,180 a year. Bartenders employed in full-service restaurants tended to earn somewhat less, averaging about $22,130 a year. Bartenders employed by bars earned an average of $20,230 per year, and bartenders who worked for civic and social organizations earned an average of $18,970 a year. The median earnings of bartenders during this period were $9.06 an hour and $18,850 a year. Eighty percent of bartenders in the U.S. reported annual incomes of between $16,170 and $31,860."}, {"is_selected": 0, "url": "http://www.celebritynetworth.com/articles/how-much-does/how-much-does-bartender-make/", "passage_text": "Tips make up half or more of bartender's salaries. If a bartender earned $6.00 an hour, their tips generally average out to $12.00 to $18.00 an hour as additional income. A bartender in an average bar will typically earn $15.00 $30.00 an hour between their wages and tips. According to bartending.org, bartenders in a high volume resort or establishment can earn $50,000 to $75,000 per year between hourly wages and tips. Indeed.com 2010 results show bartenders in restaurants at median salary rates can make a good salary per year: Bartender $73,000."}], "query_id": 9653, "answers": ["$21,550 per year", "The average hourly wage for a bartender is $10.36 and the average yearly take-home is $21,550."], "query_type": "numeric", "query": "how much do bartenders make"}
Now these blocks repeat without any enclosing {} and with no comma separating them. Clearly this can't be JSON.
My question is:
Is this really json as the dataset claims to be?
Any library (preferably in python) to parse it to extract all the individual fields.
I tried ijson in python but then this really doesn't look like real json.
This file is made of one JSON document per line: this is JSON Lines.
The file as a whole is not a valid JSON document. You should read line by line and feed each line into a JSON parser:
#!/usr/bin/env python3
from sys import argv
import json
with open(argv[1], "r", encoding="UTF-8") as f:
lines = [ json.loads(line) for line in f ]
Note: Another similar format is JSON-seq documented in RFC7464.