We Keep Coding

Is this even JSON? - python

I tried to download the dataset from http://www.msmarco.org/dataset.aspx. Glad to see that the dataset is in JSON.
Now, if you open the dev dataset, you can see that its constituent elements are like:
{"passages": [{"is_selected": 1, "url": "http://www.indeed.com/cmp/Walgreens/salaries", "passage_text": "The average Walgreens salary ranges from approximately $15,000 per year for Customer Service Associate / Cashier to $179,900 per year for District Manager. Average Walgreens hourly pay ranges from approximately $7.35 per hour for Laboratory Technician to $68.90 per hour for Pharmacy Manager. Salary information comes from 7,810 data points collected directly from employees, users, and jobs on Indeed."}, {"is_selected": 0, "url": "http://www.answers.com/Q/What_is_the_average_gross_sales_volume_of_a_single_Walgreen's_Store", "passage_text": "The average revenue in 2011 of a Starbuck Store was $1,078,000, up from $1,011,000 in 2010. The average ticket (total purchase) at domestic Starbuck stores in No … vember 2007 was reported at $6.36. In 2008, the average ticket was flat (0.0% change)."}, {"is_selected": 0, "url": "http://news.walgreens.com/fact-sheets/frequently-asked-questions.htm", "passage_text": "In fiscal 2014, Walgreens opened a total of 184 new locations and acquired 84 locations, for a net decrease of 273 after relocations and closings. How big are your stores? The average size for a typical Walgreens is about 14,500 square feet and the sales floor averages about 11,000 square feet. How do we select locations for new stores? There are several factors that Walgreens takes into account, such as major intersections, traffic patterns, demographics and locations near hospitals."}, {"is_selected": 0, "url": "http://www.babson.edu/executive-education/thought-leadership/retailing/Documents/walgreens-strategic-evolution.pdf", "passage_text": "th store in 1984, reaching $4 billion in sales in 1987, and $5 billion two years later. Walgreens ended the 1980s with 1,484 stores, $5.3 billion in revenues and $154 million in profits. However, profit margins remained just below 3 percent of sales, and returns on assets of less than 10 percent."}, {"is_selected": 0, "url": "http://www.trefis.com/stock/wag/articles/199532/key-trends-driving-walgreens-business/2013-08-07", "passage_text": "The number of Walgreen stores has risen from 5,000 in 2005 to more than 8,000 at present. The average square footage per store stood at approximately 10,200 and we forecast the figure to remain constant over our review period. Walgreen earned $303 as average front-end revenue per store square foot in 2012."}, {"is_selected": 0, "url": "http://www.walgreens.com/storelocator/find.jsp?requestType=locator", "passage_text": "Your Walgreens Store. Select a store from the search results to make it Your Walgreens Store and save time getting what you need. Your Walgreens Store will be the default location for picking up prescriptions, photos, in store orders and finding deals in the Weekly Ad."}], "query_id": 9652, "answers": ["Approximately $15,000 per year."], "query_type": "numeric", "query": "walgreens store sales average"}
{"passages": [{"is_selected": 0, "url": "http://www.breakintobartending.com/how-much-do-bartenders-make/", "passage_text": "A bartender’s income is comprised mostly of tips– 55% to be exact. In some states, employers aren’t even required to pay their bartenders the minimum wage and can pay as low as $2.13 per hour, and they depend on their tips almost entirely. Bartending can be a lot of things. For some it is exciting, for others exhausting. At times there is a lot of fun to be had, at others it is rather dull. But for the most part, bartending is almost always rewarding in the financial sense, as long as you stick with it."}, {"is_selected": 1, "url": "http://www.breakintobartending.com/how-much-do-bartenders-make/", "passage_text": "According to the Bureau of Labor Statistics, the average hourly wage for a bartender is $10.36, and the average yearly take-home is $21,550. Bartending can be a lot of things. For some it is exciting, for others exhausting. At times there is a lot of fun to be had, at others it is rather dull. But for the most part, bartending is almost always rewarding in the financial sense, as long as you stick with it."}, {"is_selected": 0, "url": "http://careerswiki.com/how-much-do-bartenders-make/", "passage_text": "About 551,100 individuals are employed as bartenders, with half of this number working part-time. The average annual salary for bartenders is $19,050 or an equivalent of $9.16 per hour, including tips. No formal training is needed for one to get a job as a bartender as all it takes are good customer service skills and a comprehensive knowledge about beverages and recipes. "}, {"is_selected": 0, "url": "http://www.answers.com/Q/How_much_do_bartenders_in_Las_Vegas_make", "passage_text": "Confidence votes 11. Bartenders in Vegas can make up to $7-$15 wage plus $10-$50 in tips per hour. But that totally depends on personality, gender and outlet. A hot young female can make $1000 a day tending bar in a strip club, but that is very exceptional. A bartender makes as much as a bartender wants to make. I wouldn't say there is a cap on how much a bartender make in one year. If the service is good and the coversation is i … nteresting a bartender can make alot of money. Location is also a big factor."}, {"is_selected": 0, "url": "https://answers.yahoo.com/question/index?qid=20100102233548AAZEutT", "passage_text": "Best Answer: An average bartender makes about...2 to 3 dollars an hour. but all the money is made off tips depending on the popularity of the bar. I used to make $700 to $1000 a night. but that is in Atlanta. If the bar is busy and you are a good bartender you will make quite a bit. I dont know how much, because I live in a town with a population of 2000 so there is not alot going on around here. Im sure the bartenders make a hundred to two hundred a night total (on a good night)...just depends."}, {"is_selected": 0, "url": "https://answers.yahoo.com/question/index?qid=20100102233548AAZEutT", "passage_text": "Report Abuse. no way to tell you how much bartenders make. in wages anything from minimum to $15 an hour. tips, anywhere from $20 to $300 or more a night. depends on a lot of things. those top dollar jobs only come after a lot of experience. If the bar is busy and you are a good bartender you will make quite a bit. I dont know how much, because I live in a town with a population of 2000 so there is not alot going on around here. Im sure the bartenders make a hundred to two hundred a night total (on a good night)...just depends."}, {"is_selected": 0, "url": "http://work.chron.com/much-bartender-make-annually-7503.html", "passage_text": "Pay by Employment Setting. Bartenders who worked in hotels and hotel restaurants generally reported the highest incomes in 2011, an average of $26,180 a year. Bartenders employed in full-service restaurants tended to earn somewhat less, averaging about $22,130 a year. Bartenders employed by bars earned an average of $20,230 per year, and bartenders who worked for civic and social organizations earned an average of $18,970 a year. The median earnings of bartenders during this period were $9.06 an hour and $18,850 a year. Eighty percent of bartenders in the U.S. reported annual incomes of between $16,170 and $31,860."}, {"is_selected": 0, "url": "http://www.celebritynetworth.com/articles/how-much-does/how-much-does-bartender-make/", "passage_text": "Tips make up half or more of bartender's salaries. If a bartender earned $6.00 an hour, their tips generally average out to $12.00 to $18.00 an hour as additional income. A bartender in an average bar will typically earn $15.00 $30.00 an hour between their wages and tips. According to bartending.org, bartenders in a high volume resort or establishment can earn $50,000 to $75,000 per year between hourly wages and tips. Indeed.com 2010 results show bartenders in restaurants at median salary rates can make a good salary per year: Bartender $73,000."}], "query_id": 9653, "answers": ["$21,550 per year", "The average hourly wage for a bartender is $10.36 and the average yearly take-home is $21,550."], "query_type": "numeric", "query": "how much do bartenders make"}
Now these blocks repeat without any enclosing {} and with no comma separating them. Clearly this can't be JSON.
My question is:
Is this really json as the dataset claims to be?
Any library (preferably in python) to parse it to extract all the individual fields.
I tried ijson in python but then this really doesn't look like real json.

This file is made of one JSON document per line: this is JSON Lines.
The file as a whole is not a valid JSON document. You should read line by line and feed each line into a JSON parser:
#!/usr/bin/env python3
from sys import argv
import json
with open(argv[1], "r", encoding="UTF-8") as f:
lines = [ json.loads(line) for line in f ]
Note: Another similar format is JSON-seq documented in RFC7464.

Related

Question #1: (use while loop only)
An auto repair shop in Miami estimates that, your vehicle will cost $1689 to repair. Assuming that you could not afford to come up with such amount up front, the repair shop offers to let you pay the total repair cost ($1689) by installment, but at an annual interest rate of 3.00% compounded monthly.
Using a while loop,
Display the remaining balance after each month until pay off
Example/Hint: Balance will be $1,493.22 after 1 month.
Using the output from item 1 above, determine how many months will the vehicle be three quarter
(¾) paid off? Use a print statement to display the message.
Example: "Repair cost total will be 3/4 paid off in 2 months"
I am struggling to figure out how to attack this problem. I'm not sure how to calculate how much each monthly payment will be. There is a hint within the question that in one month, your repair cost goes from $1689 to $1493.22, implying that for the first month, you paid off $195.78 of the repair cost. Any help on how to even start this problem would be much appreciated, thank you.

Your first step is to calculate how much the client is paying per month and with that information apply the "compound interest with monthly withdrawals" formula.
To calculate that, you first calculate how much the balance will be after one month with interest:
monthly_interest_rate = annual_interest_rate / 12
balance += (balance * monthly_interest_rate)
Now subtract the balance after one month gave as a tip from the balance calculated above:
balance_after_one_month = 1493.22
paid_monthly_by_customer = balance - balance_after_one_month

I have this title, where the variable option indicates the regional level.
option=3
year=2017
ax.set_title('Household Gas Consumption, Water Heating,NUTS %.i'%option,fontdict={'fontsize':'10','fontweight':'3'})
How could I also add the variable year so the title looks like this?
Household Gas Consumption, Water Heating, NUTS 3, Year: 2017
I tried different ways but I could't make it work. Thanks in advance.

'Household Gas Consumption, Water Heating,NUTS %i Year: %i' %(option , year)

I want to get the first sentence from a paragraph using python.
the paragraph as below
ECONOMYNEXT -Sri Lanka rupee closed steady at 176.40/50 rupees to the US dollar on Friday and gilt yields edged higher on profit taking in the secondary market even as the Central Bank cut policy rates to revive credit demand, while stocks ended 0.26 percent lower, market participants said.
The rupee ended at 176.40/50 rupees against the greenback in the spot market on Thursday.
which i was written from the below code was extracting the sentence until
the decimal place. Thanks for help.
import requests
#from pprint import pprint
from IPython.display import HTML
import json
txt = ''' ECONOMYNEXT -Sri Lanka rupee closed steady at 176.40/50 rupees to the US dollar on Friday and gilt yields edged higher on profit taking in the secondary market even as the Central Bank cut policy rates to revive credit demand, while stocks ended 0.26 percent lower, market participants said.
The rupee ended at 176.40/50 rupees against the greenback in the spot market on Thursday. '''
if len(txt) > 100:
txt = txt.partition('.')[0] + '.'
print(txt)

try to split with '. '(with a space) and '.\n'

You can try this...
txt = " ECONOMYNEXT -Sri Lanka rupee closed steady at 176.40/50 rupees to the US
dollar on Friday and gilt yields edged higher on profit taking in the
secondary market even as the Central Bank cut policy rates to revive credit
demand, while stocks ended 0.26 percent lower, market participants said. The
rupee ended at 176.40/50 rupees against the greenback in the spot market on
Thursday. "
sentence_index = txt.find('. ')
print(txt[0: sentence_index])
You will get output like as follow
ECONOMYNEXT -Sri Lanka rupee closed steady at 176.40/50 rupees to the US dollar on Friday and gilt yields edged higher on profit taking in the secondary market even as the Central Bank cut policy rates to revive credit demand, while stocks ended 0.26 percent lower, market participants said

I'm new to Python and API and am trying to start with some basics like making a list/plot of old BTC prices. I imported the Coinbase Wallet Client and used client.get_historic_prices(), which gave me a list of the price at midnight UTC for 365 days.
How can I adjust the parameters to get different date ranges and data resolution, for example each minute for two years? Is there a way to search the historic values of buy, sell, and spot separately?
from coinbase.wallet.client import Client
hist_price = client.get_historic_prices()
xx=[]
yy=[]
for ii in range(365):
xx.append(ii*-1) # x coordinate being "days ago"
yy.append(float(hist_price['prices'][ii]['price']))
Returns (this is just from a print statement of print(hist_price['prices'][0:3]). So it's midnight once a day.
prices
length = 365
{
"price": "974.39",
"time": "2017-02-01T00:00:00Z"
}
{
"price": "944.29",
"time": "2017-01-31T00:00:00Z"
}
{
"price": "920.47",
"time": "2017-01-30T00:00:00Z"
}

Get_historic_prices is not clearly documented anywhere. This is a mini guide of what I have managed to discover about it and its usage. It's not much but it should be somewhere.
get_historic_prices supports one argument called period which can take the following values:
hour
day
week
month
year
all
Each of them except all returns approximately a list of 360 price points distributed more or less uniformly in the previous hour (day, week, month, year respectively).
all returns a list of price points one for each day at 00:00:00 UTC (i think).
get_historic_prices should also support a currency_pair argument as do the get_buy_price, get_sell_price, and get_spot_price do. Unfortunately, although I have submitted a PR, it has not been merged yet.

I am query AWS pricing using boto ec2 in python.
Firstly,I am finding all offering instances of particular instance type using get_all_reserved_instances_offerings ,
then for all instances return by above, I am checking amount and fixed price in 'hourly frequency'. I do this by this code:
for ins in each_ins.recurring_charges:
if ins.frequency == 'Hourly':
print float(ins.amount)
print float(each_ins.fixed_price)
each_ins.fixed_price prints upfront correctly
"d2.2xlarge": 3844.0,
"m3.2xlarge": 1961.0,
it shows correct price as shown in picture with red mark:
But ins.amount prints:
"d2.2xlarge": 0.438,
"m3.2xlarge": 0.248,
I think it should be 0.8768 as shown in above picture with green mark.

The values in the green box are the overall effective cost of the instances per hour over the year (i.e. accounting for the per hour charge over the year plus the initial up-front cost spread over the year.)
However the ins.amount value is the cost charged per hour each month, ignoring the initial up-front cost. Roughly calculating over a month from the monthly cost gives an hourly cost of ~0.444, pretty much corresponding to the values you're seeing.
AWS says the following with respect to the continuing monthly charges:
*This is the average monthly payment over the course of the RI term. For each month, the actual monthly payment will equal the actual number of hours in that month multiplied by the hourly usage rate. The hourly usage rate is equivalent to the total average monthly payments over the term of the RI divided by the total number of hours (based on a 365 day year) over the term of the RI.