This question already has answers here:
How can I parse (read) and use JSON?
(5 answers)
What are the differences between the urllib, urllib2, urllib3 and requests module?
(11 answers)
Closed last month.
I want to dynamically query Google Maps through the Google Directions API. As an example, this request calculates the route from Chicago, IL to Los Angeles, CA via two waypoints in Joplin, MO and Oklahoma City, OK:
http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false
It returns a result in the JSON format.
How can I do this in Python? I want to send such a request, receive the result and parse it.
I recommend using the awesome requests library:
import requests
url = 'http://maps.googleapis.com/maps/api/directions/json'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
resp = requests.get(url=url, params=params)
data = resp.json() # Check the JSON Response Content documentation below
JSON Response Content: https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content
The requests Python module takes care of both retrieving JSON data and decoding it, due to its builtin JSON decoder. Here is an example taken from the module's documentation:
>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json()
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...
So there is no use of having to use some separate module for decoding JSON.
requests has built-in .json() method
import requests
requests.get(url).json()
import urllib
import json
url = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
result = json.load(urllib.urlopen(url))
Use the requests library, pretty print the results so you can better locate the keys/values you want to extract, and then use nested for loops to parse the data. In the example I extract step by step driving directions.
import json, requests, pprint
url = 'http://maps.googleapis.com/maps/api/directions/json?'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
data = requests.get(url=url, params=params)
binary = data.content
output = json.loads(binary)
# test to see if the request was valid
#print output['status']
# output all of the results
#pprint.pprint(output)
# step-by-step directions
for route in output['routes']:
for leg in route['legs']:
for step in leg['steps']:
print step['html_instructions']
just import requests and use from json() method :
source = requests.get("url").json()
print(source)
OR you can use this :
import json,urllib.request
data = urllib.request.urlopen("url").read()
output = json.loads(data)
print (output)
Try this:
import requests
import json
# Goole Maps API.
link = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
# Request data from link as 'str'
data = requests.get(link).text
# convert 'str' to Json
data = json.loads(data)
# Now you can access Json
for i in data['routes'][0]['legs'][0]['steps']:
lattitude = i['start_location']['lat']
longitude = i['start_location']['lng']
print('{}, {}'.format(lattitude, longitude))
Also for pretty Json on console:
json.dumps(response.json(), indent=2)
possible to use dumps with indent. (Please import json)
Related
I am extracting data from this API
I was able to save the JSON file on my local machine.
I want to run the requests for several stocks.
How do I do it?
I tried to play with for loops but not good came out of this. I attached the code below.
the out put is:
AAPL
[]
TSLA
[]
Thank you, Tal
try:
# For Python 3.0 and later
from urllib.request import urlopen
except ImportError:
# Fall back to Python 2's urllib2
from urllib2 import urlopen
import requests
import json
import time
def get_jsonparsed_data(url):
"""
Receive the content of ``url``, parse it as JSON and return the object.
Parameters
----------
url : str
Returns
-------
dict
"""
stock_symbol = ["AAPL","TSLA"]
for symbol in stock_symbol:
print (symbol)
#Sending the API request
r = requests.get('https://financialmodelingprep.com/api/v3/income-statement/symbol={stock_symbol}?limit=120&apikey={removed by me})
packages_JSON = r.json()
print(packages_JSON)
#Exporting the data into JSON file
with open('stocks_data321.json', 'w', encoding='utf-8') as f:
json.dump(packages_JSON, f, ensure_ascii=False, indent=4)
Querying multiple APIs iterativelly will take a lot of time. Consider using theading or AsyncIO to do requests simultaniously and speed up the process.
In a nutshell you should do something like this for each API:
import threading
for provider in [...]: # list of APIs to query
t = threading.Thread(target=api_request_function, args=(provider, ...))
t.start()
However better read this great article first to understand whats and whys of threading approach.
So, according to Python-Binance's documentation, the thing I am getting back is a dictionary JSON response. When I get the response it looks like this:
{'symbol': 'BTCUSDT', 'price': '37256.90000000'}
What I want to do is take that and reformat it so it looks like this:
'$BTC # 37256.900'
Basically removing the braces and all the other garbage that isn't needed.
The code I was using from the Binance API with Python-Binance and the way I was printing it:
price = client.get_symbol_ticker(symbol="BTCUSDT")
from crypto_tracker import price
print(price)
Is this possible at all?
Python has a great library called json to help you take a json string and create a python dict from it.
From your example I have created the following code:
import json
## Given example string
json_string = '{"symbol": "BTCUSDT", "price": "37256.90000000"}'
## Create a python dict from json string using json library json.loads
json_dict = json.loads(json_string)
## What the new python dict looks like:
## json_dict = {'symbol': 'BTCUSDT', 'price': '37256.90000000'}
## Print your expected output
print(f'$BTC # {json_dict["price"]}')
Print Output:
$BTC # 37256.90000000
Using the python "json" library will automatically convert the json string to a python dict which you can then use any way you like (i.e. automatically removing all the extra "braces and ... other garbage that isn't needed.").
Edit:
Per more back and forth in comments I took a look at the Binance API. The API endpoint your using is:
GET /api/v3/ticker/price
To get information from this you would need to make an HTTP request to:
https://api.binance.com/api/v3/ticker/price
Here is example working code:
import requests
import json
## Send a GET request to binance api
## Since this is a get request no apikey / secret is needed
response = requests.get("https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT")
## Print the text from the get request
## Output: {"symbol":"BTCUSDT","price":"37211.17000000"}
print(response.text)
## Response.text is a string so we can use json.loads on it
ticket = json.loads(response.text)
## Now that ticket is a python dict of the json string
## response from binance we can print what you want
print(f'$BTC # {ticket["price"]}')
## Output: $BTC # 37221.87000000
I'm trying to calculate the exchange from US Dolar to Brazilian Reais.
I found an REST API from brazilian central bank.
My Python code is receive the API return in JSON format, like that:
{'#odata.context': 'https://was-p.bcnet.bcb.gov.br/olinda/servico/PTAX/versao/v1/odata$metadata#_CotacaoDolarDia(cotacaoVenda)', 'value': [{'cotacaoVenda': 3.8344}]}
In my code I could isolate this part of resulte "[{'cotacaoVenda': 3.8344}]", but I can't isolate only the value "3.8344".
Follow my code:
# Cotação do Dólar V.01
import json
import requests
r = requests.get("https://olinda.bcb.gov.br/olinda/servico/PTAX/versao/v1/odata/CotacaoDolarDia(dataCotacao=#dataCotacao)?#dataCotacao='03-15-2019'&$top=1&$format=json&$select=cotacaoVenda")
if r.status_code == 200:
cotacao = json.loads(r.content)
print(cotacao['value'])
Any idea how can I isolate only the "3.8344" contained in JSON return?
Thank you
The variable cotacao, is a list, which has only one item. So we access it with index [0]. That object, is a dictionary, which we can access its fields using their key:
import json
import requests
r = requests.get("https://olinda.bcb.gov.br/olinda/servico/PTAX/versao/v1/odata/CotacaoDolarDia(dataCotacao=#dataCotacao)?#dataCotacao='03-15-2019'&$top=1&$format=json&$select=cotacaoVenda")
if r.status_code == 200:
cotacao = json.loads(r.content)
print(cotacao['value'][0]['cotacaoVenda'])
I used python2 to make a request to RNAcentral database, and I read the response as JSON format by the use of this command: response.json().
This let me read the data as a dictionary data type, so I used the corresponding syntax to obtain the data from cross references, which contained some links to other databases, but when I try to make the request for each link using the command mentioned above, I can't read it as JSON, because I can only obtain the response content as HTML.
So I need to know how to read make a request to each link from cross references and read it as JSON using python language.
Here is the code:
direcc = 'http://rnacentral.org/api/v1/rna/'+code+'/?flat=true.json'
resp = requests.get(direcc)
datos=resp.json()
d={}
links = []
for diccionario in datos['xrefs']['results']:
if diccionario['taxid']==9606:
base_datos=diccionario['database']
for llave,valor in diccionario['accession'].iteritems():
d[base_datos]={'url':diccionario['accession']['url'],
'expert_db_url':diccionario['accession']['expert_db_url'],
'source_url':diccionario['accession']['source_url']}
for key,value in d.iteritems():
links.append(d[key]['expert_db_url'])
for item in links:
response = requests.get(item)
r = response.json()
And this is the error I get: ValueError: No JSON object could be decoded.
Thank you.
This question already has answers here:
HTTP requests and JSON parsing in Python [duplicate]
(8 answers)
Closed 8 months ago.
How would I parse a json api response with python?
I currently have this:
import urllib.request
import json
url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'
def response(url):
with urllib.request.urlopen(url) as response:
return response.read()
res = response(url)
print(json.loads(res))
I'm getting this error:
TypeError: the JSON object must be str, not 'bytes'
What is the pythonic way to deal with json apis?
Version 1: (do a pip install requests before running the script)
import requests
r = requests.get(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
print(r.json())
Version 2: (do a pip install wget before running the script)
import wget
fs = wget.download(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
with open(fs, 'r') as f:
content = f.read()
print(content)
you can use standard library python3:
import urllib.request
import json
url = 'http://www.reddit.com/r/all/top/.json'
req = urllib.request.Request(url)
##parsing response
r = urllib.request.urlopen(req).read()
cont = json.loads(r.decode('utf-8'))
counter = 0
##parcing json
for item in cont['data']['children']:
counter += 1
print("Title:", item['data']['title'], "\nComments:", item['data']['num_comments'])
print("----")
##print formated
#print (json.dumps(cont, indent=4, sort_keys=True))
print("Number of titles: ", counter)
output will be like this one:
...
Title: Maybe we shouldn't let grandma decide things anymore.
Comments: 2018
----
Title: Carrie Fisher and Her Stunt Double Sunbathing on the Set of Return of The Jedi, 1982
Comments: 880
----
Title: fidget spinner
Comments: 1537
----
Number of titles: 25
I would usually use the requests package with the json package. The following code should be suitable for your needs:
import requests
import json
url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'
r = requests.get(url)
print(json.loads(r.content))
Output
[11008076,
11006915,
11008202,
....,
10997668,
10999859,
11001695]
The only thing missing in the original question is a call to the decode method on the response object (and even then, not for every python3 version). It's a shame no one pointed that out and everyone jumped on a third party library.
Using only the standard library, for the simplest of use cases :
import json
from urllib.request import urlopen
def get(url, object_hook=None):
with urlopen(url) as resource: # 'with' is important to close the resource after use
return json.load(resource, object_hook=object_hook)
Simple use case :
data = get('http://url') # '{ "id": 1, "$key": 13213654 }'
print(data['id']) # 1
print(data['$key']) # 13213654
Or if you prefer, but riskier :
from types import SimpleNamespace
data = get('http://url', lambda o: SimpleNamespace(**o)) # '{ "id": 1, "$key": 13213654 }'
print(data.id) # 1
print(data.$key) # invalid syntax
# though you can still do
print(data.__dict__['$key'])
With Python 3
import requests
import json
url = 'http://IP-Address:8088/ws/v1/cluster/scheduler'
r = requests.get(url)
data = json.loads(r.content.decode())