I am using JSON library and trying to import a page feed to an CSV file. Tried many a ways to get the result however every time code execute it Gives JSON not serialzable. No Facebook use auth code which I have and used it so connection string will change however if you use a page which has public privacy you will still be able to get the result from below code.
following is the code
import urllib3
import json
import requests
#from pprint import pprint
import csv
from urllib.request import urlopen
page_id = "abcd" # username or id
api_endpoint = "https://graph.facebook.com"
fb_graph_url = api_endpoint+"/"+page_id
try:
#api_request = urllib3.Requests(fb_graph_url)
#http = urllib3.PoolManager()
#api_response = http.request('GET', fb_graph_url)
api_response = requests.get(fb_graph_url)
try:
#print (list.sort(json.loads(api_response.read())))
obj = open('data', 'w')
# write(json_dat)
f = api_response.content
obj.write(json.dumps(f))
obj.close()
except Exception as ee:
print(ee)
except Exception as e:
print( e)
Tried many approach but not successful. hope some one can help
api_response.content is the text content of the API, not a Python object so you won't be able to dump it.
Try either:
f = api_response.content
obj.write(f)
Or
f = api_response.json()
obj.write(json.dumps(f))
requests.get(fb_graph_url).content
is probably a string. Using json.dumps on it won't work. This function expects a list or a dictionary as the argument.
If the request already returns JSON, just write it to the file.
Related
This question already has answers here:
How can I parse (read) and use JSON?
(5 answers)
What are the differences between the urllib, urllib2, urllib3 and requests module?
(11 answers)
Closed last month.
I want to dynamically query Google Maps through the Google Directions API. As an example, this request calculates the route from Chicago, IL to Los Angeles, CA via two waypoints in Joplin, MO and Oklahoma City, OK:
http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false
It returns a result in the JSON format.
How can I do this in Python? I want to send such a request, receive the result and parse it.
I recommend using the awesome requests library:
import requests
url = 'http://maps.googleapis.com/maps/api/directions/json'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
resp = requests.get(url=url, params=params)
data = resp.json() # Check the JSON Response Content documentation below
JSON Response Content: https://requests.readthedocs.io/en/master/user/quickstart/#json-response-content
The requests Python module takes care of both retrieving JSON data and decoding it, due to its builtin JSON decoder. Here is an example taken from the module's documentation:
>>> import requests
>>> r = requests.get('https://github.com/timeline.json')
>>> r.json()
[{u'repository': {u'open_issues': 0, u'url': 'https://github.com/...
So there is no use of having to use some separate module for decoding JSON.
requests has built-in .json() method
import requests
requests.get(url).json()
import urllib
import json
url = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
result = json.load(urllib.urlopen(url))
Use the requests library, pretty print the results so you can better locate the keys/values you want to extract, and then use nested for loops to parse the data. In the example I extract step by step driving directions.
import json, requests, pprint
url = 'http://maps.googleapis.com/maps/api/directions/json?'
params = dict(
origin='Chicago,IL',
destination='Los+Angeles,CA',
waypoints='Joplin,MO|Oklahoma+City,OK',
sensor='false'
)
data = requests.get(url=url, params=params)
binary = data.content
output = json.loads(binary)
# test to see if the request was valid
#print output['status']
# output all of the results
#pprint.pprint(output)
# step-by-step directions
for route in output['routes']:
for leg in route['legs']:
for step in leg['steps']:
print step['html_instructions']
just import requests and use from json() method :
source = requests.get("url").json()
print(source)
OR you can use this :
import json,urllib.request
data = urllib.request.urlopen("url").read()
output = json.loads(data)
print (output)
Try this:
import requests
import json
# Goole Maps API.
link = 'http://maps.googleapis.com/maps/api/directions/json?origin=Chicago,IL&destination=Los+Angeles,CA&waypoints=Joplin,MO|Oklahoma+City,OK&sensor=false'
# Request data from link as 'str'
data = requests.get(link).text
# convert 'str' to Json
data = json.loads(data)
# Now you can access Json
for i in data['routes'][0]['legs'][0]['steps']:
lattitude = i['start_location']['lat']
longitude = i['start_location']['lng']
print('{}, {}'.format(lattitude, longitude))
Also for pretty Json on console:
json.dumps(response.json(), indent=2)
possible to use dumps with indent. (Please import json)
I am extracting data from this API
I was able to save the JSON file on my local machine.
I want to run the requests for several stocks.
How do I do it?
I tried to play with for loops but not good came out of this. I attached the code below.
the out put is:
AAPL
[]
TSLA
[]
Thank you, Tal
try:
# For Python 3.0 and later
from urllib.request import urlopen
except ImportError:
# Fall back to Python 2's urllib2
from urllib2 import urlopen
import requests
import json
import time
def get_jsonparsed_data(url):
"""
Receive the content of ``url``, parse it as JSON and return the object.
Parameters
----------
url : str
Returns
-------
dict
"""
stock_symbol = ["AAPL","TSLA"]
for symbol in stock_symbol:
print (symbol)
#Sending the API request
r = requests.get('https://financialmodelingprep.com/api/v3/income-statement/symbol={stock_symbol}?limit=120&apikey={removed by me})
packages_JSON = r.json()
print(packages_JSON)
#Exporting the data into JSON file
with open('stocks_data321.json', 'w', encoding='utf-8') as f:
json.dump(packages_JSON, f, ensure_ascii=False, indent=4)
Querying multiple APIs iterativelly will take a lot of time. Consider using theading or AsyncIO to do requests simultaniously and speed up the process.
In a nutshell you should do something like this for each API:
import threading
for provider in [...]: # list of APIs to query
t = threading.Thread(target=api_request_function, args=(provider, ...))
t.start()
However better read this great article first to understand whats and whys of threading approach.
I have around 100 machines running Mersive Solstice, which is a wireless display tool. I'm trying to gather a few important pieces of information, in particular the fulfillment ID for the license for each installed instance.
Using the Solstice OpenControl API, found here, I whipped up a python script to grab everything I needed using a json GET. However, even when using the example GET from the documentation,
import requests
import json
url = ‘http://ip-of-machine/api/stats’
r = requests.get(url)
jsonStats = json.loads(r.text)
usersConnected = jsonStats.m_statistics.m_connectedUsers
I encounter:
Traceback (most recent call last):
File "C:/Python27/test.py", line 7, in <module>
usersConnected = jsonStats.m_statistics.m_connectedUsers
AttributeError: 'dict' object has no attribute 'm_statistics'
Which is very confusing. I've found plenty of similar questions on SO regarding this problem, but not one that's been specifically regarding wrong GET requests from the API Reference guide.
Additionally, here is my script:
import requests
import json
from time import sleep
url = 'test'
f = open("ip.txt", "r")
while(url != ""):
url = f.readline()
url = url.rstrip('\n')
print(url)
try:
r = requests.get(url)
except:
sleep(5)
jsonConfig = json.loads(r.text)
displayName = jsonConfig.m_displayInformation.m_displayName
hostName = jsonConfig.m_displayInformation.m_hostName
ipv4 = jsonConfig.m_displayInformation.m_ipv4
fulfillmentId = jsonConfig.m_licenseCuration.fulfillmentId
r.close()
f.close
I import the URL's from a text document for easy keeping. I'm able to make the connection to the /api/config JSON, and when the URL is put into a browser it does spit out the JSON records:
Json uses "Dicts" which are a type of array. You are just using them in the wrong way. I recommend reading Python Data Structures.
Json.Loads()
Returns a dictionary not a object. Do:
dict['key']['key']
Here is how your code should look:
import requests
import json
from time import sleep
url = 'test'
f = open("ip.txt", "r")
while(url != ""):
url = f.readline()
url = url.rstrip('\n')
print(url)
try:
response = requests.get(url)
json_object = json.loads(response .text)
displayName = json_object['m_displayInformation']['m_displayName']
hostName = json_object['m_displayInformation']['m_hostName']
ipv4 = json_object['m_displayInformation']['m_ipv4']
fulfillmentId = json_object['m_licenseCuration']['fulfillmentId']
except:
pass
response .close()
f.close()
I hope this was helpful!
This question already has answers here:
HTTP requests and JSON parsing in Python [duplicate]
(8 answers)
Closed 8 months ago.
How would I parse a json api response with python?
I currently have this:
import urllib.request
import json
url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'
def response(url):
with urllib.request.urlopen(url) as response:
return response.read()
res = response(url)
print(json.loads(res))
I'm getting this error:
TypeError: the JSON object must be str, not 'bytes'
What is the pythonic way to deal with json apis?
Version 1: (do a pip install requests before running the script)
import requests
r = requests.get(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
print(r.json())
Version 2: (do a pip install wget before running the script)
import wget
fs = wget.download(url='https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty')
with open(fs, 'r') as f:
content = f.read()
print(content)
you can use standard library python3:
import urllib.request
import json
url = 'http://www.reddit.com/r/all/top/.json'
req = urllib.request.Request(url)
##parsing response
r = urllib.request.urlopen(req).read()
cont = json.loads(r.decode('utf-8'))
counter = 0
##parcing json
for item in cont['data']['children']:
counter += 1
print("Title:", item['data']['title'], "\nComments:", item['data']['num_comments'])
print("----")
##print formated
#print (json.dumps(cont, indent=4, sort_keys=True))
print("Number of titles: ", counter)
output will be like this one:
...
Title: Maybe we shouldn't let grandma decide things anymore.
Comments: 2018
----
Title: Carrie Fisher and Her Stunt Double Sunbathing on the Set of Return of The Jedi, 1982
Comments: 880
----
Title: fidget spinner
Comments: 1537
----
Number of titles: 25
I would usually use the requests package with the json package. The following code should be suitable for your needs:
import requests
import json
url = 'https://hacker-news.firebaseio.com/v0/topstories.json?print=pretty'
r = requests.get(url)
print(json.loads(r.content))
Output
[11008076,
11006915,
11008202,
....,
10997668,
10999859,
11001695]
The only thing missing in the original question is a call to the decode method on the response object (and even then, not for every python3 version). It's a shame no one pointed that out and everyone jumped on a third party library.
Using only the standard library, for the simplest of use cases :
import json
from urllib.request import urlopen
def get(url, object_hook=None):
with urlopen(url) as resource: # 'with' is important to close the resource after use
return json.load(resource, object_hook=object_hook)
Simple use case :
data = get('http://url') # '{ "id": 1, "$key": 13213654 }'
print(data['id']) # 1
print(data['$key']) # 13213654
Or if you prefer, but riskier :
from types import SimpleNamespace
data = get('http://url', lambda o: SimpleNamespace(**o)) # '{ "id": 1, "$key": 13213654 }'
print(data.id) # 1
print(data.$key) # invalid syntax
# though you can still do
print(data.__dict__['$key'])
With Python 3
import requests
import json
url = 'http://IP-Address:8088/ws/v1/cluster/scheduler'
r = requests.get(url)
data = json.loads(r.content.decode())
I'm trying to write a simple script in python using urllib2 and json where I print the json to the console.
Currently I'm having trouble getting the auth_signature value. I already have the url variable setup with the appropriate keys except for the auth_signature. How do I go about this?
Here is what I have:
import json
import urllib2
import oauth2
timestamp = oauth2.generate_timestamp
nonce = oauth2.generate_nonce
url = "http://api.yelp.com/v2/search?term=food&location=Seattle&callback=callbackYelpAuth&oauth_consumer_key=XXX&oauth_consumer_secret=XXX&oauth_token=XXX&oauth_signature_method=HMAC-SHA1&oauth_timestamp=" + str(timestamp) + "&oauth_nonce=" + str(nonce) + "&oauth_signature=" + str(????)
json_obj = urllib2.urlopen(url)
data = json.load(json_obj)
print data
Following code is working fine. Hope it will help you.
import urlparse
url = "http://api.yelp.com/v2/search?term=food&location=Seattle&callback=callbackYelpAuth&oauth_consumer_key=XXX&oauth_consumer_secret=XXX&oauth_token=XXX&oauth_signature_method=HMAC-SHA1&oauth_timestamp=temstamp_value&oauth_nonce=nonce_value&oauth_signature=signature_value"
parsed = urlparse.urlparse(url)
params = urlparse.parse_qsl(parsed.query)
for x,y in params:
print "Parameter = "+x,", Value = "+y