import requests
import json
import csv
# These our are demo API keys, you can use them!
#location = ""
api_key = 'simplyrets'
api_secret = 'simplyrets'
#api_url = 'https://api.simplyrets.com/properties?q=%s&limit=1' % (location)
api_url = 'https://api.simplyrets.com/properties'
response = requests.get(api_url, auth=(api_key, api_secret))
response.raise_for_status()
houseData = json.loads(response.text)
#different parameters we need to know
p = houseData['property']
roof = p["roof"]
cooling = p["cooling"]
style = p["style"]
area = p["area"]
bathsFull = p["bathsFull"]
bathsHalf = p["bathsHalf"]
This is a snippet of the code that I am working with to try and take the information from the JSON provided by the API and put them into variables that I can actually use.
I thought that when you loaded it with json.loads() it would become a dictionary.
Yet it is telling me that I cannot do p = houseData['property'] because "list indices must be integers, not str".
Am I wrong that houseData should be a dictionary?
There are hundreds of properties returned, all of which are in a list.
You'll need to specify which property you want, so for the first one:
p = houseData[0]['property']
From https://docs.python.org/2/library/json.html :
json.loads(s[, encoding[, cls[, object_hook[, parse_float[, parse_int[, parse_constant[, object_pairs_hook[, **kw]]]]]]]])
Deserialize s (a str or unicode instance containing a JSON document) to a Python object using this conversion table.
If s is a str instance and is encoded with an ASCII based encoding other than UTF-8 (e.g. latin-1), then an appropriate encoding name must be specified. Encodings that are not ASCII based (such as UCS-2) are not allowed and should be decoded to unicode first.
The other arguments have the same meaning as in load().
If your JSON starts as an array at the outermost layer, it will be an array. If your JSON's outermost layer is an associative array, then please post your JSON and we can look into it a little further.
The problem is that json.loads() doesn't necessarily return a dictionary. If the outside container of the JSON is a list, then json.loads() will return a list, where the elements could be lists or dictionaries. Try iterating through the list returned by json.loads(). It's possible the dictionary you're looking for is simply json.loads()[0] or some other element.
There are 2 different types of JSON elements: nodes and arrays.
A node looks like:
node = {
foo = 7
bar = "Hello World!"
}
A array looks like this:
array = [ "one", "two", 3, 4, "5ive" ]
Your JSON element is probably a array. You can verify whether it's an array, dict, or other by using:
isinstance(json_element, dict)
isinstance(json_element, list)
Hope this helps!
There are some minor changes you should do:
Your API response is returning a list, so you have to iterate over it.
The requests library already supports converting to JSON so you don't have to worry about it.
import requests
# These our are demo API keys, you can use them!
#location = ""
api_key = 'simplyrets'
api_secret = 'simplyrets'
#api_url = 'https://api.simplyrets.com/properties?q=%s&limit=1' % (location)
api_url = 'https://api.simplyrets.com/properties'
response = requests.get(api_url, auth=(api_key, api_secret))
response.raise_for_status()
houseData = response.json()
# different parameters we need to know
for data in houseData:
p = data['property']
roof = p["roof"]
cooling = p["cooling"]
style = p["style"]
area = p["area"]
bathsFull = p["bathsFull"]
bathsHalf = p["bathsHalf"]
If you want to make sure you will have only one result, do an if statement to check this.
if len(houseData) != 1:
raise ValueError("Expecting only 1 houseData.")
data = houseData[0]
...
Related
I am trying to pass in a JSON file and convert the data into a dictionary.
So far, this is what I have done:
import json
json1_file = open('json1')
json1_str = json1_file.read()
json1_data = json.loads(json1_str)
I'm expecting json1_data to be a dict type but it actually comes out as a list type when I check it with type(json1_data).
What am I missing? I need this to be a dictionary so I can access one of the keys.
Your JSON is an array with a single object inside, so when you read it in you get a list with a dictionary inside. You can access your dictionary by accessing item 0 in the list, as shown below:
json1_data = json.loads(json1_str)[0]
Now you can access the data stored in datapoints just as you were expecting:
datapoints = json1_data['datapoints']
I have one more question if anyone can bite: I am trying to take the average of the first elements in these datapoints(i.e. datapoints[0][0]). Just to list them, I tried doing datapoints[0:5][0] but all I get is the first datapoint with both elements as opposed to wanting to get the first 5 datapoints containing only the first element. Is there a way to do this?
datapoints[0:5][0] doesn't do what you're expecting. datapoints[0:5] returns a new list slice containing just the first 5 elements, and then adding [0] on the end of it will take just the first element from that resulting list slice. What you need to use to get the result you want is a list comprehension:
[p[0] for p in datapoints[0:5]]
Here's a simple way to calculate the mean:
sum(p[0] for p in datapoints[0:5])/5. # Result is 35.8
If you're willing to install NumPy, then it's even easier:
import numpy
json1_file = open('json1')
json1_str = json1_file.read()
json1_data = json.loads(json1_str)[0]
datapoints = numpy.array(json1_data['datapoints'])
avg = datapoints[0:5,0].mean()
# avg is now 35.8
Using the , operator with the slicing syntax for NumPy's arrays has the behavior you were originally expecting with the list slices.
Here is a simple snippet that read's in a json text file from a dictionary. Note that your json file must follow the json standard, so it has to have " double quotes rather then ' single quotes.
Your JSON dump.txt File:
{"test":"1", "test2":123}
Python Script:
import json
with open('/your/path/to/a/dict/dump.txt') as handle:
dictdump = json.loads(handle.read())
You can use the following:
import json
with open('<yourFile>.json', 'r') as JSON:
json_dict = json.load(JSON)
# Now you can use it like dictionary
# For example:
print(json_dict["username"])
The best way to Load JSON Data into Dictionary is You can user the inbuilt json loader.
Below is the sample snippet that can be used.
import json
f = open("data.json")
data = json.load(f))
f.close()
type(data)
print(data[<keyFromTheJsonFile>])
I am working with a Python code for a REST API, so this is for those who are working on similar projects.
I extract data from an URL using a POST request and the raw output is JSON. For some reason the output is already a dictionary, not a list, and I'm able to refer to the nested dictionary keys right away, like this:
datapoint_1 = json1_data['datapoints']['datapoint_1']
where datapoint_1 is inside the datapoints dictionary.
pass the data using javascript ajax from get methods
**//javascript function
function addnewcustomer(){
//This function run when button click
//get the value from input box using getElementById
var new_cust_name = document.getElementById("new_customer").value;
var new_cust_cont = document.getElementById("new_contact_number").value;
var new_cust_email = document.getElementById("new_email").value;
var new_cust_gender = document.getElementById("new_gender").value;
var new_cust_cityname = document.getElementById("new_cityname").value;
var new_cust_pincode = document.getElementById("new_pincode").value;
var new_cust_state = document.getElementById("new_state").value;
var new_cust_contry = document.getElementById("new_contry").value;
//create json or if we know python that is call dictionary.
var data = {"cust_name":new_cust_name, "cust_cont":new_cust_cont, "cust_email":new_cust_email, "cust_gender":new_cust_gender, "cust_cityname":new_cust_cityname, "cust_pincode":new_cust_pincode, "cust_state":new_cust_state, "cust_contry":new_cust_contry};
//apply stringfy method on json
data = JSON.stringify(data);
//insert data into database using javascript ajax
var send_data = new XMLHttpRequest();
send_data.open("GET", "http://localhost:8000/invoice_system/addnewcustomer/?customerinfo="+data,true);
send_data.send();
send_data.onreadystatechange = function(){
if(send_data.readyState==4 && send_data.status==200){
alert(send_data.responseText);
}
}
}
django views
def addNewCustomer(request):
#if method is get then condition is true and controller check the further line
if request.method == "GET":
#this line catch the json from the javascript ajax.
cust_info = request.GET.get("customerinfo")
#fill the value in variable which is coming from ajax.
#it is a json so first we will get the value from using json.loads method.
#cust_name is a key which is pass by javascript json.
#as we know json is a key value pair. the cust_name is a key which pass by javascript json
cust_name = json.loads(cust_info)['cust_name']
cust_cont = json.loads(cust_info)['cust_cont']
cust_email = json.loads(cust_info)['cust_email']
cust_gender = json.loads(cust_info)['cust_gender']
cust_cityname = json.loads(cust_info)['cust_cityname']
cust_pincode = json.loads(cust_info)['cust_pincode']
cust_state = json.loads(cust_info)['cust_state']
cust_contry = json.loads(cust_info)['cust_contry']
#it print the value of cust_name variable on server
print(cust_name)
print(cust_cont)
print(cust_email)
print(cust_gender)
print(cust_cityname)
print(cust_pincode)
print(cust_state)
print(cust_contry)
return HttpResponse("Yes I am reach here.")**
So I'm making an HTTP GET request for a query which returns me some data.
The answer I'm getting is exactly what I want/what I expected. Now I'm wondering how I can take only what I want from the returned data.
This is the code I'm running:
r = requests.get(url = URL, params = PARAMS, auth = AUTH)
print (r.text)
And this is what I get from it:
{"status":"success","data":{"resultType":"matrix","result":[{"metric":{},"values":[[1567169541,"1"],[1567169556,"1"],[1567169571,"1"],[1567169586,"1"],[1567169601,"1"]]}]}}
What I need from this is only the number inside the "values". So there's a whole lot of data I don't need. Inside the brackets there's also a timestamp which I don't need.
I really ony need the number.
So how can I do this?
r.text is the literal body of the response; in this case, it contains a JSON value that needs to be decoded to get a dict:
d = json.decodes(r.text)
However, the Response object has a method to do that for you.
d = r.json()
Now it's just a matter of indexing the Python data structure appropriately.
# d['data']['result'] == [{"metric":{},"values":[[1567169541,"1"],[1567169556,"1"],[1567169571,"1"],[1567169586,"1"],[1567169601,"1"]]}]
# x only takes a single value in this case, as the above is a singleton list
# y == [1567169541,"1"]
# [1567169556,"1"]
# etc
values = [y[0] for x in d['data']['result'] for y in x['values']]
Input Data:
val1 = '[{"EmpID":123456,"AnalystID": "8aa18b9c59XXXXXb20cc2534","173f48XXXXXX427f3f14516dd0"]}]'
Expected Output:
val_op = {"EmpID":123456,"AnalystID": "8aa18b9c59XXXXXb20cc2534","173f48XXXXXX427f3f14516dd0"]}
type(val1) is str
type(val_op) is dict
(basically I just need to remove first and last single quote which define Val1 as String).
Approach i tried:
>>> strlen = len(val1)
>>> payloadStr = val1[1:(strlen-1)]
'{"EmpID":123456,"AnalystID":
"8aa18b9c59XXXXXb20cc2534","173f48XXXXXX427f3f14516dd0"]}'
>>> import json
>>>json.loads(payloadsStr)
{'EmpID':123456,'AnalystID':
'8aa18b9c59XXXXXb20cc2534','173f48XXXXXX427f3f14516dd0']}
You don't have to treat the [] separately, json.loads can handle json list objects.
import json
payload = json.loads(val1)[0]
Also note that the list value in your string is missing an opening bracket and should instead be...
val1 = '[{"EmpID":123456,"AnalystID": ["8aa18b9c59XXXXXb20cc2534","173f48XXXXXX427f3f14516dd0"]}]'
# ^
Since you mentionned the string comes from a web request, this means whoever made that request misformatted their json.
To clarify, this is a string. From the string, you want to extract a dictionary, that is contained in a list.
This makes it more difficult, as opposed to a simple list that contains a dictionary.
One approach you could do is:
val1 = val1.str.split('[')[1].str.split(']')[0]
This should get rid of the brackets, and return an array of 1 index, which contains your dictionary in a string.
Another way, more straightforward is:
val1 = val1.replace('[','').replace(']','')
From there
import json
json.loads(payloadsStr)
Should get you the dictionary.
Thank you all for all the solutions. Since Desired output need to have double quote, hence json.loads(val1) and dict(*eval(val1)) can not be used.
I just tried to replace the [] from original string and when passed in Post Request as data payload, it worked fine.
strlen = len(val1)
payloadStr = val1[1:(strlen-1)]
response = requests.request("POST", url, verify=False, data=payloadsStr, headers=header)
response.text
got the output. Apologies for any confusion. Thanks Again
I'm using requests in python for dealing with sending data (Post + JSON)
Here is my problem:
payload = {"credentials": {"deviceid": "XXX","pin": "XXX"},"data": [{"id": "","name": "","type": "","data": [{"value": "","date": ""}]}]}
Then I want to fill the blank values ("" not XXX) by variables in Python
payload["id"] = payload_id
payload["name"] = payload_name
payload["type"] = payload_type
payload["value"] = payload_value
payload["date"] = payload_datetime
If I print now
payload["id"]
everything works.
But if I print the whole
print payload
Then the value is not changed. But the value has been added. Than I have two of each. One empty and one with the correct value.
You should go down to the correct level i.e. payload["data"][0]["id"] instead of payload["id"]
Thus, it should be :
payload["data"][0]["id"] = payload_id
payload["data"][0]["name"] = payload_name
payload["data"][0]["type"] = payload_type
payload["data"][0]["data"][0]["value"] = payload_value
payload["data"][0]["data"][0]["date"] = payload_datetime
The id key is part of a dictionary that is a member of a list that contains another dictionary that you must access with the data key.
payload["data"][0]["id"]
I have managed to get the json data from a API and pull the section I needed. I wanted to parse just the public url section only as the output. This is my code:
import json
import httplib as client
headers = {"Content-type": "application/json"}
params = json.dumps({"auth": {"passwordCredentials": {"username": "myusername", "password": "mypassword"}}})
conn = client.HTTPSConnection("lon.identity.api.rackspacecloud.com:443")
conn.request("POST", "/v2.0/tokens", params, headers)
response = conn.getresponse()
output = response.read()
data = json.loads(output)
token = data['access']['serviceCatalog']
print(token)
This is the output data:
[{"endpoints": [{"region": "LON","publicURL": "https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_bfe4a309-40cf-49b8-acd0-d15f4865542f3","internalURL": "https://snet-storage101.lon3.clouddrive.com/v1/MossoCloudFS_bfe4a309-40cf-49b8-acd0-d15f48vdwds3","tenantId": "MossoCloudFS_bfe4a309-40cf-49b8-acd0-d15f486e82f3"}],"type": "object-store","name": "cloudFiles"}]
Now in my code I have tried:
token = data['access']['serviceCatalog']['endpoints']['publicURL']
However I get: TypeError: list indices must be integers, not str
How can I parse the publicURL?
Many thanks
data['access']['serviceCatalog'] doesn't contain a dict directly, but instead a list whose sole element is a dict. Similarly the endpoints dict's value is a list whose sole member is a dict. Try this instead:
token = data['access']['serviceCatalog'][0]['endpoints'][0]['publicURL']
Answering the question from the comment below, if there are multiple endpoints you can use a list comprehension to get all the public URLs.
urls = [x['endpoints'][0]['publicURL'] for x in data['access']['serviceCatalog']]
data is a list (with one element which happens to be a dictionary), but you are trying to use it like a dictionary.
>>> type(data)
<class 'list'>
Try
data = data[0]
>>> type(data)
<class 'dict'>
The output data is a array, you need access to that data with a index.
In this case is:
token = data[0][xxx][xxx]
Though with the output data, the correct syntax for access to data of "publicURL" is:
data[0]["endpoints"][0]["publicURL"]
Example:
>>> data = [{"endpoints": [{"region": "LON","publicURL": "https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_bfe4a309-40cf-49b8-acd0-d15f4865542f3","internalURL": "https://snet-storage101.lon3.clouddrive.com/v1/MossoCloudFS_bfe4a309-40cf-49b8-acd0-d15f48vdwds3","tenantId": "MossoCloudFS_bfe4a309-40cf-49b8-acd0-d15f486e82f3"}],"type": "object-store","name": "cloudFiles"}]
>>> data[0]["endpoints"][0]["publicURL"]
'https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_bfe4a309-40cf-49b8-acd0-d15f4865542f3'