I have JSON objects coming in:
{"Name":"server1","NodeID":1063,"RowID":"3","Vendor":"HP","Load":"41"},
{"Name":"server2","NodeID":1064,"RowID":"7","Vendor":"HP","Load":"12"},
{"Name":"server82","NodeID":1064,"RowID":12","Vendor":"HP","Load":"2"},
{"Name":"server242","NodeID":1064,"RowID":"4","Vendor":"HP","Load":"1"},
{"Name":"server572","NodeID":1064,"RowID":"4","Vendor":"HP","Load":"44"},
{"Name":"server8","NodeID":1064,"RowID":"2","Vendor":"HP","Load":"23"},
{"Name":"server8","NodeID":1064,"RowID":"7","Vendor":"HP","Load":"1"},
...
And I am outputting a different format, here is the dict before I feed it to urllib.request.
import json
import urllib.request
machine = {}
machine['_type'] = "general_data"
machine['_system'] = "LinuxDataCpu"
machine['_device'] = i['Name']
machine['CPUload1'] = i['Load']
If this was a bash script, CPUload$RowID would likely form the basis of the solution.
I have some machines with 1 CPU, some with 8, and some with 12.
I have a JSON object machine() that will be created and pushed out in a post.
Each JSON object will contain only one CPU value, so the key needs to reflect which cpu is being reported (1 to 12).
How do I use a variable for a key such that I can indicate a different key name for each CPU?
So the first line for server1 would get:
machine['CPUload3'] = i['Load']
And the second line gets:
machine['CPUload7'] = i['Load']
For the respective output key: value pair. Each line only reports one row at a time, so a 12 cpu machine would randomly send reports listing individual cpu's. On the back end nothing is random, but the raw sql generating the data is not ordering the data in any way. The data is actually a large block of data, but the API I am sending the json to can only take one key value pair for the data. it should let me pass the whole json object, but decisions were made out of my control, and one payload variable is allowd per post.
Michael Butscher offers: "['CPUload' + i['RowID']]"... this was what I was looking for. (yeah I'm a rookie hack with python).
Type it up Michael and I'll mark it to give you the credit.
Thanks everyone!!
Just create a string variable, pass the incremental server number via a loop and pass the string variable into the key.
Something like:
incrementalCPUnumber = 1
key = 'CPUload{0}'.format(incrementalCPUnumber)
machine[x] = i['Load']
I think you'll need to clarify more, but on face value, this is what it seems like you are trying to do.
Related
I am new to python and wanted to store the recentAveragePrice inside a variable (from a string like this one)
{"assetStock":null,"sales":250694,"numberRemaining":null,"recentAveragePrice":731,"originalPrice":null,"priceDataPoints":[{"value":661,"date":"2022-08-11T05:00:00Z"},{"value":592,"date":"2022-08-10T05:00:00Z"},{"value":443,"date":"2022-08-09T05:00:00Z"}],"volumeDataPoints":[{"value":155,"date":"2022-08-11T05:00:00Z"},{"value":4595,"date":"2022-08-10T05:00:00Z"},{"value":12675,"date":"2022-08-09T05:00:00Z"},{"value":22179,"date":"2022-08-08T05:00:00Z"},{"value":15181,"date":"2022-08-07T05:00:00Z"},{"value":14541,"date":"2022-08-06T05:00:00Z"},{"value":15310,"date":"2022-08-05T05:00:00Z"},{"value":14146,"date":"2022-08-04T05:00:00Z"},{"value":13083,"date":"2022-08-03T05:00:00Z"},{"value":14460,"date":"2022-08-02T05:00:00Z"},{"value":16809,"date":"2022-08-01T05:00:00Z"},{"value":17571,"date":"2022-07-31T05:00:00Z"},{"value":23907,"date":"2022-07-30T05:00:00Z"},{"value":39007,"date":"2022-07-29T05:00:00Z"},{"value":38823,"date":"2022-07-28T05:00:00Z"}]}
My current solution is this:
var = sampleStr[78] + sampleStr[79] + sampleStr[80]
It works for the current string but if the recentAveragePrice was above 999 it would stop working and i was wondering if instead of getting a fixed number i could search for it inside the string.
Your replit code shows that you're acquiring JSON data from some website. Here's an example based on the URL that you're using. It shows how you check the response status, acquire the JSON data as a Python dictionary then print a value associated with a particular key. If the key is missing, it will print None:
import requests
(r := requests.get('https://economy.roblox.com/v1/assets/10159617728/resale-data')).raise_for_status()
jdata = r.json()
print(jdata.get('recentAveragePrice'))
Output:
640
Since this is json you should just be able to parse it and access recentAveragePrice:
import json
sample_string = '''{"assetStock":null,"sales":250694,"numberRemaining":null,"recentAveragePrice":731,"originalPrice":null,"priceDataPoints":[{"value":661,"date":"2022-08-11T05:00:00Z"},{"value":592,"date":"2022-08-10T05:00:00Z"},{"value":443,"date":"2022-08-09T05:00:00Z"}],"volumeDataPoints":[{"value":155,"date":"2022-08-11T05:00:00Z"},{"value":4595,"date":"2022-08-10T05:00:00Z"},{"value":12675,"date":"2022-08-09T05:00:00Z"},{"value":22179,"date":"2022-08-08T05:00:00Z"},{"value":15181,"date":"2022-08-07T05:00:00Z"},{"value":14541,"date":"2022-08-06T05:00:00Z"},{"value":15310,"date":"2022-08-05T05:00:00Z"},{"value":14146,"date":"2022-08-04T05:00:00Z"},{"value":13083,"date":"2022-08-03T05:00:00Z"},{"value":14460,"date":"2022-08-02T05:00:00Z"},{"value":16809,"date":"2022-08-01T05:00:00Z"},{"value":17571,"date":"2022-07-31T05:00:00Z"},{"value":23907,"date":"2022-07-30T05:00:00Z"},{"value":39007,"date":"2022-07-29T05:00:00Z"},{"value":38823,"date":"2022-07-28T05:00:00Z"}]}'''
data = json.loads(sample_string)
recent_price = data['recentAveragePrice']
print(recent_price)
outputs:
731
Your data is in a popular format called JSON (JavaScript Object Notation). It's commonly used to exchange data between different systems like a server and a client, or a Python program and JavaScript program.
Now Python doesn't use JSON per-se, but it has a data type called a dictionary that behaves very similarly to JSON. You can access elements of a dictionary as simply as:
print(my_dictionary["recentAveragePrice"])
Python has a built-in library meant specifically to handle JSON data, and it includes a function called loads() that can convert a string into a Python dictionary. We'll use that.
Finally, putting all that together, here is a more robust program to help parse your string and pick out the data you need. Dictionaries can do a lot more cool stuff, so make sure you take a look at the links above.
# import the JSON library
# specifically, we import the `loads()` function, which will convert a JSON string into a Python object
from json import loads
# let's store your string in a variable
original_string = """
{"assetStock":null,"sales":250694,"numberRemaining":null,"recentAveragePrice":731,"originalPrice":null,"priceDataPoints":[{"value":661,"date":"2022-08-11T05:00:00Z"},{"value":592,"date":"2022-08-10T05:00:00Z"},{"value":443,"date":"2022-08-09T05:00:00Z"}],"volumeDataPoints":[{"value":155,"date":"2022-08-11T05:00:00Z"},{"value":4595,"date":"2022-08-10T05:00:00Z"},{"value":12675,"date":"2022-08-09T05:00:00Z"},{"value":22179,"date":"2022-08-08T05:00:00Z"},{"value":15181,"date":"2022-08-07T05:00:00Z"},{"value":14541,"date":"2022-08-06T05:00:00Z"},{"value":15310,"date":"2022-08-05T05:00:00Z"},{"value":14146,"date":"2022-08-04T05:00:00Z"},{"value":13083,"date":"2022-08-03T05:00:00Z"},{"value":14460,"date":"2022-08-02T05:00:00Z"},{"value":16809,"date":"2022-08-01T05:00:00Z"},{"value":17571,"date":"2022-07-31T05:00:00Z"},{"value":23907,"date":"2022-07-30T05:00:00Z"},{"value":39007,"date":"2022-07-29T05:00:00Z"},{"value":38823,"date":"2022-07-28T05:00:00Z"}]}
"""
# convert the string into a dictionary object
dictionary_object = loads(original_string)
# access the element you need
print(dictionary_object["recentAveragePrice"])
Output upon running this program:
$ python exp.py
731
Currently, I am working on a Boot Sequence in Python for a larger project. For this specific part of the sequence, I need to access a .JSON file (specs.json), establish it as a dictionary in the main program. I then need to take a value from the .JSON file, and add 1 to it, using it's key to find the value. Once that's done, I need to push the changes to the .JSON file. Yet, every time I run the code below, I get the error:
bootNum = spcInfDat['boot_num']
KeyError: 'boot_num'`
Here's the code I currently have:
(Note: I'm using the Python json library, and have imported dumps, dump, and load.)
# Opening of the JSON files
spcInf = open('mki/data/json/specs.json',) # .JSON file that contains the current system's specifications. Not quite needed, but it may make a nice reference?
spcInfDat = load(spcInf)
This code is later followed by this, where I attempt to assign the value to a variable by using it's dictionary key (The for statement was a debug statement, so I could visibly see the Key):
for i in spcInfDat['spec']:
print(CBL + str(i) + CEN)
# Loacting and increasing the value of bootNum.
bootNum = spcInfDat['boot_num']
print(str(bootNum))
bootNum = bootNum + 1
(Another Note: CBL and CEN are just variables I use to colour text I send to the terminal.)
This is the interior of specs.json:
{
"spec": [
{
"os":"name",
"os_type":"getwindowsversion",
"lang":"en",
"cpu_amt":"cpu_count",
"storage_amt":"unk",
"boot_num":1
}
]
}
I'm relatively new with .JSON files, as well as using the Python json library; I only have experience with them through some GeeksforGeeks tutorials I found. There is a rather good chance that I just don't know how .JSON files work in conjunction with the library, but I figure that it would still be worth a shot to check here. The GeeksForGeeks tutorial had no documentation about this, as well as there being minimal I know about how this works, so I'm lost. I've tried searching here, and have found nothing.
Issue Number 2
Now, the prior part works. But, when I attempt to run the code on the following lines:
# Changing the values of specDict.
print(CBL + "Changing values of specDict... 50%" + CEN)
specDict ={
"os":name,
"os_type":ost,
"lang":"en",
"cpu_amt":cr,
"storage_amt":"unk",
"boot_num":bootNum
}
# Writing the product of makeSpec to `specs.json`.
print(CBL + "Writing makeSpec() result to `specs.json`... 75%" + CEN)
jsonobj = dumps(specDict, indent = 4)
with open('mki/data/json/specs.json', "w") as outfile:
dump(jsonobj, outfile)
I get the error:
TypeError: Object of type builtin_function_or_method is not JSON serializable.
Is there a chance that I set up my dictionary incorrectly, or am I using the dump function incorrectly?
You can show the data using:
print(spcInfData)
This shows it to be a dictionary, whose single entry 'spec' has an array, whose zero'th element is a sub-dictionary, whose 'boot_num' entry is an integer.
{'spec': [{'os': 'name', 'os_type': 'getwindowsversion', 'lang': 'en', 'cpu_amt': 'cpu_count', 'storage_amt': 'unk', 'boot_num': 1}]}
So what you are looking for is
boot_num = spcInfData['spec'][0]['boot_num']
and note that the value obtained this way is already an integer. str() is not necessary.
It's also good practice to guard against file format errors so the program handles them gracefully.
try:
boot_num = spcInfData['spec'][0]['boot_num']
except (KeyError, IndexError):
print('Database is corrupt')
Issue Number 2
"Not serializable" means there is something somewhere in your data structure that is not an accepted type and can't be converted to a JSON string.
json.dump() only processes certain types such as strings, dictionaries, and integers. That includes all of the objects that are nested within sub-dictionaries, sub-arrays, etc. See documentation for json.JSONEncoder for a complete list of allowable types.
I am setting up a weather camera which will provide a live stream of the current conditions outside, but I also would like to overlay continuously updated weather conditions (temperature, wind speed/direction, current weather) from a local National Weather Service weather station, from a browser API source provided in JSON format.
I have had success extracting the desired values from a different API source using a Python script I wrote; however long story short that API source is unreliable. Therefore I am using API from the official National Weather Service ASOS station at my nearby airport. The output from the new API source I am polling from is rather complicated, however, with various tiers of indentation. I have not worked with Python very long and tutorials and guides online have either been for other languages (Java or C++ mostly) or have not worked for my specific case.
First off, here is the structure of the JSON that I am receiving:
I underlined the values I am trying to extract. They are listed under the OBSERVATIONS section, associated with precip_accum_24_hour_value_1, wind_gust_value_1, wind_cardinal_direction_value_1d, and so on. The issue is there are two values underneath each observation so the script I have tried isn't returning the values I want. Here is the code I have tried:
import urllib.request
import json
f = urllib.request.urlopen('https://api.synopticdata.com/v2/stations/latest?token=8c96805fbf854373bc4b492bb3439a67&stid=KSTC&complete=1&units=english&output=json')
json_string = f.read()
parsed_json = json.loads(json_string)
for each in parsed_json['STATION']:
observations = each['OBSERVATIONS']
print(observations)
This prints out everything underneath the OBSERVATIONS in the JSON as expected, as one long string.
{'precip_accum_24_hour_value_1': {'date_time': '2018-12-06T11:53:00Z', 'value': 0.01}, 'wind_gust_value_1': {'date_time': '2018-12-12T01:35:00Z', 'value': 14.0},
to show a small snippet of the output I am receiving. I was hoping I could individually extract the values I want from this string, but everything I have attempted is not working. I would really appreciate some guidance for finishing this piece of code so I can return the values I am looking for. I realize it may be some kind of loop or special syntax.
Try something like this:
for each in parsed_json['STATION']:
observations = each['OBSERVATIONS']
for k, v in observations.items():
print(k, v["value"])
JSON maps well into python's dictionary and list types, so accessing substructures can be done with a[<index-or-key>] syntax. Iteration over key-value pairs of a dictionary can be done as I've shown above. If you're not familiar with dictionaries in python yet, I'd recommend reading about them. Searching online should yield a lot of good tutorials.
Does this help?
When you say the JSON is complicated, it really is just nested dictionaries within the main JSON response. You would access them in the same way as you would the initial JSON blob:
import urllib.request
import json
f = urllib.request.urlopen('https://api.synopticdata.com/v2/stations/latest?token=8c96805fbf854373bc4b492bb3439a67&stid=KSTC&complete=1&units=english&output=json')
json_string = f.read()
parsed_json = json.loads(json_string)
for each in parsed_json['STATION']:
for value in each:
print(value, each[value])
A bit lost after much research. My code below parses the JSON to a dictionary I have thought using json load
response = json.load(MSW) # -->Will take a JSON String & Turn it into a python dict
Using the iteration below I return a series like this which is fine
{u'swell': {u'components': {u'primary': {u'direction': 222.5}}}}
{u'swell': {u'components': {u'primary': {u'direction': 221.94}}}}
ourResult = response
for rs in ourResult:
print rs
But how oh how do I access the 222.5 value. The above appears to just be one long string eg response[1] and not a dictionary structure at all.
In short all I need is the numerical value (which I assume is a part of that sting) so I can test conditions in the rest of my code. Is is a dictionary? With thanks as new and lost
You have to use python syntax as follows:
>>> print response['swell']['components']['primary']['direction']
222.5
Just access the nested dictionaries, unwrapping each layer with an additional key:
for rs in ourResult:
print rs['components']['primary']['direction']
I'm trying to load JSON back into an object. The "loads" method seems to work without error, but the object doesn't seem to have the properties I expect.
How can I go about examining/inspecting the object that I have (this is web-based code).
results = {"Subscriber": {"firstname": "Neal", "lastname": "Walters"}}
subscriber = json.loads(results)
for item in inspect.getmembers(subscriber):
self.response.out.write("<BR>Item")
for subitem in item:
self.response.out.write("<BR> SubItem=" + subitem)
The attempt above returned this:
Item
SubItem=__class__
I don't think it matters, but for context:
The JSON is actually coming from a urlfetch in Google App Engine to
a rest web service created using this utility:
http://code.google.com/p/appengine-rest-server.
The data is being retrieved from a datastore with this definition:
class Subscriber(db.Model):
firstname = db.StringProperty()
lastname = db.StringProperty()
Thanks,
Neal
Update #1: Basically I'm trying to deserialize JSON back into an object.
In theory it was serialized from an object, and I want to now get it back into an object.
Maybe the better question is how to do that?
Update #2: I was trying to abstract a complex program down to a few lines of code, so I made a few mistakes in "pseudo-coding" it for purposes of posting here.
Here's a better code sample, now take out of website where I can run on PC.
results = '{"Subscriber": {"firstname": "Neal", "lastname": "Walters"}}'
subscriber = json.loads(results)
for key, value in subscriber.items():
print " %s: %s" %(key, value)
The above runs, what it displays doesn't look any more structured than the JSON string itself. It displays this:
Subscriber: {u'lastname': u'Walters', u'firstname': u'Neal'}
I have more of a Microsoft background, so when I hear serialize/deserialize, I think going from an object to a string, and from a string back to an object. So if I serialize to JSON, and then deserialize, what do I get, a dictionary, a list, or an object? Actually, I'm getting the JSON from a REST webmethod, that is on my behalf serializing my object for me.
Ideally I want a subscriber object that matches my Subscriber class above, and ideally, I don't want to write one-off custom code (i.e. code that would be specific to "Subscriber"), because I would like to do the same thing with dozens of other classes. If I have to write some custom code, I will need to do it generically so it will work with any class.
Update #3: This is to explain more of why I think this is a needed tool. I'm writing a huge app, probably on Google App Engine (GAE). We are leaning toward a REST architecture for several reasons, but one is that our web GUI should access the data store via a REST web layer. (I'm a lot more used to SOAP, so switching to REST is a small challenge in itself). So one of the classic ways of getting and update data is through a business or data tier. By using the REST utility mention above, I have the choice of XML or JSON. I'm hoping to do a small working prototype of both before we develop the huge app). Then, suppose we have a successful app, and GAE doubles it prices. Then we can rewrite just the data tier, and take our Python/Django user tier (web code), and run it on Amazon or somewhere else.
If I'm going to do all that, why would I want everything to be dictionary objects. Wouldn't I want the power of full-blown class structure? One of the next tricks is sort of an object relational mapping (ORM) so that we don't necessarily expose our exact data tables, but more of a logical layer.
We also want to expose a RESTful API to paying users, who might be using any language. For them, they can use XML or JSON, and they wouldn't use the serialize routine discussed here.
json only encodes strings, floats, integers, javascript objects (python dicts) and lists.
You have to create a function to turn the returned dictionary into a class and then pass it to a json.loads using the object_hook keyword argument along with the json string. Heres some code that fleshes it out:
import json
class Subscriber(object):
firstname = None
lastname = None
class Post(object):
author = None
title = None
def decode_from_dict(cls,vals):
obj = cls()
for key, val in vals.items():
setattr(obj, key, val)
return obj
SERIALIZABLE_CLASSES = {'Subscriber': Subscriber,
'Post': Post}
def decode_object(d):
for field in d:
if field in SERIALIZABLE_CLASSES:
cls = SERIALIZABLE_CLASSES[field]
return decode_from_dict(cls, d[field])
return d
results = '''[{"Subscriber": {"firstname": "Neal", "lastname": "Walters"}},
{"Post": {"author": {"Subscriber": {"firstname": "Neal",
"lastname": "Walters"}}},
"title": "Decoding JSON Objects"}]'''
result = json.loads(results, object_hook=decode_object)
print result
print result[1].author
This will handle any class that can be instantiated without arguments to the constructor and for which setattr will work.
Also, this uses json. I have no experience with simplejson so YMMV but I hear that they are identical.
Note that although the values for the two subscriber objects are identical, the resulting objects are not. This could be fixed by memoizing the decode_from_dict class.
results in your snippet is a dict, not a string, so the json.loads would raise an exception. If that is fixed, each subitem in the inner loop is then a tuple, so trying to add it to a string as you are doing would raise another exception. I guess you've simplified your code, but the two type errors should already show that you simplified it too much (and incorrectly). Why not use an (equally simplified) working snippet, and the actual string you want to json.loads instead of one that can't possibly reproduce your problem? That course of action would make it much easier to help you.
Beyyond peering at the actual string, and showing some obvious information such as type(subscriber), it's hard to offer much more help based on that clearly-broken code and such insufficient information:-(.
Edit: in "update2", the OP says
It displays this: Subscriber: {u'lastname': u'Walters', u'firstname': u'Neal'}
...and what else could it possibly display, pray?! You're printing the key as string, then the value as string -- the key is a string, and the value is another dict, so of course it's "stringified" (and all strings in JSON are Unicode -- just like in C# or Java, and you say you come from a MSFT background, so why does this surprise you at all?!). str(somedict), identically to repr(somedict), shows the repr of keys and values (with braces around it all and colons and commas as appropriate separators).
JSON, a completely language-independent serialization format though originally centered on Javascript, has absolutely no idea of what classes (if any) you expect to see instances of (of course it doesn't, and it's just absurd to think it possibly could: how could it possibly be language-independent if it hard-coded the very concept of "class", a concept which so many languages, including Javascript, don't even have?!) -- so it uses (in Python terms) strings, numbers, lists, and dicts (four very basic data types that any semi-decent modern language can be expected to have, at least in some library if not embedded in the language proper!). When you json.loads a string, you'll always get some nested combination of the four datatypes above (all strings will be unicode and all numbers will be floats, BTW;-).
If you have no idea (and don't want to encode by some arbitrary convention or other) what class's instances are being serialized, but absolutely must have class instances back (not just dicts etc) when you deserialize, JSON per se can't help you -- that metainformation cannot possibly be present in the JSON-serialized string itself.
If you're OK with the four fundamental types, and just want to see some printed results that you consider "prettier" than the default Python string printing of the fundamental types in question, you'll have to code your own recursive pretty-printing function depending on your subjective definition of "pretty" (I doubt you'd like Python's own pprint standard library module any more than you like your current results;-).
My guess is that loads is returning a dictionary. To iterate over its content, use something like:
for key, value in subscriber.items():
self.response.out.write("%s: %s" %(key, value))