So i want to be able to pull data based on a certain condition, from this data i then want to be able to print multiple items from this query...here's what ive done so far:
def rec():
qe = JsonQ(r"C:\ShopFloor\data.json")
res = qe.at('data').where('Status', '=', 1).get()
for Process, Shortnum in res:
print(Process['Process'] + " " + Shortnum['Shortnum'])
rec()
this is from the following json file:
{
"data": [
{
"Shortnum": "34567",
"Process": "SPA",
"Status": 1,
"Start_Time": "2016-12-14 15:54:35",
"Finish_Time": "2016-12-14 15:56:02"
},
{
"Shortnum": "34567",
"Process": "Figure",
"Status": 0,
"Start_Time": "2016-12-08 15:34:05",
"Finish_Time": "2016-12-08 15:34:09"
},
How can I get this to work properly? Ideally I am looking for this kind of response from the print:
SPA 34567
cannot get a current output, i get this error... i realise I am passing too many arguments however i couldn't think of a proper way to do it...
Exception has occurred: ValueError
too many values to unpack (expected 2)
File "C:\ShopFloor\main.py", line 101, in rec
for Process, Shortnum in res:
File "C:\ShopFloor\main.py", line 106, in <module>
rec()
The typical approach to working with JSON in Python is to load the JSON object as a Python dict:
import json
with open('C:/ShopFloor/data.json', 'r') as f:
qe = json.load(f)
for item in qe['data']:
if item['Status'] == 1:
print(f'{item["Process"]} {item["Shortnum"]}')
Note this uses Python 3's f-strings (be sure to alternate single and double quotes when accessing dictionary values in an f-string). In Python 2, replace last line with:
print('{} {}'.format(item['Process'], item['Shortnum']))
Related
I have .json documents generated from the same code. Here multiple nested dicts are being dumped to the json documents. While loadling with json.load(opened_json), I get the json.decoder.JSONDecodeError: Extra data: line 30 column 2 (char 590) like error for some of of the files whereas not for others. It is not understood why. What is the proper way to dump multiple dicts (maybe nested) into json docs and in my current case what is way to read them all? (Extra: Dicts can be over multiple lines, so 'linesplitting' does not work probably.)
Ex: Say I am json.dump(data, file) with data = {'meta_data':{some_data}, 'real_data':{more_data}}.
Let us take these two fake files:
{
"meta_data": {
"id": 0,
"start": 1238397024.0,
"end": 1238397056.0,
"best": []
},
"real_data": {
"YAS": {
"t1": [
1238397047.2182617
],
"v1": [
5.0438767766574255
],
"v2": [
4.371670270544587
]
}
}
}
and
{
"meta_data": {
"id": 0,
"start": 1238397056.0,
"end": 1238397088.0,
"best": []
},
"real_data": {
"XAS": {
"t1": [
1238397047.2182617
],
"v1": [
5.0438767766574255
],
"v2": [
4.371670270544587
]
}
}
}
and try to load them using json.load(open(file_path)) for duplicatling the problem.
You chose not to offer a
reprex.
Here is the code I'm running
which is intended to represent what you're running.
If there is some discrepancy, update the original
question to clarify the details.
import json
from io import StringIO
some_data = dict(a=1)
more_data = dict(b=2)
data = {"meta_data": some_data, "real_data": more_data}
file = StringIO()
json.dump(data, file)
file.seek(0)
d = json.load(file)
print(json.dumps(d, indent=4))
output
{
"meta_data": {
"a": 1
},
"real_data": {
"b": 2
}
}
As is apparent, over the circumstances you have
described the JSON library does exactly what we
would expect of it.
EDIT
Your screenshot makes it pretty clear
that a bunch of ASCII NUL characters are appended
to the 1st file.
We can easily reproduce that JSONDecodeError: Extra data
symptom by adding a single line:
json.dump(data, file)
file.write(chr(0))
(Or perhaps chr(0) * 80 more closely matches the truncated screenshot.)
If your file ends with extraneous characters, such as NUL,
then it will no longer be valid JSON and compliant
parsers will report a diagnostic message when they
attempt to read it.
And there's nothing special about NUL, as a simple
file.write("X") suffices to produce that same
diagnostic.
You will need to trim those NULs from the file's end
before attempting to parse it.
For best results, use UTF8 unicode encoding with no
BOM.
Your editor should have settings for
switching to utf8.
Use $ file foo.json to verify encoding details,
and $ iconv --to-code=UTF-8 < foo.json
to alter an unfortunate encoding.
You need to read the file, you can do both of these.
data = json.loads(open("data.json").read())
or
with open("data.json", "r") as file:
data = json.load(file)
I had .xlsx file, and I converted it to JSON. I am using python to get the data from this json file. I am able for example to search for Build# and then I get the corresponding level, but when I search for values in, for example, "14H0232" or "14H4812" it throws a KeyError.
'''
import json
f = open('try.json')
data = json.load(f)
input= input('Enter the value: ')
for i in data['F6']:
if i['14H0232'] == input:
print(i['LEVEL'])
f.close()
'''
A Snippet of the json file.
'''
{
"F6": [
{
"LEVEL": "2.0.6.0",
"ID": "dataID",
"Build#": "9",
"prod/dev": "prod ",
"14H4812": "data1\r\ndata2",
"14H4826": "data",
"14H4813": "data1\r\ndata2"
}
],
"F5": [
{
"LEVEL": "2.0.5.1",
"ID": "dataID",
"Build#": "18",
"prod/dev": "prod",
"14H0232": "data1: data1\r\ndata2: data2\r\ndata3: data3",
"14H12321": "data1\r\ndata2"
}
],
"F4": [
{
"LEVEL": "2.0.4.1",
"ID": "dataID",
"Build#": "18",
"prod/dev": "prod",
"14H0232": "data1: data1\r\ndata2: data2\r\ndata3: data3",
"14H12321": "data1\r\ndata2"
}
]
}
'''
The problem is in your loop. When you are trying to access the value for "14H0232" it just doesnt exist in your json file. The case for 'Build#' is different I assume, as the key is always there. The example you shared is also not showing that F6 has a key with that id you specified.
So to avoid this kind of errors you can put your 'if' statement in a try block and catch the error.
try:
if i['14H0232'] == input:
print(i['LEVEL'])
except KeyError:
print("The key is not found but the code continues to execute")
I am getting the above error when trying to parse a JSON file.
Code:
import json
data = open('output.json').read()
for host in data['ASSET_DATA_REPORT']['HOST_LIST']['HOST']:
print(host['IMAGE_ID'])
Traceback:
Traceback (most recent call last):
File "json_format.py", line 11, in <module>
for host in data['ASSET_DATA_REPORT']['HOST_LIST']['HOST']:
TypeError: string indices must be integers, not str
JSON:
{
"ASSET_DATA_REPORT": {
"HOST_LIST": {
"HOST": [
{
"IP": {
"network_id": "0"
},
"TRACKING_METHOD": "EC2",
"ASSET_TAGS": {
"ASSET_TAG": [
"EC2 Running",
"IF - Database - MySQL",
]
},
"DNS": "i-xxxxxxx",
"EC2_INSTANCE_ID": "i-xxxxxx",
"EC2_INFO": {
"PUBLIC_DNS_NAME": "ec2-xxxxxxxx.amazonaws.com",
"IMAGE_ID": "ami-xxxxxx",
"VPC_ID": "vpc-xxxxxx",
"INSTANCE_STATE": "RUNNING",
"PRIVATE_DNS_NAME": "ip-xxxx.ec2.internal",
"INSTANCE_TYPE": "m3.xlarge"
}
}
]
}
}
}
It seems like host is a string for some reason and I'm not sure how to overcome this error.
Importing json is not enough. data = open('output.json').read() just treats it as any other file.
TypeError: string indices must be integers, not str is not complaining about the 'HOST' key; data['ASSET_DATA_REPORT'] on its own won't be valid either because the whole thing is a string.
Try:
with open('output.json') as infile:
data = json.load(infile)
As pointed out by #Milton Arango G there is an error in the JSON you posted. Change:
"IF - Database - MySQL",
to:
"IF - Database - MySQL"
After that, you can obtain the 'IMAGE_ID' field with:
print(data['ASSET_DATA_REPORT']['HOST_LIST']['HOST'][0]['EC2_INFO']['IMAGE_ID'])
You have a couple problems, some in your code, some in your JSON.
First, the JSON --- you have an extra comma after the last list entry:
"ASSET_TAG": [
"EC2 Running",
"IF - Database - MySQL",
]
Your code has two problems.
First is that you never convert the contents of the file to JSON --- it remains a string:
data = open('output.json').read()
You want something like
roganjosh already described:
with open('output.json') as f:
data = json.load(f)
Your followup problem is that the structure of the JSON doesn't match your code.
'IMAGE_ID' isn't a key in the
(unnamed)
dictionary stored in the 'HOST' list --- it's a key of the 'EC2_INFO' dictionary, which is contained inside that nameless dictionary.
This:
print(host['IMAGE_ID'])
Should be something like:
print(host['EC2_INFO']['IMAGE_ID'])
The output is a string:
ami-xxxxxx
That is not a good way to open a json file.
open('output.json').read()
return your file as string.
A better way is :
import json
with open('output.json', 'r') as my_file:
data = json.load(my_file)
for host in data['ASSET_DATA_REPORT']['HOST_LIST']['HOST']:
print(host['IMAGE_ID'])
I've got a json file that I've pulled from a web service and am trying to parse it. I see that this question has been asked a whole bunch, and I've read whatever I could find, but the json data in each example appears to be very simplistic in nature. Likewise, the json example data in the python docs is very simple and does not reflect what I'm trying to work with. Here is what the json looks like:
{"RecordResponse": {
"Id": blah
"Status": {
"state": "complete",
"datetime": "2016-01-01 01:00"
},
"Results": {
"resultNumber": "500",
"Summary": [
{
"Type": "blah",
"Size": "10000000000",
"OtherStuff": {
"valueOne": "first",
"valueTwo": "second"
},
"fieldIWant": "value i want is here"
The code block in question is:
jsonFile = r'C:\Temp\results.json'
with open(jsonFile, 'w') as dataFile:
json_obj = json.load(dataFile)
for i in json_obj["Summary"]:
print(i["fieldIWant"])
Not only am I not getting into the field I want, but I'm also getting a key error on trying to suss out "Summary".
I don't know how the indices work within the array; once I even get into the "Summary" field, do I have to issue an index manually to return the value from the field I need?
The example you posted is not valid JSON (no commas after object fields), so it's hard to dig in much. If it's straight from the web service, something's messed up. If you did fix it with proper commas, the "Summary" key is within the "Results" object, so you'd need to change your loop to
with open(jsonFile, 'w') as dataFile:
json_obj = json.load(dataFile)
for i in json_obj["Results"]["Summary"]:
print(i["fieldIWant"])
If you don't know the structure at all, you could look through the resulting object recursively:
def findfieldsiwant(obj, keyname="Summary", fieldname="fieldIWant"):
try:
for key,val in obj.items():
if key == keyname:
return [ d[fieldname] for d in val ]
else:
sub = findfieldsiwant(val)
if sub:
return sub
except AttributeError: #obj is not a dict
pass
#keyname not found
return None
I am having trouble with converting a dictionary to a string in python. I am trying to extract the information from one of my variables but cannot seem to remove the square brackets surrounding the information
for line in str(object):
if line.startswith ('['):
new_object = object.replace('[', '')
Is there a way to remove the square brackets or do I have to find another way of taking the information out of the dictionary?
Edit:
in more detail what i am trying to do here is the following
import requests
city = 'dublin'
country = 'ireland'
info = requests.get('http://api.openweathermap.org/data/2.5/weather?q='+city +','+ country +'&mode=json')
weather = info.json()['weather']
fh = open('/home/Ricky92d3/city.txt', 'w')
fh.write(str(weather))
fh.close()
fl = open('/home/Ricky92d3/city.txt')
Object = fl.read()
fl.close()
for line in str(Object):
if line.startswith ('['):
new_Object = Object.replace('[', '')
if line.startswith ('{'):
new_Object = Object.replace('{u', '')
print new_Object
i hope this makes what i am trying to do a little more clear
The object returned by info.json() is a Python dictionary, so you can access its contents using normal Python syntax. I admit that it can get a little bit tricky, since JSON dictionaries often contain other dictionaries and lists, but it's generally not too hard to figure out what's what if you print the JSON object out in a nicely formatted way. The easiest way to do that is by using the dumps() function in the standard Python json module.
The code below retrieves the JSON data into a dict called data.
It then prints the 'description' string from the list in the 'weather' item of data.
It then saves all the data (not just the 'weather' item) as an ASCII-encoded JSON file.
It then reads the JSON data back in again to a new dict called newdata, and pretty-prints it.
Finally, it prints the weather description again, to verify that we got back what we saw earlier. :)
import requests, json
#The base URL of the weather service
endpoint = 'http://api.openweathermap.org/data/2.5/weather'
#Filename for saving JSON data to
fname = 'data.json'
city = 'dublin'
country = 'ireland'
params = {
'q': '%s,%s' % (city, country),
'mode': 'json',
}
#Fetch the info
info = requests.get(endpoint, params=params)
data = info.json()
#print json.dumps(data, indent=4)
#Extract the value of 'description' from the list in 'weather'
print '\ndescription: %s\n' % data['weather'][0]['description']
#Save data
with open(fname, 'w') as f:
json.dump(data, f, indent=4)
#Reload data
with open(fname, 'r') as f:
newdata = json.load(f)
#Show all the data we just read in
print json.dumps(newdata, indent=4)
print '\ndescription: %s\n' % data['weather'][0]['description']
output
description: light intensity shower rain
{
"clouds": {
"all": 75
},
"name": "Dublin",
"visibility": 10000,
"sys": {
"country": "IE",
"sunset": 1438374108,
"message": 0.0118,
"type": 1,
"id": 5237,
"sunrise": 1438317600
},
"weather": [
{
"description": "light intensity shower rain",
"main": "Rain",
"id": 520,
"icon": "09d"
}
],
"coord": {
"lat": 53.340000000000003,
"lon": -6.2699999999999996
},
"base": "stations",
"dt": 1438347600,
"main": {
"pressure": 1014,
"humidity": 62,
"temp_max": 288.14999999999998,
"temp": 288.14999999999998,
"temp_min": 288.14999999999998
},
"id": 2964574,
"wind": {
"speed": 8.1999999999999993,
"deg": 210
},
"cod": 200
}
description: light intensity shower rain
I'm not quite sure what you're trying to do here (without seeing your dictionary) but if you have a string like x = "[myString]" you can just do the following:
x = x.replace("[", "").replace("]", "")
If this isn't working, there is a high chance you're actually getting a list returned. Though if that was the case you should see an error like this:
>>> x = [1,2,3]
>>> x.replace("[", "")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'list' object has no attribute 'replace'
Edit 1:
I think there's a misunderstanding of what you're getting back here. If you're just looking for a csv output file with the weather from your api try this:
import requests
import csv
city = 'dublin'
country = 'ireland'
info = requests.get('http://api.openweathermap.org/data/2.5/weather?q='+city +','+ country +'&mode=json')
weather = info.json()['weather']
weather_fieldnames = ["id", "main", "description", "icon"]
with open('city.txt', 'w') as f:
csvwriter = csv.DictWriter(f, fieldnames=weather_fieldnames)
for w in weather:
csvwriter.writerow(w)
This works by looping through the list of items you're getting and using a csv.DictWriter to write it as a row in the csv file.
Bonus
Don't call your dictionary object - It's a reserved word for the core language.