I'm trying to extract data from a JSON file with Python.
Mainly, I want to pull out the date and time from the "Technicals" section, to put that in one column of a dataframe, as well as pulling the "AKG" number and putting that in the 2nd col of the dataframe. Yes, I've looked at similar questions, but this issue is different. Thanks for your help.
A downNdirty example of the JSON file is below:
{ 'Meta Data': { '1: etc'
'2: etc'},
'Technicals': { '2017-05-04 12:00': { 'AKG': '64.8645'},
'2017-05-04 12:30': { 'AKG': '65.7834'},
'2017-05-04 13:00': { 'AKG': '63.2348'}}}
As you can see, and what's stumping me, is while the date stays the same the time advances. 'AKG' never changes, but the number does. Some of the relevant code I've been using is below. I can extract the date and time, but I can't seem to reach the AKG numbers. Note, I don't need the "AKG", just the number.
I'll mention: I'm creating a DataFrame because this will be easier to work with when creating plots with the data...right? I'm open to an array of lists et al, or anything easier, if that will ultimately help me with the plots.
akg_time = []
akg_akg = []
technicals = akg_data['Technicals'] #akg_data is the entire json file
for item in technicals: #this works
akg_time.append(item)
for item in technicals: #this not so much
symbol = item.get('AKG')
akg_akg.append(symbol)
pp.pprint(akg_akg)
error: 'str' object has no attribute 'get'
You've almost got it. You don't even need the second loop. You can append the akg value in the first one itself:
for key in technicals: # renaming to key because that is a clearer name
akg_time.append(key)
akg_akg.append(technicals[key]['AKG'])
Your error is because you believe item (or key) is a dict. It is not. It is just a string, one of the keys of the technicals dictionary, so you'd actually need to use symbols = technicals[key].get('AKG').
Although Coldspeed answer is right: when you have a dictionary you loop through keys and values like this:
Python 3
for key,value in technicals.items():
akg_time.append(key)
akg_akg.append(value["akg"])
Python 2
for key,value in technicals.iteritems():
akg_time.append(key)
akg_akg.append(value["akg"])
Related
cities={
'city1':{
'name':'sydney',
'country':'australia',
'desc':'beautiful'
},
'city2':{
'name':'toronto',
'country':'canada',
'desc':'amazing',
}
}
cities['city3']:"{'name':'Tokyo','country':'japan','desc':'lots of earthquakes’}"
for keys,values in cities.items():
print(f"{keys}--->{values}”)
This is my code. I am new to python and learning dictionaries as of now. I am trying to add a dictionary to an existing dictionary but it doesn’t work. I have no errors and still only get the first two cities info. I think my syntax must be wrong. Can anyone help me with this please>?
Output:
Try to change your insertion code to:
cities['city3'] = {'name':'Tokyo','country':'japan','desc':'lots of earthquakes'}
You probably don't want to add it as a string, so leave away the quatation marks. Furthermore, there is an erroneous quatation mark at the end of the description.
Use Chainmap from collections, it chains all dictionary into one.
from collections import ChainMap
city3 = {'city3': {'name':'Tokyo','country':'japan','desc':'lots of earthquakes'}}
cities = ChainMap(cities,city3)
print(dict(cities))
I'm trying to iterate through a JSON object/list to get the prices from the list. It works fine for the first "for" loop, but then for my program I need to have another "for" loop nested inside the first loop that effectively continues iterating again where the first loop left off.
edit: I apologise as I probably wasn't very clear as why I'm structuring this code this way. It probably isn't the best solution as I'm not very experienced with python but I'm trying to simulate the looping flow of price data based on historical prices from this JSON. So when the price is equal to something in the first loop, I was trying to continue the flow of data from where the last loop left off (the last price it read) in a nested loop with different if statements, now that the first one has been triggered. I understand this is probably a very poor way of getting around this problem and I'd be happy to hear suggestions on how to do it in a much simpler way.
for i in json:
time.sleep(1)
price = i['p']
if price == "0.00000183":
print("foo")
for i in json:
time.sleep(1)
price = float(i['p'])
if price == "0.00000181":
print("sold")
else:
continue
elif price == "0.00000185":
print ("sold")
break
else:
continue
Example segment of the JSON list:
[
{
"a":4886508,
"p":"0.00000182",
"q":"10971.00000000",
"f":5883037,
"l":5883037,
"T":1566503952430,
"m":1,
"M":1
},
{
"a":4886509,
"p":"0.00000182",
"q":"10971.00000000",
"f":5883038,
"l":5883038,
"T":1566503953551,
"m":1,
"M":1
},
{
"a":4886510,
"p":"0.00000182",
"q":"10971.00000000",
"f":5883039,
"l":5883039,
"T":1566503954431,
"m":1,
"M":1
}
]
You have 3 nested dicts inside the list and you’re using one value price to store all 3; therefore overwriting the value for every iteration.
I suggest using a simple list comprehension to get the price for each sub-dict.
[i['p'] for i in json]
This results in:
['0.00000182', '0.00000182', '0.00000182']
Although as a forewarning there is a builtin module called json so should you want to use that in the future you will need to use a more descriptive name; as well as your other naming conventions are very non informative in the first place consider better conventions.
P.S the price values are strings thus it’s meaningless to compare to int’s
Your sample code doesn't make much sense, so it's not clear what you're try to accomplish.
Ignoring that, if you use json.load() to read your file, the result will be a Python list of dictionaries (which are equivalent to JSON objects). Afterwards you can then just iterate through this list — you don't need to do anything else to go to the next dictionary/object.
Within each dictionary/object you can randomly access the key/value pairs it contains. If yoy want to iterate over them for some reason, you can used obj.keys(), obj.values(), or obj.items(), as you can with any dictionary.
Here's a small example of what I'm saying:
import json
filename = 'list_sample.json'
with open(filename) as json_file:
object_list = json.load(json_file)
for obj in object_list:
if obj['p'] == "0.00000183":
print("foo")
price = float(obj['p'])
print(price)
Use pandas
import pandas as pd
The question was how do I iterate through a json
I think the real question and goal is, how do I access my data from this object and gather insight from it.
That's the power of pandas. Now all of the information is available and easy to use.
If the data is inside a file, just as it's shown:
df = pd.read_json('file.json')
If the data is just a named object (e.g. data = [{...}, {...}, {...}])
df = pd.DataFrame(data)
Output of df:
Get p, or any of the other information:
p_list = df.p.tolist()
print(p_list)
>>> ['0.00000182', '0.00000182', '0.00000182']
You can write two loops using the same iterator object. You just need to create the iterator from your json list of dictionaries:
iterator = iter(json_data)
for i in iterator:
# stuff
for j in iterator: # start looping from where we left off
... # more stuff
Note that this only makes sense if the inner loop will be interrupted at some point by a break statement. Otherwise it will consume all of the iterator's contents and the outer loop will have nothing left. If you were expecting that, you should probably just break out of the outer loop and write the (formerly) inner loop at the same level:
for i in iterator:
if something():
break
for j in iterator: # iterate over the rest of the iterator's values
...
Note that I used json_data in my example code, rather than your variable name json. Using json as a variable name may be a bad idea because it may be the name of the module you're using to parse your data. Reusing the name may get confusing (json = json.load(filename) for instance can only be run once).
Apologies guys, I should've made it more clear that what I was looking for was probably simpler than what I made it out to be or how I described it. I basically just wanted to continue the second loop from where the first left off, with something like this which I have now found to be an alright solution.
elementCount = 0
for i in json:
time.sleep(1)
price = i['p']
elementCount = elementCount + 1
if price == "0.00000183":
print("foo")
for i in json[elementCount:]:
time.sleep(1)
price = float(i['p'])
if price == "0.00000181":
print("sold")
else:
continue
elif price == "0.00000185":
print ("sold")
break
else:
continue
I basically just didn't understand the format for continuing the iteration from a certain element in the list, like this [40:]. Thanks for all your answers and help.
I am using the python code below to extract some values from an excel spreadsheet and then push them to an html page for further processing. I would like to modify the code below so that I can add additional values against each task, any help
the code below does spit out the following:
{'line items': {'AMS Upgrade': '30667', 'BMS works':
'35722'}}
How can I revise the code below so that I can add 2 more values against each task i.e. AMS Upgrade and BMS works
and get the likes of (note the structure below could be wrong)
{'line items': {'AMS Upgrade': {'30667','100%', '25799'}},{'BMS works':
{'10667','10%', '3572'}} }
Code:
book = xlrd.open_workbook("Example - supporting doc.xls")
first_sheet = book.sheet_by_index(-1)
nested_dict = {}
nested_dict["line items"] = {}
for i in range(21,175):
Line_items = first_sheet.row_slice(rowx=i, start_colx=2, end_colx=8)
if str(Line_items[0].value) and str(Line_items[1].value):
if not Line_items[5].value ==0 :
nested_dict["line items"].update({str(Line_items[0].value) : str(Line_items[1].value)})
print nested_dict
print json.dumps(nested_dict)
*** as requested see excel extract below
In Python, each key of a dict can only be associated with a single value. However that single value can be a dict, list, set, etc that holds many values.
You will need to decide the type to use for the value associated with the 'AMS Upgrade' key, if you want it to hold multiple values like '30667','10%', '222'.
Note: what you have written:
{'30667','100%', '25799'}
Is a set literal in Python.
I have a json code from API and I want to get new chat members with the code below but I only get the first two results and not the last (Tester). Why? It should itereate through the whole json file, shouldn't it?
r = requests.get("https://api.../getUpdates").json()
chat_members = []
a = 0
for i in r:
chat_members.append(r['result'][a]['message']['new_chat_members'][0]['last_name'])
a = a + 1
Json here:
{"ok":true,"result":[{"update_id":213849278,
"message":{"message_id":37731,"from":{"id":593029363,"is_bot":false,"first_name": "#tutu"},"chat":{"id":-1001272017174,"title":"tester account","username":"v_glob","type":"supergroup"},"date":1537470595,"new_chat_participant":{"id":593029363,"is_bot":false,"first_name":"tutu "},"new_chat_member":{"id":593029363,"is_bot":false,"first_name":"\u7535\u62a5\u589e\u7c89\uff0c\u4e2d\u82f1\u6587\u5ba2\u670d\uff0c\u62c9\u4eba\u6e05\u5783\u573e\u8f6f\u4ef6\uff0c\u5e7f\u544a\u63a8\u5e7f\uff0cKYC\u6750\u6599\u8ba4\u8bc1\uff0c","last_name":"#tutupeng"},"new_chat_members":[{"id":593029363,"is_bot":false,"first_name":"\u7535\u62a5\u589e\u7c89\uff0c\u4e2d\u82f1\u6587\u5ba2\u670d\uff0c\u62c9\u4eba\u6e05\u5783\u573e\u8f6f\u4ef6\uff0c\u5e7f\u544a\u63a8\u5e7f\uff0cKYC\u6750\u6599\u8ba4\u8bc1\uff0c","last_name":"#tutu"}]}},{"update_id":213849279,
"message":{"message_id":37732,"from":{"id":658150956,"is_bot":false,"first_name":"Rebecca","last_name":"Lawson"},"chat":{"id":-10012720,"title":"v glob OFFICIAL","username":"v_glob","type":"supergroup"},"date":1537484441,"new_chat_participant":{"id":65815,"is_bot":false,"first_name":"Rebecca","last_name":"Lawson"},"new_chat_member":{"id":65815,"is_bot":false,"first_name":"Rebecca","last_name":"Lawson"},"new_chat_members":[{"id":65815,"is_bot":false,"first_name":"Rebecca","last_name":"Lawson"}]}},{"update_id":213849280,
"message":{"message_id":12,"from":{"id":696749142,"is_bot":false,"first_name":"daniel","language_code":"cs-cz"},"chat":{"id":696749142,"first_name":"daniel","type":"private"},"date":1537537013,"text":"/stat","entities":[{"offset":0,"length":5,"type":"bot_command"}]}},{"update_id":213849281,
"message":{"message_id":37740,"from":{"id":669620,"is_bot":false,"first_name":"Ivan","last_name":"Tester"},"chat":{"id":-100127201,"title":"test account","username":"v_glob","type":"supergroup"},"date":1537537597,"new_chat_participant":{"id":669620191,"is_bot":false,"first_name":"Ivan","last_name":"Tester"},"new_chat_member":{"id":669620191,"is_bot":false,"first_name":"Ivan","last_name":"Tester"},"new_chat_members":[{"id":669620191,"is_bot":false,"first_name":"Ivan","last_name":"Tester"}]}}]}
Because you iterate over the entire response dict. The top level only has two items, so that's what you iterate over. Note that you don't actually use the iterator variable, and you have a completely unnecessary separate counter.
Instead, you should be iterating over the result dict:
for result in r['result']:
if "new_chat_members" in result['message']:
chat_members.append(result['message']['new_chat_members'][0]['last_name'])
A colleague of mine has come up with a solution:
for i in l['result']:
chat_members.append(i['message']['new_chat_member']['first_name'])
To sum up: Iterate through 'result' with no positional arguments
I have a dict created in a for loop in Python dict = {year:{month:{day:[title]}}} where year, month, day, and title are all variables. I then use data = json.dumps(dict) which works perfectly. But if the day is the same, I'd like it to add another [title] aspect to the array, so it would be
for title in x:
dict = {year:{month:{day:[title]}}}
data = json.dumps(dict)
if day==day:
//insert another [title] right next to [title]
I've tried using append, update, and insert, but none of them work.
How would I go about doing this?
Note that as user2357112 mentioned, you are creating a Python dict -- not a Python list (aka a JSON "array"). Thus, when you say "[title] right next to [title]" there is a bit of confusion. Dicts do not use the order you are expecting (they use a hash-ordering).
That, and you are attempting to add a field after you've dumped the JSON to a string. You should do that before you dump it. More so, you're throwing away both your dict and data variables every loop. As written, your code will only have access to the variables in the last iteration of the loop.
And another important note: don't overload dict. Rename your variable to something else.
Also, your line day==day will always return True...
Here is what I think you are trying to do: you are creating a "calendar" of sorts that is organized into years, then months, then days. Each day has a list of "titles."
# Variables I'm assuming exist:
# `title`, `year`, `month`, `day`, `someOtherDay`, `titles`, `someOtherTitle`
myDict = {}
for title in titles: #Renamed `x` to `titles` for clarity.
# Make sure myDict has the necessary keys.
if not myDict[year]:
myDict[year] = {}
if not myDict[year][month]:
myDict[year][month] = {}
# Set the day to be a list with a single `title` (and possibly two).
myDict[year][month][day] = [title]
if day==someOtherDay:
myDict[year][month][day].append(someotherTitle)
# And FINALLY dump the result to a string.
data = json.dumps(myDict)