We Keep Coding

get data from a json

I want to get the data from a json. I have the idea of a loop to access all levels.
I have only been able to pull data from a single block.
print(output['body']['data'][0]['list'][0]['outUcastPkts'])
How do I get the other data?
import json,urllib.request
data = urllib.request.urlopen("http://172.0.0.0/statistic").read()
output = json.loads(data)
for elt in output['body']['data']:
print(output['body']['data'][0]['inUcastPktsAll'])
for elt in output['list']:
print(output['body']['data'][0]['list'][0]['outUcastPkts'])
{
"body": {
"data": [
{
"inUcastPktsAll": 3100617019,
"inMcastPktsAll": 7567,
"inBcastPktsAll": 8872,
"outPktsAll": 8585575441,
"outUcastPktsAll": 8220240108,
"outMcastPktsAll": 286184143,
"outBcastPktsAll": 79151190,
"list": [
{
"outUcastPkts": 117427359,
"outMcastPkts": 1990586,
"outBcastPkts": 246120
},
{
"outUcastPkts": 0,
"outMcastPkts": 0,
"outBcastPkts": 0
}
]
},
{
"inUcastPktsAll": 8269483865,
"inMcastPktsAll": 2405765,
"inBcastPktsAll": 124466,
"outPktsAll": 3101194852,
"outUcastPktsAll": 3101012296,
"outMcastPktsAll": 173409,
"outBcastPktsAll": 9147,
"list": [
{
"outUcastPkts": 3101012296,
"outMcastPkts": 90488,
"outBcastPkts": 9147
},
{
"outUcastPkts": 0,
"outMcastPkts": 0,
"outBcastPkts": 0
}
]
}
],
"msgs": [ "successful" ]
},
"header": {
"opCode": "1",
"token": "",
"state": "",
"version": 1
}
}

output = json.loads(data) #Type of output is a dictionary.
#Try to use ".get()" method.
print(output.get('body')) #Get values of key 'body'
print(output.get('body').get('data')) #Get a list of key 'data'
If a key doesn't exist, the '.get()' method will return None.
https://docs.python.org/3/library/stdtypes.html#dict.get

In python you can easily iterate over the objects of a list like so:
>>> l = [1, 2, 3, 7]
>>> for elem in l:
... print(elem)
...
1
2
3
7
This works regarding what can of object do you have in the list (integers, tuples, dictionaries). Having that in mind, your solution was not far off, you only to do the following changes:
for entry in output['body']['data']:
print(entry['inUcastPktsAll'])
for list_element in entry['list']:
print(list_element['outUcastPkts'])
This will give you the following for the json object you have provided:
3100617019
117427359
0
8269483865
3101012296
0

How to create a dictionary with duplicate keys and form a list of dictionary

I am trying to write a program where I am having a list of dictionaries in the following manner
[
{
'unique':1,
'duplicate':2,
},
{
'unique':1,
'duplicate':2,
},
{
'unique':1,
'duplicate':2,
},
{
'unique':1,
'duplicate':2,
}
]
Can we form it as a dictionary, where the first key in tuple should become unique Key in a dictionary
and it's corresponding values as a list for that values
Example:
[
{
'unique':1,
'duplicate':2,
},
{
'unique':1,
'duplicate':8,
},
{
'unique':2,
'duplicate':2,
},
{
'unique':1,
'duplicate':4,
}
]
The above list should be converted into the following
---- Expected Outcome ---
[
{
'unique':1,
'duplicates':[2,8,4]
},
{
'unique':2,
'duplicates':[2]
}
]
PS: I am doing this in python
Thanks for the code in advance

you can also use itertools.groupby:
from itertools import groupby
from operator import itemgetter
l = [
{
'unique':1,
'duplicate':2,
},
{
'unique':1,
'duplicate':8,
},
{
'unique':2,
'duplicate':2,
},
{
'unique':1,
'duplicate':4,
}
]
key = itemgetter('unique')
result = [{'unique':k, 'duplicate': list(map(itemgetter('duplicate'), g))}
for k, g in groupby(sorted(l, key=key ), key = key)]
print(result)
output:
[{'unique': 1, 'duplicate': [2, 8, 4]}, {'unique': 2, 'duplicate': [2]}]

I think this list comprehension can solve your problem:
result = [{'unique': id, 'duplicates': [d['duplicate'] for d in l if d['unique'] == id]} for id in set(map(lambda d: d['unique'], l))]

This might help you:
l = [
{
'unique':1,
'duplicate':2,
},
{
'unique':1,
'duplicate':8,
},
{
'unique':2,
'duplicate':2,
},
{
'unique':1,
'duplicate':4,
}
]
a = set()
for i in l:
a.add(i['unique'])
d = {i:[] for i in a }
for i in l:
d[i['unique']].append(i['duplicate'])
output = [{'unique': i, 'duplicate': j}for i, j in d.items()]
The output will be:
[{'unique': 1, 'duplicate': [2, 8, 4]}, {'unique': 2, 'duplicate': [2]}]

defaultdict(list) may help you here:
from collections import defaultdict
# data = [ {'unique': 1, 'duplicate': 2}, ... ] # your data
dups = defaultdict(list) # {unique: [duplicate]}
for dd in data:
dups[dd['unique']].append(dd['duplicate'])
answer = [dict(unique = k, duplicates = v) for k, v in dups.items()]
If you don't know the name of unique key, then replace 'unique' with something like
unique_key = list(data[0].keys())[0]

unique=[]
duplicate ={}
for items in data:
if items['unique'] not in unique:
unique.append(items['unique'])
duplicate[items['unique']]=[items['duplicate']]
else:
duplicate[items['unique']].append(items['duplicate'])
new_data=[]
for key in unique:
new_data.append({'unique':key,'duplicate':duplicate[key]})
Explanation: In the first for loop, I am appending unique keys to 'unique'. If the key doesn't exists in 'unique', I will append it in 'unique' & add a key in 'duplicate' with value as single element list. If the same key is found again, I simply append that value to 'duplicate' corresponding the key. In the 2nd loop, I am creating a 'new_dict' where I am adding these unique keys & its duplicate value list

Get all parents keys in nested dictionary for all items

I want to get all parent keys for all items in a nested python dictionary with unlimited levels. Take an analogy, if you think of a nested dictionary as a directory containing sub-directories, the behaviour I want is similar to what glob.glob(dir, recursive=True) does.
For example, suppose we have the following dictionary:
sample_dict = {
"key_1": {
"sub_key_1": 1,
"sub_key_2": 2,
},
"key_2": {
"sub_key_1": 3,
"sub_key_2": {
"sub_sub_key_1": 4,
},
},
}
I want to get the full "path" of every value in the dictionary:
["key_1", "sub_key_1", 1]
["key_1", "sub_key_2", 2]
["key_2", "sub_key_1", 3]
["key_2", "sub_key_2", "sub_sub_key_1", 4]
Just wondering if there is a clean way to do that?

Using generators can often simplify the code for these type of tasks and make them much more readable while avoiding passing explicit state arguments to the function. You get a generator instead of a list, but this is a good thing because you can evaluate lazily if you want to. For example:
def getpaths(d):
if not isinstance(d, dict):
yield [d]
else:
yield from ([k] + w for k, v in d.items() for w in getpaths(v))
result = list(getpaths(sample_dict))
Result will be:
[['key_1', 'sub_key_1', 1],
['key_1', 'sub_key_2', 2],
['key_2', 'sub_key_1', 3],
['key_2', 'sub_key_2', 'sub_sub_key_1', 4]]

You can solve it recursively
sample_dict = {
"key_1": {
"sub_key_1": 1,
"sub_key_2": 2,
},
"key_2": {
"sub_key_1": 3,
"sub_key_2": {
"sub_sub_key_1": 4,
},
}
}
def full_paths(sample_dict, paths=[], parent_keys=[]):
for key in sample_dict.keys():
if type(sample_dict[key]) is dict:
full_paths(sample_dict[key], paths=paths, parent_keys=(parent_keys + [key]))
else:
paths.append(parent_keys + [key] + [sample_dict[key]])
return paths
print(full_paths(sample_dict))

You can use this solution.
sample_dict = {
"key_1": {
"sub_key_1": 1,
"sub_key_2": 2,
},
"key_2": {
"sub_key_1": 3,
"sub_key_2": {
"sub_sub_key_1": 4,
},
},
}
def key_find(sample_dict, li=[]):
for key, val in sample_dict.items():
if isinstance(val, dict):
key_find(val, li=li + [key])
else:
print(li + [key] + [val])
key_find(sample_dict)

Python: Efficient way of sorting dictionary that has n dictionaries within it

I'll just go straight to example:
Here we have a dictionary with a test name, and another dict which contains the level categorization.
EDIT
Input:
test_values={
{
"name":"test1",
"level_map":{
"system":1,
"system_apps":2,
"app_test":3
}
},
{
"name":"test2",
"level_map":{
"system":1,
"system_apps":2,
"app_test":3
}
},
{
"name":"test3",
"level_map":{
"system":1,
"memory":2,
"memory_test":3
}
}
}
Output:
What I want is this:
dict_obj:
{
"system":{
"system_apps":{
"app_test":{
test1 object,
test2 object
},
"memory":{
"memory_test":{
test3 object
}
}
}
}
I just can't wrap my head around the logic and I'm struggling to even come up with an approach. If someone could guide me, that would be great.

Let's start with level_map. You can sort keys on values to get the ordered levels:
>>> level_map = { "system": 1, "system_apps": 2, "app_test": 3}
>>> L = sorted(level_map.keys(), key=lambda k: level_map[k])
>>> L
['system', 'system_apps', 'app_test']
Use these elements to build a tree:
>>> root = {}
>>> temp = root
>>> for k in L[:-1]:
... temp = temp.setdefault(k, {}) # create new inner dict if necessary
...
>>> temp.setdefault(L[-1], []).append("test") # and add a name
>>> root
{'system': {'system_apps': {'app_test': ['test']}}}
I split the list before the last element, because the last element will be associated to a list, not a dict (leaves of the tree are lists in your example).
Now, the it's easy to repeat this with the list of dicts:
ds = [{ "name": "test1",
"level_map": { "system": 1, "system_apps": 2, "app_test": 3}
}, { "name": "test2",
"level_map": { "system": 1, "system_apps": 2, "app_test": 3}
}, { "name": "test3",
"level_map": { "system": 1, "memory": 2, "memory_test": 3}
}]
root = {}
for d in ds:
name = d["name"]
level_map = d["level_map"]
L = sorted(level_map.keys(), key=lambda k: level_map[k])
temp = root
for k in L[:-1]:
temp = temp.setdefault(k, {})
temp.setdefault(L[-1], []).append(name)
# root is: {'system': {'system_apps': {'app_test': ['test1', 'test2']}, 'memory': {'memory_test': ['test3']}}}

replace information in Json string based on a condition

I have a very large json file with several nested keys. From whaat I've read so far, if you do:
x = json.loads(data)
Python will interpret it as a dictionary (correct me if I'm wrong). The fourth level of nesting in the json file contains several elements named by an ID number and all of them contain an element called children, something like this:
{"level1":
{"level2":
{"level3":
{"ID1":
{"children": [1,2,3,4,5]}
}
{"ID2":
{"children": []}
}
{"ID3":
{"children": [6,7,8,9,10]}
}
}
}
}
What I need to do is to replace all items in all the "children" elements with nothing, meaning "children": [] if the ID number is in a list called new_ids and then convert it back to json. I've been reading on the subject for a few hours now but I haven't found anything similar to this to try to help myself.
I'm running Python 3.3.3. Any ideas are greatly appreciated!!
Thanks!!
EDIT
List:
new_ids=["ID1","ID3"]
Expected result:
{"level1":
{"level2":
{"level3":
{"ID1":
{"children": []}
}
{"ID2":
{"children": []}
}
{"ID3":
{"children": []}
}
}
}
}

First of all, your JSON is invalid. I assume you want this:
{"level1":
{"level2":
{"level3":
{
"ID1":{"children": [1,2,3,4,5]},
"ID2":{"children": []},
"ID3":{"children": [6,7,8,9,10]}
}
}
}
}
Now, load your data as a dictionary:
>>> with open('file', 'r') as f:
... x = json.load(f)
...
>>> x
{u'level1': {u'level2': {u'level3': {u'ID2': {u'children': []}, u'ID3': {u'children': [6, 7, 8, 9, 10]}, u'ID1': {u'children': [1, 2, 3, 4, 5]}}}}}
Now you can loop over the keys in x['level1']['level2']['level3'] and check whether they are in your new_ids.
>>> new_ids=["ID1","ID3"]
>>> for key in x['level1']['level2']['level3']:
... if key in new_ids:
... x['level1']['level2']['level3'][key]['children'] = []
...
>>> x
{u'level1': {u'level2': {u'level3': {u'ID2': {u'children': []}, u'ID3': {u'children': []}, u'ID1': {u'children': []}}}}}
You can now write x back to a file like this:
with open('myfile', 'w') as f:
f.write(json.dumps(x))
If your new_ids list is large, consider making it a set.

If you have simple dictionary like this
data_dict = {
"level1": {
"level2":{
"level3":{
"ID1":{"children": [1,2,3,4,5]},
"ID2":{"children": [] },
"ID3":{"children": [6,7,8,9,10]},
}
}
}
}
than you need only this:
data_dict = {
"level1": {
"level2":{
"level3":{
"ID1":{"children": [1,2,3,4,5]},
"ID2":{"children": [] },
"ID3":{"children": [6,7,8,9,10]},
}
}
}
}
new_ids=["ID1","ID3"]
for idx in new_ids:
if idx in data_dict['level1']["level2"]["level3"]:
data_dict['level1']["level2"]["level3"][idx]['children'] = []
print data_dict
'''
{
'level1': {
'level2': {
'level3': {
'ID2': {'children': []},
'ID3': {'children': []},
'ID1': {'children': []}
}
}
}
}
'''
but if you have more complicated dictionary
data_dict = {
"level1a": {
"level2a":{
"level3a":{
"ID2":{"children": [] },
"ID3":{"children": [6,7,8,9,10]},
}
}
},
"level1b": {
"level2b":{
"level3b":{
"ID1":{"children": [1,2,3,4,5]},
}
}
}
}
new_ids =["ID1","ID3"]
for level1 in data_dict.values():
for level2 in level1.values():
for level3 in level2.values():
for idx in new_ids:
if idx in level3:
level3[idx]['children'] = []
print data_dict
'''
{
'level1a': {
'level2a': {
'level3a': {
'ID2': {'children': []},
'ID3': {'children': []}
}
}
},
'level1b': {
'level2b': {
'level3b': {
'ID1': {'children': []}
}
}
}
}
'''