We Keep Coding

pythonic way to break a dict which value is a list to several dict

description
I need to break a dict which value is a list to several dict, keep the other parts.
The key I want to break may have different name,but the cond only and always get one value which type is list.
example
input
cond = {"type":"image","questionType":["3","4","5"]}
cond = {"type":"example","fieldToBreak":["1","2","3"],"fieldInt":1,"fieldFloat":0.1}
output
[
{'type': 'image', 'questionType': '3'},
{'type': 'image', 'questionType': '4'},
{'type': 'image', 'questionType': '5'}
]
[
{'type': 'example', 'fieldToBreak': '1', 'fieldInt': 1, 'fieldFloat': 0.1},
{'type': 'example', 'fieldToBreak': '2', 'fieldInt': 1, 'fieldFloat': 0.1},
{'type': 'example', 'fieldToBreak': '3', 'fieldInt': 1, 'fieldFloat': 0.1}
]
what I have tried
cond_queue = []
for k,v in cond.items():
if isinstance(v,list):
for ele in v:
cond_copy = cond.copy()
cond_copy[k] = ele
cond_queue.append(cond_copy)
break
It works, but I think it is not the best pythonic solution.
question:
Any better pythonic solution?

Possible approach utilizing python's built-in functions and standard library. The code should work with any number of keys. It creates all combinations of values' elements in case of multiple lists presented in the original dict. Not sure if this logic a correct one.
import itertools
def dict_to_inflated_list(d):
ans, keys, vals = list(), list(), list()
# copy keys and 'listified' values in the same order
for k, v in d.items():
keys.append(k)
vals.append(v if isinstance(v, list) else [v])
# iterate over all possible combinations of elements of all 'listified' values
for combination in itertools.product(*vals):
ans.append({k: v for k, v in zip(keys, combination)})
return ans
if __name__ == '__main__':
cond = {'type': 'image', 'questionType': ['3', '4', '5']}
print(dict_to_inflated_list(cond))
cond = {'a': 0, 'b': [1, 2], 'c': [10, 20]}
print(dict_to_inflated_list(cond))
Output:
[{'type': 'image', 'questionType': '3'}, {'type': 'image', 'questionType': '4'}, {'type': 'image', 'questionType': '5'}]
[{'a': 0, 'b': 1, 'c': 10}, {'a': 0, 'b': 1, 'c': 20}, {'a': 0, 'b': 2, 'c': 10}, {'a': 0, 'b': 2, 'c': 20}]

something like the below (the solution is based on the input from the post which I assume represents the general case)
cond = {"type": "image", "questionType": ["3", "4", "5"]}
data = [{"type": "image", "questionType": e} for e in cond['questionType']]
print(data)
output
[{'type': 'image', 'questionType': '3'}, {'type': 'image', 'questionType': '4'}, {'type': 'image', 'questionType': '5'}]

This little function does the job without any extra argument except the input dictionary
def unpack_dict(d):
n = [len(v) for k,v in d.items() if type(v) is list][0] #number of items in the list
r = []
for i in range(n):
_d = {}
for k,v in d.items():
if type(v) is list:
_d[k] = v[i]
else:
_d[k] = v
r.append(_d)
return r
cond = {"type":"example","fieldToBreak":["1","2","3"],"fieldInt":1,"fieldFloat":0.1}
unpack_dict(cond)
[{'type': 'example', 'fieldToBreak': '1', 'fieldInt': 1, 'fieldFloat': 0.1},
{'type': 'example', 'fieldToBreak': '2', 'fieldInt': 1, 'fieldFloat': 0.1},
{'type': 'example', 'fieldToBreak': '3', 'fieldInt': 1, 'fieldFloat': 0.1}]
The function determines how many items (n) there are in the list entry and uses that info to extract the right value to be inserted in the dictionary. Looping over n (for i in range(n):) is used to append the correct number of dictionaries in the final output. That's it. Quite simple to read and understand.

try this:
lens = 0
for index, item in enumerate(cond):
if isinstance(cond[item], list):
lens = len(cond[item])
idx = index
break
print([{k : v if i!=idx else v[j] for i,(k,v) in enumerate(cond.items()) } for j in range(lens)])
output:
# cond = {"type":"image","questionType":["3","4","5"]}
[{'type': 'image', 'questionType': '3'},
{'type': 'image', 'questionType': '4'},
{'type': 'image', 'questionType': '5'}]
# cond = {"type":"example","fieldToBreak":["1","2","3"],"fieldInt":1,"fieldFloat":0.1}
[{'type': 'example', 'fieldToBreak': '1', 'fieldInt': 1, 'fieldFloat': 0.1},
{'type': 'example', 'fieldToBreak': '2', 'fieldInt': 1, 'fieldFloat': 0.1},
{'type': 'example', 'fieldToBreak': '3', 'fieldInt': 1, 'fieldFloat': 0.1}]
if dict have another shape:
# cond = {"questionType":["3","4","5"], "type":"image"}
[{'questionType': '3', 'type': 'image'},
{'questionType': '4', 'type': 'image'},
{'questionType': '5', 'type': 'image'}]

How to create an empty list of dictionaries and populate afterwords?

I need to initialize an empty List of Dictionary(LOD) which must have the following keys in it. "id","name","age", "gender". I want to create a loop/nested loop that starts populating the LOD. For poppulating I have a list which has ID's and the rest of the keys are generated using the random function.
The ID list looks like this: id = ['1','2','3']
The result must look something like this.
LOD = [
{
'id': '1',
'name':'122121',
'age':'2131',
'gender':'121'
},
{
'id': '2',
'name':'122121',
'age':'2131',
'gender':'121'
},
{
'id': '3',
'name':'122121',
'age':'2131',
'gender':'121'
},
]

CJDB already does what you want. But if you'd perhaps prefer another approach:
ids = ['1','2','3']
keys = ["name","age", "gender"]
LOD = []
and then populate your list with dictionaries
for i in ids:
your_dictionary = {"id": i}
for key in keys:
your_dictionary[key] = '{}_rnd_function_output'.format(key)
LOD.append(your_dictionary)
And the output would be
>>> LOD
[{'id': '1',
'name': 'name_rnd_function_output',
'age': 'age_rnd_function_output',
'gender': 'gender_rnd_function_output'},
{'id': '2',
'name': 'name_rnd_function_output',
'age': 'age_rnd_function_output',
'gender': 'gender_rnd_function_output'},
{'id': '3',
'name': 'name_rnd_function_output',
'age': 'age_rnd_function_output',
'gender': 'gender_rnd_function_output'}
]
You might consider having a sub-dictionaries within a dictionary. Your ids would be keys for main dictionary and sub-dictionaries would be values.
LOD = {}
for i in ids:
LOD[i] = {}
for key in keys:
LOD[i][key] = '{}_rnd_function_output'.format(key)
And the output
>>> LOD
{'1': {'name': 'name_rnd_function_output',
'age': 'age_rnd_function_output',
'gender': 'gender_rnd_function_output'},
'2': {'name': 'name_rnd_function_output',
'age': 'age_rnd_function_output',
'gender': 'gender_rnd_function_output'},
'3': {'name': 'name_rnd_function_output',
'age': 'age_rnd_function_output',
'gender': 'gender_rnd_function_output'}}

You can use a dictionary-comprehension for this:
ids = ['1','2','3']
LOD = [
{
'id': i,
'name':'122121',
'age':'2131',
'gender':'121'
} for i in ids
]
Output:
>>> LOD
[{'id': '1', 'name': '122121', 'age': '2131', 'gender': '121'},
{'id': '2', 'name': '122121', 'age': '2131', 'gender': '121'},
{'id': '3', 'name': '122121', 'age': '2131', 'gender': '121'}]
Or, using the random module:
import random
ids = ['1','2','3']
LOD = [
{
'id': i,
'name': str(random.randint(100000, 999999)),
'age': str(random.randint(1000, 9999)),
'gender': str(random.randint(100, 999))
} for i in ids
]
Output:
>>> LOD
[{'id': '1', 'name': '727325', 'age': '5367', 'gender': '238'},
{'id': '2', 'name': '316019', 'age': '8963', 'gender': '702'},
{'id': '3', 'name': '464023', 'age': '4324', 'gender': '155'}]
Note that you should not use id as a variable name as it shadows the builtin python id object.

You can do it by initializing dict objects in list comprehensions
keys = ['id', 'name', 'age', 'gender']
ids = ['1', '2', '3']
LOD = [dict((key, i if key == 'id' else random.randint(1, 100)) for key in keys) for i in ids]
print(LOD)
'''
[{'id': '1', 'name': 34, 'age': 10, 'gender': 57},
{'id': '2', 'name': 64, 'age': 13, 'gender': 21},
{'id': '3', 'name': 11, 'age': 17, 'gender': 2}]
'''

Iterate over list of dicts for calculation of population density

I'm new to python, so I apologise if this is straight forward. Other questions (here and here) have addressed lists of dicts, but I haven't been able to get this to work.
I have a list of dicts for each geographical area:
list_of_dicts = [{'id': 'a', 'population': '20', 'area': '10'},
{'id': 'a', 'population': '20', 'area': '10'}]
I merely want to calculate the population density for each area, by dividing the 'population': 'value', by the 'area': 'value'. This calculation should create a new item.
The results should look like this:
results = [{'id': 'a', 'population': '20', 'area': '10', 'pop_density': 2},
{'id': 'a', 'population': '30', 'area': '5', 'pop_density': 6}]

Alter the dictionaries
You can simply iterate over every dictionary, and associate 'pop_density' with the population density:
for v in list_of_dicts:
v['pop_density'] = float(v['population'])/float(v['area'])
We need to use float(..) to convert a string '20' to the number 20. We can use int(..) if all values are ints. But perhaps it is safer to work with floats.
Copy the dictionaries
In case you want to create a copy of the list_of_dicts, you can use list comprehension:
[dict(v,pop_density=float(v['population'])/float(v['area'])) for v in list_of_dicts]
Generating:
>>> [dict(v,pop_density=float(v['population'])/float(v['area'])) for v in list_of_dicts]
[{'population': '20', 'area': '10', 'pop_density': 2.0, 'id': 'a'}, {'population': '20', 'area': '10', 'pop_density': 2.0, 'id': 'a'}]

Changing original dictionaries
You can simply iterate over your list of dictionaries and the calculations. Make sure to round the result since you want an integer:
>>> list_of_dicts = [{'id': 'a', 'population': '20', 'area': '10'},
{'id': 'a', 'population': '20', 'area': '10'}]
>>>
>>> for d in list_of_dicts:
d['pop_density'] = int(d['population']) // int(d['area']) # round result by using //
>>> list_of_dicts
[{'pop_density': 2, 'population': '20', 'id': 'a', 'area': '10'}, {'pop_density': 2, 'population': '20', 'id': 'a', 'area': '10'}]
>>>
Creating new dictionaries
Python 3
If you want a new list of dictionaries, you can use a list comprehension:
>>> list_of_dicts = [{'id': 'a', 'population': '20', 'area': '10'},
{'id': 'a', 'population': '20', 'area': '10'}]
>>>
>>> [{'pop_densitiy': int(d['population']) // int(d['area']), **d} for d in list_of_dicts]
[{'area': '10', 'population': '20', 'id': 'a', 'pop_densitiy': 2}, {'area': '10', 'population': '20', 'id': 'a', 'pop_densitiy': 2}]
>>>
Python 2
Note the above uses the dictionary unpacking operator that is only available in python 3. If using Python 2, you'll need to use the dict constructor:
>>> list_of_dicts = [{'id': 'a', 'population': '20', 'area': '10'},
{'id': 'a', 'population': '20', 'area': '10'}]
>>> [dict(d, pop_densitiy=int(d['population']) // int(d['area'])) for d in list_of_dicts]
[{'pop_densitiy': 2, 'population': '20', 'id': 'a', 'area': '10'}, {'pop_densitiy': 2, 'population': '20', 'id': 'a', 'area': '10'}]
>>>

Just iterate over the existing indices and add a new one to them:
results = []
for item in list_of_dicts:
item = item.copy() # since we want a new dict - mind you this is a shallow copy
item["pop_density"] = int(item.get("population", 0)) / float(item.get("area", 1))
results.append(item)

list_of_dicts = [{'id': 'a', 'population': '20', 'area': '10'},
{'id': 'a', 'population': '30', 'area': '5'}]
[dict(j) for j in [ list(i.items()) + [ ('pop_density', int(i['population'])/int(i['area'])) ] for i in list_of_dicts ] ]
Output:
[{'area': '10', 'id': 'a', 'pop_density': 2.0, 'population': '20'},
{'area': '5', 'id': 'a', 'pop_density': 6.0, 'population': '30'}]

How to group an array by multiple keys?

I'd like a function that can group a list of dictionaries into sublists of dictionaries depending on an arbitrary set of keys that all dictionaries have in common.
For example, I'd like the following list to be grouped into sublists of dictionaries depending on a certain set of keys
l = [{'name':'b','type':'new','color':'blue','amount':100},{'name':'c','type':'new','color':'red','amount':100},{'name':'d','type':'old','color':'gold','amount':100},{'name':'e','type':'old','color':'red','amount':100},
{'name':'f','type':'old','color':'red','amount':100},{'name':'g','type':'normal','color':'red','amount':100}]
If I wanted to group by type, the following list would result, which has a sublists where each sublist has the same type:
[[{'name':'b','type':'new','color':'blue','amount':100},{'name':'c','type':'new','color':'red','amount':100}],[{'name':'d','type':'old','color':'gold','amount':100},{'name':'e','type':'old','color':'red','amount':100},
{'name':'f','type':'old','color':'red','amount':100}],[{'name':'g','type':'normal','color':'red','amount':100}]]
If I wanted to group by type and color, the following would result where the list contains sublists that have the same type and color:
[[{'name':'b','type':'new','color':'blue','amount':100}],[{'name':'c','type':'new','color':'red','amount':100}],[{'name':'d','type':'old','color':'gold','amount':100}],[{'name':'e','type':'old','color':'red','amount':100},
{'name':'f','type':'old','color':'red','amount':100}],[{'name':'g','type':'normal','color':'red','amount':100}]]
I understand the following function can group by one key, but I'd like to group by multiple keys:
def group_by_key(l,i):
l = [list(grp) for key, grp in itertools.groupby(sorted(l, key=operator.itemgetter(i)), key=operator.itemgetter(i))]
This is my attempt using the group_by_function above
def group_by_multiple_keys(l,*keys):
for key in keys:
l = group_by_key(l,key)
l = [item for sublist in l for item in sublist]
return l
The issue there is that it ungroups it right after it grouped it by a key. Instead, I'd like to re-group it by another key and still have one list of sublists.

itertools.groupby() + operator.itemgetter() will do what you want. groupby() takes an iterable and a key function, and groups the items in the iterable by the value returned by passing each item to the key function. itemgetter() is a factory that returns a function, which gets the specified items from any item passed to it.
from __future__ import print_function
import pprint
from itertools import groupby
from operator import itemgetter
def group_by_keys(iterable, keys):
key_func = itemgetter(*keys)
# For groupby() to do what we want, the iterable needs to be sorted
# by the same key function that we're grouping by.
sorted_iterable = sorted(iterable, key=key_func)
return [list(group) for key, group in groupby(sorted_iterable, key_func)]
dicts = [
{'name': 'b', 'type': 'new', 'color': 'blue', 'amount': 100},
{'name': 'c', 'type': 'new', 'color': 'red', 'amount': 100},
{'name': 'd', 'type': 'old', 'color': 'gold', 'amount': 100},
{'name': 'e', 'type': 'old', 'color': 'red', 'amount': 100},
{'name': 'f', 'type': 'old', 'color': 'red', 'amount': 100},
{'name': 'g', 'type': 'normal', 'color': 'red', 'amount': 100}
]
Examples:
>>> pprint.pprint(group_by_keys(dicts, ('type',)))
[[{'amount': 100, 'color': 'blue', 'name': 'b', 'type': 'new'},
{'amount': 100, 'color': 'red', 'name': 'c', 'type': 'new'}],
[{'amount': 100, 'color': 'gold', 'name': 'd', 'type': 'old'},
{'amount': 100, 'color': 'red', 'name': 'e', 'type': 'old'},
{'amount': 100, 'color': 'red', 'name': 'f', 'type': 'old'}],
[{'amount': 100, 'color': 'red', 'name': 'g', 'type': 'normal'}]]
>>>
>>> pprint.pprint(group_by_keys(dicts, ('type', 'color')))
[[{'amount': 100, 'color': 'blue', 'name': 'b', 'type': 'new'}],
[{'amount': 100, 'color': 'red', 'name': 'c', 'type': 'new'}],
[{'amount': 100, 'color': 'gold', 'name': 'd', 'type': 'old'}],
[{'amount': 100, 'color': 'red', 'name': 'e', 'type': 'old'},
{'amount': 100, 'color': 'red', 'name': 'f', 'type': 'old'}],
[{'amount': 100, 'color': 'red', 'name': 'g', 'type': 'normal'}]]

Python: Convert a list into a bidimensional mapping dictionary

I have a list of dictionaries:
[
{'student_id': 'john', 'exercise_id': '3', 'answer': 20},
{'student_id': 'john', 'exercise_id': '2', 'answer': 10},
{'student_id': 'jane', 'exercise_id': '2', 'answer': 30},
]
What is an elegant/short way to convert that into a [exercise x student] "mapping table" dictionary?
Like so:
{
'3':{
'john': {'student_id': 'john', 'exercise_id': '3', 'answer': 20}
},
'2': {
'john': {'student_id': 'john', 'exercise_id': '2', 'answer': 10},
'jane': {'student_id': 'jane', 'exercise_id': '2', 'answer': 30}
}
}
You can assume the map contains at most one answer per exercise per student.

The straight forward way would be to gather them in a dictionary, like this
d = {}
for item in l:
d.setdefault(item["exercise_id"], {}).setdefault(item["student_id"], []).append(item)
print(d)
Output
{'2': {'jane': [{'answer': 30, 'exercise_id': '2', 'student_id': 'jane'}],
'john': [{'answer': 10, 'exercise_id': '2', 'student_id': 'john'}]},
'3': {'john': [{'answer': 20, 'exercise_id': '3', 'student_id': 'john'}]}}
First, if the item["exercise_id"] is not there in d, then a new dictionary will be set as the value and then in that dictionary, if item["student_id"] is not there, we set an empty list as the value and we append the current dictionary in that list.

This generates the output you want:
output = {}
for value in data:
output.setdefault(value['exercise_id'], {})[value['student_id']] = value
print output