We Keep Coding

Python: Way to build a dictionary with a variable key and append to a list as the value inside a loop

I have a list of dictionaries. I want to loop through this list of dictionary and for each specific name (an attribute inside each dictionary), I want to create a dictionary where the key is the name and the value of this key is a list which dynamically appends to the list in accordance with a specific condition.
For example, I have
d = [{'Name': 'John', 'id': 10},
{'Name': 'Mark', 'id': 21},
{'Name': 'Matthew', 'id': 30},
{'Name': 'Luke', 'id': 11},
{'Name': 'John', 'id': 20}]
I then built a list with only the names using names=[i['Name'] for i in dic1] so I have a list of names. Notice John will appear twice in this list (at the beginning and end). Then, I want to create a for-loop (for name in names), which creates a dictionary 'ID' that for its value is a list which appends this id field as it goes along.
So in the end I'm looking for this ID dictionary to have:
John: [10,20]
Mark: [21]
Matthew: [30]
Luke: [11]
Notice that John has a list length of two because his name appears twice in the list of dictionaries.
But I can't figure out a way to dynamically append these values to a list inside the for-loop. I tried:
ID={[]} #I also tried with just {}
for name in names:
ID[names].append([i['id'] for i in dic1 if i['Name'] == name])
Please let me know how one can accomplish this. Thanks.

Don't loop over the list of names and go searching for every one in the list; that's very inefficient, since you're scanning the whole list all over again for every name. Just loop over the original list once and update the ID dict as you go. Also, if you build the ID dict first, then you can get the list of names from it and avoid another list traversal:
names = ID.keys()
The easiest solution for ID itself is a dictionary with a default value of the empty list; that way ID[name].append will work for names that aren't in the dict yet, instead of blowing up with a KeyError.
from collections import defaultdict
ID = defaultdict(list)
for item in d:
ID[item['Name']].append(item['id'])
You can treat a defaultdict like a normal dict for almost every purpose, but if you need to, you can turn it into a plain dict by calling dict on it:
plain_id = dict(ID)
The Thonnu has a solution using get and list concatenation which works without defaultdict. Here's another take on a no-import solution:
ID = {}
for item in d:
name, number = item['Name'], item['id']
if name in ID:
ID[name].append(number)
else:
ID[name] = [ number ]

Using collections.defaultdict:
from collections import defaultdict
out = defaultdict(list)
for item in dic1:
out[item['Name']].append(item['id'])
print(dict(out))
Or, without any imports:
out = {}
for item in dic1:
out[item['Name']] = out.get(item['Name'], []) + [item['id']]
print(out)
Or, with a list comprehension:
out = {}
[out.update({item['Name']: out.get(item['Name'], []) + [item['id']]}) for item in dic1]
print(out)
Output:
{'John': [10, 20], 'Mark': [21], 'Matthew': [30], 'Luke': [11]}

dic1 = [{'Name': 'John', 'id': 10}, {'Name': 'Mark', 'id': 21}, {'Name': 'Matthew', 'id': 30}, {'Name': 'Luke', 'id': 11}, {'Name': 'John', 'id': 20}]
id_dict = {}
for dic in dic1:
key = dic['Name']
if key in id_dict:
id_dict[key].append(dic['id'])
else:
id_dict[key] = [dic['id']]
print(id_dict) # {'John': [10, 20], 'Mark': [21], 'Matthew': [30], 'Luke': [11]}

You can use defaultdict for this to initiate a dictionary with a default value. In this case the default value will be empty list.
from collections import defaultdict
d=defaultdict(list)
for item in dic1:
d[item['Name']].append(item['id'])
Output
{'John': [10, 20], 'Mark': [21], 'Matthew': [30], 'Luke': [11]} # by converting (not required) into pure dict dict(d)

You can do in a easy version
dic1=[{'Name': 'John', 'id':10}, {'Name': 'Mark', 'id':21},{'Name': 'Matthew', 'id':30}, {'Name': 'Luke', 'id':11}, {'Name': 'John', 'id':20}]
names=[i['Name'] for i in dic1]
ID = {}
for i, name in enumerate(names):
if name in ID:
ID[name].append(dic1[i]['id'])
else:
ID[name] = [dic1[i]['id']]
print(ID)

Create many lists from list of dictionaries that have different structures

I have a big list of dictionaries. And the dictionaries don´t have the same structure. But, I don´t know all the structures (my list has 1 million elements).
For example:
mylist = [{'name': 'Juan Carlos','age':38},{'name':'David','country':'Brazil'},
{'name':'Agustina', 'country': 'Argentina'},{'name': 'Renzo','age':24}]
I want to separate mylist into different lists, and each list has dictionaries with the same keys.
For example:
list1 = [{'name': 'Juan Carlos','age':38},{'name': 'Renzo','age':24}]
list2 = [{'name':'David','country':'Brazil'},{'name':'Agustina', 'country': 'Argentina'}]
The problem is I don´t know how many sublists I'm going to have.

To do this, you need to create a dictionary where the key is the set of keys that are present:
mylist = [{'name': 'Juan Carlos','age':38},{'name':'David','country':'Brazil'},{'name':'Agustina', 'country': 'Argentina'},{'name': 'Renzo','age':24}]
sublists = {}
for sl in mylist:
keys = tuple(set(sl.keys()))
if keys not in sublists:
sublists[keys] = []
sublists[keys].append( sl )
print( sublists )
Output:
{('name', 'age'): [{'name': 'Juan Carlos', 'age': 38}, {'name': 'Renzo', 'age': 24}], ('name', 'country'): [{'name': 'David', 'country': 'Brazil'}, {'name': 'Agustina', 'country': 'Argentina'}]}

You can use a frozenset of keys as a new key:
mylist = [{'name': 'Juan Carlos','age':38},{'name':'David','country':'Brazil'},
{'name':'Agustina', 'country': 'Argentina'},{'name': 'Renzo','age':24}]
sep = {}
for d in mylist:
sep.setdefault(frozenset(d), []).append(d)
and then you can get a list of lists:
print(sep.values())

How to get dictionary value from key in a list?

I am trying to create a new list from a list of dictionary items.
Below is an example of 1 dictionary item.
{'id': 'bitcoin',
'symbol': 'btc',
'name': 'Bitcoin',
'current_price': 11907.43,
'market_cap': 220817187069,
'market_cap_rank': 1}
I want the list to just be of the id item. So what I am trying to achieve is a list with items {'bitcoin', 'etc', 'etc}

You can use list comprehension:
my_list = [{'id': 'bitcoin', 'symbol': 'btc', ...}, ...]
[d['id'] for d in my_list]
Which translates to : for each dictionary in my_list, extract the 'id' key.

id_list = [d["id"] for d in dictlist ]
This should work for you

list = [ i['id'] for i in list_of_dict]
this should help

Simple and readable code that solves the purpose:
main_list = []
for item in main_dict:
main_list.append(item.get("id", None))
print(main_list)

Split python list in filtered list of multiple list and store them in single list

I have an exhaustive list of different categories with me:
myList = [
{'name': 'Sasha', 'category': 'Dog'},
{'name': 'Meow', 'category': 'Cat'},
{'name': 'Bark', 'category': 'Dog'}
]
I want them to break and a create smaller list within this bigList. It would be similar to this:
bigList = [
[
{'category': 'Dog', 'name': 'Sasha'},
{'category': 'Dog', 'name': 'Bark'}
],
[
{'category': 'Cat', 'name': 'Meow'}
]
]
Here is the python logic for iterating the loop:
bigList = []
prev = ''
for s in myList:
newList = []
if s['category'] != prev:
for m in myList:
if m['category'] == s['category']:
newList.append(m)
bigList.append(newList)
prev = s['category']
This has done the trick for me but I would like to know how can I optimize the above logic in for loop for more shorter and efficient code.

You can do this in two steps with groupby as commented by #roganjosh:
from itertools import groupby
# step 1: sort the list by category, we need this step because groupby only groups same
# adjacent values so we need to sort the list so that same category are close to each other
sort_list = sorted(myList, key = lambda x: x["category"])
# step 2: group by the category and create a new sub list for each group
[list(g) for _, g in groupby(sort_list, key = lambda x: x['category'])]
#[[{'category': 'Cat', 'name': 'Meow'}],
# [{'category': 'Dog', 'name': 'Sasha'}, {'category': 'Dog', 'name': 'Bark'}]]

Sorting can be expensive for large lists.
Starting with your data:
my_list = [
{'name': 'Sasha', 'category': 'Dog'},
{'name': 'Meow', 'category': 'Cat'},
{'name': 'Bark', 'category': 'Dog'}
]
This loops through all elements of your list once and rembers what its has already seen before in a dictionary:
res = []
seen = {}
for entry in my_list:
val = seen.setdefault(entry['category'], [])
if not val:
res.append(val)
val.append(entry)
It appends a new list only for not yet seen entries to res but all entries to the corresponding nested list val that it got from the seen dictionary. So, the same val is in res and in seen. Hence, appending to val will enlarge val and the effect can be seen no matter if your access val through res and or seen. The line val = seen.setdefault(entry['category'], []) gives you either an existing list, if the category was seen before or, a new empty list if the category was encountered the first time. At the same time, if the category is not yet in seen, it adds a new key with an empty list as value to seen.
This is the result:
import pprint
pprint.pprint(res)
[[{'category': 'Dog', 'name': 'Sasha'}, {'category': 'Dog', 'name': 'Bark'}],
[{'category': 'Cat', 'name': 'Meow'}]]

What is the Pythonic way to iterate over a dict of dicts and lists?

I have a dict which contains some lists and some dicts, as illustrated below.
What is the most pythonic way to iterate over the dict and print out the name and address pairs for each top level dict key?
Thanks
{
'Resent-Bcc': [],
'Delivered-To': [],
'From': {'Name': 'Steve Watson', 'Address': 'steve.watson#example.org'},
'Cc': [],
'Resent-Cc': [],
'Bcc': [ {'Name': 'Daryl Hurstbridge', 'Address': 'daryl.hurstbridge#example.org'},
{'Name': 'Sally Hervorth', 'Address': 'sally.hervorth#example.org'},
{'Name': 'Mike Merry', 'Address': 'mike.merry#example.org'},
{'Name': 'Jenny Callisto', 'Address': 'jenny.callisto#example.org'}
],
'To': {'Name': 'Darius Jedburgh', 'Address': 'darius.jedburgh#example.org'}
}

Use the iteritems() method on the dict. It's clear and easy to understand: that seems Pythonic to me. iteritems() also creates less temporary items than items(), as Preet Kukreti mentioned in the comments. First, fix your data. Right now, some of the values in the top-level dict are lists, and some are more dicts:
# list
'Delivered-To': [],
# dict
'From': {'Name': 'Steve Watson', 'Address': 'steve.watson#example.org'},
This means you have to check the type of the value and act accordingly (and you might forget to check!). Make your data consistent:
# list
'Delivered-To': [],
# also list
'From': [{'Name': 'Steve Watson', 'Address': 'steve.watson#example.org'}],
This will prevent weird type-related bugs in the future. Since Python is an interpreted language, it's very easy to make type bugs and not notice until your code is in production and crashes. Try to make your code as type-safe as possible!
Then you can use something like this:
for k, v in d.iteritems():
for row in v:
if "Name" in row and "Address" in row:
print row["Name"], ":", row["Address"]

One way is to change the lone dicts into a list containing the dict. Then all the entries can be treated the same
>>> D = {
... 'Resent-Bcc': [],
... 'Delivered-To': [],
... 'From': {'Name': 'Steve Watson', 'Address': 'steve.watson#example.org'},
... 'Cc': [],
... 'Resent-Cc': [],
... 'Bcc': [ {'Name': 'Daryl Hurstbridge', 'Address': 'daryl.hurstbridge#example.org'},
... {'Name': 'Sally Hervorth', 'Address': 'sally.hervorth#example.org'},
... {'Name': 'Mike Merry', 'Address': 'mike.merry#example.org'},
... {'Name': 'Jenny Callisto', 'Address': 'jenny.callisto#example.org'}
... ],
... 'To': {'Name': 'Darius Jedburgh', 'Address': 'darius.jedburgh#example.org'}
... }
>>> L = [v if type(v) is list else [v] for v in D.values()]
>>> [(d["Name"], d["Address"]) for item in L for d in item ]
[('Steve Watson', 'steve.watson#example.org'), ('Daryl Hurstbridge', 'daryl.hurstbridge#example.org'), ('Sally Hervorth', 'sally.hervorth#example.org'), ('Mike Merry', 'mike.merry#example.org'), ('Jenny Callisto', 'jenny.callisto#example.org'), ('Darius Jedburgh', 'darius.jedburgh#example.org')]
Or the one liner version
[(d["Name"], d["Address"]) for item in (v if type(v) is list else [v] for v in D.values())]

It's probably best to keep your data simple, by making the naked dict's be a list of one element holding the original dict. Otherwise, you're kind of asking for harder to test code.
I tend to lean away from isinstance(foo, dict) and instead use things like:
if getattr(d, 'iteritems'): print list(d.iteritems())
...It strikes me as more duck-typed this way; it opens the door to using one of the many dict-replacements - things that act like a dict, but nominally aren't a dict.

for key in header:
if header[key] and type(header[key])==type([]):
for item in header[key]:
print (item)
elif type(header[key])==type({}):
print(header[key])
# this option is not the easiest to read, so I classify it as less "pythonic"
l = [header[key] for key in header if header[key] and type(header[key])==type({})] + [header[key][i] for key in header if header[key] and type(header[key])==type([]) for i in range(len(header[key]))]
for item in l:
print(item)
if you're looking for the contents of a specific header you could modify the if statements accordingly. Both of these examples print the dictionaries, but could easily be adapted to print specific values.

for i in dict:
if 'Name' in dict[i]:
print (dict[i]['Name'],dict[i]['Address'])
this will not work for the bcc where its in a list (right now it will only print the from and to names and addresses) Do you need it to print the bcc addresses too?