We Keep Coding

Create many lists from list of dictionaries that have different structures

I have a big list of dictionaries. And the dictionaries don´t have the same structure. But, I don´t know all the structures (my list has 1 million elements).
For example:
mylist = [{'name': 'Juan Carlos','age':38},{'name':'David','country':'Brazil'},
{'name':'Agustina', 'country': 'Argentina'},{'name': 'Renzo','age':24}]
I want to separate mylist into different lists, and each list has dictionaries with the same keys.
For example:
list1 = [{'name': 'Juan Carlos','age':38},{'name': 'Renzo','age':24}]
list2 = [{'name':'David','country':'Brazil'},{'name':'Agustina', 'country': 'Argentina'}]
The problem is I don´t know how many sublists I'm going to have.

To do this, you need to create a dictionary where the key is the set of keys that are present:
mylist = [{'name': 'Juan Carlos','age':38},{'name':'David','country':'Brazil'},{'name':'Agustina', 'country': 'Argentina'},{'name': 'Renzo','age':24}]
sublists = {}
for sl in mylist:
keys = tuple(set(sl.keys()))
if keys not in sublists:
sublists[keys] = []
sublists[keys].append( sl )
print( sublists )
Output:
{('name', 'age'): [{'name': 'Juan Carlos', 'age': 38}, {'name': 'Renzo', 'age': 24}], ('name', 'country'): [{'name': 'David', 'country': 'Brazil'}, {'name': 'Agustina', 'country': 'Argentina'}]}

You can use a frozenset of keys as a new key:
mylist = [{'name': 'Juan Carlos','age':38},{'name':'David','country':'Brazil'},
{'name':'Agustina', 'country': 'Argentina'},{'name': 'Renzo','age':24}]
sep = {}
for d in mylist:
sep.setdefault(frozenset(d), []).append(d)
and then you can get a list of lists:
print(sep.values())

How can I use list comprehension to separate values in a dictionary?

name=[]
age=[]
address=[]
...
for line in pg:
for key,value in line.items():
if key == 'name':
name.append(value)
elif key == 'age':
age.append(value)
elif key == 'address':
address.append(value)
.
.
.
Is it possible to use list comprehension for above code because I need to separate lots of value in the dict? I will use the lists to write to a text file.
Source Data:
a = [{'name': 'paul', 'age': '26.', 'address': 'AU', 'gender': 'male'},
{'name': 'mei', 'age': '26.', 'address': 'NY', 'gender': 'female'},
{'name': 'smith', 'age': '16.', 'address': 'NY', 'gender': 'male'},
{'name': 'raj', 'age': '13.', 'address': 'IND', 'gender': 'male'}]

I don't think list comprehension will be a wise choice because you have multiple lists.
Instead of making multiple lists and appending to them the value if the key matches you can use defaultdict to simplify your code.
from collections import defaultdict
result = defaultdict(list)
for line in pg:
for key, value in line.items():
result[key].append(value)
You can get the name list by using result.get('name')
['paul', 'mei', 'smith', 'raj']

This probably won't work the way you want: Your'e trying to assign the three different lists, so you would need three different comprehensions. If your dict is large, this would roughly triple your execution time.
Something straightforward, such as
name = [value for for key,value in line.items() if key == "name"]
seems to be what you'd want ... three times.

You can proceed as :
pg=[{"name":"name1","age":"age1","address":"address1"},{"name":"name2","age":"age2","address":"address2"}]
name=[v for line in pg for k,v in line.items() if k=="name"]
age=[v for line in pg for k,v in line.items() if k=="age"]
address=[v for line in pg for k,v in line.items() if k=="address"]

In continuation with Vishal's answer, please dont use defaultdict. Using defaultdict is a very bad practice when you want to catch keyerrors. Please use setdefault.
results = dict()
for line in pg:
for key, value in line.items():
result.setdefault(key, []).append(value)
Output
{
'name': ['paul', 'mei', 'smith', 'raj'],
'age': [26, 26, 26, 13],
...
}
However, note that if all dicts in pg dont have the same keys, you will lose the relation/correspondence between the items in the dict

Here is a really simple solution if you want to use pandas:
import pandas as pd
df = pd.DataFrame(a)
name = df['name'].tolist()
age = df['age'].tolist()
address = df['address'].tolist()
print(name)
print(age)
print(address)
Output:
['paul', 'mei', 'smith', 'raj']
['26.', '26.', '16.', '13.']
['AU', 'NY', 'NY', 'IND']
Additionally, if your end result is a text file, you can skip the list creation and write the DataFrame (or parts thereof) directly to a CSV with something as simple as:
df.to_csv('/path/to/output.csv')

What is the best way to convert key:value pairs from bytestring to string?

Lets say I have a dict:
{b'Name': b'John', b'age': b'43'}
what would be the best way to convert that to:
{'Name': 'John', 'age': '43'}
(Considering that there can be any number of key:value pairs)
This is what I have right now:
new_d = dict()
old_d = {b'Name': b'John', b'age': b'43'}
for item in old_d:
print(item, old_d[item])
new_d[item.decode('ascii')] = old_d[item].decode('ascii')
print(new_d)
Output:
{'Name': 'John', 'age': '43'}

d = { b'Name': b'John', b'age': b'43' }
d = { x.decode('ascii'): d.get(x).decode('ascii') for x in d.keys() }

Try this:
x = {b'Name': b'John', b'age': b'43'}
y = {}
for key, value in x.items():
y[key.decode("utf-8")] = value.decode("utf-8")
y
> {'Name': 'John', 'age': '43'}

I cant test this right now, but you should be able to decode everything into a new dictionary. You should specify the encoding- I think UTF-8 is a good default.
enc = 'utf-8'
s = {k.decode(enc), v.decode(enc) for k, v in d.items()}

Python - Return list of dictionaries with unique Key:Value pair

I have a long list of dictionaries that for the most part do not overlap. However, some of the dictionaries have the same 'Name' field and I'd only like unique names in the list of dictionaries. I'd like the first occurrence of the name to be the one that stays and any thereafter be deleted from the list.
I've put a short list below to illustrate the scenario:
myList = [
{'Name':'John', 'Age':'50', 'Height':'70'},
{'Name':'Kathy', 'Age':'43', 'Height':'65'},
{'Name':'John','Age':'46','Height':'68'},
{'Name':'John','Age':'50','Height':'72'}
]
I'd like this list to return the first 'John' and Kathy, but not the second or third Johns and their related information.
An acceptable, but not optimal solution would also be not having dictionaries with the same name next to each other.

You could run over the list and keep a set of unique names. Every time you encounter a new name (i.e., a name that isn't in the set), you add it to the set and the respective dict to the result:
def uniqueNames(dicts):
names = set()
result = []
for d in dicts:
if not d['Name'] in names:
names.add(d['Name'])
result.append(d)
return result

You can easily write a for-loop for this.
def getName(name):
'''Gets first occurence of name in list of dicts.'''
for i in myList:
if i['Name'] == name:
return i

Initial list:
my_list = [
{'Name':'John', 'Age':'50', 'Height':'70'},
{'Name':'Kathy', 'Age':'43', 'Height':'65'},
{'Name':'John','Age':'46','Height':'68'},
{'Name':'John','Age':'50','Height':'72'}
]
The logical (potentially newbie-friendlier) way:
names = set()
new_list = []
for d in my_list:
name = d['Name']
if name not in names:
new_list.append(d)
names.add(d['Name'])
print new_list # [{'Age': '50', 'Name': 'John', 'Height': '70'}, {'Age': '43', 'Name': 'Kathy', 'Height': '65'}]
A one-liner way:
new_list = {d['Name']: d for d in reversed(my_list)}.values()
print new_list # [{'Age': '43', 'Name': 'Kathy', 'Height': '65'}, {'Age': '50', 'Name': 'John', 'Height': '70'}]
Note: The one-liner will contain the first occurrence of each name, but it will return an arbitrarily ordered list.

string to list of dictionaries (python)

I have a string that needs to be split 3 ways and then into a list of dictionaries.
given_string = 'name:mickey,age:58|name:minnie,age:47,weight:60'
data = []
data = [value.split(',') for value in given_string.split('|')]
data = [['name:mickey', 'age:58'], ['name:minnie', 'age:47', 'weight:60']]
Now I want to split this one more time on the ':' and have the data contain a list of two dictionaries so that when I input say data[1][age], I get 47.
Basically, I think I want this for it to work:
data = [{'name': 'mickey', 'age': '58}, {'name': 'minnie', 'age': '47', 'weight': '60'}]
I believe that ultimately, data should be a list of dictionaries but once I split the string into two lists, I get confused in splitting it on the ':' and then converting the sublists to a dictionary.

You can do with a simple list comprehension
>>> [dict(x.split(':') for x in parts.split(','))
for parts in given_string.split('|')]
[{'age': '58', 'name': 'mickey'}, {'age': '47', 'name': 'minnie', 'weight': '60'}]

Nest harder.
>>> [ dict(y.split(':') for y in x.split(',')) for x in 'name:mickey,age:58|name:minnie,age:47,weight:60'.split('|')]
[{'age': '58', 'name': 'mickey'}, {'age': '47', 'name': 'minnie', 'weight': '60'}]

given_string = 'name:mickey,age:58|name:minnie,age:47,weight:60'
data = [value.split(',') for value in given_string.split('|')]
y=[] # make a empty list
for i in data:
z={}
for v in range(len(i)):
b=i[v].split(":") # ['name", "mickey', 'age","58"]
z[b[0]]=b[1] # adding keys and values in dictionary z
y.append(z) # adding dictionary to the list