I want to remove duplicates dictionaries from list of dictionaries. I am trying to make configurable code to work on any field instead of making field specific.
Input Data
dct = {'Customer_Number': 90617174,
'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280.0]},
{'Phone_Type': 'Mobile', 'Phone': [12177218280.0]}],
'Email': [{'Email_Type': 'Primary',
'Email': ['saman.zonouz#rutgers.edu']},
{'Email_Type': 'Primary',
'Email': ['saman.zonouz#rutgers.edu']}]
}
Expected Output:
{'Customer_Number': 90617174,
'Email': [{'Email_Type': 'Primary',
'Email': ['saman.zonouz#rutgers.edu']}],
'Phone_Number': [{'Phone_Type': 'Mobile',
'Phone': [12177218280]}]}
**Code tried:**
res_list = []
for key,value in address_dic.items():
if isinstance(value,list):
for i in range(len(value)):
if value[i] not in value[i + 1:]:
res_list.append(value[i])
dic.append((res_list))
**Output Getting**
type: [[{'Email_Type': 'Primary', 'Email': ['saman.zonouz#rutgers.edu']}, {'Email_Type': 'alternate', 'Email': ['samance#gmail.com', 'saman.zonouz#rutgers.edu']}], [], [{'Phone_Type': 'Mobile', 'Phone': [12177218280.0]}, {'Phone_Type': 'work', 'Phone': [nan]}, {'Phone_Type': 'home', 'Phone': [2177218280.0]}], []]
Write a function to dedupe lists:
def dedupe_list(lst):
result = []
for el in lst:
if el not in result:
result.append(el)
return result
def dedupe_dict_values(dct):
result = {}
for key in dct:
if type(dct[key]) is list:
result[key] = dedupe_list(dct[key])
else:
result[key] = dct[key]
return result
Test it:
deduped_dict = {'Customer_Number': 90617174,
'Email': [{'Email_Type': 'Primary',
'Email': ['saman.zonouz#rutgers.edu']}],
'Phone_Number': [{'Phone_Type': 'Mobile',
'Phone': [12177218280]}]}
dedupe_dict_values(dct) == deduped_dict
## Out[12]: True
I have a json array that looks like this:
{'1': {'ID': '1', ' ': 'Aaron K###', 'Distributor': 'National Energy', 'State': 'BC', 'Brand': 'Trane', 'Cell': '778-###-####', 'email address': '', 'Notes': '', '': ''}, '2': {'ID': '2', ' ': 'Martin', 'Distributor': 'Pierce Phelps', 'State': 'PA', 'Brand': 'Bryant/Carrier', 'Cell': '267-###-####', 'email address': '', 'Notes': '', '': ''},...
and I wanted to reconfigure it so that it matched django's serialization format.
I wrote this function to do it:
def re_serialize_reg_json():
d = load_file()
for i in d:
d[i]['Name'] = d[i][' ']
d[i]['pk'] = d[i]['ID']
d[i]['model'] = 'homepage.territorymanager'
d[i]['fields'] = {
'Name' : d[i]['Name'],
'Cell' : d[i]['Cell'],
'Email Address' : d[i]['email address'],
'Notes' : d[i]['Notes'],
'Distributor' : d[i]['Distributor'],
'State' :d[i]['State'],
'Brand' :d[i]['Brand'],
}
del d[i][' ']
del d[i]['ID']
del d[i]['Name']
del d[i]['Cell']
del d[i]['email address']
del d[i]['Notes']
del d[i]['Distributor']
del d[i]['State']
del d[i]['Brand']
del d[i]['']
and it works fine:
output:
{'1': {'pk': '1', 'model': 'homepage.territorymanager', 'fields': {'Name': 'Aaron Kirkus', 'Cell': '778-875-4983', 'Email Address': '', 'Notes': '', 'Distributor': 'National Energy', 'State': 'BC', 'Brand': 'Trane'}}, '2': {'pk': '2', 'model': 'homepage.territorymanager', 'fields': {'Name': 'Aaron Martin ', 'Cell': '267-246-0522', 'Email Address': '', 'Notes': '', 'Distributor': 'Pierce Phelps', 'State': 'PA', 'Brand': 'Bryant/Carrier'}},...
But I feel like it's not a very effective method for achieving this. Any ideas appreciated.
A list to store some keys, and some loop, would make the code a lot nicer
def re_serialize_reg_json(d):
for key, values in d.items():
values.update({'Name': values[' '], 'pk': values['ID'], 'model': 'homepage.territorymanager'})
f = ['Name', 'Cell', 'email address', 'Notes', 'Distributor', 'State', 'Brand']
values['fields'] = {' '.join(w.capitalize() for w in k.split()): values[k] for k in f}
for k in f + [' ', '', 'ID']:
del values[k]
return d
I have a CSV files which has a header like this:
cpus/0/compatible clocks/HSE/compatible ../frequency memories/flash/compatible ../address ../size [and so on...]
I'm able to parse that header into a nested dictionaries which may look like this:
{'clocks': {'HSE': {'compatible': '[1]',
'frequency': '[2]'}},
'cpus': {'0': {'compatible': '[0]'}},
'memories': {'bkpsram': {'address': '[13]',
'compatible': '[12]',
'size': '[14]'},
'ccm': {'address': '[7]',
'compatible': '[6]',
'size': '[8]'},
'flash': {'address': '[4]',
'compatible': '[3]',
'size': '[5]'},
'sram': {'address': '[10]',
'compatible': '[9]',
'size': '[11]'}},
'pin-controller': {'GPIOA': {'enabled': '[16]'},
'GPIOB': {'enabled': '[17]'},
'GPIOC': {'enabled': '[18]'},
'GPIOD': {'enabled': '[19]'},
'GPIOE': {'enabled': '[20]'},
'GPIOF': {'enabled': '[21]'},
'GPIOG': {'enabled': '[22]'},
'GPIOH': {'enabled': '[23]'},
'GPIOI': {'enabled': '[24]'},
'GPIOJ': {'enabled': '[25]'},
'GPIOK': {'enabled': '[26]'},
'compatible': '[15]'}}
(it is a dict object, printed with pprint())
The values of keys which look like '[<number>]' reflect the index of column in the CSV file from which the data should be loaded.
As I mainly use C/C++ I would actually love to have pointers/references in Python, as then I would just put a pointer to a list element in each value and for each row I could modify list contents, but I think there's no way to obtain such behaviour easily in Python.
So now I plan to dump this dictionary into a string and perform following 3 modifications in a row:
replace { with {{,
replace } with }},
replace '[<number>]' with {<number>}.
After that I will be able to "load" the data with something like this ast.literal_eval(dictAsStr.format(*rowFromCsv)), but it seems like a waste of time to convert the whole dict to a string and then back to a dict...
Am I missing some other obvious solution here? The format of the CSV and the way I load the header is not fixed, I may alter that easily, but I would really like a solution which would not boil down to "visit each key recursively and load appropriate value from current row manually".
From the CSV file I load each row as a list of strings, for example:
['["ARM,Cortex-M4", "ARM,ARMv7-M"]',
'["ST,STM32-HSE", "fixed-clock"]',
'0',
'["on-chip-flash"]',
'0x8000000',
'131072',
'',
'',
'',
'["on-chip-ram"]',
'0x20000000',
'65536',
'',
'',
'',
'["ST,STM32-GPIOv2-pin-controller"]',
'False',
'False',
'False',
'',
'',
'',
'',
'False',
'',
'',
'']
Now I would like to insert the values from each loaded row (list of strings) into appropriate keys in the nested dictionary, so following with the examples above I would like to get:
{'clocks': {'HSE': {'compatible': '["ST,STM32-HSE", "fixed-clock"]',
'frequency': '0'}},
'cpus': {'0': {'compatible': '["ARM,Cortex-M4", "ARM,ARMv7-M"]'}},
'memories': {'bkpsram': {'address': '',
'compatible': '',
'size': ''},
'ccm': {'address': '',
'compatible': '',
'size': ''},
'flash': {'address': '0x8000000',
'compatible': '["on-chip-flash"]',
'size': '131072'},
'sram': {'address': '0x20000000',
'compatible': '["on-chip-ram"]',
'size': '65536'}},
'pin-controller': {'GPIOA': {'enabled': 'False'},
'GPIOB': {'enabled': 'False'},
'GPIOC': {'enabled': 'False'},
'GPIOD': {'enabled': ''},
'GPIOE': {'enabled': ''},
'GPIOF': {'enabled': ''},
'GPIOG': {'enabled': ''},
'GPIOH': {'enabled': 'False'},
'GPIOI': {'enabled': ''},
'GPIOJ': {'enabled': ''},
'GPIOK': {'enabled': ''},
'compatible': '["ST,STM32-GPIOv2-pin-controller"]'}}
For completeness, here are a few first lines from the CSV file I would like to load. The first column is not part of the dictionary presented above, as it is used for indexing.
chip,cpus/0/compatible,clocks/HSE/compatible,../frequency,memories/flash/compatible,../address,../size,memories/ccm/compatible,../address,../size,memories/sram/compatible,../address,../size,memories/bkpsram/compatible,../address,../size,pin-controller/compatible,pin-controller/GPIOA/enabled,pin-controller/GPIOB/enabled,pin-controller/GPIOC/enabled,pin-controller/GPIOD/enabled,pin-controller/GPIOE/enabled,pin-controller/GPIOF/enabled,pin-controller/GPIOG/enabled,pin-controller/GPIOH/enabled,pin-controller/GPIOI/enabled,pin-controller/GPIOJ/enabled,pin-controller/GPIOK/enabled
STM32F401CB,"[""ARM,Cortex-M4"", ""ARM,ARMv7-M""]","[""ST,STM32-HSE"", ""fixed-clock""]",0,"[""on-chip-flash""]",0x8000000,131072,,,,"[""on-chip-ram""]",0x20000000,65536,,,,"[""ST,STM32-GPIOv2-pin-controller""]",False,False,False,,,,,False,,,
STM32F401CC,"[""ARM,Cortex-M4"", ""ARM,ARMv7-M""]","[""ST,STM32-HSE"", ""fixed-clock""]",0,"[""on-chip-flash""]",0x8000000,262144,,,,"[""on-chip-ram""]",0x20000000,65536,,,,"[""ST,STM32-GPIOv2-pin-controller""]",False,False,False,,,,,False,,,
STM32F401CD,"[""ARM,Cortex-M4"", ""ARM,ARMv7-M""]","[""ST,STM32-HSE"", ""fixed-clock""]",0,"[""on-chip-flash""]",0x8000000,393216,,,,"[""on-chip-ram""]",0x20000000,98304,,,,"[""ST,STM32-GPIOv2-pin-controller""]",False,False,False,,,,,False,,,
The code used to parse the header:
import csv
with open("some-path-to-CSV-file") as csvFile:
csvReader = csv.reader(csvFile)
header = next(csvReader)
previousKeyElements = header[1].split('/')
dictionary = {}
for index, key in enumerate(header[1:]):
keyElements = key.split('/')
i = 0
while keyElements[i] == '..':
i += 1
keyElements[0:i] = previousKeyElements[0:-i]
previousKeyElements = keyElements
node = dictionary
for keyElement in keyElements[:-1]:
node = node.setdefault(keyElement, {})
node[keyElements[-1]] = '[{}]'.format(index)
What about just using the actual row index (as integer) as value in the "parsed" header, ie:
{'clocks': {'HSE': {'compatible': 1,
'frequency': 2}},
# etc
Then using recursion on a parsed header copy to populate it from the row values ?:
import csv
import sys
import copy
import pprint
def parse_header(header):
previousKeyElements = header[1].split('/')
dictionary = {}
for index, key in enumerate(header[1:]):
keyElements = key.split('/')
i = 0
while keyElements[i] == '..':
i += 1
keyElements[0:i] = previousKeyElements[0:-i]
previousKeyElements = keyElements
node = dictionary
for keyElement in keyElements[:-1]:
node = node.setdefault(keyElement, {})
node[keyElements[-1]] = index
return dictionary
def _rparse(d, k, v, row):
if isinstance(v, dict):
for subk, subv in v.items():
_rparse(v, subk, subv, row)
elif isinstance(v, int):
d[k] = row[v]
else:
raise ValueError("'v' should be either a dict or an int (got : %s(%s))" % (type(v), v))
def parse_row(header, row):
struct = copy.deepcopy(header)
for k, v in struct.items():
_rparse(struct, k, v, row)
return struct
def main(*args):
path = args[0]
with open(path) as f:
reader = csv.reader(f)
header = parse_header(next(reader))
results = [parse_row(header, row[1:]) for row in reader]
pprint.pprint(results)
if __name__ == "__main__":
main(*sys.argv[1:])
Another solution (that might actually be faster) would be to build a reverse mapping with row indices as keys and dict "path" as values ie:
{0: ("cpus", "0", "compatible"),
1: ("clocks", "HSE", "compatible"),
2: ("clocks", "HSE", "frequency"),
# etc
}
and then:
def parse_row(template, map, row):
# 'template' is your parsed header dict
struct = copy.deepcopy(template)
target = struct
for index, path in map.items():
for key in path[:-1]:
target = target[key]
target[key[-1] = row[index]
Oh and yes, as an added bonus, you may want to use ast.literal_eval() to turn your values into proper python types:
>>> import ast
>>> ast.literal_eval("False")
False
>>> ast.literal_eval('["on-chip-flash"]')
['on-chip-flash']
>>> ast.literal_eval('0x8000000')
134217728
>>> ast.literal_eval('["ARM,Cortex-M4", "ARM,ARMv7-M"]')
['ARM,Cortex-M4', 'ARM,ARMv7-M']
>>> ast.literal_eval("this should fail")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/ast.py", line 49, in literal_eval
node_or_string = parse(node_or_string, mode='eval')
File "/usr/lib/python2.7/ast.py", line 37, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<unknown>", line 1
this should fail
^
SyntaxError: invalid syntax
>>> def to_python(value):
... try:
... return ast.literal_eval(value)
... except Exception as e:
... return value
...
>>> to_python('["on-chip-flash"]')
['on-chip-flash']
>>> to_python('wtf')
'wtf'
>>>
I have the following code in Python:
def buildXmlUpdate(dfrom, roomId, ldays):
start_date_sard.text = dfrom
roomId = str(roomId)
room_id_sard.text = roomId
ldays = {'avail': str(), 'price': str()}
availability_in_data.text = ldays['avail']
price_in_data.text = ldays['price']
for n in ldays:
print (dfrom, roomId, ldays)
Now when running
buildXmlUpdate ('21/12/2015', 1, [{'avail': 1, 'price': 100}, {'avail': 3, 'price': 120}])
I get the following output
('21/12/2015', '1', {'avail': '', 'price': ''})
('21/12/2015', '1', {'avail': '', 'price': ''})
In other words:
('21/12/2015', '1', {'avail': 1, 'price': 100})
('21/12/2015', '1', {'avail': 3, 'price': 120})
As you see here, the dictionary avail and price keys are set to an empty string but I want to set them according to the ldays arguments in the method.
What am I doing wrong?
Solved:
def buildXmlUpdate(dfrom, roomId, ldays):
start_date_sard.text = dfrom
roomId = str(roomId)
room_id_sard.text = roomId
#ldays = {'avail': str(), 'price': str()}
#availability_in_data.text = ldays['avail']
#price_in_data.text = ldays['price']
for n in ldays:
print (dfrom, roomId, n)
#availability_in_data.text = get.ldays['avail']
#price_in_data.txt = get.ldays['price']
ldays[-1]['avail'] = str(ldays[-1]['avail'])
ldays[-1]['price'] =str(ldays[-1]['price'])
availability_in_data.text = ldays[-1]['avail']
price_in_data.text = ldays[-1]['price']
Thank you all!
group1= [ {
'Name': 'C21114',
'Description': '',
'num': '12321114',
'working': 'true',
'belongs': 'Default',
'Expiry_Date': '',
'\xef\xbb\xbfUser_ID': 'C21114',
'Password': '*SECRET*',
},
{
'Name': 'Mahes',
'Description': '',
'num': '1026',
'working': 'true',
'belongs': 'Default',
'Expiry_Date': '',
'\xef\xbb\xbfUser_ID': 'Mahi',
'Password': '*SECRET*',
},
{
'Name': 'pri',
'Description': '',
'num': '1027',
'working': 'true',
'belongs': 'Default',
'Expiry_Date': '',
'\xef\xbb\xbfUser_ID': 'priya',
'Password': '*SECRET*',
}]
group2= [{
'Name': 'C21114',
'Description': '',
'num': '12321114',
'working': 'true',
'belongs': 'Default',
'Expiry_Date': '',
'User_ID': 'C21114',
'Password': '*SECRET*',
},
{
'Name': 'Mahes',
'Description': '',
'num': '1026',
'working': 'true',
'belongs': 'Default',
'Expiry_Date': '',
'User_ID': 'Mahi',
'Password': '*SECRET*',
},
{
'Name': 'pri',
'Description': '',
'num': '1027',
'working': 'true',
'belongs': 'Default',
'Expiry_Date': '',
'User_ID': 'priya',
'Password': '*SECRET*',
}]
Need to compare few keys of group1 and group2 are same or not. group1 and group2 are list in that many dictionaries.I just need to compare few keys with its values between group1 and group2.Explained with one example.Example : keys_to_compare = {'name', 'num',working} from group1 and group2.
This should do what you need it to do
key_to_compare = ['Name', 'num', 'working']
for key in key_to_compare:
for d1 in group1:
for d2 in group2:
if d1[key] == d2[key]:
print "same values for %s %s %s" % (key, d1[key], d2[key])
Change the if statement to do what you would like for elements with the same value.
I had to make an assumption on what the output you wanted. I created a list of lists. The inner most list is a part of indexs (group1 then group2) of matches. This is the code:
keys_to_compare = ['Name','num','working']
matches = []
for idx1 in range(len(group1)):
ref1 = group1[idx1]
found = False
for idx2 in range(len(group2)):
ref2 = group2[idx2]
found = True
for key in keys_to_compare:
if ref1[key] != ref2[key]:
found = False
if found:
matches.append([idx1,idx2])
break
if found: continue
print 'matches=%r' % (matches)
The result:
matches=[[0, 0], [1, 1], [2, 2]]
Implemented as a function, this should give you what you want:
def compare(key):
g1 = []
g2 = []
for i in group1:
g1.append(i[key])
for j in group2:
g2.append(j[key])
return g1 == g2
Put in the key name, then it will return True if the values are same in both groups and False if not. For instance, to check the keys in your list, you'll do:
keys_to_compare = ['Name','num','working']
for x in keys_to_compare:
print compare(x)