We Keep Coding

Get all the keys of a nested dict

With xmltodict I managed to get my code from xml in a dict and now I want to create an excel.
In this excel the header of a value is going to be all the parents (keys in the dict).
For example:
dict = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
The value male will have the header: adres/roommate/gender.

Here's a way to orgainze the data in the way your question asks:
d = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
print(d)
stack = [('', d)]
headerByValue = {}
while stack:
name, top = stack.pop()
if isinstance(top, dict):
stack += (((name + '/' if name else '') + k, v) for k, v in top.items())
else:
headerByValue[name] = top
print(headerByValue)
Output:
{'adres/roommate/gender': 'male',
'adres/number': '5',
'adres/street': 'example1street',
'last-name': 'Pencil',
'name': 'Pete'}

Create new dictionary with specific keys from old dictionary

I want to make a new dictionary that prints a new object containing uuid, name, website, and email address for all rows of my dict that have values for all four of these attributes.
I thought I did this for email, name, and website below in my code but I noticed sometimes name or email wont print (because they have missing values), how do I drop those? Also, uuid is outside of the nested dictionary, how do I add that in the new dictionary too?
I attached my code and an element from my code below.
new2 = {}
for i in range (0, len(json_file)):
try:
check = json_file[i]['payload']
new = {k: v for k, v in check.items() if v is not None}
new2 = {k: new[k] for k in new.keys() & {'name', 'website', 'email'}}
print(new2)
except:
continue
Dictionary sample:
{
"payload":{
"existence_full":1,
"geo_virtual":"[\"56.9459720|-2.1971226|20|within_50m|4\"]",
"latitude":"56.945972",
"locality":"Stonehaven",
"_records_touched":"{\"crawl\":8,\"lssi\":0,\"polygon_centroid\":0,\"geocoder\":0,\"user_submission\":0,\"tdc\":0,\"gov\":0}",
"address":"The Lodge, Dunottar",
"email":"dunnottarcastle#btconnect.com",
"existence_ml":0.5694238217658721,
"domain_aggregate":"",
"name":"Dunnottar Castle",
"search_tags":[
"Dunnottar Castle Aberdeenshire",
"Dunotter Castle"
],
"admin_region":"Scotland",
"existence":1,
"category_labels":[
[
"Landmarks",
"Buildings and Structures"
]
],
"post_town":"Stonehaven",
"region":"Kincardineshire",
"review_count":"719",
"geocode_level":"within_50m",
"tel":"01569 762173",
"placerank":65,
"longitude":"-2.197123",
"placerank_ml":37.27916073464469,
"fax":"01330 860325",
"category_ids_text_search":"",
"website":"http://www.dunnottarcastle.co.uk",
"status":"1",
"geocode_confidence":"20",
"postcode":"AB39 2TL",
"category_ids":[
108
],
"country":"gb",
"_geocode_quality":"4"
},
"uuid":"3867aaf3-12ab-434f-b12b-5d627b3359c3"
}

Try using the dict.get() method:
def new_dict(input_dict, keys, fallback='payload'):
ret = dict()
for key in keys:
val = input_dict.get(key) or input_dict[fallback].get(key)
if val:
ret.update({key:val})
if len(ret) == 4: # or you could do: if set(ret.keys()) == set(keys):
print(ret)
for dicto in json_file:
new_dict(dicto, ['name','website','email','uuid'])
{'name': 'Dunnottar Castle', 'website': 'http://www.dunnottarcastle.co.uk', 'email': 'dunnottarcastle#btconnect.com', 'uuid': '3867aaf3-12ab-434f-b12b-5d627b3359c3'}

Extract matching fileds from dictonary

is there some smarter way to extract all keys from an OrderedDict into an new dictionary?
For example: I will collect all keys matching with 'pre' into the dict 'cfg_pre'. I tried this (it works):
#!/usr/bin/env python
import collections
dl=collections.OrderedDict()
dl ={
'pre_enable': True,
'pre_path': '/usr/bin/pre_script.sh',
'pre_args': '-la -w --dir',
'post_enable': True,
'post_path': '/usr/bin/post_script.sh',
'fail_enable': True,
'fail_path': '/usr/bin/failure_script.sh',
'fail_args': '--debug 4'
}
cfg_pre = dict((k,v) for k, v in dl.items() if 'pre' in k)
cfg_post= dict((k,v) for k, v in dl.items() if 'post' in k)
cfg_fail=dict((k,v) for k, v in dl.items() if 'fail' in k)
print (cfg_pre)
print ("---")
print(cfg_post)
print ("---")
print(cfg_fail)
print ("---")
The keys in the dl-dict always starts with pre, post or fail.
Thanks ;)

EDIT -> Updated to fit your need
If you would like to generalize, I would suggest going to group by methods. Something like :
import itertools as it
#define a function to split your universe (here the beginning of your keys)
def keyfunc(key):
return key.split('_')[0]
#group by prefixes
groups = {k:list(g) for k,g in it.groupby(dl, keyfunc)}
#build your output
output = {g:[(key,dl[key]) for key in v] for g,v in groups.items()}
you will get the following output:
{'fail': [('fail_enable', True),
('fail_path', '/usr/bin/failure_script.sh'),
('fail_args', '--debug 4')],
'post': [('post_enable', True), ('post_path', '/usr/bin/post_script.sh')],
'pre': [('pre_enable', True),
('pre_path', '/usr/bin/pre_script.sh'),
('pre_args', '-la -w --dir')]}

I would say that three lines is short enough for such a problem.
It would be a little cleaner with dict comprehensions:
cfg_pre = {k: v for k, v in dl.items() if 'pre' in k}
cfg_post = {k: v for k, v in dl.items() if 'post' in k}
cfg_fail = {k: v for k, v in dl.items() if 'fail' in k}
You could also remove the redundant data like this:
cfg_pre = {k.replace('pre_', ''): v for k, v in dl.items() if 'pre' in k}
cfg_post = {k.replace('post_', ''): v for k, v in dl.items() if 'post' in k}
cfg_fail = {k.replace('fail_', ''): v for k, v in dl.items() if 'fail' in k}
The dictionaries will be more consistent and the information about pre, post and fail will be stored only in the variable name:
cfg_pre = {'args': '-la -w --dir', 'enable': True, 'path': '/usr/bin/pre_script.sh'}
cfg_post = {'enable': True, 'path': '/usr/bin/post_script.sh'}
cfg_fail = {'args': '--debug 4', 'enable': True, 'path': '/usr/bin/failure_script.sh'}

Matching value of 1 dictionary with key of another dictionary using Python

I have 2 dictionaries. Is it possible to loop DictA's key with DictB's values, and if DictB's value has a match, it would return DictB's key.
​
Is it possible to be done?
Please give me some advice/tips. What am I doing wrong? Thank you.
Example:
A: {'WNT3A': '1732', 'RG59L': ' ', 'SCZD9': ' ', 'CD241': '6005', 'C2': '', 'RH': ' '}
B: {'': [''], '6005': ['RH50A', ' CD241', ' SLC42A1'], '603': [''], '6000': [''], '8787': ['PERRS', ' RGS9L', ' MGC26458'], '41': ['ACCN2', ' BNaC2', ' hBNaC2'], '8490': [''], '9628': [''], '5999': ['SCZD9']}
Result :
new_A: {'WNT3A': '1732', 'RG59L': ' ', 'SCZD9': '5999 ', 'CD241': '6005', 'C2': '', 'RH': ' '}​
So far, I've coded this but it only seems to return me a dictionary that is whole or a dictionary containing only dictionary B.
new_a = {}
for key in ref_dict:
for value in (map(lambda x: x.strip(), ref_dict[key][0])):
if(not fill_dict.has_key(key)):
continue
elif(ref_dict[value] != fill_dict[key]):
new_a[value] = (ref_dict[key], fill_dict[key])
print new_a
Result:
{'WNT3A': '1732', 'RG59L': '', 'SCZD9': '', 'CD241': '6005', 'C2': '', 'RH': ''}
Another code I've tried is :
new_dict = {}
for k, v in fill_dict.iteritems():
vals = []
if isinstance(v, list):
for i in v:
vals.append(ref_dict.get(i))
else:
vals.append(ref_dict.get(v))
if not vals:
continue
new_dict[k] = vals
print new_dict
Result:
{'WNT3A': [None], 'RG59L': [['']], 'SCZD9': [['']], 'CD241': [['RH50A', ' CD241', ' SLC42A1']], 'C2': [['']], 'RH': [['']]}

You can use a dict-comprehension with next() for this:
>>> {k: next((k1 for k1, v1 in B.items() if k in v1), v) for k, v in A.items()}
{'WNT3A': '1732', 'RG59L': ' ', 'SCZD9': '5999', 'CD241': '6005', 'C2': '', 'RH': ' '}
Here I've used next(<gen exp>, v), if the generator expression returned an empty iterator(i.e if none of the B's values contains the key k) then simply use the value v from A, otherwise use the key returned from the generator expression.
The above code can also be written as:
d = {}
for k, v in A.items():
for k1, v1 in B.items():
if k in v1:
d[k] = k1
break
else:
#this will be executed if no value in `B` contains `k`
d[k] = v

This will update dictionary a with the values you want:
for key_a in a:
for key_b, value_b in b.iteritems():
for item in value_b:
if item == key_a:
a[key_a] = key_b
If you need a whole new dict, then just make a copy of a and update that one.

requests form-urlencoded data

UPDATE:
I think I'm a little closer now, if change my data structure to the following and use:
urllib.urlencode(data, doseq=True)
data ={'Properties': [('key', 'KeyLength'), ('value', '512')], 'Category': 'keysets', 'Offset': '0', 'Limit': '10'}
I now get the following which is a lot closer, but still not quite correct:
Category=keysets&Limit=10&Properties=('key', 'KeyLength')&Properties=('value', '512')&Offset=0
I'm re-writing this question because I think I know what the problem is, but still don't quite know how to fix it.
I think the problem relates to the fact that the form data I need to send contains form fields with the same name. i.e. the 'Properties'. This is my data structure:
data = {'Properties': [{'key': 'KeyLength', 'value': '512'}], 'Category': 'keysets', 'Offset': '0', 'Limit': '100'}
and this is how it should appear when received by the web service:
Form Data:
Properties[0][key]:KeyLength
Properties[0][value]:768
Category:
Offset:0
Limit:100
I'm posting using 'requests' this:
req = requests.post('http://server1/ws1/api/data/filters/', data=data)
However it seems to end up like this:
Category=keysets&Limit=100&Properties=('key', 'KeyLength')&Offset=0
Instead of like this:
Properties[0][key]=KeyLength&Properties[0][value]=768&Category=&Offset=0&Limit=100
Can somebody please advise what I'm doing wrong.

You won't be able to get that directly. From Python Standard Library Manual, Python url encoding works that way : Convert a mapping object or a sequence of two-element tuples.
There is no support for exploding list or hash values like you want : you will have to preprocess your data. You could try a function like that :
def transform(h, resul = None, kk=None):
if resul is None:
resul = {}
for (k, v) in h.items():
key = k if kk is None else "%s[%s]" % (kk, k)
if isinstance(v, list) or isinstance(v, tuple):
for i, v1 in enumerate(v):
transform(v1, resul, '%s[%d]' % (key, i))
elif isinstance(v, dict):
for i, v1 in v.items:
transform(v1, resul, '%s[%s]' % (key, i))
else:
resul[key] = v
return resul
With you original data structure, it gives :
>>> data = {'Properties': [{'key': 'KeyLength', 'value': '512'}], 'Category': 'keysets', 'Offset': '0', 'Limit': '100'}
>>> transform(data)
{'Category': 'keysets', 'Limit': '100', 'Properties[0][key]': 'KeyLength', 'Offset': '0', 'Properties[0][value]': '512'}
And then it would be encoded as per your requirements

It actually seems that the answer contributed by Serge will not play nice with lists embedded down in the tree. (they don't have an items() method)
Here's a variation on the idea that's working for me:
def flattenForPost(h, resul = None, kk=None):
if resul is None:
resul = {}
if isinstance(h, str) or isinstance(h, bool):
resul[kk] = h
elif isinstance(h, list) or isinstance(h, tuple):
for i, v1 in enumerate(h):
flattenForPost(v1, resul, '%s[%d]' % (kk, i))
elif isinstance(h, dict):
for (k, v) in h.items():
key = k if kk is None else "%s[%s]" % (kk, k)
if isinstance(v, dict):
for i, v1 in v.items():
flattenForPost(v1, resul, '%s[%s]' % (key, i))
else:
flattenForPost(v, resul, key)
return resul