UPDATE:
I think I'm a little closer now, if change my data structure to the following and use:
urllib.urlencode(data, doseq=True)
data ={'Properties': [('key', 'KeyLength'), ('value', '512')], 'Category': 'keysets', 'Offset': '0', 'Limit': '10'}
I now get the following which is a lot closer, but still not quite correct:
Category=keysets&Limit=10&Properties=('key', 'KeyLength')&Properties=('value', '512')&Offset=0
I'm re-writing this question because I think I know what the problem is, but still don't quite know how to fix it.
I think the problem relates to the fact that the form data I need to send contains form fields with the same name. i.e. the 'Properties'. This is my data structure:
data = {'Properties': [{'key': 'KeyLength', 'value': '512'}], 'Category': 'keysets', 'Offset': '0', 'Limit': '100'}
and this is how it should appear when received by the web service:
Form Data:
Properties[0][key]:KeyLength
Properties[0][value]:768
Category:
Offset:0
Limit:100
I'm posting using 'requests' this:
req = requests.post('http://server1/ws1/api/data/filters/', data=data)
However it seems to end up like this:
Category=keysets&Limit=100&Properties=('key', 'KeyLength')&Offset=0
Instead of like this:
Properties[0][key]=KeyLength&Properties[0][value]=768&Category=&Offset=0&Limit=100
Can somebody please advise what I'm doing wrong.
You won't be able to get that directly. From Python Standard Library Manual, Python url encoding works that way : Convert a mapping object or a sequence of two-element tuples.
There is no support for exploding list or hash values like you want : you will have to preprocess your data. You could try a function like that :
def transform(h, resul = None, kk=None):
if resul is None:
resul = {}
for (k, v) in h.items():
key = k if kk is None else "%s[%s]" % (kk, k)
if isinstance(v, list) or isinstance(v, tuple):
for i, v1 in enumerate(v):
transform(v1, resul, '%s[%d]' % (key, i))
elif isinstance(v, dict):
for i, v1 in v.items:
transform(v1, resul, '%s[%s]' % (key, i))
else:
resul[key] = v
return resul
With you original data structure, it gives :
>>> data = {'Properties': [{'key': 'KeyLength', 'value': '512'}], 'Category': 'keysets', 'Offset': '0', 'Limit': '100'}
>>> transform(data)
{'Category': 'keysets', 'Limit': '100', 'Properties[0][key]': 'KeyLength', 'Offset': '0', 'Properties[0][value]': '512'}
And then it would be encoded as per your requirements
It actually seems that the answer contributed by Serge will not play nice with lists embedded down in the tree. (they don't have an items() method)
Here's a variation on the idea that's working for me:
def flattenForPost(h, resul = None, kk=None):
if resul is None:
resul = {}
if isinstance(h, str) or isinstance(h, bool):
resul[kk] = h
elif isinstance(h, list) or isinstance(h, tuple):
for i, v1 in enumerate(h):
flattenForPost(v1, resul, '%s[%d]' % (kk, i))
elif isinstance(h, dict):
for (k, v) in h.items():
key = k if kk is None else "%s[%s]" % (kk, k)
if isinstance(v, dict):
for i, v1 in v.items():
flattenForPost(v1, resul, '%s[%s]' % (key, i))
else:
flattenForPost(v, resul, key)
return resul
Related
With xmltodict I managed to get my code from xml in a dict and now I want to create an excel.
In this excel the header of a value is going to be all the parents (keys in the dict).
For example:
dict = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
The value male will have the header: adres/roommate/gender.
Here's a way to orgainze the data in the way your question asks:
d = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
print(d)
stack = [('', d)]
headerByValue = {}
while stack:
name, top = stack.pop()
if isinstance(top, dict):
stack += (((name + '/' if name else '') + k, v) for k, v in top.items())
else:
headerByValue[name] = top
print(headerByValue)
Output:
{'adres/roommate/gender': 'male',
'adres/number': '5',
'adres/street': 'example1street',
'last-name': 'Pencil',
'name': 'Pete'}
I`m trying to parse a spreadsheet with a header that looks something like this:
My problem is those nested keys below "Контрагент". I decided to parse it like this:
['Дата',
'Номер документа',
'Дебет',
'Кредит',
['Контрагент',
['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа',
'Код дебитора',
'Тип документа']
But now, I don`t really have an idea as how to map it to a flat list of values:
['21.05.2021',
'591324565436',
'0.00',
'526345428.99',
'asdasd',
'234525460140679',
'77130100123412341',
'302328105423534200000000280',
'0445252345234974',
'asdfsadfsd',
'sdfghsfgdhfdghdfgh',
'',
'dfghfgdhfdgh']
Given these variables, I want a function to return following dict:
{
"Дата": "21.05.2021",
"Номер документа": "591324565436",
"Дебет": "0.00",
"Кредит": "526345428.99",
"Контрагент": {
"Наименование": "asdasd",
"ИНН": "234525460140679",
"КПП": "77130100123412341",
"Счет": "302328105423534200000000280",
"БИК": "0445252345234974",
"Наименование банка": "asdfsadfsd"
},
"Назначение платежа": "sdfghsfgdhfdghdfgh",
"Код дебитора": "",
"Тип документа": "dfghfgdhfdgh"
}
I've gone this far without realizing it'd be raising IndexError on the 3rd line:
def map_to_schema(schema, data):
for i, elem in enumerate(data):
key = schema[i]
if isinstance(key, list):
if key[0] not in result:
result[key[0]] = {}
result[key[0]] |= {
key[1][i-len(key)]: elem
}
else:
result[key] = elem
What should I do? Maybe the structure for the schema isn't good enough? I really have no idea...
You could use a dictionary comprehension and an iterator:
headers = ['Дата', 'Номер документа', 'Дебет', 'Кредит',
['Контрагент', ['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа', 'Код дебитора', 'Тип документа']
values = ['21.05.2021', '591324565436', '0.00', '526345428.99', 'asdasd', '234525460140679', '77130100123412341',
'302328105423534200000000280', '0445252345234974', 'asdfsadfsd', 'sdfghsfgdhfdghdfgh', '',
'dfghfgdhfdgh']
it = iter(values)
out = {k[0] if (islist := isinstance(k, list)) else k:
{k2: next(it) for k2 in k[1]} if islist else next(it)
for k in headers}
output:
{'Дата': '21.05.2021',
'Номер документа': '591324565436',
'Дебет': '0.00',
'Кредит': '526345428.99',
'Контрагент': {'Наименование': 'asdasd',
'ИНН': '234525460140679',
'КПП': '77130100123412341',
'Счет': '302328105423534200000000280',
'БИК': '0445252345234974',
'Наименование банка': 'asdfsadfsd'},
'Назначение платежа': 'sdfghsfgdhfdghdfgh',
'Код дебитора': '',
'Тип документа': 'dfghfgdhfdgh'}
Thanks #mozway for this solution! This is essentially the same algorithm, using a for loop.
def map(schema, s_length, row: list):
# If len(row) was less then *true* schema length, it would have thrown StopIteration.
# I ended up just extending row list by delta elements.
if (delta := s_length - len(row)) > 0:
row.extend([""] * delta)
iter_row = iter(row)
result = {}
for key in schema:
if isinstance(key, list):
result[key[0]] = {}
for sub_key in key[1]:
result[key[0]][sub_key] = next(iter_row)
else:
result[key] = next(iter_row)
return result
This is my code:
I'm trying to use the following code to insert data into an array of dictionaries but unable to insert properly.
Code:
test_list = {'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5', 'module_name-2': 'CH1.FM6', 'module_serial-2': 'PSUXA12345681'}
def parse_subdevice_modules(row):
modules = []
module = {}
for k, v in row.items():
if v:
if re.match("module_name", k):
module['name'] = v
if re.match("module_serial", k):
module['serial'] = v
modules.append(module)
module = {}
return modules
print(parse_subdevice_modules(test_list))
Expected output:
[{'name':'CH1.FM5', serial': 'PSUXA12345680'}, {'name': 'CH1.FM6', 'serial': 'PSUXA12345681'}]
Actual output:
['serial': 'PSUXA12345680'}, {'name': 'CH1.FM6', 'serial': 'PSUXA12345681'}]
Run it here: https://repl.it/repls/WetSteelblueRange
Please note that the order of the data test_list cannot be altered as it comes via an external API so I used regex. Any ideas would be appreciated.
Your code relies on the wrong assumption that keys are ordered and that the serial will always follow the name. The proper solution here is to use a dict (actually a collections.defaultdict to make things easier) to collect and regroup the values you're interested in based on the module number (the final '-N' in the key). Note that you don't need regexps here - Python string already provide the necessary operations for this task:
from collections import defaultdict
def parse_subdevice_modules(row):
modules = defaultdict(dict)
for k, v in row.items():
# first get rid of what we're not interested in
if not v:
continue
if not k.startswith("module_"):
continue
# retrieve the key number (last char) with
# negative string indexing:
key_num = k[-1]
# retrieve the useful part of the key ("name" or "serial")
# by splitting the string:
key_name = k.split("_")[1].split("-")[0]
# and now we just have to store this in our defaultdict
modules[key_num][key_name] = v
# and return only the values.
# NB: in py2.x you don't need the call to `list`,
# you can just return `modules.values()` directly
modules = list(modules.values())
return modules
test_list = {
'profile': '', 'chassis_name': '123', 'supplier_order_num': '',
'device_type': 'mass_storage', 'device_subtype': 'flashblade',
'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5',
'module_name-2': 'CH1.FM6', 'rack_total_pos': '',
'asset_tag': '002000027493', 'module_serial-2': 'PSUXA12345681',
'purchase_order': '0004530869', 'build': 'Test_Build_for_SNOW',
'po_line_num': '00190', 'mac_address': '', 'position': '7',
'model': 'FB-528TB-10X52.8TB', 'manufacturer': 'PureStorage',
'rack': 'Test_Rack_2', 'serial': 'PMPAM1842147D', 'name': 'FB02'
}
print(parse_subdevice_modules(test_list))
You can do somthing like this also.
test_list = {'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5', 'module_name-2': 'CH1.FM6',
'module_serial-2': 'PSUXA12345681'}
def parse_subdevice_modules(row):
modules_list = []
for key, value in row.items():
if not value or key.startswith('module_name'):
continue
if key.startswith('module_serial'):
module_name_key = f'module_name-{key.split("-")[-1]}'
modules_list.append({'serial': value, 'name': row[module_name_key]})
return modules_list
print(parse_subdevice_modules(test_list))
Output:
[{'serial': 'PSUXA12345680', 'name': 'CH1.FM5'}, {'serial': 'PSUXA12345681', 'name': 'CH1.FM6'}]
You would need to check if module contains 2 elements and append it to modules:
test_list = {'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5', 'module_name-2': 'CH1.FM6', 'module_serial-2': 'PSUXA12345681'}
def parse_subdevice_modules(row):
modules = []
module = {}
for k, v in row.items():
if v:
if k.startswith('module_name'):
module['name'] = v
elif k.startswith("module_serial"):
module['serial'] = v
if len(module) == 2:
modules.append(module)
module = {}
return modules
print(parse_subdevice_modules(test_list))
Returns:
[{'serial': 'PSUXA12345680'}, {'name': 'CH1.FM5'}, {'name': 'CH1.FM6'}, {'serial': 'PSUXA12345681'}]
I want to make a new dictionary that prints a new object containing uuid, name, website, and email address for all rows of my dict that have values for all four of these attributes.
I thought I did this for email, name, and website below in my code but I noticed sometimes name or email wont print (because they have missing values), how do I drop those? Also, uuid is outside of the nested dictionary, how do I add that in the new dictionary too?
I attached my code and an element from my code below.
new2 = {}
for i in range (0, len(json_file)):
try:
check = json_file[i]['payload']
new = {k: v for k, v in check.items() if v is not None}
new2 = {k: new[k] for k in new.keys() & {'name', 'website', 'email'}}
print(new2)
except:
continue
Dictionary sample:
{
"payload":{
"existence_full":1,
"geo_virtual":"[\"56.9459720|-2.1971226|20|within_50m|4\"]",
"latitude":"56.945972",
"locality":"Stonehaven",
"_records_touched":"{\"crawl\":8,\"lssi\":0,\"polygon_centroid\":0,\"geocoder\":0,\"user_submission\":0,\"tdc\":0,\"gov\":0}",
"address":"The Lodge, Dunottar",
"email":"dunnottarcastle#btconnect.com",
"existence_ml":0.5694238217658721,
"domain_aggregate":"",
"name":"Dunnottar Castle",
"search_tags":[
"Dunnottar Castle Aberdeenshire",
"Dunotter Castle"
],
"admin_region":"Scotland",
"existence":1,
"category_labels":[
[
"Landmarks",
"Buildings and Structures"
]
],
"post_town":"Stonehaven",
"region":"Kincardineshire",
"review_count":"719",
"geocode_level":"within_50m",
"tel":"01569 762173",
"placerank":65,
"longitude":"-2.197123",
"placerank_ml":37.27916073464469,
"fax":"01330 860325",
"category_ids_text_search":"",
"website":"http://www.dunnottarcastle.co.uk",
"status":"1",
"geocode_confidence":"20",
"postcode":"AB39 2TL",
"category_ids":[
108
],
"country":"gb",
"_geocode_quality":"4"
},
"uuid":"3867aaf3-12ab-434f-b12b-5d627b3359c3"
}
Try using the dict.get() method:
def new_dict(input_dict, keys, fallback='payload'):
ret = dict()
for key in keys:
val = input_dict.get(key) or input_dict[fallback].get(key)
if val:
ret.update({key:val})
if len(ret) == 4: # or you could do: if set(ret.keys()) == set(keys):
print(ret)
for dicto in json_file:
new_dict(dicto, ['name','website','email','uuid'])
{'name': 'Dunnottar Castle', 'website': 'http://www.dunnottarcastle.co.uk', 'email': 'dunnottarcastle#btconnect.com', 'uuid': '3867aaf3-12ab-434f-b12b-5d627b3359c3'}
is there some smarter way to extract all keys from an OrderedDict into an new dictionary?
For example: I will collect all keys matching with 'pre' into the dict 'cfg_pre'. I tried this (it works):
#!/usr/bin/env python
import collections
dl=collections.OrderedDict()
dl ={
'pre_enable': True,
'pre_path': '/usr/bin/pre_script.sh',
'pre_args': '-la -w --dir',
'post_enable': True,
'post_path': '/usr/bin/post_script.sh',
'fail_enable': True,
'fail_path': '/usr/bin/failure_script.sh',
'fail_args': '--debug 4'
}
cfg_pre = dict((k,v) for k, v in dl.items() if 'pre' in k)
cfg_post= dict((k,v) for k, v in dl.items() if 'post' in k)
cfg_fail=dict((k,v) for k, v in dl.items() if 'fail' in k)
print (cfg_pre)
print ("---")
print(cfg_post)
print ("---")
print(cfg_fail)
print ("---")
The keys in the dl-dict always starts with pre, post or fail.
Thanks ;)
EDIT -> Updated to fit your need
If you would like to generalize, I would suggest going to group by methods. Something like :
import itertools as it
#define a function to split your universe (here the beginning of your keys)
def keyfunc(key):
return key.split('_')[0]
#group by prefixes
groups = {k:list(g) for k,g in it.groupby(dl, keyfunc)}
#build your output
output = {g:[(key,dl[key]) for key in v] for g,v in groups.items()}
you will get the following output:
{'fail': [('fail_enable', True),
('fail_path', '/usr/bin/failure_script.sh'),
('fail_args', '--debug 4')],
'post': [('post_enable', True), ('post_path', '/usr/bin/post_script.sh')],
'pre': [('pre_enable', True),
('pre_path', '/usr/bin/pre_script.sh'),
('pre_args', '-la -w --dir')]}
I would say that three lines is short enough for such a problem.
It would be a little cleaner with dict comprehensions:
cfg_pre = {k: v for k, v in dl.items() if 'pre' in k}
cfg_post = {k: v for k, v in dl.items() if 'post' in k}
cfg_fail = {k: v for k, v in dl.items() if 'fail' in k}
You could also remove the redundant data like this:
cfg_pre = {k.replace('pre_', ''): v for k, v in dl.items() if 'pre' in k}
cfg_post = {k.replace('post_', ''): v for k, v in dl.items() if 'post' in k}
cfg_fail = {k.replace('fail_', ''): v for k, v in dl.items() if 'fail' in k}
The dictionaries will be more consistent and the information about pre, post and fail will be stored only in the variable name:
cfg_pre = {'args': '-la -w --dir', 'enable': True, 'path': '/usr/bin/pre_script.sh'}
cfg_post = {'enable': True, 'path': '/usr/bin/post_script.sh'}
cfg_fail = {'args': '--debug 4', 'enable': True, 'path': '/usr/bin/failure_script.sh'}