Related
This is my List
List = ['function = function1', 'string = string1', 'hello = hello1', 'new = new1', 'test = test1']
I need to separate the List into two differnt List's sepearted as 'keys' and 'values'
List = ['function = function1', 'string = string1', 'hello = hello1', 'new = new1', 'test = test1']
KeyList
KeyList = ['function', 'string', 'hello', 'new', 'test']
ValueList
ValueList = ['function1', 'string1', 'hello1', 'new1', 'test1']
There are different possible approach. One is the method proposed by Tim, but if you are not familiar with re you could also do:
List = ['function = function1', 'string = string1', 'hello = hello1', 'new = new1', 'test = test1']
KeyList = []
ValueList = []
for item in List:
val = item.split(' = ')
KeyList.append(val[0])
ValueList.append(val[1])
print(KeyList)
print(ValueList)
and the output is:
['function', 'string', 'hello', 'new', 'test']
['function1', 'string1', 'hello1', 'new1', 'test1']
You can simply use split(" = ") and unzip the list of key-value pairs to two tuples:
keys, values = zip(*map(lambda s: s.split(" = "), List))
# keys
# >>> ('function', 'string', 'hello', 'new', 'test')
# values
# >>>('function1', 'string1', 'hello1', 'new1', 'test1')
This is based on the fact that zip(*a_zipped_iterable) works as an unzipping function.
We can use re.findall here:
inp = ['function = function1', 'string = string1', 'hello = hello1', 'new = new1', 'test = test1']
keys = [re.findall(r'(\w+) =', x)[0] for x in inp]
vals = [re.findall(r'\w+ = (\w+)', x)[0] for x in inp]
keys = [pair[0] for pair in pairs]
values = [pair[1] for pair in pairs]
With xmltodict I managed to get my code from xml in a dict and now I want to create an excel.
In this excel the header of a value is going to be all the parents (keys in the dict).
For example:
dict = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
The value male will have the header: adres/roommate/gender.
Here's a way to orgainze the data in the way your question asks:
d = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
print(d)
stack = [('', d)]
headerByValue = {}
while stack:
name, top = stack.pop()
if isinstance(top, dict):
stack += (((name + '/' if name else '') + k, v) for k, v in top.items())
else:
headerByValue[name] = top
print(headerByValue)
Output:
{'adres/roommate/gender': 'male',
'adres/number': '5',
'adres/street': 'example1street',
'last-name': 'Pencil',
'name': 'Pete'}
I`m trying to parse a spreadsheet with a header that looks something like this:
My problem is those nested keys below "Контрагент". I decided to parse it like this:
['Дата',
'Номер документа',
'Дебет',
'Кредит',
['Контрагент',
['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа',
'Код дебитора',
'Тип документа']
But now, I don`t really have an idea as how to map it to a flat list of values:
['21.05.2021',
'591324565436',
'0.00',
'526345428.99',
'asdasd',
'234525460140679',
'77130100123412341',
'302328105423534200000000280',
'0445252345234974',
'asdfsadfsd',
'sdfghsfgdhfdghdfgh',
'',
'dfghfgdhfdgh']
Given these variables, I want a function to return following dict:
{
"Дата": "21.05.2021",
"Номер документа": "591324565436",
"Дебет": "0.00",
"Кредит": "526345428.99",
"Контрагент": {
"Наименование": "asdasd",
"ИНН": "234525460140679",
"КПП": "77130100123412341",
"Счет": "302328105423534200000000280",
"БИК": "0445252345234974",
"Наименование банка": "asdfsadfsd"
},
"Назначение платежа": "sdfghsfgdhfdghdfgh",
"Код дебитора": "",
"Тип документа": "dfghfgdhfdgh"
}
I've gone this far without realizing it'd be raising IndexError on the 3rd line:
def map_to_schema(schema, data):
for i, elem in enumerate(data):
key = schema[i]
if isinstance(key, list):
if key[0] not in result:
result[key[0]] = {}
result[key[0]] |= {
key[1][i-len(key)]: elem
}
else:
result[key] = elem
What should I do? Maybe the structure for the schema isn't good enough? I really have no idea...
You could use a dictionary comprehension and an iterator:
headers = ['Дата', 'Номер документа', 'Дебет', 'Кредит',
['Контрагент', ['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа', 'Код дебитора', 'Тип документа']
values = ['21.05.2021', '591324565436', '0.00', '526345428.99', 'asdasd', '234525460140679', '77130100123412341',
'302328105423534200000000280', '0445252345234974', 'asdfsadfsd', 'sdfghsfgdhfdghdfgh', '',
'dfghfgdhfdgh']
it = iter(values)
out = {k[0] if (islist := isinstance(k, list)) else k:
{k2: next(it) for k2 in k[1]} if islist else next(it)
for k in headers}
output:
{'Дата': '21.05.2021',
'Номер документа': '591324565436',
'Дебет': '0.00',
'Кредит': '526345428.99',
'Контрагент': {'Наименование': 'asdasd',
'ИНН': '234525460140679',
'КПП': '77130100123412341',
'Счет': '302328105423534200000000280',
'БИК': '0445252345234974',
'Наименование банка': 'asdfsadfsd'},
'Назначение платежа': 'sdfghsfgdhfdghdfgh',
'Код дебитора': '',
'Тип документа': 'dfghfgdhfdgh'}
Thanks #mozway for this solution! This is essentially the same algorithm, using a for loop.
def map(schema, s_length, row: list):
# If len(row) was less then *true* schema length, it would have thrown StopIteration.
# I ended up just extending row list by delta elements.
if (delta := s_length - len(row)) > 0:
row.extend([""] * delta)
iter_row = iter(row)
result = {}
for key in schema:
if isinstance(key, list):
result[key[0]] = {}
for sub_key in key[1]:
result[key[0]][sub_key] = next(iter_row)
else:
result[key] = next(iter_row)
return result
I'm trying to parse the item names and it's corresponding values from the below snippet. dt tag holds names and dd containing values. There are few dt tags which do not have corresponding values. So, all the names do not have values. What I wish to do is keep the values blank against any name if the latter doesn't have any values.
These are the elements I would like to scrape data from:
content="""
<div class="movie_middle">
<dl>
<dt>Genres:</dt>
<dt>Resolution:</dt>
<dd>1920*1080</dd>
<dt>Size:</dt>
<dd>1.60G</dd>
<dt>Quality:</dt>
<dd>1080p</dd>
<dt>Frame Rate:</dt>
<dd>23.976 fps</dd>
<dt>Language:</dt>
</dl>
</div>
"""
I've tried like below:
soup = BeautifulSoup(content,"lxml")
title = [item.text for item in soup.select(".movie_middle dt")]
result = [item.text for item in soup.select(".movie_middle dd")]
vault = dict(zip(title,result))
print(vault)
It gives me messy results (wrong pairs):
{'Genres:': '1920*1080', 'Resolution:': '1.60G', 'Size:': '1080p', 'Quality:': '23.976 fps'}
My expected result:
{'Genres:': '', 'Resolution:': '1920*1080', 'Size:': '1.60G', 'Quality:': '1080p','Frame Rate:':'23.976 fps','Language:':''}
Any help on fixing the issue will be highly appreciated.
You can loop through the elements inside dl. If the current element is dt and the next element is dd, then store the value as the next element, else set the value as empty string.
dl = soup.select('.movie_middle dl')[0]
elems = dl.find_all() # Returns the list of dt and dd
data = {}
for i, el in enumerate(elems):
if el.name == 'dt':
key = el.text.replace(':', '')
# check if the next element is a `dd`
if i < len(elems) - 1 and elems[i+1].name == 'dd':
data[key] = elems[i+1].text
else:
data[key] = ''
You can use BeautifulSoup to parse the dl structure, and then write a function to create the dictionary:
from bs4 import BeautifulSoup as soup
import re
def parse_result(d):
while d:
a, *_d = d
if _d:
if re.findall('\<dt', a) and re.findall('\<dd', _d[0]):
yield [a[4:-5], _d[0][4:-5]]
d = _d[1:]
else:
yield [a[4:-5], '']
d = _d
else:
yield [a[4:-5], '']
d = []
print(dict(parse_result(list(filter(None, str(soup(content, 'html.parser').find('dl')).split('\n')))[1:-1])))
Output:
{'Genres:': '', 'Resolution:': '1920*1080', 'Size:': '1.60G', 'Quality:': '1080p', 'Frame Rate:': '23.976 fps', 'Language:': ''}
For a slightly longer, although cleaner solution, you can create a decorator to strip the HTML tags of the output, thus removing the need for the extra string slicing in the main parse_result function:
def strip_tags(f):
def wrapper(data):
return {a[4:-5]:b[4:-5] for a, b in f(data)}
return wrapper
#strip_tags
def parse_result(d):
while d:
a, *_d = d
if _d:
if re.findall('\<dt', a) and re.findall('\<dd', _d[0]):
yield [a, _d[0]]
d = _d[1:]
else:
yield [a, '']
d = _d
else:
yield [a, '']
d = []
print(parse_result(list(filter(None, str(soup(content, 'html.parser').find('dl')).split('\n')))[1:-1]))
Output:
{'Genres:': '', 'Resolution:': '1920*1080', 'Size:': '1.60G', 'Quality:': '1080p', 'Frame Rate:': '23.976 fps', 'Language:': ''}
from collections import defaultdict
test = soup.text.split('\n')
d = defaultdict(list)
for i in range(len(test)):
if (':' in test[i]) and (':' not in test[i+1]):
d[test[i]] = test[i+1]
elif ':' in test[i]:
d[test[i]] = ''
d
defaultdict(list,
{'Frame Rate:': '23.976 fps',
'Genres:': '',
'Language:': '',
'Quality:': '1080p',
'Resolution:': '1920*1080',
'Size:': '1.60G'})
The logic here is that you know that every key will have a colon. Knowing this, you can write an if else statement to capture the unique combinations, whether that is key followed by key or key followed by value
Edit:
In case you wanted to clean your keys, below replaces the : in each one:
d1 = { x.replace(':', ''): d[x] for x in d.keys() }
d1
{'Frame Rate': '23.976 fps',
'Genres': '',
'Language': '',
'Quality': '1080p',
'Resolution': '1920*1080',
'Size': '1.60G'}
The problem is that empty elements are not present. Since there is no hierarchy between the <dt> and the <dd>, I'm afraid you'll have to craft the dictionary yourself.
vault = {}
category = ""
for item in soup.find("dl").findChildren():
if item.name == "dt":
if category == "":
category = item.text
else:
vault[category] = ""
category = ""
elif item.name == "dd":
vault[category] = item.text
category = ""
Basically this code iterates over the child elements of the <dl> and fills the vault dictionary with the values.
I have 2 dictionaries. Is it possible to loop DictA's key with DictB's values, and if DictB's value has a match, it would return DictB's key.
Is it possible to be done?
Please give me some advice/tips. What am I doing wrong? Thank you.
Example:
A: {'WNT3A': '1732', 'RG59L': ' ', 'SCZD9': ' ', 'CD241': '6005', 'C2': '', 'RH': ' '}
B: {'': [''], '6005': ['RH50A', ' CD241', ' SLC42A1'], '603': [''], '6000': [''], '8787': ['PERRS', ' RGS9L', ' MGC26458'], '41': ['ACCN2', ' BNaC2', ' hBNaC2'], '8490': [''], '9628': [''], '5999': ['SCZD9']}
Result :
new_A: {'WNT3A': '1732', 'RG59L': ' ', 'SCZD9': '5999 ', 'CD241': '6005', 'C2': '', 'RH': ' '}
So far, I've coded this but it only seems to return me a dictionary that is whole or a dictionary containing only dictionary B.
new_a = {}
for key in ref_dict:
for value in (map(lambda x: x.strip(), ref_dict[key][0])):
if(not fill_dict.has_key(key)):
continue
elif(ref_dict[value] != fill_dict[key]):
new_a[value] = (ref_dict[key], fill_dict[key])
print new_a
Result:
{'WNT3A': '1732', 'RG59L': '', 'SCZD9': '', 'CD241': '6005', 'C2': '', 'RH': ''}
Another code I've tried is :
new_dict = {}
for k, v in fill_dict.iteritems():
vals = []
if isinstance(v, list):
for i in v:
vals.append(ref_dict.get(i))
else:
vals.append(ref_dict.get(v))
if not vals:
continue
new_dict[k] = vals
print new_dict
Result:
{'WNT3A': [None], 'RG59L': [['']], 'SCZD9': [['']], 'CD241': [['RH50A', ' CD241', ' SLC42A1']], 'C2': [['']], 'RH': [['']]}
You can use a dict-comprehension with next() for this:
>>> {k: next((k1 for k1, v1 in B.items() if k in v1), v) for k, v in A.items()}
{'WNT3A': '1732', 'RG59L': ' ', 'SCZD9': '5999', 'CD241': '6005', 'C2': '', 'RH': ' '}
Here I've used next(<gen exp>, v), if the generator expression returned an empty iterator(i.e if none of the B's values contains the key k) then simply use the value v from A, otherwise use the key returned from the generator expression.
The above code can also be written as:
d = {}
for k, v in A.items():
for k1, v1 in B.items():
if k in v1:
d[k] = k1
break
else:
#this will be executed if no value in `B` contains `k`
d[k] = v
This will update dictionary a with the values you want:
for key_a in a:
for key_b, value_b in b.iteritems():
for item in value_b:
if item == key_a:
a[key_a] = key_b
If you need a whole new dict, then just make a copy of a and update that one.