We Keep Coding

Efficient Way to Build Large Scale Hierarchical Data Tree Path

I have a large dataset (think: big data) of network elements that form a tree-like network.
A toy dataset looks like this:
| id | type | parent_id |
|-----:|:-------|:------------|
| 1 | D | <NA> |
| 2 | C | 1 |
| 3 | C | 2 |
| 4 | C | 3 |
| 5 | B | 3 |
| 6 | B | 4 |
| 7 | A | 4 |
| 8 | A | 5 |
| 9 | A | 3 |
Important rules:
The root nodes (in the toy example of type D) and the leaf nodes (in the toy example of type A) cannot be connected with each other and amongst each other. I.e., a D node cannot be connected with another D node (vice-versa for A nodes) and an A node cannot directly be connected with a D node.
For simplicity reasons, any other node type can randomly be connected in terms of types.
The tree depth can be arbitrarily deep.
The leaf node is always of type A.
A leaf node does not need to be connected through all intermediate nodes. In reality there are only a handful intermediary nodes that are mandatory to pass through. This circumstance can be neglected for this example here.
If you are to recommend doing it in Spark, the solution must be written with pyspark in mind.
What I would like to achieve is to build an efficient way (preferably in Spark) to calculate the tree-path for each node, like so:
| id | type | parent_id | path |
|-----:|:-------|:------------|:--------------------|
| 1 | D | <NA> | D:1 |
| 2 | C | 1 | D:1>C:2 |
| 3 | C | 2 | D:1>C:2>C:3 |
| 4 | C | 3 | D:1>C:2>C:3>C:4 |
| 5 | B | 3 | D:1>C:2>C:3>B:5 |
| 6 | B | 4 | D:1>C:2>C:3>C:4>B:6 |
| 7 | A | 4 | D:1>C:2>C:3>C:4>A:7 |
| 8 | A | 5 | D:1>C:2>C:3>B:5>A:8 |
| 9 | A | 3 | D:1>C:2>C:3>A:9 |
Note:
Each element in the tree path is constructed like this: id:type.
If you have other efficient ways to store the tree path (e.g., closure tables) and calculate them, I am happy to hear them as well. However, the runtime for calculation must be really low (less than an hour, preferably minutes) and retrieval later needs to be in the area of few seconds.
The ultimate end goal is to have a data structure that allows me to aggregate any network node underneath a certain node efficiently (runtime of a few seconds at most).
The actual dataset consisting of around 3M nodes can be constructed like this:
Note:
The commented node_counts that produces the above shown toy examples
The distribution of the node elements is close to reality.
import random
import pandas as pd
random.seed(1337)
node_counts = {'A': 1424383, 'B': 596994, 'C': 234745, 'D': 230937, 'E': 210663, 'F': 122859, 'G': 119453, 'H': 57462, 'I': 23260, 'J': 15008, 'K': 10666, 'L': 6943, 'M': 6724, 'N': 2371, 'O': 2005, 'P': 385}
#node_counts = {'A': 3, 'B': 2, 'C': 3, 'D': 1}
elements = list()
candidates = list()
root_type = list(node_counts.keys())[-1]
leaf_type = list(node_counts.keys())[0]
root_counts = node_counts[root_type]
leaves_count = node_counts[leaf_type]
ids = [i + 1 for i in range(sum(node_counts.values()))]
idcounter = 0
for i, (name, count) in enumerate(sorted(node_counts.items(), reverse=True)):
for _ in range(count):
_id = ids[idcounter]
idcounter += 1
_type = name
if i == 0:
_parent = None
else:
# select a random one that is not a root or a leaf
if len(candidates) == 0: # first bootstrap case
candidate = random.choice(elements)
else:
candidate = random.choice(candidates)
_parent = candidate['id']
_obj = {'id': _id, 'type': _type, 'parent_id': _parent}
#print(_obj)
elements.append(_obj)
if _type != root_type and _type != leaf_type:
candidates.append(_obj)
df = pd.DataFrame.from_dict(elements).astype({'parent_id': 'Int64'})
In order to produce the tree path in pure python with the above toy data you can use the following function:
def get_hierarchy_path(df, cache_dict, ID='id', LABEL = 'type', PARENT_ID = 'parent_id', node_sep='|', elem_sep=':'):
def get_path(record):
if pd.isna(record[PARENT_ID]):
return f'{record[LABEL]}{elem_sep}{record[ID]}'
else:
if record[PARENT_ID] in cache_dict:
parent_path = cache_dict[record[PARENT_ID]]
else:
try:
parent_path = get_path(df.query(f'{ID} == {record[PARENT_ID]}').iloc[0])
except IndexError as e:
print(f'Index Miss for {record[PARENT_ID]} on record {record.to_dict()}')
parent_path = f'{record[LABEL]}{elem_sep}{record[ID]}'
cache_dict[record[PARENT_ID]] = parent_path
return f"{parent_path}{node_sep}{record[LABEL]}{elem_sep}{record[ID]}"
return df.apply(get_path, axis=1)
df['path'] = get_hierarchy_path(df, dict(), node_sep='>')
What I have already tried:
Calculating in pure python with the above function on the large dataset takes me around 5.5 hours. So this is not really a solution. Anything quicker than this is appreciated.
Technically using the spark graphframes package, I could use BFS. This would give me a good solution for individual leave nodes, but it does not scale to the entire network.
I think Pregel is the way to go here. But I do not know how to construct it in Pyspark.
Thank you for your help.

My current solution for this challenge relies now no longer on Spark but on SQL.
I load the whole dataset to a Postgres DB and place a Unique Index on id, type and parent_id.
Then using the following query, I can calculate the path:
with recursive recursive_hierarchy AS (
-- starting point
select
parent_id
, id
, type
, type || ':' || id as path
, 1 as lvl
from hierarchy.nodes
union all
-- recursion
select
ne.parent_id as parent_id
, h.id
, h.type
, ne.type || ':' || ne.id || '|' || h.path as path
, h.lvl + 1 as lvl
from (
select *
from hierarchy.nodes
) ne
inner join recursive_hierarchy h
on ne.id = h.parent_id
), paths as (
-- complete results
select
*
from recursive_hierarchy
), max_lvl as (
-- retrieve the longest path of a network element
select
id
, max(lvl) as max_lvl
from paths
group by id
)
-- all results with only the longest path of a network element
select distinct
, p.id
, p.type
, p.path
from paths p
inner join max_lvl l
on p.id = l.id
and p.lvl = l.max_lvl

How to change just one column in a txt file leaving all else the same and respecting the whitespaces?

I have a file with a lot of rows. To safe space, I am just c/p in a sense that gives the shape of my file.
| Martini system from 2b97.pdb |
| 55601 |
| 1ALA BB 1 13.904 5.512 1.259 |
| 12VAL BB 12 4.199 35.292 21.353 |
| 112VAL SCC 113 4.367 5.234 21.445 |
| 1113CYS BB 1114 4.041 4.969 21.220 |
| 11113CYS SCC11115 4.088 14.816 21.041 |
| 19293DEC C55598 19.018 0.828 7.094 |
| 9.05570 9.05570 30.02670 |
I need to add 0.1 units to the last column.
Therefore, my output file should look exactly like this:
| Martini system from 2b97.pdb |
| 55601 |
| 1ALA BB 1 13.904 5.512 1.359 |
| 12VAL BB 12 4.199 35.292 21.453 |
| 112VAL SCC 113 4.367 5.234 21.545 |
| 1113CYS BB 1114 4.041 4.969 21.320 |
| 11113CYS SCC11115 4.088 14.816 21.141 |
| 19293DEC C55598 19.018 0.828 7.194 |
| 9.05570 9.05570 30.02670 |
The most important thing is that my output file should have exactly the same whitespaces, format and dtype. Everything in this file is a string.
If whitespaces, format and dtype are not respected then I cannot use the output file to run in the program I need.
Just in case, I do not need to keep the initial file (although I think this part is irrelevant for me to add).
Thanks for your help.
I have tried but my problem is that I cannot keep the same shape with python.

Like another answer, I would use string slicing to get just the final column, and string concatenation to put the line back together again. However, I would use decimal.Decimal for the fixed-point math:
import fileinput
import decimal
import sys
files = ['x.txt']
for line in fileinput.input(files, inplace=True):
number = line[38:46]
try:
number = decimal.Decimal(number)
number += decimal.Decimal('.1')
number = '{:8}'.format(number)
line = line[:38] + number + line[46:]
except decimal.InvalidOperation:
pass
sys.stdout.write(line)

Similar to other answers, but here's another option. String reversing done to do the string replace right->left.
with open('in.txt', 'r') as fi, open('out.txt', 'w') as fo:
fo.write(fi.readline())
fo.write(fi.readline())
for line in fi.readlines():
try:
old = line[-10:-3]
new = '{:>7.3f}'.format(float(old) + 0.1)
line = line[::-1].replace(old[::-1], new[::-1], 1)[::-1]
except ValueError as _:
pass
finally:
fo.write(line)

I also used decimal math and string slicing. Here's my version:
from decimal import Decimal, InvalidOperation
def add_zero_point_one(data):
new_data = []
for l in data.split('\n'):
try:
d = Decimal(l[38:46]) + Decimal('0.1')
l = l[:38] + str(d).rjust(8) + l[46:]
except InvalidOperation:
pass
new_data.append(l)
return '\n'.join(new_data)
This worked with the provided sample, but assumes:
That all data out in slice 38:46 is column data you want to increment
That the column widths are fixed
Here's my full working sample:
from decimal import Decimal, InvalidOperation
data = '''| Martini system from 2b97.pdb |
| 55601 |
| 1ALA BB 1 13.904 5.512 1.259 |
| 12VAL BB 12 4.199 35.292 21.353 |
| 112VAL SCC 113 4.367 5.234 21.445 |
| 1113CYS BB 1114 4.041 4.969 21.220 |
| 11113CYS SCC11115 4.088 14.816 21.041 |
| 19293DEC C55598 19.018 0.828 7.094 |
| 9.05570 9.05570 30.02670 |'''
def add_zero_point_one(data):
new_data = []
for l in data.split('\n'):
try:
d = Decimal(l[38:46]) + Decimal('0.1')
l = l[:38] + str(d).rjust(8) + l[46:]
except InvalidOperation:
pass
new_data.append(l)
return '\n'.join(new_data)
print(data)
print(add_zero_point_one(data))

Summarize data in a list in python

In python i need to summarize data in count_list this way (like a histogram):
"""
number | occurence
0 | *
1 | **
2 | ***
3 | **
4 | **
5 | *
6 | *
7 | **
8 | ***
9 | *
10 | **
"""
But instead I get this wrong output:
"""
number | occurence
0 |
1 | **
2 |
3 |
4 |
5 |
6 | **
7 |
8 |
9 |
10 | **
"""
Here is my code:
import random
random_list = []
list_length = 20
while len(random_list) < list_length:
random_list.append(random.randint(0,10))
count_list = [0] * 11
index = 0
while index < len(random_list):
number = random_list[index]
count_list[number] = count_list[number] + 1
index = index + 1
def summerizer():
index = 0
print count_list
print '"'*3
print 'number | occurrence'
while index < len(count_list):
print '%s' %' '*(7),
print index,#the problem is here
print ' | ',#and here
print '%s' %'*'*(count_list[index])
index += 1
print '%s'%'"'*3
summerizer()

This method uses collections.Counter:
from collections import Counter
import random
random_list = []
list_length = 20
while len(random_list) < list_length:
random_list.append(random.randint(0,10))
c = Counter(random_list)
print('number | occurrence')
def summerizer(dic):
for v,d in dic.items():
print(v, '|', '%s'%'*'*c[v])
summerizer(dic)

Yes i have found the problem
It is from the ide itself !!!
This was a quiz in a course on UDACITY android application and the embedded compiler inside it make this wrong answer..
Same code i tried now from pydroid application on Android also made the answer that i need without any change
Thanks for trying to help all of you
`import random
random_list = []
list_length = 20
while len(random_list) < list_length:
random_list.append(random.randint(0,10))
count_list = [0] * 11
index = 0
while index < len(random_list):
number = random_list[index]
count_list[number] = count_list[number] + 1
index = index + 1
def summerizer():
index = 0
print count_list
print '"'*3
print 'number | occurrence'
while index < len(count_list):
print '%s' %' '*(7),
print index,
print ' | ',
print '%s' %'*'*(count_list[index])
index += 1
print '%s'%'"'*3
summerizer()`

Try this
import random
random_list = []
list_length = 20
while len(random_list) < list_length:
random_list.append(random.randint(0,10))
dic={}
for i in random_list:
dic[i]=dic.get(i,0)+1
print 'number | occurrence'
for i in range(0,11):
if(i not in dic):
print i,"|",'%s' %'*'*(0)
else:
print i,"|",'%s' %'*'*(dic[i])
Out put
[9, 8, 4, 2, 5, 4, 8, 3, 5, 6, 9, 5, 3, 8, 6, 2, 10, 10, 8, 9]
number | occurrence
0 |
1 |
2 | **
3 | **
4 | **
5 | ***
6 | **
7 |
8 | ****
9 | ***
10 | **

Python - Huffman bit compression

I need help on this. I made a Huffman compression program. For example I'll input "google", I can just print it like this:
Character | Frequency | Huffman Code
------------------------------------
'g' | 2 | 0
'o' | 2 | 11
'l' | 1 | 101
'e' | 1 | 100
I also need to print the combined bits so if I use that it will be:
011101100
Which is wrong because as far as I understand Huffman compression it should be:
011110101100
So my output should be:
Character | Frequency | Huffman Code
------------------------------------
'g' | 2 | 0
'o' | 2 | 11
'o' | 2 | 11
'g' | 2 | 0
'l' | 1 | 101
'e' | 1 | 100
Basically I need to display it based on what I input. So if I input "test" it should also print test vertically with their corresponding bits since I'm just appending the bits and displaying them. How do I achieve this? Here's the printing part:
freq = {}
for c in word:
if c in freq:
freq[c] += 1
else:
freq[c] = 1
freq = sorted(freq.items(), key=lambda x: x[1], reverse=True)
if check:
print (" Char | Freq ")
for key, c in freq:
print (" %4r | %d" % (key, c))
nodes = freq
while len(nodes) > 1:
key1, c1 = nodes[-1]
key2, c2 = nodes[-2]
nodes = nodes[:-2]
node = NodeTree(key1, key2)
nodes.append((node, c1 + c2))
nodes = sorted(nodes, key=lambda x: x[1], reverse=True)
if check:
print ("left: %s" % nodes[0][0].nodes()[0])
print ("right: %s" % nodes[0][0].nodes()[1])
huffmanCode = huffman(nodes[0][0])
print ("\n")
print (" Character | Frequency | Huffman code ")
print ("---------------------------------------")
for char, frequency in freq:
print (" %-9r | %10d | %12s" % (char, frequency, huffmanCode[char]))
P.S. I know I shouldn't be sorting them I'll remove the sorting part don't worry

How do I put data from a while loop into a table?

Basically I'm estimating pi using polygons. I have a loop which gives me a value for n, ann and bnn before running the loop again. here is what I have so far:
def printPiTable(an,bn,n,k):
"""Prints out a table for values n,2n,...,(2^k)n"""
u = (2**k)*n
power = 0
t = ((2**power)*n)
while t<=u:
if power < 1:
print(t,an,bn)
power = power + 1
t = ((2**power)*n)
else:
afrac = (1/2)*((1/an)+(1/bn))
ann = 1/afrac
bnn = sqrt(ann*bn)
print(t,ann,bnn)
an = ann
bn = bnn
power = power + 1
t = ((2**power)*n)
return
This is what I get if I run it with these values:
>>> printPiTable(4,2*sqrt(2),4,5)
4 4 2.8284271247461903
8 3.3137084989847607 3.0614674589207187
16 3.1825978780745285 3.121445152258053
32 3.1517249074292564 3.1365484905459398
64 3.1441183852459047 3.1403311569547534
128 3.1422236299424577 3.1412772509327733
I want to find a way to make it instead of printing out these values, just print the values in a nice neat table, any help?

Use string formatting. For example,
print('{:<4}{:>20f}{:>20f}'.format(t,ann,bnn))
produces
4 4.000000 2.828427
8 3.313708 3.061467
16 3.182598 3.121445
32 3.151725 3.136548
64 3.144118 3.140331
128 3.142224 3.141277
{:<4} is replaced by t, left-justified, formatted to a string of length 4.
{:>20f} is replaced by ann, right-justified, formatted as a float to a string of length 20.
The full story on the format string syntax is explained here.
To add column headers, just add a print statement like
print('{:<4}{:>20}{:>20}'.format('t','a','b'))
For fancier ascii tables, consider using a package like prettytable:
import prettytable
def printPiTable(an,bn,n,k):
"""Prints out a table for values n,2n,...,(2^k)n"""
table = prettytable.PrettyTable(['t', 'a', 'b'])
u = (2**k)*n
power = 0
t = ((2**power)*n)
while t<=u:
if power < 1:
table.add_row((t,an,bn))
power = power + 1
t = ((2**power)*n)
else:
afrac = (1/2)*((1/an)+(1/bn))
ann = 1/afrac
bnn = sqrt(ann*bn)
table.add_row((t,ann,bnn))
an = ann
bn = bnn
power = power + 1
t = ((2**power)*n)
print(table)
printPiTable(4,2*sqrt(2),4,5)
yields
+-----+---------------+---------------+
| t | a | b |
+-----+---------------+---------------+
| 4 | 4 | 2.82842712475 |
| 8 | 3.31370849898 | 3.06146745892 |
| 16 | 3.18259787807 | 3.12144515226 |
| 32 | 3.15172490743 | 3.13654849055 |
| 64 | 3.14411838525 | 3.14033115695 |
| 128 | 3.14222362994 | 3.14127725093 |
+-----+---------------+---------------+
Perhaps it is overkill for this sole purpose, but Pandas can make nice tables too, and can export them in other formats, such as HTML.

You can use output formatting to make it look pretty. Look here for an example:
http://docs.python.org/release/1.4/tut/node45.html