We Keep Coding

Find connected components recursively in a data frame

Consider the following data frame:
import numpy as np
import pandas as pd
df = pd.DataFrame(
{
"main": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
"component": [
[1, 2],
[np.nan],
[3, 8],
[np.nan],
[1, 5, 6],
[np.nan],
[7],
[np.nan],
[9, 10],
[np.nan],
[np.nan],
],
}
)
The column main represents a certain approach. Each approach consists of components. A component itself could also be an approach and is then called sub-approach.
I want to find all connected sub-approaches/components for a certain approach.
Suppose, for instance, I want to find all connected sub-approaches/components for the main approach '0'.
Then, my desired output would look like this:
target = pd.DataFrame({
"main": [0, 0, 2, 2, 8, 8],
"component": [1, 2, 3, 8, 9, 10]
})
Ideally, I want to be able to just choose the approach and then get all sub-connections.
I am convinced that there is a smart approach to do so using networkx. Any hint is appreciated.
Ultimately, I want to create a graph that looks somewhat like this (for approach 0):
Additional information:
You can explode the data frame and then remove all components from the main column (components are approaches that do not have any component).
df_exploded = df.explode(column="component").dropna(subset="component")
The graph can be constructed as follows:
import networkx as nx
import graphviz
G = nx.Graph()
G.add_edges_from([(i, j) for i, j in target.values])
graph_attr = dict(rankdir="LR", nodesep="0.2")
g = graphviz.Digraph(graph_attr=graph_attr)
for k, v in G.nodes.items():
g.node(str(k), shape="box", style="filled", height="0.35")
for n1, n2 in G.edges:
g.edge(str(n2), str(n1))
g

You can use nx.dfs_edges
edges = df.explode(column='component').dropna(subset='component')
G = nx.from_pandas_edgelist(edges, source='main', target='component', create_using=nx.DiGraph)
target = pd.DataFrame(nx.dfs_edges(G, 0), columns=['main', 'component'])
Output:
>>> target
main component
0 0 1
1 0 2
2 2 3
3 2 8
4 8 9
5 8 10
To extract the subgraph, use:
H = G.edge_subgraph(nx.dfs_edges(G, 0))

Dictionary values and list values within a function

I have a dictionary with product names and prices:
products = {'a': 2, 'b': 3, 'c': 4, 'd': 5, 'e': 6, 'f': 7, 'g': 8}
And a list with amounts of each product:
amounts = [3, 0, 5, 1, 3, 2, 0]
I want to get an output shown there total price of that order.
Not using functions I seem to get it right:
products = {'a': 2, 'b': 3, 'c': 4, 'd': 5, 'e': 6, 'f': 7, 'g': 8}
amounts = [3, 0, 5, 1, 3, 2, 0]
res_list = []
order = []
for value in products.values():
res_list.append(value)
for i in range(0, len(res_list)):
order.append(amounts[i] * res_list[i])
total = sum(order)
print(res_list)
print(order) #this line and the one above are not really necessary
print(total)
Output : 63
But when I try using this code within a function I am having some problems. this is what I have tried:
products = {'a': 2, 'b': 3, 'c': 4, 'd': 5, 'e': 6, 'f': 7, 'g': 8}
amounts = [3, 0, 5, 1, 3, 2, 0]
#order = []
def order(prod):
res_list = []
for value in prod.values():
res_list.append(value)
return res_list
prices = order(products)
print(prices)
def order1(prices):
order =[]
for i in range(0, len(prices)):
order.append(amounts[i] * prices[i])
total = sum(order)
return total
print(order1(prices))
Not working the way it is intended.
Thanks for all the help I am learning.

The immediate problem is that your lines:
total = sum(order)
return total
are indented too much, so that they are inside the for loop. Outside of a function, the bug does not matter too much, because all that happens is that the total is recalculated on every iteration but the final value is the one that is used. But inside the function, what will happen is that it will return on the first iteration.
Reducing the indentation so that it is outside the for loop will fix this.
def order1(prices):
order =[]
for i in range(0, len(prices)):
order.append(amounts[i] * prices[i])
total = sum(order)
return total
However, separate from that, you are relying on the order within the dictionary, which is only guaranteed for Python 3.7 and more recent. If you want to allow the code to be run reliably on earlier versions of Python, you can use an OrderedDict.
from collections import OrderedDict
products = OrderedDict([('a', 2), ('b', 3), ('c', 4), ('d', 5),
('e', 6), ('f', 7), ('g', 8)])
Incidentally, your order function is unnecessary. If you want to convert products.values() (a dictionary values iterator) to a list, just use:
prices = list(products.values())
Also, in order1 it is unnecessary to build up an order list and sum it - you could use:
total = 0
for i in range(0, len(prices)):
total += amounts[i] * prices[i]
That is probably enough to be getting on with for now, but if you wish to make a further refinement, then look up about how zip is used, and think how it could be used with your loop over amounts and prices.

Just zip products.values() and amounts, find the product of each pair, and then finally sum the result
>>> products = {'a': 2, 'b': 3, 'c': 4, 'd': 5, 'e': 6, 'f': 7, 'g': 8}
>>> amounts = [3, 0, 5, 1, 3, 2, 0]
>>>
>>> sum(i*j for i,j in zip(products.values(), amounts))
63

You can do this.
products = {'a': 2, 'b': 3, 'c': 4, 'd': 5, 'e': 6, 'f': 7, 'g': 8}
amounts = [3, 0, 5, 1, 3, 2, 0]
def order(products, amounts):
res_list = []
order = []
for value in products.values():
res_list.append(value)
for i in range(0, len(res_list)):
order.append(amounts[i] * res_list[i])
total = sum(order)
print(res_list)
print(order) #this line and the one above are not really necessary
print(total)
return(total)
order(products, amounts)

You don't really need to iterate twice assuming that the amount of items in products and in amounts is the same.
products = {'a': 2, 'b': 3, 'c': 4, 'd': 5, 'e': 6, 'f': 7, 'g': 8}
amounts = [3, 0, 5, 1, 3, 2, 0]
def order(products: dict, amounts: list):
total = 0
for idx, (_key, val) in enumerate(products.items()):
total = total + amounts[idx] * val
return total
print(order(products, amounts))
Note: The order of the items in the dictionary is not guaranteed, you might want to look into different data structures that link together products and amounts in a better way, i.e.:
products = {'a': 2, 'b': 3, 'c': 4, 'd': 5, 'e': 6, 'f': 7, 'g': 8}
amounts = {'a': 3, 'b': 0, 'c': 5, 'd': 1, 'e': 3, 'f': 2, 'g': 0}
In this way you could do this:
def order(products: dict, amounts: dict):
total = 0
for key, val in products.items():
total = total + val * amounts[key]
return total
print(order(products, amounts))

Once we're at it, let's get fancy with numpy, since in the end, you just want the dot product prices x amounts:
import numpy as np
total = np.dot(list(products.values()), amounts)
63
But seriously, I'd strictly use either lists or dicts for both datasets, not mix them up, since that can seriously cause problems with order syncronisation between them, even if you are on Python 3.7 with the changes made there as mentioned.

Regarding dictionary value manipulation in python

CONTEXT:
The code is to be used for representing graphs for use in implementations of graph search algorithms (like Breadth-First Search).
I want to store the graph in form of a dictionary, where keys represent the nodes and each key has three corresponding values. First is a set of nodes with which the "key" shares an edge. Second is a Boolean flag for showing visited/not visited. Third is distance of the "key" from starting node.
"""
The 'test.txt' file contains the following:
1 2 3 4 5
2 1 3 4 5
3 1 2 5
4 1 2
5 1 2 3
"""
import math as m
def readGraph(path):
a = {}
file = open(path)
data = file.readlines()
for line in data:
items = line.split()
items = [int(i) for i in items]
a[items[0]] = items[1:len(items) + 1], 0, m.inf
return a
if __name__ == '__main__':
G = readGraph('test.txt')
print(G)
The dictionary (stored in 'G') for the given file is:
G = {1: ([2, 3, 4, 5], 0, inf), 2: ([1, 3, 4, 5], 0, inf), 3: ([1, 2, 5], 0, inf), 4: ([1, 2], 0, inf), 5: ([1, 2, 3], 0, inf)}
DOUBT:
Suppose now I want to change the second value of key 1, from 0 to 1.
Typing G[1] = G[1][0], 1, G[1][2] does not seem efficient.
Is there a better approach?
UPDATE:
I tried saving the dictionary entries as lists, but that is undesirable as it would change the format of dictionary, which I want to implement.
The following is a solution, but still I want to use the dictionary in its default form, with the elements of each key stored as tuple.
if __name__ == '__main__':
G = readGraph('test.txt')
print(G)
G[1] = list(G[1])
G[1][1] = 1
print(G)

One way of doing this, you can use nested dictionary to store the node. Here is how graph G will look like.
G = {
1: {
'nodes': [2, 3, 4, 5],
'is_visited': 0,
'distance': 'inf'
},
2: {
'nodes': [1, 3, 4, 5],
'is_visited': 0,
'distance': 'inf'
}
}
and then you can get values by indexing.
G[1]['is_visited'] = 1

You can store the values for each node as a list instead of a tuple:
a[items[0]] = [items[1:len(items) + 1], 0, m.inf]
and then just update the value you want directly:
G[1][1] = 1
Another option (thanks to Bilal for his suggestion of this approach) is nested dictionaries:
a[items[0]] = {"edges": items[1:len(items) + 1], "is_visited": 0, "dist": m.inf}
Then you could access the individual elements as:
G[1]["is_visited"] = 1

How to return directly dependent nodes on a graph

I want to obtain directly dependent nodes of the given node, if possible.
For example, on the following example nx.ancestors(G, 5) returns {0, 1, 2, 3, 4}, these nodes are iteratively dependent on node 5. But I want to obtain {3, 4}, where these nodes are directly connected to node 5.
Also, nx.descendants(G, 0) returns {1, 2, 3, 4, 5}, where I want to obtain {1, 2} that are directly connected to node 0.
import networkx as nx
import matplotlib.pyplot as plt
g = nx.Graph()
G = nx.DiGraph()
# add 5 nodes, labeled 0-4:
map(G.add_node, range(5))
# 1,2 depend on 0:
G.add_edge(0,1)
G.add_edge(0,2)
# 3 depends on 1,2
G.add_edge(1,3)
G.add_edge(2,3)
# 4 depends on 1
G.add_edge(1,4)
# 5 depends on 3 and 4
G.add_edge(3,5)
G.add_edge(4,5)
print(nx.ancestors(G, 5))
print(nx.descendants(G, 0))
Output:
{0, 1, 2, 3, 4}
{1, 2, 3, 4, 5}

You can use predecessors and successors:
set(G.predecessors(5))
Output:
{3, 4}
And,
set(G.successors(0))
Output:
{1, 2}

Create a hierarchy from a dictionary of lists

I have a dictionary of lists:
a = {
'a': [1, 2, 3],
'b': [1, 2, 4],
'c': [1, 2],
'd': [1, 2, 3, 4, 5],
'e': [3],
'f': [3, 7],
'g': [3, 3],
'h': [3, 3, 3, 3, 3],
'i': [3, 3, 3, 3, 4],
}
And I would like to create hierarchical structure from this dictionary which will group items in the similar manner (exact structure does not matter, as well as the relation between elements is preserved):
/ \
/ \
e c
/\ /\
f g a b
/\ |
h i d
The hierarchy goes as follows: array g is a prefix of array h and i and therefore it is their ancestor. But e is a prefix of g, so it e is an ancestor of g.
Here is my idea how to achieve this result.
Sort the dictionary based on the number of elements in the list, which I was able to achieve with s = sorted(a.items(), key=lambda e: len(e[1])). This will give me the following structure:
.
('e', [3])
('c', [1, 2])
('g', [3, 3])
('f', [3, 7])
('a', [1, 2, 3])
('b', [1, 2, 4])
('d', [1, 2, 3, 4, 5])
('h', [3, 3, 3, 3, 3])
Right now I can find first parents by iterating through elements and checking if an element is a prefix of other elements. Starting with the first one. e is a prefix of g, f, and h. And c is a prefix of a, b, d. So these two elements are the parents.
right now I understand that I have to use recursion to enter inside of each parent and to perform the same operation, but I was not able to come up with a right solution.
So does anyone knows how to approach this problem. Or am I over-complicating things and there is an easier way to achieve the solution.
P.S. this is not a homework assignment or interview question (also it might be). This is just my abstraction from a problem I am trying to solve.

Other people already give the methord, I just write some code here:
First sort:
t = sorted(a.items(), key=lambda x: x[1])
The build the structure
ret = {}
def build(ret, upv):
if not t:
return (None, None)
k, v = t.pop(0)
while k and v:
if upv and v[:len(upv)] != upv:
return (k, v)
r = {}
ret[k] = r
k, v = build(r, v)
return None, None
build(ret, None)
print ret

given an object that has a list of children, and an is_prefix function, and your sorted list of objects, I don't see why this wouldn't work
for indx, potential_prefix in enumerate(your_list):
for potential_child in your_list[indx:]:
if is_prefix(potential_prefix, potential_child):
potential_prefix.add_child(potential_child)
# and optionally
potential_child.add_parent(potential_prefix)

How about building the tree with a set of nested dictionaries, so that you'd access the e node by tree[3] and the h node by tree[3][3][3][3][3]:
from collections import nested
def nested():
return defaultdict(nested)
def build_tree(data):
tree = nested()
for name, path in data.items():
d = tree
for p in path:
d = d[p]
d["value"] = name
return tree
Example output:
>>> a = {
'a': [1, 2, 3],
'b': [1, 2, 4],
'c': [1, 2],
'd': [1, 2, 3, 4, 5],
'e': [3],
'f': [3, 7],
'g': [3, 3],
'h': [3, 3, 3, 3, 3],
'i': [3, 3, 3, 3, 4],
}
>>> import json # for pretty printing, note that in python the keys are ints, not str
>>> print(json.dumps(build_tree(a), indent=4))
{
"1": {
"2": {
"3": {
"4": {
"5": {
"value": "d"
}
},
"value": "a"
},
"4": {
"value": "b"
},
"value": "c"
}
},
"3": {
"7": {
"value": "f"
},
"3": {
"3": {
"3": {
"3": {
"value": "h"
},
"4": {
"value": "i"
}
}
},
"value": "g"
},
"value": "e"
}
}

Just sort arrays in lexicographical order:
(c,[1,2]),
(a,[1,2,3]),
(d,[1,2,3,4,5]),
(b,[1,2,4]),
(e,[3]),
(g,[3,3]),
(h,[3,3,3,3,3]),
(i,[3,3,3,3,4]),
(f,[3,7])
Then solution is pretty obvious.
root
Lc
|La
||Ld
|Lb
Le
Lg
|Lh
|Li
Lf
You need only track path form parent by prefix. From previous line. You will form somethink like stack. root has empty set so push it on stack. c has (empty) prefix as root so root is parent of c. Push c on stack. a has prefix which is c on top of stack so c is parent of a. push a on stack. d has prefix same as a on top of stack so a is parent of d and push on stack. b doesn't have prefix d on top of stack so pop. Same for a then pop. Now there is c which is prefix so b has parent c. Push b on stack. And continue in same way.
In Erlang simply:
-module(tree_from_prefix).
-export([tree/1]).
is_prefix(_, []) -> true;
is_prefix([H|A], [H|B]) -> is_prefix(A, B);
is_prefix(_, _) -> false.
tree(L) ->
tree(lists:keysort(2, L), [{root, []}]).
tree([], _) -> [];
tree([{X, L} = Record|T] = List, [{Parent, Prefix}|R] = Stack) ->
case is_prefix(L, Prefix) of
true -> [{Parent, X}|tree(T, [Record|Stack])];
false -> tree(List, R)
end.
And result
1> tree_from_prefix:tree([{e,[3]},{c,[1, 2]},{g,[3, 3]},{f,[3, 7]},{a,[1, 2, 3]},{b, [1, 2, 4]},{d,[1, 2, 3, 4, 5]},{h,[3, 3, 3, 3, 3]},{i,[3, 3, 3, 3, 4]}]).
[{root,c},
{c,a},
{a,d},
{c,b},
{root,e},
{e,g},
{g,h},
{g,i},
{e,f}]
In python it will not be so elegant but same algorithm will work too.