We Keep Coding

loop/recursion to handle hierarchical structures in Python

I was wondering if there's a more efficient way of looping through a set of objects that might have some dependencies (child-parent relationship) between them, e.g.:
dependencies = [(child1,parent1),(child2,parent2),...]
obj_stack = [{'id':'child1'},{'id':'child2'},{'id':'parent1'},..]
new_stack = [{'id':'child1'},{'id':'child3'}]
check_and_add_to_new_stack(obj) # function that decides if an obj should be added to the new stack
I want to check each obj_stack and add objs into new_stack under two conditions:
it has no parent and check_and_add_to_new_stack(obj) adds it (let's assume it does it correctly)
it has a parent, then add it directly (no need to use the above function)
This means that I need to check if an element has a parent, and then check&add their parent first. If it gets added, then I come back to that element. I am kinda getting stuck on the recursive loop.
Here is the pseudo-code:
def check_and_add_to_new_stack(obj,stack):
if passed_checks(obj):
return add_to_new_stack(obj,stack)
return stack
def myFunction(obj_stack, new_stack, dependencies):
for obj in obj_stack:
if obj is not in new_stack:
if obj has parent in dependencies:
myFunction([parent], new_stack, dependencies)
else: # here the original obj should be thrown back into the function
new_stack += check_and_add_to_new_stack(obj)
return stack
Edit: adding the result that I am expecting and more details:
Let's assume that
passed_checks(parent1) = False
passed_checks(parent2) = True
passed_checks(child1) = True
passed_checks(child2) = False
The expected result is:
myFunction(obj_stack, new_stack, dependencies)
> [{'id':'child1'},{'id':'child3'},{'id':'child2'},{'id':'parent2'}]
Even though passed_checks(child2) = False it has a parent2 for which passed_checks = True, so both get added to the resulting set. child1 was already in new_stack. parent1 did not get added because passed_checks = False.

I think you might be looking for something like this.
walk_ancestry yields node IDs for a given node and all of its parents based on the parents dict (which is basically dict(dependencies) from your original code)
check is your check function – I just copied your condition there
The all_passing set comprehension iterates over all of our known object names (obj_stack in your original code) and uses the built-in any function to see if any of the nodes in that node's ancestry pass the check() test. If so, it's considered passing.
This could be made faster by caching and memoization, but for small enough graphs, I'm pretty sure this works out alright.
def walk_ancestry(parents, node):
while True:
yield node
node = parents.get(node)
if not node:
break
def check(node):
return node in {"parent2", "child1"}
parents = {
"child1": "parent1",
"child2": "parent2",
"parent2": "parent3",
}
all_objects = {
"parent1",
"parent2",
"parent3",
"child1",
"child2",
}
all_passing = {
node
for node in all_objects
if any(check(n) for n in walk_ancestry(parents, node))
}
print(all_passing)
The output is
['parent2', 'child2', 'child1']

Implementing a depth-first tree iterator in Python

I'm trying to implement an iterator class for not-necessarily-binary trees in Python. After the iterator is constructed with a tree's root node, its next() function can be called repeatedly to traverse the tree in depth-first order (e.g., this order), finally returning None when there are no nodes left.
Here is the basic Node class for a tree:
class Node(object):
def __init__(self, title, children=None):
self.title = title
self.children = children or []
self.visited = False
def __str__(self):
return self.title
As you can see above, I introduced a visited property to the nodes for my first approach, since I didn't see a way around it. With that extra measure of state, the Iterator class looks like this:
class Iterator(object):
def __init__(self, root):
self.stack = []
self.current = root
def next(self):
if self.current is None:
return None
self.stack.append(self.current)
self.current.visited = True
# Root case
if len(self.stack) == 1:
return self.current
while self.stack:
self.current = self.stack[-1]
for child in self.current.children:
if not child.visited:
self.current = child
return child
self.stack.pop()
This is all well and good, but I want to get rid of the need for the visited property, without resorting to recursion or any other alterations to the Node class.
All the state I need should be taken care of in the iterator, but I'm at a loss about how that can be done. Keeping a visited list for the whole tree is non-scalable and out of the question, so there must be a clever way to use the stack.
What especially confuses me is this--since the next() function, of course, returns, how can I remember where I've been without marking anything or using excess storage? Intuitively, I think of looping over children, but that logic is broken/forgotten when the next() function returns!
UPDATE - Here is a small test:
tree = Node(
'A', [
Node('B', [
Node('C', [
Node('D')
]),
Node('E'),
]),
Node('F'),
Node('G'),
])
iter = Iterator(tree)
out = object()
while out:
out = iter.next()
print out

If you really must avoid recursion, this iterator works:
from collections import deque
def node_depth_first_iter(node):
stack = deque([node])
while stack:
# Pop out the first element in the stack
node = stack.popleft()
yield node
# push children onto the front of the stack.
# Note that with a deque.extendleft, the first on in is the last
# one out, so we need to push them in reverse order.
stack.extendleft(reversed(node.children))
With that said, I think that you're thinking about this too hard. A good-ole' (recursive) generator also does the trick:
class Node(object):
def __init__(self, title, children=None):
self.title = title
self.children = children or []
def __str__(self):
return self.title
def __iter__(self):
yield self
for child in self.children:
for node in child:
yield node
both of these pass your tests:
expected = ['A', 'B', 'C', 'D', 'E', 'F', 'G']
# Test recursive generator using Node.__iter__
assert [str(n) for n in tree] == expected
# test non-recursive Iterator
assert [str(n) for n in node_depth_first_iter(tree)] == expected
and you can easily make Node.__iter__ use the non-recursive form if you prefer:
def __iter__(self):
return node_depth_first_iter(self)

That could still potentially hold every label, though. I want the
iterator to keep only a subset of the tree at a time.
But you already are holding everything. Remember that an object is essentially a dictionary with an entry for each attribute. Having self.visited = False in the __init__ of Node means you are storing a redundant "visited" key and False value for every single Node object no matter what. A set, at least, also has the potential of not holding every single node ID. Try this:
class Iterator(object):
def __init__(self, root):
self.visited_ids = set()
...
def next(self):
...
#self.current.visited = True
self.visited_ids.add(id(self.current))
...
#if not child.visited:
if id(child) not in self.visited_ids:
Looking up the ID in the set should be just as fast as accessing a node's attribute. The only way this can be more wasteful than your solution is the overhead of the set object itself (not its elements), which is only a concern if you have multiple concurrent iterators (which you obviously don't, otherwise the node visited attribute couldn't be useful to you).

check the presence of a child node recursively in a tree

I have created a tree object in python using the following code:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import re,sys,codecs
neg_markers_en=[u'not',u"napt",u'no',u'nobody',u'none',u'never']
class Node:
def __init__(self,name=None,parent=None,sentence_number=0):
self.name=name
self.next=list()
self.parent=parent
self.depth=0
self.n_of_neg=0
self.subordinate=None
self.foo=None
def print_node(self):
print self.name,'contains',[(x.name,x.depth,x.foo) for x in self.next]
for x in self.next:
x.print_node()
def get_negation(self):
for x in self.next:
if x.n_of_neg!=0:
print unicode(x.depth)+u' |||',
try:
x.look_for_parent_vp()
except: print 'not in a VP',
try:
x.look_for_parent_sent()
except: print '***'
x.get_negation()
def look_for_parent_vp(self):
if self.parent.name=='VP':
self.parent.print_nont()
else:
self.parent.look_for_parent_vp()
def look_for_parent_sent(self):
if self.parent.name=='S' or self.parent.name=='SBAR':
#This is to send out to a text file, along with what it covers
print '||| '+ self.parent.name,
try:
self.parent.check_subordinate()
self.parent.print_nont()
print '\n'
except:
print u'no sub |||',
self.parent.print_nont()
print '\n'
elif self.parent=='None': print 'root |||'
else:
self.parent.look_for_parent_sent()
def print_nont(self):
for x in self.next:
if x.next==[]:
print unicode(x.name),
else: x.print_nont()
def mark_subordinate(self):
for x in self.next:
if x.name=='SBAR':
x.subordinate='sub'
else: x.subordinate='main'
x.mark_subordinate()
def check_subordinate(self):
if self.subordinate=='sub':
print u'sub |||',
else:
self.parent.check_subordinate()
def create_tree(tree):
#replace "n't" with 'napt' so to avoid errors in splitting
tree=tree.replace("n't",'napt')
lista=filter(lambda x: x!=' ',re.findall(r"\w+|\W",tree))
start_node=Node(name='*NULL*')
current_node=start_node
for i in range(len(lista)-1):
if lista[i]=='(':
next_node=Node()
next_node.parent=current_node
next_node.depth=current_node.depth+1
current_node.next.append(next_node)
current_node=next_node
elif lista[i]==')':
current_node=current_node.parent
else:
if lista[i-1]=='(' or lista[i-1]==')':
current_node.name=lista[i]
else:
next_node=Node()
next_node.name=lista[i]
next_node.parent=current_node
#marks the depth of the node
next_node.depth=current_node.depth+1
if lista[i] in neg_markers_en:
current_node.n_of_neg+=1
current_node.next.append(next_node)
return start_node
Now all the nodes are linked so that the children nodes of a parent node are appended to a list and each one of these child nodes are referred back to their parent through the instance parent.
I have the following problem:
For each node whose name is 'S' or 'SBAR' (let's call it node_to_check), I have to look if any of its children node's name is either 'S' or 'SBAR'; if this is NOT the case I want to transform .foo attribute of the node_to_check into 'atom'.
I was thinking of something like this:
def find_node_to_check(self):
for next in self.next:
if next.name == 'S' or next.name == 'SBAR':
is_present = check_children(next)
if is_present == 'no':
find_node_to_check(next)
else:
self.foo = 'atom'
def check_children(self):
for next in self.next:
# is this way of returning correct?
if next.name == 'S' or next.name == 'SBAR':
return 'no'
else:
check_sents(next)
return 'yes'
I included in my question also the code that I have written so far. A tree structure is created in the function create_tree(tree); the input tree is a bracketed notation from the Stanford Parser.

When trying to design a novel class, knowing what you need it to do informs how you construct it. Stubbing works well here, for example:
class Node:
"""A vertex of an n-adic tree"""
def __init__(self, name):
"""since you used sentence, I assumed n-adic
but that may be wrong and then you might want
left and right children instead of a list or dictionary
of children"""
pass
def append_children(self, children):
"""adds a sequence of child Nodes to self"""
pass
def create_child(self, name):
"""creates a new Named node and adds it as a child"""
pass
def delete_child(self, name):
"""deletes a named child from self or throws exception"""
pass
And so on. Do children need to be ordered? Do you ever need to delete a node (and descendants)? Would you be able to pre-build a list of children or would you have to do it one at a time. Do you really want to store the fact that a Node is terminal (that's redundant) or do you want is_terminal() to return children is None?

How to add to a loop I'm iterating through in Python

I'm searching through a list of scripts, and in each script, I'm parsing it out and among other things, finding the subscripts.
Whenever I find a subscript, I want to add it to the list of scripts I'm searching through.
I came up with this while loop:
while keep_checking == True:
TMP = deepcopy(FILE_LIST)
for fname in TMP:
if not fname in processed:
SCL_FILE = fname
break
handleSCL(SCL_FILE)
processed.add(SCL_FILE)
if processed == FILE_LIST:
keep_checking = False
break
The code above does the job, but I feel like dirty. handleSCL() is searching for the file and adding any new subscripts to FILE_LIST.
Is there a cleaner way of doing this?

I would use a method similar to the A* pathfinding algorithm (just without the pathfinding part).
Open list: placesfiles not yet examined.
Closed list: placesfiles already examined.
Start by adding your first file to openlist; then iterate across every element in openlist. For each element, find all files, and for each new file, check if it's in either list. If it's in neither, add it to openlist. When finished with the element, add it to closedlist.
This is a pretty effective and clean way of going through all of the elements without duplication.
EDIT: upon further consideration, you could use one ordered list, and iterate through it, adding new files to the end of the list. [beginning-current] is the closedlist, and [current-end] is the openlist. A* requires two lists because of sorting and path cost calculations, but you are doing a full search, so you don't need that feature. Then you just need a "add if not exist" for the single list.

Your loop needs some cleanups!
break will break out of the while loop, no need for keep_checking. Also no need for TMP, use it directly in the for loop.
while processed != FILE_LIST:
for fname in deepcopy(FILE_LIST):
if not fname in processed:
SCL_FILE = fname
break
handleSCL(SCL_FILE)
processed.add(SCL_FILE)
will do the same work in less code.

After much thinking, I ended up writing a quick custom queue.
class PerQueue(object):
def __init__(self):
self._init()
self.all_files = set()
self.current = None
self.files = set()
self._init = False
self.cur_files = set()
def _setflag(self, value):
self._init = value
for item in self.all_files:
if item.startswith('ss'):
self.cur_files.add(item)
def _getflag(self):
return self._init
def empty(self):
n = self._empty()
return n
def pushMany(self, itemList):
for item in itemList:
self.push(item)
def push(self, item):
if not item in self.all_files and not item in self.files:
self._put(item)
def pop(self):
# I don't want errors
if not self.empty():
self.current = self._get()
self.all_files.add(self.current)
if self.init:
self.cur_files.add(self.current)
else:
self.current = None
return self.current
def _init(self):
self.files = set()
def _empty(self):
return not self.files
def _get(self):
return self.files.pop()
def _put(self, item):
self.files.add(item)
init = property(_getflag, _setflag)
This allowed me to handle a couple of special conditions (using all_files and cur_files) along with the init flag. At most we have a couple of hundred files to process at any time, so I wasn't worried about size constraints.

This would be way cleaner... and probably unnecessary at this point:
for fname in deepcopy(FILE_LIST):
handleSCL(fname)

Wxpython: TreeCtrl: Iteration over a tree

I am using the following method to iterate over all the nodes of a wxpython treectrl.
def get_desired_parent(self, name, selectednode = None):
if selectednode == None:
selectednode = self.treeCtrl.RootItem
# First perform the action on the first object separately
childcount = self.treeCtrl.GetChildrenCount(selectednode, False)
if childcount == 0:
return None
(item,cookie) = self.treeCtrl.GetFirstChild(selectednode)
if self.treeCtrl.GetItemText(item) == name:
return item
while childcount > 1:
childcount = childcount - 1
# Then iterate over the rest of objects
(item,cookie) = self.treeCtrl.GetNextChild(item,cookie)
if self.treeCtrl.GetItemText(item) == name:
return item
return None
This problem of excess code becomes even more apparent when I am iterating inside the structure recursively.
Is there another way of performing the same actions in more compact manner, to make my code more concise / pythonic.

You could use a function that is inside this one (in its namespace only) that will check if it matches the conditiin or not. If it does return the item if it doesn't, continue.
Otherwise you could check your condition just after the while line. This way the item variable will be defined by the first child before the loop and evaluated like any other.
Still another way: (or a mix of the two)
(child, cookie) = self.GetFirstChild(item)
while child.IsOk():
do_something(child)
(child, cookie) = self.GetNextChild(item, cookie)

Here a full example that traverses the tree going depth first. The function was bound to the right button.
def OnRightDown(self, event):
def showChildren(item,cookie):
# functions goes recursively down the tree
if item.IsOk():
child, cookie = self.tree.GetFirstChild(item)
while child.IsOk():
child, cookie = self.tree.GetNextChild(child, cookie)
if child:
print(self.tree.GetItemText(child)) #show child label name
showChildren(child,cookie)
pt = event.GetPosition()
item, flags = self.tree.HitTest(pt)
if item:
print(self.tree.GetItemText(item)) #show parent label name
showChildren(item,0) #iterate depth first into the tree

The best way to make your code highly readable here is to make it short and highly functional.
If you need to iterate through all the tree items and do so through depth first. Here's that as a single quick function. Hand it a function that gets each item, and where you start (usually self.root). It's also quite reusable since you might be doing this a lot.
def depth_first_tree(self, funct, item):
(child, cookie) = self.tree.GetFirstChild(item)
while child.IsOk():
self.depth_first_tree(funct, child)
funct(child)
(child, cookie) = self.tree.GetNextChild(item, cookie)