We Keep Coding

Generator expressions vs generator functions and surprisingly eager evaluation

For reasons which are not relevant I am combining some data structures in a certain way, whilst also replacing the Python 2.7's default dict with OrderedDict. The data structures use tuples as keys in dictionaries. Please ignore those details (the replacement of the dict type is not useful below, but it is in the real code).
import __builtin__
import collections
import contextlib
import itertools
def combine(config_a, config_b):
return (dict(first, **second) for first, second in itertools.product(config_a, config_b))
#contextlib.contextmanager
def dict_as_ordereddict():
dict_orig = __builtin__.dict
try:
__builtin__.dict = collections.OrderedDict
yield
finally:
__builtin__.dict = dict_orig
This works as expected initially (dict can take non-string keyword arguments as a special case):
print 'one level nesting'
with dict_as_ordereddict():
result = combine(
[{(0, 1): 'a', (2, 3): 'b'}],
[{(4, 5): 'c', (6, 7): 'd'}]
)
print list(result)
print
Output:
one level nesting
[{(0, 1): 'a', (4, 5): 'c', (2, 3): 'b', (6, 7): 'd'}]
However, when nesting calls to the combine generator expression, it can be seen that the dict reference is treated as OrderedDict, lacking the special behaviour of dict to use tuples as keyword arguments:
print 'two level nesting'
with dict_as_ordereddict():
result = combine(combine(
[{(0, 1): 'a', (2, 3): 'b'}],
[{(4, 5): 'c', (6, 7): 'd'}]
),
[{(8, 9): 'e', (10, 11): 'f'}]
)
print list(result)
print
Output:
two level nesting
Traceback (most recent call last):
File "test.py", line 36, in <module>
[{(8, 9): 'e', (10, 11): 'f'}]
File "test.py", line 8, in combine
return (dict(first, **second) for first, second in itertools.product(config_a, config_b))
File "test.py", line 8, in <genexpr>
return (dict(first, **second) for first, second in itertools.product(config_a, config_b))
TypeError: __init__() keywords must be strings
Furthermore, implementing via yield instead of a generator expression fixes the problem:
def combine_yield(config_a, config_b):
for first, second in itertools.product(config_a, config_b):
yield dict(first, **second)
print 'two level nesting, yield'
with dict_as_ordereddict():
result = combine_yield(combine_yield(
[{(0, 1): 'a', (2, 3): 'b'}],
[{(4, 5): 'c', (6, 7): 'd'}]
),
[{(8, 9): 'e', (10, 11): 'f'}]
)
print list(result)
print
Output:
two level nesting, yield
[{(0, 1): 'a', (8, 9): 'e', (2, 3): 'b', (4, 5): 'c', (6, 7): 'd', (10, 11): 'f'}]
Questions:
Why does some item (only the first?) from the generator expression get evaluated before required in the second example, or what is it required for?
Why is it not evaluated in the first example? I actually expected this behaviour in both.
Why does the yield-based version work?

Before going into the details note the following: itertools.product evaluates the iterator arguments in order to compute the product. This can be seen from the equivalent Python implementation in the docs (the first line is relevant):
def product(*args, **kwds):
pools = map(tuple, args) * kwds.get('repeat', 1)
...
You can also try this with a custom class and a short test script:
import itertools
class Test:
def __init__(self):
self.x = 0
def __iter__(self):
return self
def next(self):
print('next item requested')
if self.x < 5:
self.x += 1
return self.x
raise StopIteration()
t = Test()
itertools.product(t, t)
Creating the itertools.product object will show in the output that all the iterators items are immediately requested.
This means, as soon as you call itertools.product the iterator arguments are evaluated. This is important because in the first case the arguments are just two lists and so there's no problem. Then you evaluate the final result via list(result after the context manager dict_as_ordereddict has returned and so all calls to dict will be resolved as the normal builtin dict.
Now for the second example the inner call to combine works still fine, now returning a generator expression which is then used as one of the arguments to the second combine's call to itertools.product. As we've seen above these arguments are immediately evaluated and so the generator object is asked to generate its values. In order to do so, it needs to resolve dict. However now we're still inside the context manager dict_as_ordereddict and for that reason dict will be resolved as OrderedDict which doesn't accept non-string keys for keyword arguments.
It is important to notice here that the first version which uses return needs to create the generator object in order to return it. That involves creating the itertools.product object. That means this version is as lazy as itertools.product.
Now to the question why the yield version works. By using yield, invoking the function will return a generator. Now this is a truly lazy version in the sense that execution of the function body doesn't start until items are requested. This means neither the inner nor the outer call to convert will start executing the function body and thus invoking itertools.product until the items are requested via list(result). You can check that by putting an additional print statement inside that function and right behind the context manager:
def combine(config_a, config_b):
print 'start'
# return (dict(first, **second) for first, second in itertools.product(config_a, config_b))
for first, second in itertools.product(config_a, config_b):
yield dict(first, **second)
with dict_as_ordereddict():
result = combine(combine(
[{(0, 1): 'a', (2, 3): 'b'}],
[{(4, 5): 'c', (6, 7): 'd'}]
),
[{(8, 9): 'e', (10, 11): 'f'}]
)
print 'end of context manager'
print list(result)
print
With the yield version we'll notice that it prints the following:
end of context manager
start
start
I.e. the generators are started only when the results are requested via list(result). This is different from the return version (uncomment in the above code). Now you'll see
start
start
and before the end of the context manager is reached the error is already raised.
On a side note, in order for your code to work, the replacement of dict needs to be ineffective (and it is for the first version), so I don't see why you would use that context manager at all. Secondly, dict literals are not ordered in Python 2, and neither are keyword arguments so that also defeats the purpose of using OrderedDict. Also note that in Python 3 that non-string keyword arguments behavior of dict has been removed and the clean way to update dictionaries of any keys is to use dict.update.

Python change namespace to access attributes of a class

I have a class with a lot of attributes to be set. In order to do so, I do:
opts.a = 1
opts.b = 2
# ...
opts.xyz = 0
After repeatedly writing opts. at the beginning of variables I am wondering: Is it possible to wrap this into a function or context so that the namespace is set to the class attributes so that I don't have to write opts all the time? I.e. that I only have to do something like:
with opts:
a = 1
b = 2
# ...
xyz = 3
I thought of moving the code inside a function of the class, but that doesn't make things easier to read or write, since then I'd need to write self instead of opts everytime.
One side condition for my case: the __setattr__ of the class should be called, since I did override that with a custom function to store the order in which attributes are set.

If I take your question and the comments together, you want to use the class-as-enum approach, but with mutable values, you want a quick way to update multiple attributes, and you want to store the order in which the attributes are set.
Here is a partial implementation that does what I think you want:
class MyClass:
def __init__(self):
self.__dict__['values'] = collections.OrderedDict()
def __setattr__ (self,a,v):
self.__dict__['values'][a]=v
def __getattr__ (self,a):
return self.__dict__['values'][a]
def __repr__(self):
return repr(self.__dict__['values'])
Given this definition you can do:
>>> m = MyClass()
>>> m.z = 56
>>> m.y = 22
>>> m.b = 34
>>> m.c = 12
>>> m
OrderedDict([('z', 56), ('y', 22), ('b', 34), ('c', 12)])
Quick update:
>>> m.values.update(f=2, g=3, h=5)
>>> m
OrderedDict([('z', 56), ('y', 22), ('b', 34), ('c', 12), ('h', 5), ('g', 3), ('f', 2)])
It is true that the relative order of f-g is a bit surprising, but since the update happens in a single statement, there is an argument for saying that the updates are simultaneous and so the relative order is arbitrary.

Trying to find majority element in a list

I'm writing a function to find a majority in a Python list.
Thinking that if I can write a hash function that can map every element to a single slot in the new array or to a unique identifier, perhaps for a dictionary, that should be the best and it should be undoable. I am not sure how to progress. My hash function is obviously useless, any tips on what I can/should do, or if this is even a reasonable approach?
def find_majority(k):
def hash_it(q):
return q
map_of = [0]*len(k)
for i in k:
mapped_to = hash_it(i) #hash function
map_of[mapped_to]+=1
find_majority([1,2,3,4,3,3,2,4,5,6,1,2,3,4,5,1,2,3,4,6,5])

Python has a built-in class called Counter that will do this for you.
>>> from collections import Counter
>>> c = Counter([1,2,3,4,3,3,2,4,5,6,1,2,3,4,5,1,2,3,4,6,5])
>>> c.most_common()
[(3, 5), (2, 4), (4, 4), (1, 3), (5, 3), (6, 2)]
>>> value, count = c.most_common()[0]
>>> print value
3
See the docs.
http://docs.python.org/2/library/collections.html#collections.Counter

There is an easy way to realize like this
l = [1,2,3,4,3,3,2,4,5,6,1,2,3,4,5,1,2,3,4,6,5]
print(max(set(l), key = l.count)) # 3

I think your approach is to use another array as big as k as your "hash map". If k is huge but the number of unique elements is not so huge, you would be wasting a lot of space. Furthermore, to find the majority, you would have to loop through your map_of hashmap/array to find the max.
On the other hand, a dictionary/set (where hashing is not your concern, and the underlying array structure will probably be more compact for average cases) seems a little more appropriate. Needless to say, with the occurring elements as keys and their occurrences as values, you can find what you want in one single iteration.
So, something like:
def find_majority(k):
myMap = {}
maximum = ( '', 0 ) # (occurring element, occurrences)
for n in k:
if n in myMap: myMap[n] += 1
else: myMap[n] = 1
# Keep track of maximum on the go
if myMap[n] > maximum[1]: maximum = (n,myMap[n])
return maximum
And as expected, we get what we want.
>>> find_majority([1,2,3,4,3,3,2,4,5,6,1,2,3,4,5,1,2,3,4,6,5])
(3, 5)
Of course, Counters and other cool modules will let you do what you want in finer syntax.

looking at a number in a tuple

Right now my function is not recoginizing my numbers in the list coeff as numbers. I am trying to pair up items from the two list and then sort them into a different list based on the value of mul. But everything is going into the negative list. How to i make sure it is considering mul as a number going into each if statement.
def balance_equation(species,coeff):
data=zip(coeff,species)
positive=[]
negative=[]
for (mul,el) in data:
if mul<0:
negative.append((el,mul))
if mul>0:
positive.append((el,mul))
Edit;
I ment to originally include this
balance_equation(['H2O','A2'],['6','-4'])

Your problem is that in the way you call it (balance_equation(['H2O','A2'],['6','-4'])), mul is a string rather than an int ('6' or '-4' rather than 6 or -4). Change your if statement to:
if int(mul)<0:
negative.append((el,mul))
if int(mul)>0:
positive.append((el,mul))
This converts mul to an integer before comparing it to 0.

Well, the first problem is that your function just returns None, just throwing away the two lists, so there's no way to even see whether it's doing the right thing.
If you fix that, you'll see that it is doing the right thing.
def balance_equation(species,coeff):
data=zip(coeff,species)
positive=[]
negative=[]
for (mul,el) in data:
if mul<0:
negative.append((el,mul))
if mul>0:
positive.append((el,mul))
return negative, positive
>>> n, p = balance_equation(balance_equation('abcdef', range(-3,3))
>>> n
[('a', -3), ('b', -2), ('c', -1)]
>>> p
[('e', 1), ('f', 2)]
So, there are two possibilities:
Since the code you pasted is clearly not the actual code you're running, maybe you fixed the bug while rewriting it to post here.
You're not calling it with sensible inputs. For example, if you pass the parameters backward, since species is presumably a collection of strings, they'll all end up positive. Or, likewise, if you pass the coeffs as string representations of integers.
If it's the last problem—you're passing, say, 'abcdef', ['-3', '-2', '-1', '0', '1', '2', '3'], and you want to deal with that within balance_equation instead of in the calling code, that's easy. Just add this line before the zip:
coeff = [int(x) for x in coeff]
Or change your zip to:
data = zip((int(x) for x in coeff), species)
By the way, I'm assuming you're on CPython 2. In Python 3, trying to compare a string to 0 will raise a TypeError instead of always returning True, while in other Python 2 implementations it might always return False instead of True…

I think you have your answer, but there's also a simpler way of doing this in Python:
for (mul, el) in data:
append_to = negative.append if mul < 0 else positive.append
append_to(el)
Not sure what "should happen" to 0 though

transitive closure python tuples

Does anyone know if there's a python builtin for computing transitive closure of tuples?
I have tuples of the form (1,2),(2,3),(3,4) and I'm trying to get (1,2),(2,3),(3,4),(1,3)(2,4)
Thanks.

There's no builtin for transitive closures.
They're quite simple to implement though.
Here's my take on it:
def transitive_closure(a):
closure = set(a)
while True:
new_relations = set((x,w) for x,y in closure for q,w in closure if q == y)
closure_until_now = closure | new_relations
if closure_until_now == closure:
break
closure = closure_until_now
return closure
call:
transitive_closure([(1,2),(2,3),(3,4)])
result:
set([(1, 2), (1, 3), (1, 4), (2, 3), (3, 4), (2, 4)])
call:
transitive_closure([(1,2),(2,1)])
result:
set([(1, 2), (1, 1), (2, 1), (2, 2)])

Just a quick attempt:
def transitive_closure(elements):
elements = set([(x,y) if x < y else (y,x) for x,y in elements])
relations = {}
for x,y in elements:
if x not in relations:
relations[x] = []
relations[x].append(y)
closure = set()
def build_closure(n):
def f(k):
for y in relations.get(k, []):
closure.add((n, y))
f(y)
f(n)
for k in relations.keys():
build_closure(k)
return closure
Executing it, we'll get
In [3]: transitive_closure([(1,2),(2,3),(3,4)])
Out[3]: set([(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)])

We can perform the "closure" operation from a given "start node" by repeatedly taking a union of "graph edges" from the current "endpoints" until no new endpoints are found. We need to do this at most (number of nodes - 1) times, since this is the maximum length of a path. (Doing things this way avoids getting stuck in infinite recursion if there is a cycle; it will waste iterations in the general case, but avoids the work of checking whether we are done i.e. that no changes were made in a given iteration.)
from collections import defaultdict
def transitive_closure(elements):
edges = defaultdict(set)
# map from first element of input tuples to "reachable" second elements
for x, y in elements: edges[x].add(y)
for _ in range(len(elements) - 1):
edges = defaultdict(set, (
(k, v.union(*(edges[i] for i in v)))
for (k, v) in edges.items()
))
return set((k, i) for (k, v) in edges.items() for i in v)
(I actually tested it for once ;) )

Suboptimal, but conceptually simple solution:
def transitive_closure(a):
closure = set()
for x, _ in a:
closure |= set((x, y) for y in dfs(x, a))
return closure
def dfs(x, a):
"""Yields single elements from a in depth-first order, starting from x"""
for y in [y for w, y in a if w == x]:
yield y
for z in dfs(y, a):
yield z
This won't work when there's a cycle in the relation, i.e. a reflexive point.

Here's one essentially the same as the one from #soulcheck that works on adjacency lists rather than edge lists:
def inplace_transitive_closure(g):
"""g is an adjacency list graph implemented as a dict of sets"""
done = False
while not done:
done = True
for v0, v1s in g.items():
old_len = len(v1s)
for v2s in [g[v1] for v1 in v1s]:
v1s |= v2s
done = done and len(v1s) == old_len

If you have a lot of tupels (more than 5000), you might want to consider using the scipy code for matrix powers (see also http://www.ics.uci.edu/~irani/w15-6B/BoardNotes/MatrixMultiplication.pdf)
from scipy.sparse import csr_matrix as csr
def get_closure(tups):
index2id = list(set([tup[0] for tup in tups]) | set([tup[1] for tup in tups]));
id2index = {index2id[i]:i for i in xrange(len(index2id))};
tups_re = tups + [(index2id[i],index2id[i],) for i in xrange(len(index2id))]; # Unfortunately you have to make the relation reflexive first - you could also add the diagonal to M
M = csr( ([True for tup in tups_re],([id2index[tup[0]] for tup in tups_re],[id2index[tup[1]] for tup in tups_re])),shape=(len(index2id),len(index2id)),dtype=bool);
M_ = M**n; # n is maximum path length of your relation
temp = M_.nonzero();
#TODO: You might want to remove the added reflexivity tupels again
return [(index2id[temp[0][i]],index2id[temp[1][i]],) for i in xrange(len(temp[0]))];
In the best case, you can choose n wisely if you know a bit about your relation/graph -- that is how long the longest path can be. Otherwise you have to choose M.shape[0], which might blow up in your face.
This detour also has its limits, in particular you should be sure than the closure does not get too large (the connectivity is not too strong), but you would have the same problem in the python implementation.

You can create a graph from those tuples then use connnected components algorithm from the created graph. Networkx is library that supports connnected components algorithm.