Related
i have a nested tuple like this one :
this_one = (w,h,(1,2,4,8,16),(0,2,3,4,6),("0"),("0"),(0,1))
It will be used to feed:
itertools.product(*this_one)
w and h have to be both generators.
Generated values from h depends on generated values from w as in this function:
def This_Function(maxvalue):
for i in range(maxvalue):
w_must_generate = i
for j in range(i+1):
h_must_generate = j
Iv tried using a yield in This_function(maxvalue):
def This_Function(maxvalue):
for i in range(maxvalue):
for j in range(i+1):
yield i,j
Returning a single generator object which gives:
which_gives = itertools.product(*this_one)
for result in which_gives:
print(result)
...
((23,22),(16),(6),(0),(0),(0))
((23,22),(16),(6),(0),(0),(1))
((23,23),(1),(0),(0),(0),(0))
...
A single tuple at result[0] holding 2 values.
And this is not what i want .
The result i want has to be like the following:
the_following = itertools.product(*this_one)
for result in the_following:
print(result)
...
((23),(22),(16),(6),(0),(0),(0))
((23),(22),(16),(6),(0),(0),(1))
((23),(23),(1),(0),(0),(0),(0))
...
Without having to do something like :
the_following = itertools.product(*this_one)
for result in the_following:
something_like = tuple(result[0][0],result[0][1],result[1],...,result[5])
print(something_like)
...
((23),(22),(16),(6),(0),(0),(0))
((23),(22),(16),(6),(0),(0),(1))
((23),(23),(1),(0),(0),(0),(0))
...
Any ideas ?
That's fundamentally not how product works. From the documentation:
Before product() runs, it completely consumes the input iterables, keeping pools of values in memory to generate the products. Accordingly, it is only useful with finite inputs.
Thus, having the second one be dependent won't get you the result you want, assuming that is for h values to always be w+1..max in the resulting output sequence.
I'm not sure why you couldn't just run it with your This_function approach and destructure it with a generator expression:
result = ((a, b, c, d, e, f, g)
for ((a, b), c, d, e, f, g) in itertools.product(*this_one))
Since the values are all entirely determined, this works and is fairly legible:
from itertools import product
maxvalue = 5
w = (i for i in range(maxvalue) for j in range(i + 1))
h = (j for i in range(maxvalue) for j in range(i + 1))
this_one = (w, h, (1, 2, 4, 8, 16), (0, 2, 3, 4, 6), ("0"), ("0"), (0, 1))
result = product(*this_one)
The values yielded from w are basically 0, 1, 1, 2, 2, 2, 3, 3, 3, 3, ... and the values yielded from h are 0, 0, 1, 0, 1, 2, 0, 1, 2, 3, .... Plus, the code pretty much looks like your description of the desired behaviour, which is generally a good thing.
Both w and h are generators, no space is wasted and the solution is fairly optimal. I just can't tell if result is what you actually need, but that depends on what you need and expect product() to do here.
I want to write a Rem(a, b) which return a new tuple that is like a, with the first appearance of element b is removed. For example
Rem((0, 1, 9, 1, 4), 1) which will return (0, 9, 1, 4).
I am only allowed to use higher order functions such as lambda, filter, map, and reduce.
I am thinking about to use filter but this will delete all of the match elements
def myRem(T, E):
return tuple(filter(lambda x: (x!=E), T))
myRem((0, 1, 9, 1, 4), 1) I will have (0,9,4)
The following works (Warning: hacky code):
tuple(map(lambda y: y[1], filter(lambda x: (x[0]!=T.index(E)), enumerate(T))))
But I would never recommend doing this unless the requirements are rigid
Trick with temporary list:
def removeFirst(t, v):
tmp_lst = [v]
return tuple(filter(lambda x: (x != v or (not tmp_lst or v != tmp_lst.pop(0))), t))
print(removeFirst((0, 1, 9, 1, 4), 1))
tmp_lst.pop(0) - will be called only once (thus, excluding the 1st occurrence of the crucial value v)
not tmp_lst - all remaining/potential occurrences will be included due to this condition
The output:
(0, 9, 1, 4)
For fun, using itertools, you can sorta use mostly higher-order functions...
>>> from itertools import *
>>> data = (0, 1, 9, 1, 4)
>>> not1 = (1).__ne__
>>> tuple(chain(takewhile(not1, data), islice(dropwhile(not1, data), 1, None)))
(0, 9, 1, 4)
BTW, here's some timings comparing different approaches for dropping a particular index in a tuple:
>>> timeit.timeit("t[:i] + t[i+1:]", "t = tuple(range(100000)); i=50000", number=10000)
10.42419078599778
>>> timeit.timeit("(*t[:i], *t[i+1:])", "t = tuple(range(100000)); i=50000", number=10000)
20.06185237201862
>>> timeit.timeit("(*islice(t,None, i), *islice(t, i+1, None))", "t = tuple(range(100000)); i=50000; from itertools import islice", number=10000)
>>> timeit.timeit("tuple(chain(islice(t,None, i), islice(t, i+1, None)))", "t = tuple(range(100000)); i=50000; from itertools import islice, chain", number=10000)
19.71128663700074
>>> timeit.timeit("it = iter(t); tuple(chain(islice(it,None, i), islice(it, 1, None)))", "t = tuple(range(100000)); i=50000; from itertools import islice, chain", number=10000)
17.6895881179953
Looks like it is hard to beat the straightforward: t[:i] + t[i+1:], which is not surprising.
Note, this one is shockingly less performant:
>>> timeit.timeit("tuple(j for i, j in enumerate(t) if i != idx)", "t = tuple(range(100000)); idx=50000", number=10000)
111.66658291200292
Which makes me thing all these solutions using takewhile, filter and lambda will all suffer pretty bad...
Although:
>>> timeit.timeit("not1 = (i).__ne__; tuple(chain(takewhile(not1, t), islice(dropwhile(not1, t), 1, None)))", "t = tuple(range(100000)); i=50000; from itertools import chain, takewhile,dropwhile, islice", number=10000)
62.22159145199112
Almost twice as fast as the generator expression, which goes to show, generator overhead can be quite large. However, takewhile and dropwhile are implemented in C, albeit this implementation has redundancy (take-while and dropwhile will pass the dropwhile areas twice).
Another interesting observation, if we simply wrap the substitute a list-comp for the generator expression, it is significantly faster despite the fact that the list-comprehension + tuple call iterates over the result twice compared to only once with the generator expression:
>>> timeit.timeit("tuple([j for i, j in enumerate(t) if i != idx])", "t = tuple(range(100000)); idx=50000", number=10000)
82.59887028901721
Goes to show how steep the generator-expression price can be...
Here is a solution that only uses lambda, filter(), map(), reduce() and tuple().
def myRem(T, E):
# map the tuple into a list of tuples (value, indicator)
M = map(lambda x: [(x, 1)] if x == E else [(x,0)], T)
# make the indicator 0 once the first instance of E is found
# think of this as a boolean mask of items to remove
# here the second reduce can be changed to the sum function
R = reduce(
lambda x, y: x + (y if reduce(lambda a, b: a+b, map(lambda z: z[1], x)) < 1
else [(y[0][0], 0)]),
M
)
# filter the reduced output based on the indicator
F = filter(lambda x: x[1]==0, R)
# map the output back to the desired format
O = map(lambda x: x[0], F)
return tuple(O)
Explanation
A good way to understand what's going on is to print the outputs of the intermediate steps.
Step 1: First Map
For each value in the tuple, we return a tuple with the value and a flag to indicate if it's the value to remove. These tuples are encapsulated in a list because it makes combining easier in the next step.
# original example
T = (0, 1, 9, 1, 4)
E = 1
M = map(lambda x: [(x, 1)] if x == E else [(x,0)], T)
print(M)
#[[(0, 0)], [(1, 1)], [(9, 0)], [(1, 1)], [(4, 0)]]
Step 2: Reduce
This returns a list of tuples in a similar structure to the contents of M, but the flag variable is set to 1 for the first instance of E, and 0 for all subsequent instances. This is achieved by calculating the sum of the indicator up to that point (implemented as another reduce()).
R = reduce(
lambda x, y: x + (y if reduce(lambda a, b: a+b, map(lambda z: z[1], x)) < 1
else [(y[0][0], 0)]),
M
)
print(R)
#[(0, 0), (1, 1), (9, 0), (1, 0), (4, 0)]
Now the output is in the form of (value, to_be_removed).
Step 3: Filter
Filter out the value to be removed.
F = filter(lambda x: x[1]==0, R)
print(F)
#[(0, 0), (9, 0), (1, 0), (4, 0)]
Step 4: Second map and conversion to tuple
Extract the value from the filtered list, and convert it to a tuple.
O = map(lambda x: x[0], F)
print(tuple(O))
#(0, 9, 1, 4)
This violates your requirement for "only using higher order functions" - but since it's not clear why this is a requirement, I include the below solution.
def myRem(tup, n):
idx = tup.index(n)
return tuple(j for i, j in enumerate(tup) if i != idx)
myRem((0, 1, 9, 1, 4), 1)
# (0, 9, 1, 4)
Here is a numpy solution (still not using higher-order functions):
import numpy as np
def myRem(tup, n):
tup_arr = np.array(tup)
return tuple(np.delete(tup_arr, np.min(np.nonzero(tup_arr == n)[0])))
myRem((0, 1, 9, 1, 4), 1)
# (0, 9, 1, 4)
I need to perform a special type of tensor contraction. I want something of this kind:
A_{bg} = Sum_{a,a',a''} ( B_{a} C_{a'b} D_{a''g} )
where all the indices can have values 0,1 and the sum over a, a' and a'' is carried for all cases where a+a'+a'' = 1 or a+a'+a'' = 2. So it is like the reverse of the Einstein summation convention: I want to sum only when one of the three indices is different to the others.
Moreover, I want some flexibility with the number of indices that are not being summed: in the example the resulting tensor has 2 indices, and the sum is over products of elements of 3 tensors, one with one index, the other two with two indices. These numbers of indices are going to vary, so in general I would like to be able to write something like this:
A_{...} = Sum_{a,a',a''} ( B_{a...} C_{a...} D_{a''...} )
I want to point that the number of indices is not fixed, but it is controlled: I can know and specify how many indices every tensor has in each step.
I tried np.einsum(), but then apparently I am forced to sum over repeated indices in the standard Einstein convention, and I don't know how to implement the condition I exposed here.
And I cannot write everything with various for because, as I said, the number of indices of the tensors involved is not fixed.
Anyone has an idea?
From comments:
I would write what I put here in programming language like this:
tensa = np.zeros((2,2))
for be in range(2):
for ga in range(2):
for al in range(2):
for alp in range(2):
for alpp in range(res(al,alp),prod(al,alp)):
tensa[be,ga] += tensb[al] * tensc[alp,be] * tensd[alpp,ga]
where res and prod are two functions that ensure that al+alp+alpp = 1 or 2. The problem with this is that I need to specify all the indices involved, and I cannot do that in the general calculation for all the lattice.
First, lets write your example out in Python loops, to have a baseline for comparisons. If I understood you correctly, this is what you want to do:
b, g = 4, 5
B = np.random.rand(2)
C = np.random.rand(2, b)
D = np.random.rand(2, g)
out = np.zeros((b, g))
for j in (0, 1):
for k in (0, 1):
for l in (0, 1):
if j + k + l in (1, 2):
out += B[j] * C[k, :, None] * D[l, None, :]
When I run this, I get this output:
>>> out
array([[ 1.27679643, 2.26125361, 1.32775173, 1.5517918 , 0.47083151],
[ 0.84302586, 1.57516142, 1.1335904 , 1.14702252, 0.34226837],
[ 0.70592576, 1.34187278, 1.02080112, 0.99458563, 0.29535054],
[ 1.66907981, 3.07143067, 2.09677013, 2.20062463, 0.65961165]])
You can't get at this directly with np.einsum, but you can run it twice and get your result as the difference of these two:
>>> np.einsum('i,jk,lm->km', B, C, D) - np.einsum('i,ik,im->km', B, C, D)
array([[ 1.27679643, 2.26125361, 1.32775173, 1.5517918 , 0.47083151],
[ 0.84302586, 1.57516142, 1.1335904 , 1.14702252, 0.34226837],
[ 0.70592576, 1.34187278, 1.02080112, 0.99458563, 0.29535054],
[ 1.66907981, 3.07143067, 2.09677013, 2.20062463, 0.65961165]])
The first call to np.einsum is adding everything up, regardless of what the indices add up to. The second only adds up those where all three indices are the same. So obviously your result is the difference of the two.
Ideally, you could now go on to write something like:
>>>(np.einsum('i...,j...,k...->...', B, C, D) -
... np.einsum('i...,i...,i...->...', B, C, D))
and get your result regardless of the dimensions of your C and D arrays. If you try the first, you will get the following error message:
ValueError: operands could not be broadcast together with remapped shapes
[original->remapped]: (2)->(2,newaxis,newaxis) (2,4)->(4,newaxis,2,newaxis)
(2,5)->(5,newaxis,newaxis,2)
The problem is that, since you are not specifying what you want to do with the b and g dimensions of your tensors, it tries to broadcast them together, and since they are different, it fails. You can get it to work by adding extra dimensions of size 1:
>>> (np.einsum('i...,j...,k...->...', B, C, D[:, None]) -
... np.einsum('i...,i...,i...->...', B, C, D[:, None]))
array([[ 1.27679643, 2.26125361, 1.32775173, 1.5517918 , 0.47083151],
[ 0.84302586, 1.57516142, 1.1335904 , 1.14702252, 0.34226837],
[ 0.70592576, 1.34187278, 1.02080112, 0.99458563, 0.29535054],
[ 1.66907981, 3.07143067, 2.09677013, 2.20062463, 0.65961165]])
If you wanted all the axes of B to be placed before all the axes of C, and these before all the axes of D, the following seems to work, at least as far as creating an output of the right shape, although you may want to double check that the result is really what you want:
>>> B = np.random.rand(2, 3)
>>> C = np.random.rand(2, 4, 5)
>>> D = np.random.rand(2, 6)
>>> C_idx = (slice(None),) + (None,) * (B.ndim - 1)
>>> D_idx = C_idx + (None,) * (C.ndim - 1)
>>> (np.einsum('i...,j...,k...->...', B, C[C_idx], D[D_idx]) -
... np.einsum('i...,i...,i...->...', B, C[C_idx], D[D_idx])).shape
(3L, 4L, 5L, 6L)
EDIT From the comments, if instead of just the first axis of each tensor having to be reduced over, it is the first two, then the above could be written as:
>>> B = np.random.rand(2, 2, 3)
>>> C = np.random.rand(2, 2, 4, 5)
>>> D = np.random.rand(2, 2, 6)
>>> C_idx = (slice(None),) * 2 + (None,) * (B.ndim - 2)
>>> D_idx = C_idx + (None,) * (C.ndim - 2)
>>> (np.einsum('ij...,kl...,mn...->...', B, C[C_idx], D[D_idx]) -
... np.einsum('ij...,ij...,ij...->...', B, C[C_idx], D[D_idx])).shape
(3L, 4L, 5L, 6L)
More generally, if reducing over d indices, C_idx and D_idx would look like:
>>> C_idx = (slice(None),) * d + (None,) * (B.ndim - d)
>>> D_idx = C_idx + (None,) * (C.ndim - d)
and the calls to np.einsum would need to have d letters in the indexing, unique in the first call, repeating in the second.
EDIT 2 So what exactly goes on with C_idx and D_idx? Take the last example, with B, C and D with shapes (2, 2, 3), (2, 2, 4, 5) and (2, 2, 6). C_idx is made up of two empty slices, plus as many Nones as the number of dimensions of B minus 2, so when we take C[C_idx] the result has shape (2, 2, 1, 4, 5). Similarly D_idx is C_idx plus as many Nones as the number of dimensions of C minus 2, so the result of D[D_idx] has shape (2, 2, 1, 1, 1, 6). These three arrays don't braodcast together, but np.einsum adds additional dimensions of size 1, i.e. the "remapped" shapes of the error above, so the resulting arrays turn out to have extra trailing ones, and the shapes amtch as follows:
(2, 2, 3, 1, 1, 1)
(2, 2, 1, 4, 5, 1)
(2, 2, 1, 1, 1, 6)
The first two axes are reduced, so the disappear from the output, and in the other cases, broadcasting applies, where a dimension of size 1 is replicated to match a larger one, so the output is (3, 4, 5, 6) as we wanted.
#hpaulj proposes a method using "Levi-Civita like" tensors, that should in theory be faster, see my comments to the original question. Here's some code for comparison:
b, g = 5000, 2000
B = np.random.rand(2)
C = np.random.rand(2, b)
D = np.random.rand(2, g)
def calc1(b, c, d):
return (np.einsum('i,jm,kn->mn', b, c, d) -
np.einsum('i,im,in->mn', b, c, d))
def calc2(b, c, d):
return np.einsum('ijk,i,jm,kn->mn', calc2.e, b, c, d)
calc2.e = np.ones((2,2,2))
calc2.e[0, 0, 0] = 0
calc2.e[1, 1, 1] = 0
But when running it:
%timeit calc1(B, C, D)
1 loops, best of 3: 361 ms per loop
%timeit calc2(B, C, D)
1 loops, best of 3: 643 ms per loop
np.allclose(calc1(B, C, D), calc2(B, C, D))
Out[48]: True
A surprising result, which I can't explain...
I have two arrays,
a = [3, 6, 8, 2, 5, 5]
b = [2, 7, 9]
and I need to create a new array c which takes the values and adds them like this: a[0+0]*b[0] + a[0+1]*b[1] + a[0+2]*b[2] = (3*2) + (6*7) + (9*8) = 6 + 42 + 72 which means c[0] = 120
I'm completely lost on how to do this anything to point me in the right direction would be awesome.
If c[k] = a[k+0]*b[0] + a[k+1]*b[1] + a[k+2]*b[2]
then
>>> c = [sum(i*j for i,j in zip(a[k:], b)) for k in range(4)]
>>> c
[120, 86, 75, 84]
total = 0
for n in range(0, min(len(a), len(b))):
total += a[n] * b[n]
range function
I think this will do what you want. It borrows some of the code from #DukeSilver's answer and makes it build a list, rather than just calculating a single value. My assumption is that a is always longer than b.
c = [sum(a[i+j]*b[j] for j in range(len(b))) for i in range(len(a) - len(b) + 1)]
numpy.convolve or do you want to write your own function?
I want to create a tuple of length m, with a 1 in each position except for one n in position k.
e.g.: m=5, n=7, k=3 should yield (1,1,1,7,1) (length 5 with a 7 in position 3)
How can I do this?
>>> m, n, k = 5, 7, 3
>>> tuple(n if i == k else 1 for i in range(m))
(1, 1, 1, 7, 1)
First construct a list, then turn it into a tuple if you really need to:
def make_tuple(m, n, k):
a = [1] * m
a[k] = n
return tuple(a)
Example:
>>> make_tuple(m=5, n=7, k=3)
(1, 1, 1, 7, 1)
(1,) * k + (n,) + (1,) * (m-k-1)