We Keep Coding

Numpy: Conserving sum in average over two arrays of integers

I have two arrays of positive integers A and B that each sum to 10:
A = [1,4,5]
B = [5,5,0]
I want to write a code (that will work for a general size of the array and the sum) to calculate the array C who is also a array of positive integers that also sums to 10 that is the closest to the element-wise average as possible:
Pure average C = (A + B) / 2: C=[3,4.5,2.5]
Round C = np.ceil((A + B) / 2).astype(int): C=[3,5,3], (sum=11, incorrect!)
Fix the sum C = SOME CODE: c=[3,4,3], (sum=10, correct!)
Any value can be adjusted to make the sum correct, as long as all elements remain positive integers.
What should C = SOME CODE be?
Minimum reproducible example:
A = np.array([1,4,5])
B = np.array([5,5,0])
C = np.ceil((A + B) / 2).astype(int)
print(np.sum(C))
11
This should give 10.

You can ceil/floor every other non-int element. This works for any shape/size and any sum value (in fact you do not need to know the sum at all. It is enough if A and B have same sum):
C = (A + B) / 2
C_c = np.ceil(C)
C_c[np.flatnonzero([C!=C.astype(int)])[::2]] -= 1
print(C_c.sum())
#10.0
print(C_c.astype(int))
#[3 4 3]

Ok so based off what you're saying, this could work:
C = ((a + b) / 2) # array([3, 4, 2])
curr_sum = sum(C) # 9
adjust_amount = sum(a) - curr_sum # 10-9 = 1
if adjust_amount > 0:
C[-1] += adjust_amount # array([3, 4, 3])
# Otherwise if it's negative just grab the largest and subtract to ensure you still remain >0
else:
C[np.argmax(C)] += adjust_amount

Elegant and/or fast equivalents of Excel SUMXMY2 (sum of square of differences of corresponding array elements) in python

Test Sample:
a = [0.1357678 0.27303184 -0.75600229]
b = [0.3813097 -0.72613616 0.18361217]
I would like to implement SUMXMY2(a, b) in Python without for loops
How can I do this?

As far as I know, - is not a valid operator for lists, so I would use a list comprehension. It does technically use a for loop, however I'd call it "elegant enough".
c = [(b[i] - a[i]) ** 2 for i in range(len(b))]
result = sum(c)
To make it more compact but less readable:
c = sum([(b[i] - a[i]) ** 2 for i in range(len(b))])
If you're dealing with lists of different lengths, use this:
c = [(b[i] - a[i]) ** 2 for i in range(min(len(b), len(a)))]
result = sum(c)

Squared difference is given by:
c = ((a - b) ** 2)
The sum is then simply given by
c = c.sum()
if a and b are lists you can convert them to pandas series first:
a = pd.Series(a)
or to numpy array as:
a = np.asarray(a)

Python: Finite sum with variable range "TypeError: only integer scalar arrays can be converted to a scalar index"

Aim: plot V vs. MF
import numpy as np
V = np.arange(3,46, step = 6)
A = 3
# 'n' is a sequence of odd numbers (i.e. 1,3,5,7, ...)
n = V/A
mm = (n+1)/2
MF = sum((np.power(-1, m)) * np.exp(m * m * (V/n)) for m in range(1,mm))
I want m to form a sequence of 1, 2, 3, ... (n+1)/2.
The problem I'm having is with the final line range(1,mm) which returns
TypeError: only integer scalar arrays can be converted to a scalar index
Seems to me the problem is that m cannot be variable for different numbers in the same sequence. Is there a way around this?

mm variable is a numpy array containing the sequence of 1, 2, 3 ... so range(1, mm) part doesn't make sense since range expects its arguments to be integers. If you want to iterate on mm, you can just
MF = sum((np.power(-1, m)) * np.exp(m * m * (V/n)) for m in mm)
Note : This is probably not what you want because V is a np.array, it makes resulting expression an array.
Since you are using numpy, you do not need to iterate on mm yourself. You can use mm directly:
MF = np.sum((-1) ** mm * np.exp(mm ** 2 * A))
>>> 2.4240441494100796e+83

import numpy as np
V = np.arange(3,46, step = 6)
A = 3
'n' is a sequence of odd numbers (i.e. 1,3,5,7, ...)
n = V/A
'm' is the upper limit of summation; indices of summation are every integer between 1 to m.
m = (n+1)/2
'y' represents the element to be summed
y = np.power(-1, m) * np.exp((m**2)*A)
MF = np.add.accumulate(y)
print(n)
print(MF)

Matlab to Python

I have a simple code in MATLAB which I am trying to translate to python, but I am stuck in a simple for loop:
Here is the situation:
Matlab
f0 = constant
fn = (nx1) matrix
b = (nx1) matrix
d and x are constant
mthd = 1 or 2
s = 1:-0.1:0.1;
for i = 1:10
f = fn * s(i)
switch mthd
case 1
v(:,i) = d *(1 + 1./b.*(f0./f)).^x
case 2
v(:,i) = log(f0./f)./b;
v(:,i) = v./(1+v)
end
v(1,:) = min(vp(2,:));
The output in Matlab results v with nx1 matrix
Assuming it is a simple equation with element wise operation in matlab,
I went ahead and wrote a code in python like this:
s = np.linspace(1,0.1,num=10)
for i in range(1,11)
f = fn * s[i]
if mthd ==1:
v = d *(1 + 1/b*(f0/f))^x
elif mthd ==2:
v = log(f0/f)/b;
v = v/(1+v)
Clearly, this is not the right one and I get stuck right from f = fn* s[i]
Any suggestion in this conversion will be of great help.
Thank you

Clearly this is not the right one and I get stuck right from f = fn* s[i]
What error message are you getting here? Make sure your vectors fn and b are numpy arrays and not lists.
for i in range(1,11)
Python uses zero indexing, whereas Matlab uses 1-indexing. Therefore your for loop should use for i in range(10), which iterates from 0 to 9 instead of 1 to 10.
v = d *(1 + 1/b*(f0/f))^x
Assuming fn and b are numpy arrays in your Python implementation, if you really want this to mirror the Matlab code you can still use indexing such as v[:,i]. However you need to initialize v as a numpy array with the correct size first.
v = log(f0/f)/b;
You probably want np.log here.
Hopefully this is helpful, let me know if you still have questions. You may also find this website helpful.

The code block below should be closer to what you want. Here are a few things to look out for:
Phyton arrays are indexed from 0. In base Python you handle powers with ** e.g. 2 ** 2 equals 4
When performing scalar multiplication and divide of arrays, better to use np.multiply and np.divide
Use np.log for logarithm and np.power for exponentiation with numpy matrices.
Use np.add to add a scalar to a numpy array.
import numpy as np
f0 = 5 # constant
fn = np.matrix([[5], [4], [3], [2], [1]]) # 5 x 1 matrix
b = np.matrix([[9], [8], [7], [6], [5]]) # 5 x 1 matrix
# d and x are constant
d = 4
x = 8
# mthd = 1 or 2
mthd = 1
s = np.linspace(1,0.1,num=10)
# python arrays are indexed from 0
for i in range(0,len(s)):
f = fn * s[i]
if mthd == 1:
v = np.power(np.multiply(d, (1 + np.divide(1., np.multiply
(b, np.divide(f0, f) ) ) ) ), x)
elif mthd ==2:
v = np.divide(np.log(np.divide(f0,f)), b);
v = np.divide(v, (np.add(1, v)) )

Sparse matrix multiplication when results' sparsity is known (in python|scipy|cython)

Suppose we want to compute C=A*B for given sparse matrices A,B but are interested in a very small subset of entries of C, represented by a list of index pairs:
rows=[i1, i2, i3 ... ]
cols=[j1, j2, j3 ... ]
Both A and B are quite large (say 50Kx50K), but very sparse (<1% of entries is non-zero).
How can we compute this subset of the multiplication?
Here's a naive implementation that works really slow:
def naive(A, B, rows, cols):
N = len(rows)
vals = []
for n in xrange(N):
v = A.getrow(rows[n]) * B.getcol(cols[n])
vals.append(v[0, 0])
R = sps.coo_matrix((np.array(vals), (np.array(rows), np.array(cols))), shape=(A.shape[0], B.shape[1]), dtype=np.float64)
return R
even for small matrices this is quite bad:
import scipy.sparse as sps
import numpy as np
D = 1000
A = np.random.randn(D, D)
A[np.abs(A) > 0.1] = 0
A = sps.csr_matrix(A)
B = np.random.randn(D, D)
B[np.abs(B) > 0.1] = 0
B = sps.csr_matrix(B)
X = np.random.randn(D, D)
X[np.abs(X) > 0.1] = 0
X[X != 0] = 1
X = sps.csr_matrix(X)
rows, cols = X.nonzero()
naive(A, B, rows, cols)
On my machine, naive() finishes after 1 minute, and most of the effort is spent on structuring the rows/cols (in getrow(), getcol()).
Of course, converting this (very small) example to dense matrices, the computation takes about 100ms:
A0 = np.array(A.todense())
B0 = np.array(B.todense())
X0 = np.array(X.todense())
A0.dot(B0) * X0
Any thoughts on how to efficiently compute such matrix multiplication?
Note: This question is almost identical to the following question:
Subset of a matrix multiplication, fast, and sparse
However, there, A and B are full matrices, and, one of the dimensions is very low (say, 10)
the proposed solutions seem to benefit from both.

The format of your sparse matrices is important here. You always need a row form A and a column from B. So, store A as a csr and B as csc to get rid of the getrow/getcol overhead. Unfortunately, this is only a small part of the story.
The best solution depends a lot on the structure of your sparse matrix (a lot of sparse columns/rows, etc), but you might try one based on dictionaries and sets. For matrix A for each row the following are kept:
a set with all non-zero column indices on that row
a dictionary with the non-zero indices as keys and the corresponding non-zero values as values
For matrix B similar dicts and sets are kept for each column.
To calculate element (M, N) in the multiplication result, row M of A is multiplied with column N of B. The multiplication:
find the set intersection of the non-zero sets
calculate the sum of multiplications of the non-zero elements (i.e. the intersection above)
In most cases this should be very fast, as in a sparse matrix the set intersection is usually very small.
Some code:
class rowarray():
def __init__(self, arr):
self.rows = []
for row in arr:
nonzeros = np.nonzero(row)[0]
nzvalues = { i: row[i] for i in nonzeros }
self.rows.append((set(nonzeros), nzvalues))
def __getitem__(self, key):
return self.rows[key]
def __len__(self):
return len(self.rows)
class colarray(rowarray):
def __init__(self, arr):
rowarray.__init__(self, arr.T)
def maybe_less_naive(A, B, rows, cols):
N = len(rows)
vals = []
for n in xrange(N):
nz1,v1 = A[rows[n]]
nz2,v2 = B[cols[n]]
# list of common non-zeros
nz = nz1.intersection(nz2)
# sum of non-zeros
vals.append(sum([ v1[i]*v2[i] for i in nz]))
R = sps.coo_matrix((np.array(vals), (np.array(rows), np.array(cols))), shape=(len(A), len(B)), dtype=np.float64)
return R
D = 1000
Ap = np.random.randn(D, D)
Ap[np.abs(Ap) > 0.1] = 0
A = rowarray(Ap)
Bp = np.random.randn(D, D)
Bp[np.abs(Bp) > 0.1] = 0
B = colarray(Bp)
X = np.random.randn(D, D)
X[np.abs(X) > 0.1] = 0
X[X != 0] = 1
X = sps.csr_matrix(X)
rows, cols = X.nonzero()
maybe_less_naive(A, B, rows, cols)
This is a bit more efficient, the multiplication takes approximately 2 seconds for the test (80 000 elements). The results seem to be essentially the same.
A few comments on the performance.
There are two operations performed for each output element:
set intersection
multiplication
The complexity of set intersection should be O(min(m,n)) where m and n are the numbers of non-zeros in each operand. This is invariant of the size of the matrix, only the average number of non-zeros per row/column is important.
The number of multiplications (and dict lookups) depends on the number of non-zeros found in the intersection above.
If both matrices have randomly distributed non-zeros with probability (density) p, and the row/column length is n, then:
set intersection: O(np)
dictionary lookup, multiplication: O(np^2)
This shows that with really sparse matrices finding the intersections is the critical point. This can also be verified by profiling; most of the time is spent calculating the intersections.
When this is reflected to the real world, we seem to spend around 20 us for a row/column of 80 non-zeros. This is not blindingly fast, and the code can certainly be made faster. Cython may be one solution, but this may be one of the problems where Python is not the best possible solution. A simple linear matching (merge sort -type algorithm) for sorted integers should be at least an order of magnitude faster when written in C.
One important thing to note is that the algorithm can be done in parallel for several elements at a time. There is no need to settle for a single thread, as the calculations are independent as far as one thread handles one output point.