Related
I've got an optimization problem in which I need to minimize the sum product of two uneven but consecutive arrays, say:
A = [1, 2, 3]
B = [4, 9, 5, 3, 2, 10]
Shuffling of values is not allowed i.e. the index of the arrays must remain the same.
In other words, it is a distribution minimization of array A over array B in consecutive order.
Or: Given that len(B)>=len(A) Minimize the sum product the values of Array A of length n over n values of array B without changing the order of array A or B.
In this case, the minimum would be:
min_sum = 1*4 + 2*3 + 3*2 = 16
A brute force approach to this problem would be:
from itertools import combinations
sums = [sum(a*b for a,b in zip(A,b)) for b in combinations(B,len(A))]
min_sum = min(sums)
I need to do this for many sets of arrays however. I see a lot of overlap with the knapsack problem and I have the feeling that it should be solved with dynamic programming. I am stuck however in how to write an efficient algorithm to perform this.
Any help would be greatly appreciated!
Having two lists
A = [1, 2, 3]
B = [4, 9, 5, 3, 2, 10]
the optimal sum product can be found using:
min_sum = sum(a*b for a,b in zip(sorted(A), sorted(B)[:len(A)][::-1]))
In case A is always given sorted, this simplified version can be used:
min_sum = sum(a*b for a,b in zip(A, sorted(B)[:len(A)][::-1]))
The important part(s) to note:
You need factors of A sorted. sorted(A) will do this job, without modifying the original A (in contrast to A.sort()). In case A is already given sorted, this step can be left out.
You need the N lowest values from B, where N is the length of A. This can be done with sorted(B)[:len(A)]
In order to evaluate the minimal sum of products, you need to multiply the highest number of A with the lowest of B, the second hightst of A with the second lowest of B. That is why after getting the N lowest values of B the order gets reversed with [::-1]
Output
print(min_sum)
# 16
print(A)
# [1, 2, 3] <- The original list A is not modified
print(B)
# [4, 9, 5, 3, 2, 10] <- The original list B is not modified
With Python, you can easily sort and flip sets. The code you are looking for is
A, B = sorted(A), sorted(B)[:len(A)]
min_sum = sum([a*b for a,b in zip(A, B[::-1])])
You may need to get the values one by one from B, and keep the order of the list by having each value assigned to a key.
A = [1, 3, 2]
B = [4, 9, 5, 3, 2, 10]
#create a new dictionary with key value pairs of B array values
new_dict = {}
j=0
for k in B:
new_dict[j] = k
j+= 1
#create a new list of the smallest values in B up to length of array A
min_Bmany =[]
for lp in range(0,len(A)):
#get the smallest remaining value from dictionary new_dict
rmvky= min(zip(new_dict.values(), new_dict.keys()))
#append this item to minimums list
min_Bmany.append((rmvky[1],rmvky[0]))
#delete this key from the dictionary new_dict
del new_dict[rmvky[1]]
#sort the list by the keys(instead of the values)
min_Bmany.sort(key=lambda r: r[0])
#create list of only the values, but still in the same order as they are in original array
min_B =[]
for z in min_Bmany:
min_B.append(z[1])
print(A)
print(min_B)
ResultStr = ""
Result = 0
#Calculate the result
for s in range(0,len(A)):
ResultStr = ResultStr + str(A[s]) +"*" +str(min_B[s])+ " + "
Result = Result + A[s]*min_B[s]
print(ResultStr)
print("Result = ",Result)
The output will be as follows:
A = [1, 3, 2]
B = [4, 9, 5, 3, 2, 10]
1*4 + 3*3 + 2*2 +
Result = 17
Then change the A, and the output becomes:
A = [1, 2, 3]
B = [4, 9, 5, 3, 2, 10]
1*4 + 2*3 + 3*2 +
Result = 16
Not sure if this is helpful, but anyway.
This can be formulated as a mixed-integer programming (MIP) problem. Basically, an assignment problem with some side constraints.
min sum((i,j),x(i,j)*a(i)*b(j))
sum(j, x(i,j)) = 1 ∀i "each a(i) is assigned to exactly one b(j)"
sum(i, x(i,j)) ≤ 1 ∀j "each b(j) can be assigned to at most one a(i)"
v(i) = sum(j, j*x(i,j)) "position of each a(i) in b"
v(i) ≥ v(i-1)+1 ∀i>1 "maintain ordering"
x(i,j) ∈ {0,1} "binary variable"
v(i) ≥ 1 "continuous (or integer) variable"
Example output:
---- 40 VARIABLE z.L = 16.000
---- 40 VARIABLE x.L assign
j1 j4 j5
i1 1.000
i2 1.000
i3 1.000
---- 40 VARIABLE v.L position of a(i) in b
i1 1.000, i2 4.000, i3 5.000
Cute little MIP model.
Just as an experiment I generated a random problem with len(a)=50 and len(b)=500. This leads to a MIP with 650 rows and 25k columns. Solved in 50 seconds (to proven global optimality) on my slow laptop.
It turns out using a shortest path algorithm on a direct graph is pretty fast. Erwin did a post showing a MIP model. As you can see in the comments section there, a few of us independently tried shortest path approaches, and on examples with 100 for the length of A and 1000 for the length of B we get optimal solutions in the vicinity of 4 seconds.
The graph can look like:
Nodes are labeled n(i,j) indicating that visiting the node means assigning a(i) to b(j). The costs a(i)*b(j) can be associated with any incoming (or any outgoing) arc. After that calculate the shortest path from src to snk.
BTW can you tell a bit about the background of this problem?
I have two arrays A, B, filled with n integers. I want to find a way to implement this summation:
Σ(k=2 to n-2) (B[k] * A[n-k])
but considering that I have to use this summation in a for loop that costs O(n).
The problem is to find a way to re-use the previous result of the summation to save it in a variable and don't have to sum all the values in every loop.
I add the values in the two arrays:
[32, 164, 752, 3348, ...]
[10, 18, 38, 84, ...]
The values in A are filled thanks to this formula, so I can't use the summation in the xth iteration without fill the x-1 position of A.
You can try something like this:
A = [1, 2, 3, 4, 5, 6, 7]
B = [1, 2, 3, 4, 5, 6, 7]
sum(a * b for a, b in zip(A[-3: 1: -1], B[2: -2]))
A[-3: 1: -1] flips list A and does not take into account the 2 first and 2 last elements.
B[2: -2] does not take into account the 2 first and 2 last elements.
It will sum 5*3 + 4*4 + 3*5 and gives 46.
The simple solution is to compute that sum before you start your other loop and store the result in a variable, e.g.
def my_sum(a, b):
summation = 0
for k in range(2, n-1):
summation += b[k] * A[n-k]
return summation
# and in your main code
a = [1, 2, 3, 4, 5, 6]
b = [4, 5, 6, 7, 8, 9]
c = my_sum(a, b)
for n in range(1000):
do_something_with(c)
Now, none of that code is very pythonic, but I presume you are starting out, so aim for working code first, then fast or beautiful code.
If you want it to be more efficient in practice, you should have a look into numpy, which provides faster operations than just using basic lists like what I show here. The function you are probably looking for is called convolve (and the operation you are looking for is convolution).
Do note that this kind of operation (or at least the direct implementation) is inherently O(n^2), so you would need a better algorithm to get below that kind of behavior. The numpy docs already hint toward a faster algorithm using the Fast Fourier Transform. I'll leave that one up to you to look into :-)
You can try this :
def somme(A,B):
n = len(B)
somme = 0
for i in range(2, n-1):
somme = somme + ( B[i] * A[n-i] )
return somme
You can use a numpy array. It doesn't require creating the list of pruducts before the sum.
import numpy as np
A=[1,2,3,4,5,6,7,8]
B =[9,10,11,12,13,14,15,16]
n = len(A)
k0 = 2 # start offset
n0 = 2 # end offset
A1 = np.array(A[k0 : n-n0])
B1 = np.array(B[-1-n0:-n+k0-1:-1])
answer = (A1*B1).sum()
I have three NumPy arrays of ints, same number of columns, arbitrary number of rows each. I am interested in all instances where a row of the first one plus a row of the second one gives a row of the third one ([3, 1, 4] + [1, 5, 9] = [4, 6, 13]).
Here is a pseudo-code:
for i, j in rows(array1), rows(array2):
if i + j is in rows(array3):
somehow store the rows this occured at (eg. (1,2,5) if 1st row of
array1 + 2nd row of array2 give 5th row of array3)
I will need to run this for very big matrices so I have two questions:
(1) I can write the above using nested loops but is there a quicker way, perhaps list comprehensions or itertools?
(2) What is the fastest/most memory-efficient way to store the triples? Later I will need to create a heatmap using two as coordinates and the first one as the corresponding value eg. point (2,5) has value 1 in the pseudo-code example.
Would be very grateful for any tips - I know this sounds quite simple but it needs to run fast and I have very little experience with optimization.
edit: My ugly code was requested in comments
import numpy as np
#random arrays
A = np.array([[-1,0],[0,-1],[4,1], [-1,2]])
B = np.array([[1,2],[0,3],[3,1]])
C = np.array([[0,2],[2,3]])
#triples stored as numbers with 2 coordinates in a otherwise-zero matrix
output_matrix = np.zeros((B.shape[0], C.shape[0]), dtype = int)
for i in range(A.shape[0]):
for j in range(B.shape[0]):
for k in range(C.shape[0]):
if np.array_equal((A[i,] + B[j,]), C[k,]):
output_matrix[j, k] = i+1
print(output_matrix)
We can leverage broadcasting to perform all those summations and comparison in a vectorized manner and then use np.where on it to get the indices corresponding to the matching ones and finally index and assign -
output_matrix = np.zeros((B.shape[0], C.shape[0]), dtype = int)
mask = ((A[:,None,None,:] + B[None,:,None,:]) == C).all(-1)
I,J,K = np.where(mask)
output_matrix[J,K] = I+1
(1) Improvements
You can use sets for the final result in the third matrix, as a + b = c must hold identically. This already replaces one nested loop with a constant-time lookup. I will show you an example of how to do this below, but we first ought to introduce some notation.
For a set-based approach to work, we need a hashable type. Lists will thus not work, but a tuple will: it is an ordered, immutable structure. There is, however, a problem: tuple addition is defined as appending, that is,
(0, 1) + (1, 0) = (0, 1, 1, 0).
This will not do for our use-case: we need element-wise addition. As such, we subclass the built-in tuple as follows,
class AdditionTuple(tuple):
def __add__(self, other):
"""
Element-wise addition.
"""
if len(self) != len(other):
raise ValueError("Undefined behaviour!")
return AdditionTuple(self[idx] + other[idx]
for idx in range(len(self)))
Where we override the default behaviour of __add__. Now that we have a data-type amenable to our problem, let's prepare the data.
You give us,
A = [[-1, 0], [0, -1], [4, 1], [-1, 2]]
B = [[1, 2], [0, 3], [3, 1]]
C = [[0, 2], [2, 3]]
To work with. I say,
from types import SimpleNamespace
A = [AdditionTuple(item) for item in A]
B = [AdditionTuple(item) for item in B]
C = {tuple(item): SimpleNamespace(idx=idx, values=[])
for idx, item in enumerate(C)}
That is, we modify A and B to use our new data-type, and turn C into a dictionary which supports (amortised) O(1) look-up times.
We can now do the following, eliminating one loop altogether,
from itertools import product
for a, b in product(enumerate(A), enumerate(B)):
idx_a, a_i = a
idx_b, b_j = b
if a_i + b_j in C: # a_i + b_j == c_k, identically
C[a_i + b_j].values.append((idx_a, idx_b))
Then,
>>>print(C)
{(2, 3): namespace(idx=1, values=[(3, 2)]), (0, 2): namespace(idx=0, values=[(0, 0), (1, 1)])}
Where for each value in C, you get the index of that value (as idx), and a list of tuples of (idx_a, idx_b) whose elements of A and B together sum to the value at idx in C.
Let us briefly analyse the complexity of this algorithm. Redefining the lists A, B, and C as above is linear in the length of the lists. Iterating over A and B is of course in O(|A| * |B|), and the nested condition computes the element-wise addition of the tuples: this is linear in the length of the tuples themselves, which we shall denote k. The whole algorithm then runs in O(k * |A| * |B|).
This is a substantial improvement over your current O(k * |A| * |B| * |C|) algorithm.
(2) Matrix plotting
Use a dok_matrix, a sparse SciPy matrix representation. Then you can use any heatmap-plotting library you like on the matrix, e.g. Seaborn's heatmap.
I have a list of n arrays with 4 elements each, i.e (n=2):
l = [[1, 2, 3, 4], [5, 6, 7, 8]]
and am trying to find all elements of the list that are 'non-dominated' - that is they are not dominated by any other element in the list. An array dominates another array if each item inside it is less than or equal to the corresponding item in the other array. So
dominates([1, 2, 3, 4], [5, 6, 7, 8]) == True
as 1 <= 5 and 2 <= 6 and 3 <= 7 and 4 <= 8. But
dominates([1, 2, 3, 9], [5, 6, 7, 8]) == False
as 9 > 8. This function is relatively easy to write, for example:
def dominates(a, b):
return all(i <= j for i, j in zip(a, b))
More succinctly, given l = [a1, a2, a3, .., an] where the a are length 4 arrays, I'm looking to find all a that are not dominated by any other a in l.
I have the following solution:
def get_non_dominated(l):
to_remove = set()
for ind, item_1 in enumerate(l):
if item_2 in to_remove:
continue
for item_2 in l[ind + 1:]:
if dominates(item_2, item_1):
to_remove.add(item_1)
break
elif dominates(item_1, item_2):
to_remove.add(item_2)
return [i for i in l if i not in to_remove]
So get_non_dominated([[1, 2, 3, 4], [5, 6, 7, 8]]) should return [[1, 2, 3, 4]]. Similarly get_non_dominated([[1, 2, 3, 9], [5, 6, 7, 8]]) should return the list unchanged by the logic above (nothing dominates anything else).
But this check happens a lot and l is potentially quite large. I was wondering if anyone had ideas on a way to speed this up? My first thought was to try and vectorize this code with numpy, but I have relatively little experience with it and am struggling a bit. You can assume l has all unique arrays. Any ideas are greatly appreciated!
Another version of #Nyps answer:
def dominates(a, b):
return (np.asarray(a) <= b).all()
It is the vectorized approach of your code using numpy.
This might still be slow if you have to loop through all the rows you have. If you have a list with all the rows and you want to compare them pairwise, you could use scipy to create a N x N array (where N is the number of rows).
import numpy as np
a = np.random.randint(0, 10, size=(1000, 10))
a here is a 1000 x 10 array, simulating 1000 rows of 10 elements it:
from scipy.spatial.distance import cdist
X = cdist(a, a, metric=dominates).astype(np.bool)
X is now a 1000 x 1000 matrix containing the pairwise comparison between all the entries. This is, X[i, j] contains True if sample i dominates sample j or False otherwise.
You can now extract fancy results from X, such as the sample that dominates them all:
>>> a[50] = 0 # set a row to all 0s to fake a dominant row
>>> X = cdist(a, a, metric=dominates).astype(np.bool)
>>> non_dominated = np.where(X.all(axis=1))[0]
>>> non_dominated
array([50])
Sample at position 50 is the ruler if your population, you should watch it closely.
Now, if you want to preserve only the dominated you can do:
if non_dominated.size > 0:
return [a[i] for i in non_dominated]
else: # no one dominates every other
return a
As a recap:
import numpy as np
from scipy.spatial.distance import cdist
def get_ruler(a):
X = cdist(a, a, metric=dominates).astype(np.bool)
rulers = np.where(X.all(axis=1))[0]
if rulers.size > 0:
return [a[i] for i in rulers]
else: # no one dominates every other
return a
How about:
import numpy as np
np.all((np.asarry(l[1])-np.asarry(l[0]))>=0)
You can go a simliar way in case you are able to create your list as numpy array straight away, i.e. type(l) == np.ndarray. Then the syntax would be:
np.all(p[1])-p[0])>=0)
Say that I have 4 numpy arrays
[1,2,3]
[2,3,1]
[3,2,1]
[1,3,2]
In this case, I've determined [1,2,3] is the "minimum array" for my purposes, as it is one of two arrays with lowest value at index 0, and of those two arrays it has the the lowest index 1. If there were more arrays with similar values, I would need to compare the next index values, and so on.
How can I extract the array [1,2,3] in that same order from the pile?
How can I extend that to x arrays of size n?
Thanks
Using the python non-numpy .sort() or sorted() on a list of lists (not numpy arrays) automatically does this e.g.
a = [[1,2,3],[2,3,1],[3,2,1],[1,3,2]]
a.sort()
gives
[[1,2,3],[1,3,2],[2,3,1],[3,2,1]]
The numpy sort seems to only sort the subarrays recursively so it seems the best way would be to convert it to a python list first. Assuming you have an array of arrays you want to pick the minimum of you could get the minimum as
sorted(a.tolist())[0]
As someone pointed out you could also do min(a.tolist()) which uses the same type of comparisons as sort, and would be faster for large arrays (linear vs n log n asymptotic run time).
Here's an idea using numpy:
import numpy
a = numpy.array([[1,2,3],[2,3,1],[3,2,1],[1,3,2]])
col = 0
while a.shape[0] > 1:
b = numpy.argmin(a[:,col:], axis=1)
a = a[b == numpy.min(b)]
col += 1
print a
This checks column by column until only one row is left.
numpy's lexsort is close to what you want. It sorts on the last key first, but that's easy to get around:
>>> a = np.array([[1,2,3],[2,3,1],[3,2,1],[1,3,2]])
>>> order = np.lexsort(a[:, ::-1].T)
>>> order
array([0, 3, 1, 2])
>>> a[order]
array([[1, 2, 3],
[1, 3, 2],
[2, 3, 1],
[3, 2, 1]])