I have a set of N items that I want to split in K subsets of size n1, n2, ..., nk (with n1 + n2 + ... + nk = N)
I also have constraints on which item can belong to which subset.
For my problem, at least one solution always exist.
I'm looking to implement an algorithm in Python to generate (at least) one solution.
Exemple :
Possibilities :
Item\Subset
0
1
2
A
True
True
False
B
True
True
True
C
False
False
True
D
True
True
True
E
True
False
False
F
True
True
True
G
False
False
True
H
True
True
True
I
True
True
False
Sizes constraints : (3, 3, 3)
Possible solution : [0, 0, 2, 1, 0, 1, 2, 2, 1]
Implementation :
So far, I have tried brute force with success, but I now want to find a more optimized algorithm.
I was thinking about backtracking, but I'm not sure it is the right method, nor if my implementation is right :
import pandas as pd
import numpy as np
import string
def solve(possibilities, constraints_sizes):
solution = [None] * len(possibilities)
def extend_solution(position):
possible_subsets = [index for index, value in possibilities.iloc[position].iteritems() if value]
for subset in possible_subsets:
solution[position] = subset
unique, counts = np.unique([a for a in solution if a is not None], return_counts=True)
if all(length <= constraints_sizes[sub] for sub, length in zip(unique, counts)):
if position >= len(possibilities)-1 or extend_solution(position+1):
return solution
return None
return extend_solution(0)
if __name__ == '__main__':
constraints_sizes = [5, 5, 6]
possibilities = pd.DataFrame([[False, True, False],
[True, True, True],
[True, True, True],
[True, True, True],
[True, False, False],
[True, True, True],
[True, True, True],
[True, True, True],
[True, False, False],
[True, True, True],
[True, True, True],
[True, True, True],
[False, True, True],
[True, True, True],
[True, True, True],
[True, False, False]],
index=list(string.ascii_lowercase[:16]))
solution = solve(possibilities, constraints_sizes)
One possible expected solution : [1, 0, 0, 1, 0, 1, 1, 1, 0, 2, 2, 2, 2, 2, 2, 0]
Unfortunately, this code fails to find a solution (eventhough it works with the previous example).
What am I missing ?
Thank you very much.
This problem can be solved by setting up a bipartite flow network with Items on one side, Subsets on the other, a surplus of 1 at each Item, a deficit of (Subset's size) at each Subset, and arcs of capacity 1 from each Item to each Subset to which it can belong. Then you need a maximum flow on this network; OR-Tools can do this, but you have a lot of options.
#David Eisenstat mentioned OR-Tools as a package to solve this kind of problem.
Thanks to him, I've found out that this problem could match one of their example, an Assignement with Task Sizes problem
It matches my understanding of the problem better than what I understood from the suggested "Flow network" concept, but I'd be happy to discuss about that.
Here is the solution I implemented, based on their example :
from ortools.sat.python import cp_model
def solve(possibilities, constraint_sizes):
# Transform possibilities into costs (0 if possible, 1 otherwise)
costs = [[int(not row[subset]) for row in possibilities] for subset in range(len(possibilities[0]))]
num_subsets = len(costs)
num_items = len(costs[0])
model = cp_model.CpModel()
# Variables
x = {}
for subset in range(num_subsets):
for item in range(num_items):
x[subset, item] = model.NewBoolVar(f'x[{subset},{item}]')
# Constraints :
# Each subset should should contain a given number of item
for subset, size in zip(range(num_subsets), constraint_sizes):
model.Add(sum(x[subset, item] for item in range(num_items)) <= size)
# Each item is assigned to exactly one subset
for item in range(num_items):
model.Add(sum(x[subset, item] for subset in range(num_subsets)) == 1)
# Objective
objective_terms = []
for subset in range(num_subsets):
for item in range(num_items):
objective_terms.append(costs[subset][item] * x[subset, item])
model.Minimize(sum(objective_terms))
# Solve
solver = cp_model.CpSolver()
status = solver.Solve(model)
if status == cp_model.OPTIMAL or status == cp_model.FEASIBLE:
solution = []
for item in range(num_items):
for subset in range(num_subsets):
if solver.BooleanValue(x[subset, item]):
solution.append(subset)
return solution
return None
The trick here is to tranform the possibilities into costs (0 only if possible), and to optimize the total cost.
An acceptable solution should then have a 0 total cost.
It gives a right solution, for the previous problem :
possibilities = [[False, True, False],
[True, True, True],
[True, True, True],
[True, True, True],
[True, False, False],
[True, True, True],
[True, True, True],
[True, True, True],
[True, False, False],
[True, True, True],
True, True, True],
[True, True, True],
[False, True, True],
[True, True, True],
[True, True, True],
[True, False, False]]
constraint_sizes = [5, 5, 6]
solution = solver(possibilities, constraint_sizes)
print(solution) # [1, 2, 1, 0, 0, 0, 2, 1, 0, 1, 2, 2, 2, 2, 1, 0]
I have now two more questions :
Can we transform the optimization objective (minimize the cost) into a hard constraint (cost should equal to 0) ? I guess it could lower the computing time.
How can I get other solutions and not only one ?
I am also still looking for a plain Python solution without any library...
Thank you
Related
Suppose I have a 2d boolean array with shape (nrows,ncols). I'm trying to efficiently extract the indices of the topmost True value for each column in the array. If the column has all False values, then no indices are returned for that column. Below is an example of a boolean array with shape (4,6) where the indices of the bold Trues would be the desired output.
False False False False False False
True False False True False False
True False True False False True
True False True True False False
Desired output of indices (row,col): [(1,0),(2,2),(1,3),(2,5)]
I tried using numpy.where and also an implementation of the skyline algorithm but both options are slow. Is there a more efficient way to solve this problem?
Thank you in advance for your help.
You can use np.argmax to detect the first True values.
Prepare the example array.
import numpy as np
a = np.array(
[[0,0,0,0,0,0],
[1,0,0,1,0,0],
[1,0,1,0,0,1],
[1,0,1,1,0,0]]).astype('bool')
a
Output
array([[False, False, False, False, False, False],
[ True, False, False, True, False, False],
[ True, False, True, False, False, True],
[ True, False, True, True, False, False]])
Stack one row of False to deal with columns without a True. Find first True in every column with np.argmax and append an arange for the row indices. You have to adjust the column indices by -1 because we added one row to the array. Then select the columns where the True's index was greater than 0
b = np.vstack([np.zeros_like(a[0]),a])
t = b.argmax(axis=0)
np.vstack([t - 1, np.arange(len(a[0]))]).T[t > 0]
Output
array([[1, 0],
[2, 2],
[1, 3],
[2, 5]])
Translating #HenryYik answer to numpy gives a one line solution
np.vstack([a.argmax(axis=0), np.arange(len(a[0]))]).T[a.sum(0) > 0]
Output
array([[1, 0],
[2, 2],
[1, 3],
[2, 5]])
If you are open to using pandas, you can construct a df, drop columns with False only and then idxmax:
arr = [[False, False, False, False, False, False],
[True, False, False, True, False, False],
[True, False, True, False, False, True],
[True, False, True, True, False, False]]
df = pd.DataFrame(arr, columns=range(len(arr[0])))
s = df.loc[:, df.sum()>0].idxmax()
print (s)
Result:
0 1
2 2
3 1
5 2
dtype: int64
Which is col value vs row value. You can convert it back to your desired form:
print (list(zip(s, s.index)))
[(1, 0), (2, 2), (1, 3), (2, 5)]
I suggest you try this:
def get_topmost(ar: np.ndarray):
return [(row.index(True), i) for i, row in enumerate(ar.T.tolist()) if True in row]
Example: (should works as is)
>>> test = np.array([
[False, False, False, False, False, False],
[True, False, False, True, False, False],
[True, False, True, False, False, True],
[True, False, True, True, False, False],
])
>>> print(get_topmost(test))
[(1, 0), (2, 2), (1, 3), (2, 5)]
I have the following array:
[(True,False,True), (False,False,False), (False,False,True)]
If any element contains a True then they should all be true. So the above should become:
[(True,True,True), (False,False,False), (True,True,True)]
My below code attempts to do that but it simply converts all elements to True:
a = np.array([(True,False,True), (False,False,False), (False,True,False)], dtype='bool')
aint = a.astype('int')
print(aint)
aint[aint.sum() > 0] = (1,1,1)
print(aint.astype('bool'))
The output is:
[[1 0 1]
[0 0 0]
[0 1 0]]
[[ True True True]
[ True True True]
[ True True True]]
You could try np.any, which tests whether any array element along a given axis evaluates to True.
Here's a quick line of code that uses a list comprehension to get your intended result.
lst = [(True,False,True), (False,False,False), (False,False,True)]
result = [(np.any(x),) * len(x) for x in lst]
# result is [(True, True, True), (False, False, False), (True, True, True)]
I'm no numpy wizard but this should return what you want.
import numpy as np
def switch(arr):
if np.any(arr):
return np.ones(*arr.shape).astype(bool)
return arr.astype(bool)
np.apply_along_axis(switch, 1, a)
array([[ True, True, True],
[False, False, False],
[ True, True, True]])
ndarray.any along axis=1 and np.tile will get job done
np.tile(a.any(1)[:,None], a.shape[1])
array([[ True, True, True],
[False, False, False],
[ True, True, True]])
Create an array of True's based on the original array's second dimension and assign it to all rows that have a True in it.
>>> a
array([[ True, False, True],
[False, False, False],
[False, True, False]])
>>> a[a.any(1)] = np.ones(a.shape[1], dtype=bool)
>>> a
array([[ True, True, True],
[False, False, False],
[ True, True, True]])
>>>
Relies on Broadcasting.
I was debugging a code written by someone who has left the organization and came across a line, which uses np.less_equal.outer & np.greater_equal.outer functions. I know that np.outer creates a Cartesian cross product of two 1-dimensional arrays and creates two arrays, and np.less_equal compares the element of two arrays and returns true or false. Can someone please explain how this combined form works.
Thanks!
less_equal and greater_equal are special types of numpy functions called ufuncs, in that they have extendible functionalities, including accumulate, at, and outer.
In this case ufunc.outer extends the function to work similarly to the outer product - but while the actual outer product would be multiply.outer, this instead does the greater or less than comparison.
So you get a 2d array of booleans corresponding to each element of the first array, and whether they are greater or less than each of the elements in the second array.
np.less_equal.outer(range(1,18),range(1,13))
Out[]:
array([[ True, True, True, ..., True, True, True],
[False, True, True, ..., True, True, True],
[False, False, True, ..., True, True, True],
...,
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False],
[False, False, False, ..., False, False, False]], dtype=bool)
EDIT: a much more pythonic way of doing this would be:
np.triu(np.ones((18, 13), dtype = bool), 0)
That is, the upper triangle of a boolean array of shape (18, 13)
From the documentation, we have that for one-dimensional arrays A and B, the operation np.less_equal.outer(A, B) is equivalent to:
m = len(A)
n = len(B)
r = empty(m, n)
for i in range(m):
for j in range(n):
r[i,j] = (A[i] <= B[j])
Here's the mathematical representation of the result:
here is an example:
np.less_equal([4, 2, 1], [2, 2, 2])
array([False, True, True])
np.greater_equal([4, 2, 1], [2, 2, 2])
array([ True, True, False], dtype=bool)
and first the outer function
np.outer(range(1,2), range(1,3))
array([[1 2 3],
[2 4 6],
)
hope that helps.
Im trying to delete specific rows in my numpy array that following certain conditions.
This is an example:
a = np.array ([[1,1,0,0,1],
[0,0,1,1,1],
[0,1,0,1,1],
[1,0,1,0,1],
[0,0,1,0,1],
[1,0,1,0,0]])
I want to able to delete all rows, where specific columns are zero, this array could be a lot bigger.
In this example, if first two element are zero, or if last two elements are zero, the rows will be deleted.
It could be any combination, no only first element or last ones.
This should be the final:
a = np.array ([[1,1,0,0,1],
[0,1,0,1,1],
[1,0,1,0,1]])
For example If I try:
a[:,0:2] == 0
After reading:
Remove lines with empty values from multidimensional-array in php
and this question: How to delete specific rows from a numpy array using a condition?
But they don't seem to apply to my case, or probably I'm not understanding something here as nothing works my case.
This gives me all rows there the first two cases are zero, True, True
array([[False, False],
[ True, True],
[ True, False],
[False, True],
[ True, True],
[False, True]])
and for the last two columns being zero, the last row should be deleted too. So at the end I will only be left with 2 rows.
a[:,3:5] == 0
array([[ True, False],
[False, False],
[False, False],
[ True, False],
[ True, False],
[ True, True]])
Im trying something like this, but I don't understand now how to tell it to only give me the rows that follow the condition, although this only :
(a[a[:,0:2]] == 0).all(axis=1)
array([[ True, True, False, False, False],
[False, False, True, True, False],
[False, False, False, False, False],
[False, False, False, False, False],
[False, False, True, True, False],
[False, False, False, False, False]])
(a[((a[:,0])& (a[:,1])) ] == 0).all(axis=1)
and this shows everything as False
could you please guide me a bit?
thank you
Just adding in the question, that the case it wont always be the first 2 or the last 2. If my matrix has 35 columns, it could be the column 6th to 10th, and then column 20th and 25th. An user will be able to decide which columns they want to get deleted.
Try this
idx0 = (a[:,0:2] == 0).all(axis=1)
idx1 = (a[:,-2:] == 0).all(axis=1)
a[~(idx0 | idx1)]
The first two steps select the indices of the rows that match your filtering criteria. Then do an or (|) operation, and the not (~) operation to obtain the final indices you want.
If I understood correctly you could do something like this:
import numpy as np
a = np.array([[1, 1, 0, 0, 1],
[0, 0, 1, 1, 1],
[0, 1, 0, 1, 1],
[1, 0, 1, 0, 1],
[0, 0, 1, 0, 1],
[1, 0, 1, 0, 0]])
left = np.count_nonzero(a[:, :2], axis=1) != 0
a = a[left]
right = np.count_nonzero(a[:, -2:], axis=1) != 0
a = a[right]
print(a)
Output
[[1 1 0 0 1]
[0 1 0 1 1]
[1 0 1 0 1]]
Or, a shorter version:
left = np.count_nonzero(a[:, :2], axis=1) != 0
right = np.count_nonzero(a[:, -2:], axis=1) != 0
a = a[(left & right)]
Use the following mask:
[np.any(a[:,:2], axis=1) & np.any(a[:,:-2], axis=1)]
if you want to create a filtered view:
a[np.any(a[:,:2], axis=1) & np.any(a[:,:-2], axis=1)]
if you want to create a new array:
np.delete(a,np.where(~(np.any(a[:,:2], axis=1) & np.any(a[:,:-2], axis=1))), axis=0)
I am new to Python. I want to do following.
Input: A list of integers of size n. Each integer is in a range of 0 to 3.
Output: A multi-column (4 column in this case as integer range in 0-3 = 4) numpy list of size n. Each row of the new list will have the column corresponding to the integer value of Input list as True and rest of the columns as False.
E.g. Input list : [0, 3, 2, 1, 1, 2], size = 6, Each integer is in range of 0-3
Output list :
Row 0: True False False False
Row 1: False False False True
Row 2: False False True False
Row 3: False True False False
Row 4: False True False False
Row 5: False False True False
Now, I can start with 4 columns. Traverse through the input list and create this as follows,
output_columns[].
for i in Input list:
output_column[i] = True
Create an output numpy list with output columns
Is this the best way to do this in Python? Especially for creating numpy list as an output.
If yes, How do I merge output_columns[] at the end to create numpy multidimensional list with each dimension as a column of output_columns.
If not, what would be the best (most time efficient way) to do this in Python?
Thank you,
Is this the best way to do this in Python?
No, a more Pythonic and probably the best way is to use a simple broadcasting comparison as following:
In [196]: a = np.array([0, 3, 2, 1, 1, 2])
In [197]: r = list(range(0, 4))
In [198]: a[:,None] == r
Out[198]:
array([[ True, False, False, False],
[False, False, False, True],
[False, False, True, False],
[False, True, False, False],
[False, True, False, False],
[False, False, True, False]])
You are creating so called one-hot vector (each row in matrix is a one-hot vector meaning that only one value is True).
mylist = [0, 3, 2, 1, 1, 2]
one_hot = np.zeros((len(mylist), 4), dtype=np.bool)
for i, v in enumerate(mylist):
one_hot[i, v] = True
Output
array([[ True, False, False, False],
[False, False, False, True],
[False, False, True, False],
[False, True, False, False],
[False, True, False, False],
[False, False, True, False]], dtype=bool)