I have CVXPY problem defined with a volume array, and a cost array to match each of volumes. The problem has 192 variables and 3 constraints which I have defined.
My goal is to minimize the cost in this problem to deliver a specific volume and avoid multiple periods where I get a 0, 1, 0, 1.
My current output could look something like follows:
[0, 0, 1, 1, 0, 1... 0, 1, 0, 1]
The ideal solution would avoid an amount. So if the selection decides a 1 at a point, the next 2 points should be 0. Such as below:
[0, 0, 1, 1, 0, 0... 0, 1, 0, 0]
I am unsure how to write such a constraint to include my selection with the problem I have currently programmed as can be seen here:
import cvxpy as cp
import numpy as np
# Volume and cost
full_cost = [[0, data] for data in [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45,0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.45, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4]]
cost_ = np.array(full_cost)
ex = np.array([[0, 17100] for data in [i for i in range(0, 96)]])
# Minimum volume required
v_min = 300000
# Selection variable
selection = cp.Variable(shape=ex.shape, boolean=True)
# Constraints
assignment_constraint = cp.sum(selection,axis=1) == 1
volume_= cp.sum(cp.multiply(ex,selection))
volume_constraint = volume_ >= v_min
cost_constraint = cp.sum(cp.multiply(cost_, selection))
constraints = [assignment_constraint, volume_constraint, cost_constraint]
cost_ = cp.sum(cp.multiply(cost_,selection))
# Problem definition
assign_problem = cp.Problem(cp.Minimize(cost_), constraints)
assign_problem.solve(solver=cp.CPLEX, verbose=True)
# Find solution in ex variable
assignments = [np.where(r==1)[0][0] for r in selection.value]
c = [ ex[i][assignments[i]] for i in range(len(assignments)) ]
best_volume = np.sum(np.multiply(ex,selection.value))
best_cost = np.sum(np.multiply(cost_,selection.value))
print(best_cost)
print(c)
I believe that the constraint should be based around my selection variable, but I am struggling to see how to include it as a constraint.
If I understand correctly the it looks like you want to impose the condition
if x[i]==1 and x[i+1]==0 then x[i+2]==0
for binary variables x. That is equivalent to
x[i+2] <= 1 - x[i] + x[i+1]
I'm looking for a similar function to tf.unsorted_segment_sum, but I don't want to sum the segments, I want to get every segment as a tensor.
So for example, I have this code:
(In real, I have a tensor with shapes of (10000, 63), and the number of segments would be 2500)
to_be_sliced = tf.constant([[0.1, 0.2, 0.3, 0.4, 0.5],
[0.3, 0.2, 0.2, 0.6, 0.3],
[0.9, 0.8, 0.7, 0.6, 0.5],
[2.0, 2.0, 2.0, 2.0, 2.0]])
indices = tf.constant([0, 2, 0, 1])
num_segments = 3
tf.unsorted_segment_sum(to_be_sliced, indices, num_segments)
The output would be here
array([sum(row1+row3), row4, row2]
What I am looking for is 3 tensor with different shapes (maybe a list of tensors), first containing the first and third rows of the original (shape of (2, 5)), the second contains the 4th row (shape of (1, 5)), the third contains the second row, like this:
[array([[0.1, 0.2, 0.3, 0.4, 0.5],
[0.9, 0.8, 0.7, 0.6, 0.5]]),
array([[2.0, 2.0, 2.0, 2.0, 2.0]]),
array([[0.3, 0.2, 0.2, 0.6, 0.3]])]
Thanks in advance!
You can do that like this:
import tensorflow as tf
to_be_sliced = tf.constant([[0.1, 0.2, 0.3, 0.4, 0.5],
[0.3, 0.2, 0.2, 0.6, 0.3],
[0.9, 0.8, 0.7, 0.6, 0.5],
[2.0, 2.0, 2.0, 2.0, 2.0]])
indices = tf.constant([0, 2, 0, 1])
num_segments = 3
result = [tf.boolean_mask(to_be_sliced, tf.equal(indices, i)) for i in range(num_segments)]
with tf.Session() as sess:
print(*sess.run(result), sep='\n')
Output:
[[0.1 0.2 0.3 0.4 0.5]
[0.9 0.8 0.7 0.6 0.5]]
[[2. 2. 2. 2. 2.]]
[[0.3 0.2 0.2 0.6 0.3]]
For your case, you can do Numpy slicing in Tensorflow. So this will work:
sliced_1 = to_be_sliced[:3, :]
# [[0.4 0.5 0.5 0.7 0.8]
# [0.3 0.2 0.2 0.6 0.3]
# [0.3 0.2 0.2 0.6 0.3]]
sliced_2 = to_be_sliced[3, :]
# [0.3 0.2 0.2 0.6 0.3]
Or a more general option, you can do it in the following way:
to_be_sliced = tf.constant([[0.1, 0.2, 0.3, 0.4, 0.5],
[0.3, 0.2, 0.2, 0.6, 0.3],
[0.9, 0.8, 0.7, 0.6, 0.5],
[2.0, 2.0, 2.0, 2.0, 2.0]])
first_tensor = tf.gather_nd(to_be_sliced, [[0], [2]])
second_tensor = tf.gather_nd(to_be_sliced, [[3]])
third_tensor = tf.gather_nd(to_be_sliced, [[1]])
concat = tf.concat([first_tensor, second_tensor, third_tensor], axis=0)
I have a dataframe, like so,
import pandas as pd
import numpy as np
df = pd.DataFrame({'a': [0, 0.5, 0.2],
'b': [1,1,0.3]})
print (df)
a b
0 0.0 1.0
1 0.5 1.0
2 0.2 0.3
I want to generate a Series that looks like
pd.Series ([np.arange ( start = 0, stop = 1, step = 0.1),
np.arange ( start = 0.5, stop = 1, step = 0.1),
np.arange ( start = 0.2, stop = 0.3, step = 0.1)])
0 [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, ...
1 [0.5, 0.6, 0.7, 0.8, 0.9]
2 [0.2]
dtype: object
I am trying to do this with a lambda function and getting an error, like so
foo = lambda x: np.arange(start = x.a, stop = x.b, step = 0.1)
print (df.apply(foo, axis =1))
ValueError: Shape of passed values is (3, 10), indices imply (3, 2)
I am not sure what this means. Is there a better/correct way to do this?
I'd use a comprehension
pd.Series([np.arange(a, b, .1) for a, b in zip(df.a, df.b)], df.index)
0 [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, ...
1 [0.5, 0.6, 0.7, 0.8, 0.9]
2 [0.2]
dtype: object
Use itertuples with Series constructor:
s = pd.Series([np.arange(x.a, x.b, .1) for x in df.itertuples()], index=df.index)
print (s)
0 [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, ...
1 [0.5, 0.6, 0.7, 0.8, 0.9]
2 [0.2]
dtype: object
s = pd.Series([np.arange(x.a, x.b, .1) for i, x in df.iterrows()], index=df.index)
print (s)
0 [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, ...
1 [0.5, 0.6, 0.7, 0.8, 0.9]
2 [0.2]
dtype: object
With apply works only converting to tuple:
foo = lambda x: tuple(np.arange(start = x.a, stop = x.b, step = 0.1))
print (df.apply(foo, axis = 1))
0 (0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, ...
1 (0.5, 0.6, 0.7, 0.8, 0.9)
2 (0.2,)
dtype: object
I have a list of lists like this:
[[12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.1, 0.15, 0.2, 0.1, 0.15, 0.15, 0.15, 0.15], [12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]], etc.]
If the first and second element of an inner list is the same as the first and second element of another inner list (like the example above), I want to create a function that adds the remaining values and merges them into one list. The example output would be like this:
[12411.0, 31937, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.25, 0.2, 0.25, 0.3, 0.2, 0.25, 0.25, 0.25, 0.25]
I'm having trouble how to tell Python to initially recognize and compare the two elements of the list before merging them together. Here is my best attempt so far:
def group(A):
for i in range(len(A)):
for j in range(len(A[i])):
if A[i][0:1] == A[i: ][0:1]:
return [A[i][0], A[i][1], sum(A[i][j+2], A[i: ][j+2])]
I get an index error, I believe, because of the A[i: ] and A[i: ][j+2] parts of the code. I don't know how to phrase it though in Python to tell the function to add any other lines that meet the criteria.
Here's a function that will merge all sublists where the first two entries match. It also handles cases where the sub-lists are not the same length:
from itertools import izip_longest
l = [[1,3,4,5,6], [1,3,2,2,2], [2,3,5,6,6], [1,1,1,1,1], [1,1,2,2,2], [1,3,6,2,1,1,2]]
l2 = [[12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.1, 0.15, 0.2, 0.1, 0.15, 0.15, 0.15, 0.15], [12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]]
def merge(l):
d = {}
for ent in l:
key = tuple(ent[0:2])
merged = d.get(key, None)
if merged is None:
d[key] = ent
else:
merged[2:] = [a+b for a,b in izip_longest(merged[2:], ent[2:], fillvalue=0)]
return d.values()
print merge(l)
print merge(l2)
Output:
[[1, 3, 12, 9, 9, 1, 2], [2, 3, 5, 6, 6], [1, 1, 3, 3, 3]]
[[12411.0, 31937.0, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.25, 0.2, 0.25, 0.30000000000000004, 0.2, 0.25, 0.25, 0.25, 0.25]]
It's implemented by maintaining a dict where the keys are the first two entries of a sub-list (stored as a tuple). As we iterate over the sublists, we check to see if there's an entry in the dict. If there isn't, we store the current sublist in the dict. If there already is an entry, we add up all their values from index 2 onward, and update the dict. Once we're one iterating, we just return all the values from the dict.
This is one way to do it:
>>> a_list = [[12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.1, 0.15, 0.2, 0.1, 0.15, 0.15, 0.15, 0.15], [12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]]
>>> result = [a + b for a, b in zip(*a_list)]
>>> result[:2] = a_list[0][:2]
>>> result
[12411.0, 31937.0, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.25, 0.2, 0.25, 0.30000000000000004, 0.2, 0.25, 0.25, 0.25, 0.25]
This works by blindly adding up corresponding elements in all the sub-lists by doing:
[a + b for a, b in zip(*a_list)]
And then rewriting the first two elements of the result which according to the question does not change, by doing:
result[:2] = a_list[0][:2]
It is not evident from your question, as to what should the behavior be if the first two elements of the sub lists do not match. But the following snippet will help you check if the first two elements of the sub lists match. Lets assume a_list contains sublists whose first two elements do not match:
>>> a_list = [[12411.0, 31937.0, 0.1, 0.1], [12411.3, 31937.0, 0.1, 0.1]]
then, this condition:
all([True if list(a)[1:] == list(a)[:-1] else False for a in list(zip(*a_list))[:2]])
will return False. True otherwise. The code extracts the first elements and second elements of all the sub lists and then checks if they are equal.
You can include the above check in your code and modify your code accordingly for the expected behavior.
To sum it up:
a_list = [[12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.15, 0.1, 0.15, 0.2, 0.1, 0.15, 0.15, 0.15, 0.15], [12411.0, 31937.0, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1]]
check = all([True if list(a)[1:] == list(a)[:-1] else False for a in list(zip(*a_list))[:2]])
result = []
if check:
result = [a + b for a, b in zip(*a_list)]
result[:2] = a_list[0][:2]
else:
# whatever the behavior should be.
This is a function that will take a list of lists A and check internal list i and j using your criteria. It will then either return the summed list you want or None if the first two elements don't match.
def check_internal_ij(A,i,j):
""" checks internal list i against internal list j """
if A[i][0:2] == A[j][0:2]:
new = [x+y for x,y in zip( A[i], A[j] )]
new[0:2] = A[i][0:2]
return new
else:
return None
Then you can run the function over all combinations of internal lists you want to check.
If you are fond of itertools with a little effort, this can easily be solved by playing around with groupby, islice, izip, imap and chain.
And off course you should also remember to use operator.itemgetter
Implementation
# Create a group of lists where the key (the first two elements of the lists) matches
groups = groupby(sorted(l, key = itemgetter(0, 1)), key = itemgetter(0, 1))
# zip the lists and then chop of the first two elements. Sum the elements of the resultant list
# Remember to add the newly accumulated list with the first two elements
groups_sum = ([k, imap(sum, islice(izip(*g), 2, None))] for k, g in groups )
# Reformat the final list to match the output format
[list(chain.from_iterable(elem)) for elem in groups_sum]
Implementation (If you are a fan of single liner)
[list(chain.from_iterable([k, imap(sum, islice(izip(*g), 2, None))]))
for k, g in groupby(sorted(l, key = itemgetter(0, 1)), key = itemgetter(0, 1))]
Sample Input
l = [[10,20,0.1,0.2,0.3,0.4],
[11,22,0.1,0.2,0.3,0.4],
[10,20,0.1,0.2,0.3,0.4],
[11,22,0.1,0.2,0.3,0.4],
[20,30,0.1,0.2,0.3,0.4],
[10,20,0.1,0.2,0.3,0.4]]
Sample Output
[[10, 20, 0.3, 0.6, 0.9, 1.2],
[11, 22, 0.2, 0.4, 0.6, 0.8],
[20, 30, 0.1, 0.2, 0.3, 0.4]]
Dissection
groups = groupby(sorted(l, key = itemgetter(0, 1)), key = itemgetter(0, 1))
# After grouping, similar lists gets clustered together
[((10, 20),
[[10, 20, 0.1, 0.2, 0.3, 0.4],
[10, 20, 0.1, 0.2, 0.3, 0.4],
[10, 20, 0.1, 0.2, 0.3, 0.4]]),
((11, 22), [[11, 22, 0.1, 0.2, 0.3, 0.4], [11, 22, 0.1, 0.2, 0.3, 0.4]]),
((20, 30), [[20, 30, 0.1, 0.2, 0.3, 0.4]])]
groups_sum = ([k, imap(sum, islice(izip(*g), 2, None))] for k, g in groups )
# Each group is accumulated from the second element onwards
[[(10, 20), [0.3, 0.6, 0.9, 1.2]],
[(11, 22), [0.2, 0.4, 0.6, 0.8]],
[(20, 30), [0.1, 0.2, 0.3, 0.4]]]
[list(chain.from_iterable(elem)) for elem in groups_sum]
# Now its just a matter of representing in the output format
[[10, 20, 0.3, 0.6, 0.9, 1.2],
[11, 22, 0.2, 0.4, 0.6, 0.8],
[20, 30, 0.1, 0.2, 0.3, 0.4]]