Related
I have two lists I and i. I want to find for each element of i, how many values are less than in I and add the total number of such values to the specific i element. For example, element 15 in i has two values less than itself in I i.e. [8,11]. So 2 should be added to 15 and the combination stored in Values. I present the expected output.
I = [8, 11, 19, 37, 40, 42]
i=[15, 17, 27, 28, 31, 41]
The expected output is
New i=[17,19,30,31,34,46]
Values=[[8,11],[8,11],[8,11,19],[8,11,19],[8,11,19],[8,11,19,37,40]]
Assuming your list I is sorted, you can use bisect_left to get insertion point in your list I for each element in i and then slice the list. It uses binary search.
With that you can do:
from bisect import bisect_left
Values = [I[:bisect_left(I, e)] for e in i]
New_i = [e + len(Values[j]) for j, e in enumerate(i)]
print(Values):
[[8, 11], [8, 11], [8, 11, 19], [8, 11, 19], [8, 11, 19], [8, 11, 19, 37, 40]]
print(New_i):
[17, 19, 30, 31, 34, 46]
BTW I highly recommend not to use I and i for your variable names.
You can use two list comprehensions
>>> # for each element in i, add the elements in I to a list if they are smaller
>>> values = [[e for e in I if e < n] for n in i]
>>> # add the element of i to the number of elements in I that are smaller
>>> new_i = [sum(x) for x in zip(i, map(len, values))]
>>> values
[[8, 11], [8, 11], [8, 11, 19], [8, 11, 19], [8, 11, 19], [8, 11, 19, 37, 40]]
>>> new_i
[17, 19, 30, 31, 34, 46]
the other solutions are 100% correct, here's another solution but oldschool and more readable:
l1 = [15, 17, 27, 28, 31, 41]
l2 = [8, 11, 19, 37, 40, 42]
comp = []
for i in l1:
c = []
for j in l2:
if i > j:
i += 1
c.append(j)
comp.append(c)
print(l1)
print(comp)
input >>
l1 = [15, 17, 27, 28, 31, 41]
l2 = [8, 11, 19, 37, 40, 42]
output >>
[15, 17, 27, 28, 31, 41]
[[8, 11], [8, 11], [8, 11, 19], [8, 11, 19], [8, 11, 19], [8, 11, 19, 37, 40, 42]]
One way to do this is to use numpy which allows quick operations on lists by putting them in matrix form and is more optimized than operations in lists. An example of code could be:
import numpy as np
list_1 = [8, 11, 19, 37, 40, 42]
list_2 = [15, 17, 27, 28, 31, 41]
arr_1, arr_2 = np.array(list_1), np.array(list_2)
# Broadcast to 2D array where elem (i, j) is whether list_2[i] < list_1[j] or not
arr_sup = (arr_2[:, None] - arr_1[None, :]) > 0
# Add sum of rows to list_2 to create the new list
new_list = (np.sum(arr_sup, axis=1) + list_2).tolist()
# Get indexes (i, j) where arr_sup[i, j] is True
idx_sup = np.where(arr_sup)
values = []
for i, j in zip(idx_sup[0], idx_sup[1]): # browse i and j together
if len(values) < i + 1:
# Add a new list for each new i
values.append([])
# Add value j of list 1 to i-th list in values
values[i].append(list_1[j])
print(new_list) # [17, 19, 30, 31, 34, 46]
print(values) # [[8, 11], [8, 11], [8, 11, 19], [8, 11, 19], [8, 11, 19], [8, 11, 19, 37, 40]]
It works even if the lists are not sorted.
Let's say I have the following (fictitious) NumPy array:
arr = np.array(
[[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16],
[17, 18, 19, 20],
[21, 22, 23, 24],
[25, 26, 27, 28],
[29, 30, 31, 32],
[33, 34, 35, 36],
[37, 38, 39, 40]
]
)
And for row indices idx = [0, 2, 3, 5, 8, 9] I'd like to repeat the values in each row downward until it reaches the next row index:
np.array(
[[1, 2, 3, 4],
[1, 2, 3, 4],
[9, 10, 11, 12],
[13, 14, 15, 16],
[13, 14, 15, 16],
[21, 22, 23, 24],
[21, 22, 23, 24],
[21, 22, 23, 24],
[33, 34, 35, 36],
[37, 38, 39, 40]
]
)
Note that idx will always be sorted and have no repeat values. While I can accomplish this by doing something like:
for start, stop in zip(idx[:-1], idx[1:]):
for i in range(start, stop):
arr[i] = arr[start]
# Handle last index in `idx`
start, stop = idx[-1], arr.shape[0]
for i in range(start, stop):
arr[i] = arr[start]
Unfortunately, I have many, many arrays like this and this can become slow as the size of the array gets larger (in both the number of rows as well as the number of columns) and the length of idx also increases. The final goal is to plot these as a heatmaps in matplotlib, which I already know how to do. Another approach that I tried was using np.tile:
for start, stop in zip(idx[:-1], idx[1:]):
reps = max(0, stop - start)
arr[start:stop] = np.tile(arr[start], (reps, 1))
# Handle last index in `idx`
start, stop = idx[-1], arr.shape[0]
arr[start:stop] = np.tile(arr[start], (reps, 1))
But I am hoping that there's a way to get rid of the slow for-loop.
Try np.diff to find the repetition for each row, then np.repeat:
# this assumes `idx` is a standard list as in the question
np.repeat(arr[idx], np.diff(idx+[len(arr)]), axis=0)
Output:
array([[ 1, 2, 3, 4],
[ 1, 2, 3, 4],
[ 9, 10, 11, 12],
[13, 14, 15, 16],
[13, 14, 15, 16],
[21, 22, 23, 24],
[21, 22, 23, 24],
[21, 22, 23, 24],
[33, 34, 35, 36],
[37, 38, 39, 40]])
Let's say there's an np array like this:
k = np.array([[13, 29],
[17, 18],
[19, 27]])
Now, I need to subtract 4 from column 1 and 8 from column 2 without looping with Numpy.
I tried k - 4 but it seems to subtract even from axis 1.
k - [4, 8]
Demo:
>>> >>> import numpy as np
>>> k = np.array([[13, 29],
[17, 18],
[19, 27]])
>>> k - [4, 8]
array([[ 9, 21],
[13, 10],
[15, 19]])
You can just index the column, row or whatever you want (guide by scipy here) and then subtract whatever you want from each element in that section with -=:
>>> a
array([[13, 29],
[17, 18],
[19, 27]])
>>> a[:,0] -= 4
>>> a
array([[ 9, 29],
[13, 18],
[15, 27]])
>>> a[:,1] -= 8
>>> a
array([[ 9, 21],
[13, 10],
[15, 19]])
The Story:
Currently, I have a function-under-test that expects a list of lists of integers with the following rules:
number of sublists (let's call it N) can be from 1 to 50
number of values inside sublists is the same for all sublists (rectangular form) and should be >= 0 and <= 5
values inside sublists cannot be more than or equal to the total number of sublists. In other words, each value inside a sublist is an integer >= 0 and < N
Sample valid inputs:
[[0]]
[[2, 1], [2, 0], [3, 1], [1, 0]]
[[1], [0]]
Sample invalid inputs:
[[2]] # 2 is more than N=1 (total number of sublists)
[[0, 1], [2, 0]] # 2 is equal to N=2 (total number of sublists)
I'm trying to approach it with property-based-testing and generate different valid inputs with hypothesis library and trying to wrap my head around lists() and integers(), but cannot make it work:
the condition #1 is easy to approach with lists() and min_size and max_size arguments
the condition #2 is covered under Chaining strategies together
the condition #3 is what I'm struggling with - cause, if we use the rectangle_lists from the above example, we don't have a reference to the length of the "parent" list inside integers()
The Question:
How can I limit the integer values inside sublists to be less than the total number of sublists?
Some of my attempts:
from hypothesis import given
from hypothesis.strategies import lists, integers
#given(lists(lists(integers(min_value=0, max_value=5), min_size=1, max_size=5), min_size=1, max_size=50))
def test(l):
# ...
This one was very far from meeting the requirements - list is not strictly of a rectangular form and generated integer values can go over the generated size of the list.
from hypothesis import given
from hypothesis.strategies import lists, integers
#given(integers(min_value=0, max_value=5).flatmap(lambda n: lists(lists(integers(min_value=1, max_value=5), min_size=n, max_size=n), min_size=1, max_size=50)))
def test(l):
# ...
Here, the #1 and #2 are requirements were being met, but the integer values can go larger than the size of the list - requirement #3 is not met.
There's a good general technique that is often useful when trying to solve tricky constraints like this: try to build something that looks a bit like what you want but doesn't satisfy all the constraints and then compose it with a function that modifies it (e.g. by throwing away the bad bits or patching up bits that don't quite work) to make it satisfy the constraints.
For your case, you could do something like the following:
from hypothesis.strategies import builds, lists, integers
def prune_list(ls):
n = len(ls)
return [
[i for i in sublist if i < n][:5]
for sublist in ls
]
limited_list_strategy = builds(
prune_list,
lists(lists(integers(0, 49), average_size=5), max_size=50, min_size=1)
)
In this we:
Generate a list that looks roughly right (it's a list of list of integers and the integers are in the same range as all possible indices that could be valid).
Prune out any invalid indices from the sublists
Truncate any sublists that still have more than 5 elements in them
The result should satisfy all three conditions you needed.
The average_size parameter isn't strictly necessary but in experimenting with this I found it was a bit too prone to producing empty sublists otherwise.
ETA: Apologies. I've just realised that I misread one of your conditions - this doesn't actually do quite what you want because it doesn't ensure each list is the same length. Here's a way to modify this to fix that (it gets a bit more complicated, so I've switched to using composite instead of builds):
from hypothesis.strategies import composite, lists, integers, permutations
#composite
def limisted_lists(draw):
ls = draw(
lists(lists(integers(0, 49), average_size=5), max_size=50, min_size=1)
)
filler = draw(permutations(range(50)))
sublist_length = draw(integers(0, 5))
n = len(ls)
pruned = [
[i for i in sublist if i < n][:sublist_length]
for sublist in ls
]
for sublist in pruned:
for i in filler:
if len(sublist) == sublist_length:
break
elif i < n:
sublist.append(i)
return pruned
The idea is that we generate a "filler" list that provides the defaults for what a sublist looks like (so they will tend to shrink in the direction of being more similar to eachother) and then draw the length of the sublists to prune to to get that consistency.
This has got pretty complicated I admit. You might want to use RecursivelyIronic's flatmap based version. The main reason I prefer this over that is that it will tend to shrink better, so you'll get nicer examples out of it.
You can also do this with flatmap, though it's a bit of a contortion.
from hypothesis import strategies as st
from hypothesis import given, settings
number_of_lists = st.integers(min_value=1, max_value=50)
list_lengths = st.integers(min_value=0, max_value=5)
def build_strategy(number_and_length):
number, length = number_and_length
list_elements = st.integers(min_value=0, max_value=number - 1)
return st.lists(
st.lists(list_elements, min_size=length, max_size=length),
min_size=number, max_size=number)
mystrategy = st.tuples(number_of_lists, list_lengths).flatmap(build_strategy)
#settings(max_examples=5000)
#given(mystrategy)
def test_constraints(list_of_lists):
N = len(list_of_lists)
# condition 1
assert 1 <= N <= 50
# Condition 2
[length] = set(map(len, list_of_lists))
assert 0 <= length <= 5
# Condition 3
assert all((0 <= element < N) for lst in list_of_lists for element in lst)
As David mentioned, this does tend to produce a lot of empty lists, so some average size tuning would be required.
>>> mystrategy.example()
[[24, 6, 4, 19], [26, 9, 15, 15], [1, 2, 25, 4], [12, 8, 18, 19], [12, 15, 2, 31], [3, 8, 17, 2], [5, 1, 1, 5], [7, 1, 16, 8], [9, 9, 6, 4], [22, 24, 28, 16], [18, 11, 20, 21], [16, 23, 30, 5], [13, 1, 16, 16], [24, 23, 16, 32], [13, 30, 10, 1], [7, 5, 14, 31], [31, 15, 23, 18], [3, 0, 13, 9], [32, 26, 22, 23], [4, 11, 20, 10], [6, 15, 32, 22], [32, 19, 1, 31], [20, 28, 4, 21], [18, 29, 0, 8], [6, 9, 24, 3], [20, 17, 31, 8], [6, 12, 8, 22], [32, 22, 9, 4], [16, 27, 29, 9], [21, 15, 30, 5], [19, 10, 20, 21], [31, 13, 0, 21], [16, 9, 8, 29]]
>>> mystrategy.example()
[[28, 18], [17, 25], [26, 27], [20, 6], [15, 10], [1, 21], [23, 15], [7, 5], [9, 3], [8, 3], [3, 4], [19, 29], [18, 11], [6, 6], [8, 19], [14, 7], [25, 3], [26, 11], [24, 20], [22, 2], [19, 12], [19, 27], [13, 20], [16, 5], [6, 2], [4, 18], [10, 2], [26, 16], [24, 24], [11, 26]]
>>> mystrategy.example()
[[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []]
>>> mystrategy.example()
[[], [], [], [], [], [], [], [], [], [], [], [], [], [], []]
>>> mystrategy.example()
[[6, 8, 22, 21, 22], [3, 0, 24, 5, 18], [16, 17, 25, 16, 11], [2, 12, 0, 3, 15], [0, 12, 12, 12, 14], [11, 20, 6, 6, 23], [5, 19, 2, 0, 12], [16, 0, 1, 24, 10], [2, 13, 21, 19, 15], [2, 14, 27, 6, 7], [22, 25, 18, 24, 9], [26, 21, 15, 18, 17], [7, 11, 22, 17, 21], [3, 11, 3, 20, 16], [22, 13, 18, 21, 11], [4, 27, 21, 20, 25], [4, 1, 13, 5, 13], [16, 19, 6, 6, 25], [19, 10, 14, 12, 14], [18, 13, 13, 16, 3], [12, 7, 26, 26, 12], [25, 21, 12, 23, 22], [11, 4, 24, 5, 27], [25, 10, 10, 26, 27], [8, 25, 20, 6, 23], [8, 0, 12, 26, 14], [7, 11, 6, 27, 26], [6, 24, 22, 23, 19]]
Pretty late, but for posterity: the easiest solution is to pick dimensions, then build up from the element strategy.
from hypothesis.strategies import composite, integers, lists
#composite
def complicated_rectangles(draw, max_N):
list_len = draw(integers(1, max_N))
sublist_len = draw(integers(0, 5))
element_strat = integers(0, min(list_len, 5))
sublist_strat = lists(
element_strat, min_size=sublist_len, max_size=sublist_len)
return draw(lists(
sublist_strat, min_size=list_len, max_size=list_len))
I am working with images through numpy. I want to set a chunk of the image to its average color. I am able to do this, but I have to re-index the array, when I would like to use the original view to do this. In other words, I would like to use that 4th line of code, but I'm stuck with the 3rd one.
I have read a few posts about the as_strided function, but it is confusing to me, and I was hoping there might be a simpler solution. So is there a way to slightly modify that last line of code to do what I want?
box = im[x-dx:x+dx, y-dy:y+dy, :]
avg = block(box) #returns a 1D numpy array with 3 values
im[x-dx:x+dx, y-dy:y+dy, :] = avg[None,None,:] #sets box to average color
#box = avg[None,None,:] #does not affect original array
box = blah
just reassigns the box variable. The array that the box variable previously referred to is unaffected. This is not what you want.
box[:] = blah
is a slice assignment. It modifies the contents of the array. This is what you want.
Note that slice assignment is dependent on the syntactic form of the statement. The fact that box was assigned by box = im[stuff] does not make further assignments to box slice assignments. This is similar to how if you do
l = [1, 2, 3]
b = l[2]
b = 0
the assignment to b doesn't affect l.
Gray-scale Images
This will set a chunk of an array to its average (mean) value:
im[2:4, 2:4] = im[2:4, 2:4].mean()
For example:
In [9]: im
Out[9]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
In [10]: im[2:4, 2:4] = im[2:4, 2:4].mean()
In [11]: im
Out[11]:
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 12, 12],
[12, 13, 12, 12]])
Color Images
Suppose that we want to average over each component of color separately:
In [22]: im = np.arange(48).reshape((4,4,3))
In [23]: im
Out[23]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[30, 31, 32],
[33, 34, 35]],
[[36, 37, 38],
[39, 40, 41],
[42, 43, 44],
[45, 46, 47]]])
In [24]: im[2:4, 2:4, :] = im[2:4, 2:4, :].mean(axis=0).mean(axis=0)[np.newaxis, np.newaxis, :]
In [25]: im
Out[25]:
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17],
[18, 19, 20],
[21, 22, 23]],
[[24, 25, 26],
[27, 28, 29],
[37, 38, 39],
[37, 38, 39]],
[[36, 37, 38],
[39, 40, 41],
[37, 38, 39],
[37, 38, 39]]])