We Keep Coding

Python Return closest number above and below given x

I have a problem. In my code I need to do the following:
I have an array like this: [1, 5, 7, 9, 10, 11, 14, 15]
Now I need to determine in the most efficient way what the closest values are for a given x. For example:
array = [1, 5, 7, 9, 10, 11, 14, 15]
above, below = myFunction(array, 8)
Should return 7 and 9
If there is no number higher or lower than the given number, the value that couldn't be determined should be None. If the same number is given as in the array, for example: 7. The values 5 and 9 should be returned.
I know there is a min() and max() function, but I couldn't find anything in my case. The only thing close to what I want, was manual looping through the array, but I was wondering if there was a more efficient way, because this code will be executed arround 500.000 times, so speed is important!
Is there a efficient way to determine those values?

This answer based on #Mark Tolonen hint about bisect
Let's say that you need to append one number on every iteration but initial array is the same.
All elements in array should be unique. Initial array may be not sorted.
import bisect as b
init_array = [5, 9, 10, 11, 14, 15, 1, 7]
init_array.sort()
numbers_to_append = [22, 4, 12, 88, 99, 109, 999, 1000, 1001]
numbers_to_check = [12, 55, 23, 55, 0, 55, 10, 999]
for (_, n) in enumerate(numbers_to_check):
# get index of closer right to below
i = b.bisect_left(init_array, n)
# get above
if i >= (len(init_array)):
above = None
else:
if init_array[i] == n:
try:
above = init_array[i + 1]
except IndexError:
above = None
else:
above = init_array[i]
# get below
below = init_array[i - 1] if i - 1 >= 0 and init_array[i - 1] - n < 0 else None
print('---------')
print('array is', init_array)
print('given x is', n)
print('below is', below)
print('above is', above)
print('appending', numbers_to_append[_])
# check if number trying to append doesn't exists in array
# WARNING: this check may be slow and added only for showing that we need to add only unique numbers
# real check should be based on real conditions and numbers that we suppose to add
i_to_append = b.bisect_left(init_array, numbers_to_append[_])
if numbers_to_append[_] not in init_array[i_to_append:i_to_append+1]:
init_array.insert(i_to_append, numbers_to_append[_])
output:
---------
array is [1, 5, 7, 9, 10, 11, 14, 15]
given x is 12
below is 11
above is 14
appending 22
---------
array is [1, 5, 7, 9, 10, 11, 14, 15, 22]
given x is 55
below is 22
above is None
appending 4
---------
array is [1, 4, 5, 7, 9, 10, 11, 14, 15, 22]
given x is 23
below is 22
above is None
appending 12
---------
array is [1, 4, 5, 7, 9, 10, 11, 12, 14, 15, 22]
given x is 55
below is 22
above is None
appending 88
---------
array is [1, 4, 5, 7, 9, 10, 11, 12, 14, 15, 22, 88]
given x is 0
below is None
above is 1
appending 99
---------
array is [1, 4, 5, 7, 9, 10, 11, 12, 14, 15, 22, 88, 99]
given x is 55
below is 22
above is 88
appending 109
---------
array is [1, 4, 5, 7, 9, 10, 11, 12, 14, 15, 22, 88, 99, 109]
given x is 10
below is 9
above is 11
appending 999
---------
array is [1, 4, 5, 7, 9, 10, 11, 12, 14, 15, 22, 88, 99, 109, 999]
given x is 999
below is 109
above is None
appending 1000

Assuming that your array is not sorted, you will be indeed forced to scan the entire array.
But if it is sorted you can use a binary search approach that will run in O(log(n)), something like this:
def search_nearest_elems(array, elem):
"""
Uses dichotomic search approach.
Runs in O(log(n)), with n = len(array)
"""
i = len(array) // 2
left, right = 0, len(array) - 1
while right - left > 1:
if array[i] == elem:
return elem, elem
elif array[i] > elem:
right, i = i, (left + i) // 2
else:
left, i = i, (right + i) // 2
return array[left], array[right]
array = [1, 5, 7, 9, 10, 11, 14, 15]
assert search_nearest_elems(array, 8) == (7, 9)
assert search_nearest_elems(array, 9) == (9, 9)
assert search_nearest_elems(array, 14.5) == (14, 15)
assert search_nearest_elems(array, 2) == (1, 5)

import bisect
def find_nearest(arr,value):
arr.sort()
idx = bisect.bisect_left(arr, value)
if idx == 0:
return None, arr[idx]
elif idx == len(arr):
return arr[idx - 1], None
elif arr[idx] == value:
return arr[idx], arr[idx]
else:
return arr[idx - 1], arr[idx]
array = [1, 5, 7, 9, 10, 11, 14, 15]
print(find_nearest(array, 0))
print(find_nearest(array, 4))
print(find_nearest(array, 8))
print(find_nearest(array, 10))
print(find_nearest(array, 20))
Output:
(None, 1)
(1, 5)
(7, 9)
(10, 10)
(15, None)
Helper source

Alternative to for loops for calculating 15^6 combinations in Python

Today, I have a nested for loop in python to calculate the value of all different combinations in a horse racing card consisting of six different races; i.e. six different arrays (of different lengths, but up to 15 items per array). It can be up to 11 390 625 combinations (15^6).
For each horse in each race, I calculate a value (EV) which I want to multiply.
Array 1: 1A,1B,1C,1D,1E,1F
Array 2: 2A,2B,2C,2D,2E,2F
Array 3: 3A,3B,3C,3D,3E,3F
Array 4: 4A,4B,4C,4D,4E,4F
Array 5: 5A,5B,5C,5D,5E,5F
Array 6: 6A,6B,6C,6D,6E,6F
1A * 1B * 1C * 1D * 1E * 1F = X,XX
.... .... .... .... ... ...
6A * 6B * 6C * 6D * 6E * 6F 0 X,XX
Doing four levels is OK. It takes me about 3 minutes.
I have yet not been able to do six levels.
I need help in creating a better way of doing this, and have no idea how to proceed. Does numpy perhaps offer help here? Pandas? I've tried compiling the code with Cython, but it did not help much.
My function takes in a list containing the horses in numerical order and their EV. (Since horse starting numbers do not start with zero, I add 1 to the index). I iterate through all the different races, and save the output for the combination into a dataframe.
def calculateCombos(horses_in_race_1,horses_in_race_2,horses_in_race_3,horses_in_race_4,horses_in_race_5,horses_in_race_6,totalCombinations, df):
totalCombinations = 0
for idx1, hr1_ev in enumerate(horses_in_race_1):
hr1_no = idx1 + 1
for idx2, hr2_ev in enumerate(horses_in_race_2):
hr2_no = idx2 + 1
for idx3, hr3_ev in enumerate(horses_in_race_3):
hr3_no_ = idx3 + 1
for idx4, hr4_ev in enumerate(horses_in_race_4):
hr4_no = idx4 + 1
for idx5, hr5_ev in enumerate(horses_in_race_5):
hr5_no = idx5 + 1
for idx6, hr6_ev in enumerate(horses_in_race_6):
hr6_no = idx6 + 1
totalCombinations = totalCombinations + 1
combinationEV = hr1_ev * hr2_ev * hr3_ev * hr4_ev * hr5_ev * hr6_ev
new_row = {'Race1':str(hr1_no),'Race2':str(hr2_no),'Race3':str(hr3_no),'Race4':str(hr4_no),'Race5':str(hr5_no),'Race6':str(hr6_no), 'EV':combinationEV}
df = appendCombinationToDF(df, new_row)
return df

Why don't you try this and see if you can run the function without any issues? This works on my laptop (I'm using PyCharm). If you can't run this, then I would say that you need a better PC perhaps. I did not encounter any memory error.
Assume that we have the following:
horses_in_race_1 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
horses_in_race_2 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
horses_in_race_3 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
horses_in_race_4 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
horses_in_race_5 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
horses_in_race_6 = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
I have re-written the function as follows - made a change in enumeration. Also, not using df as I do not know what function this is - appendCombinationToDF
def calculateCombos(horses_in_race_1,horses_in_race_2,horses_in_race_3,horses_in_race_4,horses_in_race_5,horses_in_race_6):
for idx1, hr1_ev in enumerate(horses_in_race_1, start = 1):
for idx2, hr2_ev in enumerate(horses_in_race_2, start = 1):
for idx3, hr3_ev in enumerate(horses_in_race_3, start = 1):
for idx4, hr4_ev in enumerate(horses_in_race_4, start = 1):
for idx5, hr5_ev in enumerate(horses_in_race_5, start = 1):
for idx6, hr6_ev in enumerate(horses_in_race_6, start = 1):
combinationEV = hr1_ev * hr2_ev * hr3_ev * hr4_ev * hr5_ev * hr6_ev
new_row = {'Race1':str(idx1),'Race2':str(idx2),'Race3':str(idx3),'Race4':str(idx4),'Race5':str(idx5),'Race6':str(idx6), 'EV':combinationEV}
l.append(new_row)
#df = appendCombinationToDF(df, new_row)
l = [] # df = ...
calculateCombos(horses_in_race_1, horses_in_race_2, horses_in_race_3, horses_in_race_4, horses_in_race_5, horses_in_race_6)
Executing len(l), I get:
11390625 # maximum combinations possible. This means that above function ran successfully and computation succeeded.
If the above can be executed, replace list l with df and see if function can execute without encountering memory error. I was able to run the above in less than 20-30 seconds.

Python Numpy Array geht values of neighbours

I want to get all neighbour values of a np.array.
The array looks like:
x = np.array([ [1, 2, 3, 4 ],
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16] ])
What I have is:
i = 2
j = 2
n = x[i,j-1], x[i,j], x[i,j+1], x[i-1,j], x[i+1,j], x[i-1,j-1], x[i+1,j+1], x[i+1,j-1], x[i-1,j+1]
This returns (what I want)
(10, 11, 12, 7, 15, 6, 16, 14, 8)
But also got bugs for example when i want the neightbour values of
i = 3
j = 3
That gives:
Exception has occurred: IndexError
index 4 is out of bounds for axis 1 with size 4
An other soultion is:
def find_neighbors(m, i, j, dist=1):
return [row[max(0, j-dist):j+dist+1] for row in m[max(0,-1):i+dist+1]]
and
n = find_neighbors(x, i, j)
Which gives me an array of the neightbours but also gives me not all neightbours when I set
i = 0
j = 0
because it only gives me:
[array([1, 2]), array([5, 6])]
Does anybody have a solution for this?
Thank you!

You can take advantage of python indexing wrapping around for negative indices.
def wrap_nb(x,i,j):
return x[np.ix_(*((z-1, z, z+1-S) for z,S in zip((i,j), x.shape)))].ravel()
This requires i and j to be nonnegative and less than the shape of x.
If that is not guaranteed:
def wrap_nb(x,i,j):
return x[np.ix_(*(np.r_[z-1:z+2]%S for z,S in zip((i,j), x.shape)))].ravel()
Examples:
>>> wrap_nb(x,1,-2)
array([ 2, 3, 4, 6, 7, 8, 10, 11, 12])
>>> wrap_nb(x,0,-1)
array([15, 16, 13, 3, 4, 1, 7, 8, 5])
>>> wrap_nb(x,0,0)
array([16, 13, 14, 4, 1, 2, 8, 5, 6])

# function to find the start row and column
def find_start(x):
start = x-1 if x-1 >= 0 else 0
return start
# function to find the end row and column
def find_end(x, shape):
end = x+1 if x+1 <= shape else shape
return end
def find_neighbors(a, i, j):
neighbors = []
row_start, row_end = find_start(i), find_end(i, a.shape[0])
col_start, col_end = find_start(j), find_end(j, a.shape[1])
for y in range(a.shape[0]):
for z in range(a.shape[1]):
if y >= row_start and y <= row_end:
if z >= col_start and z <= col_end:
neighbors.append(a[y][z])
return neighbors
i, j = 0, 0
neighbors = find_neighbors(a, i, j)
print(neighbors)
Output: [1, 2, 5, 6]
i, j = 3, 3
neighbors = find_neighbors(a, i, j)
neighbors
Output: [11, 12, 15, 16]
i, j = 2, 2
neighbors = find_neighbors(a, i, j)
neighbors
Output: [6, 7, 8, 10, 11, 12, 14, 15, 16]
This would cover all the edge cases.

I got the following solution form an mate:
New array:
homes = np.array([ [1, 2, 3, 4 ],
[5, 6, 7, 8],
[9, 10, 11, 12],
[13, 14, 15, 16] ])
Code for returning the neighbour values:
neighbour = []
neighbour += [homes[i][j]] # value itself
neighbour += [homes[i][(j + 1) % n]] # value right
neighbour += [homes[i][(j - 1) % n]] # value left
neighbour += [homes[(i + 1) % n][j]] # value down
neighbour += [homes[(i + 1) % n][(j + 1) % n]] # value right down
neighbour += [homes[(i + 1) % n][(j - 1) % n]] # value left down
neighbour += [homes[(i - 1) % n][j]] # vlaue up
neighbour += [homes[(i - 1) % n][(j + 1) % n]] # vlaue right up
neighbour += [homes[(i - 1) % n][(j - 1) % n]] # value left up
Which returns me:
i = 0
j = 0
[16, 13, 15, 4, 1, 3, 12, 9, 11]
Thats what I need but I'am still interessed in solution like the one from Abdur

Find Value that Partitions two Numpy Arrays Equally

I have two arrays (x1 and x2) of equal length that have overlapping ranges of values.
I need to find a value q such that l1-l2 is minimized, and
l1 = x1[np.where(x1 > q)].shape[0]
l2 = x2[np.where(x2 < q)].shape[0]
I need this to be reasonably high-performance since the arrays can be large. A solution using native numpy routines would be preferred.

There may be a smarter way to look for a value, but you can do an exhaustive search as follows:
>>> x1 = np.random.rand(10)
>>> x2 = np.random.rand(10)
>>> x1.sort()
>>> x2.sort()
>>> x1
array([ 0.12568451, 0.30256769, 0.33478133, 0.41973331, 0.46493576,
0.52173197, 0.72289189, 0.72834444, 0.78662283, 0.78796277])
>>> x2
array([ 0.05513774, 0.21567893, 0.29953634, 0.37426842, 0.40000622,
0.54602497, 0.7225469 , 0.80116148, 0.82542633, 0.86736597])
We can compute l1 if q is one of the items in x1 as:
>>> l1_x1 = len(x1) - np.arange(len(x1)) - 1
>>> l1_x1
array([9, 8, 7, 6, 5, 4, 3, 2, 1, 0])
And l2 for the same q as:
>>> l2_x1 = np.searchsorted(x1, x2)
>>> l2_x1
array([ 0, 1, 1, 3, 3, 6, 6, 10, 10, 10], dtype=int64)
You can similarly get values for l1 and l2 when q is in x2:
>>> l2_x2 = np.arange(len(x2))
>>> l2_x2
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> l1_x2 = len(x1) - np.searchsorted(x1, x2, side='right')
>>> l1_x2
array([10, 9, 9, 7, 7, 4, 4, 0, 0, 0], dtype=int64)
And you then simply check for the minimum of l1 - l2:
>>> np.concatenate((l1_x1 - l2_x1, l1_x2 - l2_x2))
array([ 9, 7, 6, 3, 2, -2, -3, -8, -9, -10, 10, 8, 7,
4, 3, -1, -2, -7, -8, -9], dtype=int64)
>>> q_idx = np.argmin(np.abs(np.concatenate((l1_x1 - l2_x1, l1_x2 - l2_x2))))
>>> q = x1[q_idx] if q_idx < len(x1) else x2[q_idx - len(x1)]
>>> q
0.54602497466094291
>>> x1[x1 > q].shape[0]
4L
>>> x2[x2 < q].shape[0]
5L

I think I may have found a fairly simple way to do it.
x1 = (50 - 10) * np.random.random(10000) + 10
x2 = (75 - 25) * np.random.random(10000) + 25
x1.sort()
x2.sort()
x2 = x2[::-1] # reverse the array
# The overlap point should fall where the difference is smallest
diff = np.abs(x1 - x2)
# get the index of where the minimum occurs
loc = np.where(diff == np.min(diff))
q1 = x1[loc] # 38.79087351
q2 = x2[loc] # 38.79110941
M4rtini's solution produces q = 38.7867527.

This is fundamentally an interval problem so you might want to do some reading on Interval trees, but you don't need to understand interval trees to solve this problem.
If you think of every (x1[i], x2[i]) as being an interval, you're looking for the value q which splits the intervals into two groups as evenly as possible ignoring intervals that overlap q. Lets take the easy case first:
from numpy import array
x1 = array([19, 32, 47, 13, 56, 1, 87, 48])
x2 = array([44, 38, 50, 39, 85, 26, 92, 64])
x1sort = np.sort(x1)
x2sort = np.sort(x2)[::-1]
diff = abs(x2sort - x1sort)
mindiff = diff.argmin()
print mindiff, x2sort[mindiff], x1sort[mindiff]
# 4 44 47
#xvtk's solution works well in this case and gives us a range of [44, 47]. Because no intervals overlap the range, all values of q in the range are equivalent and yield an optimal result. Here is an example that is a little more tricky:
x1 = array([12, 65, 46, 81, 71, 77, 37])
x2 = array([ 20, 85, 59, 122, 101, 87, 58])
x1sort = np.sort(x1)
x2sort = np.sort(x2)[::-1]
diff = abs(x2sort - x1sort)
mindiff = diff.argmin()
print mindiff, x2sort[mindiff], x1sort[mindiff], x1sort[mindiff-1]
# 59 71 65
Here the solution gives us a range of [59, 71] but notice that not all values in the range are equivalent. Anything to the left of the green line will produce 3 and 4 intervals on the left and right respectively, while anything to the right of the green line will produce 3 intervals on both sides.
I'm pretty sure that the optimal solution is guaranteed to be in the range produced by #xvtk's solution. It's possible that one of the red lines is guaranteed to be an optimal solution, though I'm not sure on this point. Hope that helps.

Maybe use some of the optimizing functions in scipy to minimize the difference.
Like this for example
import numpy as np
from scipy.optimize import fmin
def findQ(q, *x):
x1, x2 = x
l1 = x1[np.where(x1 > q)].shape[0]
l2 = x2[np.where(x2 < q)].shape[0]
return abs(l1-l2)
x1 = (50 - 10) * np.random.random(10000) + 10
x2 = (75 - 25) * np.random.random(10000) + 25
q0 = (min(x2) + max(x1))/2.0
q = fmin(findQ, q0, (x1,x2))

Sorting bytes in words, tuples in python

I looked around and I can't seem to find the proper way of sorting a 32 entry tuple by inverting every odd and even entry.
ex:
1 0 3 2 5 4 7 6 9 8
to
0 1 2 3 4 5 6 7 8 9
My current code looks like this
i=0
nd = []
while i < len(self.r.ipDeviceName):
print(i)
if i%2:
nd[i]=self.r.ipDeviceName[i-1]
else:
nd[i]=self.r.ipDeviceName[i+1]
dn = "".join(map(chr,nd))
devicenameText.SetValue(dn)
the type of self.r.ipDeviceName is tuple and I either get a IndexError or a tuple doesn't suport assignation depending on variations of the code
I also tried this with the same results
nd = self.r.ipDeviceName
for i in nd:
if i&0x01:
nd[i]=self.r.ipDeviceName[i-1]
else:
nd[i]=self.r.ipDeviceName[i+1]
dn = "".join(map(chr,nd))
devicenameText.SetValue(dn)
With the same results. Something very simple seems to elude me. Thanks for your help and time.

Tuples are immutable - you can't modify them once they are created. To modify individual elements you want to store the data in a mutable collection such as a list instead. You can use the built-in functions list and tuple to convert from tuple to list or vice versa.
Alternatively you could use zip and a functional style approach to create a new tuple from your existing tuple without modifying the original:
>>> t = tuple(range(10))
>>> tuple(x for i in zip(t[1::2], t[::2]) for x in i)
(1, 0, 3, 2, 5, 4, 7, 6, 9, 8)
Or using itertools.chain:
>>> import itertools
>>> tuple(itertools.chain(*zip(t[1::2], t[::2])))
(1, 0, 3, 2, 5, 4, 7, 6, 9, 8)
Note that the use of zip here assumes that your tuple has an even number of elements (which is the case here, according to your question).

You can't change a tuple, they're immutable. However you can replace them with a new one arranged the way you want (I wouldn't call what you want "sorted"). To do it, all that is needed it to swap each pair of items that are in the original tuple.
Here's a straight-forward implementation. Note it leaves the last entry alone if there are an odd number of them since you never said how you wanted that case handled. Dealing with that possibility complicates the code slightly.
def swap_even_odd_entries(seq):
tmp = list(seq)+[seq[-1]] # convert sequence to mutable list and dup last
for i in xrange(0, len(seq), 2):
tmp[i],tmp[i+1] = tmp[i+1],tmp[i] # swap each entry with following one
return tuple(tmp[:len(seq)]) # remove any excess
a = (1, 0, 3, 2, 5, 4, 7, 6, 9, 8)
a = swap_even_odd_entries(a)
b = (91, 70, 23, 42, 75, 14, 87, 36, 19, 80)
b = swap_even_odd_entries(b)
c = (1, 0, 3, 2, 5)
c = swap_even_odd_entries(c)
print a
print b
print c
# output
# (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
# (70, 91, 42, 23, 14, 75, 36, 87, 80, 19)
# (0, 1, 2, 3, 5)
The same thing can also be done in a less-readable way as a long single expression. Again the last entry remains unchanged if the length is odd.
swap_even_odd_entries2 = lambda t: tuple(
v for p in [(b,a) for a,b in zip(*[iter(t)]*2) + [(t[-1],)*2]]
for v in p)[:len(t)]
a = (1, 0, 3, 2, 5, 4, 7, 6, 9, 8)
a = swap_even_odd_entries2(a)
b = (91, 70, 23, 42, 75, 14, 87, 36, 19, 80)
b = swap_even_odd_entries2(b)
c = (1, 0, 3, 2, 5)
c = swap_even_odd_entries2(c)
print
print a
print b
print c
# output
# (0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
# (70, 91, 42, 23, 14, 75, 36, 87, 80, 19)
# (0, 1, 2, 3, 5)

If you add the functions grouper and flatten (see itertools recipes) to your toolset, you can do:
xs = [1, 0, 3, 2, 5, 4, 7, 6, 9, 8]
xs2 = flatten((y, x) for (x, y) in grouper(2, xs))
# list(xs2) => [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
You could even write flatten(imap(reversed, grouper(2, xs)) but I guess only die-hard functional guys would like it.