We Keep Coding

How can I apply a piecewise function to every element of a 2D numpy array

Lets say I have a 2D numpy array, like
arr = array([[0, 0.001 , 0.002], [0.03, 0.04, 0.05], [0.01, 0.002, 0.5], [0.05, 0.8, 0.003]])
and I want to perform a piecewise function on it, say
def gammacor(x):
return np.piecewise(x, [x <= 0.00313, x > 0.00313], [12.92*x, 1.055*x**(1/2.4)-0.055])
gcarr = gammacor(arr)
When I do this, I get an error:
TypeError: NumPy boolean array indexing assignment requires a 0 or 1-dimensional input, input has 2 dimensions
If I try to run the function on the flattened array (with the plan to reshape back to n x 3 after running the function), I get the error:
ValueError: NumPy boolean array indexing assignment cannot assign 3 input values to the 0 output values where the mask is true
Is there an easy way to apply a piecewise function to all elements of a 2D (or ND) array?

The third parameter of np.piecewise is a funclist.
They should be callables:
import numpy as np
arr = np.array([[0, 0.001, 0.002], [0.03, 0.04, 0.05], [0.01, 0.002, 0.5],
[0.05, 0.8, 0.003]])
p = np.piecewise(arr, [arr <= 0.00313, arr > 0.00313],
[lambda v: 12.92 * v,
lambda v: 1.055 * v ** (1 / 2.4) - 0.055])
print(p)
Output:
[[0. 0.01292 0.02584 ]
[0.18974828 0.22091636 0.24780053]
[0.09985282 0.02584 0.73535698]
[0.24780053 0.90633175 0.03876 ]]
def gammacor(x):
return np.piecewise(x, [x <= 0.00313, x > 0.00313],
[lambda v: 12.92 * v,
lambda v: 1.055 * v ** (1 / 2.4) - 0.055])
gcarr = gammacor(arr)

Recursively dividing up a list, based on the endpoints

Here is what I am trying to do.
Take the list:
list1 = [0,2]
This list has start point 0 and end point 2.
Now, if we were to take the midpoint of this list, the list would become:
list1 = [0,1,2]
Now, if we were to recursively split up the list again (take the midpoints of the midpoints), the list would becomes:
list1 = [0,.5,1,1.5,2]
I need a function that will generate lists like this, preferably by keeping track of a variable. So, for instance, let's say there is a variable, n, that keeps track of something. When n = 1, the list might be [0,1,2] and when n = 2, the list might be [0,.5,1,1.5,2], and I am going to increment the value of to keep track of how many times I have divided up the list.
I know you need to use recursion for this, but I'm not sure how to implement it.
Should be something like this:
def recursive(list1,a,b,n):
"""list 1 is a list of values, a and b are the start
and end points of the list, and n is an int representing
how many times the list needs to be divided"""
int mid = len(list1)//2
stuff
Could someone help me write this function? Not for homework, part of a project I'm working on that involves using mesh analysis to divide up rectangle into parts.
This is what I have so far:
def recursive(a,b,list1,n):
w = b - a
mid = a + w / 2
left = list1[0:mid]
right = list1[mid:len(list1)-1]
return recursive(a,mid,list1,n) + mid + recursive(mid,b,list1,n)
but I'm not sure how to incorporate n into here.
NOTE: The list1 would initially be [a,b] - I would just manually enter that but I'm sure there's a better way to do it.

You've generated some interesting answers. Here are two more.
My first uses an iterator to avoid
slicing the list and is recursive because that seems like the most natural formulation.
def list_split(orig, n):
if not n:
return orig
else:
li = iter(orig)
this = next(li)
result = [this]
for nxt in li:
result.extend([(this+nxt)/2, nxt])
this = nxt
return list_split(result, n-1)
for i in range(6):
print(i, list_split([0, 2], i))
This prints
0 [0, 2]
1 [0, 1.0, 2]
2 [0, 0.5, 1.0, 1.5, 2]
3 [0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2]
4 [0, 0.125, 0.25, 0.375, 0.5, 0.625, 0.75, 0.875, 1.0, 1.125, 1.25, 1.375, 1.5, 1.625, 1.75, 1.875, 2]
5 [0, 0.0625, 0.125, 0.1875, 0.25, 0.3125, 0.375, 0.4375, 0.5, 0.5625, 0.625, 0.6875, 0.75, 0.8125, 0.875, 0.9375, 1.0, 1.0625, 1.125, 1.1875, 1.25, 1.3125, 1.375, 1.4375, 1.5, 1.5625, 1.625, 1.6875, 1.75, 1.8125, 1.875, 1.9375, 2]
My second is based on the observation that recursion isn't necessary if you always start from two elements. Suppose those elements are mn and mx. After N applications of the split operation you will have 2^N+1 elements in it, so the numerical distance between the elements will be (mx-mn)/(2**N).
Given this information it should therefore be possible to deterministically compute the elements of the array, or even easier to use numpy.linspace like this:
def grid(emin, emax, N):
return numpy.linspace(emin, emax, 2**N+1)
This appears to give the same answers, and will probably serve you best in the long run.

You can use some arithmetic and slicing to figure out the size of the result, and fill it efficiently with values.
While not required, you can implement a recursive call by wrapping this functionality in a simple helper function, which checks what iteration of splitting you are on, and splits the list further if you are not at your limit.
def expand(a):
"""
expands a list based on average values between every two values
"""
o = [0] * ((len(a) * 2) - 1)
o[::2] = a
o[1::2] = [(x+y)/2 for x, y in zip(a, a[1:])]
return o
def rec_expand(a, n):
if n == 0:
return a
else:
return rec_expand(expand(a), n-1)
In action
>>> rec_expand([0, 2], 2)
[0, 0.5, 1.0, 1.5, 2]
>>> rec_expand([0, 2], 4)
[0,
0.125,
0.25,
0.375,
0.5,
0.625,
0.75,
0.875,
1.0,
1.125,
1.25,
1.375,
1.5,
1.625,
1.75,
1.875,
2]

You could do this with a for loop
import numpy as np
def add_midpoints(orig_list, n):
for i in range(n):
new_list = []
for j in range(len(orig_list)-1):
new_list.append(np.mean(orig_list[j:(j+2)]))
orig_list = orig_list + new_list
orig_list.sort()
return orig_list
add_midpoints([0,2],1)
[0, 1.0, 2]
add_midpoints([0,2],2)
[0, 0.5, 1.0, 1.5, 2]
add_midpoints([0,2],3)
[0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2]

You can also do this totally non-recursively and without looping. What we're doing here is just making a binary scale between two numbers like on most Imperial system rulers.
def binary_scale(start, stop, level):
length = stop - start
scale = 2 ** level
return [start + i * length / scale for i in range(scale + 1)]
In use:
>>> binary_scale(0, 10, 0)
[0.0, 10.0]
>>> binary_scale(0, 10, 2)
[0.0, 2.5, 5.0, 7.5, 10.0]
>>> binary_scale(10, 0, 1)
[10.0, 5.0, 0.0]

Fun with anti-patterns:
def expand(a, n):
for _ in range(n):
a[:-1] = sum(([a[i], (a[i] + a[i + 1]) / 2] for i in range(len(a) - 1)), [])
return a
print(expand([0, 2], 2))
OUTPUT
% python3 test.py
[0, 0.5, 1.0, 1.5, 2]
%

python numpy - is there a faster way to convolve?

I have a numpy array that is very large (1 million integers). I'm using np.convolve in order to find the "densest" area of that array. By "desnsest" area I mean the window of a fixed length that has the the highest numbers when the window is summed. Let me show you in code:
import numpy as np
example = np.array([0,0,0,1,1,1,1,1,1,1,0,1,1,1,1,0,0,0,1,0,0,1,1,0,1,0,0,0,0,0,1,0])
window_size = 10
density = np.convolve(example, np.ones([window_size]), mode='valid')
print(density)
# [7.0, 7.0, 8.0, 9.0, 9.0, 9.0, 8.0, 7.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 4.0, 3.0]
I can then use np.argmax(density) to get the starting index of the highest density area 3.
Anyway, with this example it runs fast. but when convolving over million element array and with a window size of 10,000 it takes 2 seconds to complete. if I choose a windows_size of 500,000 it takes 3 minutes to complete.
Is there a better way to sum over the array with a certain window size to speed this up? If I converted this into a pandas series instead could I perhaps use something there?
Thanks for your help!

Try using scipy.signal.convolve. It has the option to compute the convolution using the fast Fourier transform (FFT), which should be much faster for the array sizes that you mentioned.
Using an array example with length 1000000 and convolving it with an array of length 10000, np.convolve took about 1.45 seconds on my computer, and scipy.signal.convolve took 22.7 milliseconds.

cumsum = np.cumsum(np.insert(example, 0, 0))
density2 = cumsum[window_size:]-cumsum[:-window_size]
np.all(density2 == density)
True
(remove insertion if you can live without the first value...)

This is how you can use the built-in NumPy real FFT functions to convolve in 1 dimension:
import numpy, numpy.fft.fftpack_lite
def fftpack_lite_rfftb(buf, s):
n = len(buf)
m = (n - 1) * 2
temp = numpy.empty(m, buf.dtype)
numpy.divide(buf, m, temp[:n])
temp[n:m] = 0
return numpy.fft.fftpack_lite.rfftb(temp[:m], s)
def fftconvolve(x, y):
xn = x.shape[-1]
yn = y.shape[-1]
cn = xn + yn - (xn + yn > 0)
m = 1 << cn.bit_length()
s = numpy.fft.fftpack_lite.rffti(m) # Initialization; can be factored out for performance
xpad = numpy.pad(x, [(0, 0)] * (len(x.shape) - 1) + [(0, m - xn)], 'constant')
a = numpy.fft.fftpack_lite.rfftf(xpad, s) # Forward transform
ypad = numpy.pad(y, [(0, 0)] * (len(y.shape) - 1) + [(0, m - yn)], 'constant')
b = numpy.fft.fftpack_lite.rfftf(ypad, s) # Forward transform
numpy.multiply(a, b, b) # Spectral multiplication
c = fftpack_lite_rfftb(b, s) # Backward transform
return c[:cn]
# Verify convolution is correct
assert (lambda a, b: numpy.allclose(fftconvolve(a, b), numpy.convolve(a, b)))(numpy.random.randn(numpy.random.randint(1, 32)), numpy.random.randn(numpy.random.randint(1, 32)))
Bear in mind that this padding is inefficient for convolution of vectors with significantly different sizes (> 100%); you'll want to use a linear combination technique like overlap-add to do smaller convolution.

Masking array X based on a condition with array Y of different size

I have two 1-dim arrays X and Y of different size. I am trying to build the 2-dim array resulting from a condition on X and Y. For instance:
X = np.array([0.3, 2.1, 4.3])
Y = np.array([1.5, 3.5])
mask = X > Y[:,np.newaxis]
and now I would like to perform something akin X[mask] = X[mask] + 1, so that for the example above it would result in:
newX = np.array([0.3, 3.1, 5.3],[0.3, 2.1, 5.3]])
I managed to get this result by doing:
newX = np.append(X, X).reshape(2,3)
newX[mask]=newX[mask]+1
But this hardcodes the length of the Y array (2 in the example), and includes a copy with np.append which is costly in cases where X and Y are actually large arrays (and it is probably quite ugly as well). Is there a correct way to do this?

In this particular case, where you wish to add 1 wherever mask is True,
perhaps the simplest way is to take advantage of broadcasting and dtype
promotion -- that is, booleans are treated as ints in numeric context.
In [49]: X + mask
Out[49]:
array([[ 0.3, 3.1, 5.3],
[ 0.3, 2.1, 5.3]])
Use broadcasting instead of an explicitly tiled copy of X if possible.
However, if you need newX you could use
In [54]: np.tile(X, (Y.size,1))
Out[54]:
array([[ 0.3, 2.1, 4.3],
[ 0.3, 2.1, 4.3]])
np.tile avoids the multiplications done by np.outer so it is faster for this purpose.
For example, with this setup:
import numpy as np
import timeit
import collections
import matplotlib.pyplot as plt
timing = collections.defaultdict(list)
Ns = np.linspace(10, 10000, 5).astype(int)
Ms = np.linspace(10, 10000, 5).astype(int)
for N, M in zip(Ns, Ms):
X = np.random.random(N)
Y = np.random.random(M)
timing['tile'].append(timeit.timeit(
'np.tile(X, (Y.size,1))',
'from __main__ import np, X, Y',
number=10))
timing['outer'].append(timeit.timeit(
'np.outer(np.ones_like(Y),X)',
'from __main__ import np, X, Y',
number=10))
plt.plot(Ns*Ms, timing['tile'], label='tile')
plt.plot(Ns*Ms, timing['outer'], label='outer')
plt.legend(loc='best')
plt.show()
As the size of the arrays get larger, the difference between tile and outer
should diminish relative to total time because the time required to
allocate/manage large arrays in RAM/swap overwhelms the relatively small
computational costs.

Find intersection of numpy float arrays

How can I find the intersection of two numpy float arrays?:
a = np.arange(2, 3, 0.1)
b = np.array([2.3, 2.4, 2.5])
out_data = np.intersect1d(a, b)
the result is
out_data -> ndarray: []

Because of the way floats work, in your example a[3] is not 2.3, but 2.3000000000000003. This is because 0.1 does not have an exact representation in IEEE double precision floats. The intersect1d method in numpy is really only well suited for integers. To solve this, you should implement your own method that takes in a tolerance to decide if two floats are sufficiently close.

Here's a vectorized approach using NumPy's broadcasting capability -
tol = 1e-5 # tolerance
out = b[(np.abs(a[:,None] - b) < tol).any(0)]
Sample run -
In [31]: a
Out[31]: array([ 2. , 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9])
In [32]: b
Out[32]: array([ 2.3 , 2.4 , 2.5 , 2.25, 2.1 ])
In [33]: tol = 1e-5 # tolerance
In [34]: b[(np.abs(a[:,None] - b) < tol).any(0)]
Out[34]: array([ 2.3, 2.4, 2.5, 2.1])

Putting my comments in function form (assuming both lists are sorted, which you should do ahead of time):
import numpy as np
from itertools import islice
def findOverlap(self, a, b, rtol = 1e-05, atol = 1e-08):
ovr_a = []
ovr_b = []
start_b = 0
for i, ai in enumerate(a):
for j, bj in islice(enumerate(b), start_b, None):
if np.isclose(ai, bj, rtol=rtol, atol=atol, equal_nan=False):
ovr_a.append(i)
ovr_b.append(j)
elif bj > ai: # (more than tolerance)
break # all the rest will be farther away
else: # bj < ai (more than tolerance)
start_b += 1 # ignore further tests of this item
return (ovr_a, ovr_b)
EDIT: getting rid of equal_nan -- if you're going to sort you may as well ditch the nans
EDIT: using islice instead of array slice
EDIT: fixed bug

The following routine will return the indexes of common values within specified tolerance(s) relative to list a.
def findOverlap(self, a, b, rtol = 1e-05, atol = 1e-08, equal_nan = False):
overlap_indexes = []
for i, item_a in enumerate(a):
for item_b in b:
if np.isclose(item_a, item_b, rtol = rtol, atol = atol, equal_nan = equal_nan):
overlap_indexes.append(i)
return overlap_indexes
eg
a = np.arange(2, 3, 0.1).tolist()
b = np.array([2.3, 2.4, 2.5]).tolist()
self.findOverlap(a, b)
-> overlap_indexes:[3, 4, 5]