A device sends an array of multiplexed error codes. You can think of the multiplexed error codes as some sort of FIFO ring buffer that can have a variable length depending on the number of simultaneously active error code. I wish to demultiplex the error codes into individual boolean arrays.
I'm looking for an efficient way (i.e. removing the for loop) to implement the following code:
import numpy as np
def get_error_vector(error_mux_vector, error_id, period):
index = np.where(error_mux_vector == error_id)[0]
error_vector = np.zeros(np.size(error_mux_vector))
for i in range(0, np.size(index) - 1):
if (index[i + 1] - index[i]) <= 1 / period:
error_vector[index[i]:index[i + 1] + 1] = 1
return error_vector
Here are mock values to illustrate the problem. 0 means no error; 1, 2, and 3 are error codes. We assume the error signal has a frequency of 5 Hz (period of 0.2s) :
import matplotlib.pyplot as plt
error_signal = np.array([0,0,0,0,0,1,2,3,1,2,3,2,3,2,3,0,0,0,0,0,1,1,1,2,3,1,3,1,3,1,3,0,0,0,0,0,2,0,2,2])
error_vector_1 = get_error_vector(error_signal, 1, 0.2)
error_vector_2 = get_error_vector(error_signal, 2, 0.2)
error_vector_3 = get_error_vector(error_signal, 3, 0.2)
plt.plot(error_signal)
plt.plot(error_vector_1)
plt.plot(error_vector_2)
plt.plot(error_vector_3)
plt.legend(['array', 'error 1', 'error 2', 'error 3'])
plt.show()
The actual device data can have from 50k to 10M points, with around 100 possible error codes. This means that a for loop is really inefficient for the use case. I would like to improve this code but I haven't found an efficient solution so far.
Here is a vectorized approach that creates all vectors in one go. It comes in two flavors. On my random test case the second is faster but that may depend on the exact stats of your signal.
import numpy as np
# dense strat
def demultiplex(signal,maxdist):
n = signal.max()
aux = np.zeros((n,len(signal)+1),np.int16)
nz = signal.nonzero()[0]
signal = signal[nz]
idx = signal.argsort(kind="stable")
valid = ((nz[idx[1:]]<=nz[idx[:-1]]+maxdist)&
(signal[idx[1:]]==signal[idx[:-1]])).nonzero()[0]
aux[signal[idx[valid]]-1,nz[idx[valid]]] = 1
aux[signal[idx[valid+1]]-1,nz[idx[valid+1]]+1] -= 1
out = (aux[:,:-1].cumsum(1) > 0).view(np.int8)
return out
# sparse strat
def demultiplex2(signal,maxdist):
n = signal.max()
m = signal.size
nz = signal.nonzero()[0]
signal = signal[nz]
idx = signal.argsort(kind="stable")
delta = nz[idx[1:]] - nz[idx[:-1]]
valid = ((delta<=maxdist)&(signal[idx[1:]]==signal[idx[:-1]])).nonzero()[0]
delta = delta[valid]
nz = nz[idx[valid]]
nz[1:] -= nz[:-1] + delta[:-1]
offsets = (delta+1).cumsum()
x = np.ones(offsets[-1],int)
x[0] = nz[0]
x[offsets[:-1]] = nz[1:]
out = np.zeros((n,m),np.uint8)
out[(signal[idx[valid]]-1).repeat(delta+1),x.cumsum()] = 1
return out
# OP
def get_error_vector(error_mux_vector, error_id, period):
index = np.where(error_mux_vector == error_id)[0]
error_vector = np.zeros(np.size(error_mux_vector),np.int8)
for i in range(0, np.size(index) - 1):
if (index[i + 1] - index[i]) <= 1 / period:
error_vector[index[i]:index[i + 1] + 1] = 1
return error_vector
#error_signal = np.array([0,0,0,0,0,1,2,3,1,2,3,2,3,2,3,0,0,0,0,0,1,1,1,2,3,1,3,1,3,1,3,0,0,0,0,0,2,0,2,2])
error_signal = np.random.randint(0,101,1000000)
import time
t=[]
t.append(time.time())
error_vector_1 = get_error_vector(error_signal, 1, 0.02)
error_vector_2 = get_error_vector(error_signal, 2, 0.02)
error_vector_3 = get_error_vector(error_signal, 3, 0.02)
t.append(time.time())
sol = demultiplex(error_signal,50)
t.append(time.time())
sol2 = demultiplex2(error_signal,50)
t.append(time.time())
print("time per error id [OP, pp, pp2]",np.diff(t)/(3,100,100))
print("results equal",end=" ")
print((error_vector_1==sol[0]).all(),end=" ")
print((error_vector_2==sol[1]).all(),end=" ")
print((error_vector_3==sol[2]).all(),end=" ")
print((error_vector_1==sol2[0]).all(),end=" ")
print((error_vector_2==sol2[1]).all(),end=" ")
print((error_vector_3==sol2[2]).all())
Sample run:
time per error id [OP, pp, pp2] [0.02730425 0.00912964 0.00440736]
results equal True True True True True True
BIt of explanation:
we argsort signal to easily identify those error codes which are followed by themselves in close enough time.
we arrange the error vectors in a stack so points that should be set can be addressed by coordinate signal[t],t
to set stretches of time points we set the first to 1 and the one after the last to -1 and form the cumsum - to remedy overlapping stretches we check >0 and cast the resulting boolean to int
I'm doing aperture photometry on a cluster of stars, and to get easier detection of background signal, I want to only look at stars further apart than n pixels (n=16 in my case).
I have 2 arrays, xs and ys, with the x- and y-values of all the stars' coordinates:
Using np.where I'm supposed to find the indexes of all stars, where the distance to all other stars is >= n
So far, my method has been a for-loop
import numpy as np
# Lists of coordinates w. values between 0 and 2000 for 5000 stars
xs = np.random.rand(5000)*2000
ys = np.random.rand(5000)*2000
# for-loop, wherein the np.where statement in question is situated
n = 16
for i in range(len(xs)):
index = np.where( np.sqrt( pow(xs[i] - xs,2) + pow(ys[i] - ys,2)) >= n)
Due to the stars being clustered pretty closely together, I expected a severe reduction in data, though even when I tried n=1000 I still had around 4000 datapoints left
Using just numpy (and part of the answer here)
X = np.random.rand(5000,2) * 2000
XX = np.einsum('ij, ij ->i', X, X)
D_squared = XX[:, None] + XX - 2 * X.dot(X.T)
out = np.where(D_squared.min(axis = 0) > n**2)
Using scipy.spatial.pdist
from scipy.spatial import pdist, squareform
D_squared = squareform(pdist(x, metric = 'sqeuclidean'))
out = np.where(D_squared.min(axis = 0) > n**2)
Using a KDTree for maximum fast:
from scipy.spatial import KDTree
X_tree = KDTree(X)
in_radius = np.array(list(X_tree.query_pairs(n))).flatten()
out = np.where(~np.in1d(np.arange(X.shape[0]), in_radius))
np.random.seed(seed=1)
xs = np.random.rand(5000,1)*2000
ys = np.random.rand(5000,1)*2000
n = 16
mask = (xs>=0)
for i in range(len(xs)):
if mask[i]:
index = np.where( np.sqrt( pow(xs[i] - x,2) + pow(ys[i] - y,2)) <= n)
mask[index] = False
mask[i] = True
x = xs[mask]
y = ys[mask]
print(len(x))
4220
You can use np.subtract.outer for creating the pairwise comparisons. Then you check for each row whether the distance is below 16 for exactly one item (which is the comparison with the particular start itself):
distances = np.sqrt(
np.subtract.outer(xs, xs)**2
+ np.subtract.outer(ys, ys)**2
)
indices = np.nonzero(np.sum(distances < 16, axis=1) == 1)
I am wondering why y is being returned as an array. If it helps, x is a numpy array.
I have not gotten to the bit where I have to store the y values in an array yet, but I am trying to find the average of N_obs_photon(t) over a finely scaled grid to give a smoothed graph of N_obs_photon(t) over a coarsely scaled grid. I think that the part with the sum(array)/len(array) should be giving me a float but instead it is returning an array (more correctly 1001 arrays). Even if I try setting x to some array with only one value, y is still being returned as an array. And how should I write it so that I am being given the average for y?
x = np.linspace(0,1,1001)
ftc = 11
L = 10**(-3)
i = x/L
j = np.arange(0,11001,1) #end number is max(i)*ftc + 1
'''reference:
fine_grid_number = j
coarse_grid_number = i
coase_grid_width = L
fine_grid_width = L/ftc #denominator is ftc
coarse_grid_position = i*L
fine_grid_position = j*L/ftc #denominator is ftc'''
#Plan is to create an array of y values, then plot x vs y
for n in i:
q = np.arange(-0.5*(ftc-1),0.5*ftc,1)
#temp_array = np.empty([])
temp_array = []
for w in q:
t = ftc*n + w #t is fine grid number
t = t*L/ftc #convert to fine grid position
if t < 0: #drop t when it would be less than zero
t = 0
temp_array.append(t) #add to array
else:
t = N_obs_photon(t) #take through function
temp_array.append(t) #add to array
y = sum(temp_array)/len(temp_array) #average the array
print(y) #test if y is a number
#store result in y array
I am trying to create a 3d tensor with the dimension (101, 2, 1000). I had to translate the R code I found to python and there is just a problem that r starts to iterate at 1 and python at zero.
Is there a way to solve this problem below?
Thanks in advance!
df1 = pd.DataFrame(np.random.rand(1000, 2), columns=['Col1', 'Col2'])
a = np.array(df1)
a_stdnorm = stats.norm.ppf(a)
n_rows = a.shape[0]
n_cols = a.shape[1]
samples = 100
if samples % 2 == 0:
samples = samples + 1 # force an odd number
samples_increment = samples - 1 # to cater for 1 based indices
tensor = np.zeros((samples, n_cols, n_rows))
sum_col = a[:,0] + a[:,1]
sort = np.argsort(sum_col)
block_half = samples // 2
start = 0
end = start + samples_increment
for n in range(n_rows):
if (n + 1) - block_half > 0 and n + block_half <= n_rows:
start = n - block_half
end = start + samples_increment
dx = sort[start:end]
data = a_stdnorm[dx,:]
tensor[:,:,sort[n]] = data
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-50-1b371b208d20> in <module>()
22 dx = sort[start:end]
23 all_data = a_stdnorm[dx,:]
---> 24 tensor[:,:,sort[n]] = all_data
ValueError: could not broadcast input array from shape (100,2) into shape (101,2)
The problem appears to be a size mismatch. Specifically, tensor has slices of size (101,2) while data ends up having size (100,2).
Removing these lines:
if samples % 2 == 0:
samples = samples + 1 # force an odd number
And just making this line do nothing:
samples_increment = samples # to cater for 1 based indices (change #1)
Where we don't do anything to make the number of samples odd seems to do the trick.
If you do need the number of samples to be odd, you'll first need to change this line not to do anything again:
samples_increment = samples # to cater for 1 based indices
Then you'll find that at the last iteration, the loop will fail. This is because we round down when computing block_half because of the odd number of examples. As such, you'll have to do a bit of massaging to update end properly for this case.
I tried to optimize the code below but I cannot figure out how to improve computation speed. I tried Cthon but the performance is like in python.
Is it possible to improve the performance without rewrite everything in C/C++?
Thanks for any help
import numpy as np
heightSequence = 400
widthSequence = 400
nHeights = 80
DOF = np.zeros((heightSequence, widthSequence), dtype = np.float64)
contrast = np.float64(np.random.rand(heightSequence, widthSequence, nHeights))
initDOF = np.zeros([heightSequence, widthSequence], dtype = np.float64)
initContrast = np.zeros([heightSequence, widthSequence, nHeights], dtype = np.float64)
initHeight = np.float64(np.r_[0:nHeights:1.0])
initPixelContrast = np.array(([0 for ii in range(nHeights)]), dtype = np.float64)
# for each row
for row in range(heightSequence):
# for each col
for col in range(widthSequence):
# initialize variables
height = initHeight # array ndim = 1
c = initPixelContrast # array ndim = 1
# for each height
for indexHeight in range(0, nHeights):
# get contrast profile for current pixel
tempC = contrast[:, :, indexHeight]
c[indexHeight] = tempC[row, col]
# save original contrast
# originalC = c
# originalHeight = height
# remove profile before maximum and after minumum contrast
idxMaxContrast = np.argmax(c)
c = c[idxMaxContrast:]
height = height[idxMaxContrast:]
idxMinContrast = np.argmin(c) + 1
c = c[0:idxMinContrast]
height = height[0:idxMinContrast]
# remove some refraction
if (len(c) <= 1) | (np.max(c) <= 0):
DOF[row, col] = 0
else:
# linear fitting of profile contrast
P = np.polyfit(height, c, 1)
m = P[0]
q = P[1]
# remove some refraction
if m >= 0:
DOF[row, col] = 0
else:
DOF[row, col] = -q / m
print 'row=%i/%i' %(row, heightSequence)
# set range of DOF
DOF[DOF < 0] = 0
DOF[DOF > nHeights] = 0
By looking at the code it seems that you can get rid of the two outer loops completely, converting the code to a vectorised form. However, the np.polyfit call must then be replaced by some other expression, but the coefficients for a linear fit are easy to find, also in vectorised form. The last if-else can then be turned into a np.where call.