Related
strong text
I have matrix (3-d array)
strong text
"a" and "b" shape: (5, 3, depths), depths == can varaible, in this example is 2, but sometimes could be 3 or 4 or 6, I am looking for a function that works with different depths.
Blockquote
a= [[[10,15,10,9,45], [2,21,78,14,96], [2,2,78,14,96], [3,34,52,87,21], [52,14,45,85,74] ], [[52,14,45,85,74], [2,2,78,14,96], [15,41,48,48,74], [3,34,52,87,21], [14,71,84,85,41]]]
Blockquote
b= [[[0,1,0,1,1], [2,2,1,0,0], [2,2,1,1,0], [0,0,0,0,1], [0,0,1,1,1] ], [[0,0,0,1,1], [0,1,1,1,2], [2,2,2,2,0], [0,0,0,1,1], [1,0,0,0,1]]]
strong text
I want a matrix "c", "c" should be the copy of "a", but when a value in "b" is == 0, "c" will also be == 0
Blockquote
c= [[[0,15,0,9,45], [2,21,78,0,0], [2,21,78,14,0], [0,0,0,0,21], [0,0,45,85,74] ], [[0,0,0,85,74], [0,2,78,14,96], [15,41,48,48,0], [0,0,0,87,21], [14,0,0,0,41]]]
strong text
thank for yourl help
Use numpy arrays and element-wise multiplication:
import numpy as np
a= [[[10,15,10,9,45], [2,21,78,14,96], [2,2,78,14,96], [3,34,52,87,21], [52,14,45,85,74] ], [[52,14,45,85,74], [2,2,78,14,96], [15,41,48,48,74], [3,34,52,87,21], [14,71,84,85,41]]]
b= [[[0,1,0,1,1], [2,2,1,0,0], [2,2,1,1,0], [0,0,0,0,1], [0,0,1,1,1] ], [[0,0,0,1,1], [0,1,1,1,2], [2,2,2,2,0], [0,0,0,1,1], [1,0,0,0,1]]]
result = np.array(a) * np.array(b)
result = result.tolist() # if you want the final result as a list
print(result)
[[[0, 15, 0, 9, 45], [4, 42, 78, 0, 0], [4, 4, 78, 14, 0], [0, 0, 0, 0, 21], [0, 0, 45, 85, 74]], [[0, 0, 0, 85, 74], [0, 2, 78, 14, 192], [30, 82, 96, 96, 0], [0, 0, 0, 87, 21], [14, 0, 0, 0, 41]]]
Note : In the question, you are talking about 5x3 blocks, but your example is 5x5. i'll assume the correct format is 5x5xDepth
Without using numpy
Let's define our depth and a result blockquote
depth = len(a)
result = []
So we can iterate trought our blockquote :
for x in range(depth):
# This is a 5 x 5 array
2d_block = []
for y in range(5):
# This is our final dimension, array of lengh 5
1d_block = []
for z in range(5):
# Check if b is 0
if b[x][y][z] == 0:
1d_block.append(0)
else:
1d_block.append(a[x][y][z])
# Add our block to the current 2D block
2d_block.append(1d_block)
# Add our blocks to the result
result.append(2d_block)
Recursive alternative
A more advanced solution
def convert_list(a, b):
if isinstance(a, list):
# Recursive call on all sub list
return [convert_list(a[i], b[i]) for i in range(len(a))]
else
# When we find an element, return a if b is not 0
return a if b!=0 else 0
This is a recursive function so you don't need to mind about the dimensions of your blockquote (as look as a and b have the same lengh)
Using numpy
Inspired by msamsami anwser, with a step to convert all non-zero numbers in b to 1 to avoid multiplying the result (zeros stay 0 so we can filter a values)
# Convert b to an array with 0 and 1, to avoid multiplying by 2
def toBinary(item):
if isinstance(item, list):
return [toBinary(x) for x in item]
else:
return item != 0
filter = toBinary(b)
result = np.array(a) * np.array(filter)
I am trying to solve the following system of linear equations:
10x1+ 40x2+ 70x3= 300
20x1+ 50x2+ 80x3= 360
30x1+ 60x2+ 80x3= 390
by using Cramer's method implementing a function by scratch:
def cramer(mat, constant): # takes the matrix and the costants
D = np.linalg.det(mat) # calculating the determinant of the original matrix
# substitution of the column with costant and creating new matrix
mat1 = np.array([constant, mat[1], mat[2]])
mat2 = np.array([mat[0], constant, mat[2]])
mat3 = np.array([mat[0], mat[1], constant])
#calculatin determinant of the matrix
D1 = np.linalg.det(mat1)
D2 = np.linalg.det(mat2)
D3 = np.linalg.det(mat3)
#finding the X1, X2, X3
X1 = D1/D
X2 = D2/D
X3 = D3/D
print(X1, X2, X3)
By using the above function on the system
a = np.array([[10, 40, 70],
[20, 50, 80],
[30, 60, 80]])
b = np.array([300, 360, 390])
I get the following result:
cramer(a,b)
-22.99999999999996 21.999999999999964 2.999999999999993
I have solved the system using the numpy function np.linalg.solve and I get another result:
x = np.linalg.solve(a, b)
[1. 2. 3.]
I cannot spot the formula error in the function I have witten. What should I adjust in the fuction in order to make it working properly?
The main problem is how you select the columns of a, i.e. you are actually selecting the rows of a rather than the columns. You can fix it by changing the matrix initializations to this:
mat1 = np.array([constant, mat[:,1], mat[:,2]])
mat2 = np.array([mat[:,0], constant, mat[:,2]])
mat3 = np.array([mat[:,0], mat[:,1], constant])
Basically mat[:,1] is saying something like mat[all rows, column 1].
TL;DR Optimal solution at the bottom.
To fix your current solution you need to use the second dimensions and pass all three matrices to compute determinants together (this way you will get stable floating point values):
def cramer(mat, constant):
D = np.linalg.det(mat)
mat1 = np.array([constant, mat[:, 1], mat[:, 2]])
mat2 = np.array([mat[:, 0], constant, mat[:, 2]])
mat3 = np.array([mat[:, 0], mat[:, 1], constant])
Dx = np.linalg.det([mat1, mat2, mat3])
X = Dx/D
print(X)
However, you don't need to create all these matrices one by one either. Instead, use a series of numpy manipulations described below.
First, create the mask to so you can then use it to replace values in a by values from b:
>>> mask = np.broadcast_to(np.diag([1,1,1]), [3, 3, 3]).swapaxes(0, 1)
array([[[1, 0, 0],
[1, 0, 0],
[1, 0, 0]],
[[0, 1, 0],
[0, 1, 0],
[0, 1, 0]],
[[0, 0, 1],
[0, 0, 1],
[0, 0, 1]]])
Then use np.where to get three matrices, each with one column replaced by b:
>>> Ms = np.where(mask, np.repeat(b, 3).reshape(3, 3), a)
array([[[300, 40, 70],
[360, 50, 80],
[390, 60, 80]],
[[ 10, 300, 70],
[ 20, 360, 80],
[ 30, 390, 80]],
[[ 10, 40, 300],
[ 20, 50, 360],
[ 30, 60, 390]]])
Then, compute three determinants and divide the determinant of a itself:
>>> np.linalg.det(Ms) / np.linalg.det(a)
array([1., 2., 3.])
Putting it all together:
def cramer(a, b):
mask = np.broadcast_to(np.diag([1,1,1]), [3, 3, 3]).swapaxes(0, 1)
Ms = np.where(mask, np.repeat(b, 3).reshape(3, 3), a)
return np.linalg.det(Ms) / np.linalg.det(a)
I'm working in python using numpy (could be a pandas series too) and am trying to make the following calculation:
Lets say I have an array corresponding to points on the x axis:
2, 9, 5, 6, 55, 8
For each element in this array I would like to get the distance to the closest element so the output would look like the following:
3, 1, 1, 1, 46, 1
I am trying to find a solution that can scale to 2D (distance to nearest XY point) and ideally would avoid a for loop. Is that possible?
There seems to be a theme with O(N^2) solutions here. For 1D, it's quite simple to get O(N log N):
x = np.array([2, 9, 5, 6, 55, 8])
i = np.argsort(x)
dist = np.diff(x[i])
min_dist = np.r_[dist[0], np.minimum(dist[1:], dist[:-1]), dist[-1]])
min_dist = min_dist[np.argsort(i)]
This clearly won't scale well to multiple dimensions, so use scipy.special.KDTree instead. Assuming your data is N-dimensional and has shape (M, N), you can do
k = KDTree(data)
dist = k.query(data, k=2)[0][:, -1]
Scipy has a Cython implementation of KDTree, cKDTree. Sklearn has a sklearn.neighbors.KDTree with a similar interface as well.
Approach 1
You can use broadcasting in order to get matrix of distances:
>>> data = np.array([2,9,5,6,55,8])
>>> dst_matrix = data - data[:, None]
>>> dst_matrix
array([[ 0, 7, 3, 4, 53, 6],
[ -7, 0, -4, -3, 46, -1],
[ -3, 4, 0, 1, 50, 3],
[ -4, 3, -1, 0, 49, 2],
[-53, -46, -50, -49, 0, -47],
[ -6, 1, -3, -2, 47, 0]])
Then we can eliminate diagonal as proposed in this post:
dst_matrix = dst_matrix[~np.eye(dst_matrix.shape[0],dtype=bool)].reshape(dst_matrix.shape[0],-1)
>>> dst_matrix
array([[ 7, 3, 4, 53, 6],
[ -7, -4, -3, 46, -1],
[ -3, 4, 1, 50, 3],
[ -4, 3, -1, 49, 2],
[-53, -46, -50, -49, -47],
[ -6, 1, -3, -2, 47]])
Finally, mininum items can be found:
>>> np.min(np.abs(dst_matrix), axis=1)
array([ 3, 1, 1, 1, 46, 1])
Approach 2
If you're looking for time and memory efficient solution, the best option is scipy.spatial.cKDTrees which packs points (of any dimension) into specific data structure that is optimized for querying closest points. It can also be extended to 2D or 3D.
import scipy.spatial
data = np.array([2,9,5,6,55,8])
ckdtree = scipy.spatial.cKDTree(data[:,None])
distances, idx = ckdtree.query(data[:,None], k=2)
output = distances[:,1] #distances to not coincident points
For each point querying first two closest points is required here because first of them is expected to be coincident. This is the only solution I found between all the proposed answers that doesn't take ages (the average performance is 4secs for 1M points). Warning: you need to filter duplicated points before applying this method.
There are many ways of achieving it. Some readable and generalizable ways are:
Approach 1:
dist = np.abs(a[:,None]-a)
np.min(dist, where=~np.eye(len(a),dtype=bool), initial=dist.max(), axis=1)
#[ 3 1 1 1 46 1]
Approach 2:
dist = np.abs(np.subtract.outer(a,a))
np.min(dist, where=~np.eye(len(a),dtype=bool), initial=dist.max(), axis=1)
For a 2-D case approach 1 (assumes Euclidean distance. Any other is also possible):
from scipy.spatial.distance import cdist
dist = cdist(a,a)
np.min(dist, where=~np.eye(len(a),dtype=bool), initial=dist.max(), axis=1)
For a 2-D case approach 2 with numpy only:
dist=np.sqrt(((a[:,None]-a)**2).sum(-1))
np.min(dist, where=~np.eye(len(a),dtype=bool), initial=dist.max(), axis=1)
You can achieve a faster distance calculation by using np.dot.
You can do some list comprehension on a pandas series:
s = pd.Series([2,9,5,6,55,8])
s.apply(lambda x: min([abs(x - s[y]) for y in s.index if s[y] != x]))
Out[1]:
0 3
1 1
2 1
3 1
4 46
5 1
Then you can just add .to_list() or .to_numpy() to the end to get rid of the series index:
s.apply(lambda x: min([abs(x - s[y]) for y in s.index if s[y] != x])).to_numpy()
array([ 3, 1, 1, 1, 46, 1], dtype=int64)
I have a numpy array such as this
[[ 0, 57],
[ 7, 72],
[ 2, 51],
[ 8, 67],
[ 4, 42]]
I want to find out for each row, how many elements in the 2nd column are within a certain distance (say, 10) of the 2nd column value for that row. So in this example, here the solution would be
[[ 0, 57, 3],
[ 7, 72, 2],
[ 2, 51, 3],
[ 8, 67, 3],
[ 4, 42, 2]]
So [first row, third column] is 3, because there are 3 elements in the 2nd column (57,51,67) which are within distance 10 from 57. Similarly for each row
Any help would be appreciated!
Here's one approach leveraging broadcasting with outer-subtraction -
(np.abs(a[:,1,None] - a[:,1]) <= 10).sum(1)
With outer subtract builtin and count_nonzero for counting -
np.count_nonzero(np.abs(np.subtract.outer(a[:,1],a[:,1]))<=10,axis=1)
Sample run -
# Input array
In [23]: a
Out[23]:
array([[ 0, 57],
[ 7, 72],
[ 2, 51],
[ 8, 67],
[ 4, 42]])
# Get count
In [24]: count = (np.abs(a[:,1,None] - a[:,1]) <= 10).sum(1)
In [25]: count
Out[25]: array([3, 2, 3, 3, 2])
# Stack with input
In [26]: np.c_[a,count]
Out[26]:
array([[ 0, 57, 3],
[ 7, 72, 2],
[ 2, 51, 3],
[ 8, 67, 3],
[ 4, 42, 2]])
Alternatively with SciPy's cdist -
In [53]: from scipy.spatial.distance import cdist
In [54]: (cdist(a[:,None,1],a[:,1,None], 'minkowski', p=2)<=10).sum(1)
Out[54]: array([3, 2, 3, 3, 2])
For million rows in the input, we might want to resort to a loopy one -
n = len(a)
count = np.empty(n, dtype=int)
for i in range(n):
count[i] = np.count_nonzero(np.abs(a[:,1]-a[i,1])<=10)
Here's a non-broadcasting approach, which takes advantage of the fact that to know how many numbers are within 3 of 10, you can subtract the number of numbers <= 13 from those strictly less than 7.
import numpy as np
def broadcast(x, width):
# for comparison
return (np.abs(x[:,None] - x) <= width).sum(1)
def largest_leq(arr, x, allow_equal=True):
maybe = np.searchsorted(arr, x)
maybe = maybe.clip(0, len(arr) - 1)
above = arr[maybe] > x if allow_equal else arr[maybe] >= x
maybe[above] -= 1
return maybe
def faster(x, width):
uniq, inv, counts = np.unique(x, return_counts=True, return_inverse=True)
counts = counts.cumsum()
low_bounds = uniq - width
low_ix = largest_leq(uniq, low_bounds, allow_equal=False)
low_counts = counts[low_ix]
low_counts[low_ix < 0] = 0
high_bounds = uniq + width
high_counts = counts[largest_leq(uniq, high_bounds)]
delta = high_counts - low_counts
out = delta[inv]
return out
This passes my tests:
for width in range(1, 10):
for window in range(5):
for trial in range(10):
x = np.random.randint(0, 10, width)
b = broadcast(x, window).tolist()
f = faster(x, window).tolist()
assert b == f
and behaves pretty well even at larger sizes:
In [171]: x = np.random.random(10**6)
In [172]: %time faster(x, 0)
Wall time: 386 ms
Out[172]: array([1, 1, 1, ..., 1, 1, 1], dtype=int64)
In [173]: %time faster(x, 1)
Wall time: 372 ms
Out[173]: array([1000000, 1000000, 1000000, ..., 1000000, 1000000, 1000000], dtype=int64)
In [174]: x = np.random.randint(0, 10, 10**6)
In [175]: %timeit faster(x, 3)
10 loops, best of 3: 83 ms per loop
I have some data that looks like:
data = [1,2,4,5,9] (random pattern of increasing integers)
And I want to plot it in a binary horizontal line so that y=1 for every x value specified in data and zero otherwise.
I have a few different data arrays that I'd like to stack, similar to this style (this is CCD clocking data but the plot format looks ideal)
I think I need to create a list of ones for my data array, but how do I specify the zero value for everything not in the array?
Thanks
You got the point. You can create a list with 1 in any position specified in data and 0 elsewhere. This can be done very easily with a list comprehension
def binary_data(data):
return [1 if x in data else 0 for x in range(data[-1] + 1)]
which will act like this:
>>> data = [1, 2, 4, 5, 9]
>>> bindata = binary_data(data)
>>> bindata
[0, 1, 1, 0, 1, 1, 0, 0, 0, 1]
Now all you have to do is plot it... or better step it since it's binary data and step() looks way better:
import numpy as np
from matplotlib.pyplot import step, show
def binary_data(data):
return [1 if x in data else 0 for x in range(data[-1] + 1)]
data = [1, 2, 4, 5, 9]
bindata = binary_data(data)
xaxis = np.arange(0, data[-1] + 1)
yaxis = np.array(bindata)
step(xaxis, yaxis)
show()
To plot multiple data arrays stacked on the same figure you could tweak binary_data() like this:
def binary_data(data, yshift=0):
return [yshift+1 if x in data else yshift for x in range(data[-1] + 1)]
so now you can set yshift parameter to shift data arrays on the y-axis. E.g,
>>> data = [1, 2, 4, 5, 9]
>>> bindata1 = binary_data(data)
>>> bindata1
[0, 1, 1, 0, 1, 1, 0, 0, 0, 1]
>>> bindata2 = binary_data(data, 2)
>>> bindata2
[2, 3, 3, 2, 3, 3, 2, 2, 2, 3]
Let's say you have data1, data2 and data3 to plot stacked, you'd go like:
import numpy as np
from matplotlib.pyplot import step, show
def binary_data(data, yshift=0):
return [yshift+1 if x in data else yshift for x in range(data[-1] + 1)]
data1 = [1, 2, 4, 5, 9]
bindata1 = binary_data(data1)
x1 = np.arange(0, data1[-1] + 1)
y1 = np.array(bindata1)
data2 = [1, 4, 9]
bindata2 = binary_data(data2, 2)
x2 = np.arange(0, data2[-1] + 1)
y2 = np.array(bindata2)
data3 = [1, 2, 8, 9]
bindata3 = binary_data(data3, 4)
x3 = np.arange(0, data3[-1] + 1)
y3 = np.array(bindata3)
step(x1, y1, x2, y2, x3, y3)
show()
that you can easily edit to make it work with an arbitrary amount of data arrays:
data = [ [1, 2, 4, 5, 9],
[1, 4, 9],
[1, 2, 8, 9] ]
for shift, d in enumerate(data):
bindata = binary_data(d, 2 * shift)
x = np.arange(0, d[-1] + 1)
y = np.array(bindata)
step(x, y)
show()
Finally if you are dealing with data arrays with different length (say [1,2] and [15,16]) and you don't like plots that vanish in the middle of the figure you can tweak binary_data() again to force its range to the maximum range of your data.
import numpy as np
from matplotlib.pyplot import step, show
def binary_data(data, limit, yshift=0):
return [yshift+1 if x in data else yshift for x in range(limit)]
data = [ [1, 2, 4, 5, 9, 12, 13, 14],
[1, 4, 10, 11, 20, 21, 22],
[1, 2, 3, 4, 15, 16, 17, 18] ]
# find out the longest data to plot
limit = max( [ x[-1] + 1 for x in data] )
x = np.arange(0, limit)
for shift, d in enumerate(data):
bindata = binary_data(d, limit, 2 * shift)
y = np.array(bindata)
step(x, y)
show()
Edit: As #ImportanceOfBeingErnest suggested, if you prefer to perform data to bindata conversion without having to define your own binary_data() function you could use numpy.zeros_like(). Just pay more attention when you stack them:
import numpy as np
from matplotlib.pyplot import step, show
data = [ [1, 2, 4, 5, 9, 12, 13, 14],
[1, 4, 10, 11, 20, 21, 22],
[1, 2, 3, 4, 15, 16, 17, 18] ]
# find out the longest data to plot
limit = max( [ x[-1] + 1 for x in data] )
x = np.arange(0, limit)
for shift, d in enumerate(data):
y = np.zeros_like(x)
y[d] = 1
# don't forget to shift
y += 2*shift
step(x, y)
show()
You can create an array with all zeros and assign 1 for those elements in data
import numpy as np
data = [1,2,4,5,9]
t = np.arange(0,data[-1]+1)
x = np.zeros_like(t)
x[data] = 1
You might then plot it with the step function
import matplotlib.pyplot as plt
plt.step(t,x, where="post")
plt.show()
or with where = "pre", depending on how to interprete your data