Related
I am building a cholesky factorisation algorthim as proposed from the book:
Linear Algebra and Optimisation for Machine Learning
They provide the following algorthim:
I have attempted this with python with the following algorithm:
def CF(array):
array = np.array(array, float)
arr = np.linalg.eigvals(array)
for o in arr:
if o < 0:
return print('Cannot apply Cholesky Factorisation')
else:
continue
L = np.zeros_like(array)
n,_ = array.shape
for j in range(n):
sumkj = 0
for k in range(j):
sumkj += L[j,k]**2
print(array[j,j], sumkj)
L[j,j] = np.sqrt(array[j,j] - sumkj)
for i in range(j+1):
sumki = 0
for ki in range(i):
sumki += L[i,ki]
L[i,j] = (array[i,j] - sumki*sumkj)/L[j,j]
return L
When testing this on a matrix I get upper triangular as opposed to lower triangular. Where was I mistaken?
a = np.array([[4, 2, 1], [2, 6, 1], [2, 8, 2]])
CF(a)
>array([[2. , 0.81649658, 0.70710678],
[0. , 2.44948974, 0.70710678],
[0. , 0. , 1.41421356]])
You don't even need to look at the detail of the algorithm.
Just see that in your book algorithm you have
for j=1 to d
L(j,j) = something
for i=j+1 to d
L(i,j) = something
So, necessarily, all elements whose line number i is greater or equal than column number j are filled. Rest is 0. Hence a lower triangle
In your algorithm on another hand you have
for j in range(n):
L[j,j] = something()
for i in range(j+1):
L[i,j] = something()
So all elements whose line number i is < to column number j, +1 (since i is in range(j+1), that is i<j+1, that is i<=j) are filled with something. Rest is 0.
Hence upper triangle
You probably wanted to
for i in range(j+1, n)
Let's say I have a simple array, like this one:
import numpy as np
a = np.array([1,2,3])
Which returns me, obviously:
array([1, 2, 3])
I'm trying to add calculated values between consecutive values in this array. The calculation should return me n equally spaced values between it's bounds.
To express myself in numbers, let's say I want to add 1 value between each pair of consecutive values, so the function should return me a array like this:
array([1, 1.5, 2, 2.5, 3])
Another example, now with 2 values between each pair:
array([1, 1.33, 1.66, 2, 2.33, 2.66, 3])
I know the logic and I can create myself a function which will do the work, but I feel numpy has specific functions that would make my code so much cleaner!
If your array is
import numpy as np
n = 2
a = np.array([1,2,5])
new_size = a.size + (a.size - 1) * n
x = np.linspace(a.min(), a.max(), new_size)
xp = np.linspace(a.min(), a.max(), a.size)
fp = a
result = np.interp(x, xp, fp)
returns: array([1. , 1.33333333, 1.66666667, 2. , 2.66666667, 3.33333333, 4. ])
If your array is always evenly spaced, you can just use
new_size = a.size + (a.size - 1) * n
result = np.linspace(a.min(), a.max(), new_size)
Using linspace should do the trick:
a = np.array([1,2,3])
n = 1
temps = []
for i in range(1, len(a)):
temps.append(np.linspace(a[i-1], a[i], num=n+1, endpoint=False))
# Add last final ending point
temps.append(np.array([a[-1]]))
new_a = np.concatenate(temps)
print(new_a)
Try with np.arange:
a = np.array([1,2,3])
n = 2
print(np.arange(a.min(), a.max(), 1 / (n + 1)))
Output:
[1. 1.33333333 1.66666667 2. 2.33333333 2.66666667]
I have a numpy array and I need to mask it.
My function looks like this:
def mask_arr(arr, min, max):
for i in range(arr.size-1):
if arr[i] < min:
arr[i] = 0
elif arr[i] > max:
arr[i] = 1
else:
arr[i] = 10
Problem is the array is huge and it takes a long time to mask it.
How can i achieve the same result but faster?
You can use use nested np.where like the following:
import numpy as np
q = np.random.rand(4,4)
# array([[0.86305369, 0.88477713, 0.58776518, 0.69122533],
# [0.52591559, 0.33155238, 0.50139987, 0.66812239],
# [0.83240284, 0.70147098, 0.17118681, 0.59652636],
# [0.82031661, 0.32032657, 0.55088698, 0.28931661]])
np.where(q > 0.8, 1, np.where(q < 0.3, 0, 10))
# array([[ 1, 1, 10, 10],
# [10, 10, 10, 10],
# [ 1, 10, 0, 10],
# [ 1, 10, 10, 0]])
Edit:
Based on your question, In the case you want to change the value in the case the element of the array is not greater then maxVal or smaller then minVal you can do or any other logic that you want:
import numpy as np
q = q = np.random.rand(4,4)
minVal = 0.3
maxVal = 0.9
qq = np.where(q > 0.8, 1, np.where(q < 0.3, 0, 2 * q))
Where q is:
[[0.63604995 0.18637738 0.90680287 0.64617278]
[0.97435344 0.04670638 0.3510053 0.71613776]
[0.17973416 0.50296747 0.35085383 0.853201 ]
[0.27820978 0.69438172 0.96186074 0.96625938]]
And qq is:
[[1.27209991 0. 1. 1.29234556]
[1. 0. 0.7020106 1.43227553]
[0. 1.00593493 0.70170767 1. ]
[0. 1.38876345 1. 1. ]]
Solution
You could use three simple assignments based on your rules. This uses the native vectorization available in numpy and hence will be quite faster compared to what you have tried.
# minval, maxval = 0.3, 0.8
condition = np.logical_and(a>=minval, a<=maxval)
a[a<minval] = 0
a[a>maxval] = 1
a[condition] = 10 # if a constant value of 10
a[condition] *= 2 # if each element gets multiplied by 2
Output:
[[10. 0. 10. 1. 0.]
[10. 10. 10. 0. 10.]
[ 1. 10. 10. 1. 1.]
[ 0. 1. 10. 10. 0.]
[ 0. 0. 10. 10. 10.]]
Dummy Data
a = np.random.rand(5,5)
Output:
array([[0.68554168, 0.27430639, 0.4382025 , 0.97162651, 0.16740865],
[0.32530579, 0.3415287 , 0.45920916, 0.09422211, 0.75247522],
[0.91621921, 0.65845783, 0.38678723, 0.83644281, 0.95865701],
[0.26290637, 0.83810284, 0.55327399, 0.3406887 , 0.26173914],
[0.24974815, 0.08543414, 0.78509214, 0.64663201, 0.61502744]])
Convenience Function
Since you mentioned that you could also self-multiply the target elements by a factor of two, I extended that functionality to either absolute assignment (setting a value of 10) or relative update (add, subtract, multiply, divide) w.r.t the current values of the array.
def mask_arr(arr,
minval: float = 0.3,
maxval: float = 0.8,
update_type: str = 'abs',
update_value: float = 10,
rel_update_method: str = '*',
mask_floor: float = 0.0,
mesk_ceiling: float = 1.0):
"""Returns the array arr after setting lower-bound (mask_floor),
upper-bound (mask_ceiling), and logic-for-in-between-values.
"""
# minval, maxval = 0.3, 0.8
condition = np.logical_and(arr>=minval, arr<=maxval)
arr[arr<minval] = lowerbound
arr[arr>maxval] = upperbound
if update_type=='abs':
# absolute update
arr[condition] = update_value
if update_type=='rel':
# relative update
if rel_update_method=='+':
arr[condition] += update_value
if rel_update_method=='-':
arr[condition] -= update_value
if rel_update_method=='*':
arr[condition] *= update_value
if rel_update_method=='/':
arr[condition] /= update_value
return arr
Example
# declare all inputs
arr = mask_arr(arr,
minval = 0.3,
maxval = 0.8,
update_type = 'rel',
update_value = 2.0,
rel_update_method = '*',
mask_floor = 0.0,
mesk_ceiling = 1.0)
# using defaults for
# mask_floor = 0.0,
# mesk_ceiling = 1.0
arr = mask_arr(arr,
minval = 0.3,
maxval = 0.8,
update_type = 'rel',
update_value = 2.0,
rel_update_method = '*')
# using defaults as before and
# setting a fixed value of 10
arr = mask_arr(arr,
minval = 0.3,
maxval = 0.8,
update_type = 'abs',
update_value = 10.0)
With numpy you don't need to do loops for such operations.
More, I would recommend you not using 'min' and 'max' as variable names given they are reserved names.
Try the following
arr[arr < min_val]=0
arr[arr > max_val]=1
arr[(arr<=max_val) & (arr>=min_val)]=10
I have a list of indices
a = [
[1,2,4],
[0,2,3],
[1,3,4],
[0,2]]
What's the fastest way to convert this to a numpy array of ones, where each index shows the position where 1 would occur?
I.e. what I want is:
output = array([
[0,1,1,0,1],
[1,0,1,1,0],
[0,1,0,1,1],
[1,0,1,0,0]])
I know the max size of the array beforehand. I know I could loop through each list and insert a 1 into at each index position, but is there a faster/vectorized way to do this?
My use case could have thousands of rows/cols and I need to do this thousands of times, so the faster the better.
How about this:
ncol = 5
nrow = len(a)
out = np.zeros((nrow, ncol), int)
out[np.arange(nrow).repeat([*map(len,a)]), np.concatenate(a)] = 1
out
# array([[0, 1, 1, 0, 1],
# [1, 0, 1, 1, 0],
# [0, 1, 0, 1, 1],
# [1, 0, 1, 0, 0]])
Here are timings for a 1000x1000 binary array, note that I use an optimized version of the above, see function pp below:
pp 21.717635259992676 ms
ts 37.10938713003998 ms
u9 37.32933565042913 ms
Code to produce timings:
import itertools as it
import numpy as np
def make_data(n,m):
I,J = np.where(np.random.random((n,m))<np.random.random((n,1)))
return [*map(np.ndarray.tolist, np.split(J, I.searchsorted(np.arange(1,n))))]
def pp():
sz = np.fromiter(map(len,a),int,nrow)
out = np.zeros((nrow,ncol),int)
out[np.arange(nrow).repeat(sz),np.fromiter(it.chain.from_iterable(a),int,sz.sum())] = 1
return out
def ts():
out = np.zeros((nrow,ncol),int)
for i, ix in enumerate(a):
out[i][ix] = 1
return out
def u9():
out = np.zeros((nrow,ncol),int)
for i, (x, y) in enumerate(zip(a, out)):
y[x] = 1
out[i] = y
return out
nrow,ncol = 1000,1000
a = make_data(nrow,ncol)
from timeit import timeit
assert (pp()==ts()).all()
assert (pp()==u9()).all()
print("pp", timeit(pp,number=100)*10, "ms")
print("ts", timeit(ts,number=100)*10, "ms")
print("u9", timeit(u9,number=100)*10, "ms")
This might not be the fastest way. You will need to compare execution times of these answers using large arrays in order to find out the fastest way. Here's my solution
output = np.zeros((4,5))
for i, ix in enumerate(a):
output[i][ix] = 1
# output ->
# array([[0, 1, 1, 0, 1],
# [1, 0, 1, 1, 0],
# [0, 1, 0, 1, 1],
# [1, 0, 1, 0, 0]])
In case you can and want to use Cython you can create a readable (at least if you don't mind the typing) and fast solution.
Here I'm using the IPython bindings of Cython to compile it in a Jupyter notebook:
%load_ext cython
%%cython
cimport cython
cimport numpy as cnp
import numpy as np
#cython.boundscheck(False) # remove this if you cannot guarantee that nrow/ncol are correct
#cython.wraparound(False)
cpdef cnp.int_t[:, :] mseifert(list a, int nrow, int ncol):
cdef cnp.int_t[:, :] out = np.zeros([nrow, ncol], dtype=int)
cdef list subl
cdef int row_idx
cdef int col_idx
for row_idx, subl in enumerate(a):
for col_idx in subl:
out[row_idx, col_idx] = 1
return out
To compare the performance of the solutions presented here I use my library simple_benchmark:
Note that this uses logarithmic axis to simultaneously show the differences for small and large arrays. According to my benchmark my function is actually the fastest of the solutions, however it's also worth pointing out that all of the solutions aren't too far off.
Here is the complete code I used for the benchmark:
import numpy as np
from simple_benchmark import BenchmarkBuilder, MultiArgument
import itertools
b = BenchmarkBuilder()
#b.add_function()
def pp(a, nrow, ncol):
sz = np.fromiter(map(len, a), int, nrow)
out = np.zeros((nrow, ncol), int)
out[np.arange(nrow).repeat(sz), np.fromiter(itertools.chain.from_iterable(a), int, sz.sum())] = 1
return out
#b.add_function()
def ts(a, nrow, ncol):
out = np.zeros((nrow, ncol), int)
for i, ix in enumerate(a):
out[i][ix] = 1
return out
#b.add_function()
def u9(a, nrow, ncol):
out = np.zeros((nrow, ncol), int)
for i, (x, y) in enumerate(zip(a, out)):
y[x] = 1
out[i] = y
return out
b.add_functions([mseifert])
#b.add_arguments("number of rows/columns")
def argument_provider():
for n in range(2, 13):
ncols = 2**n
a = [
sorted(set(np.random.randint(0, ncols, size=np.random.randint(0, ncols))))
for _ in range(ncols)
]
yield ncols, MultiArgument([a, ncols, ncols])
r = b.run()
r.plot()
May not be the best way but the only way I can think of:
output = np.zeros((4,5))
for i, (x, y) in enumerate(zip(a, output)):
y[x] = 1
output[i] = y
print(output)
Which outputs:
[[ 0. 1. 1. 0. 1.]
[ 1. 0. 1. 1. 0.]
[ 0. 1. 0. 1. 1.]
[ 1. 0. 1. 0. 0.]]
How about using array indexing? If you knew more about your input, you could get rid of the penalty for having to convert to a linear array first.
import numpy as np
def main():
row_count = 4
col_count = 5
a = [[1,2,4],[0,2,3],[1,3,4],[0,2]]
# iterate through each row, concatenate all indices and convert them to linear
# numpy append performs copy even if you don't want it, list append is faster
b = []
for row_idx, row in enumerate(a):
b.append(np.array(row, dtype=np.int64) + (row_idx * col_count))
linear_idxs = np.hstack(b)
#could skip previous steps if given index inputs well before hand, or in linear index order.
c = np.zeros(row_count * col_count)
c[linear_idxs] = 1
c = c.reshape(row_count, col_count)
print(c)
if __name__ == "__main__":
main()
#output
# [[0. 1. 1. 0. 1.]
# [1. 0. 1. 1. 0.]
# [0. 1. 0. 1. 1.]
# [1. 0. 1. 0. 0.]]
Depending on your use case, you might look into using sparse matrices. The input matrix looks suspiciously like a Compressed Sparse Row (CSR) matrix. Perhaps something like
import numpy as np
from scipy.sparse import csr_matrix
from itertools import accumulate
def ragged2csr(inds):
offset = len(inds[0])
lens = [len(x) for x in inds]
indptr = list(accumulate(lens))
indptr = np.array([x - offset for x in indptr])
indices = np.array([val for sublist in inds for val in sublist])
n = indices.size
data = np.ones(n)
return csr_matrix((data, indices, indptr))
Again, if it fits in your use case, a sparse matrix would allow elementwise/masking operations to scale with the number of nonzeros, rather than the number of elements (rows*columns), which could bring significant speedup (for a sparse enough matrix).
Another good introduction to CSR matrices is section 3.4 of Iterative Methods. In this case, data is aa, indices is ja and indptr is ia. This format also has the benefit of being very popular among different packages/libraries.
I a very new in Python. Please, apologize if the question is too simple. I have a function which return the standard deviation from the surrounding pixels of a principal pixel, something like*:
def sliding_window(arr, window_size):
""" Construct a sliding window view of the array"""
arr = np.asarray(arr)
window_size = int(window_size)
if arr.ndim != 2:
raise ValueError("need 2-D input")
if not (window_size > 0):
raise ValueError("need a positive window size")
shape = (arr.shape[0] - window_size + 1,
arr.shape[1] - window_size + 1,
window_size, window_size)
if shape[0] <= 0:
shape = (1, shape[1], arr.shape[0], shape[3])
if shape[1] <= 0:
shape = (shape[0], 1, shape[2], arr.shape[1])
strides = (arr.shape[1]*arr.itemsize, arr.itemsize,
arr.shape[1]*arr.itemsize, arr.itemsize)
return as_strided(arr, shape=shape, strides=strides)
def std(arr, i, j, d):
"""Return d-th neighbors of cell (i, j)"""
w = sliding_window(arr, 2*d+1)
ix = np.clip(i - d, 0, w.shape[0]-1)
jx = np.clip(j - d, 0, w.shape[1]-1)
i0 = max(0, i - d - ix)
j0 = max(0, j - d - jx)
i1 = w.shape[2] - max(0, d - i + ix)
j1 = w.shape[3] - max(0, d - j + jx)
return nu.std(w[ix, jx][i0:i1,j0:j1].ravel())
Now I want to apply this function to each element of the array and get as result an array with the same structure:
For example:
array = [[2,3,4,4],
[3,4,3,5],
[4,5,6,6],
[3,6,7,7]]
formula = [[std(array, 0,0,2), std(array, 1,0,2),std(array, 2,0,2),std(array, 3,0,2)],
[std(array, 0,1,2), std(array, 1,1,2),std(array, 2,1,2),std(array, 3,1,2)],
[std(array, 0,2,2), std(array, 1,2,2),std(array, 2,2,2),std(array, 3,2,2)],
[std(array, 0,3,2), std(array, 1,3,2),std(array, 2,3,2),std(array, 3,3,2)]]
result = [[0.70710678118654757, 0.9574271077563381, 1.1989578808281798, 1.0671873729054748],
[0.68718427093627676, 1.1331154474650633, 1.4624940645653537, 1.4229164972072998],
[0.8660254037844386, 1.1873172373979173, 1.5, 1.4409680388158819],
[0.68718427093627676, 1.0657403385139377, 1.35400640077266, 1.2570787221094177]]
I was trying to make a loop. Something like:
For k, v in array[i][j]:
sd(array, i,j, n)
But until now loops are very frustrating....I hope you can help me.
from here
You can do it with
array = [[2,3,4,4],
[3,4,3,5],
[4,5,6,6],
[3,6,7,7]]
print [[[std(array,x,i,2) for x in xrange(len(array[i]))] for i in xrange(len(array))]]
# Result
[[[0.70710678118654757, 0.9574271077563381, 1.1989578808281798, 1.0671873729054748], [0.68718427093627676, 1.1331154474650633, 1.4624940645653537, 1.4229164972072998], [0.8660254037844386, 1.1873172373979173, 1.5, 1.4409680388158819], [0.68718427093627676, 1.0657403385139377, 1.35400640077266, 1.2570787221094177]]]
It looks like you can use the scipy function 'ndimage.generic_filter'.
We give it a footprint (your window_size), and map a function (np.nanstd) onto a 1d array of each item which matches in that footprint.
Because we have to worry about borders, we can give use np.nanstd, and pad the array with nans (cval = np.nan).
import numpy as np
import scipy.ndimage as ndimage
results = np.empty(shape = (4,4), dtype = 'float') #to avoid type conversion
footprint = np.ones((2,2)) #change for your window size
#footprint[0,0] = 0 #not sure if this is intended
array = [[2,3,4,4],
[3,4,3,5],
[4,5,6,6],
[3,6,7,7]]
ndimage.generic_filter(array, np.nanstd, footprint=footprint, mode = 'constant', cval= np.nan, output = results, origin = -1)
results
array([[ 0.70710678, 0.5 , 0.70710678, 0.5 ],
[ 0.70710678, 1.11803399, 1.22474487, 0.5 ],
[ 1.11803399, 0.70710678, 0.5 , 0.5 ],
[ 1.5 , 0.5 , 0. , 0. ]])
The results are different from yours - not sure if I have something wrong in the understanding, or the footprint/origin/function.