I'm having trouble finding the proper way to do something I think should be trivial using numpy. I have an array (1000x1000) and I want to calculate the sum of a specific pattern across the array.
For example:
If I have this array and want to calculate the sum of a two-cell-right diagonal I would expect [7,12,11,8,12,6,11,7] (a total of 8 sums).
How can I do this?
This operation is called a 2-dimensional convolution:
>>> import numpy as np
>>> from scipy.signal import convolve2d
>>> kernel = np.eye(2, dtype=int)
>>> a = np.array([[5,3,7,1,2],[3,2,9,4,7],[8,9,4,2,3]])
>>> convolve2d(a, kernel, mode='valid')
array([[ 7, 12, 11, 8],
[12, 6, 11, 7]])
Should you want to generalize it to arbitrary dimensions, there is also scipy.ndimage.convolve available. It will also work for this 2d case, but does not offer the mode='valid' convenience.
l = [[5,3,7,1,2],[3,2,9,4,7],[8,9,4,2,3]]
[q+l[w+1][t+1] for w,i in enumerate(l[:-1]) for t,q in enumerate(i[:-1])]
then you can avoid using numpy :) and the output is
[7,12,11,8,12,6,11,7]
Related
I'm looking to implement a hardware-efficient multiplication of a list of large matrices (on the order of 200,000 x 200,000). The matrices are very nearly the identity matrix, but with some elements changed to irrational numbers.
In an effort to reduce the memory footprint and make the computation go faster, I want to store the 0s and 1s of the identity as single bytes like so.
import numpy as np
size = 200000
large_matrix = np.identity(size, dtype=uint8)
and just change a few elements to a different data type.
import sympy as sp
# sympy object
irr1 = sp.sqrt(2)
# float
irr2 = e
large_matrix[123456, 100456] = irr1
large_matirx[100456, 123456] = irr2
Is is possible to hold only these elements of the matrix with a different data type, while all the other elements are still bytes? I don't want to have to change everything to a float just because I need one element to be a float.
-----Edit-----
If it's not possible in numpy, then how can I find a solution without numpy?
Maybe you can have a look at the SciPy's Coordinate-based sparse matrix. In that case SciPy creates a sparse matrix (optimized for such large empty matrices) and with its coordinate format you can access and modify the data as you intend.
From its documentation:
>>> from scipy.sparse import coo_matrix
>>> # Constructing a matrix using ijv format
>>> row = np.array([0, 3, 1, 0])
>>> col = np.array([0, 3, 1, 2])
>>> data = np.array([4, 5, 7, 9])
>>> m = coo_matrix((data, (row, col)), shape=(4, 4))
>>> m.toarray()
array([[4, 0, 9, 0],
[0, 7, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 5]])
It does not create a matrix but a set of coordinates with values, which takes much less space than just filling a matrix with zeros.
>>> from sys import getsizeof
>>> getsizeof(m)
56
>>> getsizeof(m.toarray())
176
By definition, NumPy arrays only have one dtype. You can see in the NumPy documentation:
A numpy array is homogeneous, and contains elements described by a dtype object. A dtype object can be constructed from different combinations of fundamental numeric types.
Further reading: https://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.html
I'm not certain of the best way of asking this question, so I apologize ahead of time.
I'm trying find a peak on each row of an NxM numpy array of audio signals. Each row in the array is treated individually and I'd like to get all values a certain number of standard deviations above the noise floor for each N in the array in frequency space. In this experiment I know that I do not have a signal above 400Hz so I'm using that as my noise floor. I'm running into issues when trying to mask. Here is my code snippet:
from scipy import signal
import numpy as np
Pxx_den = signal.periodogram(input, fs=sampleRate ,nfft=sampleRate,axis=1)
p = np.array(Pxx_den)[1].astype(np.float)
noiseFloor = np.mean(p[:,400:],axis=1)
stdFloor = np.std(p[:,400:],axis=1)
p = np.ma.masked_less(p,noiseFloor+stdFloor*2)
This example will generate an error of:
ValueError: operands could not be broadcast together with shapes (91,5001) (91,)
I've deduced that this is because ma.masked_less works with a single value and does not take in an array. I would like the output to be an NxM array of values greater than the condition. Is there a Numpy way of doing what I'd like or an efficient alternative?
I've also looked at some peak detection routines such as peakUtils and scipy.signal.find_peaks_cwt() but they seem to only act on 1D arrays.
Thanks in advance
Before getting too far into using masked arrays makes sure that the following code handles them. It has to be aware of how masked arrays works, or defer to masked array methods.
As to the specific problem, I think this recreates it:
In [612]: x=np.arange(10).reshape(2,5)
In [613]: np.ma.masked_less(x,np.array([3,6]))
...
ValueError: operands could not be broadcast together with shapes (2,5) (2,)
I have a 2d array, and I try to apply the < mask with different values for each row.
Instead I can generate the mask as a 2d array matching x:
In [627]: mask= x<np.array([3,6])[:,None]
In [628]: np.ma.masked_where(mask,x)
Out[628]:
masked_array(data =
[[-- -- -- 3 4]
[-- 6 7 8 9]],
mask =
[[ True True True False False]
[ True False False False False]],
fill_value = 999999)
I can also select the values, though in a way that looses the 2d structure.
In [631]: x[~mask]
Out[631]: array([3, 4, 6, 7, 8, 9])
In [632]: np.ma.masked_where(mask,x).compressed()
Out[632]: array([3, 4, 6, 7, 8, 9])
I am trying to do inverse of numpy array,
from numpy import mat
from numpy import *
from numpy import matrix
from numpy import linalg
d =array ([ (0, 1, 2, 3, 4),
( 5, 6, 7, 8, 9),
(10, 11, 12, 13, 14)])
print d.T
print d.I
print d.diagonal
#above line gives <built-in method diagonal of numpy.ndarray object at 0x7fdf40a263f0>
print numpy.linalg.inv(d)
I am getting
AttributeError: 'numpy.ndarray' object has no attribute 'I'
any suggestion for this to get inverse and diagonal?
I would suggest changing all these imports:
from numpy import mat
from numpy import *
from numpy import matrix
from numpy import linalg
to just one:
import numpy as np
Then you can do
d = np.array(...)
# d = np.arange(15).reshape(3,5)
M = np.matrix(d)
M.I # a matrix has an I property, but an array does not
d.diagonal() # diagonal is a method, not a property
np.diagonal(d) # diagonal is also a function
np.linalg.inv(d) does not work - it gives an error objecting that the array is not square. So evidently M.I is returning a different kind of inverse. See my note below about pinv.
numpy (and Python in general) has functions, methods, and properties (attributes). Are the distinctions clear?
Many numpy functions end up calling the corresponding method for the main array object. Usually that doesn't matter much, except as a calling convenience.
np.matrix is one subclass that has many of its own methods. Note, for example
In [817]: M.diagonal()
Out[817]: matrix([[ 0, 6, 12]])
In [818]: d.diagonal()
Out[818]: array([ 0, 6, 12])
diagonal has returned the same numbers, but for M, it returns an object of the same class, and which by class definition is 2d.
d.T, M.I access properties. These don't require the () that a method does, but in many ways they are the same. np.matrix has defined I, but np.array has not.
If you are used to working with matrices in MATLAB, the np.matrix class may ease the transition. But if this is your first experience with arrays like this, I'd suggest sticking the np.array. The np.matrix class will just add confusion.
M.I is the same as M.getI().
It's code is (use help(M.getI) to read its docs)
def getI(self):
M, N = self.shape
if M == N:
from numpy.dual import inv as func
else:
from numpy.dual import pinv as func
return asmatrix(func(self))
So that means that M.I use pinv rather than inv. np.linalg.pinv(d) works.
You will get these methods if you transform d to a np.matrix, by doing:
d = np.matrix(d)
then:
d.I
d.diagonal()
will work.
A.diagonal is a method of numpy.ndarray, just as the print out suggests. Therefore, the solution of #Saullo Castro works for numpy arrays as well, without the need to convert to np.matrix.
import numpy as np
A = np.arange(25).reshape((5,5))
diag = A.diagonal()
# array([ 0, 6, 12, 18, 24])
Numpy Arrays have no method to calculate the inverse of a matrix, but you can easily do that with numpy.linalg.inv, just as you already tried according to your code example.
I have a simple, one dimensional Python array with random numbers. What I want to do is convert it into a numpy Matrix of a specific shape. My current attempt looks like this:
randomWeights = []
for i in range(80):
randomWeights.append(random.uniform(-1, 1))
W = np.mat(randomWeights)
W.reshape(8,10)
Unfortunately it always creates a matrix of the form:
[[random1, random2, random3, ...]]
So only the first element of one dimension gets used and the reshape command has no effect. Is there a way to convert the 1D array to a matrix so that the first x items will be row 1 of the matrix, the next x items will be row 2 and so on?
Basically this would be the intended shape:
[[1, 2, 3, 4, 5, 6, 7, 8],
[9, 10, 11, ... , 16],
[..., 800]]
I suppose I can always build a new matrix in the desired form manually by parsing through the input array. But I'd like to know if there is a simpler, more eleganz solution with built-in functions I'm not seeing. If I have to build those matrices manually I'll have a ton of extra work in other areas of the code since all my source data comes in simple 1D arrays but will be computed as matrices.
reshape() doesn't reshape in place, you need to assign the result:
>>> W = W.reshape(8,10)
>>> W.shape
(8,10)
You can use W.resize(), ndarray.resize()
I have some numpy/scipy issue. I have a 3D array that represent an ellipsoid in a binary way [ 0 out of the ellipsoid].
The thing is I would like to rotate my shape of a certain degree. Do you think it's possible ?
Or is there an efficient way to write directly the ellipsoid equation with the rotation ?
Just a short answer. If you need more informations or you don't know how to do it, then I will edit this post and add a small example.
The right way to rotate your matrix of data points is to do a matrix multiplication. Your rotation matrix would be probably an n*n-matrix and you have to multiply it with every point. If you have your 3d-matrix you have some thing like i*j*k-points for plotting. This means for your case you have to do it i*j*k-times to find the new points. Maybe you should consider an other matrix for plotting which is just a 2D matrix and just store the plotting points and no zero values.
There are some algorithm to calculate faster the results for low valued matrix, but just google for this.
Did you understood me or do you still have some questions? Sorry for this rough overview.
Best regards
Take a look at the command numpy.shape
I used it once to transpose an array, but I don't know if it might fit your needs.
Cheers!
rotating by a non rectangular degree is tricky, because the rotated square does no longer fit in the matrix.
The simplest way would be to transpose a 2D array/matrix by:
import numpy as np
... x = np.array([[1,2,3],[4,5,6],[7,8,9]])
x
array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
x.T # transpose the array
array([[1, 4, 7],
[2, 5, 8],
[3, 6, 9]])
if you require an explicit array rotation check the ndimage package from scipy library:
from scipy import ndimage, datasets
img = datasets.ascent()
img_45 = ndimage.rotate(img, 45, reshape=False)