I'm working on a project where I need to load a large number of numpy arrays saved on the disk using a for loop. The system I'm using is Linux.
The image below shows the memory usage during the process
As you can see, the part of unavailable memory under buff/cache can be even larger than the used memory. What is saved on this part of the memory? How can I reduce it?
The script used for loading the arrays is something like this:
import numpy as np
tmp = []
slice1, slice2 = [], []
for item in hashes:
# np.load(item) has a shape (50, 96)
tmp.append(np.load(item))
tmp = np.concatenate(tmp, axis=0)
mask1 = # a mask used for slicing, a third of the entries will be selected
mask2 = # a different mask for slicing, a third of the entries will be selected
slice1 = tmp[mask1]
slice2 = tmp[mask2]
Based on the short code segment shown, you may be converting Numpy ndarray objects into list objects while manipulating them. Try using all Numpy objects and methods. Also try to avoid for loops and use Numpy vectorized operations instead. 56GB is a huge amount of memory. Yikes! :-)
Possible Numpy codes:
import numpy as np
tmp = np.array([0, 1, 2, 3, 4], ndim=2)
tmp = np.zeros((50, 96))
load_object = np.load(filename)
tmp = np.array(load_object)
# slice1 = [mask range 1]
# slice2 = [mask range 2]
# Numpy vertical stack
result = np.vstack((tmp[slice1], tmp[slice2]))
Numpy Vertical Stack method docs:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.vstack.html
a = np.array([1, 2, 3])
b = np.array([2, 3, 4])
np.vstack((a,b))
(output)
array([[1, 2, 3],
[2, 3, 4]])
Hopefully that will solve your problem. I will do some more testing later, and edit my answer.
Is there a simpler and more memory efficient way to do the following in numpy alone.
import numpy as np
ar = np.array(a[l:r])
ar += c
a = a[0:l] + ar.tolist() + a[r:]
It may look primitive but it involves obtaining a subarray copy of the given array, then prepare two more copies of the same to append in left and right direction in addition to the scalar add. I was hoping to find some more optimized way of doing this. I would like a solution that is completely in Python list or NumPy array, but not both as converting from one form to another as shown above would cause serious overhead when the data is huge.
You can just do the assignment inplace as follows:
import numpy as np
a = np.array([1, 1, 1, 1, 1])
a[2:4] += 5
>>> a
array([1, 1, 6, 6, 1])
I am trying to do inverse of numpy array,
from numpy import mat
from numpy import *
from numpy import matrix
from numpy import linalg
d =array ([ (0, 1, 2, 3, 4),
( 5, 6, 7, 8, 9),
(10, 11, 12, 13, 14)])
print d.T
print d.I
print d.diagonal
#above line gives <built-in method diagonal of numpy.ndarray object at 0x7fdf40a263f0>
print numpy.linalg.inv(d)
I am getting
AttributeError: 'numpy.ndarray' object has no attribute 'I'
any suggestion for this to get inverse and diagonal?
I would suggest changing all these imports:
from numpy import mat
from numpy import *
from numpy import matrix
from numpy import linalg
to just one:
import numpy as np
Then you can do
d = np.array(...)
# d = np.arange(15).reshape(3,5)
M = np.matrix(d)
M.I # a matrix has an I property, but an array does not
d.diagonal() # diagonal is a method, not a property
np.diagonal(d) # diagonal is also a function
np.linalg.inv(d) does not work - it gives an error objecting that the array is not square. So evidently M.I is returning a different kind of inverse. See my note below about pinv.
numpy (and Python in general) has functions, methods, and properties (attributes). Are the distinctions clear?
Many numpy functions end up calling the corresponding method for the main array object. Usually that doesn't matter much, except as a calling convenience.
np.matrix is one subclass that has many of its own methods. Note, for example
In [817]: M.diagonal()
Out[817]: matrix([[ 0, 6, 12]])
In [818]: d.diagonal()
Out[818]: array([ 0, 6, 12])
diagonal has returned the same numbers, but for M, it returns an object of the same class, and which by class definition is 2d.
d.T, M.I access properties. These don't require the () that a method does, but in many ways they are the same. np.matrix has defined I, but np.array has not.
If you are used to working with matrices in MATLAB, the np.matrix class may ease the transition. But if this is your first experience with arrays like this, I'd suggest sticking the np.array. The np.matrix class will just add confusion.
M.I is the same as M.getI().
It's code is (use help(M.getI) to read its docs)
def getI(self):
M, N = self.shape
if M == N:
from numpy.dual import inv as func
else:
from numpy.dual import pinv as func
return asmatrix(func(self))
So that means that M.I use pinv rather than inv. np.linalg.pinv(d) works.
You will get these methods if you transform d to a np.matrix, by doing:
d = np.matrix(d)
then:
d.I
d.diagonal()
will work.
A.diagonal is a method of numpy.ndarray, just as the print out suggests. Therefore, the solution of #Saullo Castro works for numpy arrays as well, without the need to convert to np.matrix.
import numpy as np
A = np.arange(25).reshape((5,5))
diag = A.diagonal()
# array([ 0, 6, 12, 18, 24])
Numpy Arrays have no method to calculate the inverse of a matrix, but you can easily do that with numpy.linalg.inv, just as you already tried according to your code example.
Let me preface this post by saying that I'm pretty new to Python and NumPy, so I'm sure I'm overlooking something simple. What I'm trying to do is image processing over a PGM (grayscale) file using a mask (a mask convolution operation); however, I don't want to do it using the SciPy all-in-one imaging processing libraries that are available—I'm trying to implement the masking and processing operations myself. What I want to do is the following:
Iterate a 3x3 sliding window over a 256x256 array
At each iteration, I want to perform an operation with a 3x3 image mask (array that consists of fractional values < 1 ) and the 3x3 window from my original array
The operation is that the image mask gets multiplied by the 3x3 window, and that the results get summed up into one number, which represents a weighted average of the original 3x3 area
This sum should get inserted back into the center of the 3x3 window, with the original surrounding values left untouched
However, the output of one of these operations shouldn't be the input of the next operation, so a new array should be created or the original 256x256 array shouldn't be updated until all operations have completed.
The process is sort of like this, except I need to put the result of the convolved feature back into the center of the window it came from:
(source: stanford.edu)
So, in this above example, the 4 would go back into the center position of the 3x3 window it came from (after all operations had concluded), so it would look like [[1, 1, 1], [0, 4, 1], [0, 0, 1]] and so on for every other convolved feature obtained. A non-referential copy could also be made of the original and this new value inserted into that.
So, this is what I've done so far: I have a 256x256 2D numpy array which is my source image. Using as_strided, I convert it into a 4D numpy array of 3x3 slices. The main problem I'm facing is that I want to execute the operation I've specified over each slice. I'm able to perform it on one slice, but in npsum operations I've tried, it adds up all the slices' results into one value. After this, I either want to create a new 256x256 array with the results, in the fashion that I've described, or iterate over the original, replacing the middle values of each 3x3 window as appropriate. I've tried using ndenumerate to change just the same value (v, x, 1, 1) of my 4D array each time, but since the index returned from my 4D array is of the form (v, x, y, z), I can't seem to figure out how to only iterate through (v, x) and leave the last two parts as constants that shouldn't change at all.
Here's my code thus far:
import numpy as np
from numpy.lib import stride_tricks
# create 256x256 NumPy 2D array from image data and image size so we can manipulate the image data, then create a 4D array of strided windows
# currently, it's only creating taking 10 slices to test with
imageDataArray = np.array(parsedPGMFile.imageData, dtype=int).reshape(parsedPGMFile.numRows, parsedPGMFile.numColumns)
xx = stride_tricks.as_strided(imageDataArray, shape=(1, 10, 3, 3), strides=imageDataArray.strides + imageDataArray.strides)
# create the image mask to be used
mask = [1,2,1,2,4,2,1,2,1]
mask = np.array(mask, dtype=float).reshape(3, 3)/16
# this will execute the operation on just the first 3x3 element of xx, but need to figure out how to iterate through all elements and perform this operation individually on each element
result = np.sum(mask * xx[0,0])
Research from sources like http://wiki.scipy.org/Cookbook/GameOfLifeStrides, http://www.johnvinyard.com/blog/?p=268, and http://chintaksheth.wordpress.com/2013/07/31/numpy-the-tricks-of-the-trade-part-ii/ were very helpful (as well as SO), but they don't seem to address what I'm trying to do exactly (unless I'm missing something obvious). I could probably use a ton of for loops, but I'd rather learn how to do it using these awesome Python libraries we have. I also realize I'm combining a few questions together, but that's only because I have the sneaking suspicion that this can all be done very simply! Thanks in advance for any help!
When you need to multiply element-wise, then reduce with addition, think np.dot or np.einsum:
from numpy.lib.stride_tricks import as_strided
arr = np.random.rand(256, 256)
mask = np.random.rand(3, 3)
arr_view = as_strided(arr, shape=(254, 254, 3, 3), strides=arr.strides*2)
arr[1:-1, 1:-1] = np.einsum('ijkl,kl->ij', arr_view, mask)
Based on the example illustration:
In [1]: import numpy as np
In [2]: from scipy.signal import convolve2d
In [3]: image = np.array([[1,1,1,0,0],[0,1,1,1,0],[0,0,1,1,1],[0,0,1,1,0],[0,1,1,0,0]])
In [4]: m = np.array([[1,0,1],[0,1,0],[1,0,1]])
In [5]: convolve2d(image, m, mode='valid')
Out[5]:
array([[4, 3, 4],
[2, 4, 3],
[2, 3, 4]])
And putting it back where it came from:
In [6]: image[1:-1,1:-1] = convolve2d(image, m, mode='valid')
In [7]: image
Out[7]:
array([[1, 1, 1, 0, 0],
[0, 4, 3, 4, 0],
[0, 2, 4, 3, 1],
[0, 2, 3, 4, 0],
[0, 1, 1, 0, 0]])
For example, I have a variable which point to a vector contains many elements in memory, I want to copy element in vector to a numpy array, what should I do except one by one copy? Thx
I am assuming that your vector can be represented like that:-
import array
x = array('l', [1, 3, 10, 5, 6]) # an array using python's built-in array module
Casting it as a numpy array will then be:-
import numpy as np
y = np.array(x)
If the data is packed in a buffer in native float format:
a = numpy.fromstring(buf, dtype=float, count=N)