The following simple python code:
class Node:
NumberOfNodes = 0
def __init__(self):
Node.NumberOfNodes += 1
if __name__ == '__main__':
nodes = []
for i in xrange(1, 7 * 1000 * 1000):
if i % 1000 == 0:
print i
nodes.append(Node())
takes gigabytes of memory;
Which I think is irrational. Is that normal in python?
How could I fix that?(in my original code, I have about 7 million objects each with 10 fields and that takes 8 gigabytes of RAM)
If you have fixed number of fields then you can use __slots__ to save quite a lot of memory. Note that __slots__ do have some limitations, so make sure your read the Notes on using __slots__ carefully before choosing to use them in your application:
>>> import sys
>>> class Node(object):
NumberOfNodes = 0
def __init__(self):
Node.NumberOfNodes += 1
...
>>> n = Node()
>>> sys.getsizeof(n)
64
>>> class Node(object):
__slots__ = ()
NumberOfNodes = 0
def __init__(self):
Node.NumberOfNodes += 1
...
>>> n = Node()
>>> sys.getsizeof(n)
16
Python is an inherently memory heavy programming language. There are some ways you can get around this. __slots__ is one way. Another, more extreme approach is to use numpy to store your data. You can use numpy to create a structured array or record -- a complex data type that uses minimal memory, but suffers a substantial loss of functionality compared to a normal python class. That is, you are working with the numpy array class, rather than your own class -- you cannot define your own methods on your array.
import numpy as np
# data type for a record with three 32-bit ints called x, y and z
dtype = [(name, np.int32) for name in 'xyz']
arr = np.zeros(1000, dtype=dtype)
# access member of x of a record
arr[0]['x'] = 1 # name based access
# or
assert arr[0][0] == 1 # index based access
# accessing all x members of records in array
assert arr['x'].sum() == 1
# size of array used to store elements in memory
assert arr.nbytes == 12000 # 1000 elements * 3 members * 4 bytes per int
See more here.
Related
I am working on a framework for processing incoming data.
The data is received from a socket and added to numpy an array A (used as buffer) using shifting, sth like:
A[:-1] = A[1:]
A[-1] = value
The framework allows loading processing units as classes that have an access to incoming data using array view pointing to A. Everytime new data is received and stored in A, a method execute() is called:
def execute(self,):
newSample = self.data[-1]
What is important is that new sample is always under index = -1.
A user can also create his own array views in __init__ function:
def __init__(self,):
self.myData = self.data[-4:] # view that contains last 4 samples
Everything works nicely when I am shifting array A and adding new value at the end. However, for offline testing, I want to load all the data at the start of the framework and run everything else as before (i.e. the same classes implementing data processing).
Of course, I can again create A buffer using zeros array and shift it with new values. However, this involves copying of data between two arrays that is absolutely not necessary - takes time and memory.
What I was thinking about is to provide a way to change the boundaries of the numpy array or change A.data pointer. However, all the solutions are not allowed or lead to the warning message.
Finally, I am trying to change an internal offset of array A, so that I can advance it and thus make more data available for algorithms. What is important, self.data[-1] has to always point to the newly appeared sample and standard numpy array API should be used.
I have subclassed np.ndarray:
class MyArrayView(np.ndarray):
def __new__(cls, input_array):
obj = np.asarray(input_array).view(cls)
# add the new attribute to the created instance
obj._offset = 0
# Finally, we must return the newly created object:
return obj
def __array_finalize__(self, obj):
if obj is None:
return
self._offset = getattr(obj, '_offset', None)
def advance_index(self):
self._offset += 1
def __str__(self):
return super(MyArrayView, self[:]).__str__()
def __repr__(self):
return super(MyArrayView, self[:]).__repr__()
def __getitem__(self, idx):
if isinstance(idx, slice):
start = 0
stop = self._offset
step = idx.step
idx = slice(start, stop, step)
else:
idx = self._offset + idx
return super(MyArrayView, self).__getitem__(idx)
that allows me to do the following:
a = np.array([1,2,3,4,5,6,7,8,9,10])
myA = MyArrayView(a)
b = myA
print("b :", b)
for i in range(1,5):
myA.advance_index()
print(b[:], b[-1])
print("b :", b)
print("b + 10 :", b + 10)
print("b[:] + 20 :", b[:] + 20)
and gives following output:
b : []
[1] 1
[1 2] 2
[1 2 3] 3
[1 2 3 4] 4
b : [1 2 3 4]
b + 10 : [11 12 13 14]
b[:] + 20 : [21 22 23 24]
so far so good. However if I check the shape:
print("shape", b[:].shape) # shape (4,)
print("shape", b.shape) # shape (10,)
it is different in those two cases. I have tried to change it using: shape=(self.internalIndex,) but it leads me only to an error message.
I want to ask if you think this is the right way what I am doing and it only requires to overload more functions in a np.ndarray class. Or should I completely abandon this solution and fallback to shifting array with a new sample? Or is it may be possible to be achieved using standard np.ndarray implementation as I need to use standard numpy API.
I also tried this:
a = np.array([1,2,3,4,5,6,7,8,9,10])
b = a.view()[5:]
print(a.data) # <memory at 0x7f09e01d8f48>
print(b.data) # <memory at 0x7f09e01d8f48> They point to the same memory start!
print(np.byte_bounds(a)) # (50237824, 50237904)
print(np.byte_bounds(b)) # (50237864, 50237904) but the byte_bounds are different
So having this in mind, I would say I need to create a view of array a and extend it (or at least move it like a window on top of a).
However, all my tries to change the byte_bounds did not bring any effects.
I admire your bravery, but am quite sure sub-classing numpy arrays is overkill for your problem and can cause you a huge lot of headache. In the end it might cause a performance hit somewhere that by far outruns the array copying you are trying to avoid.
Why not make the slice (i.e. [-4:] or slice(-4, None)) a parameter to your __init__ function or a class attribute and override that in your test?
def __init__(self, lastfour=slice(-4, None)):
self.myData = self.data[lastfour]
I have been modelling a stochastic process with Python and Numpy and witnessing weird behavior with the following code:
import numpy as np
class Example( object ):
def __init__( self ):
self.x = 0
def add_random( self ):
self.x += np.random.randn(1)
return self.x
if __name__ == '__main__':
example = Example()
state = []
for x in range(10):
state.append( example.add_random() )
print state
This will return an array of 10 identical random numbers as opposed to 10 different random numbers as expected. Eliminating the object.__iadd__ operator and/or replacing np.random.randn(.) with a constant will solve the issue. Anybody has an idea what is the root of this?
np.random.randn(1) returns an array containing a single element:
In [27]: np.random.randn(1)
Out[27]: array([-1.90409169])
The first time this line is executed
self.x += np.random.randn(1)
self.x--initially a Python integer--is replaced by a numpy array. Subsequent execution of that line modifies x in-place, because that's how numpy arrays implement in-place addition. So return self.x always returns the same array. Thus the list that you create in the main section is a list containing the same object repeated 10 times.
One way to fix this is to use np.random.randn() instead of np.random.randn(1). np.random.randn() returns a scalar, so the assignment self.x += np.random.randn(1) creates a new self.x object each time it is executed.
You are returning the array self.x by reference, so you have 10 pointers to the same array. Every time the array is modified, all ten pointers point to the same modified version.
If you want to return separate copies of the array each time, you could return self.x.copy() in the add_random function.
An alternative fix would be to replace np.random.rand(1) with np.random.rand() so self.x would remain a scalar rather than being up-cast to an array.
I've got this code
class coordenates:
x = 0
y = 0
coor = coordenates()
coor.x=0
coor.y=0
list = []
list.append(coor)
list.append(coor)
Now, the problem is that when I update
list[0].x=100
it is also modifing list[1].x somehow!
print str(list[0].x)
>> 100
print str(list[1].x)
>> 100
which must remain in 0 since I haven't update it. Is append() creating the same object pointing in the same position in memory in positions 0 and 1? why creating 2 different objects solves the problem?
In your current code, x and y are class-level attributes. I suspect that you want them to be instance-level attributes. If so, set them in __init__():
class Coordinates:
def __init__(self):
self.x = 0
self.y = 0
More important, if you append the same coor to the list twice, any mutation of the coor will be reflected in "both" coordinates (because the list is just holding a reference to the same underlying coordinate in both positions of the list). Maybe you want something like this instead, where you create two independent coordinate instances?
list = []
list.append(Coordinates())
list.append(Coordinates())
You can see an illustration of your problem with this code:
c = Coordinates()
cs = []
cs.append(c)
cs.append(c)
for c in cs:
print id(c) # Both elements of the list refer to the same object.
I'm trying to create efficient broadcast arrays in numpy, e.g. a set of shape=[1000,1000,1000] arrays that have only 1000 elements, but repeated 1e6 times. This can be achieved both through np.lib.stride_tricks.as_strided and np.broadcast_arrays.
However, I am having trouble verifying that there is no duplication in memory, and this is critical since tests that actually duplicate the arrays in memory tend to crash my machine leaving no traceback.
I've tried examining the size of the arrays using .nbytes, but that doesn't seem to correspond to the actual memory usage:
>>> import numpy as np
>>> import resource
>>> initial_memuse = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
>>> pagesize = resource.getpagesize()
>>>
>>> x = np.arange(1000)
>>> memuse_x = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
>>> print("Size of x = {0} MB".format(x.nbytes/1e6))
Size of x = 0.008 MB
>>> print("Memory used = {0} MB".format((memuse_x-initial_memuse)*resource.getpagesize()/1e6))
Memory used = 150.994944 MB
>>>
>>> y = np.lib.stride_tricks.as_strided(x, [1000,10,10], strides=x.strides + (0, 0))
>>> memuse_y = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
>>> print("Size of y = {0} MB".format(y.nbytes/1e6))
Size of y = 0.8 MB
>>> print("Memory used = {0} MB".format((memuse_y-memuse_x)*resource.getpagesize()/1e6))
Memory used = 201.326592 MB
>>>
>>> z = np.lib.stride_tricks.as_strided(x, [1000,100,100], strides=x.strides + (0, 0))
>>> memuse_z = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
>>> print("Size of z = {0} MB".format(z.nbytes/1e6))
Size of z = 80.0 MB
>>> print("Memory used = {0} MB".format((memuse_z-memuse_y)*resource.getpagesize()/1e6))
Memory used = 0.0 MB
So .nbytes reports the "theoretical" size of the array, but apparently not the actual size. The resource checking is a little awkward, as it looks like there are some things being loaded & cached (perhaps?) that result in the first striding taking up some amount of memory, but future strides take none.
tl;dr: How do you determine the actual size of a numpy array or array view in memory?
One way would be to examine the .base attribute of the array, which references the object from which an array "borrows" its memory. For example:
x = np.arange(1000)
print(x.flags.owndata) # x "owns" its data
# True
print(x.base is None) # its base is therefore 'None'
# True
a = x.reshape(100, 10) # a is a reshaped view onto x
print(a.flags.owndata) # it therefore "borrows" its data
# False
print(a.base is x) # its .base is x
# True
Things are slightly more complicated with np.lib.stride_tricks:
b = np.lib.stride_tricks.as_strided(x, [1000,100,100], strides=x.strides + (0, 0))
print(b.flags.owndata)
# False
print(b.base)
# <numpy.lib.stride_tricks.DummyArray object at 0x7fb40c02b0f0>
Here, b.base is a numpy.lib.stride_tricks.DummyArray instance, which looks like this:
class DummyArray(object):
"""Dummy object that just exists to hang __array_interface__ dictionaries
and possibly keep alive a reference to a base array.
"""
def __init__(self, interface, base=None):
self.__array_interface__ = interface
self.base = base
We can therefore examine b.base.base:
print(b.base.base is x)
# True
Once you have the base array then its .nbytes attribute should accurately reflect the amount of memory it occupies.
In principle it's possible to have a view of a view of an array, or to create a strided array from another strided array. Assuming that your view or strided array is ultimately backed by another numpy array, you could recursively reference its .base attribute. Once you find an object whose .base is None, you have found the underlying object from which your array is borrowing its memory:
def find_base_nbytes(obj):
if obj.base is not None:
return find_base_nbytes(obj.base)
return obj.nbytes
As expected,
print(find_base_nbytes(x))
# 8000
print(find_base_nbytes(y))
# 8000
print(find_base_nbytes(z))
# 8000
I am trying to make an class = that extends from list return a slice of itself instead of a list type. The reason I want to do this is because I have many other methods to manipulate the instance of A.
I am running python 2.7.3
Say I have:
class B():
def __init__(self, t, p):
self.t = t
self.p = p
class Alist(list):
def __init__(self, a_list_of_times = []):
for a_time in a_list_of_times:
self.append(a_time )
def __getslice__(self, i, j):
return super(Alist, self).__getslice__(i,j)
def plot_me(self):
pass
# other code goes here!
alist1 = Alist()
for i in range(0, 1000000):
alist1.append(B(i, i)) # yes ten million, very large list!
alist = alist1[1000:200000] # will return a list!
alist2 = Alist(alist) # will return Alist istance
The problem is that remaking the entire list as seen in making variable b is VERY VERY SLOW (comparative to the slice). What I want to do is simply change the class of alist (currently of type list)to Alist
When I try:
alist.__class__ = Alist
>>>> TypeError: __class__ assignment: only for heap types.
Which is very sad since I can do this for my own object types.
I understand that it is not standard, but it is done.
Reclassing an instance in Python.
Is there a way around this? Also I have obviously simplified the problem, where my objects a bit more complex. Mainly what I am finding is that remaking the list into my Alist version is slow. And I need to do this operation a lot (unavoidable). Is there a way to remake A? or a solution to this to make it speed up?
In my version, I can do about a 10,000 (size of my slice) slice in 0.07 seconds, and converting it to my version of Alist takes 3 seconds.
The UserList class (moved to collections in Python 3) is perfectly designed for this. It is a list by all other means but has a data attribute that you can store an underlying list in without copying.
from UserList import UserList
class Alist(UserList):
def __init__(self, iterable, copy=True):
if copy:
super(Alist, self).__init__(iterable)
else:
self.data = iterable
def plot_me(self):
pass