We Keep Coding

Python- Size of int, float etc from sys.getsizeof()

I am trying to compare sizes of data types in Python with sys.getsizeof(). However, for integers and floats, it returns same - 24 (not customary 4 or 8 bytes). Also, size of an array declared with array.array() with 4 integer elements is returned 72 (not 96). and with 4 float elements- 88 (not 96). What is going on?
import array, sys
arr1 = array.array('d', [1,2,3,4])
arr2 = array.array('i', [1,2,3,4])
print sys.getsizeof(arr1[1]), sys.getsizeof(arr2[1]) # 24, 24
print sys.getsizeof(arr1), sys.getsizeof(arr2) # 88, 72

The function sys.getsizeof() returns the amount of space the Python object takes. Not the amount of space you would need to represent the data in that object in the memory of the underlying system.
Python objects have overhead to cover reference counting (for garbage collection) and other implementation-related stuff. In addition, an array is not a naive sequence of floats or ints; the data structure has a fair amount of stuff under the hood that keeps track of datatype, number of elements and so on. That's where the 'd' or 'i' lives, for example.
To get the answers I think you are expecting, try
print (arr1.itemsize * len(arr1))
print (arr2.itemsize * len(arr2))

Size of objects in memory during an IPython session (with Guppy?)

I recall reading that it is hard to pin down the exact memory usage of objects in Python. However, that thread is from 2009, and since then I have read about various memory profilers in Python (see the examples in this thread). Also, IPython has matured substantially in recent months (version 1.0 was released a few days ago)
IPython already has a magic called whos, that prints the variable names, their types and some basic Data/Info.
In a similar fashion, is there any way to get the size in memory of each of the objects returned by who
? Any utilities available for this purpose already in IPython?
Using Guppy
Guppy (suggested in this thread) has a command that allows one to get the cummulative memory usage per object type, but unfortunately:
It does not show memory usage per object
It prints the sizes in bytes (not in human readable format)
For the second one, it may be possible to apply bytes2human from this answer, but I would need to first collect the output of h.heap() in a format that I can parse.
But for the first one (the most important one), is there any way to have Guppy show memory usage per object?
In [6]: import guppy
In [7]: h = guppy.hpy()
In [8]: h.heap()
Out[8]:
Partition of a set of 2871824 objects. Total size = 359064216 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 522453 18 151469304 42 151469304 42 dict (no owner)
1 451503 16 36120240 10 187589544 52 numpy.ndarray
2 425700 15 34056000 9 221645544 62 sklearn.grid_search._CVScoreTuple
3 193439 7 26904688 7 248550232 69 unicode
4 191061 7 22696072 6 271246304 76 str
5 751128 26 18027072 5 289273376 81 numpy.float64
6 31160 1 12235584 3 301508960 84 list
7 106035 4 9441640 3 310950600 87 tuple
8 3300 0 7260000 2 318210600 89 dict of 0xb8670d0
9 1255 0 3788968 1 321999568 90 dict of module
<1716 more rows. Type e.g. '_.more' to view.>

Why not use something like:
h.heap().byid
But this will only show you immediate sizes (i.e. not the total size of a list including the other lists it might refer to).
If you have a particular object you wish to get the size of you can use:
h.iso(object).domisize
To find the approximate amount of memory that would freed if it were deleted.

How to find total amount of memory used by python process/object in windows

I have some script that loads a lot of data to memory.
I want to know how efficient the data stored in memory.
So, I want to be able to know how many memory was used by python before I loaded data, and after I loaded data.
Also I wondering, if it is some way to check memory usage of complex object.
Let say i have nested dictionary with different types of data inside. How can i know how many memory used by all data in this dictionary.
Thanks,
Alex

As far as I know there is no easy way to see what the memory consumption of a certain object is. It would be a non-trivial thing to do because references could be shared among objects.
Here are my two favourite workarounds:
Use the process manager. Have the program pause for before allocation. Write down the memory used before allocation. Allocate. Write down memory after allocation. It's a low-tech method but it works.
Alternatively you can use pickle.dump to serialize your data structure. The resulting pickle will be comparable (not identical!) in size to the space needed to store the data structure in memory. For better results, use the binary pickle protocol.

In order to analyze how much memory an object uses, you could use Pympler:
>>> from pympler import asizeof
>>> obj = dict(nested=dict(trash=[1,2,3]))
>>> asizeof.asizeof(obj)
800
>>> asizeof.asizeof(obj['nested'])
480
>>> asizeof.asizeof(obj['nested']['trash'])
160
>>> asizeof.asizeof(obj['nested']['trash'][0])
24

You can take a look at the guppy package, which can give you informations about memory used by every loaded object. Unfortunately, it doesn't seem to work under python>=2.6, but it's good if you are using at most python 2.5.
Its usage is really simple, just put these lines in your code, where you want to collect memory informations:
from guppy import hpy
hp = hpy()
print hp.heap()
Which will give you an output like this:
Partition of a set of 25961 objects. Total size = 1894868 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 11901 46 775408 41 775408 41 str
1 6040 23 219964 12 995372 53 tuple
2 1718 7 116824 6 1112196 59 types.CodeType
3 73 0 113608 6 1225804 65 dict of module
4 348 1 107232 6 1333036 70 dict (no owner)
5 196 1 100192 5 1433228 76 dict of type
6 1643 6 92008 5 1525236 80 function
7 209 1 90572 5 1615808 85 type
8 144 1 76800 4 1692608 89 dict of class
9 984 4 35424 2 1728032 91 __builtin__.wrapper_descriptor

An alternative is that you could use windows's performance counters through pywin32

Python memory usage: Which of my objects is hogging the most memory?

The program I've written stores a large amount of data in dictionaries. Specifically, I'm creating 1588 instances of a class, each of which contains 15 dictionaries with 1500 float to float mappings. This process has been using up the 2GB of memory on my laptop pretty quickly (I start writing to swap at about the 1000th instance of the class).
My question is, which of the following is using up my memory?
34 million some pairs of floats?
The overhead of 22,500 dictionaries?
the overhead of 1500 classes?
To me it seems like the memory hog should be the huge number of floating point numbers that I'm holding in memory. However, If what I've read so far is correct, each of my floating point numbers take up 16 bytes. Since I have 34 million pairs, this should be about 108 million bytes, which should be just over a gigabyte.
Is there something I'm not taking into consideration here?

The floats do take up 16 bytes apiece, and a dict with 1500 entries about 100k:
>> sys.getsizeof(1.0)
16
>>> d = dict.fromkeys((float(i) for i in range(1500)), 2.0)
>>> sys.getsizeof(d)
98444
so the 22,500 dicts take over 2GB all by themselves, the 68 million floats another GB or so. Not sure how you compute 68 million times 16 equal only 100M -- you may have dropped a zero somewhere.
The class itself takes up a negligible amount, and 1500 instances thereof (net of the objects they refer to of course, just as getsizeof gives us such net amounts for the dicts) not much more than a smallish dict each, so, that's hardly the problem. I.e.:
>>> sys.getsizeof(Sic)
452
>>> sys.getsizeof(Sic())
32
>>> sys.getsizeof(Sic().__dict__)
524
452 for the class, (524 + 32) * 1550 = 862K for all the instances, as you see that's not the worry when you have gigabytes each in dicts and floats.

Huge memory usage of loading large dictionaries in memory

I have a file on disk that's only 168MB. It's just a comma separated list of word,id.
The word can be 1-5 characters long. There's 6.5 million lines.
I created a dictionary in python to load this up into memory so I can search incoming text against that list of words. When python loads it up into memory it shows 1.3 GB's of RAM space used. Any idea why that is?
So let's say my word file looks like this...
1,word1
2,word2
3,word3
Then add 6.5 million to that.
I then loop through that file and create a dictionary (python 2.6.1):
def load_term_cache():
"""will load the term cache from our cached file instead of hitting mysql. If it didn't
preload into memory it would be 20+ million queries per process"""
global cached_terms
dumpfile = os.path.join(os.getenv("MY_PATH"), 'datafiles', 'baseterms.txt')
f = open(dumpfile)
cache = csv.reader(f)
for term_id, term in cache:
cached_terms[term] = term_id
f.close()
Just doing that blows up the memory. I view activity monitor and it pegs the memory to all available up to around 1.5GB of RAM On my laptop it just starts to swap. Any ideas how to most efficiently store key/value pairs in memory with python?
Update: I tried to use the anydb module and after 4.4 million records it just dies
the floating point number is the elapsed seconds since I tried to load it
56.95
3400018
60.12
3600019
63.27
3800020
66.43
4000021
69.59
4200022
72.75
4400023
83.42
4600024
168.61
4800025
338.57
You can see it was running great. 200,000 rows every few seconds inserted until I hit a wall and time doubled.
import anydbm
i=0
mark=0
starttime = time.time()
dbfile = os.path.join(os.getenv("MY_PATH"), 'datafiles', 'baseterms')
db = anydbm.open(dbfile, 'c')
#load from existing baseterm file
termfile = os.path.join(os.getenv("MY_PATH"), 'datafiles', 'baseterms.txt.LARGE')
for line in open(termfile):
i += 1
pieces = line.split(',')
db[str(pieces[1])] = str(pieces[0])
if i > mark:
print i
print round(time.time() - starttime, 2)
mark = i + 200000
db.close()

Lots of ideas. However, if you want practical help, edit your question to show ALL of your code. Also tell us what is the "it" that shows memory used, what it shows when you load a file with zero entries, and what platform you are on, and what version of Python.
You say that "the word can be 1-5 words long". What is the average length of the key field in BYTES? Are the ids all integer? If so what are the min and max integer? If not, what is the average length if ID in bytes? To enable cross-achecking of all of above, how many bytes are there in your 6.5M-line file?
Looking at your code, a 1-line file word1,1 will create a dict d['1'] = 'word1' ... isn't that bassackwards?
Update 3: More questions: How is the "word" encoded? Are you sure you are not carrying a load of trailing spaces on any of the two fields?
Update 4 ... You asked "how to most efficiently store key/value pairs in memory with python" and nobody's answered that yet with any accuracy.
You have a 168 Mb file with 6.5 million lines. That's 168 * 1.024 ** 2 / 6.5 = 27.1 bytes per line. Knock off 1 byte for the comma and 1 byte for the newline (assuming it's a *x platform) and we're left with 25 bytes per line. Assuming the "id" is intended to be unique, and as it appears to be an integer, let's assume the "id" is 7 bytes long; that leaves us with an average size of 18 bytes for the "word". Does that match your expectation?
So, we want to store an 18-byte key and a 7-byte value in an in-memory look-up table.
Let's assume a 32-bit CPython 2.6 platform.
>>> K = sys.getsizeof('123456789012345678')
>>> V = sys.getsizeof('1234567')
>>> K, V
(42, 31)
Note that sys.getsizeof(str_object) => 24 + len(str_object)
Tuples were mentioned by one answerer. Note carefully the following:
>>> sys.getsizeof(())
28
>>> sys.getsizeof((1,))
32
>>> sys.getsizeof((1,2))
36
>>> sys.getsizeof((1,2,3))
40
>>> sys.getsizeof(("foo", "bar"))
36
>>> sys.getsizeof(("fooooooooooooooooooooooo", "bar"))
36
>>>
Conclusion: sys.getsizeof(tuple_object) => 28 + 4 * len(tuple_object) ... it only allows for a pointer to each item, it doesn't allow for the sizes of the items.
A similar analysis of lists shows that sys.getsizeof(list_object) => 36 + 4 * len(list_object) ... again it is necessary to add the sizes of the items. There is a further consideration: CPython overallocates lists so that it doesn't have to call the system realloc() on every list.append() call. For sufficiently large size (like 6.5 million!) the overallocation is 12.5 percent -- see the source (Objects/listobject.c). This overallocation is not done with tuples (their size doesn't change).
Here are the costs of various alternatives to dict for a memory-based look-up table:
List of tuples:
Each tuple will take 36 bytes for the 2-tuple itself, plus K and V for the contents. So N of them will take N * (36 + K + V); then you need a list to hold them, so we need 36 + 1.125 * 4 * N for that.
Total for list of tuples: 36 + N * (40.5 + K + v)
That's 26 + 113.5 * N (about 709 MB when is 6.5 million)
Two parallel lists:
(36 + 1.125 * 4 * N + K * N) + (36 + 1.125 * 4 * N + V * N)
i.e. 72 + N * (9 + K + V)
Note that the difference between 40.5 * N and 9 * N is about 200MB when N is 6.5 million.
Value stored as int not str:
But that's not all. If the IDs are actually integers, we can store them as such.
>>> sys.getsizeof(1234567)
12
That's 12 bytes instead of 31 bytes for each value object. That difference of 19 * N is a further saving of about 118MB when N is 6.5 million.
Use array.array('l') instead of list for the (integer) value:
We can store those 7-digit integers in an array.array('l'). No int objects, and no pointers to them -- just a 4-byte signed integer value. Bonus: arrays are overallocated by only 6.25% (for large N). So that's 1.0625 * 4 * N instead of the previous (1.125 * 4 + 12) * N, a further saving of 12.25 * N i.e. 76 MB.
So we're down to 709 - 200 - 118 - 76 = about 315 MB.
N.B. Errors and omissions excepted -- it's 0127 in my TZ :-(

Take a look (Python 2.6, 32-bit version)...:
>>> sys.getsizeof('word,1')
30
>>> sys.getsizeof(('word', '1'))
36
>>> sys.getsizeof(dict(word='1'))
140
The string (taking 6 bytes on disk, clearly) gets an overhead of 24 bytes (no matter how long it is, add 24 to its length to find how much memory it takes). When you split it into a tuple, that's a little bit more. But the dict is what really blows things up: even an empty dict takes 140 bytes -- pure overhead of maintaining a blazingly-fast hash-based lookup take. To be fast, a hash table must have low density -- and Python ensures a dict is always low density (by taking up a lot of extra memory for it).
The most memory-efficient way to store key / value pairs is as a list of tuples, but lookup of course will be very slow (even if you sort the list and use bisect for the lookup, it's still going to be extremely slower than a dict).
Consider using shelve instead -- that will use little memory (since the data reside on disk) and still offer pretty spiffy lookup performance (not as fast as an in-memory dict, of course, but for a large amount of data it will be much faster than lookup on a list of tuples, even a sorted one, can ever be!-).

convert your data into a dbm (import anydbm, or use berkerley db by import bsddb ...), and then use dbm API to access it.
the reason to explode is that python has extra meta information for any objects, and the dict needs to construct a hash table (which would require more memory). you just created so many objects (6.5M) so the metadata becomes too huge.
import bsddb
a = bsddb.btopen('a.bdb') # you can also try bsddb.hashopen
for x in xrange(10500) :
a['word%d' %x] = '%d' %x
a.close()
This code takes only 1 second to run, so I think the speed is OK (since you said 10500 lines per second).
btopen creates a db file with 499,712 bytes in length, and hashopen creates 319,488 bytes.
With xrange input as 6.5M and using btopen, I got 417,080KB in ouput file size and around 1 or 2 minute to complete insertion. So I think it's totally suitable for you.

I have the same problem though I'm later. The others has answered this question well. And I offer an easy to use(maybe not so easy :-) ) and rather efficient alternative, that's pandas.DataFrame. It performs well in memory usage when saving large data.