We Keep Coding

Fastest method to modify bytestring over loop in Python?

I have an image stored as a bytestring b'' and am performing per-pixel operations. Right now the fastest way I've found is to use the struct crate to pack and unpack the bytes during modification, then save the pixels to a bytearray
# retrieve image data. Stored as bytestring
pixels = buff.get(rect, 1.0, "CIE LCH(ab) alpha double",
Gegl.AbyssPolicy.CLAMP)
# iterator split into 32-byte chunks for each pixel's 8-byte LCHA channels
pixels_iter = (pixels[x:x + 32] for x in range(0, len(pixels), 32))
new_pixels = bytearray()
# when using `pool.map`, the loop was placed in its own function.
for pixel in pixels_iter:
l, c, h, a = struct.unpack('dddd', pixel)
# simple operation for now: lower chroma if bright and saturated
c = c - (l * c) / 100
new_pixels += struct.pack('dddd', l, c, h, a)
# save new data. everything hereout handled by GEGL instead of myself.
shadow.set(rect, "CIE LCH(ab) alpha double", bytes(new_pixels))
Problem is this takes about 3 1/2 seconds for a 7MP image on my workstation. Fair but not ideal if updates are frequently requested. From what I've gathered, it seems the constant array modification and possibly struct [un]packing are the main culprits. I've refactored this probably a dozen times and I think I'm out of ideas for optimizing this.
I've tried:
struct.unpacking the whole bytestring once instead of each pixel as-needed. Lost about 20% efficiency.
collections.deque Admittedly not familiar with its technicalities. Lost 10-30% depending on implementation
similar results with other iterator helpers like map/join
numpy.array Also admittedly know basically nothing about general numpy. Similar results to deque
multiprocessing seemed to be bottlenecked when I appended the pool.map results to new_pixels. Actually lost about 10% which seems wild, as usually I can just lazily throw threads at problems. The pixels_iter was grouped again into equally sized sublists for each thread, so new_pixels concatenated 8 large lists instead of a few million small lists, which I thought would be faster. Tempted to retry this one as I might've botched it somehow with my 4 am implementation.
In theory it could also work by saving multiple small sections of the image buffer to avoid concatenating to new_pixels entirely, but that would vastly increase code complexity elsewhere.
Converting pixels itself into a bytearray and modifying it in-place using slice ranges. Lost ~30% but also halved memory usage.
Completely separate interpreters like Pypy are off the table, as I'm not the one bundling the Python version.

NumPy should produce much faster results than a manual loop, if you use it properly. Using it properly means using NumPy operations over whole arrays, not just looping manually over a NumPy array.
For example,
new_pixels = bytearray(pixels)
as_numpy = numpy.frombuffer(new_pixels, dtype=float)
as_numpy[1::4] *= 1 - as_numpy[::4] / 100
Now new_pixels contains the adjusted values.

Python 1500% Memory Overhead from loading binary file

I'm trying to load one binary file containing 120mb of data using the following routine:
from struct import *
def loadBinaryData():
d = []
size = calcsize('iif')
with open("ratings.bin",'rb') as f:
while True:
data = f.read(size)
if not data: break
(a,b,c) = unpack_from('iif',data)
d.append(((a,b),c))
return d
However, when I execute this, my python process starts to use up to 2.2gb of RAM, which - to me - feels very wrong. Are there any obvious errors that can explain this behavior? Am I misusing any very wasteful python features?
One more thing, I don't wanna use generator functions for this, I actually need all of the data in memory.

I would try converting more data at once, to cut down on per-object memory overhead. Your data seems to be a 3-tuple of around 12 bytes, so for a 120-MB file you have around 10 million tuples.
If you look at this article, you can see that:
An integer has an overhead of 24 bytes.
A float does, too.
A tuple has 63 bytes of overhead.
10 million tuples at (2 * 24 + 24 + 63) bytes each is going to weigh in at 1.35 GB, but perhaps there is additional garbage created by the growing of the d list.

Looking for efficient python program for this following python script

I am looking for a memory efficient python script for the following script. The following script works well for smaller dimension but the dimension of the matrix is 5000X5000 in my actual calculation. Therefore, it takes very long time to finish it. Can anyone help me how can I do that?
def check(v1,v2):
if len(v1)!=len(v2):
raise ValueError,"the lenght of both arrays must be the same"
pass
def d0(v1, v2):
check(v1, v2)
return dot(v1, v2)
import numpy as np
from pylab import *
vector=[[0.1, .32, .2, 0.4, 0.8], [.23, .18, .56, .61, .12], [.9, .3, .6, .5, .3], [.34, .75, .91, .19, .21]]
rav= np.mean(vector,axis=0)
#print rav
#print vector
m= vector-rav
corr_matrix=[]
for i in range(0,len(vector)):
tmp=[]
x=sqrt(d0(m[i],m[i]))
for j in range(0,len(vector)):
y=sqrt(d0(m[j],m[j]))
z=d0(m[i],m[j])
w=z/(x*y)
tmp.append(w)
corr_matrix.append(tmp)
print corr_matrix

Make your matrix (and your vector) into numpy arrays instead of Python lists. That will make it take much less memory (and also run faster).
To understand why:
A Python list is a list of Python object instances. Each one of these has type information, pointers, and all kinds of other stuff to keep around beyond just the 8-byte number. Let's say each one ends up being 64 bytes instead of 8. So, that's 64 bytes per element, times 25M elements, equals 1600M bytes!
By contrast, a numpy array is a list of just the raw values, together with a single copy of all that extra information (in the dtype). So, instead of 64 * 25M bytes, you've got 8 * 25M + 64 bytes, which is only 1/8th the size.
As for the speed increase: If you iterate over a 5000x5000 matrix, you're calling some code in the inner loop 25M times. If you're doing a numpy expression like m + m, the code inside the loop is a few lines of C code that get compiled down to a few dozen machine-code operations, which is blazingly fast. If you're doing the loop explicitly in Python, the inside of that loop has to drive the Python interpreter every time through the loop, which is much, much slower. (On top of that, the C compiler will optimize the code, and numpy may have some explicit optimizations too.) Depending on how trivial the work inside the loop is, the speedup can be anywhere from 2x to 10000x. So, even if you have to make things a bit convoluted, try to find a way to express each step as an array broadcast rather than a loop, and it will be much faster.
So, how do you do that? Simple. Instead of this:
corr_matrix=[]
for i in range(len(vector)):
tmp=[]
# …
for j in range(len(vector)):
# …
tmp.append(w)
corr_matrix.append(tmp)
Do this:
corr_matrix=np.zeros((len(vector), len(vector))
for i in range(len(vector)):
# …
for j in range(len(vector)):
# …
corr_matrix[i, j] = w
That immediately eliminates all the memory problems caused by the overhead of keeping around 25M Python float objects, and will give you a significant speed boost too. You can't reduce the memory any further except by not keeping the whole array in memory at once, but you should be fine already. (You can boost the speed even more by using broadcast operations in place of loops, but if the memory is your problem, and the performance is fine, it may not be necessary.)

Repeatedly appending to a large list (Python 2.6.6)

I have a project where I am reading in ASCII values from a microcontroller through a serial port (looks like this : AA FF BA 11 43 CF etc)
The input is coming in quickly (38 two character sets / second).
I'm taking this input and appending it to a running list of all measurements.
After about 5 hours, my list has grown to ~ 855000 entries.
I'm given to understand that the larger a list becomes, the slower list operations become. My intent is to have this test run for 24 hours, which should yield around 3M results.
Is there a more efficient, faster way to append to a list then list.append()?
Thanks Everyone.

I'm given to understand that the larger a list becomes, the slower list operations become.
That's not true in general. Lists in Python are, despite the name, not linked lists but arrays. There are operations that are O(n) on arrays (copying and searching, for instance), but you don't seem to use any of these. As a rule of thumb: If it's widely used and idiomatic, some smart people went and chose a smart way to do it. list.append is a widely-used builtin (and the underlying C function is also used in other places, e.g. list comprehensions). If there was a faster way, it would already be in use.
As you will see when you inspect the source code, lists are overallocating, i.e. when they are resized, they allocate more than needed for one item so the next n items can be appended without need to another resize (which is O(n)). The growth isn't constant, it is proportional with the list size, so resizing becomes rarer as the list grows larger. Here's the snippet from listobject.c:list_resize that determines the overallocation:
/* This over-allocates proportional to the list size, making room
* for additional growth. The over-allocation is mild, but is
* enough to give linear-time amortized behavior over a long
* sequence of appends() in the presence of a poorly-performing
* system realloc().
* The growth pattern is: 0, 4, 8, 16, 25, 35, 46, 58, 72, 88, ...
*/
new_allocated = (newsize >> 3) + (newsize < 9 ? 3 : 6);
As Mark Ransom points out, older Python versions (<2.7, 3.0) have a bug that make the GC sabotage this. If you have such a Python version, you may want to disable the gc. If you can't because you generate too much garbage (that slips refcounting), you're out of luck though.

One thing you might want to consider is writing your data to a file as it's collected. I don't know (or really care) if it will affect performance, but it will help ensure that you don't lose all your data if power blips. Once you've got all the data, you can suck it out of the file and jam it in a list or an array or a numpy matrix or whatever for processing.

Appending to a python list has a constant cost. It is not affected by the number of items in the list (in theory). In practice appending to a list will get slower once you run out of memory and the system starts swapping.
http://wiki.python.org/moin/TimeComplexity
It would be helpful to understand why you actually append things into a list. What are you planning to do with the items. If you don't need all of them you could build a ring buffer, if you don't need to do computation you could write the list to a file, etc.

First of all, 38 two-character sets per second, 1 stop bit, 8 data bits, and no parity, is only 760 baud, not fast at all.
But anyway, my suggestion, if you're worried about having overly large lists/don't want to use one huge list, is just to store store a list on disk once it reaches a certain size and start a new list, repeating until you've gotten all the data, then combining all the lists into one once you're done receiving the data.
Though you may skip the sublists completely and just go with nmichaels' suggestion, writing the data to a file as you get it and using a small circular buffer to hold the received data that has not yet been written.

It might be faster to use numpy if you know how long the array is going to be and you can convert your hex codes to ints:
import numpy
a = numpy.zeros(3000000, numpy.int32)
for i in range(3000000):
a[i] = int(scanHexFromSerial(),16)
This will leave you with an array of integers (which you could convert back to hex with hex()), but depending on your application maybe that will work just as well for you.

Why do dicts of defaultdict(int)'s use so much memory? (and other simple python performance questions)

I do understand that querying a non-existent key in a defaultdict the way I do will add items to the defaultdict. That is why it is fair to compare my 2nd code snippet to my first one in terms of performance.
import numpy as num
from collections import defaultdict
topKeys = range(16384)
keys = range(8192)
table = dict((k,defaultdict(int)) for k in topKeys)
dat = num.zeros((16384,8192), dtype="int32")
print "looping begins"
#how much memory should this use? I think it shouldn't use more that a few
#times the memory required to hold (16384*8192) int32's (512 mb), but
#it uses 11 GB!
for k in topKeys:
for j in keys:
dat[k,j] = table[k][j]
print "done"
What is going on here? Furthermore, this similar script takes eons to run compared to the first one, and also uses an absurd quantity of memory.
topKeys = range(16384)
keys = range(8192)
table = [(j,0) for k in topKeys for j in keys]
I guess python ints might be 64 bit ints, which would account for some of this, but do these relatively natural and simple constructions really produce such a massive overhead?
I guess these scripts show that they do, so my question is: what exactly is causing the high memory usage in the first script and the long runtime and high memory usage of the second script and is there any way to avoid these costs?
Edit:
Python 2.6.4 on 64 bit machine.
Edit 2: I can see why, to a first approximation, my table should take up 3 GB
16384*8192*(12+12) bytes
and 6GB with a defaultdict load factor that forces it to reserve double the space.
Then inefficiencies in memory allocation eat up another factor of 2.
So here are my remaining questions:
Is there a way for me to tell it to use 32 bit ints somehow?
And why does my second code snippet take FOREVER to run compared to the first one? The first one takes about a minute and I killed the second one after 80 minutes.

Python ints are internally represented as C longs (it's actually a bit more complicated than that), but that's not really the root of your problem.
The biggest overhead is your usage of dicts. (defaultdicts and dicts are about the same in this description). dicts are implemented using hash tables, which is nice because it gives quick lookup of pretty general keys. (It's not so necessary when you only need to look up sequential numerical keys, since they can be laid out in an easy way to get to them.)
A dict can have many more slots than it has items. Let's say you have a dict with 3x as many slots as items. Each of these slots needs room for a pointer to a key and a pointer serving as the end of a linked list. That's 6x as many points as numbers, plus all the pointers to the items you're interested in. Consider that each of these pointers is 8 bytes on your system and that you have 16384 defaultdicts in this situation. As a rough, handwavey look at this, 16384 occurrences * (8192 items/occurance) * 7 (pointers/item) * 8 (bytes/pointer) = 7 GB. This is before I've gotten to the actual numbers you're storing (each unique number of which is itself a Python dict), the outer dict, that numpy array, or the stuff Python's keeping track of to try to optimize some.
Your overhead sounds a little higher than I suspect and I would be interested in knowing whether that 11GB was for a whole process or whether you calculated it for just table. In any event, I do expect the size of this dict-of-defaultdicts data structure to be orders of magnitude bigger than the numpy array representation.
As to "is there any way to avoid these costs?" the answer is "use numpy for storing large, fixed-size contiguous numerical arrays, not dicts!" You'll have to be more specific and concrete about why you found such a structure necessary for better advice about what the best solution is.

Well, look at what your code is actually doing:
topKeys = range(16384)
table = dict((k,defaultdict(int)) for k in topKeys)
This creates a dict holding 16384 defaultdict(int)'s. A dict has a certain amount of overhead: the dict object itself is between 60 and 120 bytes (depending on the size of pointers and ssize_t's in your build.) That's just the object itself; unless the dict is less than a couple of items, the data is a separate block of memory, between 12 and 24 bytes, and it's always between 1/2 and 2/3rds filled. And defaultdicts are 4 to 8 bytes bigger because they have this extra thing to store. And ints are 12 bytes each, and although they're reused where possible, that snippet won't reuse most of them. So, realistically, in a 32-bit build, that snippet will take up 60 + (16384*12) * 1.8 (fill factor) bytes for the table dict, 16384 * 64 bytes for the defaultdicts it stores as values, and 16384 * 12 bytes for the integers. So that's just over a megabyte and a half without storing anything in your defaultdicts. And that's in a 32-bit build; a 64-bit build would be twice that size.
Then you create a numpy array, which is actually pretty conservative with memory:
dat = num.zeros((16384,8192), dtype="int32")
This will have some overhead for the array itself, the usual Python object overhead plus the dimensions and type of the array and such, but it wouldn't be much more than 100 bytes, and only for the one array. It does store 16384*8192 int32's in your 512Mb though.
And then you have this rather peculiar way of filling this numpy array:
for k in topKeys:
for j in keys:
dat[k,j] = table[k][j]
The two loops themselves don't use much memory, and they re-use it each iteration. However, table[k][j] creates a new Python integer for each value you request, and stores it in the defaultdict. The integer created is always 0, and it so happens that that always gets reused, but storing the reference to it still uses up space in the defaultdict: the aforementioned 12 bytes per entry, times the fill factor (between 1.66 and 2.) That lands you close to 3Gb of actual data right there, and 6Gb in a 64-bit build.
On top of that the defaultdicts, because you keep adding data, have to keep growing, which means they have to keep reallocating. Because of Python's malloc frontend (obmalloc) and how it allocates smaller objects in blocks of its own, and how process memory works on most operating systems, this means your process will allocate more and not be able to free it; it won't actually use all of the 11Gb, and Python will re-use the available memory inbetween the large blocks for the defaultdicts, but the total mapped address space will be that 11Gb.

Mike Graham gives a good explanation of why dictionaries use more memory, but I thought that I'd explain why your table dict of defaultdicts starts to take up so much memory.
The way that the defaultdict (DD) is set-up right now, whenever you retrieve an element that isn't in the DD, you get the default value for the DD (0 for your case) but also the DD now stores a key that previously wasn't in the DD with the default value of 0. I personally don't like this, but that's how it goes. However, it means that for every iteration of the inner loop, new memory is being allocated which is why it is taking forever. If you change the lines
for k in topKeys:
for j in keys:
dat[k,j] = table[k][j]
to
for k in topKeys:
for j in keys:
if j in table[k]:
dat[k,j] = table[k][j]
else:
dat[k,j] = 0
then default values aren't being assigned to keys in the DDs and so the memory stays around 540 MB for me which is mostly just the memory allocated for dat. DDs are decent for sparse matrices though you probably should just use the sparse matrices in Scipy if that's what you want.