I am trying to create image database compatible with cPickle. Initially list is empty. Data of each image in the directory is added as new row to the list. Images are 224x224. Size of images is on average 8KB. After loading around 10000 images my pc hangs. There are no mouse movement. Nothing happens. needs restart. Below is code snippet which does this..
cr=csv.reader(open(csv_file,"rb"))
for row in cr:
print row[0], row[1]
try:
image=Image.open(row[0]+'.jpg').convert('LA')
pixels=[]
pixels=[f[0] for f in list(image.getdata())]
#pix=np.array(image)
dataset.append(pixels)
#dataset.append(pix)
labels.append(row[1])
del image
except:
print("image not found")
I tried reducing size of images to 28X28 and it works. But i don't want to reduce the size of the images. I am using python 64 bit executable. RAM is 4GB. Ubuntu 14.04. I suspect this is happening due to limited stack space, and list is taking more than available stack space. If so, how do i create this huge list? is there any workaround for this issue? My end goal is to create an numpy array with pixel data as its rows. Currently i am converting list into numpy array.. Is there a solution for this problem??
If the data will eventually be numpy array, maybe try using numpy.memmap. It works like "normal" numpy arrays, the difference is that the data is actually stored on the disk in binary. Only the requested chunks of array are put in RAM, thus may get rid of your problem.
If the size of data array is determined, you just need to set up the correct dimension when creating a memmap object. If not, check out
numpy.memmap.resize, and you should be able to create it anyways.
Oh, and there are other solutions such as PyTables.
Good luck!
Related
I have a collection of rather large tiff files (typical resolution is 30k x 30k to 50k x 50k). These have some interesting properties: 1. they contain several different resolutions of the same image, and 2. the data seems to be compressed as JPEG tiles (512x512). They also seem to be BigTIFFs.
I'd like to read specific regions of interest (more or less random access pattern) and I'd like it to be as fast as possible (~about as fast as decompressing the needed tiles plus a little bit of overhead). Decompressing the whole file, or even one of the resolution layers, is not practical. I'm using python.
I tried opening with skimage.io.MultiImage('test.tiff'))[level], and this works (produces correct data). It does decompress each resolution layer completely, although not the whole file; so okay at the low resolutions but not really useful for the highest resolutions. There didn't seem to be any way to select a region of interest in skimage.io, or in any of the libraries it uses (imread, PIL, ...).
I also tried using OpenSlide using img.read_region((x,y), level, (width, height)). This library seems made exactly for this type of data, and is very fast, but unfortunately produces incorrect data for some regions. Until the bug is fixed upstream, I can't use it.
Lastly, using a very recent version of tifffile=2020.6.3 and imagecodecs=2020.5.30 (older versions don't work - I think at least 2018.10.18 is needed), I could list the tiles, using code modified from the tifffile documentation:
with tifffile.TiffFile('test.tiff') as tif:
fh = tif.filehandle
for page in tif.pages:
jpegtables = page.tags.get('JPEGTables', None)
if jpegtables is not None:
jpegtables = jpegtables.value
for index, (offset, bytecount) in enumerate(
zip(page.dataoffsets, page.databytecounts)
):
fh.seek(offset)
data = fh.read(bytecount)
tile, indices, shape = page.decode(data, index, jpegtables)
print(tile.shape, indices, shape)
It seems the page.decode() call actually decompresses each tile (tile is a numpy array with pixel data). It is not obvious how to only get the index but not decompress. I'm also not sure how fast this would be. This leaves any selection of a region of interest and reassembly of tiles as an exercise to the user.
How do I efficiently read regions of interest out of files like this? Does someone have example code to do that with tifffile? Or, is there another library that would do the trick?
I'm writing some image processing routines for a micro-controller that supports MicroPython. The bad news is that it only has 0.5 MB of RAM. This means that if I want to work with relatively big images/matrices like 256x256, I need to treat it as a collection of smaller matrices (e.g. 32x32) and perform the operation on them. Leaving at aside the fact of reconstructing the final output of the orignal (256x256) matrix from its (32x32) submatrices, I'd like to focus on how to do the loading/saving from/to disk (an SD card in this case) of this smaller matrices from a big image.
Given that intro, here is my question: Assuming I have a 256x256 on disk that I'd like to apply some operation onto (e.g. convolution), what's the most convenient way of storing that image so it's easy to load it into 32x32 image patches? I've seen there is a MicroPython implementation of the pickle module, is this a good idea for my problem?
Sorry, but your question contains the answer - if you need to work with 32x32 tiles, the best format is that which represents your big image as a sequence of tiles (and e.g. not as one big 256x256 image, though reading tiles out of it is also not a rocket science and should be fairly trivial to code in MicroPython, though 32x32 tiles would be more efficient of course).
You don't describe the exact format of your images, but I wouldn't use pickle module for it, but store images as raw bytes and load them into array.array() objects (using inplace .readinto() operation).
I have seen an example of loading a TIFF sequence of say a 3D tiff stack or animation. However I cannot figure out how to do the opposite, taking say, a 3D+ numpy array and saving it as a TIFF sequence. Are there are any examples of encoding this? I can read a 2D array using PIL.fromarray. It would be nice if this method had some way of loading a multi-dimensional array but a naive method call will throw an unsupported-type exception.
Presumably if one did write such a sequence they might also want to add some headers to dictate channels, time and so forth. My particular bias is being able to open such images in applications like ImageJ/FIJI or converting from TIFF to other formats. Maybe there are better ways to go about this in the first place.
I know this is very old, but in case anyone comes here looking for the answer, this does seem to have been solved nicely.
e.g.
im.save("filename.tiff", format="TIFF", save_all=True)
https://pillow.readthedocs.io/en/latest/releasenotes/3.4.0.html
The PCIDSK (normally .pix) format allows for bands with multiple data types to be held in the same file. My files contain "U16", "U8" and "BIT" channels.
GDAL is able to interpret the dataset correctly:
ds = gdal.Open("myfile.pix")
for i in range(1,ds.RasterCount+1):
print ds.GetRasterBand(i).DataType, ds.GetRasterBand(i).GetDescription()
It is interpreting the "U16" band as gdalconst.GDT_UInt16 (2), and the "U8" and "BIT" as gdalconst.GDT_Byte (1). I can see all the bands from the original file, in the correct order, with proper band names.
The problem is when I try to extract the data to numpy arrays. pythonw.exe crashes (not even an error in the console) when I try the usual ds.GetRasterBand(1).ReadAsArray().
I noticed that I was able to extract some data correctly by specifying the region to extract but it still crashing when the region includes one specific pixel, which is different from file to file. The given pixel is problematic for all "U16" and "U8" layers.
For a file 9494 (X) x 9609 (Y), it fails only at:
ds.GetRasterBand(1).ReadAsArray(8704,9472,1,1)
For a file 9193 (X) x 9293 (Y), it fails only at:
ds.GetRasterBand(1).ReadAsArray(5376,9216,1,1)
There is nothing notable about the pixels in the original data. I noticed that the unreadable locations would be located at edge of a tile 256x256 tile interleaved in the original.
It seems to be able to handle the "BIT" layers correctly with .ReadAsArray(), but not the "U16" and "U8".
The machine has plenty of memory, and I able to read and create other datasets that are much larger.
As a side note, if any knows how to create PCIDSK files with multiple data types, it would be useful for me to create a working example of the problem.
Using Python 2.7.11, Numpy 1.10.2, gdal bindings 1.11.3.
I am trying to write a module in python which will draw a numpy array of color data (rgb) to screen. At the moment I am currently using a 3 dimensional color array like this:
numpy.ones((10,10,3),dtype=np.float32,order='F') # (for 10x10 pure white tiles)
binding it to a buffer and using a glVertexAttribArray to broadcast the data to an array of tiles (point sprites) (in this case a 10x10 array) and this works fine for a static image.
But I want to be able to change the data in the array and have buffer reflect this change without having to rebuild it from scratch.
Currently I've built the buffer with:
glBufferData(GL_ARRAY_BUFFER, buffer_data.nbytes, buffer_data, GL_DYNAMIC_DRAW)
where buffer_data is the numpy array. What (if anything) could I pass instead (some pointer into memory perhaps?)
If you want to quickly render a rapidly changing numpy array, you might consider taking a look at glumpy. If you do go with a pure pyopengl solution I'd also be curious to see how it works.
Edit: see my answer here for an example of how to use Glumpy to view a constantly updating numpy array
glBufferData is for updating the entire buffer as it will create a new buffer each time.
What you want is either:
glMapBuffer / glUnmapBuffer.
glMapBuffer copies the buffer to client memory and alter the values locally, then push the changes back to the GPU with glUnmapBuffer.
glBufferSubData
This allows you to update small sections of a buffer, instead of the entire thing.
It sounds like you also want some class that automatically picks up these changes.
I cannot confirm if this is a good idea, but you could wrap or extend numpy.array and over-ride the built in method setitem.