I have a pkl file containing an ndarray that I originally dump using a GPU. I unpickle it with the GPU and now I want to store it as whatever, that I can later uncompress using a CPU. I run everything on a supercomputer and later I want to just have access to the ndarrays on a normal computer without a fancy GPU. I looked into functions such as
np.save()
np.savez()
but save() I can't set allow_pickle=False and when I load the array stored with savez() it's empty.
This is how I save things:
I run THEANO_FLAGS="device=gpu,floatX=float32" srun -u python deep_q_rl/unpicklestuff.py
unpicklestuff.py:
import sys
import cPickle
import lasagne.layers
import os
import numpy as np
for i in os.listdir(path):
net_file = open(path+str(i), 'r')
network = cPickle.load(net_file)
q_layers = lasagne.layers.get_all_layers(network.l_out)
np.savez(savepath+str(i), q_layers)
And this is how I load them later:
q_layers = np.load(path)
Related
That is, if I want to generate a heatmap map, I process a set of 3D NumPy arrays, and finally, want to turn this set of arrays into a .nii file to represent it.
This array is not extracted from the .nii file using image.numpy(), but generated by myself using NumPy.
I was using a reader function to generate but I think it's very hard and very strict for the path parameters to be entered, so I would like to ask if there is a simpler path.
import numpy as np
import torch
import os
import torchio as tio
def numpy_reader(npy_name):
data = torch.from_numpy(np.load(npy_name)).unsqueeze(0)
affine = np.eye(4)
return data, affine
image_nii = tio.ScalarImage(npy_name, reader=numpy_reader)
image_nii.save(os.path.join('./nii_temp', savenii_name))
I am trying to read several images from archive with skimage.io.imread_collection, but for some reason it throws an error:
"There is no item named '00071198d059ba7f5914a526d124d28e6d010c92466da21d4a04cd5413362552/masks/*.png' in the archive".
I checked several times, such directory exists in archive and with *.png I just specify that I want to have all images in my collection, and imread_collection works well, when I am trying to download images not from archive, but from extracted folder.
#specify folder name
each_img_idx = '00071198d059ba7f5914a526d124d28e6d010c92466da21d4a04cd5413362552'
with zipfile.ZipFile('stage1_train.zip') as archive:
mask_ = skimage.io.imread_collection(archive.open(str(each_img_idx) + '/masks/*.png')).concatenate()
May some one explain me, what's going on?
Not all scikit-image plugins support reading from bytes, so I recommend using imageio. You'll also have to tell ImageCollection how to access the images inside the archive, which is done using a customized load_func:
from skimage import io
import imageio
archive = zipfile.ZipFile('foo.zip')
images = [f.filename for f in zf.filelist]
def zip_imread(fn):
return imageio.imread(archive.read(fn))
ic = io.ImageCollection(images, load_func=zip_imread)
ImageCollection has some benefits like not loading all images into memory at the same time. But if you simply want a long list of NumPy arrays, you can do:
collection = [imageio.imread(zf.read(f)) for f in zf.filelist]
I'm trying to learn how to use pydicom for reading and processing dicom images. I'm using Python 3.
import dicom
import numpy
ds = pydicom.read_file(lstFilesDCM[0])
print(ds.pixel_array)`
I get an error NameError: name 'pydicom' is not defined. If I change
ds = pydicom.read_file(lstFilesDCM[0])
to
ds = dicom.read_file(lstFilesDCM[0])
(using dicom.read_file instead), I get the following error:
NotImplementedError: Pixel Data is compressed in a format
pydicom does not yet handle. Cannot return array
I also verified that pydicom is properly installed and updated.
How do i fix this?
You are trying to call a class that you have not imported before:
Use:
import pydicom
import numpy
ds = pydicom.read_file(lstFilesDCM[0])
print(ds.pixel_array)
or
import dicom
ds = dicom.read_file("the_name_of_file.dcm")
Documentation: http://pydicom.readthedocs.io/en/stable/pydicom_user_guide.html
If you want to get your hands on the pixel data, I suggest to use the convert program from the ImageMagick suite. You can either call this program from Python using the subprocess module. (See this example, where I convert them to JPEG format), or you can use one of the Python bindings.
If you want to manipulate the images, using the bindings might be preferable. But note that not all the bindings have been converted to ImageMagick version 7.
I'm newbie in python and in geoprocessing. I'm writing some program to calculate ndwi. To make this, I try to open geotiff dataset with gdal, but dataset can't be opened. I tried to open different tiff files (Landsat8 multiple data, Landsat7 composite, etc), but dataset is always None.
What reason to this could be? Or how can i find it out?
Here's a part of code:
import sys, os, struct
import gdal, gdalconst
from gdalconst import *
import numpy as np
from numpy import *
class GDALCalcNDWI ():
def calcNDWI(self, outFilePath):
gdal.AllRegister()
# this allows GDAL to throw Python Exceptions
gdal.UseExceptions()
filePath = "C:\\Users\\Daria\\Desktop.TIF\\170028-2007-05-21.tif"
# Open
dataset = gdal.Open(filePath, gdal.GA_ReadOnly)
# Check
if dataset is None:
print ("can't open tiff file")
sys.exit(-1)
Thanks
Whenever you have a well-known file reader that is returning None, make sure the path to your file is correct. I doubt you have a directory called Desktop.TIF, I'm assuming you just made a typo in your source code. You probably want C:\\Users\\Dara\\Desktop\\TIF\\170028-2007-05-21.tif as the path (note that Desktop.TIF ==> Desktop\\TIF).
The safest thing to do is right click on the file, go to properties, and copy/paste that path into your python source code.
Having searched through internet, I couldn't find how a set of images with their corresponding labels can be saved as batch files in a way using the following data I'll be able to retrieve (load) them to work on them. simply saying I'm searching for a way to save them in a way to use the following code to retrieve them.
import cPickle as pickle
import numpy as np
import os
def load_CIFAR_batch(filename):
""" load single batch of cifar """
with open(filename, 'rb') as f:
datadict = pickle.load(f)
X = datadict['data']
Y = datadict['labels']
X = X.reshape(10000, 3, 32, 32).transpose(0,2,3,1).astype("float")
Y = np.array(Y)
return X, Y
e.g. the data set of CIFAR10 is already in batch format using cPickle, but I don't know how to use cPickle what they have done to save images.
CIFAR-10 Dataset link : http://www.cs.toronto.edu/~kriz/cifar.html
I'm using:
Ubuntu 14.04 LTS
Python 2.7