I am using Pytorch's custom dataset feature to create a custom dataset from separate files in one folder. Each file contains 123 rows and 123 columns, and all data points are integers.
My issue is that the resources I've come across cater to files in one .csv, mine are not. More so, opening the images after being transformed as an image doesn't run as well. I'm not sure how to proceed from here on as my code gives:
AttributeError: 'Image' object has no attribute 'read'
import os
from torch.utils.data import DataLoader, Dataset
from numpy import genfromtxt
# Custom dataset
class CONCEPTDataset(Dataset):
""" Concept Dataset """
def __init__(self, file_dir, transforms=None):
"""
Args:
file_dir (string): Directory with all the images.
transforms (optional): Changes on the data.
"""
self.file_dir = file_dir
self.transforms = transforms
self.concepts = os.listdir(file_dir)
self.concepts.sort()
self.concepts = [os.path.join(file_dir, concept) for concept in self.concepts]
def __len__(self):
return len(self.concepts)
def __getitem__(self, idx):
image = self.concepts[idx]
# csv file to a numpy array using genfromtxt
data = genfromtxt(image, delimiter=',')
data = self.transforms(data.unsqueeze(0))
return data
PIL.Image.fromarray is used to convert an array to a PIL Image while Image.open is used to load an image file from the file system. You don't need either of those two since you already have a NumPy array representing your image and are looking to return it. PyTorch will convert it to torch.Tensor automatically if you plug your dataset to a torch.data.utils.DataLoader.
Related
My problem is a bit wired. I am working on Prostate MRI dataset which contains dicom images. When I load dicom files using Simple ITK the output numpy array's dtype will be float64 . But when I load same dicom files using pydicom , the output numpy array's dtype will be uint16 And the problem isn't just this. Pixel intensities would be different when using different module. So my question is why they look different and which one is correct and why these modules load data differently?
this is the code I'm using to load dcm files.
import pydicom
import SimpleITK as sitk
path = 'dicoms/1.dcm'
def read_using_sitk():
reader = sitk.ImageFileReader()
reader.SetFileName(path)
image = reader.Execute()
numpy_array = sitk.GetArrayFromImage(image)
return numpy_array.dtype
def read_using_pydicom():
dataset = pydicom.dcmread(path)
numpy_array = dataset.pixel_array
return numpy_array.dtype
The difference is that pydicom loads the original data as saved in the dataset (which is usually uint16 for MR data), while SimpleITK does some preprocessing (most likely applying the LUT) and returns the processed data as a float array.
In pydicom, to get data suitable for display, you have to apply some lookup table yourself, usually the one coming with the image.
If you have a modality LUT (not very common for MR data), you first have to apply that using apply_modality_lut, while for the VOI LUT you use apply_voi_lut. This will apply both modality and VOI LUT as found in the dataset:
ds = dcmread(fname)
arr = ds.pixel_array
out = apply_modality_lut(arr, ds)
display_data = apply_voi_lut(out, ds, index=0)
This can be savely used even if no modality or VOI LUT is present in the dataset - in this case just the input data is returned.
Note that there can be more than one VOI LUT in a DICOM image, for example to show different kinds of tissue - thus the index argument, though that is also not very common in MR images.
I am trying to import CT scan data into ImageJ/FIJI (There is HDF5 plugin in ImageJ/Fiji, however the synchrotron CT data has so large datasets.. so it was failed to open). The scan data (Image dataset) is saved as dataset into the hdf5 file. So I have to extract image dataset from the hdf5 file, then converted it into the Tiff file.
HdF5 File path is "F:/New_ESRF/SNT_BTO4/SNT_BTO4_S1/SNT_BTO4_S1_1_1pag_db0005_vol.hdf5"
Herein, 'SNT_BTO4_S1_1_1pag_db0005_vol.hdf5' is divided into several datasets, and the image dataset is in here:/entry0000/reconstruction/results/data
At the moment, I accessed to the image dataset using h5py. However, after that, I am in stuck to extract/save the dataset separately from the hdf5 file.
Which code is required to extract the image dataset from the hdf5 file?
After that, I am thinking of using from PIL to Image then convert the image into Tiff file. Can I get any advice on the code for this?
import numpy as np
import h5py
filename = "F:/New_ESRF/SNT_BTO4/SNT_BTO4_S1/SNT_BTO4_S1_1_1pag_db0005_vol.hdf5"
with h5py.File(filename,'r') as hdf:
base_items = list (hdf.items())
print('#Items in the base directory:', base_items)
#entry0000
G1 = hdf.get ('entry0000')
G1_items = list (G1.items())
print('#Items in entry0000', G1_items)
#reconstruction
G11 = G1.get ('/entry0000/reconstruction')
G11_items = list (G11.items())
print('#Items in reconstruction', G11_items)
#results_data
G12 = G11.get ('/entry0000/reconstruction/results')
G12_items = list (G12.items())
print('#Items in results', G12_items)
Extracting image data from an HDF5 file and converting to an image is a "relatively straight forward" 2 step process:
Access the data in the HDF5 file
Convert to an image with cv2 (or PIL)
A simple example is available here: How to extract individual JPEG images from a HDF5 file.
You can apply the same process to your file. Here is some pseudo-code. It's not complete because you don't show the shape of the image dataset (and the shape affects how to read the data). Also, you didn't say how many images are in dataset /entry0000/reconstruction/results/data --- does it have a single image or multiple images. If multiple images, which axis is the image counter?
import h5py
import cv2 ## for image conversion
filename = "F:/New_ESRF/SNT_BTO4/SNT_BTO4_S1/SNT_BTO4_S1_1_1pag_db0005_vol.hdf5"
with h5py.File(filename,'r') as hdf:
# get image dataset
img_ds = hdf['/entry0000/reconstruction/results/data']
print(f'Image Dataset info: Shape={img_ds.shape},Dtype={img_ds.dtype}')
## following depends on dataset shape/schema
## code below assumes images are along axis=0
for i in range(img_ds.shape[0]):
cv2.imwrite(f'test_img_{i:03}.tiff',img_ds[i,:]) # uses slice notation
# alternately load to a numpy array first
img_arr = img_ds[i,:] # slice notation gets [i,:,:,:]
cv2.imwrite(f'test_img_{i:03}.tiff',img_arr)
Note: you don't need to use .get() to get a dataset. You can simply reference the dataset path. Also, when you use a group object, use the relative path from the dataset to the group, not the absolute path. (You should modify your code to reflect these changes.) For example, the following are equivalent
G1 = hdf['entry0000']
## is the same as G1 = hdf.get('entry0000')
G11 = hdf['entry0000/reconstruction']
## is the same as G11 = hdf.get('entry0000/reconstruction')
## OR referencing G1 group object:
G11 = G1['reconstruction']
## is the same as G11 = G1.get('reconstruction')
I have a csv file that contains three columns namely (image file names, labels(in form of 0,1...), and class names). While another folder contains all the images. I want to read the images using this csv file and use them further for the task of image classification using deep learning models on python.
You can try this:
import pandas as pd
import cv2
import numpy as np
from pathlib import Path
df = pd.read_csv('path_to_csv.csv')
path = Path('path/to/img/dir') # relative path is OK
dataset = df.apply(lambda file_name, label:
return (cv2.imread(path/file_name), label),
axis=1)
dataset = np.asarray(dataset)
Now you'll have dataset that is a numpy matrix with images as column 1 and labels as column 2
I'm using OpenCV to import image. Of course you can use others like pillow too, but OpenCV is faster than pillow
I'm facing this problem: I would like to save the plot I produced inside an .h5 file.
What I'm doing now is::
import numpy as np
import matplotlib.pyplot as plt
data = np.random.randn(10,10)
fig = plt.scatter(data[0],data[1])
plt.savefig('image.jpg')
then I create the .h5 file as follow:
import cv2
import h5py
image = cv2.imread("image.jpg")
with h5py.File("h5_file.h5", 'a') as hf:
Xset = hf.create_dataset(
name="img",
data=image,
shape=image.shape,
maxshape=image.shape)
And, visualizing with HDFView, I can see the result as follow:
I need to open it "as Image" and I can see a picture for each "table" of RGB.
What I need instead is the result of Tools>Convert Image to HDF5 in the HDFView, which is the following:
Any suggestion? Or any other h5 visualizer to be used?
The problem is not the HDF5 data. I copied your code and ran to create the 2 files. A comparison of the data (as NumPy arrays) shows them to be identical. The difference is the attributes:
When you read the image with cv2.imread() and save the data, no attributes are created.
When you use Tools>Convert Image to HDF5 to create the file, 5 attributes are added to the dataset. (Notice this dataset has a different file icon in HDFView.) The attributes are shown on the Object Attribute Info panel in HDFView and/or you can get them with the following code:
Code:
for k,v in ds2.attrs.items():
print ('attr:',k,'=',v)
Output:
attr: CLASS = b'IMAGE'
attr: IMAGE_MINMAXRANGE = [ 0 255]
attr: IMAGE_SUBCLASS = b'IMAGE_TRUECOLOR'
attr: IMAGE_VERSION = b'1.2'
attr: INTERLACE_MODE = b'INTERLACE_PIXEL'
You can add these attributes to your first file (created with cv2.imread() ) using the code below:
Xset.attrs.create('CLASS','IMAGE',dtype='S6')
Xset.attrs.create('IMAGE_MINMAXRANGE',[0, 255],dtype=np.uint8)
Xset.attrs.create('IMAGE_SUBCLASS','IMAGE_TRUECOLOR',dtype='S16')
Xset.attrs.create('IMAGE_VERSION','1.2',dtype='S4')
Xset.attrs.create('INTERLACE_MODE','INTERLACE_PIXEL',dtype='S16')
Here is the image I see in HDFView after adding the attributes. It is almost the same - no more blurring, but the data point color is a different (gold vs blue). Not sure what causes that. Maybe something to do with the default pallete?
I am trying to write a custom input pipeline in Tensorflow for my dataset containing .fits files. I have a list of strings to the locations of the files, like so
pathlist = ['/path/to/file1', 'path/to/file2', ...]
Although the path naming convention has very specific subdirectories, this is a general example. I have written a short function, which when applied to each path element of this list, will spit out a numpy.ndarray with the appropriate data
import numpy as np
from astropy.io import fits
import tensorflow as tf
def path2im(path):
print(path)
hdulist = fits.open(path)
data = hdulist[1].data
data[np.isnan(data)] = 0
return tf.convert_to_tensor(data.astype(np.float32))
It basically opens the fits file from the path, and extracts the data along with dropping the NaNs and converting the array to a tensor. I am following the guidelines set down here (Loading Images in a Directory As Tensorflow Data set) for generating a tensorflow input pipeline. I start by defining a filename dataset from the list of paths, and then mapping the function over it.
filenames = tf.data.Dataset.list_files(pathlist)
ims = filenames.map(path2im)
When this is run, it prints the path not as a string, but as
Tensor("arg0:0", shape=(), dtype=string)
Which makes sense considering the filenames dataset contains tensors, as well as a huge error block in the map function which fails at this line
->hdulist = fits.open(path)
because fits.open(path) accepts a string as an argument for the path. Is there any way to rectify this issue? I cannot find a way to convert the string tensor into a string without starting a session and using .eval(), which I do not want to do in this initialization phase.
The main idea of the Dataset API is to have your data preprocessing part of the TensorFlow graph, so for example you can just specify a filename as placeholder when you'll run the TensorFlow graph.
This is then completely expected that type of the object filenames is a Tensor, and if you want to convert it as a string, you'll have to evaluate it using a Session.
You might want to have a look at this introductory guide to datasets.