In the original Caffe framework, there was an executable under caffe/build/tools called convert_imageset, which took a directory of JPEG images and a text file with labels for each image, and output an LMDB that could be fed to a Caffe model to train, test, etc.
What is the best way to convert raw JPEG images and labels to an LMDB that Caffe2 can ingest using the AddInput() function from this MNIST tutorial on the Caffe2 website?
According to my research, you cannot simply create an LMDB file using this tool and feed a Caffe2 model.
The tutorial script just downloads two LMDBs (mnist-train-nchw-lmdb and mnist-test-nchw-lmdb) and passes them to AddInput(), but gives no insight as to how the LMDBs were created.
There is a binary called make_image_db.cc which does precisely what you are describing. It is located in caffe2/build/bin/make_image_db:
// This script converts an image dataset to a database.
//
// caffe2::FLAGS_input_folder is the root folder that holds all the images
//
// caffe2::FLAGS_list_file is the path to a file containing a list of files
// and their labels, as follows:
//
// subfolder1/file1.JPEG 7
// subfolder1/file2.JPEG 7
// subfolder2/file1.JPEG 8
// ...
As described in https://github.com/caffe2/caffe2/issues/1755 you can use the binary in the following way (also with fewer parameters):
caffe2/build/bin/make_image_db -color -db lmdb -input_folder ./some_input_folder
-list_file ./labels_file -num_threads 10 -output_db_name ./some_output_folder -raw -scale 256 -shuffle
A full Caffe2 example on how to create and read a lmdb database (for random images) can be found in the official github repository and can be used as a skeleton to adapt to your own images https://github.com/caffe2/caffe2/blob/master/caffe2/python/examples/lmdb_create_example.py. Since I have not used this method yet, I will simply copy the example. In order to create the database, one can use:
import argparse
import numpy as np
import lmdb
from caffe2.proto import caffe2_pb2
from caffe2.python import workspace, model_helper
def create_db(output_file):
print(">>> Write database...")
LMDB_MAP_SIZE = 1 << 40 # MODIFY
env = lmdb.open(output_file, map_size=LMDB_MAP_SIZE)
checksum = 0
with env.begin(write=True) as txn:
for j in range(0, 128):
# MODIFY: add your own data reader / creator
label = j % 10
width = 64
height = 32
img_data = np.random.rand(3, width, height)
# ...
# Create TensorProtos
tensor_protos = caffe2_pb2.TensorProtos()
img_tensor = tensor_protos.protos.add()
img_tensor.dims.extend(img_data.shape)
img_tensor.data_type = 1
flatten_img = img_data.reshape(np.prod(img_data.shape))
img_tensor.float_data.extend(flatten_img)
label_tensor = tensor_protos.protos.add()
label_tensor.data_type = 2
label_tensor.int32_data.append(label)
txn.put(
'{}'.format(j).encode('ascii'),
tensor_protos.SerializeToString()
)
checksum += np.sum(img_data) * label
if (j % 16 == 0):
print("Inserted {} rows".format(j))
print("Checksum/write: {}".format(int(checksum)))
return checksum
The database can then by loaded by:
def read_db_with_caffe2(db_file, expected_checksum):
print(">>> Read database...")
model = model_helper.ModelHelper(name="lmdbtest")
batch_size = 32
data, label = model.TensorProtosDBInput(
[], ["data", "label"], batch_size=batch_size,
db=db_file, db_type="lmdb")
checksum = 0
workspace.RunNetOnce(model.param_init_net)
workspace.CreateNet(model.net)
for _ in range(0, 4):
workspace.RunNet(model.net.Proto().name)
img_datas = workspace.FetchBlob("data")
labels = workspace.FetchBlob("label")
for j in range(batch_size):
checksum += np.sum(img_datas[j, :]) * labels[j]
print("Checksum/read: {}".format(int(checksum)))
assert np.abs(expected_checksum - checksum < 0.1), \
"Read/write checksums dont match"
Last but not least, there is also a tutorial on how to create a minidb database: https://github.com/caffe2/caffe2/blob/master/caffe2/python/tutorials/create_your_own_dataset.ipynb. For this, one could use the following function:
def write_db(db_type, db_name, features, labels):
db = core.C.create_db(db_type, db_name, core.C.Mode.write)
transaction = db.new_transaction()
for i in range(features.shape[0]):
feature_and_label = caffe2_pb2.TensorProtos()
feature_and_label.protos.extend([
utils.NumpyArrayToCaffe2Tensor(features[i]),
utils.NumpyArrayToCaffe2Tensor(labels[i])])
transaction.put(
'train_%03d'.format(i),
feature_and_label.SerializeToString())
# Close the transaction, and then close the db.
del transaction
del db
Features would be a tensor containing your images as numpy arrays. Labels are the corresponding true labels for the features. You would then simply call the function as
write_db("minidb", "train_images.minidb", train_features, train_labels)
Finally, you would load the images from the database by
net_proto = core.Net("example_reader")
dbreader = net_proto.CreateDB([], "dbreader", db="train_images.minidb", db_type="minidb")
net_proto.TensorProtosDBInput([dbreader], ["X", "Y"], batch_size=16)
for create database in lmbd:
create the train data folder
create train.txt file conataining filename label
create validation data folder
create val.txt file contatining filename and label
edit this file
gedit examples/imagenet/create_imagenet.sh
EXAMPLE= path to where *.lmbd folder wil be stored
DATA= path where val.txt and train.txt is present
TOOLS=build/tools
TRAIN_DATA_ROOT=test/make_caffe_data/train/ # path to trainfiles
VAL_DATA_ROOT=test/make_caffe_data/val/ # path to test_files
Set RESIZE=true to resize the images to 256x256. Leave as false if images have
already been resized using another tool.
RESIZE=true
./examples/imagenet/create_imagenet.sh
Related
Hi everyone I'm facing an issue after that I elaborate images and labels. To create an unique dataset I use the zip function. After the elaboration both images and labels are 18k and it's correct but when I call the zip(image,labels), items become 563.
Here some code to let you to understand:
# Map the load_and_preprocess_image function over the dataset of image paths
images = image_paths.map(load_and_preprocess_image)
# Map the extract_label function over the dataset of image paths
labels = image_paths.map(extract_label)
# Zip the labels and images together to create a dataset of (image, label) pairs
#HERE SOMETHING STRANGE HAPPENS
data = tf.data.Dataset.zip((images,labels))
# Shuffle and batch the data
data = data.shuffle(buffer_size=1000).batch(32)
# Split the data into train and test sets
data = data.shuffle(buffer_size=len(data))
# Convert the dataset into a collection of data
num_train = int(0.8 * len(data))
train_data = image_paths.take(num_train)
val_data = image_paths.skip(num_train)
I cannot see where is the error. Can you help me plese? Thanks
I'd like to have a dataset of 18k images,labels
tf's zip
tf.data.Dataset.zip is not like Python's zip. The tf.data.Dataset.zip's input is tf datasets. You may check the images/label return from your map function is the correct tf.Dataset object.
check tf.ds
make sure your image/label is correct tf.ds.
print("ele: ", images_dataset.element_spec)
print("num: ", images_dataset.cardinality().numpy())
print("ele: ", labels_dataset.element_spec)
print("num: ", labels_dataset.cardinality().numpy())
workaround
In your case, combine the image and label processing in one map function and return both to bypass to use tf.data.Dataset.zip:
# load_and_preprocess_image_and_label
def load_and_preprocess_image_and_label(image_path):
""" load image and label then some operations """
return image, label
# Map the load_and_preprocess_image function over the dataset of image/label paths
train_list = tf.data.Dataset.list_files(str(PATH / 'train/*.jpg'))
data = train_list.map(load_and_preprocess_image_and_label,
num_parallel_calls=tf.data.AUTOTUNE)
My data directory is as below :
ㄴTrain
ㄴpatient_ID_0
ㄴimage0.png
ㄴimage1.png
ㄴimage2.png
ㄴpatient_ID_1
ㄴTest
What I want to do is : train an image-level classifier, and use majority-vote to get patient-level predictions. Image-level labels are the same as patient-level labels (if a patient is labeled as 0, all his/her images are labeled as 0).
To do so, I would like to
save (img, file path, label) to Dataset
during test & evaluation phase, filter out image-level predictions for each patient using file path
I found a piece of matlab code that does a similar job (In matlab, ImageDatastore object manages a collection of image files, with labels saved as the property "Labels" and file paths saved as the property "Files") :
myImds = imageDatastore(['D:\Colorectal_MSI_Project\',tissueType,'_TILES_NORM_SPLIT\TEST_noBalance'],...
'IncludeSubfolders',true,'LabelSource','foldernames');
[YPred,predScoresExternal] =...
classify(myNet,myImds,...
'MiniBatchSize',1024,'ExecutionEnvironment','gpu');
trueLabels = cellstr(myImds.Labels);
predLabels = cellstr(YPred);
% find classification accuracy on a per patient level
for i = 1:numel(allUniqNames)
currName = allUniqNames{i};
currMask = contains(myImds.Files,currName);
% count hits
trueMSIMUT = sum(strcmp(trueLabels(currMask),'MSIMUT'));
predMSIMUT = sum(strcmp(predLabels(currMask),'MSIMUT'));
trueMSS = sum(strcmp(trueLabels(currMask),'MSS'));
predMSS = sum(strcmp(predLabels(currMask),'MSS'));
trueIsMSIMUT(i) = trueMSIMUT / (trueMSIMUT+trueMSS);
trueIsMSIMUTbool(i) = trueMSIMUT >= trueMSS;
predIsMSIMUT(i) = predMSIMUT / (predMSIMUT+predMSS); % majority vote
AllNumImages(i) = sum(currMask);
disp(['patient ',currName,'; # tiles ',num2str(sum(currMask)),...
'; true is MSIMUT: ',num2str(trueIsMSIMUT(i)),'; pred is MSIMUT: ',num2str(predIsMSIMUT(i)),...
'; score: ',num2str(predIsMSIMUTScore(i))]);
Any suggestions on how to implement this in tensorflow would be much appreciated.
I have a folder of jpeg images that I'm trying to convert to a folder of tfrecords. The best I can do, from this code, is to write all jpegs to one tfrecords file, but I'm not sure how to use that (large tfrecords file) AND my other starter code requires individual tfrecord files for each image. For example, I was given a folder of 5 tfrecs to use to begin with.
# Source: https://stackoverflow.com/questions/33849617/how-do-i-convert-a-directory-of-jpeg-images-to-tfrecords-file-in-tensorflow
# Note: modified from source
def _int64_feature(value):
return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
def _bytes_feature(value):
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
# images and labels array as input
def convert_to(images, labels, output_directory, name):
num_examples = labels.shape[0]
if images.shape[0] != num_examples:
raise ValueError("Images size %d does not match label size %d." %
(images.shape[0], num_examples))
rows = images.shape[1]
cols = images.shape[2]
depth = 1
filename = os.path.join(output_directory, name + '.tfrecords')
print('Writing', filename)
writer = tf.python_io.TFRecordWriter(filename)
for index in range(num_examples):
image_raw = images[index].tobytes()
example = tf.train.Example(features=tf.train.Features(feature={
'height': _int64_feature(rows),
'width': _int64_feature(cols),
'depth': _int64_feature(depth),
'label': _int64_feature(int(labels[index])),
'image_raw': _bytes_feature(image_raw)}))
writer.write(example.SerializeToString())
Above is my function convert_to, can this be changed to answer my question? Below is the rest, you can see the 2nd to last line (#s) it is correctly given the array and labels from the 300 images.
def read_image(file_name, images_path):
image = skimage.io.imread(images_path + file_name)
return image
def extract_image_index_make_label(img_name):
#remove_ext = img_name.split(".")[0]
# name, serie, repetition, char = remove_ext.split("_")
# label = int(char) + 1000 * int(repetition) + 1000_000 * int(serie)
label = random.randint(1,300)
return label
images_path = "/content/monet_jpg/"
image_list = os.listdir(images_path)
images = []
labels = []
for img_name in tqdm(image_list):
images.append(read_image(img_name, images_path))
labels.append(extract_image_index_make_label(img_name))
images_array = np.array(images)
labels = np.array(labels)
#print(images_array.shape)
print(images_array.shape, labels.shape)
# (300, 256, 256, 3) (300,)
convert_to(images_array, labels, ".", "ALL_MONET_TFREC")
Even using a folder of Tfrecs would still have efficiency benefits over a folder of jpegs correct? Anyway that is what my starter code is setup to use.
I can give you some examples from a real-case situation I have been working on:
Set of images (15000 .jpeg images, 15 classes (i.e. 1000 per class), 224x224x3). The size on disk is 398 MB, while the set of records in TFRecord is 336 MB. In this, we also consider the overhead of other metadata attachment to the TFRecord (for example, a str label attached to every protobuffer instance. Therefore, you may see a reduction of 398-336 = 62 MB, in spite of additional metadata being included. Consider that further reduction could be made if the proto contained only the serialized image.
Training speed. If using tf.data.Dataset() together with TFRecord() dataset, the speed of the training increases. For example, in the same scenario with the dataset, the length of an epoch decreased with ~30 seconds in my case when using tf.data.Dataset() + TFRecord() versus tf.data.Dataset() + tf.data.Dataset.from_tensor_slices() (without changing anything related to the pipeline except for the ones above, no other network, HParams, CPU, GPU etc.)
Indeed, those differences are specific to the task, but I have noticed overall improvements in :
Size alloted on disk (allocating one big chunk takes a bit less space than size resulting from splitting into a lot of chunks).
Training speed (which I find even more important, in my view).
Note also some other advantages not included in my previous example:
Fast I/O: the TFRecord format can be read with parallel I/O
operations, which is useful for TPUs or multiple hosts.
I need to download single or multiple images from the collection of ee. (preferably multiple but I can just put a single image code in a loop).
My main problem --> Download every month image from a start date to end date for a specific location (lat: "", long: "") with zoom 9
I am trying to download historical simple satellite data from the SKYSAT/GEN-A/PUBLIC/ORTHO/RGB. This gives an image like -->
https://developers.google.com/earth-engine/datasets/catalog/SKYSAT_GEN-A_PUBLIC_ORTHO_RGB#description
I am working on this in python. So this code will give me the collection but I can't download the entire collection, I need to select an image out of it.
import ee
# Load a Landsat 8 ImageCollection for a single path-row.
collection = (ee.ImageCollection('SKYSAT/GEN-A/PUBLIC/ORTHO/RGB').filterDate('2016-03-01', '2018-03-01'))
#pp.pprint('Collection: '+str(collection.getInfo())+'\n')
# Get the number of images.
count = collection.size()
print('Count: ', str(count.getInfo()))
image = ee.Image(collection.sort('CLOUD_COVER').first())
Here the image contains the <ee.image.Image at 0x23d6cf5dc70> property but I don't know how to download it.
Also, how do I specify I want it for a specific location(lat, long) with zoom 19.
Thanks for anything
You can download your whole images in the image collection using geetools.
You can install with pip install geetools.
Then, change the area, dates, and folder name(if the folder does not exist, it will be created on your drive) and execute this code.
import geetools
StartDate= ee.Date.fromYMD(2018,1,16)
EndDate = ee.Date.fromYMD(2021,10,17)
Area = ee.Geometry.Polygon(
[[[-97.93534101621628, 49.493287372441955],
[-97.93534101621628, 49.49105034378085],
[-97.93049158231736, 49.49105034378085],
[-97.93049158231736, 49.493287372441955]]])
collection =(ee.ImageCollection('SKYSAT/GEN-A/PUBLIC/ORTHO/RGB')
.filterDate(StartDate,EndDate)
.filterBounds(Area)
data_type = 'float32'
name_pattern = '{system_date}'
date_pattern = 'yMMdd' # dd: day, MMM: month (JAN), y: year
scale = 10
folder_name = 'GEE_Images'
tasks = geetools.batch.Export.imagecollection.toDrive(
collection=collection,
folder=folder_name ,
region=Area ,
namePattern=name_pattern,
scale=scale,
dataType=data_type,
datePattern=date_pattern,
verbose=True,
maxPixels=1e10
)
Or;
You can use the Earth Engine library to export images.
nimg = collection.toList(collection.size().getInfo()).size().getInfo()
for i in range(nimg):
img = ee.Image(collection.toList(nimg).get(i))
date = img.date().format('yyyy-MM-dd').getInfo()
task = ee.batch.Export.image.toDrive(img.toFloat(),
description=date,
folder='Folder_Name',
fileNamePrefix= date,
region = Area,
dimensions = (256,256),
# fileFormat = 'TFRecord',
maxPixels = 1e10)
task.start()
There are two file formats; TFRecord and GeoTIFF. The default format is GeoTIFF. Also, you can extract images with specific dimensions as shown above. If you want to download images with a scale factor, just remove the dimension line and add a scale factor instead of it.
You can read this document for more information.
Insert your region of analysis (geom) by constructing a bounding box. Then, use the code below to batch download the images.
// Example geometry. You could also insert points etc.
var geom = ee.Geometry.Polygon(
[[[-116.8, 44.7],
[-116.8, 42.6],
[-110.6, 42.6],
[-110.6, 44.7]]], None, False)
for (var i = 0; i < count ; i++) {
var img = ee.Image(collection.toList(1, i).get(0));
var geom = img.geometry().getInfo();
Export.image(img, img.get('system:index').getInfo(), {crs: crs, scale: scale, region: geom});
}
I have a directory of images with random ids and a text file which has ids and their corresponding label. I was wondering if there is a way to read the data directly from disk and not loading the entire dataset in ram as a matrix.
I know it can be done by using the method of python generators and then follow up with placeholders to feed the data.
def generator_(path1,filename):
.
.
yield x,y
x=tf.placeholder(tf.float32,shape=[None,w,h,3])
y=tf.placeholder(tf.float32,shape=[None,n_c])
x,y=generator_(path_image,'labels.txt')
But what is the better way to do it using tf.data api ?
Supposing your labels.txt has such as structure (comma-separated image IDs and labels):
1, 0
2, 2
3, 1
...
42, 2
and your images are stored like:
/data/
|---- image1.jpg
|---- image2.jpg
...
|---- image42.jpg
You could then use tf.data in such a way:
import tensorflow as tf
def generate_parser(separator=",", image_path=["/data/image", ".jpg"]):
image_path = [tf.constant(image_path[0]), tf.constant(image_path[1])]
def _parse_data(line):
# Split the line according to separator:
line_split = tf.string_split([line], separator)
# Convert label value to int:
label = tf.string_to_number(line_split.values[1], out_type=tf.int32)
# Build complete image path from ID:
image_filepath = image_path[0] + line_split.values[0] + image_path[1]
# Open image:
image_string = tf.read_file(image_filepath)
image_decoded = tf.image.decode_image(image_string)
return image_decoded, label
return _parse_data
label_file = "/var/data/labels.txt"
dataset = (tf.data.TextLineDataset([label_file])
.map(generate_parser(separator=",", image_path=["/data/image", ".jpg"])))
# add .batch(), .repeat(), etc.