I have a problem about reshaping dataframe for implementing CNN.
My dataframe shape : train.shape -> (230, 67502).
Then I wrote a code shown below.
Y_train = train["Label"]
X_train = train.drop(labels = ["Label"],axis = 1)
When I run this code below for plotting some images by iloc, It throws an error
img = X_train.iloc[0].to_numpy()
img = np.pad(img, (0, (67600-img.shape[0])), 'constant').reshape((260, 260))
plt.imshow(img,cmap='gray')
plt.title(train.iloc[0,0])
plt.axis("off")
plt.show()
Then I normalize X_train
X_train = X_train / 255.0
print("x_train shape: ",X_train.shape)
When I reshape X_train , it throws an error
X_train = X_train.values.reshape(-1, 260, 260)
print("x_train shape: ",X_train.shape)
ValueError: cannot reshape array of size 15525000 into shape (260,260)
How can I fix the issue?
Are you absolutely sure you need a 260 * 260 image? As 67500 == 270 * 250, you can try that and see how it looks!
Otherwise you would need to PAD - how exactly would depend upon your image.
But, one of the simplest might be to add 100 more 0's at the end to make it 67600 - hence 260 * 260
Well, 260*260 is 67600 and not 67500. So you can't cast your array into those dimensions.
To actually cast it into those dimensions you would need to pad or normalize the source image arrays. For example, check the Keras pad_sequences functionality documentation on dealing with this kind of issues.
Solution :
X_train = np.pad(X_train, ((0,0), (0, (67600-X_train.shape[1]))), 'constant').reshape(-1, 260, 260)
Related
Im new to tensorflow and Im trying to feed some data with tensorflow.Dataset. Im using Cityscape dataset with 8 different classes. Here is my code:
import os
import cv2
import numpy as np
import tensorflow as tf
H = 256
W = 256
id2cat = np.array([0,0,0,0,0,0,0, 1,1,1,1, 2,2,2,2,2,2, 3,3,3,3, 4,4, 5, 6,6, 7,7,7,7,7,7,7,7,7])
def readImage(x):
x = cv2.imread(x, cv2.IMREAD_COLOR)
x = cv2.resize(x, (W, H))
x = x / 255.0
x = x.astype(np.float32)
return x
def readMask(path):
mask = cv2.imread(path, 0)
mask = cv2.resize(mask, (W, H))
mask = id2cat[mask]
return mask.astype(np.int32)
def preprocess(x, y):
def f(x, y):
image = readImage(x)
mask = readMask(y)
return image, mask
image, mask = tf.numpy_function(f, [x, y], [tf.float32, tf.int32])
mask = tf.one_hot(mask, 3, dtype=tf.int32)
image.set_shape([H, W, 3])
mask.set_shape([H, W, 3])
return image, mask
def tf_dataset(x, y, batch=8):
dataset = tf.data.Dataset.from_tensor_slices((x, y))
dataset = dataset.shuffle(buffer_size=5000)
dataset = dataset.map(preprocess)
dataset = dataset.batch(batch)
dataset = dataset.repeat()
dataset = dataset.prefetch(2)
return dataset
def loadCityscape():
trainPath = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'datasets\\Cityscape\\train')
imagesPath = os.path.join(trainPath, 'images')
maskPath = os.path.join(trainPath, 'masks')
images = []
masks = []
print('Loading images and masks for Cityscape dataset...')
for image in os.listdir(imagesPath):
images.append(readImage(os.path.join(imagesPath, image)))
for mask in os.listdir(maskPath):
if 'label' in mask:
masks.append(readMask(os.path.join(maskPath, mask)))
print('Loaded {} images\n'.format(len(images)))
return images, masks
images, masks = loadCityscape()
dataset = tf_dataset(images, masks, batch=8)
print(dataset)
That last print(dataset) shows:
<PrefetchDataset shapes: ((None, 256, 256, 3), (None, 256, 256, 3)), types: (tf.float32, tf.int32)>
Why am I obtaining (None, 256, 256, 3) instead of (8, 256, 256, 3)? I also have some doubts about how to iterate over this dataset.
Thanks a lot.
Tensorflow is a graph based mathematical framework that abstracts for you all of those complex vectorial or matricial operations you face, particularly in machine learning.
What the developers though is that it would be unconfortable to specify every single time how many input vectors you need to pass in your model for the training, so they decided to abstract it for you.
You will not interested if your model is fed with one single or thousands samples as long as the output matches with the input dimension (but also any internal operation should match in dimensions!).
So the None size is a placeholder for a possible changing shape, that is usually the batch size of the input.
We need a placeholder because (None, 2) is a different shape with respect of just (2,), because in the first case we know we will face 2 dimensions.
Even if the None dimension is unknown when you "compile" your model, it will be evaluated only when it is strictly needed, in other words when you run it. In this way your model will be happy to run on a batch size of 64 as like as 128 samples.
For the rest a (non-scalar) Tensor behaves like a normal numpy array:
tensor1 = tf.constant([ 0, 1, 2, 3]) # shape (4, )
tensor2 = tf.constant([ [0], [1], [2], [3]]) # shape (4, 1)
for x in tensor1:
print(x) # 0, 1, 2, 3
for x in tensor2:
print(x) # Tensor([0]), Tensor([1]), Tensor([2]), Tensor([3])
The only difference is that it can be allocated into any supported device memory (CPU / Cuda GPU).
Iterating through the dataset is just like slicing it at (usually) constant sizes, where that constant is your batch size, which will fill that empty None dimension.
This line of code will be responsible of slicing your dataset into "sub-tensors" ("sub-arrays") composed by its samples:
dataset = dataset.batch(N)
# iterating over it:
for batch in dataset: # I'm taking N samples here
...
Your "runtime" shape will be (N, 256, 256, 3), but if you will try to take an element from the dataset it could still have None in the shape... That's because we can't guarantee, for example, that the dimension of the dataset is exactly divisible by the batch size, so some trailing samples of a variable shape could still be possible. You will hardly get rid off that None dimension, but in some custom methods of your model you could achieve that.
If you are still unconfortable with tensors there is the tensor.numpy() method that gives you back a numpy array, but at the cost of copying it (usually to your CPU). This is not available in every step of the process.
There are many way to define a dataset in tensorflow, I suggest to read how they think you should build an input pipeline, because it will make your life easier if you understand how much tensorflow takes your code at higher levels of abstraction.
I want to load the mnist dataset to the mobilenet V1 CNN
then, I faced with this problem
ValueError: Error when checking input: expected input_1 to have shape (32, 32, 3) but got array with shape (28, 28, 1)
Below is my code
image_data, label_data = data['image'], data['label']
idx_list = {}
for i in range(10):
idx_list[i] = np.where(label_data == i) # return tuple dtype (rows indices, column indices)
selected_test_sample_indices = {}
for label in range(10):
selected_test_sample_indices[label] = random.sample(set(idx_list[label][0]), int(len(idx_list[label][0]) * 0.2))
selected_train_sample_indicies = {}
for label in range(10):
selected_train_sample_indicies[label] = list(set(idx_list[label][0])- set(selected_test_sample_indices[label]))
train_data_indicies, test_data_indicies = [],[]
for label, indicies in selected_train_sample_indicies.items():
train_data_indicies = train_data_indicies + indicies # merge 2 list
for label, indicies in selected_test_sample_indices.items():
test_data_indicies = test_data_indicies + indicies
random.shuffle(train_data_indicies)
random.shuffle(test_data_indicies)
y_train_data = np.array([label_data[idx] for idx in train_data_indicies])
X_train_data = np.array([image_data[idx] for idx in train_data_indicies])
y_test_data = np.array([label_data[idx] for idx in test_data_indicies])
X_test_data = np.array([image_data[idx] for idx in test_data_indicies])
number_of_classes = 10
y_train = y_train_data
y_test = y_test_data
X_train = X_train_data.reshape(X_train_data.shape[0], img_rows, img_cols, 1)
X_test = X_test_data.reshape(X_test_data.shape[0], img_rows, img_cols, 1)```
Whenn I tried to reshape I got the following error
ValueError: cannot reshape array of size 11146912 into shape (14218,32,32,1)
when I change it to (4500,32,32,3), the sum is lower than 11146912
It really confused me.
Please help me to fix this bug.
The MNIST dataset contains images in grayscale with the size of 28x28 pixels. That is why the shape of each image is (28, 28, 1) with each value between 0-255. Here's another stackoverflow question with the same problem. The most valid answer is to convert the grayscale images into rgb images and then resizing the images.
Well after converting the grayscale images to rgb images the shape of your images will change from
28 x 28 x 1 to 28 x 28 x 3
Then you need to resize it to 32. You can use openCV library for that.
resized_image = cv2.resize(image, (32, 32))
Then your resized_image shape would be 32 x 32 x 3
Here I have one tensor with (1084, 1625, 3) shape.
And I need to reshape it to (none,none,none ,3).
how can i do that?
I used this code but it does not work.
image = tf.cast(img, tf.float32)
image = (image / 127.5) - 1
I don't think you can do that. I think what you're trying to do is turn a 3D tensor into a 4D tensor. I'm guessing this is the origin of your problem. You can do this to add a 4th dimension, because Tensorflow needs it:
import tensorflow as tf
tensor = tf.random.uniform((100, 100, 3), 0, 256, dtype=tf.int32)
new = tf.expand_dims(tensor, axis=0)
print(new.shape)
Out[14]: TensorShape([1, 100, 100, 3])
But then I could be wrong. If this is the case you can provide your error traceback and code.
I am running a neural network model on some images. Initially, for training, I converted all the images into a pandas dataframe of dimension (# of images in the dataset) x r x g x b, where r, g, b are the colour values of each image. Now when I am trying to test the model on a single externally downloaded image, it is giving a dimension error as, obviously, the image's dimension is only r x g x b. How do I add the number of images as a dimension into this image?
EDIT: Here's the code:
#load the data as a pandas data frame
import pandas as pd
dataset = pd.read_csv(os.path.join(data_path, 'data.csv'))
# split into input (X) and output (Y) variables
X = dataset.values[:,0]
Y = dataset.values[:,1]
# Load all the images and resize them into a single numpy array of consistent dimension
from scipy.misc import imresize
from scipy.misc import imread
import numpy as np
temp = []
for img_name in X:
img_path = os.path.join(data_dir, 'Train', img_name)
img = imread(img_path)
img = imresize(img, (32, 32))
img = img.astype('float32')
temp.append(img)
X = np.stack(temp)
# Convert the data classes from words into a number format readable by the program
from sklearn.preprocessing import LabelEncoder
lb = LabelEncoder()
Y = lb.fit_transform(Y)
Y = keras.utils.np_utils.to_categorical(Y)
# Split the data into 67% for training and 33% for testing
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33)
### Define the neural network model
### Compile and train the model on the data
### Evaluate it
# Test it on an externally downloaded image
img = imread(os.path.join(image_folder, downloaded_image)).astype('float32')
plt.imshow(imresize(img, (128, 128)))
print('X_train shape: ', X_train.shape)
print('Downloaded image shape: ', img.shape)
This returns:
X_train shape: (13338, 32, 32, 3)
Downloaded image shape: (448, 720, 3)
I want to make the downloaded image's shape to be (1, 448, 720, 3) so that it matches the dimensions of X_train's shape, because when I try to predict the class of the downloaded image, it returns a dimension error:
pred = cnn_model.predict_classes(img)
print('Predicted:', lb.inverse_transform(pred))
This returns:
ValueError: Error when checking : expected conv2d_71_input to have 4 dimensions, but got array with shape (960, 640, 3)
From your description, it seems like you don't really mean to use the number of images as a feature, but rather as a sample weight. Conceptually, you probably want to transform
k x r x g x b
to
r x g x b
... # repeat k times
r x g x b
which would naturally make the input and output dimensions identical, BTW. If this increases learning time too much, and your library has a sample weight parameter, you should consider using it.
If you'd like to just technically add a dimension, you can use np.expand_dims:
>>> np.expand_dims(np.array([[1, 2, 3], [3, 4, 5]]), axis=0).shape
(1, 2, 3)
However, I cannot say I'm sure that this is fundamentally what you what.
I am trying to implement a custom rbf kernel function. However I am getting the following error. I am not sure why it is expected a certain amount of data points?
Error occurs in this line of code:
rbf_y = rbf_kernel.predict(X_test)
Code
def myKernel(x,y):
pairwise_dists = squareform(pdist(x, 'euclidean'))
K = scip.exp(-pairwise_dists ** 2 / .08 ** 2)
return K
rbf_kernel = svm.SVC(kernel=myKernel, C=1).fit(X_train, Y_train.ravel())
rbf_y = rbf_kernel.predict(X_test)
rbf_accuracy = accuracy_score(Y_test, rbf_y)
Error:
ValueError: X.shape[1] = 15510 should be equal to 31488, the number of samples at training time
Data Shape
X_train shape: (31488, 128)
X_test shape: (15510, 128)
Y_train shape: (31488, 1)
Y_test shape: (15510, 1)
Return Shape from Kernel
myKernel(X_train, X_train).shape = (31488, 31488)
A custom kernel kernel(X, Y) should compute a similarity measure between the matrix X and the matrix Y, and the output should be of shape [X.shape[0], Y.shape[0]]. Your kernel function ignores Y, and returns a matrix of shape [X.shape[0], X.shape[0]], which leads to the error you are seeing.
To fix the issue, implement a kernel function that computes a kernel matrix of the correct shape. Scikit-learn's custom kernels documentation has some simple examples of how this might work.
In the case of your specific kernel, you might try cdist(x, y) in place of squareform(pdist(x)).