We Keep Coding

Taking the same number of elements from each class [duplicate]

I am using sklearn for multi-classification task. I need to split alldata into train_set and test_set. I want to take randomly the same sample number from each class.
Actually, I amusing this function
X_train, X_test, y_train, y_test = cross_validation.train_test_split(Data, Target, test_size=0.3, random_state=0)
but it gives unbalanced dataset! Any suggestion.

Although Christian's suggestion is correct, technically train_test_split should give you stratified results by using the stratify param.
So you could do:
X_train, X_test, y_train, y_test = cross_validation.train_test_split(Data, Target, test_size=0.3, random_state=0, stratify=Target)
The trick here is that it starts from version 0.17 in sklearn.
From the documentation about the parameter stratify:
stratify : array-like or None (default is None)
If not None, data is split in a stratified fashion, using this as the labels array.
New in version 0.17: stratify splitting

You can use StratifiedShuffleSplit to create datasets featuring the same percentage of classes as the original one:
import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit
X = np.array([[1, 3], [3, 7], [2, 4], [4, 8]])
y = np.array([0, 1, 0, 1])
stratSplit = StratifiedShuffleSplit(y, n_iter=1, test_size=0.5, random_state=42)
for train_idx, test_idx in stratSplit:
X_train=X[train_idx]
y_train=y[train_idx]
print(X_train)
# [[3 7]
# [2 4]]
print(y_train)
# [1 0]

If the classes are not balanced but you want the split to be balanced, then stratifying isn't going to help. There doesn't seem to be a method for doing balanced sampling in sklearn but it's kind of easy using basic numpy, for example a function like this might help you:
def split_balanced(data, target, test_size=0.2):
classes = np.unique(target)
# can give test_size as fraction of input data size of number of samples
if test_size<1:
n_test = np.round(len(target)*test_size)
else:
n_test = test_size
n_train = max(0,len(target)-n_test)
n_train_per_class = max(1,int(np.floor(n_train/len(classes))))
n_test_per_class = max(1,int(np.floor(n_test/len(classes))))
ixs = []
for cl in classes:
if (n_train_per_class+n_test_per_class) > np.sum(target==cl):
# if data has too few samples for this class, do upsampling
# split the data to training and testing before sampling so data points won't be
# shared among training and test data
splitix = int(np.ceil(n_train_per_class/(n_train_per_class+n_test_per_class)*np.sum(target==cl)))
ixs.append(np.r_[np.random.choice(np.nonzero(target==cl)[0][:splitix], n_train_per_class),
np.random.choice(np.nonzero(target==cl)[0][splitix:], n_test_per_class)])
else:
ixs.append(np.random.choice(np.nonzero(target==cl)[0], n_train_per_class+n_test_per_class,
replace=False))
# take same num of samples from all classes
ix_train = np.concatenate([x[:n_train_per_class] for x in ixs])
ix_test = np.concatenate([x[n_train_per_class:(n_train_per_class+n_test_per_class)] for x in ixs])
X_train = data[ix_train,:]
X_test = data[ix_test,:]
y_train = target[ix_train]
y_test = target[ix_test]
return X_train, X_test, y_train, y_test
Note that if you use this and sample more points per class than in the input data, then those will be upsampled (sample with replacement). As a result, some data points will appear multiple times and this may have an effect on the accuracy measures etc. And if some class has only one data point, there will be an error. You can easily check the numbers of points per class for example with np.unique(target, return_counts=True)

Another approach is to over- or under- sample from your stratified test/train split. The imbalanced-learn library is quite handy for this, specially useful if you are doing online learning & want to guarantee balanced train data within your pipelines.
from imblearn.pipeline import Pipeline as ImbalancePipeline
model = ImbalancePipeline(steps=[
('data_balancer', RandomOverSampler()),
('classifier', SVC()),
])

This is my implementation that I use to get train/test data indexes
def get_safe_balanced_split(target, trainSize=0.8, getTestIndexes=True, shuffle=False, seed=None):
classes, counts = np.unique(target, return_counts=True)
nPerClass = float(len(target))*float(trainSize)/float(len(classes))
if nPerClass > np.min(counts):
print("Insufficient data to produce a balanced training data split.")
print("Classes found %s"%classes)
print("Classes count %s"%counts)
ts = float(trainSize*np.min(counts)*len(classes)) / float(len(target))
print("trainSize is reset from %s to %s"%(trainSize, ts))
trainSize = ts
nPerClass = float(len(target))*float(trainSize)/float(len(classes))
# get number of classes
nPerClass = int(nPerClass)
print("Data splitting on %i classes and returning %i per class"%(len(classes),nPerClass ))
# get indexes
trainIndexes = []
for c in classes:
if seed is not None:
np.random.seed(seed)
cIdxs = np.where(target==c)[0]
cIdxs = np.random.choice(cIdxs, nPerClass, replace=False)
trainIndexes.extend(cIdxs)
# get test indexes
testIndexes = None
if getTestIndexes:
testIndexes = list(set(range(len(target))) - set(trainIndexes))
# shuffle
if shuffle:
trainIndexes = random.shuffle(trainIndexes)
if testIndexes is not None:
testIndexes = random.shuffle(testIndexes)
# return indexes
return trainIndexes, testIndexes

This is the function I am using. You can adapt it and optimize it.
# Returns a Test dataset that contains an equal amounts of each class
# y should contain only two classes 0 and 1
def TrainSplitEqualBinary(X, y, samples_n): #samples_n per class
indicesClass1 = []
indicesClass2 = []
for i in range(0, len(y)):
if y[i] == 0 and len(indicesClass1) < samples_n:
indicesClass1.append(i)
elif y[i] == 1 and len(indicesClass2) < samples_n:
indicesClass2.append(i)
if len(indicesClass1) == samples_n and len(indicesClass2) == samples_n:
break
X_test_class1 = X[indicesClass1]
X_test_class2 = X[indicesClass2]
X_test = np.concatenate((X_test_class1,X_test_class2), axis=0)
#remove x_test from X
X_train = np.delete(X, indicesClass1 + indicesClass2, axis=0)
Y_test_class1 = y[indicesClass1]
Y_test_class2 = y[indicesClass2]
y_test = np.concatenate((Y_test_class1,Y_test_class2), axis=0)
#remove y_test from y
y_train = np.delete(y, indicesClass1 + indicesClass2, axis=0)
if (X_test.shape[0] != 2 * samples_n or y_test.shape[0] != 2 * samples_n):
raise Exception("Problem with split 1!")
if (X_train.shape[0] + X_test.shape[0] != X.shape[0] or y_train.shape[0] + y_test.shape[0] != y.shape[0]):
raise Exception("Problem with split 2!")
return X_train, X_test, y_train, y_test

Divide the data into multiple sets using ShuffleSplit then storing each in a set

I have loaded the CIFAR10 dataset but I want to divide it into multiple splits.
Here is how I downloaded the dataset
(x_train, y_train), (x_test, y_test) = cifar10.load_data()
Then I used ShuffleSplit to create a generator to split the data like this:
from sklearn.model_selection import ShuffleSplit
rs = ShuffleSplit(n_splits=3, test_size=0.1, random_state=0)
splits = rs.split(x_train)
I know I can iterate over the generated splits using:
for train_index, test_index in splits:
#train_index is a np array which hold the indies
print("TRAIN:", train_index, "TEST:", test_index)
Assuming I want to have at the end.
x_train1, y_train1, x_train2, y_train2, x_train3, y_train3
How can I divide the data based on the generated indices such that one training split which contain both the training and the testing indices?
I tried combining the indices into list or contacte the arrays but it did not work.

I was able to solve the problem by using a different approach the code is below:
partitions_train_x = []
partitions_train_y = []
partitions_test_x = []
partitions_test_y = []
x = np.arange(len(y_train))
np.random.shuffle(x)
indices = np.split(x, num_partitions)
for data_indices in zip(indices):
x = x_train[data_indices]
y = y_train[data_indices]
partions_train_x.append(x)
partitions_train_y.append(y)
x = np.arange(len(y_test))
indices = np.split(x, num_partitions)
for data_indices in zip(indices):
x = x_test[data_indices]
y = y_test[data_indices]
partitions_test_x.append(x)
partitions_test_y.append(y)
I know this may not be the best way to do, but it works.

A better way to do it :)
num_shreds = 10
shred_size = len(X_train)//num_shreds
X_train, y_train = shuffle(X_train, y_train)
shred_X = [X_train[i:i + shred_size] for i in range(0, shred_size* num_shreds, shred_size)]
shred_y = [y_train[i:i + shred_size] for i in range(0, shred_size* num_shreds, shred_size)]

Shuffle and split 2 numpy arrays so as to maintain their ordering with respect to each other

I have 2 numpy arrays X and Y, with shape X: [4750, 224, 224, 3] and Y: [4750,1].
X is the training dataset and Y is the correct output label for each entry.
I want to split the data into train and test so as to validate my machine learning model. Therefore, I want to split them randomly so that they both have the correct ordering after random split is applied on X and Y. ie- every row of X is correctly has its corresponding label unchanged after the split.
How can I achieve the above objective ?

This is how I would do it
def split(x, y, train_ratio=0.7):
x_size = x.shape[0]
train_size = int(x_size * train_ratio)
test_size = x_size - train_size
train_indices = np.random.choice(x_size, size=train_size, replace=False)
mask = np.zeros(x_size, dtype=bool)
mask[train_indices] = True
x_train, y_train = x[mask], y[mask]
x_test, y_test = x[~mask], y[~mask]
return (x_train, y_train), (x_test, y_test)
I simply choose the required number of indices I need (randomly) for my train set, remaining will be for the test set.
Then use a mask to select the train and test samples.

You can also use the scikit-learn train_test_split to split your data using just 2 lines of code :
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.33)

sklearn.model_selection.train_test_split is a good choice!
But to craft one of your own
import numpy as np
def my_train_test_split(X, Y, train_ratio=0.8):
"""return X_train, Y_train, X_test, Y_test"""
n = X.shape[0]
split = int(n * train_ratio)
index = np.arange(n)
np.random.shuffle(index)
return X[index[:split]], Y[index[:split]], X[index[split:]], Y[index[split:]]

Create train and test variables from loaded arff file

I want perform multilabel classification. A have a dataset in arff format which I load. However I don't now how convert import data to X and y vectors in order to apply sklearn/train_test_split.
How can I get X and y?
data, meta = scipy.io.arff.loadarff('../yeast-train.arff')
df = pd.DataFrame(data)
#Get X, y
X, y = ??? <---
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

Ok. Its a multilabel data in which features are in the columns Att1, Att2, Att3.... Att20 and targets are in the columns Class1, Class2, .... Class14.
So you need to use those columns for getting the X and y. Do it like this:
# Fill the .... with all other column names
feature_cols = ['Att1', 'Att2', 'Att3', 'Att4', 'Att5' .... 'Att20']
target_cols = ['Class1', 'Class2', 'Class3', 'Class4', .... 'Class14']
X, y = df[feature_cols], df[target_cols]

numpy: How can I select specific indexes in an np array for k-fold cross validation?

I have a training data set in matrix form of dimensions 5000 x 3027 (CIFAR-10 data set). Using array_split in numpy, I partitioned it into 5 different parts, and I want to select just one of the parts as the cross validation fold. However my problem comes when I use something like
XTrain[[Indexes]] where indexes is an array like [0,1,2,3], because doing this gives me a 3D tensor of dimensions 4 x 1000 x 3027, and not a matrix. How do I collapse the "4 x 1000" into 4000 rows, to get a matrix of 4000 x 3027?
for fold in range(len(X_train_folds)):
indexes = np.delete(np.arange(len(X_train_folds)), fold)
XTrain = X_train_folds[indexes]
X_cv = X_train_folds[fold]
yTrain = y_train_folds[indexes]
y_cv = y_train_folds[fold]
classifier.train(XTrain, yTrain)
dists = classifier.compute_distances_no_loops(X_cv)
y_test_pred = classifier.predict_labels(dists, k)
num_correct = np.sum(y_test_pred == y_test)
accuracy = float(num_correct/num_test)
k_to_accuracy[k] = accuracy

Perhaps you can try this instead (new to numpy so if I am doing something inefficient/wrong, would be happy to be corrected)
X_train_folds = np.array_split(X_train, num_folds)
y_train_folds = np.array_split(y_train, num_folds)
k_to_accuracies = {}
for k in k_choices:
k_to_accuracies[k] = []
for i in range(num_folds):
training_data, test_data = np.concatenate(X_train_folds[:i] + X_train_folds[i+1:]), X_train_folds[i]
training_labels, test_labels = np.concatenate(y_train_folds[:i] + y_train_folds[i+1:]), y_train_folds[i]
classifier.train(training_data, training_labels)
predicted_labels = classifier.predict(test_data, k)
k_to_accuracies[k].append(np.sum(predicted_labels == test_labels)/len(test_labels))

I would suggest using scikit-learn package. It already comes with plenty of common machine learning tools, such as K-fold cross-validation generator:
>>> from sklearn.cross_validation import KFold
>>> X = # your data [samples x features]
>>> y = # gt labels
>>> kf = KFold(X.shape[0], n_folds=5)
And then, iterate through kf:
>>> for train_index, test_index in kf:
X_train, X_test = X[train_index], X[test_index]
y_train, y_test = y[train_index], y[test_index]
# do something
The above loop will be executed n_folds times, each time with different training and testing indexes.