l want to split data into train and test and also a vector that contains names (it serves me as an index and reference).
name_images has a shape of (2440,)
My data are :
data has a shape of (2440, 3072)
labels has a shape of (2440,)
from sklearn.cross_validation import train_test_split
x_train, x_test, y_train, y_test= train_test_split(data, labels, test_size=0.3)
but l want also to split my name_images into name_images_train and name_images_test with respect to the split of data and labels
l tried
x_train, x_test, y_train, y_test,name_images_train,name_images_test= train_test_split(data, labels,name_images, test_size=0.3)
it doesn't preserve the order
Any suggestions
thank you
EDIT1:
x_train, x_test, y_train, y_test= train_test_split(data, labels,test_size=0.3, random_state=42)
name_images_train, name_images_test=train_test_split(name_images,
test_size=0.3,
random_state=42)
EDIT1 don't preserve the order
There are multiple ways to accomplish this.
The most straight forward is to use random_state parameter of train_test_split. As the documentation states:
random_state : int or RandomState :-
Pseudo-random number generator state used for random sampling.
When you fix the random_state, the indices which are generated for splitting the arrays into train and test are exact same each time.
So change your code to:
x_train, x_test,
y_train, y_test,
name_images_train, name_images_test=train_test_split(data, labels, name_images,
test_size=0.3,
random_state=42)
For more understanding on random_state, see my answer here:
https://stackoverflow.com/a/42197534/3374996
In my case, I realize that my input arrays were not in proper order in the first place. So for future Googlers--you may want to double-check if (data, labels) are in the same order or not.
Related
I'm using train_test_split to split image data for a convolutional neural network in Python:
x_train, x_test, y_train, y_test = train_test_split(X, Y)
For each image in X, how can I figure out whether it was sent to the x_train or x_test set? Since all the data in the x_train or x_test datasets are in tensor form and randomized, I'm not sure how to relate a given instance in x_train/x_test back to its original place in X. My confusion matrix is printing inconsistent information, so I'm trying to figure out if the way the data is split being training and testing is the reason.
Edit 1: Folder Structure
All the images are in one array (X = np.array(X_images)) which I derived from collecting image from folders such that:
Data
Class_1
Class_2
...
Class_n
I then used: Y = np_utils.to_categorical(labels, num_classes) to get the Y values
If you are able to re-run the experiment, try generating indicates instead of raw arrays. Then use indicates to extract train and dev sets.
# this is slightly modified example from the sklearn documentation:
import numpy as np
from sklearn.model_selection import KFold
X = np.array([["a", "b"], ["c", "d"], ["e", "f"], ["g", "h"]])
y = np.array(["a1", "c1", "e1", "g1"])
kf = KFold(n_splits=2)
for train_index, test_index in kf.split(X):
print("indicates", "TRAIN:", train_index, "TEST:", test_index)
X_train, X_test = X[train_index], X[test_index]
y_train, y_test = y[train_index], y[test_index]
train_test_split takes as arguments an arbitrary number of arrays/vectors, so you could just pass an additional list/array containing some identifier to that call, e.g. X_train, X_test, y_train, y_test, id_train, id_test = train_test_split(X, y, ids), where ids is some list/array containing the identifiers corresponding to each element in X/y. Then, the data point at index i in X_train/y_train will correspond to the identifier at id_train[i], and so on for the "test" data. If you don't have a row-identifier column handy, you could just use the index of X, e.g. ids = list(range(X.shape[0])).
The following solved my problem. I created a numpy array from the original image data:
indices = np.arange(X.shape[0])
Fed this into the train_test_split function call and added two more that created an index corresponding to the image's respective X index:
x_train, x_test, y_train, y_test, x_train_ind, x_test_ind = train_test_split(X, Y, indices, test_size=0.2, random_state=2)
After getting the index of an image in the x_train dataset, we can plug that into x_train_ind to get the index in the original X dataset
I have data about Parkinson patients stored in the dataframe X and whether a patient has Parkinson indicated by y (0 or 1). This is retrieved by:
X=pd.read_csv('parkinsons.data',index_col=0)
y=X['status']
X=X.drop(['status'],axis=1)
Then, I create training and test samples:
X_train, y_train, X_test, y_test = train_test_split(X,y,test_size=0.3,random_state=7)
I want to use SVC on this training data:
svc=SVC()
svc.fit(X_train,y_train)
Then, I get the error:
ValueError: bad input shape (59, 22).
What did I do wrong and how can I get rid of this error?
You have problems with the definition of train_test_split Careful! train_test_split outputs the X part first followed by the Y part. You are actually naming y_train what is X_test. Change this and it should work:
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3,random_state=7)
Either use this
X_train, y_train, X_test, y_test =train_test_split(X,y,test_size=0.3,random_state=7)
svc=SVC()
svc.fit(X_train,X_test)
Or this
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size=0.3,random_state=7)
svc=SVC()
svc.fit(X_train,y_train)
I prefer using the second one
Please help me. After splitting my data into
X_train, y_train, X_test, y_test = train_test_split(X,y)
then passing it to my linear regression model I.e
linereg = LinearRegression().fit(X_train, y_train)
It brings out an error saying array must be 2D not 1D array. How can I make it a 2D array.
first split the data correctly
X_train, x_test, Y_train,y_test=train_test_split(features,labels,train_size=0.7, test_size=0.3, random_state=2)
try reshaping the x_train and x_test using reshape method.
x_test=x_test.reshape(-1,1)
x_train=x_train.reshape(-1,1)
Basically i wanted to split my dataset into training,testing and validation set. I therefore have used train_test_split function twice. I have a dataset of around 10-Million rows.
On the first split i have split training and testing dataset into 70-Million training and 30-Million testing. Now to get validation set i am bit confused whether to use splitted testing data or training data as an input parameter of train-test-split in order to get validation set. Give some advise. TIA
X = features
y = target
# dividing X, y into train and test and validation data 70% training dataset with 15% testing and 15% validation set
from sklearn.model_selection import train_test_split
#features and label splitted into 70-30
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.3, random_state = 0)
#furthermore test data is splitted into test and validation set 15-15
x_test, x_val, y_test, y_val = train_test_split(X_test, y_test, test_size=0.5)
Don't make a testing set too small. A 20% testing dataset is fine. It would be better, if you splitted you training dataset into training and validation (80%/20% is a fair split). Considering this, you shall change your code in this way:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
x_test, x_val, y_test, y_val = train_test_split(X_train, y_train, test_size=0.25)
This is a common practice to split a dataset like this.
I have a training set consisting of X and Y, The X is of shape (4000,32,1) and Y is of shape (4000,1).
I would like to create a training/validation set based on split. Here is what I have been trying to do
from sklearn.model_selection import StratifiedShuffleSplit
sss = StratifiedShuffleSplit(test_size=0.1, random_state=23)
for train_index, valid_index in sss.split(X, Y):
X_train, X_valid = X[train_index], X[valid_index]
y_train, y_valid = Y[train_index], Y[valid_index]
Running the program gives the following error message related to the above code segment
for train_index, valid_index in sss.split(X, Y):
ValueError: The least populated class in y has only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2.
I am not very clear about the above error message, what's the right way to create a training/validation split for the training set as above?
It's a little bit weird because I copy/pasted your code with sklearn's breast cancer dataset as follow
from sklearn.datasets import load_breast_cancer
cancer = load_breast_cancer()
X, Y = cancer.data, cancer.target
from sklearn.model_selection import StratifiedShuffleSplit
sss = StratifiedShuffleSplit(test_size=0.1, random_state=23)
for train_index, valid_index in sss.split(X, Y):
X_train, X_valid = X[train_index], X[valid_index]
y_train, y_valid = Y[train_index], Y[valid_index]
Here X.shape = (569, 30) and Y.shape = (569,) and I had no error, for example y_valid.shape = 57 or one tenth of 569.
I suggest you to reshape X into (4000,32) (and so Y into (4000)), because Python may see it as a list of ONE big element (I am using python 2-7 by the way).
To answer your question, you can alternatively use train_test_split
from sklearn.model_selection import train_test_split
which according to the help
Split arrays or matrices into random train and test subsets Quick utility that wraps input validation and
``next(ShuffleSplit().split(X, y))`
Basically a wrapper of what you wanted to do. You can then specify the training and the test sizes, the random_state, if you want to stratify your data or to shuffle it etc.
It's easy to use for example:
X_train, X_valid, y_train, y_valid = train_test_split(X,Y, test_size = 0.1, random_state=0)