We Keep Coding

XGBoost XGBRegressor predict with different dimensions than fit

I am using the xgboost XGBRegressor to train on a data of 20 input dimensions:
model = xgb.XGBRegressor(objective='reg:squarederror', n_estimators=20)
model.fit(trainX, trainy, verbose=False)
trainX is 2000 x 19, and trainy is 2000 x 1.
In another word, I am using the 19 dimensions of trainX to predict the 20th dimension (the one dimension of trainy) as the training.
When I am making a prediction:
yhat = model.predict(x_input)
x_input has to be 19 dimensions.
I am wondering if there is a way to keep using the 19 dimensions to train prediction the 20th dimension. But during the prediction, x_input has only 4 dimensions to predict the 20th dimension. It is kinda of a transfer learning to different input dimension.
Does xgboost supports such a feature? I tried just to fill x_input's other dimensions to None, but that yields to terrible prediction results.

Fundamentally, you're training your model with a dense dataset (19/19 feature values), and are now wondering if you're allowed to make predictions with a sparse dataset (4/19 feature values).
Does xgboost supports such a feature?
Yes, it is technically possible with XGBoost, because XGBoost will treat the absent 15/19 feature values as missing. It will not be possible with some other ML framework (such as Scikit-Learn) that do not work with sparse input by default.
Alternatively, you can make your XGBoost model explicitly "missing-value-proof" by assembling a pipeline which contains feature imputation step(s).
I tried just to fill x_input's other dimensions to None, but that yields to terrible prediction results.
You should represent missing values as float("NaN") (not as None).

If I understand your question correctly, you are trying to train a model with 19 features, but then feed it only 1 feature to make a prediction.
That's not going to be possible. When you train a model, you are assuming that your data points are drawn from a probability distribution P(X,Y), where Y is your label and X is your features. If you try to change the dimensionality of X, it'll no longer belong to that distribution (at least intuitively, I am not a mathematician so, I cannot come up with a proof for this).
For instance, let's assume your data lies on a 3D cube. That means that you need three coordinate axes to represent a point on it. You cannot place a point using 2 dimensions without assuming the value of the remaining dimension.
You can assume the values of the features you try to drop, but they may not represent the data you originally trained on.

Understand shap values for binary classification

I have trained my imbalanced dataset (binary classification) using CatboostClassifer. Now, I am trying to interpret the model using the SHAP library. Below is the code to fit the model and calculate shap values:
weights = y.value_counts()[0] / y.value_counts()[1]
catboost_clf = CatBoostClassifier(loss_function='Logloss', iterations=100, verbose=True, \
l2_leaf_reg=6, scale_pos_weight=weights,eval_metric="MCC")
catboost_clf.fit(X, y)
trainx_preds = catboost_clf.predict(X_test)
explainer = shap.TreeExplainer(catboost_clf)
shap_values = explainer.shap_values(Pool(X,y))
#Class 0 samples 1625125
#Class 1 samples 122235
The size of shap values is (1747360, 13) i.e. (number of instances, number of features). I was expecting the shap values to be a 3d array i.e. (number of classes,number of instances, number of features). Shap values for each of the positive and negative class. How do I achieve that? How do I extract class wise shapley values to better understanding of the model.
Also, explainer.expected_value shows one base value instead of two.
Is there anything missing or incorrect in the code?
Thanks in advance!

Adding 'Multicalss' to the loss_function solved the problem. Referred to the documentation: Catboost
model = CatBoostClassifier(loss_function = 'MultiClass')

Is it possible to extract features from half of a UNet?

I have a 3D UNet that I trained for a couple thousand epochs and now I want to do some clustering on a similar dataset. However first I want to breakdown the volumes to a feature set array and perform the clustering on the features rather than the volumetric array. I would like to have the output shape from conv3d_9 (Conv3D)
Is it possible to take the features from only the down half of a UNet to extract those features?

Supposing that you already trained your model, you can use the Functional API to achieve this.
For example,
from tensorflow.keras.models import Model
feature_extraction_model = Model(inputs= model.inputs, outputs=model.layers[-18].output)
features_prediction = feature_extraction_model(input_3d_image)
Note that -18 is the index of the conv3d_9.

You can get intermediate layer by index or name shown below
feature = model.get_layer('conv3d_9d')

Dividing the training/validation sets by spatial parcellation using KMeans: making sure each subset of training/validation includes all labels

I am hoping to get some help here. I want to use Kmeans clustering to divide the training set for the supervised learning in the next stage, which is called spatial parcellation in the literature.
I have extracted features (X_train) from 3D image data (within a dilated mask), which includes the x, y, and z (positions of voxels), and mask includes two labels (y_train={0 : background,1: obj1}). It means each voxel can be background or object with the mask
I used Kmeans clustering in scikit-learn, and clustered the training dataset's voxel position ([x,y,z]) into 80 different clusters.
Problem the problem is that once I have divided the training set based on 80 clusters,
from sklearn.cluster import KMeans
pos_ind=[0,1,2]
kmeans_model= KMeans(n_clusters=80, random_state=rng).fit(X_TRAIN[:,pos_ind])
later I load the kmeans_model and assign the validation set to the clusters:
Dval_clusters = kmeans_model.predict(X_DVAL[:, pos_ind])
and then I find the row indices of validation set that it falls into cidx th cluster
cluster_idx = np.unique(kmeans_model.labels_)
Dval_rows = np.where(Dval_clusters==cluster_idx[cidx])[0] # find the rows of X_dval that belongs to cidx th cluster
X_dval = X_Dval[Dval_rows]
y_dval = y_Dval[Dval_rows]
Once I am fitting the validation set on the trained model, the trained model may have both labels 0 and 1, however, it is showing an error during the fitting the validation set, which means during assigning the validation samples to a cluster, it has only picked up voxels either with background or obj voxels (i.e., all samples in one validation set of a cluster has only 1 class label).
ValueError: could not broadcast input array from shape (30527,1) into shape (30527,2)
Question I know that clustering does not use labels, but is it possible to enforce the clustering that during the clustering, both labels are sampled? Or is there any trick for doing so? because background is just a class label and we need to to separate the object class (class:1) from the background (class:0)
I would really appreciate if you leave your expert opinion here.

Compute cross entropy loss for classification in pytorch

I am trying to build two neural network for classification. One for Binary and the second is for multi-class classification. I am trying to use the torch.nn.CrossEntropyLoss() as a loss function, but I try to train my first neural network I get the following error:
multi-target not supported at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THNN/generic/ClassNLLCriterion.c:22
From my analysis, I found that the my dataset has two problems that caused the error.
My data set is one hot encoded. I used one hot encoding to pre processes my dataset. The first target Y_binary variable has the shape of torch.Size([125973, 1]) full of 0s and 1 indicating classes 'No' and 'Yes'.
My data has the wrong dimensions? I found that I can't use a simple vector with the cross entropy loss function. Some people used the following code to reshape their target vector before feeding to the loss function.
out = out.permute(0, 2, 3, 1).contiguous().view(-1, class_number)
But I didn't really understand the reasoning behind this code. But it seems for my that I need to keep track of the following variables: Class_Number, Batch_size, Dimension_Output. For my code here are the dimensions
X_train.shape: (125973, 122)
Y_train2.shape: (125973, 1)
batch_size = 64
K = len(set(Y_train2)) # Binary classification For multi class classification use K = len(set(Y_train5))
Should the target value be one hot encoded? If not, how I can feed a nominal feature to the loss function?
If I use reshape the output, can you help me do this for my code ?
I am trying to use this loss function for both my neural networks.
Thank you in advance,

The error is due to the usage of torch.nn.CrossEntropyLoss() which can be used if you want to predict 1 class out of N classes. For multiclass classification, you should use torch.nn.BCEWithLogitsLoss() which combines a Sigmoid layer and the BCELoss in one single class.
In case of multi-class, and if you use Sigmoid + BCELoss, then you need the target to be one-hot encoding, i.e. something like this per sample: [0 1 0 0 0 1 0 0 1 0], where 1 will be at the locations of classes present.