We Keep Coding

How can i sort an array based on the mean of each column in python?

Input:
array([[ 1. , 5. , 1. ],
[ 10. , 7. , 1.5],
[ 6.9, 5. , 1. ],
[ 19. , 9. , 100. ],
[ 11. , 11. , 11. ]])
Expected Output:
array([[ 19. , 9. , 100. ],
[ 11. , 11. , 11. ],
[ 10. , 7. , 1.5],
[ 6.9, 5. , 1. ],
[ 1. , 5. , 1. ]])
i tried doing the below:
for i in M:
ls = i.mean()
x = np.append(i,ls)
print(x) #found the mean
After this i am unable to arrange each column based on the mean value in each row. All i can do
is to arrange each row in descending order but that is not what i wanted.

You can do this:
In [405]: row_idxs = np.argsort(np.mean(a * -1, axis=1))
In [406]: a[row_idxs, :]
Out[406]:
array([[ 19. , 9. , 100. ],
[ 11. , 11. , 11. ],
[ 10. , 7. , 1.5],
[ 6.9, 5. , 1. ],
[ 1. , 5. , 1. ]])
Using argsort will sort the indices. Multiplying by -1 allows you to get descending order.

Numpy sorting a matrix by column by cycle

I have this numpy array:
array(
[[ 1. , 9. , 565.98653513],
[ 1. , 1. , 973.18466261],
[ 1. , 25. , 803.17747373],
[ 2. , 9. , 82.56336897],
[ 2. , 1. , 104.69517373],
[ 2. , 25. , 627.01127514],
[ 3. , 21. , 334.07622382],
[ 3. , 34. , 921.37623107],
[ 3. , 20. , 342.08177942],
... ... ... ...
[ 10. , 7. , 424.29338026],
[ 10. , 0. , 232.71475407],
[ 10. , 1. , 330.44846202]])
But I want to sort the matrix by the first column in a cycle: 1, 2, 3, ...,10. It should look like this:
array(
[[ 1. , 9. , 565.98653513],
[ 2. , 9. , 82.56336897],
[ 3. , 21. , 334.07622382],
... ... ... ...
[ 10. , 7. , 424.29338026],
[ 1. , 1. , 973.18466261],
[ 2. , 1. , 104.69517373],
[ 3. , 34. , 921.37623107],
... ... ... ...
[ 10. , 0. , 232.71475407],
[ 1. , 25. , 803.17747373],
[ 2. , 25. , 627.01127514],
[ 3. , 20. , 342.08177942],
... ... ... ...
[ 10. , 1. , 330.44846202]])
How can I do this?
I was thinking of converting it to a dataframe (i.e. pandas) for more sorting options, then covert back to an array...but I don't see a straight forward function to do this.
I appreciate any help or ideas.

Assuming your array is named x:
y = [x[i::10] for i in range(int(len(x)/10))]
y = np.array(y)
y.reshape(x.shape)
print(y)
The x[i:j:k] notation means x from i to j with step k. So x[i::10] mean x from i to the end with step 10.
See more here.

Sort 2D NumPy array by one of the columns

I though this would be super easy but I am struggling a little. I have a data structure as follows
array([[ 5. , 3.40166205],
[ 10. , 2.72778882],
[ 15. , 2.31881804],
[ 20. , 2.50643777],
[ 1. , 3.94076063],
[ 2. , 3.80598599],
[ 3. , 3.67121134],
[ 6. , 3.2668874 ],
[ 7. , 3.13211276],
[ 8. , 2.99733811],
[ 9. , 2.86256347],
[ 11. , 2.64599467],
[ 12. , 2.56420051],
[ 13. , 2.48240635],
[ 14. , 2.4006122 ],
[ 16. , 1.8280531 ],
[ 17. , 1.74625894],
[ 18. , 1.66446479],
[ 19. , 1.58267063],
[ 20. , 1.50087647]])
And I want to sort it ONLY on the first column ... so it is ordered as follows:
array([[1. , 3.9],
[2. , 3.8],
... ,
[20. , 1.5]])
np.sort doesn't seem to work as it moves array to a flat structure. I've also used itemgetter
from operator import itemgetter
sorted(data, key=itemgetter(1))
But this doesn't give me the output I'm looking for.
Help appreciated!

This is a common numpy idiom. You can use argsort (on the first column) + numpy indexing here -
x[x[:, 0].argsort()]
array([[ 1. , 3.94076063],
[ 2. , 3.80598599],
[ 3. , 3.67121134],
[ 5. , 3.40166205],
[ 6. , 3.2668874 ],
[ 7. , 3.13211276],
[ 8. , 2.99733811],
[ 9. , 2.86256347],
[ 10. , 2.72778882],
[ 11. , 2.64599467],
[ 12. , 2.56420051],
[ 13. , 2.48240635],
[ 14. , 2.4006122 ],
[ 15. , 2.31881804],
[ 16. , 1.8280531 ],
[ 17. , 1.74625894],
[ 18. , 1.66446479],
[ 19. , 1.58267063],
[ 20. , 2.50643777],
[ 20. , 1.50087647]])

Python: Generate a new array with more columns based on another array

I have this array:
I need create a new array like this:
I guess I need use a conditional, but I don't know how create an array with 7 columns, based on values of a 5 columns array.
If anyone could help me, I thank!

I'm going to assume you want to convert your last column into one hot concodings and then concat it to your original array. You can initialise an array of zeros, and then set the appropriate indices to 1. Finally concat the OHE array to your original.
MCVE:
print(arr)
array([[ -9.95, 15.27, 9.08, 1. ],
[ -6.81, 11.87, 8.38, 2. ],
[ -3.02, 11.08, -8.5 , 1. ],
[ -5.73, -2.29, -2.09, 2. ],
[ -7.01, -0.9 , 12.91, 2. ],
[-11.64, -10.3 , 2.09, 2. ],
[ 17.85, 13.7 , 2.14, 0. ],
[ 6.34, -9.49, -8.05, 2. ],
[ 18.62, -9.43, -1.02, 1. ],
[ -2.15, -23.65, -13.03, 1. ]])
c = arr[:, -1].astype(int)
ohe = np.zeros((c.shape[0], c.max() + 1))
ohe[np.arange(c.shape[0]), c] = 1
arr = np.hstack((arr[:, :-1], ohe))
print(arr)
array([[ -9.95, 15.27, 9.08, 0. , 1. , 0. ],
[ -6.81, 11.87, 8.38, 0. , 0. , 1. ],
[ -3.02, 11.08, -8.5 , 0. , 1. , 0. ],
[ -5.73, -2.29, -2.09, 0. , 0. , 1. ],
[ -7.01, -0.9 , 12.91, 0. , 0. , 1. ],
[-11.64, -10.3 , 2.09, 0. , 0. , 1. ],
[ 17.85, 13.7 , 2.14, 1. , 0. , 0. ],
[ 6.34, -9.49, -8.05, 0. , 0. , 1. ],
[ 18.62, -9.43, -1.02, 0. , 1. , 0. ],
[ -2.15, -23.65, -13.03, 0. , 1. , 0. ]])

One-line version of #COLDSPEED using the np.eye trick:
np.hstack([arr[:,:-1], np.eye(arr[:,-1].astype(int).max() + 1)[arr[:,-1].astype(int)]])

Having trouble determining a 2D feature matrix structure to feed into machine learning algorithm

I am training an emotion recognition system that detects emotions through facial movement as a result, I have formed a 4 dimensional matrix that I am trying to reduce to 2 dimensions.
Features that makes up the 4D matrix:
Number of videos (and each video will be assigned emotion label)
Number of frames per video
Direction of the facial landmarks per frame
Speed of the facial landmarks per frame
The important features that I am trying to train with:
The left side is the speed (hypotenuse between same facial landmark each frame)
The right side is direction (arctan of the x and y values of the same facial landmark each frame)
The 4D matrix that I am stuck with and trying to reduce to 2D
>> main.shape
(60, 17, 68, 2)
# 60 videos, 17 frames per video, 68 facial landmarks, 2 features (direction and speed)
>> main
array([[[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ]],
[[ 1. , 1. ],
[ 1.41421356, 0.78539816],
[ 1.41421356, 0.78539816],
...,
[ 3. , 1. ],
[ 3. , 1. ],
[ 3. , 1. ]],
[[ 0. , 0. ],
[ -1.41421356, 0.78539816],
[ -1.41421356, 0.78539816],
...,
[ 2. , 1. ],
[ 3. , 1. ],
[ 3. , 1. ]],
...,
[[ 1. , 1. ],
[ 1.41421356, -0.78539816],
[ 1.41421356, -0.78539816],
...,
[ -1.41421356, 0.78539816],
[ 1. , 1. ],
[ -1.41421356, 0.78539816]],
[[ 2.23606798, -0.46364761],
[ 2.82842712, -0.78539816],
[ 2.23606798, -0.46364761],
...,
[ 1. , 0. ],
[ 0. , 0. ],
[ 1. , 1. ]],
[[ -1.41421356, -0.78539816],
[ -2.23606798, -0.46364761],
[ -2.23606798, -0.46364761],
...,
[ 1.41421356, -0.78539816],
[ 1.41421356, -0.78539816],
[ 2.23606798, -1.10714872]]],
[[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ]],
[[ 2. , 1. ],
[ 2.23606798, -1.10714872],
[ 1.41421356, -0.78539816],
...,
[ -2. , -0. ],
[ -1. , -0. ],
[ -1.41421356, -0.78539816]],
[[ 2. , 1. ],
[ -2.23606798, 1.10714872],
[ -1.41421356, 0.78539816],
...,
[ 1. , 1. ],
[ -1. , -0. ],
[ -1. , -0. ]],
...,
[[ -2. , -0. ],
[ -3. , -0. ],
[ -4.12310563, -0.24497866],
...,
[ 0. , 0. ],
[ -1. , -0. ],
[ -2.23606798, 1.10714872]],
[[ -2.23606798, 1.10714872],
[ -1.41421356, 0.78539816],
[ -2.23606798, 1.10714872],
...,
[ -2.23606798, 0.46364761],
[ -1.41421356, 0.78539816],
[ -1.41421356, 0.78539816]],
[[ 2. , 1. ],
[ 1.41421356, 0.78539816],
[ 2.82842712, 0.78539816],
...,
[ 1. , 1. ],
[ 1. , 1. ],
[ -2.23606798, -1.10714872]]],
[[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ]],
[[ 1. , 1. ],
[ 0. , 0. ],
[ 1. , 1. ],
...,
[ -3. , -0. ],
[ -2. , -0. ],
[ 0. , 0. ]],
[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ 1.41421356, 0.78539816],
[ 1. , 0. ],
[ 0. , 0. ]],
...,
[[ 1. , 0. ],
[ 1. , 1. ],
[ 0. , 0. ],
...,
[ 2. , 1. ],
[ 3. , 1. ],
[ 3. , 1. ]],
[[ -7.28010989, 1.29249667],
[ -7.28010989, 1.29249667],
[ -8.54400375, 1.21202566],
...,
[-22.02271555, 1.52537305],
[ 22.09072203, -1.48013644],
[ 22.36067977, -1.39094283]],
[[ 1. , 0. ],
[ 1.41421356, -0.78539816],
[ 1. , 0. ],
...,
[ -1.41421356, -0.78539816],
[ 1. , 1. ],
[ 1.41421356, 0.78539816]]],
...,
[[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ]],
[[ 5.38516481, 0.38050638],
[ 5.09901951, 0.19739556],
[ 4.47213595, -0.46364761],
...,
[ -1.41421356, 0.78539816],
[ -2.82842712, 0.78539816],
[ -5. , 0.64350111]],
[[ -6.32455532, 0.32175055],
[ -6.08276253, -0.16514868],
[ -5.65685425, -0.78539816],
...,
[ 3.60555128, 0.98279372],
[ 5. , 0.92729522],
[ 5.65685425, 0.78539816]],
...,
[[ -3.16227766, -0.32175055],
[ -3.60555128, -0.98279372],
[ 5. , 1. ],
...,
[ 12.08304597, 1.14416883],
[ 13.15294644, 1.418147 ],
[ 14.31782106, 1.35970299]],
[[ 3.60555128, -0.5880026 ],
[ 4.47213595, -1.10714872],
[ 6. , 1. ],
...,
[-20.39607805, 1.37340077],
[-21.02379604, 1.52321322],
[-22.09072203, 1.48013644]],
[[ 1. , 1. ],
[ -1.41421356, 0.78539816],
[ 1. , 1. ],
...,
[ 4.12310563, 1.32581766],
[ 4. , 1. ],
[ 4.12310563, 1.32581766]]],
[[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ]],
[[ 0. , 0. ],
[ 1. , 1. ],
[ -2.23606798, 1.10714872],
...,
[ -3.16227766, 0.32175055],
[ 1. , 1. ],
[ 1.41421356, -0.78539816]],
[[ 1. , 1. ],
[ 1. , 1. ],
[ 1. , 1. ],
...,
[ 3. , 1. ],
[ 2. , 1. ],
[ -1.41421356, 0.78539816]],
...,
[[ 5.38516481, -1.19028995],
[ 4.47213595, -1.10714872],
[ 4.12310563, -1.32581766],
...,
[ 2.23606798, -0.46364761],
[ 1. , 1. ],
[ -1. , -0. ]],
[[ -5.38516481, 1.19028995],
[ -4.12310563, 1.32581766],
[ -3.16227766, 1.24904577],
...,
[ 0. , 0. ],
[ 1. , 0. ],
[ 1.41421356, -0.78539816]],
[[ 8.06225775, 1.44644133],
[ -7.07106781, -1.42889927],
[ 6. , 1. ],
...,
[ -3.16227766, -0.32175055],
[ -3.16227766, -0.32175055],
[ -3.16227766, -0.32175055]]],
[[[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ],
...,
[ 0. , 0. ],
[ 0. , 0. ],
[ 0. , 0. ]],
[[ -2.23606798, 0.46364761],
[ -1.41421356, 0.78539816],
[ -2.23606798, 0.46364761],
...,
[ 1. , 0. ],
[ 1. , 0. ],
[ 1. , 1. ]],
[[ -2.23606798, -0.46364761],
[ -1.41421356, -0.78539816],
[ 2. , 1. ],
...,
[ 0. , 0. ],
[ 1. , 0. ],
[ 1. , 0. ]],
...,
[[ 1. , 0. ],
[ 1. , 1. ],
[ -2.23606798, -1.10714872],
...,
[ 19.02629759, 1.51821327],
[ 19. , 1. ],
[-19.10497317, -1.46591939]],
[[ 3.60555128, 0.98279372],
[ 3.60555128, 0.5880026 ],
[ 5. , 0.64350111],
...,
[ 7.28010989, -1.29249667],
[ 7.61577311, -1.16590454],
[ 8.06225775, -1.05165021]],
[[ -7.28010989, 1.29249667],
[ -5. , 0.92729522],
[ -5.83095189, 0.5404195 ],
...,
[ 20.09975124, 1.47112767],
[ 21.02379604, 1.52321322],
[-20.22374842, -1.42190638]]]])
The direction and speed features are quite valuable (the most important features) as it represents the movement of each facial landmark per frame and I am trying to get the machine learning algorithm to train base on that
I tried to reshape three of the dimensions into one long vector (just mushed speed, direction, and frame all together) and finally formed a 2D matrix, I fed it into sklearn SVM function and it produced a rather low accuracy. I expected this as I figured there is no way the ml algorithm would recognize that the difference between the features in the giant single matrix and assume that everything in the vector is the same features.
The 2D matrix I was forced to make to feed in sklearn SVM by forcing speed, direction, and video per frame all into one vector, and got a low accuracy with:
>> main
array([[ 0. , 0. , 0. , ..., -0.78539816,
2.23606798, -1.10714872],
[ 0. , 0. , 0. , ..., 1. ,
-2.23606798, -1.10714872],
[ 0. , 0. , 0. , ..., 1. ,
1.41421356, 0.78539816],
...,
[ 0. , 0. , 0. , ..., 1. ,
4.12310563, 1.32581766],
[ 0. , 0. , 0. , ..., -0.32175055,
-3.16227766, -0.32175055],
[ 0. , 0. , 0. , ..., 1.52321322,
-20.22374842, -1.42190638]])
>> main.shape
(60, 2312)
I want to preserve the speed and direction features, but have to represent them in a 2D matrix that takes into account the frames in the video.
The emotion label will be attached to each of the 17 frames in each video. (so basically, the 17 frame video will be labeled as an emotion)
Is there any smart way in reshaping and reducing the 4D matrix that would accomplish this?

So, the way that you've framed the question you will absolutely see poor accuracy and there's very little you can do to change it. Assigning a single emotion to a video (depending on your corpus), is generally inaccurate enough that any machine learning algorithm will have trouble learning the signal you're trying to pull out.
Additionally, you've framed the problem as a time-series problem, which is going to make your life a headache, especially if you're using off-the-shelf sklearn algorithms, which are very poorly suited for this kind of task.
If at all possible, you should instead frame your problem as a computer vision problem. You should attempt to predict on each individual frame, what the emotion content is. If you don't have a dataset with that level of granularity, you just aren't going to see great accuracy.
It's a little bit of a departure from the way in which you asked the question, but the way in which you've asked the question it's non-tractable. Here is, instead, the way that you should approach the problem:
Label individual frames with emotional content
Train an image-based algorithm to categorize those tagged frames
Convolutional neural networks will likely give you the best performance for any image-based problem where you have a decently-sized dataset
If that is not an option, you need to develop a 1d feature representation of the image. I would personally suggest using indico's image features API. Once you have this representation a typical algorithm like an SVM will work great.
If accuracy is not quite to your liking, but is getting close I would recommend using a pre-processing/data-augmentation pipeline like the one details here. Granted, that example is for plankton identification, the basic approach is identical
If the accuracy still isn't up to snuff, and you need to predict on the entire video you will then want to aggregate your results to give accurate results over the entire video
One method is to train a convolutional neural network on a graph of the predictions you've made over the video. This is kind of weird, but might work pretty well
A good approach would be to use a bayesian method, assuming each prediction has a certain level of confidence, and combining the prediction distributions over the video.
The best approach is to treat this as an ensemble learning problem. Luckily, ensemble learning is a very well-studied and understood problem. You can find details of how to combine multiple predictions in this format here.
I hope this has been helpful! Let me know if you have any more questions.
Disclaimer: I am the CEO of indico, so I may be biased in recommending its use.