Convert list of PNGImageFile to array of array - python

I have a dataset organized in this way: /dataset/train/class/images.png (the same for test) (and I have 2 classes, positive and negative).
I want to obtain x_train, y_train, x_test and y_test, so I am using this python script:
x_train = []
y_train = []
x_test = []
y_test = []
base_dir_train = 'Montgomery_real_splitted/TRAIN/'
base_dir_test = 'Montgomery_real_splitted/TEST/'
for f in sorted(os.listdir(base_dir_train)):
if os.path.isdir(base_dir_train+f):
print(f"{f} is a target class")
for i in sorted(os.listdir(base_dir_train+f)):
y_train.append(f)
im = Image.open(base_dir_train+f+'/'+i)
x_train.append(im)
for f in sorted(os.listdir(base_dir_test)):
if os.path.isdir(base_dir_test+f):
print(f"{f} is a target class")
for i in sorted(os.listdir(base_dir_test+f)):
y_test.append(f)
imt=Image.open(base_dir_test+f+'/'+i)
x_test.append(imt)
y_train = np.array(y_train)
y_test = np.array(y_test)
Basically I obtain what I want, for example x_train is this:
[<PIL.PngImagePlugin.PngImageFile image mode=L size=4892x4020 at 0x10A98B280>,
<PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x10A98B040>,
...
<PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x11BA5D940>,
<PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x11BA5D9A0>]
And y_train is:
array(['neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg',
...
'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos'],
dtype='<U3')
However, I want that x_train is in this format:
array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint8)
How can I convert it?
EDIT for #Pyaive Oleg If I do im = np.array(im) then the result is this, which is different from the one that I want. The same for the tensor
[array([[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
...,
[ 5, 6, 7, ..., 14, 9, 5],
[ 4, 5, 6, ..., 12, 8, 4],
[ 0, 1, 2, ..., 3, 2, 0]], dtype=uint8),
array([[ 1, 1, 1, ..., 8, 246, 0],
[ 1, 1, 1, ..., 0, 7, 11],
[ 1, 1, 1, ..., 0, 0, 6],
...,
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
...
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
array([[ 0, 0, 0, ..., 0, 255, 1],
[ 0, 0, 0, ..., 0, 3, 11],
[ 0, 0, 0, ..., 2, 0, 7],
...,
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
array([[ 1, 1, 1, ..., 19, 246, 0],
[ 1, 1, 1, ..., 0, 16, 0],
[ 1, 1, 1, ..., 2, 0, 12],
...,
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),


Before appending imt to x_train, do this:
imt = np.array(imt)
The following also can help:
from torchvision import transforms
imt = transforms.ToTensor()(imt)

Related

Multi-agent grid world Q-learning observation parameter

I am trying to train agents in a multi-agent grid world using q-learning. I am using the existing environment 'collection-game': https://reposhub.com/python/deep-learning/ArnaudFickinger-gym-multigrid.html#articleHeader2. My problem is the observation/state-parameter that the environment algorithm returns. I want to use this parameter to choose the best policy given the state the agent is currently in. An example of this is shown below. Here the state parameter returns an integer that can be used to index the q-table.
Example of using the state to pick policy
The environment that I am using however returns an array of arrays that shows the observations for all agents in the environment:
[array([[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[10, 2, 0, 0, 2, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 1, 0, 0, 0, 1]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0]]], dtype=uint8), array([[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 1, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 2, 0, 0, 2, 1]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 0],
[ 6, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0]]], dtype=uint8), array([[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 1]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]]], dtype=uint8)]
My question is: how to interpret this variable, and how to use it to pick the best policy for each agent.

How to add label to image?

Hi I used this line of code to see the format of images in mnist dataset:
print(mnsit.load_data())
The output is this:
(array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint8), array([5, 0, 4, ..., 5, 6, 8], dtype=uint8))
The main array contains an array at the end. This array is a set of labels that I give to input of a neural network.
Now I need to add an array of labels at the end of images to create a dataset like mnist dataset. Are there functions I can use to do this? I tried append and insert but it didn't work.

You could do something like that:
dataset = [image_array, label_array]
That creates an list with the two array as a elements.

Move axis in tensorflow

I have two tensors. The main tensor is as follows:
array([[[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217]],
[[ 450, 607, 493, 662],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 950, 1277, 1028, 1335],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]]], dtype=int32)
I want to move this tensor according to the following tensor:
array([0, 2, 5], dtype=int32)
The above tensor contains the axis we want the current axis to move to.
The final tensor should look like this:
array([[[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 450, 607, 493, 662],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 950, 1277, 1028, 1335],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]]], dtype=int32)

You can use the tensorflow scatter function tf.scatter_nd for achieving this.
Define your input tensor:
input = tf.constant([[[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217]],
[[ 450, 607, 493, 662],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 950, 1277, 1028, 1335],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]]])
Since we are interested in only the first 3 elements along zeroth dimension, let's slice it into a new tensor:
sliced_input = tf.slice(input, [0, 0, 0], [3, -1, -1])
Define your target indices:
indices = tf.constant([[0], [2], [5]])
Define shapes of your target output, here same as your input shape:
shape = tf.shape(input)
Now use the scatter function to get your output:
output = tf.scatter_nd(indices, sliced_input, shape)
output:
array([[[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217],
[ 298, 1217, 298, 1217]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 450, 607, 493, 662],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 950, 1277, 1028, 1335],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]],
[[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0],
[ 0, 0, 0, 0]]], dtype=int32)

multi label classification confusion matrix have wrong number of labels

i am feeding in y_test and y_pred to a confusion matrix. My data is for multi label classification so the row values are one hot encodings.
my data has 30 labels but after feeding into the confusion matrix, the output only has 11 rows and cols which is confusing me. I thought i should have a 30X30.
Their formats are numpy arrays. (y_test and y_pred are dataframes of which i convert to numpy arrays using dataframe.values)
y_test.shape
(8680, 30)
y_test
array([[1, 0, 0, ..., 0, 0, 0],
[1, 0, 0, ..., 0, 0, 0],
[1, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]])
y_pred.shape
(8680, 30)
y_pred
array([[1, 0, 0, ..., 0, 0, 0],
[1, 0, 0, ..., 0, 0, 0],
[1, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]])
I transform them to confusion matrix usable format:
y_test2 = y_test.argmax(axis=1)
y_pred2 = y_pred.argmax(axis=1)
conf_mat = confusion_matrix(y_test2, y_pred2)
here is what my confusion matrix look like:
conf_mat.shape
(11, 11)
conf_mat
array([[4246, 77, 13, 72, 81, 4, 6, 3, 0, 0, 4],
[ 106, 2010, 20, 23, 21, 0, 5, 2, 0, 0, 0],
[ 143, 41, 95, 32, 10, 3, 14, 1, 1, 1, 2],
[ 101, 1, 0, 351, 36, 0, 0, 0, 0, 0, 0],
[ 346, 23, 7, 10, 746, 5, 6, 4, 3, 3, 2],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
Why does my confusion matrix only have 11 X 11 shape? shouldn't it be 30X30?

I think you are not quit clear the definition of confusion_matrix
y_true = [2, 0, 2, 2, 0, 1]
y_pred = [0, 0, 2, 2, 0, 2]
confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
[0, 0, 1],
[1, 0, 2]])
Which in data frame is
pd.DataFrame(confusion_matrix(y_true, y_pred),columns=[0,1,2],index=[0,1,2])
Out[245]:
0 1 2
0 2 0 0
1 0 0 1
2 1 0 2
The column and index are the category of input.
You have (11,11), which means you only have 11 categories in your data

All this means is that some labels are unused.
y_test.any(axis=0)
y_pred.any(axis=0)
Should show that only 11 of the columns have any 1s in them.
Here's what it would look like if that was not the case:
from sklearn.metrics import confusion_matrix
y_test = np.zeros((8680, 30))
y_pred = np.zeros((8680, 30))
y_test[np.arange(8680), np.random.randint(0, 30, 8680)] = 1
y_pred[np.arange(8680), np.random.randint(0, 30, 8680)] = 1
y_test2 = y_test.argmax(axis=1)
y_pred2 = y_pred.argmax(axis=1)
confusion_matrix(y_test2, y_pred2).shape # (30, 30)

Concatenating or adding elements of an array

i have a numpy array of 27 elements,Im trying concatenate or add all the elements inside the array,but i cant come up with anything right,
I tried,
for index,value in enumerate(array):
np.concatenate(array[index],array[index])
but this throws
TypeError: only integer scalar arrays can be converted to a scalar index
I tried
array[1]+array[2]+array[3]
this works for me, but im not sure how to put this in a loop,
Any suggestions on this front would be really helpful
Thanks in advance.
EDIT:
array looks like this
array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,

You want to sum over one axis (the first, I think). Like this:
array.sum(axis=0)

Categories