We Keep Coding

Multi-agent grid world Q-learning observation parameter

I am trying to train agents in a multi-agent grid world using q-learning. I am using the existing environment 'collection-game': https://reposhub.com/python/deep-learning/ArnaudFickinger-gym-multigrid.html#articleHeader2. My problem is the observation/state-parameter that the environment algorithm returns. I want to use this parameter to choose the best policy given the state the agent is currently in. An example of this is shown below. Here the state parameter returns an integer that can be used to index the q-table.
Example of using the state to pick policy
The environment that I am using however returns an array of arrays that shows the observations for all agents in the environment:
[array([[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[10, 2, 0, 0, 2, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 1, 0, 0, 0, 1]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0]]], dtype=uint8), array([[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 1, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 2, 0, 0, 2, 1]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 0],
[ 6, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0]]], dtype=uint8), array([[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 1]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]]], dtype=uint8)]
My question is: how to interpret this variable, and how to use it to pick the best policy for each agent.

How to add label to image?

Hi I used this line of code to see the format of images in mnist dataset:
print(mnsit.load_data())
The output is this:
(array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint8), array([5, 0, 4, ..., 5, 6, 8], dtype=uint8))
The main array contains an array at the end. This array is a set of labels that I give to input of a neural network.
Now I need to add an array of labels at the end of images to create a dataset like mnist dataset. Are there functions I can use to do this? I tried append and insert but it didn't work.

You could do something like that:
dataset = [image_array, label_array]
That creates an list with the two array as a elements.

Convert list of PNGImageFile to array of array

I have a dataset organized in this way: /dataset/train/class/images.png (the same for test) (and I have 2 classes, positive and negative).
I want to obtain x_train, y_train, x_test and y_test, so I am using this python script:
x_train = []
y_train = []
x_test = []
y_test = []
base_dir_train = 'Montgomery_real_splitted/TRAIN/'
base_dir_test = 'Montgomery_real_splitted/TEST/'
for f in sorted(os.listdir(base_dir_train)):
if os.path.isdir(base_dir_train+f):
print(f"{f} is a target class")
for i in sorted(os.listdir(base_dir_train+f)):
y_train.append(f)
im = Image.open(base_dir_train+f+'/'+i)
x_train.append(im)
for f in sorted(os.listdir(base_dir_test)):
if os.path.isdir(base_dir_test+f):
print(f"{f} is a target class")
for i in sorted(os.listdir(base_dir_test+f)):
y_test.append(f)
imt=Image.open(base_dir_test+f+'/'+i)
x_test.append(imt)
y_train = np.array(y_train)
y_test = np.array(y_test)
Basically I obtain what I want, for example x_train is this:
[<PIL.PngImagePlugin.PngImageFile image mode=L size=4892x4020 at 0x10A98B280>,
<PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x10A98B040>,
...
<PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x11BA5D940>,
<PIL.PngImagePlugin.PngImageFile image mode=L size=4020x4892 at 0x11BA5D9A0>]
And y_train is:
array(['neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg', 'neg',
...
'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos', 'pos'],
dtype='<U3')
However, I want that x_train is in this format:
array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint8)
How can I convert it?
EDIT for #Pyaive Oleg If I do im = np.array(im) then the result is this, which is different from the one that I want. The same for the tensor
[array([[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
...,
[ 5, 6, 7, ..., 14, 9, 5],
[ 4, 5, 6, ..., 12, 8, 4],
[ 0, 1, 2, ..., 3, 2, 0]], dtype=uint8),
array([[ 1, 1, 1, ..., 8, 246, 0],
[ 1, 1, 1, ..., 0, 7, 11],
[ 1, 1, 1, ..., 0, 0, 6],
...,
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
...
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
array([[ 0, 0, 0, ..., 0, 255, 1],
[ 0, 0, 0, ..., 0, 3, 11],
[ 0, 0, 0, ..., 2, 0, 7],
...,
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),
array([[ 1, 1, 1, ..., 19, 246, 0],
[ 1, 1, 1, ..., 0, 16, 0],
[ 1, 1, 1, ..., 2, 0, 12],
...,
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0],
[ 0, 0, 0, ..., 0, 0, 0]], dtype=uint8),

Before appending imt to x_train, do this:
imt = np.array(imt)
The following also can help:
from torchvision import transforms
imt = transforms.ToTensor()(imt)

python - Create sparse tiles in numpy

I looked through the numpy documentation, but I don't know the exact terminology of the thing I want to do. I want to do the following:
a=np.array([[0,1,2],
[3,4,5],
[6,7,8],
[,9,10,11]])
b=np.sparse_tiles(a,(1,2),dtype=a.dtype)
b
array([[0,0,0,0,0,0,0,0,0],
[0,0,0,0,0,1,0,0,2],
[0,0,0,0,0,0,0,0,0],
[0,0,3,0,0,4,0,0,5],
[0,0,0,0,0,0,0,0,0],
[0,0,6,0,0,7,0,0,8],
[0,0,0,0,0,0,0,0,0],
[0,0,9,0,0,10,0,0,11]])
Is there anything built into numpy that can facilitate this? I prefer way of doing it without using loops.

You can use np.kron:
>>> np.kron(a, [[0,0,0],[0,0,1]])
array([[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 1, 0, 0, 2],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 3, 0, 0, 4, 0, 0, 5],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 6, 0, 0, 7, 0, 0, 8],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 9, 0, 0, 10, 0, 0, 11]])

Stepped slices do the trick:
In [210]: b=np.zeros((8,9),int)
In [211]: b[1::2, 2::3] = a
In [212]: b
Out[212]:
array([[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 1, 0, 0, 2],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 3, 0, 0, 4, 0, 0, 5],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 6, 0, 0, 7, 0, 0, 8],
[ 0, 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 9, 0, 0, 10, 0, 0, 11]])

Concatenating or adding elements of an array

i have a numpy array of 27 elements,Im trying concatenate or add all the elements inside the array,but i cant come up with anything right,
I tried,
for index,value in enumerate(array):
np.concatenate(array[index],array[index])
but this throws
TypeError: only integer scalar arrays can be converted to a scalar index
I tried
array[1]+array[2]+array[3]
this works for me, but im not sure how to put this in a loop,
Any suggestions on this front would be really helpful
Thanks in advance.
EDIT:
array looks like this
array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,

You want to sum over one axis (the first, I think). Like this:
array.sum(axis=0)