Concatenating or adding elements of an array - python

i have a numpy array of 27 elements,Im trying concatenate or add all the elements inside the array,but i cant come up with anything right,
I tried,
for index,value in enumerate(array):
np.concatenate(array[index],array[index])
but this throws
TypeError: only integer scalar arrays can be converted to a scalar index
I tried
array[1]+array[2]+array[3]
this works for me, but im not sure how to put this in a loop,
Any suggestions on this front would be really helpful
Thanks in advance.
EDIT:
array looks like this
array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,


You want to sum over one axis (the first, I think). Like this:
array.sum(axis=0)

Related

Multi-agent grid world Q-learning observation parameter

I am trying to train agents in a multi-agent grid world using q-learning. I am using the existing environment 'collection-game': https://reposhub.com/python/deep-learning/ArnaudFickinger-gym-multigrid.html#articleHeader2. My problem is the observation/state-parameter that the environment algorithm returns. I want to use this parameter to choose the best policy given the state the agent is currently in. An example of this is shown below. Here the state parameter returns an integer that can be used to index the q-table.
Example of using the state to pick policy
The environment that I am using however returns an array of arrays that shows the observations for all agents in the environment:
[array([[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[10, 2, 0, 0, 2, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 1, 0, 0, 0, 1]],
[[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0]]], dtype=uint8), array([[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 1, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 2, 0, 0, 2, 1]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 0],
[ 6, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0]]], dtype=uint8), array([[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[10, 3, 0, 0, 3, 1]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]],
[[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 2, 5, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0],
[ 6, 0, 0, 0, 0, 0],
[ 1, 0, 0, 0, 0, 0]]], dtype=uint8)]
My question is: how to interpret this variable, and how to use it to pick the best policy for each agent.

How to add label to image?

Hi I used this line of code to see the format of images in mnist dataset:
print(mnsit.load_data())
The output is this:
(array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint8), array([5, 0, 4, ..., 5, 6, 8], dtype=uint8))
The main array contains an array at the end. This array is a set of labels that I give to input of a neural network.
Now I need to add an array of labels at the end of images to create a dataset like mnist dataset. Are there functions I can use to do this? I tried append and insert but it didn't work.

You could do something like that:
dataset = [image_array, label_array]
That creates an list with the two array as a elements.

python/numpy combine subarrays 4 rows at a time

I have a numpy array that is split by each row:
splitArray:
[[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]
I was hoping to merge said splitArray every 4 rows, and the last subarray not necessarily having to be 4, but just the remainder of what's left.
Below is the array I hope to have:
joinedArray:
[[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0]]

Using a list-comp:
[a[i:i+4] for i in range(0, len(a), 4)]
#[array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]]),
# array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]]),
# array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]]),
# array([[0, 0, 0, 0, 0, 0, 0],
# [0, 0, 0, 0, 0, 0, 0]])]

As a pure Numpythonic approach you can find all the desired indexes for splitting your array by creating a range from chunking number to number of rows with the chunking number as thestep arg of the range. Then use np.split() to split your array:
In [24]: def chunk_array(arr, ch):
...: x = arr.shape[0]
...: return np.split(a, np.arange(ch, x, ch))
...:
...:
Demo:
In [25]: chunk_array(a, 4)
Out[25]:
[array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]])]
In [26]: chunk_array(a, 3)
Out[26]:
[array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]), array([[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]])]
If you want the chunked arrays to be concatenated you can use #jpp's answer with np.concatenate() and map or slightly different in a list comprehension.
In [75]: def chunk_array(arr, ch):
...: x = arr.shape[0]
...: return [np.concatenate(subs) for subs in np.split(arr, np.arange(ch, x, ch))]

That can be done using the infamous grouper recipe.
>>> from itertools import zip_longest
>>> import numpy as np
>>>
>>> data = [7 * [0] for i in range(14)]
>>> i=iter(data); list(map(np.concatenate, zip_longest(*4*(i,), fillvalue=[])))
[array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0]), array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0]), array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0]), array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])]

You can use np.concatenate with np.split. If required, you can adjust the below example to output a list of lists instead of a list of arrays.
As mentioned, a single jagged numpy array is not a good idea.
A = np.zeros((14, 3))
res = list(map(np.concatenate, np.split(A, np.arange(4, A.shape[0], 4))))
print(res)
[array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
array([ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
array([ 0., 0., 0., 0., 0., 0.])]

Create boolean mask on TensorFlow

Suppose I have a list
x = [0, 1, 3, 5]
And I want to get a tensor with dimensions
s = (10, 7)
Such that the first column of the rows with indexes defined in x are 1, and 0 otherwise.
For this particular example, I want to obtain the tensor containing:
T = [[1, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]]
Using numpy, this would be the equivalent:
t = np.zeros(s)
t[x, 0] = 1
I found this related answer, but it doesn't really solve my problem.

Try this:
import tensorflow as tf
indices = tf.constant([[0, 1],[3, 5]], dtype=tf.int64)
values = tf.constant([1, 1])
s = (10, 7)
st = tf.SparseTensor(indices, values, s)
st_ordered = tf.sparse_reorder(st)
result = tf.sparse_tensor_to_dense(st_ordered)
sess = tf.Session()
sess.run(result)
Here is the output:
array([[0, 1, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 1, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]], dtype=int32)
I slightly modified your indexes so you can see the x,y format of the indices
To obtain what you originally asked, set:
indices = tf.constant([[0, 0], [1, 0],[3, 0], [5, 0]], dtype=tf.int64)
Output:
array([[1, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[1, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0, 0]], dtype=int32)

scipy block_diag does not preserve complex numbers

I wrote a function that puts tensor products of Pauli matrices on the diagonal, using the block_diag function.
When I implement the function , I obtain:
array([[ 1, 0, 0, 0, 0, 0, 0, 0],
[ 0, 1, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 1, 0, 0, 0, 0],
[ 0, 0, 1, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0, 1, 0],
[ 0, 0, 0, 0, 0, 0, 0, -1]])
as you can see, the matrix
array([[ 0 , -1j],[1j,0]])
is missing, being replaced by a 2x2 0 matrix.
The warning it gives me is :
/usr/lib/python2.7/dist-packages/scipy/linalg/special_matrices.py:541: ComplexWarning: Casting complex values to real discards the imaginary part
out[r:r + rr, c:c + cc] = arrs[i]
Any ideas on how I can overcome this?

Categories