We Keep Coding

How to set values in a 2d numpy array given 1D indices for each row?

In numpy you can set the indices of a 1d array to a value
import numpy as np
b = np.array([0, 0, 0, 0, 0])
indices = [1, 3]
b[indices] = 1
b
array([0, 1, 0, 1, 0])
I'm trying to do this with multi-rows and an index for each row in the most programmatically elegant and computationally efficient way possible. For example
b = np.array([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]])
indices = [[1, 3], [0, 1], [0, 3]]
The desired result is
array([[0, 1, 0, 1, 0],
[1, 1, 0, 0, 0],
[1, 0, 0, 1, 0]])
I tried b[indices] and b[:,indices] but they resulted in an error or undesired result.
From searching, there are a few work arounds, but each tends to need at least 1 loop in python.
Solution 1: Run a loop through each row of the 2d array. The draw back for this is that the loop runs in python, and this part won't take advantage of numpy's c processing.
Solution 2: Use numpy put. The draw back is put works on a flattened version of the input array, so the indices need to be flattened too, and altered by the row size and number of rows, which would use a double for loop in python.
Solution 3: put_along_axis seems to only be able to set 1 value per row, so I would need to repeat this function for the number of values per row.
What would be the most computationally and programatically elegant solution? Anything where numpy would handle all the operations?

In [330]: b = np.zeros((3,5),int)
To set the (3,2) columns, the row indices need to be (3,1) shape (matching by broadcasting):
In [331]: indices = np.array([[1,3],[0,1],[0,3]])
In [332]: b[np.arange(3)[:,None], indices] = 1
In [333]: b
Out[333]:
array([[0, 1, 0, 1, 0],
[1, 1, 0, 0, 0],
[1, 0, 0, 1, 0]])
put along does the same thing:
In [335]: b = np.zeros((3,5),int)
In [337]: np.put_along_axis(b, indices,1,axis=1)
In [338]: b
Out[338]:
array([[0, 1, 0, 1, 0],
[1, 1, 0, 0, 0],
[1, 0, 0, 1, 0]])

On solution to build the indices in each dimension and then use a basic indexing:
from itertools import chain
b = np.array([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]])
# Find the indices along the axis 0
y = np.arange(len(indices)).repeat(np.fromiter(map(len, indices), dtype=np.int_))
# Flatten the list and convert it to an array
x = np.fromiter(chain.from_iterable(indices), dtype=np.int_)
# Finaly set the items
b[y, x] = 1
It works even for indices lists with variable-sized sub-lists like indices = [[1, 3], [0, 1], [0, 2, 3]]. If your indices list always contains the same number of items in each sub-list then you can use the (more efficient) following code:
b = np.array([[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]])
indices = np.array(indices)
n, m = indices.shape
y = np.arange(n).repeat(m)
x = indices.ravel()
b[y, x] = 1

Simple one-liner based on Jérôme's answer (requires all items of indices to be equal-length):
>>> b[np.arange(np.size(indices)) // len(indices[0]), np.ravel(indices)] = 1
>>> b
array([[0, 1, 0, 1, 0],
[1, 1, 0, 0, 0],
[1, 0, 0, 1, 0]])

How to loop by specific step

I have a nested loop data=
[[1,43,344,546],[2,34,45,565]....[10,1,1,1],
[1,15,111,151],[2,152,28,19]....[10,2,2,2],
[1,21,45,1647],[2,288,65,90]....[10,3,3,3]
.....]
so basically, all the inside lists can be grouped by 10lists with the first element always starting from 1 to 10. taking the every 10th list as a key, so I want to calculate a newlist by subtracting every list's number by the 10th list accordingly, for example
[[1,43-1,344-2,546-3], [2,34-1,45-2,565-3].... [10,1,2,3],
[1,15-21,111-22,151-23],[2,152-21,28-22,19-23]....[10,21,22,23],
[1,21-31,45-32,1647-33],[2,288-31,65-32,90-33]....[10,31,32,33]
.....]
My code seems don't work, can someone plz help with this? thanks
line = 0
while line <= (len(data) - 10):
for i in range(line, line + 10):
temp = data[i]
if temp[0] == 10: #find the every 10 th keys and store them to x, y, z
x = temp[1]
y = temp[2]
z = temp[3]
break
for sublist in data:
sublist[1] = sublist[1] - x# assign new elements to original list data
sublist[2] = sublist[2] - y
sublist[3] = sublist[3] - z
line += 10
return data

This is one way to do what you want
from pprint import pprint
def next_mul(n):
"returns next multiple of 10 greater than n"
return n + (10 - n % 10)
# sample data
data = [[i for i in range(4)] for i in range(20)]
for i in range(len(data)):
if i % 10 == 0:
continue
data[i-1][1:] = [k-l for k,l in zip(data[next_mul(i)-1][1:], data[i-1][1:])]
pprint(data)
Output (originally every sublist of data was just [0, 1, 2, 3]):
[[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 1, 2, 3],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 0, 0, 0],
[0, 1, 2, 3]]
Hope this helps. Please let me know if there are any questions/concerns!

How to construct matrix based on condition and position?

I have a matrix A=
np.matrix([[0, 0, 0, 0, 0],
[0, 0, 0, 1, 1],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 1, 1, 1]]
I wanna build a matrix B where B[i,j]=5 if A[i,j]=1 and (i+1)%3=0; B[i,j]=0 otherwise.
The B should be: B=
np.matrix([[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 5, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 5, 0, 0]]
Is there any possible method to achieve this without using for loop, just like matrix calculation? Thank you.

UPDATED ANSWER:
I have not thought of a way to eliminate the for-loop in the list comprehension for filtering on the remainder condition, but the "heavy" part of the computation exploits numpy's optimizations.
import numpy as np
newdata = 5 * np.array([(i + 1) % 3 == 0 for i in range(data.shape[-1])]) * np.array(data)
ORIGINAL ANSWER (prior to condition that for-loops cannot be used):
Assuming your matrix is stored as data, then you can use list comprehension syntax to get what you want.
newdata = [[5 if val == 1 and (idx + 1) % 3 == 0 else 0
for idx, val in enumerate(row)]
for row in data]

Python - how to inspect / discover list has references?

One of the more confusing aspects in python is instantiating a list of lists (assuming one isn't using numpy) - for instance, if one tries to do it via simpler multiplication you end up with reference copies:
In [1]: a = [[0] * 4] * 4
In [2]: a
Out[2]: [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
In [3]: a[0][1] = 1
In [4]: a
Out[4]: [[0, 1, 0, 0], [0, 1, 0, 0], [0, 1, 0, 0], [0, 1, 0, 0]]
As mentioned in various other SO post such as this one, a proper way to instantiate without any references would be as follows:
In [5]: b = [[0 for i in range(4)] for i in range(4)]
In [6]: b
Out[6]: [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
In [7]: b[0][1] = 1
In [8]: b
Out[8]: [[0, 1, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
The question is this - assume one does define their list as done with list a, is there a way to inspect the array in such a way that it will show that it is using references? Merely printing the array will not reveal the references.

Some terminology first: you are talking about lists here (not arrays), which always store references to their elements.
A quick way to check whether all references in a list refer to different objects is
>>> l1 = [[0, 1], [0, 1]]
>>> l2 = [[0, 1]]*2
>>>
>>> len(set(map(id, l1))) == len(l1) # no duplicates
True
>>> len(set(map(id, l2))) == len(l2) # duplicates
False
which simply checks whether there are n unique ids for the objects in a list of length n.
If your list has a very large number of elements, it might be more efficient to do this lazily and return False on the first duplicate id.
def all_unique(lst):
seen = set()
for x in lst:
id_ = id(x)
if id_ in seen:
return False
seen.add(id_)
return True
... works like this:
>>> all_unique(l1)
True
>>> all_unique(l2)
False

You can use the id function:
>>> a = [[0] * 4] * 4
>>> a
[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
>>> [id(sublist) for sublist in a]
[1975671202696, 1975671202696, 1975671202696, 1975671202696]
>>> b = [[0 for i in range(4)] for i in range(4)]
>>> b
[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
>>> [id(sublist) for sublist in b]
[1975671204808, 1975671205128, 1975671205000, 1975671204872]
As you can see, in a, the ids are all the same, while in b, they are different.

Python 2.7 creating a multidimensional list

In Python I want an intuitive way to create a 3 dimensional list.
I want an (n by n) list. So for n = 4 it should be:
x = [[[],[],[],[]],[[],[],[],[]],[[],[],[],[]],[[],[],[],[]]]
I've tried using:
y = [n*[n*[]]]
y = [[[]]* n for i in range(n)]
Which both appear to be creating copies of a reference.
I've also tried naive application of the list builder with little success:
y = [[[]* n for i in range(n)]* n for i in range(n)]
y = [[[]* n for i in range(1)]* n for i in range(n)]
I've also tried building up the array iteratively using loops, with no success. I also tried this:
y = []
for i in range(0,n):
y.append([[]*n for i in range(n)])
Is there an easier or more intuitive way of doing this?

I think your list comprehension versions were very close to working. You don't need to do any list multiplication (which doesn't work with empty lists anyway). Here's a working version:
>>> y = [[[] for i in range(n)] for i in range(n)]
>>> print y
[[[], [], [], []], [[], [], [], []], [[], [], [], []], [[], [], [], []]]

looks like the most easiest way is as follows:
def create_empty_array_of_shape(shape):
if shape: return [create_empty_array_of_shape(shape[1:]) for i in xrange(shape[0])]
it's work for me

i found this:
Matrix = [[0 for x in xrange(5)] for x in xrange(5)]
You can now add items to the list:
Matrix[0][0] = 1
Matrix[4][0] = 5
print Matrix[0][0] # prints 1
print Matrix[4][0] # prints 5
from here: How to define two-dimensional array in python

A very simple and elegant way is:
a = [([0] * 5) for i in range(5)]
a
[[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]

In Python I made a little factory method to create a list of variable dimensions and variable sizes on each of those dimensions:
def create_n_dimensional_matrix(self, n):
dimensions = len(n)
if (dimensions == 1):
return [0 for i in range(n[0])]
if (dimensions == 2):
return [[0 for i in range(n[0])] for j in range(n[1])]
if (dimensions == 3):
return [[[0 for i in range(n[0])] for j in range(n[1])] for k in range(n[2])]
if (dimensions == 4):
return [[[[0 for i in range(n[0])] for j in range(n[1])] for k in range(n[2])] for l in range(n[3])]
run it like this:
print(str(k.create_n_dimensional_matrix([2,3])))
print(str(k.create_n_dimensional_matrix([3,2])))
print(str(k.create_n_dimensional_matrix([1,2,3])))
print(str(k.create_n_dimensional_matrix([3,2,1])))
print(str(k.create_n_dimensional_matrix([2,3,4,5])))
print(str(k.create_n_dimensional_matrix([5,4,3,2])))
Which prints:
The two dimensional lists (2x3), (3x2)
The three dimensional lists (1x2x3),(3x2x1)
The four dimensional lists (2x3x4x5),(5x4x3x2)
[[0, 0], [0, 0], [0, 0]]
[[0, 0, 0], [0, 0, 0]]
[[[0], [0]], [[0], [0]], [[0], [0]]]
[[[0, 0, 0], [0, 0, 0]]]
[[[[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]]], [[[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]]], [[[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]]], [[[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]]], [[[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]], [[0, 0], [0, 0], [0, 0]]]]
[[[[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]], [[[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]], [[0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0], [0, 0, 0, 0, 0]]]]

How about this:
class MultiDimList(object):
def __init__(self, shape):
self.shape = shape
self.L = self._createMultiDimList(shape)
def get(self, ind):
if(len(ind) != len(self.shape)): raise IndexError()
return self._get(self.L, ind)
def set(self, ind, val):
if(len(ind) != len(self.shape)): raise IndexError()
return self._set(self.L, ind, val)
def _get(self, L, ind):
return self._get(L[ind[0]], ind[1:]) if len(ind) > 1 else L[ind[0]]
def _set(self, L, ind, val):
if(len(ind) > 1):
self._set(L[ind[0]], ind[1:], val)
else:
L[ind[0]] = val
def _createMultiDimList(self, shape):
return [self._createMultiDimList(shape[1:]) if len(shape) > 1 else None for _ in range(shape[0])]
def __repr__(self):
return repr(self.L)
You can then use it as follows
L = MultiDimList((3,4,5)) # creates a 3x4x5 list
L.set((0,0,0), 1)
L.get((0,0,0))

import copy
dimensions = 2, 3, 4
z = 0
genList = lambda size,value: [copy.deepcopy(value) for i in range(size)]
for i in dimensions: z = genList(i, z)

I am amazed no one tried to devise a generic way to do it.
See my answer here: https://stackoverflow.com/a/33460217/5256940
import copy
def ndlist(init, *args): # python 2 doesn't have kwarg after *args
dp = init
for x in reversed(args):
dp = [copy.deepcopy(dp) for _ in xrange(x)] # Python 2 xrange
return dp
l = ndlist(0, 1, 2, 3, 4) # 4 dimensional list initialized with 0's
l[0][1][2][3] = 1
Edit: Built on user2114402's answer: added default value parameter
def ndlist(s, v):
return [ndlist(s[1:], v) for i in xrange(s[0])] if s else v

Here's one that will give you an N dimensional "matrix" filled up with copies of a copiable object.
Edit: This is a slight modification of pterodragon's original answer, which I much prefer to user2114402's less readable answer. In fact, outside of a doc-string the only difference from pterodragon's solution is that I explicitly use a list of dimension sizes, rather than have the user pass them as arguments.
import copy
def instantiate_mdl(dim_maxes, base=0):
""" Instantiate multi-dimensional list, that is a list of list of list ...
Arguments:
dim_maxes (list[int]): a list of dimension sizes, for example
[2, 4] represents a matrix (represented by lists) of 2 rows and
4 columns.
base (object): an optional argument indicating the object copies
of which will reside at the lowest level in the datastructure.
Returns:
base (list[base]): a multi-dimensional list of lists structure,
which is filled with clones of the base parameter.
"""
for dim_max in reversed(dim_maxes):
base = [copy.deepcopy(base) for i in range(dim_max)]
return base
data = instantiate_mdl([3, 5])
data[0][0] = 99999
data[1][1] = 88888
data[2][4] = 77777
for r in data:
print(r)
>>> # Output
>>> [99999, 0, 0, 0, 0]
>>> [0, 88888, 0, 0, 0]
>>> [0, 0, 0, 0, 77777]

Here is a losution that works for any number of dimention :
def multi_dimensional_list(dimensions, filling=None):
if len(dimensions) == 1:
return [filling] * dimensions[0]
else:
return [
multi_dimensional_list(dimensions[1:], filling)
for _ in range(dimensions[0])
]
print(multi_dimensional_list([2, 3, 4], 0))
"""
output :
[
[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]],
[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
]
"""

Here is a more generic way of doing it.
def ndlist(shape, dtype=list):
t = '%s for v%d in xrange(shape[%d])'
cmd = [t % ('%s', i + 1, i) for i in xrange(len(shape))]
cmd[-1] = cmd[-1] % str(dtype())
for i in range(len(cmd) - 1)[::-1]:
cmd[i] = cmd[i] % ('[' + cmd[i + 1] + ']')
return eval('[' + cmd[0] + ']')
list_4d = ndlist((2, 3, 4))
list_3d_int = ndlist((2, 3, 4), dtype=int)
print list_4d
print list_3d_int
Result:
[[[[], [], [], []], [[], [], [], []], [[], [], [], []]], [[[], [], [], []], [[], [], [], []], [[], [], [], []]]]
[[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]]

You can also build a 2D list with different length rows using the append method.
e.g.
sf_gcov_cell = []
sf_gcov_cell.append(['r1_c1', 'r2_c2_', 'r3_c3__', 'r4_c4'])
sf_gcov_cell.append(['r2_c1', 'r2_c2'])
sf_gcov_cell.append(['r3_c1', 'r3_c2___', 'r3_c3_'])
print(sf_gcov_cell)
Result:
[['r1_c1', 'r2_c2_', 'r3_c3__', 'r4_c4'], ['r2_c1', 'r2_c2'], ['r3_c1', 'r3_c2___', 'r3_c3_']]

Just use a simple recursive function that continuously appends new arrays to the initialized arrays within itself.
# Counts the maximum amount of dimensions in an array
def dcounter(array: list, total = 1):
for arr in array:
return dcounter(arr, total + 1)
return total
# Create array with specified amount of dimensions
def nd_array(dimensions = 1, array = list(), i = 0):
if dimensions > 1:
array.append([])
nd_array(dimensions - 1, array[i])
return array
# Create a 5D array
array = nd_array(5)
print(array) # => "[[[[[]]]]]"
# Get the maximum amount of dimensions in the array
amount = dcounter(array)
print(amount) # => "5"
If you wanted to specify the size of each dimension, the amount of arrays within each dimension, then you could modify nd_array into doing so.