We Keep Coding

How can I manipulate a numpy array without nested loops?

If I have a MxN numpy array denoted arr, I wish to index over all elements and adjust the values like so
for m in range(arr.shape[0]):
for n in range(arr.shape[1]):
arr[m, n] += x**2 * np.cos(m) * np.sin(n)
Where x is a random float.
Is there a way to broadcast this over the entire array without needing to loop? Thus, speeding up the run time.

You are just adding zeros, because sin(2*pi*k) = 0 for integer k.
However, if you want to vectorize this, the function np.meshgrid could help you.
Check the following example, where I removed the 2 pi in the trigonometric functions to add something unequal zero.
x = 2
arr = np.arange(12, dtype=float).reshape(4, 3)
n, m = np.meshgrid(np.arange(arr.shape[1]), np.arange(arr.shape[0]), sparse=True)
arr += x**2 * np.cos(m) * np.sin(n)
arr
Edit: use the sparse argument to reduce memory consumption.

You can use nested generators of two-dimensional arrays:
import numpy as np
from random import random
x = random()
n, m = 10,20
arr = [[x**2 * np.cos(2*np.pi*j) * np.sin(2*np.pi*i) for j in range(m)] for i in range(n)]

In [156]: arr = np.ones((2, 3))
Replace the range with arange:
In [157]: m, n = np.arange(arr.shape[0]), np.arange(arr.shape[1])
And change the first array to (2,1) shape. A (2,1) array broadcasts with a (3,) to produce a (2,3) result.
In [158]: A = 0.23**2 * np.cos(m[:, None]) * np.sin(n)
In [159]: A
Out[159]:
array([[0. , 0.04451382, 0.04810183],
[0. , 0.02405092, 0.02598953]])
In [160]: arr + A
Out[160]:
array([[1. , 1.04451382, 1.04810183],
[1. , 1.02405092, 1.02598953]])
The meshgrid suggested in the accepted answer does the same thing:
In [161]: np.meshgrid(m, n, sparse=True, indexing="ij")
Out[161]:
[array([[0],
[1]]),
array([[0, 1, 2]])]
This broadcasting may be clearer with:
In [162]: m, n
Out[162]: (array([0, 1]), array([0, 1, 2]))
In [163]: m[:, None] * 10 + n
Out[163]:
array([[ 0, 1, 2],
[10, 11, 12]])

Difference between single and double bracket Numpy array?

What is the difference between these two numpy objects?
import numpy as np
np.array([[0,0,0,0]])
np.array([0,0,0,0])

In [71]: np.array([[0,0,0,0]]).shape
Out[71]: (1, 4)
In [72]: np.array([0,0,0,0]).shape
Out[72]: (4,)
The former is a 1 x 4 two-dimensional array, the latter a 4 element one-dimensional array.

The difference between single and double brackets starts with lists:
In [91]: ll=[0,1,2]
In [92]: ll1=[[0,1,2]]
In [93]: len(ll)
Out[93]: 3
In [94]: len(ll1)
Out[94]: 1
In [95]: len(ll1[0])
Out[95]: 3
ll is a list of 3 items. ll1 is a list of 1 item; that item is another list. Remember, a list can contain a variety of different objects, numbers, strings, other lists, etc.
Your 2 expressions effectively make arrays from two such lists
In [96]: np.array(ll)
Out[96]: array([0, 1, 2])
In [97]: _.shape
Out[97]: (3,)
In [98]: np.array(ll1)
Out[98]: array([[0, 1, 2]])
In [99]: _.shape
Out[99]: (1, 3)
Here the list of lists has been turned into a 2d array. In a subtle way numpy blurs the distinction between the list and the nested list, since the difference between the two arrays lies in their shape, not a fundamental structure. array(ll)[None,:] produces the (1,3) version, while array(ll1).ravel() produces a (3,) version.
In the end result the difference between single and double brackets is a difference in the number of array dimensions, but we shouldn't loose sight of the fact that Python first creates different lists.

When you defined an array with two brackets, what you were really doing was declaring an array with an array with 4 0's inside. Therefore, if you wanted to access the first zero you would be accessing
your_array[0][0] while in the second array you would just be accessing your array[0]. Perhaps a better way to visualize it is
array: [
[0,0,0,0],
]
vs
array: [0,0,0,0]

Convert a numpy array to an array of numpy arrays

How can I convert numpy array a to numpy array b in a (num)pythonic way. Solution should ideally work for arbitrary dimensions and array lengths.
import numpy as np
a=np.arange(12).reshape(2,3,2)
b=np.empty((2,3),dtype=object)
b[0,0]=np.array([0,1])
b[0,1]=np.array([2,3])
b[0,2]=np.array([4,5])
b[1,0]=np.array([6,7])
b[1,1]=np.array([8,9])
b[1,2]=np.array([10,11])

For a start:
In [638]: a=np.arange(12).reshape(2,3,2)
In [639]: b=np.empty((2,3),dtype=object)
In [640]: for index in np.ndindex(b.shape):
b[index]=a[index]
.....:
In [641]: b
Out[641]:
array([[array([0, 1]), array([2, 3]), array([4, 5])],
[array([6, 7]), array([8, 9]), array([10, 11])]], dtype=object)
It's not ideal since it uses iteration. But I wonder whether it is even possible to access the elements of b in any other way. By using dtype=object you break the basic vectorization that numpy is known for. b is essentially a list with numpy multiarray shape overlay. dtype=object puts an impenetrable wall around those size 2 arrays.
For example, a[:,:,0] gives me all the even numbers, in a (2,3) array. I can't get those numbers from b with just indexing. I have to use iteration:
[b[index][0] for index in np.ndindex(b.shape)]
# [0, 2, 4, 6, 8, 10]
np.array tries to make the highest dimension array that it can, given the regularity of the data. To fool it into making an array of objects, we have to give an irregular list of lists or objects. For example we could:
mylist = list(a.reshape(-1,2)) # list of arrays
mylist.append([]) # make the list irregular
b = np.array(mylist) # array of objects
b = b[:-1].reshape(2,3) # cleanup
The last solution suggests that my first one can be cleaned up a bit:
b = np.empty((6,),dtype=object)
b[:] = list(a.reshape(-1,2))
b = b.reshape(2,3)
I suspect that under the covers, the list() call does an iteration like
[x for x in a.reshape(-1,2)]
So time wise it might not be much different from the ndindex time.
One thing that I wasn't expecting about b is that I can do math on it, with nearly the same generality as on a:
b-10
b += 10
b *= 2
An alternative to an object dtype would be a structured dtype, e.g.
In [785]: b1=np.zeros((2,3),dtype=[('f0',int,(2,))])
In [786]: b1['f0'][:]=a
In [787]: b1
Out[787]:
array([[([0, 1],), ([2, 3],), ([4, 5],)],
[([6, 7],), ([8, 9],), ([10, 11],)]],
dtype=[('f0', '<i4', (2,))])
In [788]: b1['f0']
Out[788]:
array([[[ 0, 1],
[ 2, 3],
[ 4, 5]],
[[ 6, 7],
[ 8, 9],
[10, 11]]])
In [789]: b1[1,1]['f0']
Out[789]: array([8, 9])
And b and b1 can be added: b+b1 (producing an object dtype). Curiouser and curiouser!

Based on hpaulj I provide a litte more generic solution. a is an array of dimension N which shall be converted to an array b of dimension N1 with dtype object holding arrays of dimension (N-N1).
In the example N equals 5 and N1 equals 3.
import numpy as np
N=5
N1=3
#create array a with dimension N
a=np.random.random(np.random.randint(2,20,size=N))
a_shape=a.shape
b_shape=a_shape[:N1] # shape of array b
b_arr_shape=a_shape[N1:] # shape of arrays in b
#Solution 1 with list() method (faster)
b=np.empty(np.prod(b_shape),dtype=object) #init b
b[:]=list(a.reshape((-1,)+b_arr_shape))
b=b.reshape(b_shape)
print "Dimension of b: {}".format(len(b.shape)) # dim of b
print "Dimension of array in b: {}".format(len(b[0,0,0].shape)) # dim of arrays in b
#Solution 2 with ndindex loop (slower)
b=np.empty(b_shape,dtype=object)
for index in np.ndindex(b_shape):
b[index]=a[index]
print "Dimension of b: {}".format(len(b.shape)) # dim of b
print "Dimension of array in b: {}".format(len(b[0,0,0].shape)) # dim of arrays in b

Forming matrix from 2 vectors in Numpy, with repetition of 1 vector

Using numpy arrays I want to create such a matrix most economically:
given
from numpy import array
a = array(a1,a2,a3,...,an)
b = array(b1,...,bm)
shall be processed to matrix M:
M = array([[a1,a2,b1,...,an],
... ...,
[a1,a2,bm,...,an]]
I am aware of numpy array's broadcasting methods but couldn't figure out a good way.
Any help would be much appreciated,
cheers,
Rob

You can use numpy.resize on a first and then add b's items at the required indices using numpy.insert on the re-sized array:
In [101]: a = np.arange(1, 4)
In [102]: b = np.arange(4, 6)
In [103]: np.insert(np.resize(a, (b.shape[0], a.shape[0])), 2, b, axis=1)
Out[103]:
array([[1, 2, 4, 3],
[1, 2, 5, 3]])

You can use a combination of numpy.tile and numpy.hstack functions.
M = numpy.repeat(numpy.hstack(a, b), (N,1))
I'm not sure I understand your target matrix, though.

Dealing with N by 1 matrices in Numpy

Given a numpy array of size (n,) how do you transform it to a numpy array of size (n,1).
The reason is because I am trying to matrix multiply to numpy arrays of size (n,) and (,n) to get a (n,n) but when I do:
numpy.dot(a,b.T)
It says that you can't do it. I know as a fact that transposing a (n,) does nothing, so it would just be nice to change the (n,) and make them (n,1) and avoid this problem all together.

Use reshape (-1,1) to reshape (n,) to (n,1), see detail examples:
In [1]:
import numpy as np
A=np.random.random(10)
In [2]:
A.shape
Out[2]:
(10,)
In [3]:
A1=A.reshape(-1,1)
In [4]:
A1.shape
Out[4]:
(10, 1)
In [5]:
A.T
Out[5]:
array([ 0.6014423 , 0.51400033, 0.95006413, 0.54321892, 0.2150995 ,
0.09486603, 0.54560678, 0.58036358, 0.99914564, 0.09245124])
In [6]:
A1.T
Out[6]:
array([[ 0.6014423 , 0.51400033, 0.95006413, 0.54321892, 0.2150995 ,
0.09486603, 0.54560678, 0.58036358, 0.99914564, 0.09245124]])

You can use None for dimensions that you want to be treated as degenerate.
a = np.asarray([1,2,3])
a[:]
a[:, None]
In [48]: a
Out[48]: array([1, 2, 3])
In [49]: a[:]
Out[49]: array([1, 2, 3])
In [50]: a[:, None]
Out[50]:
array([[1],
[2],
[3]])