We Keep Coding

Shifting all element positions of an array in Python given by XOR operator

I would like to know if there is a way to move all elements of a numpy array without iterating over each entry. The shift I desire is to relabel the indices by a fixed XOR operation, in the following form:
import numpy as np
N = 2
x = np.arange(2**(2 * N)).reshape(2**N, 2**N)
z = np.zeros((2**N, 2**N))
k = 1
for i in range(2**N):
for j in range(2**N):
z[i][j] = x[i ^ k][j ^ k]
The problem I have is that latter I wish to take huge values of N, which becomes a bottleneck if we wish to iterate over each entry. Any advice on how to perform the move in a single shot will be very much appreciated.

There is simply no reason to use a loop here at all. You are not shifting anything, and the problem is completely separable across the axes:
x = np.arange(2**(2 * N)).reshape(2**N, 2**N)
z = x[np.arange(2**N)[:, None] ^ k, np.arange(2**N) ^ k]
But what is x[a, b] for some a, b to begin with? From the original definition, you can see that it's a * 2**N + b. You can therefore plug in a = np.arange(2**N)[:, None] ^ k and b = np.arange(2**N) ^ k to avoid having to generate anything but the final result:
idx = np.arange(2**N) ^ k
z = (idx[:, None] << N) + idx
It's generally faster to substitute << N for * 2**N.
Allocation
None of the solutions shown here pre-allocate z. However, if you were to do that, as in the original question, you should be careful about the type.
z = np.zeros((2**N, 2**N))
This creates an array of floats by default, which is likely not what you want.
z = np.zeros((2**N, 2**N), dtype=int)
Adding an explicit dtype make the array into integers. However, the simplest method available to you is likely
z = np.zeros_like(x)
Since you are planning on filling in every single element, you don't need to waste time filling it with zeros first. np.empty presents a better option in this case:
z = np.empty((2**N, 2**N), dtype=int)
OR
z = np.empty_like(x)

How to add a vector element to every element of the subarray along the associated index of a multidimensional array

I'm wondering if there is a more "numpythonic" or efficient way of doing the following:
Suppose I have a 1D array A of known length L, and I have a multidimensional array B, which has a dimension also with length L. Suppose I want to add (or set) the value of B[:, ..., x, ..., :] += A[x]. In other words, add the value A[x] to every value of the entire sub-array of B in the matching index x.
An extremely simple working example is this:
A = np.arange(10, 20)
B = np.random.rand(3, len(A), 3)
for iii in range(len(A)):
B[:, iii, :] += A[iii]
Clearly I can always loop over the index I want as above, but I'm curious if there's a more efficient way. If there's some more common terminology which describes this process, I'd also be interested because I'm having difficulty even constructing an appropriate Google search.
I'm also attempting to avoid creating a new array of the same shape as B and tiling the A-vector repeatedly over other indices and then adding that to B, as a more "real" world application would likely involve B being a relatively large array.

For your simple case, you can do:
B[:] = A[:, None]
This works because of broadcasting. By simulating the dimensions of B in A, you tell numpy where to place the elements unambiguously. For a more general case, where you want to place A along dimension k of B, you can do:
B[:] = np.expand_dims(A, tuple(range(A.ndim, A.ndim + B.ndim - k)))
np.expand_dims will add axes at the indices you tell it to. There are other ways too. For example, you can index A with B.ndim - k - 1 instances of None:
B[:] = A[(slice(None), *(None,) * (B.ndim - k - 1))]
You can also use np.reshape to get the correctly shaped view:
B[:] = A.reshape(-1, *np.ones(B.ndim - k - 1, dtype=int))
OR
B[:] = A.reshape(-1, *(1,) * (B.ndim - k - 1))
OR
B[:] = A.reshape((-1,) + (1,) * (B.ndim - k - 1))
In all these cases, you only need to expand the trailing dimensions of A, since that's how broadcasting works. Since broadcasting is such an integral part of numpy, you can simply relpace = with += to get the expected result.

Incrementing the value of a ROI that (possibly) partially is out of bounds in Python

I have an array that needs to be partially incremented with the values of another (smaller) array. However, even if the smaller array does start inside the bounds of bigger array, it can overflow in some cases like shown below.
In this case I still want the values to be incremented/modified with a manner that clips the overflowing indices and uses what remains inside the bounds.
Here is how I do it at the moment
h = big_image.shape[0]
w = big_image.shape[1]
small_image = make_small_image(c) # c is the edge length
if (x+c) > w:
small_image = small_image[:(w - x), :]
if (y+c) > h:
small_image = small_image[:, :(h - y)]
big_image[y:y + c, x: x + c] += small_image.T
I wonder if there is a easier and faster way to do it (an API or a better technique etc.)

Right now you are modifying the array small_image and thus creating a new copy of the data when the if-condition is True. Instead you could just create a memory-view to the cropped small image:
small_img_memview = small_image[:(w - x), :(h - y)]
This should always be correct, even for the case that the if-conditions are False and should be faster than copying.
Of course you can also just write all of your code in one line:
big_image[y:y + c, x: x + c] += small_image[:(w - x), :(h - y)].T

Obtaining an array of values for coupled nonlinear equations by iterating over input arrays

I am attempting to solve a coupled system of nonlinear equations:
x(x+2y)/(1-x-y) = A
y(x+2y)/x = B
where A and B are elements in two different arrays of identical size.
I have 10,000 values for A in an array, and 10,000 values for B in another array.
I need to determine x[i] and y[i] using A[i] and B[i]. However, when I do the following:
def Eqs(z):
x, y = z
for i in A and B:
return (x*(x + 2*y) - A[i]*(1 - x - y), y*(x + 2*y) - B[i]*x)
x, y = scopt.fsolve(Eqs, (1,1))
I get the following error message:
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Using random arrays of appropriate size for A and B, Could someone please (1) explain why I'm getting that particular error message, and (2) suggest a work-around?

The line for i in A and B: doesn't make much sense. A and B is a binary logical operation on the arrays A and B, which is neither valid nor the operation you need.
What you want to do is probably something like for a, b in zip(A, B):. This yields the scalars a and b in every iteration. So a = A[i] and b = B[i]. zip creates pairs of values from A and B, which are than available as a and b within the for-loop.
Alternatively you can iterate using the index i: for i in range(len(A)):
There is another issue with the code: return will stop the loop and leave the method in the very first iteration. So you either create two lists for the return values, fill it throughout the for-loop and return it at the end of Eqs. Or you shorten it using list comprehensions:
return [x*(x + 2*y) - a*(1 - x - y) for a in A], [y*(x + 2*y) - b*x for b in B]
This should yield two lists.

Numpy ValueError: setting an array element with a sequence. This message may appear without the existing of a sequence?

Why do I get this error message? ValueError: setting an array element with a sequence. Thank you
Z=np.array([1.0,1.0,1.0,1.0])
def func(TempLake,Z):
A=TempLake
B=Z
return A*B
Nlayers=Z.size
N=3
TempLake=np.zeros((N+1,Nlayers))
kOUT=np.zeros(N+1)
for i in xrange(N):
kOUT[i]=func(TempLake[i],Z)

You're getting the error message
ValueError: setting an array element with a sequence.
because you're trying to set an array element with a sequence. I'm not trying to be cute, there -- the error message is trying to tell you exactly what the problem is. Don't think of it as a cryptic error, it's simply a phrase. What line is giving the problem?
kOUT[i]=func(TempLake[i],Z)
This line tries to set the ith element of kOUT to whatever func(TempLAke[i], Z) returns. Looking at the i=0 case:
In [39]: kOUT[0]
Out[39]: 0.0
In [40]: func(TempLake[0], Z)
Out[40]: array([ 0., 0., 0., 0.])
You're trying to load a 4-element array into kOUT[0] which only has a float. Hence, you're trying to set an array element (the left hand side, kOUT[i]) with a sequence (the right hand side, func(TempLake[i], Z)).
Probably func isn't doing what you want, but I'm not sure what you really wanted it to do (and don't forget you can usually use vectorized operations like A*B rather than looping in numpy.) That should explain the problem, anyway.

It's a pity that both of the answers analyze the problem but didn't give a direct answer.
Let's see the code.
Z = np.array([1.0, 1.0, 1.0, 1.0])
def func(TempLake, Z):
A = TempLake
B = Z
return A * B
Nlayers = Z.size
N = 3
TempLake = np.zeros((N+1, Nlayers))
kOUT = np.zeros(N + 1)
for i in xrange(N):
# store the i-th result of
# function "func" in i-th item in kOUT
kOUT[i] = func(TempLake[i], Z)
The error shows that you set the ith item of kOUT(dtype:int) into an array. Here every item in kOUT is an int, can't directly assign to another datatype. Hence you should declare the data type of kOUT when you create it.
For example, like:
Change the statement below:
kOUT = np.zeros(N + 1)
into:
kOUT = np.zeros(N + 1, dtype=object)
or:
kOUT = np.zeros((N + 1, N + 1))
All code:
import numpy as np
Z = np.array([1.0, 1.0, 1.0, 1.0])
def func(TempLake, Z):
A = TempLake
B = Z
return A * B
Nlayers = Z.size
N = 3
TempLake = np.zeros((N + 1, Nlayers))
kOUT = np.zeros(N + 1, dtype=object)
for i in xrange(N):
kOUT[i] = func(TempLake[i], Z)
Hope it can help you.

I believe python arrays just admit values. So convert it to list:
kOUT = np.zeros(N+1)
kOUT = kOUT.tolist()

Z=np.array([1.0,1.0,1.0,1.0])
def func(TempLake,Z):
A=TempLake
B=Z
return A*B
Nlayers=Z.size
N=3
TempLake=np.zeros((N+1,Nlayers))
kOUT=np.vectorize(func)(TempLake,Z)
This works too , instead of looping , just vectorize however read below notes from the scipy documentation : https://docs.scipy.org/doc/numpy/reference/generated/numpy.vectorize.html
The vectorize function is provided primarily for convenience, not for performance. The implementation is essentially a for loop.
If otypes is not specified, then a call to the function with the first argument will be used to determine the number of outputs. The results of this call will be cached if cache is True to prevent calling the function twice. However, to implement the cache, the original function must be wrapped which will slow down subsequent calls, so only do this if your function is expensive.

To put a sequence or another numpy array into a numpy array,
Just change this line:
kOUT=np.zeros(N+1)
to:
kOUT=np.asarray([None]*(N+1))
Or:
kOUT=np.zeros((N+1), object)

KOUT[i] is a single element of a list. But you are assigning a list to this element. your func is generating a list.

You can try the expand option in Series.str.split('seperator', expand=True).
By default expand is False.
expand : bool, default False
Expand the splitted strings into separate columns.
If True, return DataFrame/MultiIndex expanding dimensionality.
If False, return Series/Index, containing lists of strings.