We Keep Coding

Raise Elements of Array to Series of Exponents

Suppose I have a numpy array such as:
a = np.arange(9)
>> array([0, 1, 2, 3, 4, 5, 6, 7, 8])
If I want to raise each element to succeeding powers of two, I can do it this way:
power_2 = np.power(a,2)
power_4 = np.power(a,4)
Then I can combine the arrays by:
np.c_[power_2,power_4]
>> array([[ 0, 0],
[ 1, 1],
[ 4, 16],
[ 9, 81],
[ 16, 256],
[ 25, 625],
[ 36, 1296],
[ 49, 2401],
[ 64, 4096]])
What's an efficient way to do this if I don't know the degree of the even monomial (highest multiple of 2) in advance?

One thing to observe is that x^(2^n) = (...(((x^2)^2)^2)...^2)
meaning that you can compute each column from the previous by taking the square.
If you know the number of columns in advance you can do something like:
import functools as ft
a = np.arange(5)
n = 4
out = np.empty((*a.shape,n),a.dtype)
out[:,0] = a
# Note: this works by side-effect!
# The optional second argument of np.square is "out", i.e. an
# array to write the result to (nonetheless the result is also
# returned directly)
ft.reduce(np.square,out.T)
out
# array([[ 0, 0, 0, 0],
# [ 1, 1, 1, 1],
# [ 2, 4, 16, 256],
# [ 3, 9, 81, 6561],
# [ 4, 16, 256, 65536]])
If the number of columns is not known in advance then the most efficient method is to make a list of columns, append as needed and only in the end use np.column_stack or np.c_ (if using np.c_ do not forget to cast the list to tuple first).

The straightforward approach is:
exponents = [2**n for n in a]
[a**e for e in exponents]
This works fine for relatively small numbers, but I see what looks like numerical overflow on the larger numbers. (Although I can compute those high powers just fine using scalars.)

The most elegant way I could think of is to not calculate the exponents beforehand. Since your exponents follow a very easy pattern, you can express everything using on list-comprehension.
result = [item**2*index for index,item in enumerate(a)]
If you are working with quite large datasets, this will cause some serious overhead. This statement will do all calculations immediately and save all calculated element in one large array. To mitigate this problem, you could you a generator expression, which will generate the data on the fly.
result = (item**2*index for index,item in enumerate(a))
See here for more details.

Load data from generator into already allocated numpy array

I have a large array
data = np.empty((n, k))
where both n and k are large. I also have a lot of generators g, each with k elements, and I want to load each generator into a row in data. I can do:
data[i] = list(g)
or something similar, but this makes a copy of the data in g. I can load with a for loop:
for j, x in enumerate(g):
data[i, j] = x
but I'm wondering if numpy has a way to do this already without copying or looping in Python.
I know that g have length k in advance and am happy to do some __len__ subclass patching if necessary. np.fromiter will accept something like that when creating a new array, but I'd rather load into this already existing array if possible, due to the constraints of my context.

There's not much you can do, as stated in the comments.
Although you can consider these two solutions:
using numpy.fromiter
Instead of creating data = np.empty((n, k)) yourself, use numpy.fromiter and the count argument, which is made specifically from this case where you know the number of items in advance. This way numpy won't have to "guess" the size and re-allocate until the guess is large enough.
Using fromiter allows to run the for loop in C instead of python. This might be a tiny bit faster, but the real bottleneck will likely be in your generators anyway.
Note that fromiter only deals with flat arrays, so you need to read everything flatten (e.g. using chain.from_iterable) and only then call reshape:
from itertools import chain
n = 20
k = 4
generators = (
(i*j for j in range(k))
for i in range(n)
)
flat_gen = chain.from_iterable(generators)
data = numpy.fromiter(flat_gen, 'int64', count=n*k)
data = data.reshape((n, k))
"""
array([[ 0, 0, 0, 0],
[ 0, 1, 2, 3],
[ 0, 2, 4, 6],
[ 0, 3, 6, 9],
[ 0, 4, 8, 12],
[ 0, 5, 10, 15],
[ 0, 6, 12, 18],
[ 0, 7, 14, 21],
[ 0, 8, 16, 24],
[ 0, 9, 18, 27],
[ 0, 10, 20, 30],
[ 0, 11, 22, 33],
[ 0, 12, 24, 36],
[ 0, 13, 26, 39],
[ 0, 14, 28, 42],
[ 0, 15, 30, 45],
[ 0, 16, 32, 48],
[ 0, 17, 34, 51],
[ 0, 18, 36, 54],
[ 0, 19, 38, 57]])
"""
using cython
If you can re-use data and want to avoid re-allocation of the memory, you can't use numpy's fromiter anymore. IMHO the only way to avoid the python's for loop is to implement it in cython. Again, this is extremely likely overkill, since you still have to read the generators in python.
For reference, the C implementation of fromiter looks like that: https://github.com/numpy/numpy/blob/v1.18.3/numpy/core/src/multiarray/ctors.c#L4001-L4118

There is no faster way than the ones you described. You have to allocate each element of the numpy array, either by iterating the generator or by allocating the entire list.

Couple of things here:
1) You can just say
for whatever in g:
do_stuff
Since g is a generator, the for loop understands how to get the data out of the generator.
2) You won't have to "copy" out of the generator necessarily (since it isn't doesn't have the entire sequence loaded in memory by design) but you will need to loop through it to fill up your numpy data structure. You might be able to squeeze out some performance (since your structures are large) with tools in numpy or itertools.
So the answer is "no" since you're using generators. If you don't need to have all of the data available at once, you can just use generators to keep the memory profile small but I don't have any context for what you are doing with the data.

numpy: how to get the maximum value like this in numpy? [duplicate]

I need a fast way to keep a running maximum of a numpy array. For example, if my array was:
x = numpy.array([11,12,13,20,19,18,17,18,23,21])
I'd want:
numpy.array([11,12,13,20,20,20,20,20,23,23])
Obviously I could do this with a little loop:
def running_max(x):
result = [x[0]]
for val in x:
if val > result[-1]:
result.append(val)
else:
result.append(result[-1])
return result
But my arrays have hundreds of thousands of entries and I need to call this many times. It seems like there's got to be a numpy trick to remove the loop, but I can't seem to find anything that will work. The alternative will be to write this as a C extension, but it seems like I'd be reinventing the wheel.

numpy.maximum.accumulate works for me.
>>> import numpy
>>> numpy.maximum.accumulate(numpy.array([11,12,13,20,19,18,17,18,23,21]))
array([11, 12, 13, 20, 20, 20, 20, 20, 23, 23])

As suggested, there is scipy.maximum.accumulate:
In [9]: x
Out[9]: [1, 3, 2, 5, 4]
In [10]: scipy.maximum.accumulate(x)
Out[10]: array([1, 3, 3, 5, 5])

logical arrays and mapping in python

I'm trying to vectorize some element calculations but having difficulty doing so without creating list comprehensions for local information to global information. I was told that I can accomplish what I want to do using logical arrays, but so far the examples I've found has not been helpful. While yes I can accomplish this with list comprehensions, speed is a main concern with my code.
I have a set of values that indicate indices in the "global" calculation that should not be adjusted.
For example, these "fixed" indices are
1 2 6
If my global calculation has ten elements, I would be able to set all the "free" values by creating a list of the set of the global indices and subtracting the fixed indices.
free = list(set(range(len(global)) - set(fixed))
[0, 3, 4, 5, 7, 8, 9]
in the global calculation, I would be able to adjust the "free" elements as shown in the following code snippet
global = np.ones(10)
global[free] = global[free] * 10
which should produce:
global = [10, 1, 1, 10, 10, 10, 1, 10, 10, 10]
my "local" calculation is a subset of the global one, where the local map indicates the corresponding indices in the global calculation.
local_map = [4, 2, 1, 8, 6]
local_values = [40, 40, 40, 40, 40]
but I need the values associated with the local map to retain their order for calculation purposes.
What would the equivalent of global[free] be on the local level?
the desired output would be something like this:
local_free = list(set(range(len(local)) - set(fixed))
local_values[local_free] *= 10
OUTPUT: local_values = [400, 40, 40, 400, 40]
I apologize if the question formatting is off, the code block formatting doesn't seem to be working in my browser, so please let me know if you need clarification.

For such comparison-related operations, NumPy has tools like np.setdiff1d and np.in1d among others. To solve our case, these two would be enough. I would assume that the inputs are NumPy arrays, as then we could use vectorized indexing methods supported by NumPy.
On the first case, we have -
In [97]: fixed = np.array([1,2,6])
...: global_arr = np.array([10, 1, 1, 10, 10, 10, 1, 10, 10, 10])
...:
To get the equivalent of list(set(range(len(global_arr)) - set(fixed)) in NumPy, we could make use of np.setdiff1d -
In [98]: np.setdiff1d(np.arange(len(global_arr)),fixed)
Out[98]: array([0, 3, 4, 5, 7, 8, 9])
Next up, we have -
In [99]: local_map = np.array([4, 2, 1, 8, 6])
...: local_values = np.array([42, 40, 48, 41, 43])
...:
We were trying to get -
local_free = list(set(range(len(local)) - set(fixed))
local_values[local_free] *= 10
Here, we can use np.in1d to get a mask to be an equivalent for local_free that could be used to index and assign into local_values with NumPy's boolean-indexing method -
In [100]: local_free = ~np.in1d(local_map,fixed)
...: local_values[local_free] *= 10
...:
In [101]: local_values
Out[101]: array([420, 40, 48, 410, 43])

Create larger list from an existing list using a list comprehension or map()

I am trying to generate a list to index coordinates (x, y and z), given a set of atom indices. My problem is quite simply how to elegantly go from this list:
atom_indices = [0, 4, 5, 8]
To this list:
coord_indices = [0, 1, 2, 12, 13, 14, 15, 16, 17, 24, 25, 26]
The easiest to read/understand way of doing this I've thought of so far is simply:
coord_indices = []
for atom in atom_indices:
coord_indices += [3 * atom,
3 * atom + 1,
3 * atom + 2]
But this doesn't seem very Pythonic. Is there a better way I haven't thought of without getting a list of lists or a list of tuples?

How about:
>>> atom_indices = [0, 4, 5, 8]
>>> coords = [3*a+k for a in atom_indices for k in range(3)]
>>> coords
[0, 1, 2, 12, 13, 14, 15, 16, 17, 24, 25, 26]
We can nest loops in list comprehensions in the same order we'd write the loops, i.e. this is basically
coords = []
for a in atom_indices:
for k in range(3):
coords.append(3*a+k)
Don't be afraid of for loops, though, if they're clearer in the situation. For reasons I've never fully understood, some people feel like they're being more clever when they write code horizontally instead of vertically, even though it makes it harder to debug.