We Keep Coding

Convert array to a single float in Python

I am trying to write a function which would estimate data noise (σ2) based on three NP arrays - One augmented X-matrix and the two vectors - the y-target and the MAP weights:
This function should return the empirical data noise estimate, σ2.
I have the following function:
def estimDS (X, output_y, W):
n = X.shape[0] # observations rows
d = X.shape[1] # number of features in columns
matmul = np.matmul(aug_x, ml_weights)
mult_left = (1/(n-d))
mult_right = (output_y-matmul)**2
estimDS = mult_left * mult_right
return estimDS
And this is an example on which I run function:
output_y = np.array([208500, 181500, 223500,
140000, 250000, 143000,
307000, 200000, 129900,
118000])
aug_x = np. array([[ 1., 1710., 2003.],
[ 1., 1262., 1976.],
[ 1., 1786., 2001.],
[ 1., 1717., 1915.],
[ 1., 2198., 2000.],
[ 1., 1362., 1993.],
[ 1., 1694., 2004.],
[ 1., 2090., 1973.],
[ 1., 1774., 1931.],
[ 1., 1077., 1939.]])
W = [-2.29223802e+06 5.92536529e+01 1.20780450e+03]
sig2 = estimDS(aug_x, output_y, W)
print(sig2)
Function returns an array, but I need to get this result as a float 3700666577282.7227
[5.61083809e+07 2.17473754e+07 6.81288433e+06 4.40198178e+07
1.86225354e+06 3.95549405e+08 8.78575426e+08 3.04530677e+07
3.32164594e+07 2.87861673e+06]

You forgot to sum over i=1 to n. Therefore mult_right should be defined as:
mult_right=np.sum((output_y-matmul)**2, axis=0)

how to generate functions in a for loop and use them for several inputs

I want to run a function in a for loop. Firstly, I have a list of arrays and each arry includes some constants. These constants go to function. Then, I make function and finally import arrays stored as a list into the created function. At the moment it is only using the constants stored in the last array of the list storing constants. I want to create the first function using the first array of constants and run that function for the first array of inps. I have checked this solution but I could not solve my issue.
constants=[np.array([2., 2., 2., 2.]),
np.array([20., 40., 30., 10.])]
inps=[np.array([[1., 1., 1.],[2., 2., 2.]]),
np.array([[3., 3., 3.],[4., 4., 4.]])]
This is my code:
def fun(row=i):
row[0] = a * row[0]
for i in constants:
i=i.tolist()
a = i[0]
return row[0], row[1], row[2]
out=[]
for j in inps:
j=[j]
new = np.asarray(list(map(fun, [x for point in j for x in point])))
out.append(new)
Then, I want to get:
out= [np.array([[2., 1., 1.],
[4., 2., 2.]]),
np.array([[60., 3. , 3. ],
[80., 4. , 4. ]])]
Simply, I want to multiply first value of the first array of constants to first column of first array of inps and replace it with the result. Then, I want to multiply the second of constants tothe second array of inps and so on.
But my code is creating only one function and performs the last function created by the cnstants coming from constants[lasti] for all the arrays of inps.
It is giving me the following result:
[array([[40., 1., 1.],
[80., 2., 2.]]),
array([[120., 3., 3.],
[160., 4., 4.]])]
In advance, I appreciate any help.

Not sure you need the function at all. This produces the output you are looking for:
import numpy as np
constants = [
np.array([2.0, 2.0, 2.0, 2.0]),
np.array([20.0, 40.0, 30.0, 10.0])]
inps = [
np.array([[1.0, 1.0, 1.0], [2.0, 2.0, 2.0]]),
np.array([[3.0, 3.0, 3.0], [4.0, 4.0, 4.0]]),
]
for index, inp in enumerate(inps):
inp[:,0] *= constants[index][0]
print(inps)
Output:
[array([[2., 1., 1.],
[4., 2., 2.]]),
array([[60., 3., 3.],
[80., 4., 4.]])]

Replace values in ndarray with condirion

I need to change values in ndarray: if > 0.5 then 1, less - then 0. Example:
array([ 0.11865984, 0.49666225, 0.134077 , 0.09502114, 0.70667432,
0.1648032 , 0.6356094 , 0.36054945, 0.68379407, 0.09297716])

You can try something like
import numpy as np
arr = np.array([ 0.11865984, 0.49666225, 0.134077 , 0.09502114, 0.70667432,
0.1648032 , 0.6356094 , 0.36054945, 0.68379407, 0.09297716])
bool_condition_array = arr > 0.5
binary_array = bool_condition_array.astype(int)

Evenly round to the given number of decimals. np.around
If you have an np.array you can use np.around function:
import numpy as np
arr = np.array([ 0.11865984, 0.49666225, 0.134077 , 0.09502114, 0.70667432,
0.1648032 , 0.6356094 , 0.36054945, 0.68379407, 0.09297716])
result = np.around(arr)
print(result)
# array([0., 0., 0., 0., 1., 0., 1., 0., 1., 0.])

How to append numpy.array to other numpy.array?

I want to create 2D numpy.array knowing at the begining only its shape, i.e shape=2. Now, I want to create in for loop ith one dimensional numpy.arrays, and add them to the main matrix of shape=2, so I'll get something like this:
matrix=
[numpy.array 1]
[numpy.array 2]
...
[numpy.array n]
How can I achieve that? I try to use:
matrix = np.empty(shape=2)
for i in np.arange(100):
array = np.zeros(random_value)
matrix = np.append(matrix, array)
But as a result of print(np.shape(matrix)), after loop, I get something like:
(some_number, )
How can I append each new array in the next row of the matrix? Thank you in advance.

I would suggest working with list
matrix = []
for i in range(10):
a = np.ones(2)
matrix.append(a)
matrix = np.array(matrix)
list does not have the downside of being copied in the memory everytime you use append. so you avoid the problem described by ali_m. at the end of your operation you just convert the list object into a numpy array.

I suspect the root of your problem is the meaning of 'shape' in np.empty(shape=2)
If I run a small version of your code
matrix = np.empty(shape=2)
for i in np.arange(3):
array = np.zeros(3)
matrix = np.append(matrix, array)
I get
array([ 9.57895902e-259, 1.51798693e-314, 0.00000000e+000,
0.00000000e+000, 0.00000000e+000, 0.00000000e+000,
0.00000000e+000, 0.00000000e+000, 0.00000000e+000,
0.00000000e+000, 0.00000000e+000])
See those 2 odd numbers at the start? Those are produced by np.empty(shape=2). That matrix starts as a (2,) shaped array, not an empty 2d array. append just adds sets of 3 zeros to that, resulting in a (11,) array.
Now if you started with a 2 array with the right number of columns, and did concatenate on the 1st dimension you would get a multirow array. (rows only have meaning in 2d or larger).
mat=np.zeros((1,3))
for i in range(1,3):
mat = np.concatenate([mat, np.ones((1,3))*i],axis=0)
produces:
array([[ 0., 0., 0.],
[ 1., 1., 1.],
[ 2., 2., 2.]])
A better way of doing an iterative construction like this is with list append
alist = []
for i in range(0,3):
alist.append(np.ones((1,3))*i)
mat=np.vstack(alist)
alist is:
[array([[ 0., 0., 0.]]), array([[ 1., 1., 1.]]), array([[ 2., 2., 2.]])]
mat is
array([[ 0., 0., 0.],
[ 1., 1., 1.],
[ 2., 2., 2.]])
With vstack you can get by with np.ones((3,), since it turns all of its inputs into 2d array.
append would work, but it also requires axis=0 parameter, and 2 arrays. It gets misused, often by mistaken analogy to the list append. It is just another front end to concatenate. So I prefer not to use it.
Notice that other posters assumed your random value changed during the iteration. That would produce a arrays of differing lengths. For 1d appending that would still produce the long 1d array. But a 2d append wouldn't work, because an 2d array can't be ragged.
mat = np.zeros((2,),int)
for i in range(4):
mat=np.append(mat,np.ones((i,),int)*i)
# array([0, 0, 1, 2, 2, 3, 3, 3])

The function you are looking for is np.vstack
Here is a modified version of your example
import numpy as np
matrix = np.empty(shape=2)
for i in np.arange(3):
array = np.zeros(2)
matrix = np.vstack((matrix, array))
The result is
array([[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]])

List as element of list of lists or multidimensional lists as a grid

I am trying to create a lat/lon grid that contains an array of found indices where two conditions are met for a lat/lon combination. This approach might be too complicated, but using a meshgrid or numpy broadcasting failed also. If there is a better approach, feel free to share your knowlegde. :-)
Round lat/lon values to gridsize resolution of 1° but retain full length of array:
x = np.around(lon, decimals=0)
y = np.around(lat, decimals=0)
arrays consists of longitude/latitude values from -180 to 180 and -82° to 82°; multiple douplets possible
Check for each combination of lat/lon how many measurements are available for 1°/1° grid point:
a = arange(-180,181)
b = arange(-82,83)
totalgrid = [ [ 0 for i in range(len(b)) ] for j in range(len(a)) ]
for d1 in range(len(a)):
for d2 in range(len(b)):
totalgrid[d1][d2]=np.where((x==a[d1])&(y==b[d2]))[0]
This method fails and returns only a list of lists with empty arrays. I can't figure out why it's not working properly.
Replacing the last line by:
totalgrid[d1][d2]=np.where((x==a[0])&(y==b[0]))[0]
returns all found indices from lon/lat that are present at -180°/-82°. Unfortunately it takes a while. Am I missing a for loop somewhere?!
The Problem in more detail:
#askewchan
Unfortunately this one does not solve my original problem.
As expected the result represents the groundtrack quite well.
But besides the fact that I need the total number of points for each grid point, I also need each single index of lat/lon combinations in the lat/lon array for further computations.
Let's assume I have an array
lat(100000L,), lon(100000L,) and a third one array(100000L,)
which corresponds to the measurement at each point. I need every index of all 1°/1° combinations in lat/lon, to check this index in the array(100000L,) if a condition is met. Now lets assume that the indices[10000,10001,10002,..,10025] of lat/lon are on the same gridpoint. For those indices I need to check whether array[10000,10001,10002,..,10025] now met a condition, i.e. np.where(array==0). With cts.nonzero() I only get the index in the histogram. But then all information of each point contributing to the value of the histogram is lost. Hopefully you get what was my initial problem.

Not sure if I understand the goal here, but you want to count how many lat/lon pairs you have in each 1° section? This is what a histogram does:
lon = np.random.random(5000)*2*180 - 180
lat = np.random.random(5000)*2*82 - 82
a = np.arange(-180,181)
b = np.arange(-82,83)
np.histogram2d(lon, lat, (a,b))
#(array([[ 0., 0., 1., ..., 0., 0., 0.],
# [ 0., 2., 0., ..., 0., 0., 1.],
# [ 0., 0., 0., ..., 0., 1., 0.],
# ...,
# [ 0., 1., 0., ..., 0., 0., 0.],
# [ 0., 0., 0., ..., 0., 0., 0.],
# [ 0., 0., 0., ..., 0., 0., 0.]]),
The indices where you have a nonzero count would be at:
cts.nonzero()
#(array([ 0, 0, 0, ..., 359, 359, 359]),
# array([ 2, 23, 25, ..., 126, 140, 155]))
You can plot it too:
cts, xs, ys = np.histogram2d(lon, lat, (a,b))
pyplot.imshow(cts, extent=(-82,82,-180,180))