Related
hi I wanna create a probability vector for my 2 Dimensional array. I wrote a function myself to iterate through the elements and calculate the probability of each value. When I only enter positive values everything works, but as soon as there is a negative number I create a negative probability, which shouldn't be possible as the value must be 0<=x<=1.
def createProbabilityVector(inputArray):
vector = inputArray
probabilityVector = np.zeros(vector.shape)
for x in range(vector.shape[0]):
vectorSum = sum(vector[x])
probabilityVector[[x]] = vector[[x]] / vectorSum
return probabilityVector
is the mistake in the code or do I simply fail to understand what I want to do?
edit: some examples
input
[[ 1.62242568 1.27356428 -1.88008155 1.37183247]
[-1.10638392 0.18420085 -1.68558966 -1.59951709]
[ 1.79166467 -0.21911691 -1.29066019 0.4565108 ]
[-0.20459109 1.59912774 0.47735207 1.6398782 ]]
output:
[[ 0.67948147 0.53337625 -0.78738927 0.57453155]
[ 0.26296832 -0.04378136 0.4006355 0.38017754]
[ 2.42642012 -0.2967462 -1.74791851 0.61824459]
[-0.05825873 0.45536272 0.13592931 0.4669667 ]]
-----
input
[[ 1.50162225 -0.31502279 -1.40281248 -1.09221922]
[ 1.93663826 1.31671237 -1.14334774 1.54792572]
[ 1.21376416 -1.44547074 0.0045907 1.4099986 ]
[ 0.51903455 -0.80046238 -1.69780354 -1.29893969]]
output:
[[-1.14764998 0.24076355 1.0721323 0.83475413]
[ 0.52943577 0.3599612 -0.31256699 0.42317002]
[ 1.02610693 -1.2219899 0.00388094 1.19200202]
[-0.15833053 0.24417956 0.51791182 0.39623914]]
-----
input
[[-1.6333837 -0.50469549 -1.62305585 -1.43558978]
[ 0.29636416 -0.22401163 -1.82816273 0.10676174]
[-1.6599302 -0.2516563 -1.64843802 -0.86857615]
[ 1.31762542 0.8690911 1.5888384 -1.83204102]]
output:
[[ 0.31431022 0.09711799 0.31232284 0.27624895]
[-0.17971828 0.13584296 1.10861674 -0.06474142]
[ 0.37482047 0.05682524 0.37222548 0.1961288 ]
[ 0.67796038 0.44717514 0.81750812 -0.94264364]]
-----
input
[[ 0.15369025 1.05426071 -0.61295255 0.95033555]
[ 0.04138761 -1.41072628 1.90319561 -1.2563338 ]
[ 1.85131197 -1.24551221 -1.62731374 0.43129381]
[ 0.21235188 1.21581691 -0.57470021 -0.58482563]]
output:
[[ 0.09945439 0.68222193 -0.3966473 0.61497099]
[-0.05728572 1.95262488 -2.63426518 1.73892602]
[-3.1366464 2.11025017 2.75713 -0.73073377]
[ 0.79046139 4.52577253 -2.13927148 -2.17696245]]
You need to transform all the values of the input array into positive values, a few alternatives are:
Convert all the negatives to 0, function zeroed
Shift all the values by the absolute value of the minimum element, function shifted
Apply the exponential function to the values, function exponential
After you have converted the values of the input array you can use your function as usual, follow the definition of the transformation functions:
def zeroed(arr):
return arr.clip(min=0)
def shifted(arr):
return arr + abs(np.min(arr))
def exponential(arr):
return np.exp(arr)
In your function you can use the transformation as follows:
def createProbabilityVector(inputArray):
vector = inputArray
probabilityVector = np.zeros(vector.shape)
for x in range(vector.shape[0]):
new_vector = zeroed(vector[x])
vectorSum = sum(new_vector)
probabilityVector[[x]] = new_vector / vectorSum
return probabilityVector
The function zeroed can be replace by shifted or exponential, for the input:
array = np.array([[1.62242568, 1.27356428, -1.88008155, 1.37183247],
[-1.10638392, 0.18420085, -1.68558966, -1.59951709],
[1.79166467, -0.21911691, -1.29066019, 0.4565108],
[-0.20459109, 1.59912774, 0.47735207, 1.6398782]])
These are the results for the function zeroed:
[[0.38015304 0.29841079 0. 0.32143616]
[0. 1. 0. 0. ]
[0.79694165 0. 0. 0.20305835]
[0. 0.43029432 0.1284462 0.44125948]]
for shifted:
[[0.35350056 0.31829072 0. 0.32820872]
[0.22847732 0.73756992 0. 0.03395275]
[0.52233595 0.18158552 0. 0.29607853]
[0. 0.41655061 0.15748787 0.42596152]]
and exponential:
[[0.39778013 0.28063027 0.01198184 0.30960776]
[0.17223667 0.62606504 0.09651165 0.10518664]
[0.69307072 0.09279107 0.03177905 0.18235916]
[0.06504215 0.39494808 0.12863496 0.41137482]]
I have a numpy array:
array([[ 0.68597575, 0.05544651, -1. ],
[ 0.33494648, 0.46368367, 1. ],
[ 0.42486765, 0.89427025, 1. ],
[ 0.62408611, 0.64633939, 1. ],
[ 0.37087957, 0.53077302, -1. ],
[ 0.21664159, 0.10786084, -1. ],
[ 0.13003626, 0.18425347, -1. ]])
I want the rows having last values -1 to be multiplied by -1 and also replaced in the actual matrix.
I tried this:
def transform(data):
for row in data:
if row[-1] == -1:
row = row * -1
but I know there would be something simpler than this.
You can avoid the for loop by doing:
data[data[:, -1] == -1] *= -1
i prefer to do this in two steps for two reasons: (i) easier to understand the code; and (ii) often the boolean index created in the first step can be reused elsewhere in the data pipeline
create the Boolean index that selects the rows:
idx = M[:,-1] == -1
do the transform on the indexed data:
M[idx,] *= -1
I have a 3d mesh with points and the locations of the points are
in an array that looks like this:
mesh_vectors = np.array([[-0.85758871, 0.8965745 , -0.1427767 ],
[-0.23945311, 1.00544977, 1.45797086],
[-0.57341832, -1.07448494, -0.11827722],
[ 0.05894491, -0.97208506, 1.47583127],
[-0.71402085, -0.08872638, -0.12916484],
[-0.09181146, 1.01235461, 0.47418442],
[-0.09025362, 0.01668115, 1.46690106],
[ 0.19773833, -0.95349348, 0.49089319],
[ 0.05055711, 0.02909645, 0.48503664]])
I have two indexing arrays:
idx1 = np.array([4 2 1 6 5 0 1 5])
idx2 = np.array([6 3 0 4 7 2 3 7])
these translations correspond to the index arrays:
translate_1 = np.array([[ 0.00323021 0.00047712 -0.00422925]
[ 0.00153422 0.00022654 -0.00203258]
[ 0.00273207 0.00039626 0.00038201]
[ 0.0052439 0.00075993 0.00068843]
[-0.00414245 -0.00053918 0.00543974]
[-0.00681844 -0.00084955 0.00894626]
[ 0. 0. 0. ]
[-0.00672519 -0.00099897 -0.00090189]])
translate_2 = np.array([[ 0.00523871 0.00079512 0.00068814]
[ 0.00251901 0.00038234 0.00033379]
[ 0.00169134 0.00021078 -0.00218737]
[ 0.00324106 0.00040338 -0.00422859]
[-0.00413547 -0.00058669 0.00544016]
[-0.00681223 -0.0008921 0.00894669]
[ 0. 0. 0. ]
[-0.00672553 -0.00099677 -0.00090191]])
they are currently added to the mesh like this:
mesh_vectors[idx1] += translate_1
mesh_vectors[idx2] += translate_2
The trouble is, what I really need to add isn't the translations
but the mean of the translations where multiple translations are
applied to the same mesh point. The indexing arrays can have indices occurring in a variety of different frequencies. Could be [2,2,2,3,4,5] and [1,2,1,1,5,4] though they will always be the same size. I'm trying to do this with numpy for speed but I have the options of using loops on start to generate indexing arrays if needed.
Thanks in advance!
This works:
scaled_tr1 = translate_1 / np.bincount(idx1)[idx1,None]
np.add.at(mesh_vectors, idx1, scaled_tr1)
Note that the use of np.add.at instead of fancy indexing is required:
ufunc.at(a, indices, b=None)
Performs unbuffered in place operation on operand a for elements specified by indices. For addition ufunc, this method is equivalent to a[indices] += b, except that results are accumulated for elements that are indexed more than once. For example, a[[0,0]] += 1 will only increment the first element once because of buffering, whereas add.at(a, [0,0], 1) will increment the first element twice.
I have a numpy array which has only a few non-zero entries which can be either positive or negative. E.g. something like this:
myArray = np.array([[ 0. , 0. , 0. ],
[ 0.32, -6.79, 0. ],
[ 0. , 0. , 0. ],
[ 0. , 1.5 , 0. ],
[ 0. , 0. , -1.71]])
In the end, I would like to receive a list where each entry of this list corresponds to a row of myArray and is a cumulative product of function outputs which depend on the entries of the respective row of myArray and another list (in the example below it is called l).
The individual terms depend on the sign of the myArray entry: When it is positive, I apply "funPos", when it is negative, I apply "funNeg" and if the entry is 0, the term will be 1. So in the example array from above it would be:
output = [1*1*1 ,
funPos(0.32, l[0])*funNeg(-6.79,l[1])*1,
1*1*1,
1*funPos(1.5, l[1])*1,
1*1*funNeg(-1.71, l[2])]
I implemented this as shown below and it gives me the desired output (note: that is just a highly simplified toy example; the actual matrices are far bigger and the functions more complicated). I go through each row of the array, if the sum of the row is 0, I don't have to do any calculations and the output is just 1. If it is not equal 0, I go through this row, check the sign of each value and apply the appropriate function.
import numpy as np
def doCalcOnArray(Array1, myList):
output = np.ones(Array1.shape[0]) #initialize output
for indRow,row in enumerate(Array1):
if sum(row) != 0: #only then calculations are needed
tempProd = 1. #initialize the product that corresponds to the row
for indCol, valCol in enumerate(row):
if valCol > 0:
tempVal = funPos(valCol, myList[indCol])
elif valCol < 0:
tempVal = funNeg(valCol, myList[indCol])
elif valCol == 0:
tempVal = 1
tempProd = tempProd*tempVal
output[indRow] = tempProd
return output
def funPos(val1,val2):
return val1*val2
def funNeg(val1,val2):
return val1*(val2+1)
myArray = np.array([[ 0. , 0. , 0. ],
[ 0.32, -6.79, 0. ],
[ 0. , 0. , 0. ],
[ 0. , 1.5 , 0. ],
[ 0. , 0. , -1.71]])
l = [1.1, 2., 3.4]
op = doCalcOnArray(myArray,l)
print op
The output is
[ 1. -7.17024 1. 3. -7.524 ]
which is the desired one.
My question is whether there is a more efficient way for doing that since that is quite "expensive" for large arrays.
EDIT:
I accepted gabhijit's answer because the pure numpy solution he came up with seems to be the fastest one for the arrays I am dealing with. Please note, that there is also a nice working solution from RaJa that requires panda and also the solution from dave works fine which can serve as a nice example on how to use generators and numpy's "apply_along_axis".
Here's what I have tried - using reduce, map. I am not sure how fast this is - but is this what you are trying to do?
Edit 4: Simplest and most readable - Make l a numpy array and then greatly simplifies where.
import numpy as np
import time
l = np.array([1.0, 2.0, 3.0])
def posFunc(x,y):
return x*y
def negFunc(x,y):
return x*(y+1)
def myFunc(x, y):
if x > 0:
return posFunc(x, y)
if x < 0:
return negFunc(x, y)
else:
return 1.0
myArray = np.array([
[ 0.,0.,0.],
[ 0.32, -6.79, 0.],
[ 0.,0.,0.],
[ 0.,1.5,0.],
[ 0.,0., -1.71]])
t1 = time.time()
a = np.array([reduce(lambda x, (y,z): x*myFunc(z,l[y]), enumerate(x), 1) for x in myArray])
t2 = time.time()
print (t2-t1)*1000000
print a
Basically let's just look at last line it says cumulatively multiply things in enumerate(xx), starting with 1 (last parameter to reduce). myFunc simply takes the element in myArray(row) and element # index row in l and multiplies them as needed.
My output is not same as yours - so I am not sure whether this is exactly what you want, but may be you can follow the logic.
Also I am not so sure how fast this will be for huge arrays.
edit: Following is a 'pure numpy way' to do this.
my = myArray # just for brevity
t1 = time.time()
# First set the positive and negative values
# complicated - [my.itemset((x,y), posFunc(my.item(x,y), l[y])) for (x,y) in zip(*np.where(my > 0))]
# changed to
my = np.where(my > 0, my*l, my)
# complicated - [my.itemset((x,y), negFunc(my.item(x,y), l[y])) for (x,y) in zip(*np.where(my < 0))]
# changed to
my = np.where(my < 0, my*(l+1), my)
# print my - commented out to time it.
# Now set the zeroes to 1.0s
my = np.where(my == 0.0, 1.0, my)
# print my - commented out to time it
a = np.prod(my, axis=1)
t2 = time.time()
print (t2-t1)*1000000
print a
Let me try to explain the zip(*np.where(my != 0)) part as best as I can. np.where simply returns two numpy arrays first array is an index of row, second array is an index of column that matches the condition (my != 0) in this case. We take a tuple of those indices and then use array.itemset and array.item, thankfully, column index is available for free to us, so we can just take the element # that index in the list l. This should be faster than previous (and by orders of magnitude readable!!). Need to timeit to find out whether it indeed is.
Edit 2: Don't have to call separately for positive and negative can be done with one call np.where(my != 0).
So, let's see if I understand your question.
You want to map elements of your matrix to a new matrix such that:
0 maps to 1
x>0 maps to funPos(x)
x<0 maps to funNeg(x)
You want to calculate the product of all elements in the rows this new matrix.
So, here's how I would go about doing it:
1:
def myFun(a):
if a==0:
return 1
if a>0:
return funPos(a)
if a<0:
return funNeg(a)
newFun = np.vectorize(myFun)
newArray = newFun(myArray)
And for 2:
np.prod(newArray, axis = 1)
Edit: To pass the index to funPos, funNeg, you can probably do something like this:
# Python 2.7
r,c = myArray.shape
ctr = -1 # I don't understand why this should be -1 instead of 0
def myFun(a):
global ctr
global c
ind = ctr % c
ctr += 1
if a==0:
return 1
if a>0:
return funPos(a,l[ind])
if a<0:
return funNeg(a,l[ind])
I think this numpy function would be helpful to you
numpy.apply_along_axis
Here is one implementation. Also I would warn against checking if the sum of the array is 0. Comparing floats to 0 can give unexpected behavior due to machine accuracy constraints. Also if you have -5 and 5 the sum is zero and I'm not sure thats what you want. I used numpy's any() function to see if anything was nonzero. For simplicity I also pulled your list (my_list) into global scope.
import numpy as np
my_list = 1.1, 2., 3.4
def func_pos(val1, val2):
return val1 * val2
def func_neg(val1, val2):
return val1 *(val2 + 1)
def my_generator(row):
for i, a in enumerate(row):
if a > 0:
yield func_pos(a, my_list[i])
elif a < 0:
yield func_neg(a, my_list[i])
else:
yield 1
def reduce_row(row):
if not row.any():
return 1.0
else:
return np.prod(np.fromiter(my_generator(row), dtype=float))
def main():
myArray = np.array([
[ 0. , 0. , 0. ],
[ 0.32, -6.79, 0. ],
[ 0. , 0. , 0. ],
[ 0. , 1.5 , 0. ],
[ 0. , 0. , -1.71]])
return np.apply_along_axis(reduce_row, axis=1, arr=myArray)
There are probably faster implmentations, I think apply_along_axis is really just a loop under the covers.
I didn't test, but I bet this is faster than what you started with, and should be more memory efficient.
I've tried your example with the masking function of numpy arrays. However, I couldn't find a solution to replace the values in your array by funPos or funNeg.
So my suggestion would be to try this using pandas instead as it conserves indices while masking.
See my example:
import numpy as np
import pandas as pd
def funPos(a, b):
return a * b
def funNeg(a, b):
return a * (b + 1)
myPosFunc = np.vectorize(funPos) #vectorized form of funPos
myNegFunc = np.vectorize(funNeg) #vectorized form of funNeg
#Input
I = [1.0, 2.0, 3.0]
x = pd.DataFrame([
[ 0.,0.,0.],
[ 0.32, -6.79, 0.],
[ 0.,0.,0.],
[ 0.,1.5,0.],
[ 0.,0., -1.71]])
b = pd.DataFrame(myPosFunc(x[x>0], I)) #calculate all positive values
c = pd.DataFrame(myNegFunc(x[x<0], I)) #calculate all negative values
b = b.combineMult(c) #put values of c in b
b = b.fillna(1) #replace all missing values that were '0' in the raw array
y = b.product() #multiply all elements in one row
#Output
print ('final result')
print (y)
print (y.tolist())
My code takes about two hours to process. The bottleneck is in for loop and if
statements (see comment in code).
I'm beginner with python :) Can anyone recommend an efficient python way to replace the nested for and if statements?
I have tables of ~30 million rows, each row with (x,y,z) values:
20.0 11.3 7
21.0 11.3 0
22.0 11.3 3
...
My desired output is a table in the form x, y, min(z), count(min(z)). The last
column is a final count of the least z values at that (x,y). Eg:
20.0 11.3 7 7
21.0 11.3 0 10
22.0 11.3 3 1
...
There's only about 600 unique coordinates, so the output table will be 600x4.
My code:
import numpy as np
file = open('input.txt','r');
coordset = set()
data = np.zeros((600,4))*np.nan
irow = 0
ctr = 0
for row in file:
item = row.split()
x = float(item[0])
y = float(item[1])
z = float(item[2])
# build unique grid of coords
if ((x,y)) not in coordset:
data[irow][0] = x
data[irow][1] = y
data[irow][2] = z
irow = irow + 1 # grows up to 599
# lookup table of unique coords
coordset.add((x,y))
# BOTTLENECK. replace ifs? for?
for i in range(0, irow):
if data[i][0]==x and data[i][1]==y:
if z > data[i][2]:
continue
elif z==data[i][2]:
ctr = ctr + 1
data[i][3]=ctr
if z < data[i][2]:
data[i][2] = z
ctr = 1
data[i][3]=ctr
edit: For reference the approach by #Joowani computes in 1m26s. My original approach, same computer, same datafile, 106m23s.
edit2: #Ophion and #Sibster thanks for suggestions, I don't have enough credit to +1 useful answers.
Your solution seems slow because it iterates through the list (i.e. data) every time you make an update. A better approach would be using a dictionary, which takes O(1) as opposed to O(n) per update.
Here would be my solution using a dictionary:
file = open('input.txt', 'r')
#coordinates
c = {}
for line in file:
#items
(x, y, z) = (float(n) for n in line.split())
if (x, y) not in c:
c[(x, y)] = [z, 1]
elif c[(x, y)][0] > z:
c[(x, y)][0], c[(x, y)][1] = z, 1
elif c[(x, y)][0] == z:
c[(x, y)][1] += 1
for key in c:
print("{} {} {} {}".format(key[0], key[1], c[key][0], c[key][1]))
Why not change the last if to an elif ?
Like it is done now you will evaluate the z < data[i][2]: every iteration of the loop.
You could even just replace it with an else since you have already checked if z>data[i][2] and z == data[i][2] so the only remaining possibility is z < data[i][2]:
So following code will do the same and should be faster :
if z > data[i][2]:
continue
elif z==data[i][2]:
ctr = ctr + 1
data[i][3]=ctr
else:
data[i][2] = z
ctr = 1
data[i][3]=ctr
To do this in numpy use np.unique.
def count_unique(arr):
row_view=np.ascontiguousarray(a).view(np.dtype((np.void,a.dtype.itemsize * a.shape[1])))
ua, uind = np.unique(row_view,return_inverse=True)
unique_rows = ua.view(a.dtype).reshape(ua.shape + (-1,))
count=np.bincount(uind)
return np.hstack((unique_rows,count[:,None]))
First lets check for a small array:
a=np.random.rand(10,3)
a=np.around(a,0)
print a
[[ 0. 0. 0.]
[ 0. 1. 1.]
[ 0. 1. 0.]
[ 1. 0. 0.]
[ 0. 1. 1.]
[ 1. 1. 0.]
[ 1. 0. 1.]
[ 1. 0. 1.]
[ 1. 0. 0.]
[ 0. 0. 0.]]
print output
[[ 0. 0. 0. 2.]
[ 0. 1. 0. 1.]
[ 0. 1. 1. 2.]
[ 1. 0. 0. 2.]
[ 1. 0. 1. 2.]
[ 1. 1. 0. 1.]]
print np.sum(output[:,-1])
10
Looks good! Now lets check for a large array:
a=np.random.rand(3E7,3)
a=np.around(a,1)
output=count_unique(a)
print output.shape
(1331, 4) #Close as I can get to 600 unique elements.
print np.sum(output[:,-1])
30000000.0
Takes about 33 second on my machine and 3GB of memory, doing this all in memory for large arrays will likely be your bottleneck. For a reference #Joowani's solution took about 130 seconds, although this is a bit of an apple and oranges comparison as we start with a numpy array. Your milage may vary.
To read in the data as a numpy array I would view the question here, but it should look something like the following:
arr=np.genfromtxt("./input.txt", delimiter=" ")
Loading in that much data from a txt file I would really recommend using the pandas example in that link.