Related
This is what I saw in a blog:
(Source: https://deepnotes.io/softmax-crossentropy#cross-entropy-loss)
def delta_cross_entropy(X,y):
"""
X is the output from fully connected layer (num_examples x num_classes)
y is labels (num_examples x 1)
Note that y is not one-hot encoded vector.
It can be computed as y.argmax(axis=1) from one-hot encoded vectors of labels if required.
"""
m = y.shape[0]
grad = softmax(X)
grad[range(m),y] -= 1
grad = grad/m
return grad
Also I try it myself by replacing y(matrix) using a integer number:
import numpy as np
arr = np.array([[1,2,3], [4,5,6], [7,8,9]])
print(arr)
arr[range(2), 1] = 0
print("after range: ", arr)
As result I got:
[[1 2 3]
[4 5 6]
[7 8 9]]
after range: [[1 0 3]
[4 0 6]
[7 8 9]]
Can someone pls explain what does this range() function do here? thank you!
range(2) is like passing [0, 1] as an argument in this case. You'll get the same result in this experiment:
arr1 = np.array([[1,2,3], [4,5,6], [7,8,9]])
arr1[range(2), 1] = 0
arr2 = np.array([[1,2,3], [4,5,6], [7,8,9]])
arr2[[0, 1], 1] = 0
np.all(arr1 == arr2)
Where np.all checks if the equality is true across the whole array and it will return True
In other words arr[range(2), 1] = 0 is equal to this as well:
arr[0, 1] = 0
arr[1, 1] = 0
And another way to look at it:
for i in range(2):
arr[i, 1] = 0
Range just creates a new range object from 0 to 1 less than the value given, so for example:
>>> range(2)
[0, 1]
Numpy arrays can accept iterables of indices to index so in this case you are saying "set the {0,1} row, 1st column equal to 0".
I have a numpy array of numpy arrays like the following example:
data = [[0.4, 1.5, 2.6],
[3.4, 0.2, 0.0],
[null, 3.2, 1.0],
[1.0, 4.6, null]]
I would like an efficient way of returning the row index, column index and value if the value meets a condition.
I need the row and column values because I feed them into func_which_returns_lat_long_based_on_row_and_column(column, row) which is applied if the value meets a condition.
Finally I would like to append the value, and outputs of the function to my_list.
I have solved my problem with the nested for loop solution shown below but it is slow. I believe I should be using np.where() however I cannot figure that out.
my_list = []
for ii, array in enumerate(data):
for jj, value in enumerate(array):
if value > 1:
lon , lat = func_which_returns_lat_long_based_on_row_and_column(jj,ii)
my_list.append([value, lon, lat])
I'm hoping there is a more efficient solution than the one I'm using above.
import numpy as np
import warnings
warnings.filterwarnings('ignore')
data = [[0.4, 1.5, 2.6],
[3.4, 0.2, 0.0],
[np.nan, 3.2, 1.0],
[1.0, 4.6, np.nan]]
x = np.array(data)
i, j = np.where(x > 1 )
for a, b in zip(i, j):
print('lon: {} lat: {} value: {}'.format(a, b, x[a,b]))
Output is
lon: 0 lat: 1 value: 1.5
lon: 0 lat: 2 value: 2.6
lon: 1 lat: 0 value: 3.4
lon: 2 lat: 1 value: 3.2
lon: 3 lat: 1 value: 4.6
As there is np.nan in comparison, there will be RuntimeWarning.
you can use
result = np.where(arr == 15)
it will return a np array of indices where element is in arr
try to build a function that works on arrays. For instance a function that adds to every element of the data the corresonding column and row index could look like:
import numpy as np
def func_which_returns_lat_long_based_on_row_and_column(data,indices):
# returns element of data + columna and row index
return data + indices[:,:,0] + indices[:,:,1]
data = np.array([[0.4, 1.5, 2.6],
[3.4, 0.2, 0.0],
[np.NaN, 3.2, 1.0],
[1.0, 4.6, np.NaN]])
# create a matrix of the same shape as data (plus an additional dim because they are two indices)
# with the corresponding indices of the element in it
x_range = np.arange(0,data.shape[0])
y_range = np.arange(0,data.shape[1])
grid = np.meshgrid(x_range,y_range, indexing = 'ij')
indice_matrix = np.concatenate((grid[0][:,:,None],grid[1][:,:,None]),axis=2)
# for instance:
# indice_matrix[0,0] = np.array([0,0])
# indice_matrix[1,0] = np.array([1,0])
# indice_matrix[1,3] = np.array([1,3])
# calculate the output
out = func_which_returns_lat_long_based_on_row_and_column(data,indice_matrix)
data.shape
>> (4,3)
indice_matrix.shape
>> (4, 3, 2)
indice_matrix
>>> array([[[0, 0],
[0, 1],
[0, 2]],
[[1, 0],
[1, 1],
[1, 2]],
[[2, 0],
[2, 1],
[2, 2]],
[[3, 0],
[3, 1],
[3, 2]]])
indice_matrix[2,1]
>> array([2, 1])
I am not sure what the title of this question should be. But lets say we have 2 arrays, values and distances.
values = np.array([[-1,-1,-1],
[1, 2, 0],
[-1,-1,-1]])
distances = np.array([[1,2,3],
[6,5,4],
[7,8,9]])
I would like to get the values that are non negative, and have them in order by its corresponding distance, based on the distances array.
So with the example above, the positive values are [1,2,0] and its distances will be [6,5,4]. Thus, if sorting by its corresponding distance, I would like to have [0,2,1] as the answer.
My code is below. It works, but would like to have the solution of just using numpy. Im sure that would be more efficient than this:
import numpy as np
import heapq
def get_sorted_values(seek_val, values, distances):
r, c = np.where(values >= seek_val)
di = distances[r, c]
vals = values[r, c]
print("di", di)
print("vals", vals)
if len(di) >= 1:
heap = []
for d, v in zip(di,vals):
heapq.heappush(heap, (d,v))
lists = []
while heap:
d, v = heapq.heappop(heap)
lists.append(v)
return lists
else:
## NOTHING FOUND
return None
Input:
seek_val = 0
values = np.array([[-1,-1,-1],
[1,2,0],
[-1,-1,-1]])
distances = np.array([[1,2,3],
[6,5,4],
[7,8,9]])
print("Ans:",get_sorted_values(seek_val, values, distances))
Output:
di [6 5 4]
vals [1 2 0]
Ans: [0, 2, 1]
"one liner":
values[np.where(values >= 0)][np.argsort(distances[np.where(values >= 0)])]
Out[981]: array([0, 2, 1])
repeating np.where(values >= 0) is inefficient, could make a variable if values is big
v_indx = np.where(values >= 0)
values[v_indx][np.argsort(distances[v_indx])]
Try np.argsort
import numpy as np
values = np.array([[-1,-1,-1],
[ 1, 2, 0],
[-1,-1,-1]])
distances = np.array([[1, 2, 3],
[6, 5, 4],
[7, 8, 9]])
print(values[values >= 0])
# [1 2 0]
print(distances[values >= 0])
# [6 5 4]
print('Ans:', values[values >= 0][np.argsort(distances[values >= 0])])
# Ans: [0 2 1]
Simple problem, but I cannot seem to get it to work. I want to calculate the percentage a number occurs in a list of arrays and output this percentage accordingly.
I have a list of arrays which looks like this:
import numpy as np
# Create some data
listvalues = []
arr1 = np.array([0, 0, 2])
arr2 = np.array([1, 1, 2, 2])
arr3 = np.array([0, 2, 2])
listvalues.append(arr1)
listvalues.append(arr2)
listvalues.append(arr3)
listvalues
>[array([0, 0, 2]), array([1, 1, 2, 2]), array([0, 2, 2])]
Now I count the occurrences using collections, which returns a a list of collections.Counter:
import collections
counter = []
for i in xrange(len(listvalues)):
counter.append(collections.Counter(listvalues[i]))
counter
>[Counter({0: 2, 2: 1}), Counter({1: 2, 2: 2}), Counter({0: 1, 2: 2})]
The result I am looking for is an array with 3 columns, representing the value 0 to 2 and len(listvalues) of rows. Each cell should be filled with the percentage of that value occurring in the array:
# Result
66.66 0 33.33
0 50 50
33.33 0 66.66
So 0 occurs 66.66% in array 1, 0% in array 2 and 33.33% in array 3, and so on..
What would be the best way to achieve this?
Many thanks!
Here's an approach -
# Get lengths of each element in input list
lens = np.array([len(item) for item in listvalues])
# Form group ID array to ID elements in flattened listvalues
ID_arr = np.repeat(np.arange(len(lens)),lens)
# Extract all values & considering each row as an indexing perform counting
vals = np.concatenate(listvalues)
out_shp = [ID_arr.max()+1,vals.max()+1]
counts = np.bincount(ID_arr*out_shp[1] + vals)
# Finally get the percentages with dividing by group counts
out = 100*np.true_divide(counts.reshape(out_shp),lens[:,None])
Sample run with an additional fourth array in input list -
In [316]: listvalues
Out[316]: [array([0, 0, 2]),array([1, 1, 2, 2]),array([0, 2, 2]),array([4, 0, 1])]
In [317]: print out
[[ 66.66666667 0. 33.33333333 0. 0. ]
[ 0. 50. 50. 0. 0. ]
[ 33.33333333 0. 66.66666667 0. 0. ]
[ 33.33333333 33.33333333 0. 0. 33.33333333]]
The numpy_indexed package has a utility function for this, called count_table, which can be used to solve your problem efficiently as such:
import numpy_indexed as npi
arrs = [arr1, arr2, arr3]
idx = [np.ones(len(a))*i for i, a in enumerate(arrs)]
(rows, cols), table = npi.count_table(np.concatenate(idx), np.concatenate(arrs))
table = table / table.sum(axis=1, keepdims=True)
print(table * 100)
You can get a list of all values and then simply iterate over the individual arrays to get the percentages:
values = set([y for row in listvalues for y in row])
print [[(a==x).sum()*100.0/len(a) for x in values] for a in listvalues]
You can create a list with the percentages with the following code :
percentage_list = [((counter[i].get(j) if counter[i].get(j) else 0)*10000)//len(listvalues[i])/100.0 for i in range(len(listvalues)) for j in range(3)]
After that, create a np array from that list :
results = np.array(percentage_list)
Reshape it so we have the good result :
results = results.reshape(3,3)
This should allow you to get what you wanted.
This is most likely not efficient, and not the best way to do this, but it has the merit of working.
Do not hesitate if you have any question.
I would like to use functional-paradigm to resolve this problem. For example:
>>> import numpy as np
>>> import pprint
>>>
>>> arr1 = np.array([0, 0, 2])
>>> arr2 = np.array([1, 1, 2, 2])
>>> arr3 = np.array([0, 2, 2])
>>>
>>> arrays = (arr1, arr2, arr3)
>>>
>>> u = np.unique(np.hstack(arrays))
>>>
>>> result = [[1.0 * c.get(uk, 0) / l
... for l, c in ((len(arr), dict(zip(*np.unique(arr, return_counts=True))))
... for arr in arrays)] for uk in u]
>>>
>>> pprint.pprint(result)
[[0.6666666666666666, 0.0, 0.3333333333333333],
[0.0, 0.5, 0.0],
[0.3333333333333333, 0.5, 0.6666666666666666]]
I am trying to convert a MATLAB code in Python. I don't know how to initialize empty matrix in Python.
MATLAB Code:
demod4(1) = [];
I tried in Python
demod4[0] = array([])
but it gives error:
only length-1 arrays can be converted to Python scalars
If you are using numpy arrays, you initialize to 0, by specifying the expected matrix size:
import numpy as np
d = np.zeros((2,3))
>>> d
[[ 0. 0. 0.]
[ 0. 0. 0.]]
This would be the equivalent of MATLAB 's:
d = zeros(2,3);
You can also initialize an empty array, again using the expected dimensions/size
d = np.empty((2,3))
If you are not using numpy, the closest somewhat equivalent to MATLAB's d = [] (i.e., a zero-size matrix) would be using an empty list and then
append values (for filling a vector)
d = []
d.append(0)
d.append(1)
>>> d
[0, 1]
or append lists (for filling a matrix row or column):
d = []
d.append(range(0,2))
d.append(range(2,4))
>>> d
[[0, 1], [2, 3]]
See also:
initialize a numpy array (SO)
NumPy array initialization (fill with identical values) (SO)
How do I create an empty array and then append to it in NumPy? (SO)
NumPy for MATLAB users
You could use a nested list comprehension:
# size of matrix n x m
matrix = [ [ 0 for i in range(n) ] for j in range(m) ]
What about initializing a list, populating it, then converting to an array.
demod4 = []
Or, you could just populate at initialization using a list comprehension
demod4 = [[func(i, j) for j in range(M)] for i in range(N)]
Or, you could initialize an array of all zeros if you know the size of the array ahead of time.
demod4 = [[0 for j in range(M)] for i in range(N)]
or
demod4 = [[0 for i in range(M)]*N]
Or try using numpy.
import numpy as np
N, M = 100, 5000
np.zeros((N, M))
To init matrix with M rows and N columns you can use following pattern:
M = 3
N = 2
matrix = [[0] * N for _ in range(M)]
If you want to initialize the matrix with 0s then use the below code
# for m*n matrix
matrix = [[0] * m for i in range(n)]
M=[]
n=int(input())
m=int(input())
for j in range(n):
l=[]
for k in range(m):
l.append(0)
M.append(l)
print(M)
This is the traditional way of doing it matrix[m,n], However, python offers many cool ways of doing so as mentioned in other answers.
i find that this method is the easies method to create a matrix
rows = 3
columns = 4
matrix = [[0] * columns] * rows
my output:
[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
if you want to print the matrix use this:
for i in range(rows):
for j in range(columns):
print(matrix[i][j], end=' ')
print()
my output:
0 0 0 0
0 0 0 0
0 0 0 0
rows = 3
columns = 2
M = [[0]*columns]*rows
Or you could also use '' instead of 0
print(M)
Output:
M = [[0, 0], [0, 0], [0, 0]]