Related
Assume I have a square raster of given size, and I want to "draw" (render) a circle (or ellipse) of given radius (or major / minor axes) and center.
One way of doing this in Python with NumPy is:
import numpy as np
def ellipse(box_size, semisizes, position=0.5, n_dim=2):
shape = (box_size,) * n_dim
if isinstance(semisizes, (int, float)):
semisizes = (semisizes,) * n_dim
position = ((box_size - 1) * position,) * n_dim
grid = [slice(-x0, dim - x0) for x0, dim in zip(position, shape)]
position = np.ogrid[grid]
arr = np.zeros(shape, dtype=float)
for x_i, semisize in zip(position, semisizes):
arr += (np.abs(x_i / semisize) ** 2)
return arr <= 1.0
print(ellipse(5, 2).astype(float))
# [[0. 0. 1. 0. 0.]
# [0. 1. 1. 1. 0.]
# [1. 1. 1. 1. 1.]
# [0. 1. 1. 1. 0.]
# [0. 0. 1. 0. 0.]]
which produces a rasterization without anti-aliasing.
In particular, the pixels that are only partially included in the circle get a value of 0 (similarly to pixels excluded from the circle), while pixels entirely included in the circle gets a value of 1.
With anti-aliasing, the pixels partially included in the circle would get a value between 0 and 1 depending on how much of their area is included in the circle.
How could I modify the code from above to (possibly cheaply) include anti-aliasing?
I am struggling to see how (if?) I could use the values of arr.
Super-sampling-based methods are out of question here.
Eventually, the result should look something like:
# [[0.0 0.2 1.0 0.2 0.0]
# [0.2 1.0 1.0 1.0 0.2]
# [1.0 1.0 1.0 1.0 1.0]
# [0.2 1.0 1.0 1.0 0.2]
# [0.0 0.2 1.0 0.2 0.0]]
(where 0.2 should be a value between 0.0 and 1.0 representing how much area of that specific pixel is covered by the circle).
EDIT
I see now obvious way on how to adapt the code from Creating anti-aliased circular mask efficiently although obviously, np.clip() must be part of the solution.
One fast but not necessarily mathematically correct way of doing this (loosely based on the code from Creating anti-aliased circular mask efficiently) is:
import numpy as np
def prod(items, start=1):
for item in items:
start *= item
return start
def ellipse(box_size, semisizes, position=0.5, n_dim=2, smoothing=1.0):
shape = (box_size,) * n_dim
if isinstance(semisizes, (int, float)):
semisizes = (semisizes,) * n_dim
position = ((box_size - 1) * position,) * n_dim
grid = [slice(-x0, dim - x0) for x0, dim in zip(position, shape)]
position = np.ogrid[grid]
arr = np.zeros(shape, dtype=float)
for x_i, semisize in zip(position, semisizes):
arr += (np.abs(x_i / semisize) ** 2)
if smoothing:
k = prod(semisizes) ** (0.5 / n_dim / smoothing)
return 1.0 - np.clip(arr - 1.0, 0.0, 1.0 / k) * k
elif isinstance(smoothing, float):
return (arr <= 1.0).astype(float)
else:
return arr <= 1.0
n = 1
print(np.round(ellipse(5 * n, 2 * n, smoothing=0.0), 2))
# [[0. 0. 1. 0. 0.]
# [0. 1. 1. 1. 0.]
# [1. 1. 1. 1. 1.]
# [0. 1. 1. 1. 0.]
# [0. 0. 1. 0. 0.]]
n = 1
print(np.round(ellipse(5 * n, 2 * n, smoothing=1.0), 2))
# [[0. 0.65 1. 0.65 0. ]
# [0.65 1. 1. 1. 0.65]
# [1. 1. 1. 1. 1. ]
# [0.65 1. 1. 1. 0.65]
# [0. 0.65 1. 0.65 0. ]]
A slightly more general version of this approach has been included in the raster_geometry Python package (Disclaimer: I am the main author of it).
Let us have a single event probability prob which is a scalar between 0-1. If I want to iterate over every possible probability with 0.1 increments, then I can use:
prob = np.arange(0.01, 1, 0.1)
Now assume I have 5 events (independent, probabilities sum to 1), each with probability p_i. I would like to have multi-dimensional probability arrays such as:
1.0 - 0.0 - 0.0 - 0.0 - 0.0
0.9 - 0.1 - 0.0 - 0.0 - 0.0
0.9 - 0.0 - 0.1 - 0.0 - 0.0
0.9 - 0.0 - 0.0 - 0.1 - 0.0
0.9 - 0.0 - 0.0 - 0.0 - 0.1
0.8 - 0.1 - 0.1 - 0.0 - 0.0
0.8 - 0.1 - 0.0 - 0.1 - 0.0
. . . . .
. . . . .
. . . . .
0.2 - 0.2 - 0.2 - 0.2 - 0.2
Is there a more clever way than to consider all the combinations of 0 - 0.1 - ... - 1 and delete the rows not summing up to 1? If yes, what is the easiest way?
You can use itertools.product and filter to create all combinations that sum 10 and pass it to an array:
import itertools
f = filter(lambda x: sum(x) == 10, itertools.product(*[range(11)]*5))
x = np.array(list(f)).astype(np.float)/10
x
>> array([[0. , 0. , 0. , 0. , 1. ],
[0. , 0. , 0. , 0.1, 0.9],
[0. , 0. , 0. , 0.2, 0.8],
...,
[0.9, 0. , 0.1, 0. , 0. ],
[0.9, 0.1, 0. , 0. , 0. ],
[1. , 0. , 0. , 0. , 0. ]])
EDIT
For the record, here's a more efficient way without using filtering. Essentially you create k bins (in your example, 10), and "assign" them to "n" samples (in your example, 3) in all possible combinations, using combinations_with_replacement
Then, you count how many bins each samples gets: this is your probability. This method is more complex to understand but avoids the filter, and thus it is much more efficient. You can try it with divisions of 0.01 (k = 100)
n = 3 # number of samples
k = 100 # number of subdivisions
f = itertools.combinations_with_replacement(range(3),k) #your iterator
r = np.array(list(f)) #your array of combinations
x = np.vstack((r==i).sum(1) for i in range(n)).T/k #your probability matrix
There's likely a more elegant solution using itertools but this is probably fine and uses no dependencies?:
for i in prob:
for j in prob:
for k in prob:
for l in prob:
m = 1 - i - j - l
if m>=0:
print(i,j,k,l,m)
I am trying to use numpy to dynamically create a set of zeros based on the size of a separate numpy array.
This is a small portion of the code of a much larger project. I have posted everything relevant in this question. I have a function k means which takes in a dataset (posted below) and a k value (which is 3, for this example).
I create a variable centroids which is supposed to look something like
[[4.9 3.1 1.5 0.1]
[7.2 3. 5.8 1.6]
[7.2 3.6 6.1 2.5]]
From there, I need to create a numpy array of "labels", one corresponding to every row in the dataset, of all zeroes with the same shape as the centroids array. Meaning, for a dataset with 5 rows, it would look like:
[[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]
This is what I am trying to achieve, albiet on a dynamic scale (i.e. where the # of rows and columns in the dataset are unknown).
The following (hard coded, non numpy) satisfies that (assuming there are 150 lines in the dataset:
def k_means(dataset, k):
centroids = [[5,3,2,4.5],[5,3,2,5],[2,2,2,2]]
cluster_labels = []
for i in range(0,150):
cluster_labels.append([0,0,0,0])
print (cluster_labels)
I am trying to do this dynamically with the following:
def k_means(dataset, k):
centroids = dataset[numpy.random.choice(dataset.shape[0], k, replace=False), :]
print(centroids)
cluster_labels = []
cluster_labels = numpy.asarray(cluster_labels)
for index in range(len(dataset)):
# temp_array = numpy.zeros_like(centroids)
# print(temp_array)
cluster_labels = cluster_labels.append(cluster_labels, numpy.zeros_like(centroids))
The current result is: AttributeError: 'numpy.ndarray' object has no attribute 'append'
Or, if I comment out the cluster_labels line and uncomment the temp, I get:
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]]
I will ultimately get 150 sets of that.
Sample of Iris Dataset:
5.1 3.5 1.4 0.2
4.9 3 1.4 0.2
4.7 3.2 1.3 0.2
4.6 3.1 1.5 0.2
5 3.6 1.4 0.2
5.4 3.9 1.7 0.4
4.6 3.4 1.4 0.3
5 3.4 1.5 0.2
4.4 2.9 1.4 0.2
4.9 3.1 1.5 0.1
5.4 3.7 1.5 0.2
4.8 3.4 1.6 0.2
4.8 3 1.4 0.1
4.3 3 1.1 0.1
5.8 4 1.2 0.2
5.7 4.4 1.5 0.4
5.4 3.9 1.3 0.4
5.1 3.5 1.4 0.3
5.7 3.8 1.7 0.3
5.1 3.8 1.5 0.3
5.4 3.4 1.7 0.2
5.1 3.7 1.5 0.4
4.6 3.6 1 0.2
5.1 3.3 1.7 0.5
4.8 3.4 1.9 0.2
5 3 1.6 0.2
5 3.4 1.6 0.4
5.2 3.5 1.5 0.2
5.2 3.4 1.4 0.2
4.7 3.2 1.6 0.2
4.8 3.1 1.6 0.2
5.4 3.4 1.5 0.4
5.2 4.1 1.5 0.1
5.5 4.2 1.4 0.2
Can anybody help me dynamically use numpy to achieve what I am aiming for?
Thanks.
shape of a numpy array is the size of the array. In a 2D array shape represents (number of rows, number of columns). So, shape[0] is the number of rows and shape[1] is the number of columns. You can use numpy.zeros((dataset.shape[0], centroids.shape[1])) to create a numpy array with your desired dimensions. Here is an example code with modified version of your k-means function.
import numpy
def k_means(dataset, k):
centroids = dataset[numpy.random.choice(dataset.shape[0], k, replace=False), :]
print(centroids)
cluster_labels = numpy.zeros((dataset.shape[0], centroids.shape[1]))
print(cluster_labels)
dataset = numpy.array([[1,2,3,4,5,6,7,8,9,0],
[3,4,5,6,4,3,2,2,6,7],
[4,4,5,6,7,7,8,9,9,0],
[5,6,7,8,5,3,3,2,2,1],
[6,3,3,2,2,4,5,6,6,8]])
k_means(dataset, 2)
Output:
[[1 2 3 4 5 6 7 8 9 0]
[5 6 7 8 5 3 3 2 2 1]]
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
I used numpy.zeros((dataset.shape[0], centroids.shape[1])) to make it more similar to your code. Actually, numpy.zeros(dataset.shape) would do the same thing, because centroids.shape[1] and dataset.shape[1] is the same. The number of columns of centroids and the number columns dataset are the same, because you choose your centroids from the dataset. So, the last version should be like:
def k_means(dataset, k):
centroids = dataset[numpy.random.choice(dataset.shape[0], k, replace=False), :]
cluster_labels = numpy.zeros(dataset.shape)
Given transactions between nodes in a (potentially large ~ 2+GBs) json file, with ~ million nodes and ~10 million transactions each having 10-1000 nodes such as
{"transactions":
[
{"transaction 1": ["node1","node2","node7"], "weight":0.41},
{"transaction 2": ["node4","node2","node1","node3","node10","node7","node9"], "weight":0.67},
{"transaction 3": ["node3","node10","node11","node2","node1"], "weight":0.33},...
]
}
what would the most elegant and efficient pythonic way to convert this into a node affinity matrix, where the affinities are the sum of weighted transactions between the nodes.
affinity [i,j] = weighted transaction count between nodes[i] and nodes[j] = affinity [j,i]
e.g.
affinity[node1, node7] = [0.41 (transaction1) + 0.67 (transaction2)] / 2 = affinity[node7, node1]
Note: the affinity matrix will be symmetrical and thus computing lower triangle alone will suffice.
Values not representative*** structure example only!
node1 | node2 | node3 | node4 | .... node1 1 .4 .1 .9 ... node2 .4 1 .6 .3 ... node3 .1 .6 1 .7 ... node4 .9 .3 .7
1 ......
First of all I would clean the data and represent each node with an integer and start with a dictionary like this
data=[{'transaction': [1, 2, 7], 'weight': 0.41},
{'transaction': [4, 2, 1, 3, 10, 7, 9], 'weight': 0.67},
{'transaction': [3, 10, 11, 2, 1], 'weight': 0.33}]
Not sure if this is pythonic enough but it should be self-explanatory
def weight(i,j,data_item):
return data_item["weight"] if i in data_item["transaction"] and j in data_item["transaction"] else 0
def affinity(i,j):
if j<i: # matrix is symmetric
return affinity(j,i)
else:
weights = [weight(i,j,data_item) for data_item in data if weight(i,j,data_item)!=0]
if len(weights)==0:
return 0
else:
return sum(weights) / float(len(weights))
ln = 10 # number of nodes
A = [[affinity(i,j) for j in range(1,ln+1)] for i in range(1,ln+1)]
To view the affinity matrix
import numpy as np
print(np.array(A))
[[ 0.47 0.47 0.5 0.67 0. 0. 0.54 0. 0.67 0.5 ]
[ 0.47 0.47 0.5 0.67 0. 0. 0.54 0. 0.67 0.5 ]
[ 0.5 0.5 0.5 0.67 0. 0. 0.67 0. 0.67 0.5 ]
[ 0.67 0.67 0.67 0.67 0. 0. 0.67 0. 0.67 0.67]
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. ]
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. ]
[ 0.54 0.54 0.67 0.67 0. 0. 0.54 0. 0.67 0.67]
[ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. ]
[ 0.67 0.67 0.67 0.67 0. 0. 0.67 0. 0.67 0.67]
[ 0.5 0.5 0.5 0.67 0. 0. 0.67 0. 0.67 0.5 ]]
I would like to know is there any better way to perform element wise division operator in python. The code below suppose to perform division A1 with B1 row and A2 with B2 rows therefore my expected output is only two rows. However the division part is A1 with B1, A1 with B2, A2 with B1 and A2 with B2. Can anyone help me?
The binary file is for A,C,G,T representations using 1000,0100,0010,0001.
Division file has four columns each each A, C, G, T and therefore the values obtained
earlier must divide accordingly.
Code
import numpy as np
from numpy import genfromtxt
import csv
csvfile = open('output.csv', 'wb')
writer = csv.writer(csvfile)
#open csv file into arrays
with open('binary.csv') as actg:
actg=actg.readlines()
with open('single.csv') as single:
single=single.readlines()
with open('division.csv') as division:
division=division.readlines()
# Converting binary line and single line into 3 rows and 4 columns
# binary values using reshape
for line in actg:
myarray = np.fromstring(line, dtype=float, sep=',')
myarray = myarray.reshape((-1, 3, 4))
for line2 in single:
single1 = np.fromstring(line2, dtype=float, sep=',')
single1 = single1.reshape((-1, 4))
# This division is in 2 rows and 4 column: first column
# represents 1000, 2nd-0100, 3rd-0010, 4th-0001 in the
# binary.csv. Therefore the division part where 1000's
# value should be divided by 1st column, 0010 should be
# divided by 3rd column value
for line1 in division:
division1 = np.fromstring(line1, dtype=float, sep=',')
m=np.asmatrix(division1)
m=np.array(m)
res2 = (single1[np.newaxis,:,:] / m[:,np.newaxis,:] * myarray).sum(axis=-1)
print(res2)
writer.writerow(res2)
csvfile.close()
binary.csv
0,1,0,0,1,0,0,0,0,0,0,1
0,0,1,0,1,0,0,0,1,0,0,0
single.csv:
0.28,0.22,0.23,0.27,0.12,0.29,0.34,0.21,0.44,0.56,0.51,0.65
division.csv
0.4,0.5,0.7,0.1
0.2,0.8,0.9,0.3
Expected output
0.44,0.3,6.5
0.26,0.6,2.2
Actual output
0.44,0.3,6.5
0.275,0.6,2.16666667
0.32857143,0.3,1.1
0.25555556,0.6,2.2
Explanation on the error
Let division file as follows:
A,B,C,D
E,F,G,H
Let after single and binary computation result as follows:
1,3,4
2,2,1
Let the number 1,2,3,4 is assigned to the location A,B,C,D and next row E,F,G,H
1/A,3/C,4/D
2/F,2/F,1/E
where 1 divided by A, 3 divided by C and so on. Basically this is what the code can do. Unfortunately the division part it happened to be like what described earlier. 221 operates with BBC and 134 operates with EGH therefore the output has 4 rows which is not what I want.
I don't know if this is what you are looking for, but here is a short way to get what (I think) you want:
import numpy as np
binary = np.genfromtxt('binary.csv', delimiter = ',').reshape((2, 3, 4))
single = np.genfromtxt('single.csv', delimiter = ',').reshape((1, 3, 4))
divisi = np.genfromtxt('division.csv', delimiter = ',').reshape((2, 1, 4))
print(np.sum(single / divisi * binary, axis = -1))
Output:
[[ 0.44 0.3 6.5 ]
[ 0.25555556 0.6 2.2 ]]
The output of your program looks kind of like this:
myarray
[ 0. 1. 0. 0. 1. 0. 0. 0. 0. 0. 0. 1.]
[[[ 0. 1. 0. 0.]
[ 1. 0. 0. 0.]
[ 0. 0. 0. 1.]]]
single1
[ 0.28 0.22 0.23 0.27 0.12 0.29 0.34 0.21 0.44 0.56 0.51 0.65]
[[ 0.28 0.22 0.23 0.27]
[ 0.12 0.29 0.34 0.21]
[ 0.44 0.56 0.51 0.65]]
division
[ 0.4 0.5 0.7 0.1]
m
[[ 0.4 0.5 0.7 0.1]]
res2
[[ 0.44 0.3 6.5 ]]
division
[ 0.2 0.8 0.9 0.3]
m
[[ 0.2 0.8 0.9 0.3]]
res2
[[ 0.275 0.6 2.16666667]]
myarray
[ 0. 0. 1. 0. 1. 0. 0. 0. 1. 0. 0. 0.]
[[[ 0. 0. 1. 0.]
[ 1. 0. 0. 0.]
[ 1. 0. 0. 0.]]]
single1
[ 0.28 0.22 0.23 0.27 0.12 0.29 0.34 0.21 0.44 0.56 0.51 0.65]
[[ 0.28 0.22 0.23 0.27]
[ 0.12 0.29 0.34 0.21]
[ 0.44 0.56 0.51 0.65]]
division
[ 0.4 0.5 0.7 0.1]
m
[[ 0.4 0.5 0.7 0.1]]
res2
[[ 0.32857143 0.3 1.1 ]]
division
[ 0.2 0.8 0.9 0.3]
m
[[ 0.2 0.8 0.9 0.3]]
res2
[[ 0.25555556 0.6 2.2 ]]
So, with that in mind, it looks like your last two lines of the output, the one's you did not expect are caused by the second line in binary.csv. So don't use that line in your calculations if you don't want 4 line in your result.