So, this is more like a structural problem but I think it's looking fairy ugly at the moment, I have code looking like:
for i in range(length_of_tree):
potential_ways = np.zeros((M, 2))
for m in range(omega):
for s in range(Z):
potential_ways[m][s] = sum([quad[r][m][s] for r in range(reps)])
The code is currently working, but I've noticed that there are several ways using numpy to avoid for-loops, my question is therefore, is there a way for me to make this code a bit more minimalistic?
A sum over values in an array can always be changed into an inner product which is optimised in numpy. As has been suggested here, I don't really understand the context of your question without examples but you should be able to do something like the following:
np.random.seed(1)
# your examples
M = 2
length_of_tree,reps = 100,100
omega,Z = 2,2
# a random matrix of values of shape 100,2,2
quad = np.random.normal(0,1,size=(100,2,2))
# useful initializations
quadT = quad.T
dummy = np.ones(shape=(100,))
for i in range(length_of_tree):
# option 1
potential_ways = np.zeros((M, 2))
for m in range(omega):
for s in range(Z):
potential_ways[m][s] = sum([quad[r][m][s] for r in range(reps)])
# option 2
potential_ways = quadT.dot(dummy).T
Related
I'm new to Numpy and already read a few things about it. Repeatedly it says, that you should get rid of loops when using Numpy, because otherwise you have some Python-overhead which slows down your code. So for practicing I tried to implement a simple algorithm in a "numpythonic" way, but I can't manage to get rid of the for loop. Is there a way to improve this solution?
My main problem is, that I have some kind of "cumulative-conditional" situation and I have no idea how I can solve this without a loop.
import numpy as np
def process(data):
r = np.zeros(3)
for d in data:
ru = r / np.linalg.norm(r)
r = np.where(
np.dot(ru, d) < 0.0,
r + np.negative(d),
r + d
)
return r
data = [
[0.99558784, 0.03476669, -0.08715574],
[0.99194152, 0.1217951, -0.0348995],
[0.9864998, 0.08630755, -0.1391731]
]
print(process(data))
# Out: [ 2.97402916 0.24286934 -0.26122834]
(Besides of my main problem I'm open for general criticism or improvement suggestions of course)
A few comments:
In first loop, your ru = ... statement produces warning - it divides by 0.
In this case, np.dot() returns a single float, not an array. Hence, there is no point for using np.where(); if statement would be faster.
For the data you provided, your function is equivalent to np.sum(data, axis=0).
I have a set of 4 simultaneous equations:
0.059z = x
0.06w = y
z+w = 8093
x+y = 422
All the solutions I've found so far seem to be for equations that have all the variables present in each equation, then convert to matrices and use the solve function.
Is there an easier way to do this in R or Python using the equations in their original form?
Also, how can I ensure that only positive numbers are returned in the solution?
Hope this makes sense...many thanks for the help
You can use sympy for this:
from sympy import symbols, linsolve, Eq
x,y,z,w = symbols('x y z w')
linsolve([Eq(0.059*z, x), Eq(0.06*w, y), Eq(z+w, 8093), Eq(x+y, 422)], (x, y, z, w))
Output:
Regarding your comments about negative values - there is only one solution to the system of equations, and it has negative values for y and w. If there was more than one solution, sympy would return them, and you could filter the solutions from there to only positive values.
In R, maybe you try it like below:
library(rootSolve)
library(zeallot)
model <- function(v){
c(x,y,z,w) %<-% v
return(c(0.059*z-x, 0.06*w-y, z+w-8093, x+y-422))
}
res <- multiroot(f = model, start = c(0,0,0,0))
then you can get the solution res as
> res
[1] 3751.22 -3329.22 63580.00 -55487.00
there are a few things going on here. first as CDJB notes: if there were any positive solutions then sympy would find them. I searched for those numbers and found this paper which suggests you should be using 7088 instead of 8093. we can do a quick sanity check:
def pct(value):
return f"{value:.1%}"
print(pct(422 / 8093)) # ~5.2%
print(pct(422 / 7088)) # ~6.0%
confirming that you're going to struggle averaging ~5.9% and ~6.0% towards ~5.2%, and explaining the negative solutions in the other answers. further, these are presumably counts so all your variables also need to be whole numbers.
once this correct denominator is used, I'd comment that there are many solutions (11645 by my count) e.g:
cases = [1, 421]
pop = [17, 7071]
rates = [pct(c / p) for c, p in zip(cases, pop)]
gives the appropriate output, as does:
cases = [2, 420]
pop = [34, 7054]
this is because the data was rounded to two decimal places. you probably also don't want to use either of the above, they're just the first two valid solutions I got.
we can define a Python function to enumerate all solutions:
from math import floor, ceil
def solutions(pop, cases, rate1, rate2, err):
target = (pct(rate1), pct(rate2))
for pop1 in range(1, pop):
pop2 = pop - pop1
c1_lo = ceil(pop1 * (rate1 - err))
c1_hi = floor(pop1 * (rate1 + err))
for c1 in range(c1_lo, c1_hi+1):
c2 = cases - c1
if (pct(c1 / pop1), pct(c2 / pop2)) == target:
yield c1, c2, pop1, pop2
all_sols = list(solutions(7088, 422, 0.059, 0.060, 0.0005))
which is where I got my count of 11645 above from.
not sure what to suggest with this, but you could maybe do a bootstrap to see how much your statistic varies with different solutions. another option would be to do a Bayesian analysis which would let you put priors over the population sizes and hence cut this down a lot.
I need to store a solution of expensive FEM calculation to use it in further analysis. Browsing through tutorials I have so far discovered, that I may store my results like this:
from fenics import *
mesh = Mesh('mesh/UnitSquare8x8.xml')
V = FunctionSpace(mesh, 'P', 1)
u = TrialFunction(V)
v = TestFunction(V)
f = Constant(-6.0)
a = dot(grad(u), grad(v))*dx
L = f*v*dx
u_D = Expression('1 + x[0]*x[0] + 2*x[1]*x[1]', degree=2)
def boundary(x, on_boundary):
return on_boundary
bc = DirichletBC(V, u_D, boundary)
A = assemble(a)
b = assemble(L)
bc.apply(A, b)
u = Function(V)
solver = KrylovSolver("cg", "ilu")
solver.solve(A, u.vector(), b)
File('solution.xml') << u.vector()
and later load them like this:
from fenics import *
mesh = Mesh('mesh/UnitSquare8x8.xml')
V = FunctionSpace(mesh, 'P', 1)
u = Function(V)
File('solution.xml') >> u.vector()
Unfortunately I hardly know what exactly I am doing here. Is that a proper way of storing and loading calculated results? Is order of elements in u.vector() (for the same mesh file) fixed within/between different FEniCS versions, or it is just an implementation detail which may change any time? If it is unsafe, then what is the proper way of doing so?
I have found another (possibly even more dangerous) solution. I may use VALUES = u.vector().get_local() and u.vector().set_local(VALUES) methods, as VALUES is a numpy array which I may easily store and load.
No, according to answer to Is the order of Vector elements preserved between runs? the order of Vector elements is not guarranted to be preserved.
It is recommended to use XDMFFile.write_checkpoint() and XDMFFile.read_checkpoint() methods instead.
I have written a function for covariance matrix and the output I am getting is correct but the problem with the code is, it's taking too much time for high dimension dataset.
Could you please help me to modify the code below to take less time for output?
def cov_variance(norm_data,mean_of_mat):
col = len(norm_data)
row = len(norm_data[0])
out =[]
i = 0
sum_of_covar = 0
freezrow = 0
flag = 1
while flag<=len(mean_of_mat):
for r in range(row):
for c in range(col):
sum_of_covar+=(((norm_data[c][freezrow])-mean_of_mat[freezrow])*\
((norm_data[c][r])-mean_of_mat[i]))
freezrow=freezrow
out.append(sum_of_covar)
i+=1
sum_of_covar=0
freezrow=freezrow
flag+=1
freezrow+=1
i=0
out1 = map(lambda x : x/col-1,out)
cov_variance_output = reshape(out1,row)
return cov_variance_output
Like doctorlove already said, don't implement your own. It will almost certainly be slower and/or less versatile (speaking from my own experience).
I tried commenting with this info but my rep is too low. You can find information on calculating covariance matrices with numpy here: https://docs.scipy.org/doc/numpy/reference/generated/numpy.cov.html
I'm relatively new to python and am wondering how to make the following more efficient by avoiding explicit nested 'for' loops and using python's implicit looping instead. I'm working with image data, and in this case trying to speed up my k-means algorithm. Here's a sample of what I'm trying to do:
# shape of image will be something like 140, 150, 3
num_sets, rows_per_set, num_columns = image_values.shape
for set in range(0, num_sets):
for row in range(0, rows_per_set):
pos = np.argmin(calc_euclidean(rgb_[set][row], means_list)
buckets[pos].append(image_values[set][row])
What I have today works great but I'd like to make it more efficient.
Feedback and recommendations are greatly appreciated.
Here is a vectorised solution. I'm almost certain I got your dimensions muddled up (3 is not really the number of columns, is it?), but the principle should be recognisable anyway:
For demonstration I only collect the (flat) indices into set and row in the buckets.
import numpy as np
k = 6
rgb_=np.random.randint(0, 9, (140, 150, 3))
means_list = np.random.randint(0, 9, (k, 3))
# compute distance table; use some algebra to leverage highly optimised
# dot product
squared_dists = np.add.outer((rgb_*rgb_).sum(axis=-1),
(means_list*means_list).sum(axis=-1)) \
- 2*np.dot(rgb_, means_list.T)
# find best cluster
best = np.argmin(squared_dists, axis=-1)
# find group sizes
counts = np.bincount(best.ravel())
# translate to block boundaries
bnds = np.cumsum(counts[:-1])
# group indices by best cluster; argpartition should be
# a bit cheaper than argsort
chunks = np.argpartition(best.ravel(), bnds)
# split into buckets
buckets = np.split(chunks, bnds)
# check
num_sets, rows_per_set, num_columns = rgb_.shape
def calc_euclidean(a, b):
return ((a-b)**2).sum(axis=-1)
for set in range(0, num_sets):
for row in range(0, rows_per_set):
pos = np.argmin(calc_euclidean(rgb_[set][row], means_list))
assert pos == best[set, row]
assert rows_per_set*set+row in buckets[pos]