For a side project in my PhD, I engaged in the task of modelling some system in Python. Efficiency wise, my program hits a bottleneck in the following problem, which I'll expose in a Minimal Working Example.
I deal with a large number of segments encoded by their 3D beginning and endpoints, so each segment is represented by 6 scalars.
I need to calculate a pairwise minimal intersegment distance. The analytical expression of the minimal distance between two segments is found in this source. To the MWE:
import numpy as np
N_segments = 1000
List_of_segments = np.random.rand(N_segments, 6)
Pairwise_minimal_distance_matrix = np.zeros( (N_segments,N_segments) )
for i in range(N_segments):
for j in range(i+1,N_segments):
p0 = List_of_segments[i,0:3] #beginning point of segment i
p1 = List_of_segments[i,3:6] #end point of segment i
q0 = List_of_segments[j,0:3] #beginning point of segment j
q1 = List_of_segments[j,3:6] #end point of segment j
#for readability, some definitions
a = np.dot( p1-p0, p1-p0)
b = np.dot( p1-p0, q1-q0)
c = np.dot( q1-q0, q1-q0)
d = np.dot( p1-p0, p0-q0)
e = np.dot( q1-q0, p0-q0)
s = (b*e-c*d)/(a*c-b*b)
t = (a*e-b*d)/(a*c-b*b)
#the minimal distance between segment i and j
Pairwise_minimal_distance_matrix[i,j] = sqrt(sum( (p0+(p1-p0)*s-(q0+(q1-q0)*t))**2)) #minimal distance
Now, I realize this is extremely inefficient, and this is why I am here. I have looked extensively in how to avoid the loop, but I run into a bit of a problem. Apparently, this sort of calculations is best done with the cdist of python. However, the custom distance functions it can handle have to be binary functions. This is a problem in my case, because my vectors have specifically a length of 6, and have to bit split into their first and last 3 components. I don't think I can translate the distance calculation into a binary function.
Any input is appreciated.
You can use numpy's vectorization capabilities to speed up the calculation. My version computes all elements of the distance matrix at once and then sets the diagonal and the lower triangle to zero.
def pairwise_distance2(s):
# we need this because we're gonna divide by zero
old_settings = np.seterr(all="ignore")
N = N_segments # just shorter, could also use len(s)
# we repeat p0 and p1 along all columns
p0 = np.repeat(s[:,0:3].reshape((N, 1, 3)), N, axis=1)
p1 = np.repeat(s[:,3:6].reshape((N, 1, 3)), N, axis=1)
# and q0, q1 along all rows
q0 = np.repeat(s[:,0:3].reshape((1, N, 3)), N, axis=0)
q1 = np.repeat(s[:,3:6].reshape((1, N, 3)), N, axis=0)
# element-wise dot product over the last dimension,
# while keeping the number of dimensions at 3
# (so we can use them together with the p* and q*)
a = np.sum((p1 - p0) * (p1 - p0), axis=-1).reshape((N, N, 1))
b = np.sum((p1 - p0) * (q1 - q0), axis=-1).reshape((N, N, 1))
c = np.sum((q1 - q0) * (q1 - q0), axis=-1).reshape((N, N, 1))
d = np.sum((p1 - p0) * (p0 - q0), axis=-1).reshape((N, N, 1))
e = np.sum((q1 - q0) * (p0 - q0), axis=-1).reshape((N, N, 1))
# same as above
s = (b*e-c*d)/(a*c-b*b)
t = (a*e-b*d)/(a*c-b*b)
# almost same as above
pairwise = np.sqrt(np.sum( (p0 + (p1 - p0) * s - ( q0 + (q1 - q0) * t))**2, axis=-1))
# turn the error reporting back on
np.seterr(**old_settings)
# set everything at or below the diagonal to 0
pairwise[np.tril_indices(N)] = 0.0
return pairwise
Now let's take it for a spin. With your example, N = 1000, I get a timing of
%timeit pairwise_distance(List_of_segments)
1 loops, best of 3: 10.5 s per loop
%timeit pairwise_distance2(List_of_segments)
1 loops, best of 3: 398 ms per loop
And of course, the results are the same:
(pairwise_distance2(List_of_segments) == pairwise_distance(List_of_segments)).all()
returns True. I'm also pretty sure there's a matrix multiplication hidden somewhere in the algorithm, so there should be some potential for further speedup (and also cleanup).
By the way: I've tried simply using numba first without success. Not sure why, though.
This is more of a meta answer, at least for starters. Your problem might already be in "my program hits a bottleneck" and "I realize this is extremely inefficient".
Extremely inefficient? By what measure? Do you have comparison? Is your code too slow to finish in a reasonable amount of time? What is a reasonable amount of time for you? Can you throw more computing power at the problem? Equally important -- do you use a proper infrastructure to run your code on (numpy/scipy compiled with vendor compilers, possibly with OpenMP support)?
Then, if you have answers for all of the questions above and need to further optimize your code -- where is the bottleneck in your current code exactly? Did you profile it? It the body of the loop possibly much more heavy-weight than the evaluation of the loop itself? If so, then "the loop" is not your bottleneck and you do not need to worry about the nested loop in the first place. Optimize the body at first, possibly by coming up with unorthodox matrix representations of your data so that you can perform all these single calculations in one step -- by matrix multiplication, for instance. If your problem is not solvable by efficient linear algebra operations, you can start writing a C extension or use Cython or use PyPy (which just very recently got some basic numpy support!). There are endless possibilities for optimizing -- the questions really are: how close to a practical solution are you already, how much do you need to optimize, and how much of an effort are you willing to invest.
Disclaimer: I have done non-canonical pairwise-distance stuff with scipy/numpy for my PhD, too ;-). For one particular distance metric, I ended up coding the "pairwise" part in simple Python (i.e. I also used the doubly-nested loop), but spent some effort in getting the body as efficient as possible (with a combination of i) a cryptical matrix multiplication representation of my problem and ii) using bottleneck).
You can use it something like this:
def distance3d (p, q):
if (p == q).all ():
return 0
p0 = p[0:3]
p1 = p[3:6]
q0 = q[0:3]
q1 = q[3:6]
... # Distance computation using the formula above.
print (distance.cdist (List_of_segments, List_of_segments, distance3d))
It doesn't seem to be any faster, though, since it executes the same loop internally.
Related
I'm trying to improve a simple algorithm to obtaining the Pearson correlation coefficient from two arrays, X(m, n) and Y(n), returning me another array R of dimension (m).
In the case, I want to know the behavior each row of X regarding the values of Y. A sample (working) code is presented below:
import numpy as np
from scipy.stats import pearsonr
np.random.seed(1)
m, n = 10, 5
x = 100*np.random.rand(m, n)
y = 2 + 2*x.mean(0)
r = np.empty(m)
for i in range(m):
r[i] = pearsonr(x[i], y)[0]
For this particular case, I get: r = array([0.95272843, -0.69134753, 0.36419159, 0.27467137, 0.76887201, 0.08823868, -0.72608421, -0.01224453, 0.58375626, 0.87442889])
For small values of m (near 10k) this runs pretty fast, but I'm starting to work with m ~ 30k, and so this is taking much longer than I expected. I'm aware I could implement multiprocessing/multi-threading but I believe there's a (better) pythonic way of doing this.
I tried to use use pearsonr(x, np.ones((m, n))*y), but it returns only (nan, nan).
pearsonr only supports 1D array internally. Moreover, it computes the p-values which is not used here. Thus, it would be more efficient not to compute it if possible. Additionally, the code also recompute the y vector every time and it does not efficiently make use of vectorized Numpy operations. This is why the computation is a bit slow. You can check this in the code here.
One way to compute this is by writing your own custom implementation based on the one of Scipy:
def multi_pearsonr(x, y):
xmean = x.mean(axis=1)
ymean = y.mean()
xm = x - xmean[:,None]
ym = y - ymean
normxm = np.linalg.norm(xm, axis=1)
normym = np.linalg.norm(ym)
return np.clip(np.dot(xm/normxm[:,None], ym/normym), -1.0, 1.0)
It is 450 times faster on my machine for m = 10_000.
Note that I did not keep the checks of the Scipy code, but it may be a good idea to keep them if your input is not guaranteed to be statistically safe (ie. well formatted for the computation of the Pearson test).
equations
alpha=0.05,beta=0.1,p1=0.015,p2=0.1
Solve for the unknowns n and c!
At a first look, some information is missing regarding how your application defines an acceptable solution, but I am going to make some assumptions and provide a (somewhat naive) approximate answer.
Context first: the problem, posed as is, is a set of 2 algebraic equations in 2 unknowns, to be solved on the 2D grid of non-negative integers. I am assuming you meant the unknowns are non-negative integers, and not, say, real numbers, from the problem formulation, involving the combinatorial terms. Further, note that whenever n is smaller than c, the summation will eventually involve factorial operations on a negative number, formally undefined, hence I will further suggest assuming that we solve over the triangular domain of non-negative integer pairs where n is greater than or equal to c.
As such, to refine the definition of solution, one would have to specify an allowed numeric tolerance (requiring exact equality or allowing, from your application point of view, some error). Unless there is some guarantee, from the problem domain, of the existence of integers solving exactly, I am proceeding to assume that an approximation with known error is meaningful/valuable. Finally: I am going to assume you are looking for A solution (not all solutions) and I am going to limit the search space, as you will see in the simple script below.
With the above said, I don't have any algebraic manipulation to suggest that would lead to a solution, and am not aware of a commercial/open source solver that can solve this non-linear problem over an integer grid. Surprisingly, choosing a reasonable grid and just exhaustively checking some options, yields the following approximate solution:
Opt n: 52, opt c: 2, min_err (residual L2 norm): 0.0074985711238005955
This of course does not claim that there cannot be more accurate approximations for some part of the grid not visited. Such guarantees, if at all, can be achieved with further analysis, such as monotone behavior if the right hand side of the equations w.r.t n and c, etc.
To arrive at the simple approximation suggested above, I treated your right hand sides as functions of n and c, and performed some exhaustive evaluations, keeping track of the error w.r.t to the left hand side required value. The error is then defined by the L2 norm of the error vector - simply the difference between the RHS evaluated at the current location, and the LHS of each of the equations. This is done by the following script, which of course is not scalable for large values of n and c, and the fact that the closest candidate was in the interior of the search space, for all I know, was just chance:
import math
import numpy as np
def eval_eq(n, c, p):
res = 0
for d in range(c + 1):
res += (math.factorial(n) / (math.factorial(d) * (math.factorial(n - d)))) * (p ** d) * ((1. - p) ** (n - d))
return res
def eval_err_norm(n, c):
p1 = 0.015
p2 = 0.1
beta = 0.1
alpha = 0.05
eq1_val = eval_eq(n, c, p1)
eq2_val = eval_eq(n, c, p2)
err1 = eq1_val - (1. - alpha)
err2 = eq2_val - beta
l2_err = np.linalg.norm(np.array([err1, err2]))
return l2_err
if __name__ == '__main__':
err = np.inf
opt_n = None
opt_c = None
for n_ in range(100):
for c_ in range(100):
if n_ < c_:
continue
else:
cur_err = eval_err_norm(n_, c_)
if cur_err < err:
err = cur_err
opt_n = n_
opt_c = c_
print(f'Opt n: {opt_n}, opt c: {opt_c}, min_err (residual L2 norm): {err}')
I am trying to create a piecewise linear regression to minimize the MSE(minimum square errors) then using linear regression directly. The method should be using dynamic programming to calculate the different piecewise sizes and combinations of groups to achieve the overall MSE. I think the algorithm runtime is O(n²) and I wonder if there are ways to optimize it to O(nLogN)?
import numpy as np
from sklearn.metrics import mean_squared_error
from sklearn import linear_model
import pandas as pd
import matplotlib.pyplot as plt
x = [3.4, 1.8, 4.6, 2.3, 3.1, 5.5, 0.7, 3.0, 2.6, 4.3, 2.1, 1.1, 6.1, 4.8,3.8]
y = [26.2, 17.8, 31.3, 23.1, 27.5, 36.5, 14.1, 22.3, 19.6, 31.3, 24.0, 17.3, 43.2, 36.4, 26.1]
dataset = np.dstack((x,y))
dataset = dataset[0]
d_arg = np.argsort(dataset[:,0])
dataset = dataset[d_arg]
def calc_error(dataset):
lr_model = linear_model.LinearRegression()
x = pd.DataFrame(dataset[:,0])
y = pd.DataFrame(dataset[:,1])
lr_model.fit(x,y)
predictions = lr_model.predict(x)
mse = mean_squared_error(y, predictions)
return mse
#n is the number of points , m is the number of groups, k is the minimum number of points in a group
#（15，5，3）returns 【【3，3，3，3，3】】
#（15，5，2） returns [[2,4,3,3,3],[3,2,4,2,4],[4,2,3,3,3]....]
def all_combination(n,m,k):
result = []
if n < k*m:
print('There are not enough elements to split.')
return
combination_bottom = [k for q in range(m)]
#add greedy algorithm here?
if n == k*m:
result.append(combination_bottom.copy())
else:
combination_now = [combination_bottom.copy()]
j = k*m+1
while j < n+1:
combination_last = combination_now.copy()
combination_now = []
for x in combination_last:
for i in range (0, m):
combination_new = x.copy()
combination_new[i] = combination_new[i]+1
combination_now.append(combination_new.copy())
j += 1
else:
for x in combination_last:
for i in range (0, m):
combination_new = x.copy()
combination_new[i] = combination_new[i]+1
if combination_new not in result:
result.append(combination_new.copy())
return result #2-d list
def calc_sum_error(dataset,cb):#cb = combination
mse_sum = 0
for n in range(0,len(cb)):
if n == 0:
low = 0
high = cb[0]
else:
low = 0
for i in range(0,n):
low += cb[i]
high = low + cb[n]
mse_sum += calc_error(dataset[low:high])
return mse_sum
#k is the number of points as a group
def best_piecewise(dataset,k):
lenth = len(dataset)
max_split = lenth // k
min_mse = calc_error(dataset)
split_cb = []
all_cb = []
for i in range(2, max_split+1):
split_result = all_combination(lenth, i, k)
all_cb += split_result
for cb in split_result:
tmp_mse = calc_sum_error(dataset,cb)
if tmp_mse < min_mse:
min_mse = tmp_mse
split_cb = cb
return min_mse, split_cb, all_cb
min_mse, split_cb, all_cb = best_piecewise(dataset, 2)
print('The best split of the data is '+str(split_cb))
print('The minimum MSE value is '+str(min_mse))
x = np.array(dataset[:,0])
y = np.array(dataset[:,1])
plt.plot(x,y,"o")
for n in range(0,len(split_cb)):
if n == 0:
low = 0
high = split_cb[n]
else:
low = 0
for i in range(0,n):
low += split_cb[i]
high = low + split_cb[n]
x_tmp = pd.DataFrame(dataset[low:high,0])
y_tmp = pd.DataFrame(dataset[low:high,1])
lr_model = linear_model.LinearRegression()
lr_model.fit(x_tmp,y_tmp)
y_predict = lr_model.predict(x_tmp)
plt.plot(x_tmp, y_predict, 'g-')
plt.show()
Please let me know if I didn't make it clear in any part.
It took me some time to realize, that the problem you're describing is exactly what a decision tree regressor tries to solve.
Unfortunately, construction of an optimal decision tree is NP-hard, meaning that even with dynamic programming you can't bring the runtime down to anything like O(NlogN).
Good news is that you can directly use any well maintained decision tree implementation, DecisionTreeRegressor of sklearn.tree module for example, and can be certain about obtaining best possible performance in O(NlogN) time complexity. To enforce a minimum number of points per group, use min_samples_leaf parameter. You can also control several other properties like maximun of no. groups with max_leaf_nodes, optimization w.r.t different loss functions using criterion etc.
If you're curious how Scikit-learn's decision tree compare with the one learnt by your algorithm (i.e. split_cb in your code):
X = np.array(x).reshape(-1,1)
dt = DecisionTreeRegressor(min_samples_leaf=MIN_SIZE).fit(X,y)
split_cb = np.unique(dt.apply(X),return_counts=True)[1]
And then use the same plotting code you use. Do note that since your time complexity is considerably higher than O(NlogN)*, your implementation will often find better splits than the scikit-learn's greedy algorithm.
[1] Hyafil, L., & Rivest, R. L. (1976). Constructing optimal binary decision trees is np-complete. Information Processing Letters, 5(1), 15–17
*Although I'm not sure about the exact time complexity of your implementation, it's quite certainly worse than O(N^2), all_combination(21,4,2) took more than 5 mins.
Just so you know, this is a massive topic and there is no way to discuss all of it here. But I think we can make some good inroads and answer what you're looking for along the way.
Also, I think theory first works best because others may not be at the same point. Your code which worked out of the box - Hell yes! - kinda indicates you know what I'm getting ready to say but it leaves me with a couple of questions:
Why write in vanilla python when it isn't needed and is much slower than NumPy which you are already importing and using to some extent?
Does your example indicate that you don't fully understand the application piecewise regression? Since we're starting with the theory first, this may a bit of a non-issue.
Here's the thing about regression: It rarely models the data exactly and the closer it gets to perfectly accurate, the closer it gets to being overfit.
Your piecewise regressions are, with the exception of the first one, absolutely perfect. And they should be. Two points make a line. So, in the example you provided, you've also given an example of overfitting the data and what a brittle model would look like. Not sure if that's right? Consider what would the x values of 4.85 to 5.99 return? How about 3.11 to 3.39?
Your example is on the left (or top), standard linear regression is on the right (or bottom):
This linear regression on the right gives us y values for the full range of x values, and ones that (presumably) continue on. The continuity of a function is exactly what we're seeking. With the other example, You can throw any number of tools at it, including a decision tree regressor, but you'll either get something similarly brittle or something that violates expectations. And then what? Toss it out because it's 'wrong'? Go with it because 'that's what the computer says'? Both are equally bad.
We could stop there. But this is a really good question and it would be a disservice to not go on. So let's start with two different datasets that we know are good candidates for piecewise regression.
iterations = 500
x = np.random.normal(0, 1, iterations) * 10
y = np.where(x < 0, -4 * x + 3 , np.where(x < 10, x + 48, x + 98)) + np.random.normal(0, 3, iterations)
plt.scatter(x, y, s = 3, color = 'k')
plt.show()
... which gives us the left image. I picked this for two reasons: It's continuous over the x-values but not with the y-values. The image on the right is from a really well-done R package Which is also continuous on the x-axis, has one clear break but would still be best by three piecewise regressions. I'll mention a bit more about it later.
A couple of things to note:
One sort of obvious way to detect breakpoints is to look for where a line would one which would pass a one-sided limit test but fail a two-sided limit. That is, it's not differentiable.
Breakpoints in dummy data like this are going to be so much easier to identify because we're using code to develop them. But in real life, we will probably need a different solution. Let's set that aside for now.
One of the concerns I highlighted before was wondering why you would write vanilla python when other libraries are specifically geared towards your question are so much faster. So, let's find out how much faster and what sort of an answer you might find. And let's use the discontiguous torture test for good measure:
from scipy import optimize
def piecewise_linear(x, x0, x1, b, k1, k2, k3):
condlist = [x < x0, (x >= x0) & (x < x1), x >= x1]
funclist = [lambda x: k1*x + b, lambda x: k1*x + b + k2*(x-x0), lambda x: k1*x + b + k2*(x-x0) + k3*(x - x1)]
return np.piecewise(x, condlist, funclist)
p, e = optimize.curve_fit(piecewise_linear, x, y)
xd = np.linspace(-30, 30, iterations)
plt.plot(x, y, "ko" )
plt.plot(xd, piecewise_linear(xd, *p))
Even in a fairly extreme case like this, we get a quick, robust answer which is probably not as pretty as we would like and takes some thought about if it's optimal or not. So, take a sec and consider the graph. Is it optimal (and why) or not (and why not)?
While we're at it, let's talk about time. Running %%timeit on the roll-your-own version (imports, data, plotting -- the whole thing) took:
10.8 s ± 160 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Which was 650 times longer than doing something similar (but with additionally randomizing 500 data points) with built-in NumPy and SciPy functions.
16.5 ms ± 1.87 ms per loop (mean ± std. dev. of 7 runs, 100 loops each)
If that doesn't quite do it for you which is a very reasonable situation because (and I'm sort of tipping my hand here) we would expect a piecewise linear regression to catch any and all discontinuous breaks. So for that, let me refer you to this GitHub gist by datadog since a: there is no need to re-invent the wheel and b: they have an interesting implementation. Along with the code is an accompanying blog post that addresses the key shortcoming of dynamic programming as well as their methodology and thinking.
While dynamic programming can be used to traverse this search space
much more efficiently than a naive brute-force implementation, it’s
still too slow in practice.
Three last points.
If you tweak the random points to get a contiguous line, the results look even better but it's not perfect.
You can see why this can't all be addressed in one question. I haven't addressed things like curve fitting, Anscombe's quartet, splining, or using ML.
For now, there is no substitute for understanding what is going on. That said the R package MCP is really impressive in how it identifies inflection points using a Bayesian approach.
I am now trying to learn the ADMM algorithm (Boyd 2010) for LASSO regression.
I found out a very good example on this page.
The matlab code is shown here.
I tried to convert it into python language so that I could develop a better understanding.
Here is the code:
import scipy.io as io
import scipy.sparse as sp
import scipy.linalg as la
import numpy as np
def l1_norm(x):
return np.sum(np.abs(x))
def l2_norm(x):
return np.dot(x.ravel().T, x.ravel())
def fast_threshold(x, threshold):
return np.multiply(np.sign(x), np.fmax(abs(x) - threshold, 0))
def lasso_admm(X, A, gamma):
c = X.shape[1]
r = A.shape[1]
C = io.loadmat("C.mat")["C"]
L = np.zeros(X.shape)
rho = 1e-4
maxIter = 200
I = sp.eye(r)
maxRho = 5
cost = []
for n in range(maxIter):
B = la.solve(np.dot(A.T, A) + rho * I, np.dot(A.T, X) + rho * C - L)
C = fast_threshold(B + L / rho, gamma / rho)
L = L + rho * (B - C);
rho = min(maxRho, rho * 1.1);
cost.append(0.5 * l2_norm(X - np.dot(A, B)) + gamma * l1_norm(B))
cost = np.array(cost).ravel()
return B, cost
data = io.loadmat("lasso.mat")
A = data["A"]
X = data["X"]
B, cost = lasso_admm(X, A, gamma)
I have found the loss function did not converge after 100+ iterations. Matrix B did not tend to be sparse, on the other hand, the matlab code worked in different situations.
I have checked with different input data and compared with Matlab outputs, yet I still could not get hints.
Could anybody take a try?
Thank you in advance.
My gut feeling as to why this is not working to your expectations is your la.solve() call. la.solve() assumes that the matrix is full rank and is independent (i.e. invertible). When you use \ in MATLAB, what MATLAB does under the hood is that if the matrix is full rank, the exact inverse is found. However, should the matrix not be this way (i.e. overdetermined or underdetermined), the solution to the system is solved by least-squares instead. I would suggest you modify that call so that you're using lstsq instead of solve. As such, simply replace your la.solve() call with this:
sol = la.lstsq(np.dot(A.T, A) + rho * I, np.dot(A.T, X) + rho * C - L)
B = sol[0]
Note that lstsq returns a whole bunch of outputs in a 4-element tuple, in addition to the solution. The solution of the system is in the first element of this tuple, which is why I did B = sol[0]. What is also returned are the sums of residues (second element), the rank (third element) and the singular values of the matrix you are trying to invert when solving (fourth element).
Also some peculiarities that I have noticed:
One thing that may or may not matter is the random generation of numbers. MATLAB and Python NumPy generate random numbers differently, so this may or may not affect your solution.
In MATLAB, Simon Lucey's code initializes L to be a zero matrix such that L = zeros(size(X));. However, in your Python code, you initialize L to be this way: L = np.zeros(C.shape);. You are using different variables to ascertain the shape of L. Obviously, the
code wouldn't work if there was a dimension mismatch, but that's another thing that's different. Not sure if this will affect your solution either.
So far I haven't found anything out of the ordinary, so try that fix and let me know.
I want to solve this kind of problem:
dy/dt = 0.01*y*(1-y), find t when y = 0.8 (0<t<3000)
I've tried the ode function in Python, but it can only calculate y when t is given.
So are there any simple ways to solve this problem in Python?
PS: This function is just a simple example. My real problem is so complex that can't be solve analytically. So I want to know how to solve it numerically. And I think this problem is more like an optimization problem:
Objective function y(t) = 0.8, Subject to dy/dt = 0.01*y*(1-y), and 0<t<3000
PPS: My real problem is:
objective function: F(t) = 0.85,
subject to: F(t) = sqrt(x(t)^2+y(t)^2+z(t)^2),
x''(t) = (1/F(t)-1)*250*x(t),
y''(t) = (1/F(t)-1)*250*y(t),
z''(t) = (1/F(t)-1)*250*z(t)-10,
x(0) = 0, y(0) = 0, z(0) = 0.7,
x'(0) = 0.1, y'(0) = 1.5, z'(0) = 0,
0<t<5
This differential equation can be solved analytically quite easily:
dy/dt = 0.01 * y * (1-y)
rearrange to gather y and t terms on opposite sides
100 dt = 1/(y * (1-y)) dy
The lhs integrates trivially to 100 * t, rhs is slightly more complicated. We can always write a product of two quotients as a sum of the two quotients * some constants:
1/(y * (1-y)) = A/y + B/(1-y)
The values for A and B can be worked out by putting the rhs on the same denominator and comparing constant and first order y terms on both sides. In this case it is simple, A=B=1. Thus we have to integrate
1/y + 1/(1-y) dy
The first term integrates to ln(y), the second term can be integrated with a change of variables u = 1-y to -ln(1-y). Our integrated equation therefor looks like:
100 * t + C = ln(y) - ln(1-y)
not forgetting the constant of integration (it is convenient to write it on the lhs here). We can combine the two logarithm terms:
100 * t + C = ln( y / (1-y) )
In order to solve t for an exact value of y, we first need to work out the value of C. We do this using the initial conditions. It is clear that if y starts at 1, dy/dt = 0 and the value of y never changes. Thus plug in the values for y and t at the beginning
100 * 0 + C = ln( y(0) / (1 - y(0) )
This will give a value for C (assuming y is not 0 or 1) and then use y=0.8 to get a value for t. Note that because of the logarithm and the factor 100 multiplying t y will reach 0.8 within a relatively short range of t values, unless the initial value of y is incredibly small. It is of course also straightforward to rearrange the equation above to express y in terms of t, then you can plot the function as well.
Edit: Numerical integration
For a more complexed ODE which cannot be solved analytically, you will have to try numerically. Initially we only know the value of the function at zero time y(0) (we have to know at least that in order to uniquely define the trajectory of the function), and how to evaluate the gradient. The idea of numerical integration is that we can use our knowledge of the gradient (which tells us how the function is changing) to work out what the value of the function will be in the vicinity of our starting point. The simplest way to do this is Euler integration:
y(dt) = y(0) + dy/dt * dt
Euler integration assumes that the gradient is constant between t=0 and t=dt. Once y(dt) is known, the gradient can be calculated there also and in turn used to calculate y(2 * dt) and so on, gradually building up the complete trajectory of the function. If you are looking for a particular target value, just wait until the trajectory goes past that value, then interpolate between the last two positions to get the precise t.
The problem with Euler integration (and with all other numerical integration methods) is that its results are only accurate when its assumptions are valid. Because the gradient is not constant between pairs of time points, a certain amount of error will arise for each integration step, which over time will build up until the answer is completely inaccurate. In order to improve the quality of the integration, it is necessary to use more sophisticated approximations to the gradient. Check out for example the Runge-Kutta methods, which are a family of integrators which remove progressive orders of error term at the cost of increased computation time. If your function is differentiable, knowing the second or even third derivatives can also be used to reduce the integration error.
Fortunately of course, somebody else has done the hard work here, and you don't have to worry too much about solving problems like numerical stability or have an in depth understanding of all the details (although understanding roughly what is going on helps a lot). Check out http://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.ode.html#scipy.integrate.ode for an example of an integrator class which you should be able to use straightaway. For instance
from scipy.integrate import ode
def deriv(t, y):
return 0.01 * y * (1 - y)
my_integrator = ode(deriv)
my_integrator.set_initial_value(0.5)
t = 0.1 # start with a small value of time
while t < 3000:
y = my_integrator.integrate(t)
if y > 0.8:
print "y(%f) = %f" % (t, y)
break
t += 0.1
This code will print out the first t value when y passes 0.8 (or nothing if it never reaches 0.8). If you want a more accurate value of t, keep the y of the previous t as well and interpolate between them.
As an addition to Krastanov`s answer:
Aside of PyDSTool there are other packages, like Pysundials and Assimulo which provide bindings to the solver IDA from Sundials. This solver has root finding capabilites.
Use scipy.integrate.odeint to handle your integration, and analyse the results afterward.
import numpy as np
from scipy.integrate import odeint
ts = np.arange(0,3000,1) # time series - start, stop, step
def rhs(y,t):
return 0.01*y*(1-y)
y0 = np.array([1]) # initial value
ys = odeint(rhs,y0,ts)
Then analyse the numpy array ys to find your answer (dimensions of array ts matches ys). (This may not work first time because I am constructing from memory).
This might involve using the scipy interpolate function for the ys array, such that you get a result at time t.
EDIT: I see that you wish to solve a spring in 3D. This should be fine with the above method; Odeint on the scipy website has examples for systems such as coupled springs that can be solved for, and these could be extended.
What you are asking for is a ODE integrator with root finding capabilities. They exist and the low-level code for such integrators is supplied with scipy, but they have not yet been wrapped in python bindings.
For more information see this mailing list post that provides a few alternatives: http://mail.scipy.org/pipermail/scipy-user/2010-March/024890.html
You can use the following example implementation which uses backtracking (hence it is not optimal as it is a bolt-on addition to an integrator that does not have root finding on its own): https://github.com/scipy/scipy/pull/4904/files