We Keep Coding

Invert a 2D --> 2D non-linear function

I'm sorry if this is an unusual way to write a question here, since the scope of it seems quite large (to me). I'd be happy to be directed to pre-defined packages that already do what I'm needing, indeed I hope there is (should be?) a standardised solution to my question. I was wondering if there is any help out there? I'm still learning Python through a project I'm doing, and I feel I'm slightly weak on certain points...
Ok so here goes: I would like to invert a 2D --> 2D function in python, but none of my efforts have succeeded yet.
Let's say I have two relationships in a (non-linear) systems of equations, so
a = f(x,y)
b = g(x,y)
where both f and g are continuous and invertible, and x and y have a certain pre-defined rectangular domain. a and b also have their own rectangular domain, but it is different from that of x and y.
Some extra info on f and g: One of the functions will be linear, let's call this f. So, a = f(x,y) = qx + py + r (where q, p and r are known constants). In Python terms, I guess you would write a[ i, j ] = qx[ i ] + py[ j ]. The other function, g, has no analytic expression but looks similar to ksin(x) + lsin(y), for x and y between 0 and pi/2.
Moreover, the overall "mother-function" that I wish to make a 3D surface plot of, takes a and b as arguments. Calling the mother function M, we then have that M = M(a,b) = M(f(x,y),g(x,y)). So far so good.
The essence of the problem is that I need to first choose a pair (a,b) on the "mother-grid", then find the corresponding pair (x,y) that gives rise to this particular (a,b). f and g do not have any analytically "simple" inverses however, and I need to find these numerically.
So the basic question is, "given a[ i ] and b[ j ] as two sorted lists, and given x[ ii ] and y [ jj ] that are used to obtain each a and b, how do I find the two inverse functions x = inv1 (a,b) and y = inv2 (a,b)?"
PS. I have tried the "cheap way" of circumventing this problem by first choosing a (x,y) pair, calculating a tentative (A,B) pair and then interpolate this into the pre-defined (a,b) mesh as best as I could. However, since the (x,y) mesh and the (a,b) meshes are (quite) different, the corresponding "fitting error" always make the end result come out jagged and messy (I have a control scenario where I know what the end result should look like, before doing my own cases). This is because I am essentially forcing the values of A and B onto the height of the M function at position (a,b) if that makes sense. I've tried averaging and smoothing "cheats" to this, but it is still not passable imo. Hence, I really need to choose an (a,b) pair FIRST, and then only finding the relevant (x,y) pair after that.
Note: Some parameters in the M-function depends directly on x and y, hence the need for knowing the exact values of x and y.

Thanks for the additional information! I think you can solve this directly and then analytically.
Starting with your final result (a, b):
First solve against a to find your x-y line, e.g.:
a = qx + py + r
y = (qx + r - a) / -p
Since you said it is monotonically increasing, I just solve for y for simplicity.
Next, plug this into your non-analytic g using a simple binary search across x:
def invert(a, b):
def get_y(x):
return (Q * x + R - a) / -P
def g_constrained(x):
return g(x, func(x))
x = binary_search(g_constrained, b, min_x, max_x, guess_x)
y = get_y(x)
return x, y
Note that your function is not guaranteed to have a single solution in general, consider a planar solution for g that looks like an arc, since f is a line, you can have two intersections. You will need to decide what you want to do with this information.
Previous concerns
I am suspicious of your claim that a = f(x,y) is continuous and invertible.
Put succinctly: if your function z = f(x, y) doesn't intersect the plane z = K in exactly one point for every K, it is not invertible.
For a detailed example:
Consider some point, and 4 points around it - here I use (0,0) and unit length, for convenience.
z = f(0,0)
a = f(-1,0)
b = f(1,0)
p = f(0,-1)
q = f(0,1)
If f provides a scalar value (or anything where x < y and y < z implies x < z), then we have a problem.
Since f is continuous and invertible, either a < z < b or b < z < a, and likewise for p and q. So f-inv(z+) will map to two different values, one on the line (-1, 0) -> (1, 0) and one on the line (0, -1) -> (0, 1).

Is there a way to generate random solutions to non-square linear equations, preferably in python?

First of all, I know that these threads exist! So bear with me, my question is not fully answered by them.
As an example assume we are in a 4-dimensional vector space, i.e R^4. We are looking at the two linear equations:
3*x1 - 2* x2 + 7*x3 - 2*x4 = 6
1*x1 + 3* x2 - 2*x3 + 5*x4 = -2
The actual questions is: Is there a way to generate a number N of points that solve both of these equations making use of the linear solvers from NumPy etc?
The main problem with all python libraries I have tried so far is: they need n equations for a n-dimensional space
Solving the problem is very easy for one equation, since you can simply use n-1 randomly generated vlaues and adapt the last one such that the vector solves the equation.
My expected result would be a list of N "randomly" generated points that solve k linear equations in an n-dimensional space, where k<n.

A system of linear equations with more variables than equations is known as an underdetermined system.
An underdetermined linear system has either no solution or infinitely many solutions.
...
There are algorithms to decide whether an underdetermined system has solutions, and if it has any, to express all solutions as linear functions of k of the variables (same k as above). The simplest one is Gaussian elimination.
As you say, many functions available in libraries (e.g. np.linalg.solve) require a square matrix (i.e. n equations for n unknowns), what you are looking for is an implementation of Gaussian elimination for non square linear systems.
This isn't 'random', but np.linalg.lstsq (least square) is will solve non-square matrices:
Return the least-squares solution to a linear matrix equation.
Solves the equation a x = b by computing a vector x that minimizes the Euclidean 2-norm || b - a x ||^2. The equation may be under-, well-, or over- determined (i.e., the number of linearly independent rows of a can be less than, equal to, or greater than its number of linearly independent columns). If a is square and of full rank, then x (but for round-off error) is the “exact” solution of the equation.
For more info, see:
solving Ax =b for a non-square matrix A using python

Since you have an underdetermined system of equations (too few constraints for your solutions, or fewer equations than variables) you can just pick some arbitrary values for x3 and x4 and solve the system in x1, x2 (this has 2 variables/2 equations).
You will just need to check that the resulting system is not inconsistent (i.e. it admits no solution) and that there are no duplicate solutions.
You could for instance fix x3=0 and choosing random values of x4 generate solutions for your equations in x1, x2
Here's an example generating 10 "random" solutions
n = 10
x3 = 0
X = []
for x4 in np.random.choice(1000, n):
b = np.array([[6-7*x3+2*x4],[-2+2*x3-5*x4]])
x = np.linalg.solve(a, b)
X.append(np.append(x,[x3,x4]))
# check solution nr. 3
[x1, x2, x3, x4] = X[3]
3*x1 - 2* x2 + 7*x3 - 2*x4
# output: 6.0
1*x1 + 3* x2 - 2*x3 + 5*x4
# output: -2.0

Thanks for the answers, which both helped me and pointed me in the right direction.
I now have an easy step-by-step solution to my problem for arbitrary k<n.
1. Find one solution to all equations given. This can be done by using
solution_vec = numpy.linalg.lstsq(A,b)
this gives a solution as seen in ukemis answer. In my example above, the Matrix A is equal to the coefficients of the equations on the left side, b represents the vector on the right side.
2. Determine the null space of your matrix A.
These are all vectors v such that the skalar product v*A_i = 0 for every(!) row A_i of A. The following function, found in this thread can be used to get representatives of the null space of A:
def nullSpaceOfMatrix(A, eps=1e-15):
u, s, vh = scipy.linalg.svd(A)
null_mask = (s <= eps)
null_space = scipy.compress(null_mask, vh, axis=0)
return scipy.transpose(null_space)
3. Generate as many (N) "random" linear combinations (meaning with random coefficients) of solution_vec and resulting vectors of the nullspace of the matrix as you want! This works because the scalar product is additive and nullspace vectors have a scalar product of 0 to the vectors of the equations. Those linear combinations always must contain solution_vec, as in:
linear_combination = solution_vec + a*null_spacevec_1 + b*nullspacevec_2...
where a and b can be randomly chosen.

Constrained non-linear optimization - overdetermined system of polynomial equations

I have a system of 21 polynomial equations in a total of 12 unknowns a, ..., l. Each equation has the general form V1*abc + V2*abd + ... + V64*jkl = x, where V1, ..., V64 are each either 0 or 1, i.e., each equation contains on the left hand side the sum of some products of three different unknowns.
There is a set of constrains: a + b + c + d = 1, e + f + g + h = 1, i + j + k + l = 1. The sum of all xs (right hand sides) is equal to 1.
I have as an input a vector of xs. Is there a solver which could provide me the values of a, ..., l which yield a vector of x's as close as possible to the original xs while adhering to the constrains ? I'm looking for a python implementation.
I looked in scipy.optimize but I'm not able to establish which method is preferable for my problem.

You might want to try this python binding for IPOPT. IPOPT is an optimization library that uses an interior-point solver for finding (local) optima of functions with generalized constraints, both equality and inequality constraints. As you've described your problem, you won't care about the inequality constraints.
A candidate function for your optimization objective would be the sum of the squared differences for your 21 polynomial equations. Let's say you start with your initial x, which is a 21-element vector, then your objective would be:
(V1_0*abc + V2_0*abd + ... + V64_0*jkl - x_0)^2 + (V1_1*abc + V2_1*abd + ... + V64_1*jkl - x_1)^2 + ...(V1_{21}*abc + V2_{21}*abd + ... + V64_{21}*jkl - x_{21})^2
To use IPOPT, you will need to compute the partial derivatives of your constraints and objective wrt all of your variable a-l.
If IPOPT won't work for you, you might even be able to use scipy.optimize with this objective function. From the docs, it looks like scipy.optimize will try to pick the method appropriate for your problem based upon how you define it; if you define your constraints and objective correctly, scipy.optimize should pick the correct method.

Probability density function convolution in python: how to efficiently?

Given that this is a fairly long post, I think it's best if I start with my question: What is the best way of implementing this model in python?
I'm developling a statistical model that involves the convolution of probability density functions. Without giving away what I can't, here is my problem:
Parameter A describes the time from t0 to event A at t1. It is given by a uniform distribution with lower and upper bounds a and b respectively.
A ~ U(a, b)
Parameter B describes the time from t1 to event B at t2. It is also given by a uniform distribution with lower and upper bounds c and d respectively.
B ~ U(c, d)
Parameter G describes the time from t0 to event C, at an unknown time. It is given by a gamma distribution with shape and scale parameters alpha and beta, with mean = alpha / beta and standard deviation = alpha / (beta ^ 2).
G ~ G(alpha, beta)
We would like to know the probability that G falls between t1 and t2. We would also like to perform a grid search over these parameters, so it is important that the calculation is completed efficiently. The target end user includes civil service agencies who do not have access to high performance computing (like we do), so parallellisation is not an option.
The time at which t1 occurs is given by A, however the time at which t2 occurs is given as the sum of A and B
Y = A + B
W = A - G
Z = Y - G
As these two distributions are uniform, they may be analytically convolved. The probability that G occurs between A and Y is given as P(A < G < Y). We can introduce variables Y and Z as above:
P(A < G < Y) = P(W < 0 < Z)
P(A < G < Y) = P(Z > 0) - P(W > 0)
P(A < G < Y) = [1 - P(Z < 0)] - [1 - P(W < 0)]
P(A < G < Y) = Fw(0) - Fz(0)
Fw(0) and Fz(0) refer to the cumulitive density functions of distributions Y and Z respectively.
Fw(0) can be calculated as shown in this image.
For Fz(0), a triple integral is required.
What would be the best way of computing this maths in python? We have a working program, but it is slow even when we constrain the integration bounds and analytically convolve A and B to give Y. The strategy is to convolve numerically, using quad from scipy.integrate. This is slow and frequently gives us roundoff errors and overruns the maximum number of subdivisions.
Is there a better way?
We can verify the results of the program against two existing implementations, one programmed in fortran but utilising NAG library integration routines, and another written in matlab. The fortran version is satisfactory for our own scientific research, but our motivation for using python is to produce a distributable source code written in a freely available language using open source routines such as those in the Scipy library. The use of NAG library routines percludes this from fortran.

On ordinary differential equations (ODE) and optimization, in Python

I want to solve this kind of problem:
dy/dt = 0.01*y*(1-y), find t when y = 0.8 (0<t<3000)
I've tried the ode function in Python, but it can only calculate y when t is given.
So are there any simple ways to solve this problem in Python?
PS: This function is just a simple example. My real problem is so complex that can't be solve analytically. So I want to know how to solve it numerically. And I think this problem is more like an optimization problem:
Objective function y(t) = 0.8, Subject to dy/dt = 0.01*y*(1-y), and 0<t<3000
PPS: My real problem is:
objective function: F(t) = 0.85,
subject to: F(t) = sqrt(x(t)^2+y(t)^2+z(t)^2),
x''(t) = (1/F(t)-1)*250*x(t),
y''(t) = (1/F(t)-1)*250*y(t),
z''(t) = (1/F(t)-1)*250*z(t)-10,
x(0) = 0, y(0) = 0, z(0) = 0.7,
x'(0) = 0.1, y'(0) = 1.5, z'(0) = 0,
0<t<5

This differential equation can be solved analytically quite easily:
dy/dt = 0.01 * y * (1-y)
rearrange to gather y and t terms on opposite sides
100 dt = 1/(y * (1-y)) dy
The lhs integrates trivially to 100 * t, rhs is slightly more complicated. We can always write a product of two quotients as a sum of the two quotients * some constants:
1/(y * (1-y)) = A/y + B/(1-y)
The values for A and B can be worked out by putting the rhs on the same denominator and comparing constant and first order y terms on both sides. In this case it is simple, A=B=1. Thus we have to integrate
1/y + 1/(1-y) dy
The first term integrates to ln(y), the second term can be integrated with a change of variables u = 1-y to -ln(1-y). Our integrated equation therefor looks like:
100 * t + C = ln(y) - ln(1-y)
not forgetting the constant of integration (it is convenient to write it on the lhs here). We can combine the two logarithm terms:
100 * t + C = ln( y / (1-y) )
In order to solve t for an exact value of y, we first need to work out the value of C. We do this using the initial conditions. It is clear that if y starts at 1, dy/dt = 0 and the value of y never changes. Thus plug in the values for y and t at the beginning
100 * 0 + C = ln( y(0) / (1 - y(0) )
This will give a value for C (assuming y is not 0 or 1) and then use y=0.8 to get a value for t. Note that because of the logarithm and the factor 100 multiplying t y will reach 0.8 within a relatively short range of t values, unless the initial value of y is incredibly small. It is of course also straightforward to rearrange the equation above to express y in terms of t, then you can plot the function as well.
Edit: Numerical integration
For a more complexed ODE which cannot be solved analytically, you will have to try numerically. Initially we only know the value of the function at zero time y(0) (we have to know at least that in order to uniquely define the trajectory of the function), and how to evaluate the gradient. The idea of numerical integration is that we can use our knowledge of the gradient (which tells us how the function is changing) to work out what the value of the function will be in the vicinity of our starting point. The simplest way to do this is Euler integration:
y(dt) = y(0) + dy/dt * dt
Euler integration assumes that the gradient is constant between t=0 and t=dt. Once y(dt) is known, the gradient can be calculated there also and in turn used to calculate y(2 * dt) and so on, gradually building up the complete trajectory of the function. If you are looking for a particular target value, just wait until the trajectory goes past that value, then interpolate between the last two positions to get the precise t.
The problem with Euler integration (and with all other numerical integration methods) is that its results are only accurate when its assumptions are valid. Because the gradient is not constant between pairs of time points, a certain amount of error will arise for each integration step, which over time will build up until the answer is completely inaccurate. In order to improve the quality of the integration, it is necessary to use more sophisticated approximations to the gradient. Check out for example the Runge-Kutta methods, which are a family of integrators which remove progressive orders of error term at the cost of increased computation time. If your function is differentiable, knowing the second or even third derivatives can also be used to reduce the integration error.
Fortunately of course, somebody else has done the hard work here, and you don't have to worry too much about solving problems like numerical stability or have an in depth understanding of all the details (although understanding roughly what is going on helps a lot). Check out http://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.ode.html#scipy.integrate.ode for an example of an integrator class which you should be able to use straightaway. For instance
from scipy.integrate import ode
def deriv(t, y):
return 0.01 * y * (1 - y)
my_integrator = ode(deriv)
my_integrator.set_initial_value(0.5)
t = 0.1 # start with a small value of time
while t < 3000:
y = my_integrator.integrate(t)
if y > 0.8:
print "y(%f) = %f" % (t, y)
break
t += 0.1
This code will print out the first t value when y passes 0.8 (or nothing if it never reaches 0.8). If you want a more accurate value of t, keep the y of the previous t as well and interpolate between them.

As an addition to Krastanov`s answer:
Aside of PyDSTool there are other packages, like Pysundials and Assimulo which provide bindings to the solver IDA from Sundials. This solver has root finding capabilites.

Use scipy.integrate.odeint to handle your integration, and analyse the results afterward.
import numpy as np
from scipy.integrate import odeint
ts = np.arange(0,3000,1) # time series - start, stop, step
def rhs(y,t):
return 0.01*y*(1-y)
y0 = np.array([1]) # initial value
ys = odeint(rhs,y0,ts)
Then analyse the numpy array ys to find your answer (dimensions of array ts matches ys). (This may not work first time because I am constructing from memory).
This might involve using the scipy interpolate function for the ys array, such that you get a result at time t.
EDIT: I see that you wish to solve a spring in 3D. This should be fine with the above method; Odeint on the scipy website has examples for systems such as coupled springs that can be solved for, and these could be extended.

What you are asking for is a ODE integrator with root finding capabilities. They exist and the low-level code for such integrators is supplied with scipy, but they have not yet been wrapped in python bindings.
For more information see this mailing list post that provides a few alternatives: http://mail.scipy.org/pipermail/scipy-user/2010-March/024890.html
You can use the following example implementation which uses backtracking (hence it is not optimal as it is a bolt-on addition to an integrator that does not have root finding on its own): https://github.com/scipy/scipy/pull/4904/files