Related
I am not a mathematician but I think that what I am after is called a "multiple linear regression"; please correct me if I am wrong.
I use numpy.polyfit and numpy.poly1d on a series of angle/pulse_width values from a servo motor, to obtain a function, angles_to_pulsewidths.
angles_to_pulsewidths is a polynomial function that models the servo and represents a line of good fit for the series. Given an angle value, it returns a corresponding pulse_width.
I am now trying to do a similar thing but instead of a single angle value in my series, I have pair of x/y co-ordinates for each pulse_width. I want to obtain a function that given an x/y pair, returns a corresponding pulse_width.
This is my code for creating my angles_to_pulsewidths function:
import numpy
angles_and_pulsewidths = [
[-162, 2490],
[-144, 2270],
[-126, 2070],
[-108, 1880]
]
angles_values_array = numpy.array(angles_and_pulsewidths)[:, 0]
pulsewidths_values_array = numpy.array(angles_and_pulsewidths)[:, 1]
coefficients = numpy.polyfit(
angles_values_array,
pulsewidths_values_array,
3
)
angles_to_pulsewidths = numpy.poly1d(coefficients)
I have been trying to modify this so that instead of providing a one-dimensional array of angles I will provide a two-dimensional array of x/y values:
xy_values = [[1, 2], [3, 4], [5, 6], [6, 7]]
pulse_widths = [2490, 2270, 2070, 1880]
However in this case, I can't use polyfit, because that takes only a one-dimensional array for its x parameter.
I can use numpy.linalg.lstsq instead, but I can't work out what to do with the results it gives me.
I'm also not even sure if I am on the right track; am I? I have read numerous related questions here, and have found numerous clues, but not enough to get me to the next step.
It is possible to use scipy's curve_fit for this.
If you know the general format of the function, perhaps you think it will be something of the form:
a x ^ 2 + b x y + c y ^ 2 + d x + e y +f
then you can use scipy's curve_fit to estimate what I will refer to as "parameters": a, b, c, d, e, f.
First we need to define the general form of our function:
def func(variables, a, b, c, d, e, f):
x, y = variables
return a * x ** 2 + b * x * y + c * y ** 2 + d * x + e * y + f
Note that our function has 6 parameters, to be able to demonstrate how this works we need more data than parameters so I'm extending your example data set to have 7 pairs of xy values and 7 pulse widths:
xy_values = [[1, 2], [3, 4], [5, 6], [6, 7], [8, 9], [10, 11], [12, 13]]
pulse_widths = [2490, 2270, 2070, 1880, 2000, 500, 600]
(If you do not have more data than parameters then you probably can choose a general form of your function to have less parameters.)
We need to reshape our xy_values so that it is not pairs of values but a single pair of two sets of values (the xs and the ys). To do this I'm choosing to creating a numpy array and "transpose" it:
xy_values = np.array(xy_values).T
We can now call our func on our array:
func(variables=xy_values, a=0, b=0, c=0, d=0, e=0, f=4)
Which gives:
array([4, 4, 4, 4, 4, 4, 4])
We can now actually use our data and curve_fit to estimate the best parameters:
from scipy.optimize import curve_fit
popt, pcov = curve_fit(f=func, xdata=xy_values, ydata=pulse_widths)
pcov contains information about how good the fit is and popt is the actual values of the parameters which we can directly see and use:
popt
gives:
array([ -25.61043682, -106.84636863, 119.10145249, -374.6200899 ,
230.65326227, 2141.55126789])
and we can call the function with it on some new value of x and y:
func([0, 5], *popt)
which gives:
6272.353891536915
Choosing the correct general form of the function you want to fit is case dependant. If there is any knowledge of the problem at hand (perhaps you expect there to be some trigonometric relationship) then you can use it otherwise it's a case of trial and error and getting a relationship that's "good enough" for your use case.
EDIT: Your original suggestion of needing to use multiple linear regression (MLR) is not completely incorrect. The solution approach I've described allows you to do MLR but it just assumes a specific type of func: one where all the terms are linear.
I'm aware that we can us numpy to differentiate polynomials with the following:
f = numpy.poly1d([1, 0, 1])
f.deriv()
I've tried interpolating a set of data points and performing deriv() on the resulting polynomial.
from scipy import interpolate
x = [-2,-1,2]
y = [-2,1,-1]
f = interpolate.interp1d(x, y)
f.deriv()
But the object f is of different type.
Basically, how might I convert f to a numpy polynomial object ready for differentiation?
Thanks a lot!
The issue you're facing here is the way the interpolation actually works. Interpolation can at most guess some local function that matches the given points the best, but it can't exactly, and probably never is, except for perhaps some extreme easy cases(?), be exactly correct as the actual function given.
That said, you can approximate a function in a given range as a Taylor polynomial to a good degree. This should for relatively narrow ranges and a good guess of the initial function work sufficiently well for you.(?)
import numpy as np
from scipy import interpolate
x = [-2, -1, 2]
y = [-2, 1, -1]
f = interpolate.interp1d(x, y)
h = interpolate.approximate_taylor_polynomial(f, -1, 2, 2)
h
>>>> poly1d([-0.61111111, 1.16666667, -0.22222222])
h.deriv()
>>>> poly1d([-1.22222222, 1.16666667])
EDIT I Expanding the original answer for clarification:
I wanted to show that this approach works to a point. The above OP example used is really small MWE example and thus the results are less than convincing.
To show it's fairly close approximation I'll construct a polynomial. I'll get its values in range [-5, 5]. I'll use the range [-5, 5] and the returned values of polynomial in as interpolation arrays.
I'll approximate the interpolated function with the Taylor series expansion using the best "guesses" I have (since I constructed the original polynomial this is not really a guess tbh).
I'll compare the results in range [-5, 5] from the Taylor expansion with the original polynomial values in the range.
f = np.poly1d([1,0,1])
f([-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5])
>>> array([26, 17, 10, 5, 2, 1, 2, 5, 10, 17, 26])
x = [-5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5]
y = [26, 17, 10, 5, 2, 1, 2, 5, 10, 17, 26]
f = interpolate.interp1d(x, y)
h = interpolate.approximate_taylor_polynomial(f, 0, 2, 5)
h(x)
>>> [26., 17.12, 10.21333333, 5.28, 2.32, 1.33333333, 2.32, 5.28, 10.21333333, 17.12, 26.]
f(x)
>>> [26., 17., 10., 5., 2., 1., 2., 5., 10., 17., 26.]
Here are some examples to show how the guesses get better and better the higher order of Taylor expansion you use. Careful as the manual says the expansion is unstable once it reaches order of 30.
h = interpolate.approximate_taylor_polynomial(f, 0, 15, 5)
>>> [ 25.41043927, 17.18570392, 10.19122784, 5.09107466,
2.02363911, 1. , 2.02664952, 5.07915194,
10.22646919, 17.13871545, 26. ])
h = interpolate.approximate_taylor_polynomial(f, 0, 20, 5)
>>> [ 26. , 17.13481942, 10.10070835, 5.21247548,
2.13174692, 1.23098041, 2.13174692, 5.21247548,
10.10070835, 17.13481942, 25.9999999 ])
EDIT II Answers for questions in comments:
It's not a stupid question. I can see that Taylor series is confusing you. In math they usually show the mathematical definition for the Taylor series based on nth order derivatives of the original function in the point of expansion but don't show others so it might be confusing as how to apply it in a broader sense.
In essence it's the same as with derivatives:
f' = lim d->0 [ ( f(x+d)-f(x) )/d ]
which in numerical programing we just approximate with:
f' = f(x+d)-f(x)/d (there are other approx of the derivative as well)
and that's an ok approximation as long as the d remains really small. Taylor series of a function goes something like this:
0th order: h ~ f(a)
1st order: h ~ f(a) + f'(a)(x-a)
2nd order: h ~ f(a) + f'(a)(x-a) + f''(a)/2 * (x-a)^2 ...
...
so if we now introduce our derivative approximation into the series expansion:
1st deriv 2nd deriv
h ~ f(a) + [ (f(a+d)-f(a))/d ] (x-a) + [ ( f(a+d) - 2f(a) + f(a-d) )/d^2 ) ] * (x-a)^2 ....
so now you see why the function needs the point in which it needs to be evaluated. Now, this helps us only to get rid of the derivation of the original function we had.
So you see, we don't need to know the original function at all. All we have to be able to do is provide approximated values the original function would have had in the point of expansion. And that is exactly what interpolation gives you.
Interpolation takes in a set of points presumably attributed to some original function and then based on the behavior of those points tries to guess which points in between the given points would most likely be in the graph of the original function as well. So in an essence, interpolation tries to guess the values of would-be function whose original points we know, within the range of those original points.
Ok. But what do we do about the fact that Taylor exp. goes into infinity?
We just round it of:
h ~ 0th + 1st + 2nd + 3rd + ... + nth + P
and we call P the remainder. There are ways of estimating this remainder, some given by Taylor himself iirc.
Ok, but now why do we need a range in our function call?
What we did here is what's called a finite difference method. In essence it is just about as simple as that, in reality things can get a bit more complicated because you have to show that indeed you can do these things and not break the convergence of Taylor series. It turns out that you don't break the Taylor series BUT only for bounded continuous functions, which means that you can only approx functions on a certain interval.
Think of it this way. You can approximate a straight line with a Taylor series. Think of it as compounding more and more and more polynomial orders until their "waviness" cancels each other out. Like doing sin^2 + cos^2 which is always 1.
But if you stop the series expansion at some order, then suddenly you don't have anything stopping the series from diverging again. Because Taylor series is just one big polynomial it will either start going up into infinity or down into infinity. Look at the image bellow, it shows Taylor series approximating the original quadratic function f in the point of expansion 0 on a range of 10 around it; but plotted from -50 to 50.
Special interest is the 1st series order, which is just a straight line as you can see from the formulas above (green). Notice how as soon as the series cross -10 or 10 they start diverging from the actual function by a lot. In some cases the functions were similar enough to continue being close in value with the original function (i.e. 2nd order Taylor series is also a quadratic equation which is why it traces the original function very well).
Unfortunately because we do not have any prior knowledge about the original function in your case it's impossible to determine that some Taylor expansion estimates it perfectly. As far as we know we only approximated the function around 0. It might as well contain sine or cosine members for all we know.
As far as your question about f is concerned:
f is just some dummy function I started from. It's supposed to look like np.poly1d([1,0,1]) which is supposed to be something like f(x) = x**2 +1. I don't know where you got 2.1667 + 0.25x - 0.9167x**2 from.
I used f just to create x and y arrays, so that I can be sure that indeed those numbers belong to a function. What would be the point otherwise. I only used it once more in the end by doing f(x) to show how similar the numbers turn out.
Remember x is an array, and f(x) means "calculate the value of function f for every member of array x". Nothing more. It's just the value of the function f(x) = x**2+1 in the points [-5, -4, .... 4, 5].
All other works was just based on how to approximate a function by Taylor expansion when all you have is some fixed data set and no knowledge of the original function. And I showed that if you interpolate between the points and approximate the unknown function with Taylor expansion you can reconstruct a function that has meaningfully similar results on a bounded range of numbers x.
That approximated function is called h in my snippets and it looks something like:
h = 2.28194274e-08 + 5.37467022e-17 x - 1.98652602e-06 x^2 - 3.65181145e-15 x^3 + 7.38646849e-05 x^4 + 1.02224219e-13 x^5 + ... till 25th order would be reached
and to get its derivative in python all you would need to do is
h.deriv()
because its type is poly1d.
I am trying to call scipy.stats.multivariate_normal with four different parameters for mu and sigma. And then for each generated probability density function I need to call that pdf on an array of say, 10 values.
For simplicity let's say that above mentioned function is addXY:
def addXY(x, y):
return x+y
params=[[1,2],[1,3],[1,4],[1,5]] # mu and sigma, four versions
inputs=[1,2,3] # values, in this case 3 of them
matrix = []
for pdf_params in params:
row = []
for inp in inputs:
entry = addXY(*pdf_params)
row.append(entry*inp)
matrix.append(row)
print matrix
Is this pythonic?
Is there a way to pass params and inputs and get a matrix with all combinations in it that is more pythonic/vectorized/faster?
!Important notice: Inputs in the example are scalar values (I've set scalar values to simplify problem description, I am actually using array of n-dimensional vectors and thus multivariate_normal pdf).
Hints and tips about similar operations are welcome.
Based on your description of what you are trying to compute, you don't need multivariate_normal. You are calling the PDF method with a set of scalar values for a distribution with a scalar mu and sigma. So you can use the pdf() method of scipy.stats.norm. This method will broadcast its arguments, so by passing in arrays with the proper shape, you can compute the PDF for the different values of mu and sigma in one call. Here's an example.
Here are your x values (you called them inputs), and the parameters:
In [23]: x = np.array([1, 2, 3])
In [24]: params = np.array([[1, 2], [1, 3], [1, 4], [1, 5]])
For convenience, separate the parameters into arrays of mu and sigma values.
In [25]: mu = params[:, 0]
In [26]: sig = params[:, 1]
We'll use scipy.stats.norm to compute the PDF.
In [27]: from scipy.stats import norm
This call computes the PDF for the desired combinations of x and parameters. mu.reshape(-1, 1) and sig.reshape(-1, 1) are 2D arrays with shape (4, 1). x has shape (3,), so when these arguments are broadcast, the result has shape (4, 3). Each row is the PDF evaluated at x for one of the pairs of mu and sigma.
In [28]: p = norm.pdf(x, loc=mu.reshape(-1, 1), scale=sig.reshape(-1, 1))
In [29]: p
Out[29]:
array([[ 0.19947114, 0.17603266, 0.12098536],
[ 0.13298076, 0.12579441, 0.10648267],
[ 0.09973557, 0.09666703, 0.08801633],
[ 0.07978846, 0.07820854, 0.07365403]])
In other words, the rows of p are:
norm.pdf(x, loc=mu[0], scale=sig[0])
norm.pdf(x, loc=mu[1], scale=sig[1])
norm.pdf(x, loc=mu[2], scale=sig[2])
norm.pdf(x, loc=mu[3], scale=sig[3])
This is only my idea to shorten the code and utilize more library.
In your code, in fact, you do not use numpy, scipy. Question will be whether you would like to use numpy.array for further data processing.
Option 1: just use list to present array and list of list to present matrix:
from itertools import product
matrix_list = [sum(param)*input_x for param, input_x in product(params, inputs)]
matrix = zip(*[iter(matrix_list)]*len(inputs))
print matrix
Credit for using zip method should be given to
convert a flat list to list of list in python
Option 2: use numpy.array and numpy.matrix for further processing
from itertools import product
import numpy as np
matrix_array = np.array([sum(param)*input_x for param, input_x in product(params, inputs)])
matrix = matrix_array.reshape(len(params),len(inputs))
print matrix
I am trying to figure out how to calculate covariance with the Python Numpy function cov. When I pass it two one-dimentional arrays, I get back a 2x2 matrix of results. I don't know what to do with that. I'm not great at statistics, but I believe covariance in such a situation should be a single number. This is what I am looking for. I wrote my own:
def cov(a, b):
if len(a) != len(b):
return
a_mean = np.mean(a)
b_mean = np.mean(b)
sum = 0
for i in range(0, len(a)):
sum += ((a[i] - a_mean) * (b[i] - b_mean))
return sum/(len(a)-1)
That works, but I figure the Numpy version is much more efficient, if I could figure out how to use it.
Does anybody know how to make the Numpy cov function perform like the one I wrote?
Thanks,
Dave
When a and b are 1-dimensional sequences, numpy.cov(a,b)[0][1] is equivalent to your cov(a,b).
The 2x2 array returned by np.cov(a,b) has elements equal to
cov(a,a) cov(a,b)
cov(a,b) cov(b,b)
(where, again, cov is the function you defined above.)
Thanks to unutbu for the explanation. By default numpy.cov calculates the sample covariance. To obtain the population covariance you can specify normalisation by the total N samples like this:
numpy.cov(a, b, bias=True)[0][1]
or like this:
numpy.cov(a, b, ddof=0)[0][1]
Note that starting in Python 3.10, one can obtain the covariance directly from the standard library.
Using statistics.covariance which is a measure (the number you're looking for) of the joint variability of two inputs:
from statistics import covariance
# x = [1, 2, 3, 4, 5, 6, 7, 8, 9]
# y = [1, 2, 3, 1, 2, 3, 1, 2, 3]
covariance(x, y)
# 0.75
Using Python & Numpy, I would like to:
Consider each row of an (n columns x
m rows) matrix as a vector
Weight each row (scalar
multiplication on each component of
the vector)
Add each row to create a final vector
(vector addition).
The weights are given in a regular numpy array, n x 1, so that each vector m in the matrix should be multiplied by weight n.
Here's what I've got (with test data; the actual matrix is huge), which is perhaps very un-Numpy and un-Pythonic. Can anyone do better? Thanks!
import numpy
# test data
mvec1 = numpy.array([1,2,3])
mvec2 = numpy.array([4,5,6])
start_matrix = numpy.matrix([mvec1,mvec2])
weights = numpy.array([0.5,-1])
#computation
wmatrix = [ weights[n]*start_matrix[n] for n in range(len(weights)) ]
vector_answer = [0,0,0]
for x in wmatrix: vector_answer+=x
Even a 'technically' correct answer has been all ready given, I'll give my straightforward answer:
from numpy import array, dot
dot(array([0.5, -1]), array([[1, 2, 3], [4, 5, 6]]))
# array([-3.5 -4. -4.5])
This one is much more on with the spirit of linear algebra (and as well those three dotted requirements on top of the question).
Update:
And this solution is really fast, not marginally, but easily some (10- 15)x faster than all ready proposed one!
It will be more convenient to use a two-dimensional numpy.array than a numpy.matrix in this case.
start_matrix = numpy.array([[1,2,3],[4,5,6]])
weights = numpy.array([0.5,-1])
final_vector = (start_matrix.T * weights).sum(axis=1)
# array([-3.5, -4. , -4.5])
The multiplication operator * does the right thing here due to NumPy's broadcasting rules.