Im translating some matlab code to python code and debugging both codes i get a different result from a call to the cumtrapz function, i also verified that the input data of both is similar. This are the codes:
Python Code
from numpy import zeros, ceil, array, mean, ptp, abs, sqrt, power
from scipy.integrate import cumtrapz
def step_length_vector(ics_y, fcs_y, acc_y, l, sf):
step_length_m1 = zeros(int(ceil(len(ics_y)/2))-1)
for i in range(0, len(ics_y)-2, 2):
av = acc_y[int(ics_y[i]):int(ics_y[i+2])+1]
t = array(range(1, int((ics_y[i+2]-ics_y[i])+2)))/sf
hvel = cumtrapz(t, av - mean(av), initial=0)
h = cumtrapz(t, hvel - mean(hvel), initial=0)
hend = ptp(h)
sl = 6.8*(sqrt(abs(2*l*hend - hend**2)))
step_length_m1[int(ceil(i/2))] = sl
return step_length_m1
Matlab Code
function [StepLengthM1] = StepLengthVector(ICsY,FCsY,ACCY,l,sf)
StepLengthM1 = zeros(1,ceil(length(ICsY)/2)-1);
for i= 1:2:length(ICsY)-2
av = ACCY(ICsY(i):ICsY(i+2));
t = (1:(ICsY(i+2)-ICsY(i))+1)/sf;
hvel = cumtrapz(t,av-mean(av));
h = cumtrapz(t,hvel-mean(hvel));
hend = peak2peak(h);
sl = 6.8*(sqrt(abs(2*l*hend - hend.^2)));
StepLengthM1(ceil(i/2)) = sl;
end
end
The hvel variable is different for both codes. Maybe im using wrong the scipy cumtrapz because i asume that the initial value that recives is 0. In both cases the inputs ics_y(ICsy), fcs_y(FCsY), acc_y(ACCY) are one dimensional arrays and l and sf are scalars.
Thanks!!!
(If this question is about cumtrapz, you should simplify your tests to just a single call to cumtrapz with the same input arrays in matlab and Python. Also, be sure you read the matlab and SciPy documentation of each function carefully. The SciPy functions are typically not exact duplicates of the corresponding matlab function.)
The problem is that when you give both x and y values, the order in which they are given in matlab/octave is x, y, but in the SciPy version, it is y, x.
For example,
octave:11> t = [0 1 1.5 4 4.5 6]
t =
0.00000 1.00000 1.50000 4.00000 4.50000 6.00000
octave:12> y = [1 2 3 -2 0 1]
y =
1 2 3 -2 0 1
octave:13> cumtrapz(t, y)
ans =
0.00000 1.50000 2.75000 4.00000 3.50000 4.25000
To get the same result with scipy.integrate.cumtrapz:
In [22]: from scipy.integrate import cumtrapz
In [23]: t = np.array([0, 1, 1.5, 4, 4.5, 6])
In [24]: y = np.array([1, 2, 3, -2, 0, 1])
In [25]: cumtrapz(y, t, initial=0)
Out[25]: array([0. , 1.5 , 2.75, 4. , 3.5 , 4.25])
Related
I have this simple piece of Matlab code below I like to convert over to Python.What is the correct Python syntax for the last line? I have also posted my Python attempt below this little snippet.
%variables
x=\[1 2 3 4 5 6\]
l_Vector =5
memlen = 2
deg =2
n= 1
k=1
% create 5 x 4
x_val = np.zeros( (l_Vector,memlen\*deg) )
x_val(n,(k - 1)*memlen+(1:memlen)) = (x(n:(n + memlen-1)).*(abs(x(n:(n + memlen -1))).^(k - 1))).';
type here
I tried the following : breaking the last line down.
from scipy import linalg
import numpy as np
x= np.array([1,2,3,4,5,6])
l_Vector =5
memlen = 2
deg =2
x_terms = np.zeros( (l_Vector,memlendeg) )
term_1 = x[n:(n+memorylen)]
term_2 = np.abs(x[n:(n+memorylen)])**k
pp = np.multiply(term_1 ,term_2)
x_terms[n,(k)(memorylen) ,2] = pp.T
I'm trying to learn Python for basic work in linear algebra. I'm running into the following problem with a simple system of linear equations:
import scipy.linalg as la
import numpy as np
A = np.array([[186/450, 54/21, 30/60],
[12/450, 6/21 , 3/60],
[9/450, 6/21 , 15/60]])
l = np.array([18/450, 12/21, 30/36])
b = np.array([2, 0, 1/6])
y = np.array([180, 0, 30])
x = la.inv(np.eye(3) - A) # y
lam = np.transpose(l) # la.inv(np.eye(3) - A)
This returns
array([0.21212121, 2.12121212, 1.39393939])
which is incorrect. Performing the same operation in Julia,
A = [186/450 54/21 30/60;
12/450 6/21 3/60;
9/450 6/21 15/60]
l = [18/450, 12/21, 30/60]
b = [2, 0, 1/6]
y = [180, 0, 30]
λ = l' * inv(I - A)
yields the correct result, which is
1×3 adjoint(::Vector{Float64}) with eltype Float64:
0.181818 1.81818 0.909091
What am I missing here? I think I might be missing something in the opaque numpy array syntax.
There is a typo in l instantiation in your python code. (30/36 should be 30/60).
This code with the typo fixed produces the same result as in Julia.
import scipy.linalg as la
import numpy as np
A = np.array([[186/450, 54/21, 30/60],
[12/450, 6/21 , 3/60],
[9/450, 6/21 , 15/60]])
l = np.array([18/450, 12/21, 30/60]) #typo fixed here
b = np.array([2, 0, 1/6])
y = np.array([180, 0, 30])
x = la.inv(np.eye(3) - A) # y
lam = np.transpose(l) # la.inv(np.eye(3) - A)
Giving:
array([0.18181818, 1.81818182, 0.90909091])
I'm working on a custom likelihood function for Theano (Attempting to fit a conditional logistic regression.)
The likelihood requires summing values by group. In R we have the "ave()" function, in Python Pandas we have "groupby()". How would I do something similar in Theano?
Edited for more detail
I want to create a cox proportional hazards model (same as conditional logistic regression.) The log likelihood requires the sum of values by group:
In Pandas, this would be:
temp = df.groupby('groupid')['eta'].aggregate(np.sum)
denominator = np.log(temp).sum()
In the data, we have a column with group ID, and the values to be summed
group eta
1 2.1
1 1.8
1 0.9
2 1.2
2 0.75
2 1.42
The output for the group sums would then be:
group sum
1 4.8
2 3.37
Then, the sum of the log of the sums:
log(4.8) + log(3.37) = 2.7835
This is quick and easy to do in Pandas. How can I do something similar in Thano? Sure, could write a nexted loop, but that seems slow and I try to avoid manually coded loops when possible as they are slow.
Thanks!
Let say you have "X" (a list of all your etas), with the dim. Nx1 (I guess) and a matrix H. H is a NxG matrix that has a on-hot encoding of the groups.
The you you write something like:
import numpy as np
from numpy import newaxis as na
import theano.tensor as T
X = T.vector()
H = T.matrix()
tmp = T.sum(X[:, na] * H, axis=0)
O = T.sum(T.log(tmp))
x = np.array([5, 10, 10, 0.5, 5, 0.5])
# create a 1-hot encoding
g = np.array([1, 2, 2, 0, 1, 0])
h = np.zeros(shape=(len(x), 3))
for i,j in enumerate(g):
h[i,j] = 1.0
O.eval({X:x, H: h})
This should work as long as there is at least one eta per point (or else -inf).
The scipy.fftpack.rfft function returns the DFT as a vector of floats, alternating between the real and complex part. This means to multiply to DFTs together (for convolution) I will have to do the complex multiplication "manually" which seems quite tricky. This must be something people do often - I presume/hope there is a simple trick to do this efficiently that I haven't spotted?
Basically I want to fix this code so that both methods give the same answer:
import numpy as np
import scipy.fftpack as sfft
X = np.random.normal(size = 2000)
Y = np.random.normal(size = 2000)
NZ = np.fft.irfft(np.fft.rfft(Y) * np.fft.rfft(X))
SZ = sfft.irfft(sfft.rfft(Y) * sfft.rfft(X)) # This multiplication is wrong
NZ
array([-43.23961083, 53.62608086, 17.92013729, ..., -16.57605207,
8.19605764, 5.23929023])
SZ
array([-19.90115323, 16.98680347, -8.16608202, ..., -47.01643274,
-3.50572376, 58.1961597 ])
N.B. I am aware that fftpack contains a convolve function, but I only need to fft one half of the transform - my filter can be fft'd once in advance and then used over and over again.
You don't have to flip back to np.float64 and hstack. You can create an empty destination array, the same shape as sfft.rfft(Y) and sfft.rfft(X), then create a np.complex128 view of it and fill this view with the result of the multiplication. This will automatically fill the destination array as wanted.
If I retake your example :
import numpy as np
import scipy.fftpack as sfft
X = np.random.normal(size = 2000)
Y = np.random.normal(size = 2000)
Xf = np.fft.rfft(X)
Xf_cpx = Xf[1:-1].view(np.complex128)
Yf = np.fft.rfft(Y)
Yf_cpx = Yf[1:-1].view(np.complex128)
Zf = np.empty(X.shape)
Zf_cpx = Zf[1:-1].view(np.complex128)
Zf[0] = Xf[0]*Yf[0]
# the [...] is important to use the view as a reference to Zf and not overwrite it
Zf_cpx[...] = Xf_cpx * Yf_cpx
Zf[-1] = Xf[-1]*Yf[-1]
Z = sfft.irfft.irfft(Zf)
and that's it!
You can use a simple if statement if you want your code to be more general and handle odd lengths as explained in Jaime's answer.
Here is a function that does what you want:
def rfft_mult(a,b):
"""Multiplies two outputs of scipy.fftpack.rfft"""
assert a.shape == b.shape
c = np.empty( a.shape )
c[...,0] = a[...,0]*b[...,0]
# To comply with the rfft support of multi dimensional arrays
ar = a.reshape(-1,a.shape[-1])
br = b.reshape(-1,b.shape[-1])
cr = c.reshape(-1,c.shape[-1])
# Note that we cannot use ellipses to achieve that because of
# the way `view` work. If there are many dimensions, one should
# consider to manually perform the complex multiplication with slices.
if c.shape[-1] & 0x1: # if odd
for i in range(len(ar)):
ac = ar[i,1:].view(np.complex128)
bc = br[i,1:].view(np.complex128)
cc = cr[i,1:].view(np.complex128)
cc[...] = ac*bc
else:
for i in range(len(ar)):
ac = ar[i,1:-1].view(np.complex128)
bc = br[i,1:-1].view(np.complex128)
cc = cr[i,1:-1].view(np.complex128)
cc[...] = ac*bc
c[...,-1] = a[...,-1]*b[...,-1]
return c
You can take a view of a slice of your return array, e.g.:
>>> scipy.fftpack.fft(np.arange(8))
array([ 28.+0.j , -4.+9.65685425j, -4.+4.j ,
-4.+1.65685425j, -4.+0.j , -4.-1.65685425j,
-4.-4.j , -4.-9.65685425j])
>>> a = scipy.fftpack.rfft(np.arange(8))
>>> a
array([ 28. , -4. , 9.65685425, -4. ,
4. , -4. , 1.65685425, -4. ])
>>> a.dtype
dtype('float64')
>>> a[1:-1].view(np.complex128) # First and last entries are real
array([-4.+9.65685425j, -4.+4.j , -4.+1.65685425j])
You will need to handle even or odd sized FFTs differently:
>>> scipy.fftpack.fft(np.arange(7))
array([ 21.0+0.j , -3.5+7.26782489j, -3.5+2.79115686j,
-3.5+0.79885216j, -3.5-0.79885216j, -3.5-2.79115686j,
-3.5-7.26782489j])
>>> a = scipy.fftpack.rfft(np.arange(7))
>>> a
array([ 21. , -3.5 , 7.26782489, -3.5 ,
2.79115686, -3.5 , 0.79885216])
>>> a.dtype
dtype('float64')
>>> a[1:].view(np.complex128)
array([-3.5+7.26782489j, -3.5+2.79115686j, -3.5+0.79885216j])
i am having 2 arrays (G and G_). They have the same shape and size and i want to convolve them. i found the numpy.convolve and fftconvolve.
My Code is like:
foldedX = getFoldGradientsFFT(G, G_)
foldedY = getFoldGradientsNumpy(G, G_)
def getFoldGradientsFFT(G, G_):
# convolve via scipy fast fourier transform
X =signal.fftconvolve(G,G_, "same)
X*=255.0/numpy.max(X);
return X
def getFoldGradientsNumpy(G, G_):
# convolve via numpy.convolve
Y = ndimage.convolve(G, G_)
Y*=255.0/numpy.max(Y);
return Y
But the results aren't the same.
The result is like:
Numpy.concolve()
[ 11.60287582 3.28262652 18.80395211 52.75829556 99.61675945
147.74124258 187.66178244 215.06160439 234.1907606 229.04221552]
scipy.signal.fftconvolve:
[ -4.88130620e-15 6.74371119e-02 4.91875539e+00 1.94250997e+01
3.88227012e+01 6.70322921e+01 9.78460423e+01 1.08486302e+02
1.17267015e+02 1.15691562e+02]
I thought the result is supposed to be the same, even if the two functions convolves with a different procedure?!
i forgot to mention, that i want to convolve 2 2-dimensional arrays :S
the arrays:
G = array([[1,2],[3,4]])
G_ = array([[5,6],[7,8]])
the code
def getFoldGradientsFFT(G, G_):
X =signal.fftconvolve(G,G_,"same")
X=X.astype("int")
X*=255.0/np.max(X);
return X
def getFoldGradientsNumpy(G, G_):
# convolve via convolve
old_shape = G.shape
G = np.reshape(G, G.size)
G_ = np.reshape(G_, G.size)
Y = np.convolve(G, G_, "same")
Y = np.reshape(Y,old_shape)
Y = Y.astype("int")
Y*=255.0/np.max(Y);
return Y
def getFoldGradientsNDImage(G, G_):
Y = ndimage.convolve(G, G_)
Y = Y.astype("int")
Y *= 255.0/np.max(Y)
return Y
the results:
getFoldGradientsFFT
[[ 21 68]
[ 93 255]]
getFoldGradientsNumpy
[[ 66 142]
[250 255]]
getFoldGradientsNDImage
[[147 181]
[220 255]]
numpy.convolve is for one-dimensional data.
The following code compares the results of signal.convolve, signal.fftconvolve, and ndimage.convolve.
for ndimage.convolve, we need to set mode argument to "constant", and origin argument to -1 when N is even, and 0 when N is odd.
from scipy import signal
from scipy import ndimage
import numpy as np
np.random.seed(1)
for N in xrange(2, 20):
a = np.random.randint(0, 10, size=(N, N))
b = np.random.randint(0, 10, size=(N, N))
r1 = signal.convolve(a, b, mode="same")
r2 = signal.fftconvolve(a, b, mode="same")
r3 = ndimage.convolve(a, b, mode="constant", origin=-1 if N%2==0 else 0)
print "N=", N
print np.allclose(r1, r2)
print np.allclose(r2, r3)
getFoldGradientsNumpy is using scipy.ndimage.convolve. That does multi-dimensional convolution and is not the same as scipy.convolve.
For me, when convolving two one-dimensional arrays, scipy.convolve, scipy.signal.convolve, and scipy.signal.fftconvolve all return the same answer.