There was a phenomenal answer posted by alko for computing a partial derivative of a multivariate function numerically in this thread.
I have a follow-up question now about enhancing this function to accept an array of input values. I have some code where I'm looping through a big long list of n-dimensional points, calculating the partial derivatives with respect to each variable, and this becomes quite computationally expensive.
It's easy enough to vectorize the function in question with np.vectorize, but it causes issues with the partial_derivative wrapper:
from scipy.misc import derivative
import numpy as np
def foo(x, y):
return(x**2 + y**3)
def partial_derivative(func, var=0, point=[]):
args = point[:]
def wraps(x):
args[var] = x
return func(*args)
return derivative(wraps, point[var], dx=1e-6)
vfoo = np.vectorize(foo)
>>>foo(3,1)
>>>10
>>>vfoo([3,3], [1,1])
>>>array([10,10])
>>>partial_derivative(foo,0,[3,1])
>>>6.0
>>>partial_derivative(vfoo,0,[[3,3], [1,1]])
>>>TypeError: can only concatenate list (not "float") to list
The last line should ideally return [6.0, 6.0]. In this case the two arrays supplied to the vectorized function vfoo are essentially zipped up pairwise, so ([3,3], [1,1]) gets transformed into two points, [3,1] and [3,1]. This seems to get mangled when it gets passed to the function wraps. The point that it ends up passing to the function derivative is [3,3]. In addition, there's obviously the TypeError thrown up.
Does anyone have any recommendations or suggestions? Has anyone ever had a need to do something similar?
Edit
Sometimes I think posting on SO is just what it takes to break a mental block. I think I've got it working for anyone who might be interested:
vfoo = np.vectorize(foo)
foo(3,1)
X = np.array([3,3])
Y = np.array([1,1])
vfoo(X, Y)
partial_derivative(foo,0,[3,1])
partial_derivative(vfoo,0,[X, Y])
And the last line now returns array([ 6., 6.])
I have a small problem with args[var] = x : this might forever change args[var] , and all values have been passed by reference however small your change is. So you might not get the exact answer you are looking for. Here is an example:
In[67]: a = np.arange(9).reshape(3,3)
In[68]: b = a[:]
In[69]: b[0,0]=42
In[70]: a
Out[70]:
array([[42, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]])
you need to fix it by e.g.:
def wraps(x):
tmp = args[var]
args[var] = x
ret= func(*args)
args[var] = tmp
return ret
Also, you can use numdifftools. They seem to know what they are doing. This will do all the partial derivatives:
import numpy as np
import numdifftools as nd
def partial_function(f___,input,pos,value):
tmp = input[pos]
input[pos] = value
ret = f___(*input)
input[pos] = tmp
return ret
def partial_derivative(f,input):
ret = np.empty(len(input))
for i in range(len(input)):
fg = lambda x:partial_function(f,input,i,x)
ret[i] = nd.Derivative(fg)(input[i])
return ret
if __name__ == "__main__":
f = lambda x,y: x*x*x+y*y
input = np.array([1.0,1.0])
print ('partial_derivative of f() at: '+str(input))
print (partial_derivative(f,input))
Finally: if you want your function to take an array of the parameters, e.g.:
f = lambda x: x[0]*x[0]*x[0]+x[1]*x[1]
then replace the respective line with (removed the '*')
ret = f___(input)
Related
I have a function f(x,a) where 'x' is a variable and 'a' is a parameter. I want creat a function F(x) that is a sum of f(x,a) for a range of parameter 'a', for instance:
F(x) = f(x,a1) + f(x,a2) + f(x,a3) + ... + f(x,aN) but how I have a large range for 'a' (a=[a1,a2,a3,...,aN]) I want to write a program for this but I don't now how. For instance:
import numpy as np
# Black-Body radiation equation: 'x' is related to frequency and 'a' is related to temperature
def f(x,a):
return x**3/(np.exp(a*x) - 1)
# range for parameter a:
a = [1000,2000,3000,4000,5000,6000]
# Superposition of spectrum
def F(x):
return f(x,a[0]) + f(x,a[1]) + f(x,a[2]) + f(x,a[3]) + f(x,a[4]) + f(x,a[5])
The last line for function F(x) isn't very smart, so I tried make a loop in the above sum with sum() function
def F(x):
spectrum = []
for i in a:
spectrum = sum(f(x,i))
return spectrum
But as I don't have much experience with Python this doesn't work and I got the error:
import matplotlib.pyplot as plt
x = np.linspace(0,100,500)
plt.plot(x,F(x))
plt.show()
# ValueError: x and y must have same first dimension, but have shapes (500,) and (1,)
Does anyone know how to do this? thank you very much
From what i understand, this should do the job:
def F(x):
return sum(f(x, _a) for _a in a)
The thing I do in the sum() function is called list comprehension, feel free to look this Python feature if you are interested by Python coding: it is very powerful.
I am trying to solve a non linear system. Here is the code for a toy problem.
import collections
import numpy as np
import scipy
def flat(x):
''' flattens a shallow list
ex: [[1,2,3],[4,5],[6]] ----> flattens to [1,2,3,4,5]
numpy flatten does not work on lists.
'''
if isinstance(x, collections.Iterable):
return [a for i in x for a in flat(i)]
else:
return [x]
def func(X):
'''setups the matrix dynamic equation and the set of constraints
'''
A = [[0,1,0,1],[2,1,0,4],[1,4,1,3],[3, 2, 1,0]]
A1 = [[1,0,1,-1], [0,-1,2,1],[1,2,0,1],[1,2,0,-2]]
x = X[:-1]
alpha = X[-1]
x0 = [1,2,3,4]
y = x - x0
# x[0] = 0.5
# x[3] = 0.3
dyneqn = np.dot(A,y) + alpha * np.dot(A1,x)
cons = (1/2.0)*np.dot(x.T,np.dot(A1,x)) + np.dot([-1,1,2,-3], x) + 0.5
return flat([dyneqn, cons])
sol = scipy.optimize.root(func,[1,-1,2,0,-1])
sol.x
Problem Statement
The argument X of the objective function f has five unknowns that we are solving for. I want to set the first parameter, i.e., X[0]=0.5and the fourth parameter i.e., X[3] = 0.3 and solve for the remaining 3 unknowns. Let us assume for simplicity that such a solution exists and my initial guess is somehow a good one.
Attempt:
I know I should probably pass these arguments to the args=() argument in scipy.optimize.root. I tried setting
args = (X[0]=0.5, X[3]=0.3)
init_guess = [0.5,-1,2,0.3,-1]
scipy.optimize.root(func,init_guess, args=args)
This is obviously wrong.
Question? How can I fix this?.
Note: I added the flat function so that the code is self contained. It has nothing to do with this question.
Typically with scipy functions like root, minimize, etc
root(func, x0, args=(a, b, c, ...))
requires a func that accepts:
func(x0, a, b, c, ...)
# do something those arguments
return value
x0 is the value that root varies, a,b,c are args value that are passed unchanged to your function. Depending of the problem x0 may be an array. The nature of the args is entirely up to you.
From your example I reconstruct that you want to solve for the second and third component of some vector x as well as the parameter alpha. With the args keyword of scipy.optmize.root that would look something like
def func(x_solve, x0, x3):
#x_solve.size should be 3
x = np.empty(4)
x[0], x[3] = x0, x3
x[1:3] = x_solve[:2]
alpha = x_solve[2]
...
scipy.optimize.root(func, [-1,2,-1], args=(.5, .3))
As Azat and kazemakase pointed out, I'm also not sure if you actually want to use root, but the usage of scipy.optimize.minimize is pretty much the same.
Edit: It should be possible to have a flexible set of fixed variables by using a dictionary as an additional argument which specifies those:
def func(x_solve, fixed):
x = x_solve[:-1] # last value is alpha
for idx in fixed.keys(): # overwrite fixed entries
x[idx] = fixed[idx]
alpha = x_solve[-1]
# fixed variables, key is the index
fixed_vars = {0:.5, 3:.3}
# find roots
scipy.optimize.root(func,
[.5, -1, 2, .3, -1],
args=(fixed_vars,))
That way, when the optimizer in root numerically evaluates the Jacobian it obtains zero for the fixed variables and should therefore leave those invariant. However, that might lead to complications in the convergence of the algorithm.
Curve fitting tools such as those in scipy tend to assume that the parameters of the model functions are real-valued.
When fitting a model that depends on complex-valued parameters to a complex-valued data set, one therefore first has to create a version of the model, in which each complex parameter is replaced by two real ones.
First, a simple example:
# original function, representing a model where a,b may be complex-valued
def f(x, a, b):
return a+b*x
# modified function, complex parameters have been replaced by two real ones
def f_r(x, a_r, a_i, b_r, b_i):
return f(x, a_r + 1J*a_i, b_r+1J*b_i)
print( f(1,2+3J,4+5J) == f_r(1,2,3,4,5) )
Note: The output of the model is still complex-valued, but this can easily be taken care of by appropriately defining the residual function.
Now, instead of having to write new code for every function f, I would like to have a "function factory" to which I pass the function object f together with a list of booleans is_complex specifying which arguments of f are to be assumed complex-valued (and therefore need to be replaced by two real-valued arguments).
This list of booleans could e.g. be inferred from the initial values provided together with f.
I am new to this kind of problem, so I looked around on the web and came across the decorator module. Before going to the generic case, here is the example from above using the Functionmaker class:
import decorator
def f(x, a, b):
return a+b*x
f_r = decorator.FunctionMaker.create(
'f_r(x, a_r, a_i, b_r, b_i)',
'return f(x, a_r + 1J*a_i, b_r + 1J*b_i)',
dict(f=f))
For the generic case, one can now imagine to synthesize the two strings that are passed to the function maker:
import decorator
import inspect
def f(x, a, b):
return a+b*x
def fmaker(f,is_complex):
argspec = inspect.getargspec(f)
args = argspec.args[:]
fname = f.func_name
s1 = "{}_r(".format(fname)
s2 = "return f("
for arg, cplx in zip(args, is_complex):
if not cplx:
s1 += "{},".format(arg)
s2 += "{},".format(arg)
else:
s1 += "{}_r,".format(arg)
s1 += "{}_i,".format(arg)
s2 += "{}_r+1J*{}_i,".format(arg,arg)
s1 += ')'
s2 += ')'
return decorator.FunctionMaker.create(s1,s2,dict(f=f))
is_complex = [False, True, True]
f_r = fmaker(f,is_complex)
# prints ArgSpec(args=['x', 'a_r', 'a_i', 'b_r', 'b_i'], varargs=None, keywords=None, defaults=())
print(inspect.getargspec(f_r))
print( f(1,2+3J,4+5J) == f_r(1,2,3,4,5) )
This seems to solve the problem.
My question is: is this a reasonable way of doing this? Are there better/simpler ways in python?
P.S. I am not a computer scientist, so if I am using technical terms incorrectly, please feel free to revise.
You do not have to do any nasty string based generation, you can simply use basic function closures to create a wrapper:
def complex_unroll(f, are_complex):
# This function will have access to are_complex and f through python closure
# *args give us access to all parameters as a list
def g(*args, **kwargs):
# new_args stores new list of parameters, the complex ones
new_args = []
# arg_id is iterator used to keep track where are we in the original list
arg_id = 0
for is_complex in are_complex:
if is_complex:
# if we request complex unroll, we merge two consequtive params
new_args.append(args[arg_id] + 1J*args[arg_id+1])
# and move iterator 2 slots
arg_id += 2
else:
# otherwise, just copy the argument
new_args.append(args[arg_id])
arg_id += 1
# finally we return a call to original function f with new args
return f(*new_args, **kwargs)
# our unroll function returns a newly designed function g
return g
And now
def f(x, a, b):
return a+b*x
def f_r(x, a_r, a_i, b_r, b_i):
return f(x, a_r + 1J*a_i, b_r+1J*b_i)
f_u = complex_unroll(f, [False, True, True])
print f(1,2+3J,4+5J)
print f_r(1,2,3,4,5)
print f_u(1,2,3,4,5)
f_u2 = complex_unroll(f, [True, True, True])
print f_u2(1,0,2,3,4,5)
Works as desired.
Why I would prefer this path as compared to the proposed one in the question?
It does not use any additional modules/libraries, just a very basic mechanism of python's dealing with arguments and closures. In particular your solution does reflection, it analyzes the defined function, which is quite complex operation as compared to what you try to obtain.
It handles named arguments just fine, so if you have f(x, a, b, flag), you can still just use g = complex_unroll(f, [False, True, True]) and call g(0, 0, 0, 0, 0, flag = True), which would fail in your code. You could add support for this, though.
I am now programming on BFGS algorithm, where I need to create a function with a doulbe sum. I need to return a FUNCTION but not a number, so something like sum+= is not acceptable.
def func(X,W):
return a function of double sum of X, W
A illustrative example:
X = np.array([[1,1,1,1],[2,2,2,2],[3,3,3,3],[4,4,4,4],[5,5,5,5]])
W = np.array([[1,1,1,1],[2,2,2,2],[3,3,3,3]])
I want to get a function that, for each instance X[i] in X, and for each W[j] in W, return a function of the sum of numpy.dot(X[i],W[j]). For example, X[1] dot W[2] shoulde be 2*3+2*3+2*3+2*3
----------This contend is edited by me:-------------
When I saw the answers provided below, I think my question is not clear enough. Actually, I want to get a function:
Func = X[0]W[0]+X[0]W[1]+X[0]W[2]+ X[1]W[0]+X[1]W[1]+X[1]W[2]+
X[2]W[0]+X[2]W[1]+X[2]W[2]+ X[3]W[0]+X[3]W[1]+X[3]W[2] +
X[4]W[0]+X[4]W[1]+X[4]W[2]
-------------------end the edited content--------------
If I only got one dimension of W, the problem is easy by using numpy.sum(X,W).
However, how can I return a function of two sums with Python?
If you want to return the function f(i,j) -> X[i].W[j] :
def func(X,W):
def f(i,j): return np.dot(X[i],W[j])
return f
will work.
EDIT:
The VALUE you name Func in your edit is computed by sum([np.dot(x,w) for x in X for w in W]) or, more efficient, np.einsum('ij,kj->',X,W) .
if you want to return the FUNCTION that return Func, you can do it like that :
def func(X,W):
Func=np.einsum('ij,kj->',X,W)
return lambda : Func
Then f=func(X,W); print(f()) will print 360, the value named Func in your example.
If I got your question right, this should do exactly what you want (python-2.7):
import numpy as np
def sample_main():
X = np.array([[1,1,1,1],[2,2,2,2],[3,3,3,3],[4,4,4,4],[5,5,5,5]])
W = np.array([[1,1,1,1],[2,2,2,2],[3,3,3,3]])
f = lambda i, j : reduce (lambda a, b: a+b, map(lambda x, w: x*w, X[i], W[j]), 0)
return f
if __name__ == '__main__':
f = sample_main()
print (f(0, 0))
Just replace the sample_main function with your function that takes X and W.
Actually, I want to implement L_BFGS algorithm in my Python code. Inspired by the two answers provided by #B.M. and #siebenschlaefer, I figure out how to implement in my code:
func = np.sum(np.sum(log_p_y_xz(Y[i][t], Z[i], sigma_eta_ti(X[i],w[t],gamma[t]))+log_p_z_x(alpha, beta, X[i]) for t in range(3)) for i in range (5))
Please do not mind the details of the formula, what I want to say is that, I use two sum here and just using i in rage (5) and t in range (3) to tell the code do the sums.
Thanks again for the answers provided by #B.M. and #siebenschlaefer!!
Let's suppose that we have a Python function that takes in Numpy arrays and returns another array:
import numpy as np
def f(x, y, method='p'):
"""Parameters: x (np.ndarray) , y (np.ndarray), method (str)
Returns: np.ndarray"""
z = x.copy()
if method == 'p':
mask = x < 0
else:
mask = x > 0
z[mask] = 0
return z*y
although the actual implementation does not matter. We can assume that x and y will always be arrays of the same shape, and that the output is of the same shape as x.
The question is what would be the simplest/most elegant way of wrapping such function so it would work with both ND arrays (N>1) and scalar arguments, in a manner somewhat similar to universal functions in Numpy.
For instance, the expected output for the above function should be,
In [1]: f_ufunc(np.arange(-1,2), np.ones(3), method='p')
Out[1]: array([ 0., 0., 1.]) # random array input -> output of the same shape
In [2]: f_ufunc(np.array([1]), np.array([1]), method='p')
Out[2]: array([1]) # array input of len 1 -> output of len 1
In [3]: f_ufunc(1, 1, method='p')
Out[3]: 1 # scalar input -> scalar output
The function f cannot be changed, and it will fail if given a scalar argument for x or y.
When x and y are scalars, we transform them to 1D arrays, do the calculation then transform them back to scalars at the end.
f is optimized to work with arrays, scalar input being mostly a convenience. So writing a function that work with scalars and then using np.vectorize or np.frompyfunc would not be acceptable.
A beginning of an implementation could be,
def atleast_1d_inverse(res, x):
# this function fails in some cases (see point 1 below).
if res.shape[0] == 1:
return res[0]
else:
return res
def ufunc_wrapper(func, args=[]):
""" func: the wrapped function
args: arguments of func to which we apply np.atleast_1d """
# this needs to be generated dynamically depending on the definition of func
def wrapper(x, y, method='p'):
# we apply np.atleast_1d to the variables given in args
x = np.atleast_1d(x)
y = np.atleast_1d(x)
res = func(x, y, method='p')
return atleast_1d_inverse(res, x)
return wrapper
f_ufunc = ufunc_wrapper(f, args=['x', 'y'])
which mostly works, but will fail the tests 2 above, producing a scalar output instead of a vector one. If we want to fix that, we would need to add more tests on the type of the input (e.g. isinstance(x, np.ndarray), x.ndim>0, etc), but I'm afraid to forget some corner cases there. Furthermore, the above implementation is not generic enough to wrap a function with a different number of arguments (see point 2 below).
This seems to be a rather common problem, when working with Cython / f2py function, and I was wondering if there was a generic solution for this somewhere?
Edit: a bit more precisions following #hpaulj's comments. Essentially, I'm looking for
a function that would be the inverse of np.atleast_1d, such as
atleast_1d_inverse( np.atleast_1d(x), x) == x, where the second argument is only used to determine the type or the number of dimensions of the original object x. Returning numpy scalars (i.e. arrays with ndim = 0) instead of a python scalar is ok.
A way to inspect the function f and generate a wrapper that is consistent with its definition. For instance, such wrapper could be used as,
f_ufunc = ufunc_wrapper(f, args=['x', 'y'])
and then if we have a different function def f2(x, option=2): return x**2, we could also use
f2_ufunc = ufunc_wrapper(f2, args=['x']).
Note: the analogy with ufuncs might be a bit limited, as this corresponds to the opposite problem. Instead of having a scalar function that we transform to accept both vector and scalar input, I have a function designed to work with vectors (that can be seen as something that was previously vectorized), that I would like to accept scalars again, without changing the original function.
This doesn't fully answer the question of making a vectorized function truly behave like a ufunc, but I did recently run into a slight annoyance with numpy.vectorize that sounds similar to your issue. That wrapper insists on returning an array (with ndim=0 and shape=()) even if passed scalar inputs.
But it appears that the following does the right thing. In this case I am vectorizing a simple function to return a floating point value to a certain number of significant digits.
def signif(x, digits):
return round(x, digits - int(np.floor(np.log10(abs(x)))) - 1)
def vectorize(f):
vf = np.vectorize(f)
def newfunc(*args, **kwargs):
return vf(*args, **kwargs)[()]
return newfunc
vsignif = vectorize(signif)
This gives
>>> vsignif(0.123123, 2)
0.12
>>> vsignif([[0.123123, 123.2]], 2)
array([[ 0.12, 120. ]])
>>> vsignif([[0.123123, 123.2]], [2, 1])
array([[ 0.12, 100. ]])