I am using scipy.optimize.minimize to try to determine the optimal parameters of a probability density function (PDF). My PDF involves a discrete Gaussian kernel (https://en.wikipedia.org/wiki/Gaussian_function and https://en.wikipedia.org/wiki/Scale_space_implementation#The_discrete_Gaussian_kernel).
In theory, I know the average value of the PDF (where the PDF should be centered on). So if I were to calculate the expectation value of my PDF, I should recover the mean value that I already know. My PDF is sampled at discrete values of n (which must never be negative and should start at 0 to make any physical sense), and I am trying to determine the optimal value of t (the "scaling factor") to recover the average value of the PDF (which again, I already know ahead of time).
My minimal working example to determine the optimal "scaling factor" t is the following:
#!/usr/bin/env python3
import numpy as np
from scipy.special import iv
from scipy.optimize import minimize
def discrete_gaussian_kernel(t, n):
return np.exp(-t) * iv(n, t)
def expectation_value(t, average):
# One constraint is that the starting value
# of the range over which I sample the PDF
# should be 0.
# Method 1 - This seems to give good, consistent results
int_average = int(average)
ceiling_average = int(np.ceil(average))
N = range(int_average - ceiling_average + 1,
int_average + ceiling_average + 2)
# Method 2 - The multiplicative factor for 'end' is arbitrary.
# I should in principle be able make end be as large as
# I want since the PDF goes to zero for large values of n,
# but this seems to impact the result and I do now know why.
#start = 0
#end = 2 * int(average)
#N = range(start, end)
return np.sum([n * discrete_gaussian_kernel(t, n - average) for n in N])
def minimize_function(t, average):
return average - expectation_value(t, average)
if __name__ == '__main__':
average = 8.33342
#average = 7.33342
solution = minimize(fun = minimize_function,
x0 = 1,
args = average)
print(solution)
t = solution.x[0]
print(' solution t =', t)
print(' given average =', average)
print('recalculated average =', expectation_value(t, average))
I have two problems with my minimal working example:
1) The code works OK for some values of what I choose for the variable "average." One example of this is when the value is 8.33342. However, the code does not work for other values, for example 7.33342. In this case, I get
RuntimeWarning: overflow encountered in exp
so I was thinking that maybe scipy.optimize.minimize is choosing a bad value for t (like a large negative number). I am confident that this is the problem since I have printed out the value of t in the function expectation_value, and t becomes increasingly negative. So I would like to add bounds to the possible values of what "t" can take ("t" should not be negative). Looking at the documentation of scipy.optimize.minimize, there is a bounds keyword argument. So I tried:
solution = minimize(fun = minimize_function,
x0 = 1,
args = average,
bounds = ((0, None)))
but I get the error:
ValueError: length of x0 != length of bounds
I searched for this error on stackoverflow, and there are some other threads, but I did not find any helpful. How can I set a bound successfully?
2) My other question has to do with scipy.optimize.minimize being sensitive to the range over which I calculate the expectation value. For an average value of
average = 8.33342
and the method of calculating the range as
# Method 1 - This seems to give good, consistent results
int_average = int(average)
ceiling_average = int(np.ceil(average))
N = range(int_average - ceiling_average + 1,
int_average + ceiling_average + 2)
the "recalculated average" is 8.3329696426. But for the other method (which has a very similar range),
# Method 2 - The multiplicative factor for 'end' is arbitrary.
# I should in principle be able make end be as large as
# I want since the PDF goes to zero for large values of n,
# but this seems to impact the result and I do now know why.
start = 0
end = 2 * int(average)
N = range(start, end)
the "recalculated average" is 8.31991111857. The ranges are similar in each case, so I don't know why there is such a large change, especially since I what my recalculated average to be as close as possible to the true average. And if I were to extend the range to larger values (which I think is reasonable since the PDF goes to zero there),
start = 0
end = 4 * int(average)
N = range(start, end)
the "recalculated average" is 9.12939372912, which is even worse. So is there a consistent method to calculate the range so that the reconstructed average is always as close as possible to the true average? The scaling factor can take on any value so I would think scipy.optimize.minimize should be able to find a scaling factor to get back the true average exactly.
Related
I have a large data set, statistic, with statistic.shape = (1E10,) that I want to effectively bin (sum) into an array of zeros, out = np.zeros(1E10). Each entry in statistic has a corresponding index, idx, which tells me in which out bin it belongs. The indices are not unique so I cannot use out += statistic[idx] since this will only count the first time a particular index is encountered. Therefore I'm using np.add.at(out, idx, statistic). My problem is that for very large arrays, np.add.at() returns the wrong answer.
Below is an example script that shows this behaviour. The function check_add() should return 1.
import numpy as np
def check_add(N):
N = int(N)
out = np.zeros(N)
np.add.at(out, np.arange(N), np.ones(N))
return np.sum(out)/N
n_arr = [1E3, 1E5, 1E8, 1E10]
for n in n_arr:
print('N = {} (log(N) = {}); output ratio is {}'.format(n, np.log10(n), check_add(n)))
This example returns for me:
N = 1000.0 (log(N) = 3.0); output ratio is 1.0
N = 100000.0 (log(N) = 5.0); output ratio is 1.0
N = 100000000.0 (log(N) = 8.0); output ratio is 1.0
N = 10000000000.0 (log(N) = 10.0); output ratio is 0.1410065408
Can someone explain to me why the function fails for N=1E10?
This is an old bug, NumPy issue 13286. ufunc.at was using a too-small variable for the loop counter. It got fixed a while ago, so update your NumPy. (The fix is present in 1.16.3 and up.)
You're overflowing int32:
1E10 % (np.iinfo(np.int32).max - np.iinfo(np.int32).min + 1) # + 1 for 0
Out[]: 1410065408
There's your weird number (googling that number actually got me to here which is how I figured this out.)
Now, what's happening in your function is a bit more weird. By the documentation of ufunc.at you should just be accumulate-adding the 1 values in the indices that are lower than np.iinfo(np.int32).max and the negative indices above np.iinfo(np.int32).min - but it seems to be 1) working backwards and 2) stopping when it gets to the last overflow. Without digging into the c code I couldn't tell you why, but it's probably a good thing it does - your function would fail silently and with the "correct" mean if it had done things this way, while corrupting your results (having 2 or 3 in those indices and 0 in the middle).
It is most likely due to integer precision indeed. If you play around with the numpy data-type (e.g. you constrain it to an (unsigned) value between 0-255) by setting uint8, you will see that they ratios start declining already for the second array. I do not have enough memory to test it, but setting all dtypes to uint64 as below should help:
def check_add(N):
N = int(N)
out = np.zeros(N,dtype='uint64')
np.add.at(out, np.arange(N,dtype='uint64'), 1)
return np.sum(out)/N
To understand the behavior, I recommend setting dtype='uint8' and checking the behavior for smaller N. So what happens is that the np.arange function creates ascending integers for the vector elements until it reaches the integer limit. It then starts again at 0 and counts up again, so at the beginning (smaller Ns) you get correct sum (although your out vector contains a lot of elements >1 in the positions 0:limit and a lot of elements = 0 beyond the limit). If however you choose N large enough, the elements in your out vector start exceeding the integer limit and start again from 0. As soon as that happens your sum is vastly off. To double-check, realize that the uint8 limit is 255(256 integers) and 256^2=65536. Set N = 65536 with dtype='uint8' and check_add(65536) will return 0.
import numpy as np
def check_add(N):
N = int(N)
out = np.zeros(N,dtype='uint8')
np.add.at(out, np.arange(N,dtype='uint8'), 1)
return np.sum(out)/N
n_arr = [1E1, 1E3, 1E5,65536, 1E7]
for n in n_arr:
print('N = {} (log(N) = {}); output ratio is {}'.format(n, np.log10(n), check_add(n)))
Also note, that you don't need the np.ones vector but can simply replace it by 1, if all you care about is uniformly incrementing everything by 1.
Guessing as I couldn't run it, but could it be a problem that you are exceeding max integer value in python for the last option? Ie exceeds 2147483647.
Use longinteger type instead as per below.
Referring to: [enter link description here][1]https://docs.python.org/2.0/ref/integers.html
Hope this helps. Please let me know if it does work.
I am optimizing the behavior of battery storage combined with solar PV to generate the highest possible revenue stream.
I now want to add one more revenue stream: Peak Shaving (or Demand Charge Reduction)
My approach is as follows:
Next to the price per kWh, an industrial customer pays for the maximal amount of power (kW) he was drawing from the grid in one period (i=1:end), so called demand charges
This maximum amount is found in the vector P_Grid = P_GridLoad (energy self-consumed from the grid) + P_GridBatt (energy used to charge the battery)
There exists a price vector which tells the price per kW for all points in time
I now want to generate a vector P_GridMax that is zero for all points in time but the moment when the maximal value of P_Grid occurs (then it equals max(P_Grid).
Thus, the vector P_GridMax consists of zeros and one nonzero element (not more!)
In doing so, I can now multiply this vector with the price vector, sum up over all points in time and receive the billed demand charges
By including this vector into the objective of my model I can minimize these charges
Now, does anybody see a solution for how to formulate such a constraint (P_GridMax)? I already updated my objective function and defined P_Grid.
Any other approach would also be welcome.
This is the relevant part of my model, with P_xxx = power flow vectors, C_xxx = price vectors, ...
m.P_Grid = Var(m.i_TIME, within = NonNegativeReals)
m.P_GridMax = Var(m.i_TIME, within = NonNegativeReals)
# Minimize electricity bill
def Total_cost(m):
return ... + sum(m.P_GridMax[i] * m.C_PowerCosts[i] for i in m.i_TIME) - ...
m.Cost = Objective(rule=Total_cost)
## Peak Shaving constraints
def Grid_Def(m,i):
return m.P_Grid[i] = m.P_GridLoad[i] + m.P_GridBatt[i]
m.Bound_Grid = Constraint(m.i_TIME,rule=Grid_Def)
def Peak_Rule(m,i):
????
????
????
????
m.Bound_Peak = Constraint(m.i_TIME,rule=Peak_Rule)
Thank you very much in advance! Please be aware that I have very little experience with python/pyomo coding, I would really appreciate you giving extensive explanations :)
Best,
Mathias
Another idea is that you don't actually need to index your P_GridMax variable with time.
If you're dealing with demand costs they tend to be fixed over some period, or in your case it seems that they are fixed over the entire problem horizon (since you're only looking for one max value).
In that case you would just need to do:
m.P_GridMax = pyo.Var(domain=pyo.NonNegativeReals)
def Peak_Rule(m, i):
return m.P_GridMax >= m.P_Grid[i]
m.Bound_Peak = pyo.Constraint(m.i_TIME,rule=Peak_Rule)
if you're really set on multiplying your vectors element-wise you can also just create a new variable that represents that indexed product and apply the same principle to extract the max value.
Here is one way to do this:
introduce a binary helper variable ismax[i] for i in i_TIME. This variable is 1 if the maximum is obtained in period i and 0 otherwise. Then obviously you have a constraint sum(ismax[i] for i in i_TIME) == 1: the maximum must be attained in exactly one period.
Now you need two additional constraints:
if ismax[i] == 0 then P_GridMax[i] == 0.
if ismax[i] == 1 then for all j in i_TIME we must have P_GridMax[i] >= P_GridMax[j].
The best way to formulate this would be to use indicator constraints but I don't know Pyomo so I don't know whether it supports that (I suppose it does but I don't know how to write them). So I'll give instead a big-M formulation.
For this formulation you need to define a constant M so that P_Grid[i] can not exceed that value for any i. With that the first constraint becomes
P_GridMax[i] <= M * ismax[i]
That constraint forces P_GridMax[i] to 0 unless ismax[i] == 1. For ismax[i] == 1 it is redundant.
The second constraint would be for all j in i_TIME
P_GridMax[i] + M * (1 - ismax[i]) >= P_Grid[j]
If ismax[i] == 0 then the left-hand side of this constraint is at least M, so by the definition of M it will be satisfied no matter what the value of P_GridMax[i] is (the first constraint forces P_Grid[i] == 0 in that case). For ismax[i] == 1 the left-hand side of the constraint becomes just P_GridMax[i], exactly what we want.
I want to make Python 3.7.1 pick a number between 0 and 100. However I want a lower number to be much more likely than a higher number, in a reverse exponential-smooth-graduation-curve kind of way (doesn't have to be exact).
I guess I could start with
myrandomnumber = random.randint(0, 100)
And then link that to an array of some sort to determine differing percentages for each number. I've seen other people do that with random die rolls, but the thing is, that's quite neat for only for six possibilities, I want to do this for a hundred (or more) and don't want to sit there making a huge array with a hundred entries just for that. Of course I could do it this way I suppose, but I feel like Python probably has a really easy way to do this that I'm missing.
Thanks, folks!
What you probably want is a gamma distributed random number.
For example with a k=1 and θ=2.0:
There are algorithms for using the evenly-distributed random function to generate normal, exponential, or gamma distributed values.
But since you're in python, you could probably jump straight to using numpy's random.gamma function:
#the (1,2) shape ends basically at 20. Multiply by 5 to get my 0..100 scale
numpy.random.gamma(1, 2.0) * 5
I'm going with the assumption that you want to generate integer values over a bounded range, and that you mean non-uniformly distributed when you talk about "bias". Since you don't have a particular parametric distribution in mind, one approach is to start with a continuous distribution and take the "floor" of the outcomes using int(). You'll want to increase the upper bound by 1 so that rounding down gives values inclusive of that bound.
One easy choice is a triangular distribution. Python provides random.triangular() function, which takes 3 arguments—the lower bound, upper bound, and the mode. Here's a discretized version:
import random as rnd
import math
import sys
def triangle(upper_bound):
return int(rnd.triangular(0.0, float(upper_bound + 1) - sys.float_info.epsilon, 0.0))
I've subtracted float's epsilon from the upper bound to prevent the (extremely unlikely) chance of getting an outcome of 101 when your upper bound is 100. Another bounded distribution choice might be the beta distribution, which you could then scale and truncate.
If you want the distribution shifted even further down the scale towards 0, you could use distributions such as the exponential, or more generally the gamma, with truncation and rounding. Both of those have infinite support, so there are a couple of ways to truncate. The simpler way is to use acceptance/rejection—keep generating values until you get one in range:
def expo_w_rejection(upper_bound, scale_param = 0.4):
upper_bound += 1
while True:
candidate = rnd.expovariate(1.0 / (upper_bound * scale_param))
if candidate < upper_bound:
return int(candidate)
As before, bump the upper limit up by 1 to get outcomes that include the upper limit after truncating. I've also included an optional scale_param which should be a value strictly between 0 and 1, i.e., not inclusive of either limit. Values closer to 0 will cause the results to bunch more to the left, values closer to 1 yield less bunching.
The other method would be to use the inverse transform technique for generating, and to restrict the range of the uniform to not exceed the upper bound based on evaluating the cumulative distribution function at the target upper bound:
def trunc_exp(upper_bound, scale_param = 0.4):
upper_bound = float(upper_bound) + 1.0 - sys.float_info.epsilon
trunc = 1.0 - math.exp(-1.0 / scale_param)
return int((-upper_bound * scale_param) * math.log(1.0 - trunc * rnd.random()))
Both approaches yield distributionally simular results, as can be seen in the following screenshot. "Column 1" was generated with truncated inversion, while "Column 2" was generated with acceptance/rejection.
I was looking at the code for a particular cryptocurrency casino game (EthCrash - if you're interested). The game generates crash points using a function (I call this crash(x)) where x is an integer that is randomly drawn from the space of integers (0,2^52).
I'd like to calculate the expected value of the crash points. The code below should explain everything, but a clean picture of the function is here: https://i.imgur.com/8dPBALa.png, and what I'm trying to calculate is here: https://i.imgur.com/nllykDQ.png (apologies - can't paste pictures yet).
I wrote the following code:
import math
two52 = 2**52
def crash(x):
crash_point = math.floor((100*two52-x)/(two52-x))
return(crash_point/100)
crashes_sum = 0
for i in range(two52+1):
crashes_sum += crash(i)
expected_crash = crashes_sum/two52
Unfortunately, the loop is taking too long to run - any ideas for how I can do this faster?
ok, if you cannot do it straightforward, time to get smart, right?
So idea to get ranges where whole sum could be computed fast. I will put some pseudocode which not even compiles, could have bugs etc. Use it as illustration.
First, lets rewrite the term in the sum as
floor( 100 + 99*x/(252 - x) )
First idea - get ranges where floor is not changing due to the fact that term
n =< 99*x/(252 - x) < n+1. Obviously, for this whole range we could add to sum range_length*(100 + n), no need to do it term by term
sum = 0
r_lo = 0
for k in range(0, 2*52): # LOOP OVER RANGES
r_hi = floor(2**52/(1 + 99/n))
sum += (100 + n -1)*(r_hi - r_lo)
if r_hi-r_lo == 1:
break
r_lo = r_hi + 1
Obviously, range size will shrink till it is equal to 1, and then this method will be useless, we break out. Obviously, by that time each term would be different from previous one by 1 or more.
Ok, second idea - again ranges, where sum is arithmetic series. First we have to find range where increment is equal to 1. Then range where increment is equal to 2, etc. Looks like you have to find roots of quadratic equation for this, but code would be about the same
r_lo = pos_for_increment(1)
t_lo = ... # term at r_lo
for n in range(2, 2*52): # LOOP OVER RANGES
r_hi = pos_for_increment(n) - 1
t_hi = ... # term at r_lo
sum += (t_lo + t_hi)*(r_hi - r_lo) / 2 # arith.series sum
if r_hi > 2**52:
break
r_lo = r_hi + 1
t_lo = t_hi + n
might think about something else, but those tricks are worth trying
Using the map function might help increase the speed since it makes the computation in parallel
import math
two52 = 2**52
def crash(x):
crash_point = math.floor((100*two52-x)/(two52-x))
return(crash_point/100)
crashes_sum = sum(map(crash,range(two52)))
expected_crash = crashes_sum/two52
I have been able to speed up your code by taking advantage of numpy vectorization:
import numpy as np
import time
two52 = 2**52
crash = lambda x: np.floor( ( 100 * two52 - x ) / ( two52 - x ) ) / 100
starttime = time.time()
icur = 0
ispan = 100000
crashes_sum = 0
while icur < two52-ispan:
i = np.arange(icur, icur+ispan, 1)
crashes_sum += np.sum(crash(i))
icur += ispan
crashes_sum += np.sum(crash(np.arange(icur, two52, 1)))
expected_crash = crashes_sum / two52
print(time.time() - starttime)
The trick is to compute the sum on a moving windows to take advantage of numpy's vectorization (written in C). I tried up to 2**30 and it takes 9 seconds on my laptop (and too long for your code to be able to benchmark).
Python is probably not the most suitable language for what you want to do, you may want to try C or Fortran for that (and take advantage of threading).
You will have to use a powerful GPU if you wan't the result within some hours.
A possible CPU implementation
import numpy as np
import numba as nb
import time
two52 = 2**52
loop_to=2**30
#nb.njit(fastmath=True,parallel=True)
def sum_over_crash(two52,loop_to): #loop_to is only for testing performance
crashes_sum = nb.float64(0)
for i in nb.prange(loop_to):#nb.prange(two52+1):
crashes_sum += np.floor((100*two52-i)/(two52-i))/100
return crashes_sum/two52
sum_over_crash(two52,2)#don't measure static compilation overhead
t1=time.time()
sum_over_crash(two52,2**30)
print(time.time()-t1)
This takes 0.57s for on my quadcore i7. eg. 28 days for the whole calculation.
As the calculation can not be minimized mathematically, the only option is to calculate it step by step.
This takes a long time (as stated in other answers). Your best bet on calculating it fast is to use a lower level language than python. Since python is an interpreted language, it is rather slow to calculate this kind of thing.
Additionally you can use multithreading (if availible in the chosen language) to make it even faster.
Cloud Computing is also an option that could be suitable for this, as you are only going to calculate the number once. Amazon and Google (and many more) provide this kind of service for a relatively small fee.
But before performing any of the calculations you need to adjust your formula, as with the way it stands right now, you're going to get a ZeroDivisionError at the very last iteration of your loop.
I'm trying to calculate the error for the Taylor series I've calculated with the following code:
# Define initial values, including appropriate value of x for the series input
import numpy as np
x = -0.9
i = 1
taySum = 0
ln = np.log(1.9)
terms = 1
''' Iterate through the series while checking that
the difference between the obtained series value and ln(1.9)
exceeds 10 digits of accuracy. Stop iterating once the series
value is within 10 digit accuracy of ln(1.9).'''
while (abs(taySum - ln) > 0.5e-10) == True:
taySum += (-1) * (pow(x,i))/(i)
i += 1
terms += 1
print ('value: {}, terms: {}'.format(taySum, terms))
I need to somehow incorporate the error function which calculates the kth derivative and I'm not sure how to do this. The error formula is available at this website which is the following:
There is no way to calculate the error in a taylor series exactly unless you know the exact value it is converging to, which for something like ln 1.9 we don't. The formula you have quoted gives the error in terms of a quantity c < z < x (assuming c < x), which in this case is in the range 0 < z < 0.9, but is otherwise unknown. This means we cannot use this use formula to find the exact error. (otherwise we could find the exact value of ln 1.9, which is not possible)
What this formula does is put bounds on the error. If you look at the formula you will see it has the same form as the next term in the series with the argument of f^(n+1) changed from c, the point you are expanding around, to z, our unknown parameter. In other words it is saying that the error is approximately the same size as the next term in the series, or at least it will if the parameter you are expanding in is small.
With this in mind I would approximate the error by simply computing the next term in the series and saying it is basically that.