It's pretty easy to write a function that computes the maximum drawdown of a time series. It takes a small bit of thinking to write it in O(n) time instead of O(n^2) time. But it's not that bad. This will work:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
def max_dd(ser):
max2here = pd.expanding_max(ser)
dd2here = ser - max2here
return dd2here.min()
Let's set up a brief series to play with to try it out:
np.random.seed(0)
n = 100
s = pd.Series(np.random.randn(n).cumsum())
s.plot()
plt.show()
As expected, max_dd(s) winds up showing something right around -17.6. Good, great, grand. Now say I'm interested in computing the rolling drawdown of this Series. I.e. for each step, I want to compute the maximum drawdown from the preceding sub series of a specified length. This is easy to do using pd.rolling_apply. It works like so:
rolling_dd = pd.rolling_apply(s, 10, max_dd, min_periods=0)
df = pd.concat([s, rolling_dd], axis=1)
df.columns = ['s', 'rol_dd_10']
df.plot()
This works perfectly. But it feels very slow. Is there a particularly slick algorithm in pandas or another toolkit to do this fast? I took a shot at writing something bespoke: it keeps track of all sorts of intermediate data (locations of observed maxima, locations of previously found drawdowns) to cut down on lots of redundant calculations. It does save some time, but not a whole lot, and not nearly as much as should be possible.
I think it's because of all the looping overhead in Python/Numpy/Pandas. But I'm not currently fluent enough in Cython to really know how to begin attacking this from that angle. I was hoping someone had tried this before. Or, perhaps, that someone might want to have a look at my "handmade" code and be willing to help me convert it to Cython.
Edit:
For anyone who wants a review of all the functions mentioned here (and some others!) have a look at the iPython notebook at: http://nbviewer.ipython.org/gist/8one6/8506455
It shows how some of the approaches to this problem relate, checks that they give the same results, and shows their runtimes on data of various sizes.
If anyone is interested, the "bespoke" algorithm I alluded to in my post is rolling_dd_custom. I think that could be a very fast solution if implemented in Cython.
Here's a numpy version of the rolling maximum drawdown function. windowed_view is a wrapper of a one-line function that uses numpy.lib.stride_tricks.as_strided to make a memory efficient 2d windowed view of the 1d array (full code below). Once we have this windowed view, the calculation is basically the same as your max_dd, but written for a numpy array, and applied along the second axis (i.e. axis=1).
def rolling_max_dd(x, window_size, min_periods=1):
"""Compute the rolling maximum drawdown of `x`.
`x` must be a 1d numpy array.
`min_periods` should satisfy `1 <= min_periods <= window_size`.
Returns an 1d array with length `len(x) - min_periods + 1`.
"""
if min_periods < window_size:
pad = np.empty(window_size - min_periods)
pad.fill(x[0])
x = np.concatenate((pad, x))
y = windowed_view(x, window_size)
running_max_y = np.maximum.accumulate(y, axis=1)
dd = y - running_max_y
return dd.min(axis=1)
Here's a complete script that demonstrates the function:
import numpy as np
from numpy.lib.stride_tricks import as_strided
import pandas as pd
import matplotlib.pyplot as plt
def windowed_view(x, window_size):
"""Creat a 2d windowed view of a 1d array.
`x` must be a 1d numpy array.
`numpy.lib.stride_tricks.as_strided` is used to create the view.
The data is not copied.
Example:
>>> x = np.array([1, 2, 3, 4, 5, 6])
>>> windowed_view(x, 3)
array([[1, 2, 3],
[2, 3, 4],
[3, 4, 5],
[4, 5, 6]])
"""
y = as_strided(x, shape=(x.size - window_size + 1, window_size),
strides=(x.strides[0], x.strides[0]))
return y
def rolling_max_dd(x, window_size, min_periods=1):
"""Compute the rolling maximum drawdown of `x`.
`x` must be a 1d numpy array.
`min_periods` should satisfy `1 <= min_periods <= window_size`.
Returns an 1d array with length `len(x) - min_periods + 1`.
"""
if min_periods < window_size:
pad = np.empty(window_size - min_periods)
pad.fill(x[0])
x = np.concatenate((pad, x))
y = windowed_view(x, window_size)
running_max_y = np.maximum.accumulate(y, axis=1)
dd = y - running_max_y
return dd.min(axis=1)
def max_dd(ser):
max2here = pd.expanding_max(ser)
dd2here = ser - max2here
return dd2here.min()
if __name__ == "__main__":
np.random.seed(0)
n = 100
s = pd.Series(np.random.randn(n).cumsum())
window_length = 10
rolling_dd = pd.rolling_apply(s, window_length, max_dd, min_periods=0)
df = pd.concat([s, rolling_dd], axis=1)
df.columns = ['s', 'rol_dd_%d' % window_length]
df.plot(linewidth=3, alpha=0.4)
my_rmdd = rolling_max_dd(s.values, window_length, min_periods=1)
plt.plot(my_rmdd, 'g.')
plt.show()
The plot shows the curves generated by your code. The green dots are computed by rolling_max_dd.
Timing comparison, with n = 10000 and window_length = 500:
In [2]: %timeit rolling_dd = pd.rolling_apply(s, window_length, max_dd, min_periods=0)
1 loops, best of 3: 247 ms per loop
In [3]: %timeit my_rmdd = rolling_max_dd(s.values, window_length, min_periods=1)
10 loops, best of 3: 38.2 ms per loop
rolling_max_dd is about 6.5 times faster. The speedup is better for smaller window lengths. For example, with window_length = 200, it is almost 13 times faster.
To handle NA's, you could preprocess the Series using the fillna method before passing the array to rolling_max_dd.
For the sake of posterity and for completeness, here's what I wound up with in Cython. MemoryViews materially sped things up. There was a bit of work to do to make sure I'd properly typed everything (sorry, new to c-type languages). But in the end I think it works nicely. For typical use cases, the speedup vs regular python was ~100x or ~150x. The function to call is cy_rolling_dd_custom_mv where the first argument (ser) should be a 1-d numpy array and the second argument (window) should be a positive integer. The function returns a numpy memoryview, which works well enough in most cases. You can explicitly call np.array(result) if you need to to get a nice array of the output:
import numpy as np
cimport numpy as np
cimport cython
DTYPE = np.float64
ctypedef np.float64_t DTYPE_t
#cython.boundscheck(False)
#cython.wraparound(False)
#cython.nonecheck(False)
cpdef tuple cy_dd_custom_mv(double[:] ser):
cdef double running_global_peak = ser[0]
cdef double min_since_global_peak = ser[0]
cdef double running_max_dd = 0
cdef long running_global_peak_id = 0
cdef long running_max_dd_peak_id = 0
cdef long running_max_dd_trough_id = 0
cdef long i
cdef double val
for i in xrange(ser.shape[0]):
val = ser[i]
if val >= running_global_peak:
running_global_peak = val
running_global_peak_id = i
min_since_global_peak = val
if val < min_since_global_peak:
min_since_global_peak = val
if val - running_global_peak <= running_max_dd:
running_max_dd = val - running_global_peak
running_max_dd_peak_id = running_global_peak_id
running_max_dd_trough_id = i
return (running_max_dd, running_max_dd_peak_id, running_max_dd_trough_id, running_global_peak_id)
#cython.boundscheck(False)
#cython.wraparound(False)
#cython.nonecheck(False)
def cy_rolling_dd_custom_mv(double[:] ser, long window):
cdef double[:, :] result
result = np.zeros((ser.shape[0], 4))
cdef double running_global_peak = ser[0]
cdef double min_since_global_peak = ser[0]
cdef double running_max_dd = 0
cdef long running_global_peak_id = 0
cdef long running_max_dd_peak_id = 0
cdef long running_max_dd_trough_id = 0
cdef long i
cdef double val
cdef int prob_1
cdef int prob_2
cdef tuple intermed
cdef long newthing
for i in xrange(ser.shape[0]):
val = ser[i]
if i < window:
if val >= running_global_peak:
running_global_peak = val
running_global_peak_id = i
min_since_global_peak = val
if val < min_since_global_peak:
min_since_global_peak = val
if val - running_global_peak <= running_max_dd:
running_max_dd = val - running_global_peak
running_max_dd_peak_id = running_global_peak_id
running_max_dd_trough_id = i
result[i, 0] = <double>running_max_dd
result[i, 1] = <double>running_max_dd_peak_id
result[i, 2] = <double>running_max_dd_trough_id
result[i, 3] = <double>running_global_peak_id
else:
prob_1 = 1 if result[i-1, 3] <= float(i - window) else 0
prob_2 = 1 if result[i-1, 1] <= float(i - window) else 0
if prob_1 or prob_2:
intermed = cy_dd_custom_mv(ser[i-window+1:i+1])
result[i, 0] = <double>intermed[0]
result[i, 1] = <double>(intermed[1] + i - window + 1)
result[i, 2] = <double>(intermed[2] + i - window + 1)
result[i, 3] = <double>(intermed[3] + i - window + 1)
else:
newthing = <long>(int(result[i-1, 3]))
result[i, 3] = i if ser[i] >= ser[newthing] else result[i-1, 3]
if val - ser[newthing] <= result[i-1, 0]:
result[i, 0] = <double>(val - ser[newthing])
result[i, 1] = <double>result[i-1, 3]
result[i, 2] = <double>i
else:
result[i, 0] = <double>result[i-1, 0]
result[i, 1] = <double>result[i-1, 1]
result[i, 2] = <double>result[i-1, 2]
cdef double[:] finalresult = result[:, 0]
return finalresult
Here is a Numba-accelerated solution:
import pandas as pd
import numpy as np
import numba
from time import time
n = 10000
returns = pd.Series(np.random.normal(1.001, 0.01, n), pd.date_range("2020-01-01", periods=n, freq="1min"))
#numba.njit
def max_drawdown(cum_returns):
max_drawdown = 0.0
current_max_ret = cum_returns[0]
for ret in cum_returns:
if ret > current_max_ret:
current_max_ret = ret
max_drawdown = max(max_drawdown, 1 - ret / current_max_ret)
return max_drawdown
t = time()
rolling_1h_max_dd = returns.cumprod().rolling("1h").apply(max_drawdown, raw=True)
print("Fast:", time() - t);
def max_drawdown_slow(x):
return (1 - x / x.cummax()).max()
t = time()
rolling_1h_max_dd_slow = returns.cumprod().rolling("1h").apply(max_drawdown_slow, raw=False)
print("Slow:", time() - t);
assert rolling_1h_max_dd.equals(rolling_1h_max_dd_slow)
Output:
Fast: 0.05633878707885742
Slow: 4.540301084518433
=> 80x speedup
Simple oneliner
df['rol_dd_10'] = df['s'].rolling(10).apply(lambda s: ((s - s.cummax()) / s.cummax()).min())
Which gives you a rolling window of maximum drawdown in percent.
If you don't want percentages but rather only want the absolute value instead:
df['rol_dd_10'] = df['s'].rolling(10).apply(lambda s: (s - s.cummax()).min())
# BEGIN: TRADEWAVE MOVING AVERAGE CROSSOVER EXAMPLE
THRESHOLD = 0.005
INTERVAL = 43200
SHORT = 10
LONG = 90
def initialize():
storage.invested = storage.get('invested', False)
def tick():
short_term = data(interval=INTERVAL).btc_usd.ma(SHORT)
long_term = data(interval=INTERVAL).btc_usd.ma(LONG)
diff = 100 * (short_term - long_term) / ((short_term + long_term) / 2)
if diff >= THRESHOLD and not storage.invested:
buy(pairs.btc_usd)
storage.invested = True
elif diff <= -THRESHOLD and storage.invested:
sell(pairs.btc_usd)
storage.invested = False
plot('short_term', short_term)
plot('long_term', long_term)
# END: TRADEWAVE MOVING AVERAGE CROSSOVER EXAMPLE
##############################################################
##############################################################
# BEGIN MAX DRAW DOWN by litepresence
# vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
dd()
ROLLING = 30 # days
def dd():
dd, storage.max_dd = max_dd(0)
bnh_dd, storage.max_bnh_dd = bnh_max_dd(0)
rolling_dd, storage.max_rolling_dd = max_dd(
ROLLING*86400/info.interval)
rolling_bnh_dd, storage.max_rolling_bnh_dd = bnh_max_dd(
ROLLING*86400/info.interval)
plot('dd', dd, secondary=True)
plot('bnh_dd', bnh_dd, secondary=True)
plot('rolling_dd', rolling_dd, secondary=True)
plot('rolling_bnh_dd', rolling_bnh_dd, secondary=True)
plot('zero', 0, secondary=True)
if info.tick == 0:
plot('dd_floor', -200, secondary=True)
def max_dd(rolling):
port_value = float(portfolio.usd+portfolio.btc*data.btc_usd.price)
max_value = 'max_value_' + str(rolling)
values_since_max = 'values_since_max_' + str(rolling)
max_dd = 'max_dd_' + str(rolling)
storage[max_value] = storage.get(max_value, [port_value])
storage[values_since_max] = storage.get(values_since_max, [port_value])
storage[max_dd] = storage.get(max_dd, [0])
storage[max_value].append(port_value)
if port_value > max(storage[max_value]):
storage[values_since_max] = [port_value]
else:
storage[values_since_max].append(port_value)
storage[max_value] = storage[max_value][-rolling:]
storage[values_since_max] = storage[values_since_max][-rolling:]
dd = -100*(max(storage[max_value]) - storage[values_since_max][-1]
)/max(storage[max_value])
storage[max_dd].append(float(dd))
storage[max_dd] = storage[max_dd][-rolling:]
max_dd = min(storage[max_dd])
return (dd, max_dd)
def bnh_max_dd(rolling):
coin = data.btc_usd.price
bnh_max_value = 'bnh_max_value_' + str(rolling)
bnh_values_since_max = 'bnh_values_since_max_' + str(rolling)
bnh_max_dd = 'bnh_max_dd_' + str(rolling)
storage[bnh_max_value] = storage.get(bnh_max_value, [coin])
storage[bnh_values_since_max] = storage.get(bnh_values_since_max, [coin])
storage[bnh_max_dd] = storage.get(bnh_max_dd, [0])
storage[bnh_max_value].append(coin)
if coin > max(storage[bnh_max_value]):
storage[bnh_values_since_max] = [coin]
else:
storage[bnh_values_since_max].append(coin)
storage[bnh_max_value] = storage[bnh_max_value][-rolling:]
storage[bnh_values_since_max] = storage[bnh_values_since_max][-rolling:]
bnh_dd = -100*(max(storage[bnh_max_value]) - storage[bnh_values_since_max][-1]
)/max(storage[bnh_max_value])
storage[bnh_max_dd].append(float(bnh_dd))
storage[bnh_max_dd] = storage[bnh_max_dd][-rolling:]
bnh_max_dd = min(storage[bnh_max_dd])
return (bnh_dd, bnh_max_dd)
def stop():
log('MAX DD......: %.2f pct' % storage.max_dd)
log('R MAX DD....: %.2f pct' % storage.max_rolling_dd)
log('MAX BNH DD..: %.2f pct' % storage.max_bnh_dd)
log('R MAX BNH DD: %.2f pct' % storage.max_rolling_bnh_dd)
[2015-03-04 00:00:00] MAX DD......: -67.94 pct
[2015-03-04 00:00:00] R MAX DD....: -4.93 pct
[2015-03-04 00:00:00] MAX BNH DD..: -82.88 pct
[2015-03-04 00:00:00] R MAX BNH DD: -26.38 pct
Draw Down
Max Drawn Down
Buy and Hold Draw Down
Buy and Hold Max Draw Down
Rolling Draw Down
Rolling Max Drawn Down
Rolling Buy and Hold Draw Down
Rolling Buy and Hold Max Draw Down
No pandas, cython, or numpy dependencies. All calculations via simple lists.
Definitions are reusable for multiple rolling window sizes in the same script. You will have to edit the series input for your platform as this is designed for Bitcoin trading at tradewave.net
Hello people.
This is quite a complex problem if you want to solve this in a computationally efficient way for a rolling window.
I have gone ahead and written a solution to this in C#.
I want to share this as the effort required to replicate this work is quite high.
First, here are the results:
here we take a simple drawdown implementation and re-calculate for the full window each time
test1 - simple drawdown test with 30 period rolling window. run 100 times.
total seconds 0.8060461
test2 - simple drawdown test with 60 period rolling window. run 100 times.
total seconds 1.416081
test3 - simple drawdown test with 180 period rolling window. run 100 times.
total seconds 3.6602093
test4 - simple drawdown test with 360 period rolling window. run 100 times.
total seconds 6.696383
test5 - simple drawdown test with 500 period rolling window. run 100 times.
total seconds 8.9815137
here we compare to the results generated from my efficient rolling window algorithm where only the latest observation is added and then it does it's magic
test6 - running drawdown test with 30 period rolling window. run 100 times.
total seconds 0.2940168
test7 - running drawdown test with 60 period rolling window. run 100 times.
total seconds 0.3050175
test8 - running drawdown test with 180 period rolling window. run 100 times.
total seconds 0.3780216
test9 - running drawdown test with 360 period rolling window. run 100 times.
total seconds 0.4560261
test10 - running drawdown test with 500 period rolling window. run 100 times.
total seconds 0.5050288
At at 500 period window. We are achieving about a 20:1 improvement in calculation time.
Here is the code of the simple drawdown class used for the comparisons:
public class SimpleDrawDown
{
public double Peak { get; set; }
public double Trough { get; set; }
public double MaxDrawDown { get; set; }
public SimpleDrawDown()
{
Peak = double.NegativeInfinity;
Trough = double.PositiveInfinity;
MaxDrawDown = 0;
}
public void Calculate(double newValue)
{
if (newValue > Peak)
{
Peak = newValue;
Trough = Peak;
}
else if (newValue < Trough)
{
Trough = newValue;
var tmpDrawDown = Peak - Trough;
if (tmpDrawDown > MaxDrawDown)
MaxDrawDown = tmpDrawDown;
}
}
}
And here is the code for the full efficient implementation. Hopefully the code comments make sense.
internal class DrawDown
{
int _n;
int _startIndex, _endIndex, _troughIndex;
public int Count { get; set; }
LinkedList<double> _values;
public double Peak { get; set; }
public double Trough { get; set; }
public bool SkipMoveBackDoubleCalc { get; set; }
public int PeakIndex
{
get
{
return _startIndex;
}
}
public int TroughIndex
{
get
{
return _troughIndex;
}
}
//peak to trough return
public double DrawDownAmount
{
get
{
return Peak - Trough;
}
}
/// <summary>
///
/// </summary>
/// <param name="n">max window for drawdown period</param>
/// <param name="peak">drawdown peak i.e. start value</param>
public DrawDown(int n, double peak)
{
_n = n - 1;
_startIndex = _n;
_endIndex = _n;
_troughIndex = _n;
Count = 1;
_values = new LinkedList<double>();
_values.AddLast(peak);
Peak = peak;
Trough = peak;
}
/// <summary>
/// adds a new observation on the drawdown curve
/// </summary>
/// <param name="newValue"></param>
public void Add(double newValue)
{
//push the start of this drawdown backwards
//_startIndex--;
//the end of the drawdown is the current period end
_endIndex = _n;
//the total periods increases with a new observation
Count++;
//track what all point values are in the drawdown curve
_values.AddLast(newValue);
//update if we have a new trough
if (newValue < Trough)
{
Trough = newValue;
_troughIndex = _endIndex;
}
}
/// <summary>
/// Shift this Drawdown backwards in the observation window
/// </summary>
/// <param name="trackingNewPeak">whether we are already tracking a new peak or not</param>
/// <returns>a new drawdown to track if a new peak becomes active</returns>
public DrawDown MoveBack(bool trackingNewPeak, bool recomputeWindow = true)
{
if (!SkipMoveBackDoubleCalc)
{
_startIndex--;
_endIndex--;
_troughIndex--;
if (recomputeWindow)
return RecomputeDrawdownToWindowSize(trackingNewPeak);
}
else
SkipMoveBackDoubleCalc = false;
return null;
}
private DrawDown RecomputeDrawdownToWindowSize(bool trackingNewPeak)
{
//the start of this drawdown has fallen out of the start of our observation window, so we have to recalculate the peak of the drawdown
if (_startIndex < 0)
{
Peak = double.NegativeInfinity;
_values.RemoveFirst();
Count--;
//there is the possibility now that there is a higher peak, within the current drawdown curve, than our first observation
//when we find it, remove all data points prior to this point
//the new peak must be before the current known trough point
int iObservation = 0, iNewPeak = 0, iNewTrough = _troughIndex, iTmpNewPeak = 0, iTempTrough = 0;
double newDrawDown = 0, tmpPeak = 0, tmpTrough = double.NegativeInfinity;
DrawDown newDrawDownObj = null;
foreach (var pointOnDrawDown in _values)
{
if (iObservation < _troughIndex)
{
if (pointOnDrawDown > Peak)
{
iNewPeak = iObservation;
Peak = pointOnDrawDown;
}
}
else if (iObservation == _troughIndex)
{
newDrawDown = Peak - Trough;
tmpPeak = Peak;
}
else
{
//now continue on through the remaining points, to determine if there is a nested-drawdown, that is now larger than the newDrawDown
//e.g. higher peak beyond _troughIndex, with higher trough than that at _troughIndex, but where new peak minus new trough is > newDrawDown
if (pointOnDrawDown > tmpPeak)
{
tmpPeak = pointOnDrawDown;
tmpTrough = tmpPeak;
iTmpNewPeak = iObservation;
//we need a new drawdown object, as we have a new higher peak
if (!trackingNewPeak)
newDrawDownObj = new DrawDown(_n + 1, tmpPeak);
}
else
{
if (!trackingNewPeak && newDrawDownObj != null)
{
newDrawDownObj.MoveBack(true, false); //recomputeWindow is irrelevant for this as it will never fall before period 0 in this usage scenario
newDrawDownObj.Add(pointOnDrawDown); //keep tracking this new drawdown peak
}
if (pointOnDrawDown < tmpTrough)
{
tmpTrough = pointOnDrawDown;
iTempTrough = iObservation;
var tmpDrawDown = tmpPeak - tmpTrough;
if (tmpDrawDown > newDrawDown)
{
newDrawDown = tmpDrawDown;
iNewPeak = iTmpNewPeak;
iNewTrough = iTempTrough;
Peak = tmpPeak;
Trough = tmpTrough;
}
}
}
}
iObservation++;
}
_startIndex = iNewPeak; //our drawdown now starts from here in our observation window
_troughIndex = iNewTrough;
for (int i = 0; i < _startIndex; i++)
{
_values.RemoveFirst(); //get rid of the data points prior to this new drawdown peak
Count--;
}
return newDrawDownObj;
}
return null;
}
}
public class RunningDrawDown
{
int _n;
List<DrawDown> _drawdownObjs;
DrawDown _currentDrawDown;
DrawDown _maxDrawDownObj;
/// <summary>
/// The Peak of the MaxDrawDown
/// </summary>
public double DrawDownPeak
{
get
{
if (_maxDrawDownObj == null) return double.NegativeInfinity;
return _maxDrawDownObj.Peak;
}
}
/// <summary>
/// The Trough of the Max DrawDown
/// </summary>
public double DrawDownTrough
{
get
{
if (_maxDrawDownObj == null) return double.PositiveInfinity;
return _maxDrawDownObj.Trough;
}
}
/// <summary>
/// The Size of the DrawDown - Peak to Trough
/// </summary>
public double DrawDown
{
get
{
if (_maxDrawDownObj == null) return 0;
return _maxDrawDownObj.DrawDownAmount;
}
}
/// <summary>
/// The Index into the Window that the Peak of the DrawDown is seen
/// </summary>
public int PeakIndex
{
get
{
if (_maxDrawDownObj == null) return 0;
return _maxDrawDownObj.PeakIndex;
}
}
/// <summary>
/// The Index into the Window that the Trough of the DrawDown is seen
/// </summary>
public int TroughIndex
{
get
{
if (_maxDrawDownObj == null) return 0;
return _maxDrawDownObj.TroughIndex;
}
}
/// <summary>
/// Creates a running window for the calculation of MaxDrawDown within the window
/// </summary>
/// <param name="n">the number of periods within the window</param>
public RunningDrawDown(int n)
{
_n = n;
_currentDrawDown = null;
_drawdownObjs = new List<DrawDown>();
}
/// <summary>
/// The new value to add onto the end of the current window (the first value will drop off)
/// </summary>
/// <param name="newValue">the new point on the curve</param>
public void Calculate(double newValue)
{
if (double.IsNaN(newValue)) return;
if (_currentDrawDown == null)
{
var drawDown = new DrawDown(_n, newValue);
_currentDrawDown = drawDown;
_maxDrawDownObj = drawDown;
}
else
{
//shift current drawdown back one. and if the first observation falling outside the window means we encounter a new peak after the current trough, we start tracking a new drawdown
var drawDownFromNewPeak = _currentDrawDown.MoveBack(false);
//this is a special case, where a new lower peak (now the highest) is created due to the drop of of the pre-existing highest peak, and we are not yet tracking a new peak
if (drawDownFromNewPeak != null)
{
_drawdownObjs.Add(_currentDrawDown); //record this drawdown into our running drawdowns list)
_currentDrawDown.SkipMoveBackDoubleCalc = true; //MoveBack() is calculated again below in _drawdownObjs collection, so we make sure that is skipped this first time
_currentDrawDown = drawDownFromNewPeak;
_currentDrawDown.MoveBack(true);
}
if (newValue > _currentDrawDown.Peak)
{
//we need a new drawdown object, as we have a new higher peak
var drawDown = new DrawDown(_n, newValue);
//do we have an existing drawdown object, and does it have more than 1 observation
if (_currentDrawDown.Count > 1)
{
_drawdownObjs.Add(_currentDrawDown); //record this drawdown into our running drawdowns list)
_currentDrawDown.SkipMoveBackDoubleCalc = true; //MoveBack() is calculated again below in _drawdownObjs collection, so we make sure that is skipped this first time
}
_currentDrawDown = drawDown;
}
else
{
//add the new observation to the current drawdown
_currentDrawDown.Add(newValue);
}
}
//does our new drawdown surpass any of the previous drawdowns?
//if so, we can drop the old drawdowns, as for the remainer of the old drawdowns lives in our lookup window, they will be smaller than the new one
var newDrawDown = _currentDrawDown.DrawDownAmount;
_maxDrawDownObj = _currentDrawDown;
var maxDrawDown = newDrawDown;
var keepDrawDownsList = new List<DrawDown>();
foreach (var drawDownObj in _drawdownObjs)
{
drawDownObj.MoveBack(true);
if (drawDownObj.DrawDownAmount > newDrawDown)
{
keepDrawDownsList.Add(drawDownObj);
}
//also calculate our max drawdown here
if (drawDownObj.DrawDownAmount > maxDrawDown)
{
maxDrawDown = drawDownObj.DrawDownAmount;
_maxDrawDownObj = drawDownObj;
}
}
_drawdownObjs = keepDrawDownsList;
}
}
Example usage:
RunningDrawDown rd = new RunningDrawDown(500);
foreach (var input in data)
{
rd.Calculate(input);
Console.WriteLine(string.Format("max draw {0:0.00000}, peak {1:0.00000}, trough {2:0.00000}, drawstart {3:0.00000}, drawend {4:0.00000}",
rd.DrawDown, rd.DrawDownPeak, rd.DrawDownTrough, rd.PeakIndex, rd.TroughIndex));
}
Related
I've been trying to get same results from tradingviews RMA method but I dont know how to accomplish it
In their page RMA is computed as:
plot(rma(close, 15))
//the same on pine
pine_rma(src, length) =>
alpha = 1/length
sum = 0.0
sum := na(sum[1]) ? sma(src, length) : alpha * src + (1 - alpha) * nz(sum[1])
plot(pine_rma(close, 15))
to test I used input and their result, this is input column and the same input after applying tradingview´s rma(input,14):
data = [[588.0,519.9035093599585],
[361.98999999999984,508.62397297710436],
[412.52000000000055,501.7594034787397],
[197.60000000000042,480.0337318016869],
[208.71999999999932,460.6541795301378],
[380.1100000000006,454.90102384941366],
[537.6599999999999,460.8123792887413],
[323.5600000000013,451.0086379109742],
[431.78000000000077,449.6351637744761],
[299.6299999999992,438.9205092191563],
[225.1900000000005,423.65404427493087],
[292.42000000000013,414.28018396957873],
[357.64999999999964,410.23517082889435],
[692.5100000000003,430.3976586268306],
[219.70999999999916,415.34854015348543],
[400.32999999999987,414.2757872853794],
[604.3099999999995,427.849659622138],
[204.29000000000087,411.8811125062711],
[176.26000000000022,395.0510330415374],
[204.1800000000003,381.41738782428473],
[324.0,377.3161458368358],
[231.67000000000007,366.91284970563316],
[184.21000000000092,353.8626461552309],
[483.0,363.08674285842864],
[290.6399999999994,357.911975511398],
[107.10000000000036,339.996834403441],
[179.0,328.49706051748086],
[182.36000000000058,318.05869905194663],
[275.0,314.98307769109323],
[135.70000000000073,302.17714357030087],
[419.59000000000015,310.56377617242225],
[275.6399999999994,308.06922073153487],
[440.48999999999984,317.5278478221396],
[224.0,310.8472872634153],
[548.0100000000001,327.78748103031415],
[257.0,322.73123238529183],
[267.97999999999956,318.82043007205664],
[366.51000000000016,322.2268279240526],
[341.14999999999964,323.57848307233456],
[147.4200000000001,310.9957342814536],
[158.78000000000063,300.12318183277836],
[416.03000000000077,308.4022402732943],
[360.78999999999917,312.14422311091613],
[1330.7299999999996,384.90035003156487],
[506.92000000000013,393.61603931502464],
[307.6100000000006,387.4727507925229],
[296.7299999999996,380.991125735914],
[462.0,386.7774738976345],
[473.8099999999995,392.9940829049463],
[769.4200000000002,419.88164841173585],
[971.4799999999997,459.2815306680404],
[722.1399999999994,478.0571356203232],
[554.9799999999996,483.5516259331572],
[688.5,498.19079550936027],
[292.0,483.462881544406],
[634.9500000000007,494.2833900055199]]
# Create the pandas DataFrame
dfRMA = pd.DataFrame(data, columns = ['input', 'wantedrma'])
dfRMA['try1'] = dfRMA['input'].ewm( alpha=1/14, adjust=False).mean()
dfRMA['try2'] = numpy_ewma_vectorized(dfRMA['input'],14)
dfRMA
ewm does not give me same results, so I searched and found this but I just replicated ewma
def numpy_ewma_vectorized(data, window):
alpha = 1/window
alpha_rev = 1-alpha
scale = 1/alpha_rev
n = data.shape[0]
r = np.arange(n)
scale_arr = scale**r
offset = data[0]*alpha_rev**(r+1)
pw0 = alpha*alpha_rev**(n-1)
mult = data*pw0*scale_arr
cumsums = mult.cumsum()
out = offset + cumsums*scale_arr[::-1]
return out
I am getting these results
Do you know how to translate pinescript rma method in pandas?
I realized that using pandas ewm it seems to converge, last rows are closer and closer to the value, is this correct?
...
As far as I know Pine-script will use data that is not exported, so the weighted mean is taking into account previous records that are not in your table, meaning that you can't reproduce the results without more information.
What you need to do is load around 50-100 points (depending on alpha) of data further into the past than what you actually will use, and use a threshold for the comparing the data. You need that both python and pine-script is using data with the same or at least similar "history".
So you make the calculations using the whole dataframe and then you skip the first rows. You can see the effect of the historical data in your own example as difference between your calculation and pine-script one is quickly vanishing after the 55 point, but of course the difference will also depend on alpha.
So actually your code could be already well written. In any case you can use the pandas implementation directly, it will be easier and probably faster.
const cloneArray = (input) => [...input]
const pluck = (input, key) => input.map(element => element[key])
const pineSma = (source, length) => {
let sum = 0.0
for (let i = 0; i < length; ++i) {
sum += source[i] / length
}
return sum
}
const pineRma = (src, length, last) => {
const alpha = 1.0 / length
return alpha * src[0] + (1.0 - alpha) * last
}
const calculatePineRma = (candles, sourceKey, length) => {
const results = []
for (let i = 0; i <= candles.length - length; ++i) {
const sourceCandles = cloneArray(candles.slice(i, i + length)).reverse()
const closes = pluck(sourceCandles, sourceKey)
if (i === 0) {
results.push(pineSma(closes, length))
continue
}
results.push(pineRma(closes, length, results[results.length - 1]))
}
return results
}
I'm trying to do choices based on their weight/probability
this is what I had in python:
import random
myChoiceList = ["Attack", "Heal", "Amplify", "Defense"]
myWeights = [70, 0, 15, 15] // % probability = 100% Ex. Attack has 70% of selection
print(random.choices(myChoicelist , weights = myWeights, k = 1))
I want to do the same thing in c#, how does one do that?
does C# have any methods similar to random.choices() all I know is random.Next()
*this python code works fine randome.choice takes in (sequence, weights, k)
sequence: values,
weights: A list were you can weigh the possibility for each value,
k: the length of the returned list,
I'm looking to do the same for C#,
choose values based on there probability
There is nothing built into C# like this, however, it's not that hard to add an extension method to recreate the same basic behavior:
static class RandomUtils
{
public static string Choice(this Random rnd, IEnumerable<string> choices, IEnumerable<int> weights)
{
var cumulativeWeight = new List<int>();
int last = 0;
foreach (var cur in weights)
{
last += cur;
cumulativeWeight.Add(last);
}
int choice = rnd.Next(last);
int i = 0;
foreach (var cur in choices)
{
if (choice < cumulativeWeight[i])
{
return cur;
}
i++;
}
return null;
}
}
Then you can call it in a similar way as the Python version:
string[] choices = { "Attack", "Heal", "Amplify", "Defense" };
int[] weights = { 70, 0, 15, 15 };
Random rnd = new Random();
Console.WriteLine(rnd.Choice(choices, weights));
you can get random.next(0,100), then choose the relevant item with a simple switch case or something. your domains will be like this , [0-70 , 70-85, 85-100]. let me know if you need full code.
Random ran = new Random();
int probability = ran.Next(0, 100);
string s;
if (probability == 0)
s = "Heal";
else if (probability <= 70)
s = "Attack";
else if (probability <= 85)
s = "Amplify";
else if (probability <= 100)
s = "Defense";
Looking for the help with an algorithm for local machine or cluster (Python, R, JavaScript, any languages).
I have a list of locations with coordinates.
# R script
n <- 10
set.seed(1)
index <- paste0("id_",c(1:n))
lat <- runif(n, 32.0, 41)
lon <- runif(n, 84, 112)*(-1)
values <- as.integer(runif(n, 50, 100))
df <- data.frame(index, lat, lon, values, stringsAsFactors = FALSE)
names(df) <- c('loc_id','lat','lon', 'value')
loc_id lat lon value
1 id_1 34.38958 -89.76729 96
2 id_2 35.34912 -88.94359 60
3 id_3 37.15568 -103.23664 82
4 id_4 40.17387 -94.75490 56
5 id_5 33.81514 -105.55556 63
6 id_6 40.08551 -97.93558 69
7 id_7 40.50208 -104.09332 50
8 id_8 37.94718 -111.77337 69
9 id_9 37.66203 -94.64099 93
10 id_10 32.55608 -105.76847 67
I need to find 3 closets locations for each location in the table.
This is my code in R:
# R script
require(dplyr)
require(geosphere)
start.time <- Sys.time()
d1 <- df
sample <- 999999999999
distances <- list("init1" = sample, "init2" = sample, "init3" = sample)
d1$distances <- apply(d1, 1, function(x){distances})
n_rows = nrow(d1)
for (i in 1:(n_rows-1)) {
# current location
dot1 <- c(d1$lon[i], d1$lat[i])
for (k in (i+1):n_rows) {
# next location
dot2 <- c(d1$lon[k], d1$lat[k])
# distance between locations
meters_between <- as.integer(distm(dot1, dot2, fun = distHaversine))
# updating current location distances
distances <- d1$distances[[i]]
distances[d1$loc_id[k]] <- meters_between
d1$distances[[i]] <- distances[order(unlist(distances), decreasing=FALSE)][1:3]
# updating next location distances
distances <- d1$distances[[k]]
distances[d1$loc_id[i]] <- meters_between
d1$distances[[k]] <- distances[order(unlist(distances), decreasing=FALSE)][1:3]
}
}
But it takes too much time:
# [1] "For 10 rows and 45 iterations takes 0.124729156494141 sec. Average sec 0.00277175903320313 per row."
# [1] "For 100 rows and 4950 iterations takes 2.54944682121277 sec. Average sec 0.000515039761861165 per row."
# [1] "For 200 rows and 19900 iterations takes 10.1178169250488 sec. Average sec 0.000508433011308986 per row."
# [1] "For 500 rows and 124750 iterations takes 73.7151870727539 sec. Average sec 0.000590903303188408 per row."
I did the same in Python:
# Python script
import pandas as pd
import numpy as np
n = 10
np.random.seed(1)
data_m = np.random.uniform(0, 5, 5)
data = {'loc_id':range(1, n+1),
'lat':np.random.uniform(32, 41, n),
'lon':np.random.uniform(84, 112, n)*(-1),
'values':np.random.randint(50, 100, n)}
df = pd.DataFrame(data)[['loc_id', 'lat', 'lon', 'values']]
df['loc_id'] = df['loc_id'].apply(lambda x: 'id_{0}'.format(x))
df = df.reset_index().drop('index', axis = 1).set_index('loc_id')
from geopy.distance import distance
from datetime import datetime
start_time = datetime.now()
sample = 999999999999
df['distances'] = np.nan
df['distances'] = df['distances'].apply(lambda x: [{'init1': sample}, {'init2': sample}, {'init3': sample}])
n_rows = len(df)
rows_done = 0
for i, row_i in df.head(n_rows-1).iterrows():
dot1 = (row_i['lat'], row_i['lon'])
rows_done = rows_done + 1
for k, row_k in df.tail(n_rows-rows_done).iterrows():
dot2 = (row_k['lat'], row_k['lon'])
meters_between = int(distance(dot1,dot2).meters)
distances = df.at[i, 'distances']
distances.append({k: meters_between})
distances_sorted = sorted(distances, key=lambda x: x[next(iter(x))])[:3]
df.at[i, 'distances'] = distances_sorted
distances = df.at[k, 'distances']
distances.append({i: meters_between})
distances_sorted = sorted(distances, key=lambda x: x[next(iter(x))])[:3]
df.at[k, 'distances'] = distances_sorted
print df
Almost the same performance.
Anybody knows if there is a better approach? In my task it has to be done for 90000 locations. Even thought about Hadoop/MpRc/Spark, but have no idea how to do in distributed mode.
I am glad to hear any ideas or suggestions.
If Euclidean distance is ok then nn2 uses kd-trees and C code so it should be fast:
library(RANN)
nn2(df[2:3], k = 4)
This took a total of 0.06 to 0.11 seconds on my not particularly fast laptop to process n = 10,000 rows and a total of 1.00 to 1.25 seconds for 90,000 rows.
I can offer a python solution with scipy
from scipy.spatial import distance
from geopy.distance import vincenty
v=distance.cdist(df[['lat','lon']].values,df[['lat','lon']].values,lambda u, v: vincenty(u, v).kilometers)
np.sort(v,axis=1)[:,1:4]
Out[1033]:
array([[384.09948155, 468.15944729, 545.41393271],
[270.07677993, 397.21974571, 659.96238603],
[384.09948155, 397.21974571, 619.616239 ],
[203.07302273, 483.54687912, 741.21396029],
[203.07302273, 444.49156394, 659.96238603],
[437.31308598, 468.15944729, 494.91879983],
[494.91879983, 695.91437812, 697.27399161],
[270.07677993, 444.49156394, 483.54687912],
[530.54946479, 626.29467739, 695.91437812],
[437.31308598, 545.41393271, 697.27399161]])
Here's how to solve this problem with C++ and my library
GeographicLib (version 1.47 or later). This uses true ellipsoidal geodesic
distances and a
vantage point tree
to optimize the search for nearest neighbors.
#include <exception>
#include <vector>
#include <fstream>
#include <string>
#include <GeographicLib/NearestNeighbor.hpp>
#include <GeographicLib/Geodesic.hpp>
using namespace std;
using namespace GeographicLib;
// A structure to hold a geographic coordinate.
struct pos {
string id;
double lat, lon;
pos(const string& _id = "", double _lat = 0, double _lon = 0) :
id(_id), lat(_lat), lon(_lon) {}
};
// A class to compute the distance between 2 positions.
class DistanceCalculator {
private:
Geodesic _geod;
public:
explicit DistanceCalculator(const Geodesic& geod) : _geod(geod) {}
double operator() (const pos& a, const pos& b) const {
double d;
_geod.Inverse(a.lat, a.lon, b.lat, b.lon, d);
if ( !(d >= 0) )
// Catch illegal positions which result in d = NaN
throw GeographicErr("distance doesn't satisfy d >= 0");
return d;
}
};
int main() {
try {
// Read in pts
vector<pos> pts;
string id;
double lat, lon;
{
ifstream is("pts.txt"); // lines of "id lat lon"
if (!is.good())
throw GeographicErr("pts.txt not readable");
while (is >> id >> lon >> lat)
pts.push_back(pos(id, lat, lon));
if (pts.size() == 0)
throw GeographicErr("need at least one location");
}
// Define a distance function object
DistanceCalculator distance(Geodesic::WGS84());
// Create NearestNeighbor object
NearestNeighbor<double, pos, DistanceCalculator>
ptsset(pts, distance);
vector<int> ind;
int n = 3; // Find 3 nearest neighbors
for (unsigned i = 0; i < pts.size(); ++i) {
ptsset.Search(pts, distance, pts[i], ind,
n, numeric_limits<double>::max(),
// exclude the point itself
0.0);
if (ind.size() != n)
throw GeographicErr("unexpected number of results");
cout << pts[i].id;
for (unsigned j = 0; j < ind.size(); ++j)
cout << " " << pts[ind[j]].id;
cout << "\n";
}
int setupcost, numsearches, searchcost, mincost, maxcost;
double mean, sd;
ptsset.Statistics(setupcost, numsearches, searchcost,
mincost, maxcost, mean, sd);
long long
totcost = setupcost + searchcost,
exhaustivecost = ((pts.size() - 1) * pts.size())/2;
cerr
<< "Number of distance calculations = " << totcost << "\n"
<< "With an exhaustive search = " << exhaustivecost << "\n"
<< "Ratio = " << double(totcost) / exhaustivecost << "\n"
<< "Efficiency improvement = "
<< 100 * (1 - double(totcost) / exhaustivecost) << "%\n";
}
catch (const exception& e) {
cerr << "Caught exception: " << e.what() << "\n";
return 1;
}
}
This reads in a set of points (in the form "id lat lon") for pts.txt,
puts them in a VP tree. Then for each point it looks up the 3 nearest
neighbors and prints the id and the id's of the neighbors (ranked by
distance).
Compile this with, e.g.,
g++ -O3 -o nearest nearest.cpp -lGeographic
If pts.txt contains 90000 points, then the computation completes in
about 6 secs (or 70 μs per point) on my home computer after doing about 3380000 distance
calculations. This
is about 1200 times more efficient than a brute force calculaion
(doing all N(N − 1)/2 distance calculations).
You could speed this up (by a factor of a "few") by using a crude
approximation to the distance (e.g., spherical or euclidean); just
modify the DistanceCalculator class appropriately. For example, this
version of DistanceCalculator returns the spherical distance in
degrees:
// A class to compute the spherical distance between 2 positions.
class DistanceCalculator {
public:
explicit DistanceCalculator(const Geodesic& /*geod*/) {}
double operator() (const pos& a, const pos& b) const {
double sphia, cphia, sphib, cphib, somgab, comgab;
Math::sincosd(a.lat, sphia, cphia);
Math::sincosd(b.lat, sphib, cphib);
Math::sincosd(Math::AngDiff(a.lon, b.lon), somgab, comgab);
return Math::atan2d(Math::hypot(cphia * sphib - sphia * cphib * comgab,
cphib * somgab),
sphia * sphib + cphia * cphib * comgab);
}
};
But now you have the added burden of ensuring that the approximation
is good enough. I recommend just using the correct geodesic distance
in the first place.
Details of the implementation of VP trees in GeographicLib are given
here.
I'm trying to rewrite this function:
def smoothen_fast(heightProfile, travelTime):
smoothingInterval = 30 * travelTime
heightProfile.extend([heightProfile[-1]]*smoothingInterval)
# Get the mean of first `smoothingInterval` items
first_mean = sum(heightProfile[:smoothingInterval]) / smoothingInterval
newHeightProfile = [first_mean]
for i in xrange(len(heightProfile)-smoothingInterval-1):
prev = heightProfile[i] # the item to be subtracted from the sum
new = heightProfile[i+smoothingInterval] # item to be added
# Calculate the sum of previous items by multiplying
# last mean with smoothingInterval
prev_sum = newHeightProfile[-1] * smoothingInterval
new_sum = prev_sum - prev + new
mean = new_sum / smoothingInterval
newHeightProfile.append(mean)
return newHeightProfile
as embedded C++ Code:
import scipy.weave as weave
heightProfile = [0.14,0.148,1.423,4.5]
heightProfileSize = len(heightProfile)
travelTime = 3
code = r"""
#include <string.h>
int smoothingInterval = 30 * travelTime;
double *heightProfileR = new double[heightProfileSize+smoothingInterval];
for (int i = 0; i < heightProfileSize; i++)
{
heightProfileR[i] = heightProfile[i];
}
for (int i = 0; i < smoothingInterval; i++)
{
heightProfileR[heightProfileSize+i] = -1;
}
double mean = 0;
for (int i = 0; i < smoothingInterval; i++)
{
mean += heightProfileR[i];
}
mean = mean/smoothingInterval;
double *heightProfileNew = new double[heightProfileSize-smoothingInterval];
for (int i = 0; i < heightProfileSize-smoothingInterval-1; i++)
{
double prev = heightProfileR[i];
double newp = heightProfile[i+smoothingInterval];
double prev_sum = heightProfileNew[i] * smoothingInterval;
double new_sum = prev_sum - prev + newp;
double meanp = new_sum / smoothingInterval;
heightProfileNew[i+1] = meanp;
}
return_val = Py::new_reference_to(Py::Double(heightProfileNew));
"""
d = weave.inline(code,['heightProfile','heightProfileSize','travelTime'])
As a return type i need the heightProfileNew. I need the access it like a list in Python later.
I look at these examples:
http://docs.scipy.org/doc/scipy/reference/tutorial/weave.html
He keeps telling me that he doesn't know Py::, but in the examples there are no Py-Includes?
I know, the question is old, but I think it is still interesting.
Assuming your using weave to improve computation speed and that you know the length of your output beforehand, I suggest that you create the result before calling inline. That way you can create the result variable in python (very easy). I would also suggest using a nd.ndarray as a result because it makes shure you use the right datatype. You can iterate ndarrays in python the same way you iterate lists.
import numpy as np
heightProfileArray = np.array(heightprofile)
# heightProfileArray = np.array(heightprofile, dtype = np.float32) if you want to make shure you have the right datatype. Another choice would be np.float64
resultArray = np.zeros_like(heightProfileArray) # same array size and data type but filled with zeros
[..]
weave.inline(code,['heightProfile','heightProfileSize','travelTime','resultArray'])
for element in resultArray:
print element
In your C++-code you can then just assign values to elements of that array:
[..]
resultArray[i+1] = 5.5;
[..]
I'm implementing a simple Xor Reducer, but it is unable to return the appropriate value.
Python Code (Input):
class LazySpecializedFunctionSubclass(LazySpecializedFunction):
subconfig_type = namedtuple('subconfig',['dtype','ndim','shape','size','flags'])
def __init__(self, py_ast = None):
py_ast = py_ast or get_ast(self.kernel)
super(LazySlimmy, self).__init__(py_ast)
# [... other code ...]
def points(self, inpt):
iter = np.nditer(input, flags=['c_index'])
while not iter.finished:
yield iter.index
iter.iternext()
class XorReduction(LazySpecializedFunctionSubclass):
def kernel(self, inpt):
'''
Calculates the cumulative XOR of elements in inpt, equivalent to
Reduce with XOR
'''
result = 0
for point in self.points(inpt): # self.points is defined in LazySpecializedFunctionSubclass
result = point ^ result # notice how 'point' here is the actual element in self.points(inpt), not the index
return result
C Code (Output):
// <file: module.c>
void kernel(long* inpt, long* output) {
long result = 0;
for (int point = 0; point < 2; point ++) {
result = point ^ result; // Notice how it's using the index, point, not inpt[point].
};
* output = result;
};
Any ideas how to fix this?
The problem is that you are using point in different ways, in XorReduction kernel method you are iterating of the values in the array, but in the generated C code you are iterating over the indices of the array. Your C code xor reduction is thus done on the indices.
The generated C function should look more like
// <file: module.c>
void kernel(long* inpt, long* output) {
long result = 0;
for (int point = 0; point < 2; point ++) {
result = inpt[point] ^ result; // you did not reference your input in the question
};
* output = result;
};