Mission Impossible: MetaTrader 5 does not support testing and optimization of trading robots based on Renko indicators. – Other – February 26, 2024
This is a series of blog posts about trading according to signals from Renko charts. Before this point we have discussed several aspects of using custom symbols for Renko implementations. The latest findings are described in the article Using Custom Tick Modeling to Prevent Backtesting – the fantastic holy grail of Renko-based trading robots.
One of the most obvious questions about this series is why we use custom symbols for our Renko implementation rather than something else. In fact, “something else” is very limited. Drawing Renko on a canvas or using graphic objects is impractical because it does not allow for backtesting. It seems much more natural to use the Renko indicator instead.
Unfortunately, the indicators in MetaTrader 5 are designed in such a way that it is impossible for testers to work with Renko indicators. Here’s why:
renko indicator
To start our research, we need an indicator that computes a graphical series that looks like a Renko box. Let’s not invent the wheel and use one of the existing indicators like Blue Renko Bars.
As a result, the program required several bug fixes and improvements, the most important of which I will describe one by one. Final edits are attached to the post.
The original directive marker_buffers is incorrect because the drawing type of the plot is DRAW_CANDLES. This is because this type does not support additional buffers for colors (as opposed to DRAW_COLOR_CANDLES, which uses 4+1 buffers).
#property indicator_separate_window #property indicator_buffers 4 #property indicator_plots 1
DRAW_CANDLES requires 4 buffers. The buffer array brickColors is useless and has been removed everywhere.
double brickColors();Other buffer arrays are initialized at startup.
int OnCalculate(const int rates_total, const int prev_calculated, ...) ... if(prev_calculated == 0) ArrayInitialize(OpenBuffer, 0); ArrayInitialize(HighBuffer, 0); ArrayInitialize(LowBuffer, 0); ArrayInitialize(CloseBuffer, 0); ...
We introduced a new variable lastBar to detect the formation of new bars in the host chart. At this moment we need to initialize the buffer elements we just added (under certain conditions).
datetime lastBar; int OnCalculate(const int rates_total, const int prev_calculated, const datetime &time(),...) ... if(lastBar != time(0)) OpenBuffer(0) = 0; HighBuffer(0) = 0; LowBuffer(0) = 0; CloseBuffer(0) = 0; lastBar = time(0); ...
The counter of free renko boxes in the renkoBuffer array of MqlRates was not handled correctly in all situations and could cause an “out of bounds” exception (the indicator would stop and unload).
int OnCalculate(const int rates_total, const int prev_calculated, ...) int size = ArraySize(renkoBuffer); ... int first; if(prev_calculated == 0) ... first = (rates_total > size) ? size : rates_total; else first = size; for(int i = first - 2; i >= 0; i--) ... HighBuffer(shift + i + 1) = renkoBuffer(i).high; LowBuffer(shift + i + 1) = renkoBuffer(i).low; ...
Changed the working principle in the RenkoAdd function, which adds a new box to renkoBuffer. Wrap the call to ArrayResize with two calls to ArraySetAsSeries instead of the heavy ArrayCopy.
void RenkoAdd() int size = ArraySize(renkoBuffer); ArraySetAsSeries(renkoBuffer, false); ArrayResize(renkoBuffer, size + 1, 10000); ArraySetAsSeries(renkoBuffer, true); ...
Additionally, new elements are initialized by an empty struct.
void RenkoAdd() ... const static MqlRates zero = ; renkoBuffer(0) = zero;
N1 attempt(ChartSetSymbolPeriod)
Now let’s think a little about how the indicator works and what this means for the Renko box.
When a new bar is added to the chart, the rates and regular indicators (if applicable) will move to the left by 1 bar, but the Renko box should remain the same (since new boxes appear according to their own “schedule”). On the other hand, when a new box is created, all the old boxes must be moved to the left, but the rates and other indicators will remain in the same place.
It is important to note that all Renko indicators must address these issues.
To resolve this desynchronization, this indicator reserves a variable RedrawChart (which is also not an input parameter) to hold the number of bars to redraw. Defaults to 1, replaced by CHART_VISIBLE_BARS. As a result, renko is only accurate for the last CHART_VISIBLE_BARS bar. Also, ChartGetInteger(0, CHART_VISIBLE_BARS, 0) always returns 0 bars while testing/optimizing without visual mode! This solution is partial, not universal, and could potentially lead to miscalculations when used in automated trading.
It is particularly flawed in the following aspects: Many trading strategies use a combination of multiple indicators to generate trading signals. For example, over a series of blog posts we have been using a simple testing strategy for two MAs crossing over a Renko. To implement this with a Renko indicator, we need to apply MA to the Renko indicator.
And here’s the problem. Indicators in MetaTrader are calculated once on every bar and then recalculated in a “short-circuit” fashion. Only the most recent bar is counted. Even if you update the CHART_VISIBLE_BARS bars in the Renko indicator, the MA (or other indicator) applied on top of the Renko will only be updated on the latest bars. As a result, it is impossible to obtain the correct intersection of MAs.
To overcome this problem, we added a new feature to the renko indicator. Request a complete update of the chart including all additionally applied indicators after creation of new bars or formation of new Renko boxes. To achieve this, calls to ChartSetSymbolPeriod are added in two places: OnCalculate and RenkoAdd.
int OnCalculate(const int rates_total, const int prev_calculated, const datetime &time(),...) ... if(lastBar != time(0)) ... ChartSetSymbolPeriod(0, _Symbol, _Period); lastBar = time(0); ... void RenkoAdd() ... if(lastBar) ChartSetSymbolPeriod(0, _Symbol, _Period);
MA is now in sync with renko relocations and updated appropriately.
Renko indicator with MA for period 1 (see CHLO below to see if it works as a closing price)
OHLC -> Claw
But there is another small problem. Renko is represented by four buffers containing the open, high, low and close price of the candle. If additional indicators are applied to other indicators, the very first buffer is used. Therefore, our 2 MA applies to the open price, which is not usually the desired effect for Renko-based strategies. Instead, MA should be applied to the closing price of the Renko box. To do so, we need to swap the Open and Close buffers in the renko indicator.
The new mode is turned on or off by the new parameter SwapOpenClose.
input bool SwapOpenClose = false; int OnInit() ... if(SwapOpenClose) PlotIndexSetString(0, PLOT_LABEL, "Close;High;Low;Open"); ... int OnCalculate(...) ... for(int i = first - 2; i >= 0; i--) OpenBuffer(i + 1) = SwapOpenClose ? renkoBuffer(i).close : renkoBuffer(i).open; HighBuffer(i + 1) = renkoBuffer(i).high; LowBuffer(i + 1) = renkoBuffer(i).low; CloseBuffer(i + 1) = SwapOpenClose ? renkoBuffer(i).open : renkoBuffer(i).close; ...
This looks like a finished Renko indicator suitable for trading automation. This is a trick, but we’ll discover it in a moment and try to add other features to make it work as expected.
Professional advisor based on 2 MA crossing Renko indicator
For now, let’s apply a test EA (MA2Cross) that originally used renko custom symbols to work with the renko indicator. The name of the modified version is MA2CrossInd.mq5.
Input parameters for basic indicators are added.
input int BrickSize = 100; input bool ShowWicks = true; input bool TotalBars = false; input bool SwapOpenClose = true;
An indicator with the given parameters is created in OnInit and its handle is passed to the signal filter instead of the previous Signal_2MACross_MAPrice parameter (in practice it was always the close price).
int OnInit() ... const int handle = iCustom(_Symbol, _Period, "Blue Renko Bars", BrickSize, ShowWicks, TotalBars, SwapOpenClose); if(handle == INVALID_HANDLE) Print("Can't create indicator, ", _LastError); return(INIT_FAILED); ChartIndicatorAdd(0, 1, handle); ... filter0.MAPrice((ENUM_APPLIED_PRICE)handle); ...
This program can actually be traded on online charts! But backtesting and optimization is impossible!
The reason is that ChartSetSymbolPeriod function is not working in tester. As a result, the two MAs are not recalculated properly and give inconsistent signals.
What can we do?
N2 try(PLOT_SHIFT)
One of the ideas was to implement the Renko indicator using another principle of relocating the Renko box with respect to the regular bars. This is based on the PLOT_SHIFT property.
As you know, depending on its properties, you can move the visual representation of the indicator buffer to the right or left. Positive values shift the curve to the right for the specified number of bars, while negative values shift the curve to the left. So when a new regular bar is created, you can apply Shift +1 to the Renko box to visually keep it in the same position. And when a new Renko box is added, you can apply Shift -1 to move the old box to the left and keep it aligned with the last regular bar.
Because we cannot know in advance which direction future prices will move further along the graph, we need to reserve an empty, invisible box on the right. A spare value is provided for the corresponding input parameter and is used to initialize the variable with the current shift.
input int Reserve = 0; int shift; int OnInit() ... shift = Reserve; PlotIndexSetInteger(0, PLOT_SHIFT, shift); ...
Then, in OnCalculate, we increase the movement of the new bar. Nonzero Reserve is also used as a flag to enable new modes.
int OnCalculate(...) ... if(lastBar != time(0)) if(!Reserve) ChartSetSymbolPeriod(0, _Symbol, _Period); else PlotIndexSetInteger(0, PLOT_SHIFT, ++shift); Comment("++", shift); OpenBuffer(0) = 0; HighBuffer(0) = 0; LowBuffer(0) = 0; CloseBuffer(0) = 0; ...
It also reduces the movement of new boxes in RenkoAdd.
void RenkoAdd() ... if(!Reserve) ChartSetSymbolPeriod(0, _Symbol, _Period); else PlotIndexSetInteger(0, PLOT_SHIFT, --shift); Comment("--", shift);
Of course, you need to use shifts to adjust the index while writing data to the buffer.
int OnCalculate(...) ... for(int i = first - 2; i >= 0; i--) OpenBuffer(shift + i + 1) = SwapOpenClose ? renkoBuffer(i).close : renkoBuffer(i).open; HighBuffer(shift + i + 1) = renkoBuffer(i).high; LowBuffer(shift + i + 1) = renkoBuffer(i).low; CloseBuffer(shift + i + 1) = SwapOpenClose ? renkoBuffer(i).open : renkoBuffer(i).close; if(i == 0) OpenBuffer(shift + i) = SwapOpenClose ? close(i) : renkoBuffer(i).close; HighBuffer(shift + i) = upWick ? upWick : MathMax(renkoBuffer(i).close, renkoBuffer(i).open); LowBuffer(shift + i) = downWick ? downWick : MathMin(renkoBuffer(i).close, renkoBuffer(i).open); CloseBuffer(shift + i) = SwapOpenClose ? renkoBuffer(i).close : close(i);
Unfortunately, despite the fact that the new approach works perfectly visually, data movement is not properly detected in metrics applied on top of Renko.
This is a known and common limitation of the MetaTrader platform. The PLOT_SHIFT property cannot be detected outside of a metric, and when OnCalculate is called on a dependent metric, it receives and processes unshifted data. The short form of OnCalculate is:
int OnCalculate(const int rates_total, const int prev_calculated, const int begin, const double &data());
Here you can see that the indicator receives a similar and somewhat related property, PLOT_DRAW_BEGIN, passed through the startup parameter. However, there is nothing to tell you that the data array uses shifted indexing. The only way to overcome this is to have input for all indicators as input. However, in our case we change the moves dynamically in renko, so it is not possible to adjust the moves of the MA on the fly (unless you re-implement all the necessary indicators yourself and send the moves via chart events or global variables or something else). – Too expensive.)
Try N3 (say “no” to rate_total)
Another approach that comes to mind is not to update the Renko box at all. Stack them at the start of tester time and return the actual number in OnCalculate instead of rates_total.
A negative reserve value can be used as a flag to activate this mode. When on, the buffer index should be mapped to a range (0..size), this is done by adjusting the shift variable.
int OnCalculate(...) ... if(Reserve < 0) shift = rates_total - size; if(shift < 0) Print("Renko buffer overflow, will terminate..."); for(int i = first - 2; i >= 0; i--) OpenBuffer(shift + i + 1) = SwapOpenClose ? renkoBuffer(i).close : renkoBuffer(i).open; HighBuffer(shift + i + 1) = renkoBuffer(i).high; LowBuffer(shift + i + 1) = renkoBuffer(i).low; CloseBuffer(shift + i + 1) = SwapOpenClose ? renkoBuffer(i).open : renkoBuffer(i).close; ... ... return(Reserve < 0 ? size + 1 : rates_total);
Additionally, any calls to change the PLOT_SHIFT property must be wrapped with an appropriate protection condition.
int OnInit() ... if(Reserve > 0) shift = Reserve; PlotIndexSetInteger(0, PLOT_SHIFT, shift); ... int OnCalculate(...) ... if(Reserve > 0) PlotIndexSetInteger(0, PLOT_SHIFT, ++shift); Comment("++", shift); ... void RenkoAdd() ... else if(Reserve > 0) PlotIndexSetInteger(0, PLOT_SHIFT, --shift); Comment("--", shift);
While running in the visual tester, the indicators are calculated and displayed correctly in this mode (remember that you have to scroll to the beginning of the chart to see the boxes). But this is another trick.
To make it work as expected, we need to make some changes to the signal generation module Signal2MACross.mqh by adding the following methods (see attached Signal2MACrossDEMA.mqh):
class Signal2MACross : public CExpertSignal ... int StartIndex(void) override const int base = iBars(_Symbol, PERIOD_CURRENT) - ::BarsCalculated(m_type); return((m_every_tick ? base : base + 1));
Here m_type is the price type to which the MA indicator is applied and we can assign a handle to the renko indicator as mentioned before.
... const int handle = iCustom(_Symbol, _Period, "Blue Renko Bars", BrickSize, ShowWicks, TotalBars, SwapOpenClose); ... filter0.MAPrice((ENUM_APPLIED_PRICE)handle); ...
This will cause EA’s Renko closing price to be used in MA calculations.
Unfortunately none of this works because the value returned from BarsCalculated is always rates_total and not the actual value returned from OnCalculate.
I am attaching an example of a slightly modified DEMA indicator (DEMAtest.mq5, located in MQL5/Indicators/Examples/). This allows us to track and output the actual OnCalculate parameters received by the indicator as they are applied to the renko indicator. The number of bars (boxes) calculated has been reduced. This indicator is also used by Signal2MACrossDEMA.
int OnCalculate(const int rates_total, const int prev_calculated, const int begin, const double &price()) const int limit = (_AppliedTo >= 10) ? BarsCalculated(_AppliedTo) : rates_total; ... #ifdef _DEBUG Print(rates_total, " ", prev_calculated, " ", limit, " ", begin); #endif ...
You can ensure that the number of available bars is always reported as rates_total, and as a result the signal module above will read data from the wrong index.
I think this is a bug in the platform. Because we are passing the correct value through the startup parameter, but not through rates_total.
conclusion
At the time of this writing, it is not possible in MetaTrader 5 to run backtests or optimizations of EA based on signals from the Renko indicator. Several workarounds are available:
You can use custom symbols in renko as described in a previous blog post.
Renko can be computed virtually within EA. If you need to apply different technical indicators to Renko, this can be a difficult routine task, as it requires a re-implementation from scratch for the virtual structure.
Alternatively, one can use only a limited subset of signals that rely on the Renko box without any additional indicators. For example, Close(i) checks each other. The Blue Renko Bars indicator is ready for this scenario.
