Don’t make this mistake in backtesting! – Analysis and Forecast – June 1, 2024

When backtesting a professional advisor in a strategy test, the broker calls the data from the server. This part is a little tricky. What most people don’t know is that brokers store some sort of ‘secret data’. This means that the broker does not store 100% of the real-time tick data flow on its servers. All brokers reduce this real-time data flow to some degree, exposing strategy testers to a variety of data flows.

When we test expert advisors in Strategy Tester, we backtest them with this smoothed data. Experts can produce excellent results with this pared-down data, but when those same experts are exposed to a live data stream, backtest results can exhibit unexpected behavior. As a result, your account may be under pressure.

Let’s make this happen using EURUSD, a widely used trading instrument. For example, when comparing the backtest results of these two brokers, we see that the first broker has 100% history quality and approximately 55 million ticks.

The second broker again has 100% history quality and has 78 million ticks!

Both brokers offer 100% data quality, but the second broker offers almost 50% more tick data! In this situation, the second broker’s tick data is a better representation of reality.

Considering that the products are the same, the analysis period is the same, and the price movements of those products are also unique, both brokers should provide somewhat similar tick data numbers. But that’s not the case.

When you test a professional advisor from your first broker, the test results may not be reliable. When you test a professional advisor on a second broker, you test with more tick data, so the test results are more reliable.

Observation of results

Looking at the results, we see that the risk parameters in particular differ significantly between the two brokers. As I said earlier, this is expected because the data provided is not broker-specific.

The fewer ticks you have, the further away you are from the actual data flow.

Experts performed better with more tick data on the risk side, but received a small profit in return. In this example, more attention should be paid to the backtesting of the second broker.

In this case, if you are risk-sensitive and changed your mind about buying after testing the experts at the first broker, it will not be the right decision for you.

If you are profit oriented and buy this expert because it gives good profits, then again you will make a wrong decision because more tick data will result in less profits.

What can we do to overcome this situation?

There is a simple rule of thumb you can apply. Although it is not a 100% solution, it will help you judge professional counselors more transparently.

If you are interested in a professional advisor, you should test it out with your own broker first. If the results are successful, you should test it with at least one, and ideally two, brokers.

Ideally, if your expert advisor is robust against data anomalies, you should not see significant differences in results across different brokers. Typically a difference of up to 10% is permitted.

However, if your expert advisor fails with one of the three brokers or the results are significantly different, you should not consider using them.

Thank you for reading and please check out my other articles.

Evren Caglar