Backtesting - pro's, con's and PROBLEMS

algorithmic-trading

CavaliereVerde 2018-02-23 07:52:27 UTC #61

https://community.darwinex.com/t/kvl-cavaliereverde/331

kom75 2018-04-29 16:19:46 UTC #62

Hi,

Can someone explain what I do wrong testing my strategies? I run strategies on demo, recently did backtest on different data, time etc, but to check if everything is ok I did backtest on last 3 monts data (which on mt4 should be Darwinex data). Compare to my demo results: Demo is 648 pounds in profit, backtest is 436,8 pounds in loss. All settings are same.

Thanks in advance

PetePan15 2018-04-29 20:53:01 UTC #63

Hi @kom75, let's try to figure this out

Good move to have verified your demo-results before jumping into live-trading.

First off, are you familiar with the basic principles for running a backtest in MT4? If not, please read up on this topic. Are you sure that your data is adapted to the strategy you use? For instance, if you are opening trades at the open of an H4 candle, you may not need tick data, but if you trade a timeframe below H1 or if you use entry orders rather than market orders you do need tick data.

I don't know how complex your strategy is, i.e. do you fully understand its entry- & exit-logic and could you execute it manually? If so, you can review a portion or all of the trades visually, and see whether one of the two is not implementing the entry- and exit-rules correctly. You may even be able to discern the pattern underlying the difference between the two track-records (for example, one is not executing trades at certain times, is always closing trades early, etc.)

A couple of ideas where things could have gone wrong:
- If your backtest and demo both took the same trades, you should check: what is your average SL/TP and roughly how many trades do you have per day on average? (if your TP/SL is too small + you are trading frequently, it could be that the spread is eating all your profits)
- If your backtest and demo took different trades: are you opening your trades at the open of a new candle? Which timeframe? (if H4 or Daily you will get different signals if your data does not work with NY close like Darwinex).

Please let us know how it goes, and if you're still stuck, please give us some more detailed information about your strategy (timeframe, type of orders used, number of trades...), the data you used in demo and backtest, etc.

Good luck!

TradeSignalMachine 2018-04-30 06:48:52 UTC #64

Some great advice from @PetePan15. The only point I would add for now until you provide more information is that if you are trading higher timeframes and using opening bars in your testing, you must ensure that your EA controls trading execution so that it only acts on new bars in real time trading. If you leave your EA to potentially trade on every tick in real time, then that can in some cases cause significant disparities between your test results and actual. Oh and as a last point, don't expect to get absolutely identical results between the two. You should be able to get relatively close though. Good luck finding the issue.

kom75 2018-04-30 09:00:01 UTC #65

Thanks @PetePan15 and @TradeSignalMachine for your input. My algo has 4 strategies implemented which trades intraday, target are vary with fixed 2, 4 and 20 points and trailed target based on candle high. Also entry depends of strategy can be executed any time or at candle open. I have backtested it once again on this month data and in backtest there is 60 trades opened compare to 58 on demo, money results differs about 200 pounds which is roughly 30 points and 2% of account (trading 1 lot on 10 000 account). Checking backtest I have seen trailed positions always closes differently than on demo, sometimes even 3 points, also money result on same amount of points was different by 3 pounds. Backtest was done on 30 min candles with mt4 tick precision on Darwinex data (same as demo), spread is included but not commission, which is small at about 0,40 pound and with 60 trades can't be those difference reason

CavaliereVerde 2018-04-30 09:04:00 UTC #66

Up to my knowledge mt4 "max precision" is always on m1 candles, even if the chart comes from the broker.
But I may be wrong, I am not an expert about tick history.

kom75 2018-04-30 09:11:11 UTC #67

@CavaliereVerde i have backtested it on 1 min data with same results

CavaliereVerde 2018-04-30 09:13:17 UTC #68

So it confimrs what I am saying, on backtests you are working with m1 data, in live trading you are getting better results with ticks.

kom75 2018-04-30 09:29:10 UTC #69

I tested it now on Dukascopy data with tick precision and results are even worse. 63 trades and 400 pounds difference, can be cause by different data. The main point of my post was that using same data on demo and backtest i would expect very similar results, and don't quite understand why the differ so much

easyFX 2018-04-30 09:35:02 UTC #70

THe best is to go into visual backtest and compare trades one by one.

Also make sure you dont have any future peeking bias in the backtest (even tho it would be weird to have a worst performance). And have a look if you get some stops/limits filled within gaps (like over the weekende), i think in demo you can trick it and it get fills which never existed.

integracore2 2018-04-30 09:37:21 UTC #71

You may find this blog post helpful

The "Data Handling" section may contain answers to several of your questions.

kom75 2018-04-30 09:41:20 UTC #72

@easyFX, I did visual test and there are many differences, also this was made on 1 month data with 60 position opened. How I can do backtest on longer data if I can't trust results and visualy backtest few years data is bit hard bearing in mind this year algo opened 695 positions on 5 currencies and indices

CavaliereVerde 2018-04-30 09:44:27 UTC #73

Of course you cannot go visual on the whole backtest.
Try to do that on 2 weeks to spot the differences and their reason.

kom75 2018-04-30 09:49:53 UTC #74

thanks for link @integracore2, I have checked it and only one thing is i use 30 min TF which can be noisy with low quality data, but I also use tick data from Dukascopy and TrueFX which suposed to be much better.

kom75 2018-04-30 09:55:52 UTC #75

I have checked 1 month visually, and there are some things I don't understand like different money results on same amount of pips, or closing position 3 points lower than on demo (in both cases for losses, so both weren't closed by trailing stop)

CavaliereVerde 2018-04-30 10:02:39 UTC #76

What I can say is that despite the high Tf your trade management and exit srategy is very sensitive to noise and ticks.

Sometimes demo price feed is different fom real, as pointed out by @easyFX.

If I were in your place I woud trust becktests and would work to turn backtests to profit, ignoring demo results.
It is also an obliged choice, the other alternative would be waiting for years on demo...
Try to find the most reliable testing condition and go with backtests.

KlondikeFX 2018-04-30 10:03:47 UTC #77

You want to avoid testing strategies that make us of tight intra-bar trailing, stop-loss or take-profit values with simple MT4 Tick Backtests.
The reason being, MT4 simulates tickdata from the 1m candle. These synthetic tick data is completely unrealistic and you would be surprised what kind of amazingly looking backtests you can create that look terrible on a serious tickdata backtest.

So if you have a strategy that makes heavy use of tight trailing you absolutely have to backtest with a different setup. I assume this is the case as you are talking about 2, 4 and 20 point targets (I assume you mean pips though?).

I can recommend using Tickdata Suite (https://eareview.net/tick-data-suite) as it allows you to also include slippage and execution delays which will both happen during live trading.

Alternatively, you can take a look at MT5 which allows for true tickdata Backtests.

kom75 2018-04-30 10:17:35 UTC #78

Thanks @KlondikeFX, I have Tick Data Suite and backtest on 1 month data is even worse than using Mt4 data, which I have used only becasue last 3 months data should be same as I had on demo acc. Talking about points I backtested indices where pips we use in forex (or at least I was learnt this terminology). With this algo I trade currencies live as well and results are very similar to demo one, so would expect same from SPX. Purpose of these posts were not to find how to backtest algo (which is next step) but why demo results differ so much from backtest

KlondikeFX 2018-04-30 10:36:27 UTC #79

Assumably because the backtest is not correct. If you have your EA coded in a way that it is making decisions intra-bar you need to do very precise backtests. If you want to get to the bottom of this I would advise to record the tickdata on the demo account you are currently trading. Here is an example how to do this:

Afterwards you import this recorded tickdata into tickddata suite and compare the backtest generated from this data with your trading results. They should be very close. Otherwise you need to dig deeper inside your code. This is where the fun begins

kom75 2018-04-30 10:39:10 UTC #80

Thanks again @KlondikeFX:). I will go this way. After one week of collecting data I will compare results once again

Powered by Discourse, best viewed with JavaScript enabled

CFDs are complex instruments and come with a high risk of losing money rapidly due to leverage. 73 % of retail investor accounts lose money when trading CFDs with this provider. You should consider whether you understand how CFDs work and whether you can afford to take the high risk of losing your money.