1. Pitfalls of various trading systems
2. Pitfalls in trading system development
Designing a trading system is not easy. Financial market data is the amalgamation of fear, greed, noise, and generally, a whole salad of millions of people across the country if not the world taking bets practically simultaneously on the price of financial instruments. Given this fact, it is plain to see why so many financial advisers say "past performance is not indicative of future results".
While there is a temptation to try and create trading strategies that have wonderful performance on past data (also known as "in-sample data"), there have been numerous studies showing that when too much effort is placed into this activity, that the system fails going forward on unseen data (or what is more formally referred to as "out-of-sample data").
3. How to not overfit
This is more commonly known as overfitting, which is the act of trying to tailor a backtest to look as good as possible (smoothest line, highest profit, whatever the criteria may be), rather than seeing a backtest as an evaluation of a trading system.
In order to avoid such phenomena, trading systems should strive to have as few parameters--essentially the knobs you can turn to change settings--as possible. Why? Because the number of system configurations increases exponentially with parameters. That is, consider a grid search that tests five separate settings for each parameter. A one-parameter system would need only five runs. A two parameter system would need twenty five. A three parameter system would already need one hundred twenty five, and beyond that, the numbers start to get very high very fast.
4. Stability with system settings
Furthermore, a trading strategy should not perform much differently when the parameters are slightly altered. For instance, a trend-following system should have similar results for similar settings of its trend filter. For instance, a trend-following system that buys a stock when its 50-day moving average crosses above its 200-day moving average
5. Stability with system settings
should behave similarly if those numbers are 45 and 210, respectively.
6. Hypothesis testing
Lastly, one can conduct statistical hypothesis tests to determine whether or not the following properties are true: first of all, is there a relationship between an indicator and future returns? Secondly, is there statistical evidence for the signal process to generate outperformance against some benchmark?
While quantstrat's built-in mechanics to protect against overfitting are beyond the scope of this class, these are nevertheless important things to keep in mind.
7. Let's practice!
Let's move on to some exercises.