Forex Maths & Physics

Overfitting — The #1 Way Backtests Fool You

The most dangerous chart in trading is a gorgeous equity curve from a backtest. It feels like proof. It is usually an illusion — the result of overfitting, the single biggest reason real-money results disappoint after a "perfect" test.

Signal vs noise

Every price history contains a little genuine structure (signal) and a lot of randomness (noise). When you tune a strategy's parameters until the backtest looks spectacular, you are almost always fitting the noise — the specific accidents of that one history. Run it forward on data it has never seen and the noise is different, so the magic evaporates.

A quick intuition

Give a model enough knobs and it can "explain" anything, including pure coincidence. With twenty parameters you can fit an elephant; with twenty-one you can make it wiggle its trunk. A rule with many tunable thresholds, filters and exceptions is a rule that has memorised the past, not understood it.

Measuring the danger: PBO

Quants quantify this with the Probability of Backtest Overfitting (PBO):

In words: across many splits of the data, how often does the strategy that looked best in-sample (IS) turn out below median out-of-sample (OOS)? A high PBO means your selection process is essentially picking lucky noise — the in-sample winner is no better than a coin flip out of sample.

The defences that actually work

• Hold out data you never touch until the very end. One look is all you get.
• Walk-forward testing. Train on a window, test on the next, roll forward. It mimics how you would actually trade.
• Penalise complexity. Fewer parameters, simpler logic, economic rationale. If you cannot say *why* an edge should exist, suspect noise.
• Deflate your metrics. Adjust the Sharpe for the number of trials (see our Sharpe article).
• Demand robustness. A real edge degrades gracefully as you nudge parameters; an overfit one collapses.

The honest mindset

Treat every stunning backtest as guilty until proven innocent. The question is never "how good does it look?" but "how much of this survives on data the model has never seen?" The discipline of assuming you have fooled yourself is, paradoxically, the most reliable edge of all.

Educational content only — NOT financial advice and not a recommendation to trade. CFDs and spread bets are high-risk leveraged products; the majority of retail investor accounts lose money. Maniflood does not hold client money or trade on anyone's behalf.
Maniflood is a trading name and subsidiary of TopologixLabs LTD (registered in England & Wales, Company No. 17246838). Contact: info@topologixlabs.com · +44 7519 421551