Finertia.

How to tell if a backtest is overfit.

Every rule looks good on the years it was tuned on. Four checks that ask whether yours learned a pattern or memorised the noise.

What overfitting looks like

An overfit backtest has learned the noise in its history instead of a pattern that repeats. The tell is a strong result on the data the parameters were chosen on and a weak or negative one on data they were not. Momentum on Apple is a clean example: a Sharpe of 0.949 on the years it was tuned on, and −0.303 on the years after.

The more combinations you try, the better the best one looks by luck alone. Trying ten parameter sets and reporting the winner is not one test; it is ten, with the nine losers hidden.

Four checks, and what each asks

Walk-forward

Does the chosen combination still work on later data it was never tuned on? How it works.

Deflated Sharpe ratio

Is the winner's Sharpe higher than the best of that many tries would reach on pure noise? It compares against that expected maximum (Bailey and López de Prado, 2014) rather than against zero, and accounts for skew and fat tails.

Probability of backtest overfitting

The period is cut into eight blocks, and all 70 ways of choosing four of them as the training half are tried. Each time, the in-sample winner is ranked on the other half. The probability is how often it lands in the bottom half there. Around 0.5 means choosing by backtest was no better than choosing at random.

Permutation test

Does the timing of the trades matter, or would the same amount of time in the market have done as well? How it works.

A small grid is not a safe grid

MACD's grid here has only four combinations, which sounds too few to overfit. On the later Apple window its in-sample pick still came out with a probability of overfitting of 0.77, and a test-period Sharpe of −0.114. Few choices limit how much luck a search can find; they do not make what it finds real.

Habits that help

  • Decide the grid before looking at results, and count every run you make.
  • Prefer a rule that does reasonably across neighbouring parameters over one sharp peak.
  • Test on more than one stock and more than one split before believing a result.
  • Treat a confidence interval that contains zero as no evidence, not weak evidence.

On real stocks

On 39 US and Indian stocks, 2019–2024, the three rules were each tuned on the first 70% and scored on the rest: 21 of 117 held up. On 20 stocks none did, among them Apple, which had a Sharpe ratio of 1.21 for anyone who simply held it.

All the stocks

See it on a real strategy.

Finertia runs these checks on any stock you pick. The free plan runs the backtest; Pro adds the validation.

Create a free accountSee a full result first