The permutation test: skill, or just time in the market?

A strategy that is long most of the time in a rising market will make money. That is the market's return, not the strategy's. The permutation test asks whether when the strategy was in the market mattered, or whether being in the market that much was the whole result.

1. Take the strategy's daily positions: long, short or flat.
2. Shuffle their order 500 times. Each shuffle keeps exactly as many long, short and flat days; only the timing changes.
3. Run each shuffled series against the same returns and costs, and compute its Sharpe.
4. The p-value is the share of shuffles that matched or beat the real Sharpe. Below 0.05, the timing beat random timing.

A low p-value says the entries and exits carried information. A high one says the result could have come from the same exposure spread at random. Bollinger bands on Apple came out at p 0.066: close, but its entries could not be told apart from shuffled ones.

It is not a test of whether the strategy makes money. A rule can beat random timing and still lose after costs, or fail to beat simply holding the stock.

Shuffling single days breaks up the strategy's holding periods, so shuffled series trade more often and pay more in costs than the real one did. That tilts the test slightly in the strategy's favour, which is worth knowing when a p-value lands just under 0.05. It also says nothing about overfitting: a combination picked from a grid can beat random timing on the data it was picked on. The overfitting checks cover that.