How to read a stat-arb backtest without fooling yourself

A backtest is a story. The interesting question is which version of the truth it is telling.

Every stat-arb platform shows a backtest in its marketing material. Almost all of them have at least one of five problems that turn the equity curve into fiction. Knowing those problems lets you read a backtest the way an analyst reads a financial statement: the numbers matter less than the methodology that produced them.

What follows is a short field guide. The references at the bottom point to the literature behind each item. The full technical treatment is in the implementation paper.

A backtest with tunable parameters needs to tune those parameters on data the evaluation does not see. Otherwise it is reporting how well the parameters fit history, not how the strategy generalises.

The standard fix is walk-forward partitioning. Split the data into rolling training windows and sequential test windows. Tune on training, evaluate on the immediately following test, slide forward and repeat. The aggregated test-window P&L is the out-of-sample backtest you should actually care about.

If a report does not specify walk-forward windows and step sizes, assume the parameters were fit on the same data as the evaluation.

The historical price path that actually happened is one draw from a distribution of paths that could have happened. The fact that your strategy worked on the realised path is weak evidence that it will work on the path that follows.

Block bootstrap

Naive iid bootstrap is inappropriate for time series because of serial dependence. Block bootstrap resamples contiguous blocks to preserve local autocorrelation.

Monte Carlo simulation

Simulate alternative paths under an explicit data-generating process. A defensible backtest shows both the realised equity curve and the resampled distribution around it.

Three formal tools matter here: Probabilistic Sharpe Ratio, Deflated Sharpe Ratio and Probability of Backtest Overfitting.

PSR asks how likely the true Sharpe exceeds a benchmark once you account for non-normality. DSR adjusts that further when the reported strategy is the best of many tested strategies. PBO measures how often in-sample winners fail out of sample.

These metrics are diagnostics, not decoration. A strategy selected purely by maximising in-sample Sharpe with PBO near 1 should be discarded.

Pairs trading is high-turnover. A backtest that assumes mid-price fills and zero transaction cost usually overstates achievable P&L by more than the true edge.

• Bid-ask spread. Fills occur at the side of the book, not the midpoint.
• Slippage. Larger orders walk the book.
• Exchange fees. Including maker-vs-taker differences.
• Funding payments. For perpetual futures at venue cadence.
• Borrow costs. In equity markets.
• Latency. Between signal generation and exchange acknowledgement.

If a backtest reports a high Sharpe and does not name its fee, funding and slippage assumptions, the reported Sharpe is probably not surviving the real cost model.

A pair universe built from currently tradeable assets excludes assets that were delisted or disappeared from the record. That silently removes failures and overstates stability.

The fix is a point-in-time universe: at each historical time t, restrict the backtestable universe to assets that were actually listed, tradeable and liquid enough at that moment.

A related issue is future leakage in pair selection itself. If you use information from 2024 to pick pairs you then “trade” in 2019, you are selecting on the future.

A defensible pairs-trading backtest report should include at minimum:

1. A point-in-time universe description.
2. A method description sufficient to reproduce the strategy.
3. Walk-forward partitioning details.
4. Aggregate out-of-sample performance with PSR / DSR where relevant.
5. Robustness diagnostics such as bootstrap or Monte Carlo distributions.
6. A transaction-cost decomposition.

Reports that omit walk-forward partitioning, show only one equity curve or hide the transaction-cost model are not evidence of deployable performance. The full technical treatment is at Pairs Trading Implementation §4.