Lead
A single question can tell you whether a quantitative investment model is built to last—or just to look good on a backtest. Allocators and junior investors who rely on Sharpe ratios and drawdown charts often miss the deeper issue: specification. When predictors are chosen because they fit history rather than because they reflect an economic mechanism, apparent robustness is usually fragile.
Why specification matters
Any strategy mined from historical data—hedge funds, quant boutiques, systematic strategies—faces the same risk.
Correlation is not causation. Variables that explain past returns may do so only because of transient relationships, data quirks, or mechanical constructions. When regimes change, those relationships can vanish and the model breaks.
The one-liner test that changes a meeting
Ask this of the manager: “Why should each key variable predict future returns—briefly, in one sentence?” A defensible answer links the variable to a plausible economic mechanism (supply/demand, liquidity, behavioral bias, microstructure, etc.), states the expected sign and timing, and flags conditions under which the signal should fail. If the manager struggles, you’ve just exposed specification risk.
What good answers sound like
Strong responses do three things clearly:
– Tie each variable to an economic prior (the friction, incentive or structural force that creates the edge). – Show disciplined validation: out-of-sample tests, rolling windows, sensitivity and removal checks. – Explain when and why a predictor might stop working, and what the monitoring/recalibration plan is.
Weak or evasive answers
Watch out for these red flags:
– Heavy emphasis on in-sample metrics or p-values with no causal story. – No held-out validation or only trivial parameter tweaks presented as robustness. – Resistance to simple checks (remove the variable; show rolling performance). – Claims of “proprietary” secrecy without offering a conceptual framework you can follow.
How to probe effectively (no PhD required)
Short, disciplined questions get results:
– “What economic mechanism links this variable to returns?” – “When should the signal fail?” (Managers who can’t answer are guessing.) – “Which variables move together and why?” (Reveals collinearity.) – “Show rolling/out-of-sample performance and results when you drop each variable in turn.” – “How do turnover, capacity and transaction costs affect realized performance?”
Three practical tests to demand
1) Out-of-sample and walk-forward validation: mimic live deployment and show performance across regimes. 2) Leave-one-out and sensitivity analysis: quantify how much each variable matters. 3) Stress tests for regime shifts and liquidity shocks: simulate degradation scenarios and execution drag.
Operational governance: make variable selection part of the rulebook
Variable choice isn’t just a research detail; it belongs in risk governance. Require:
– A documented map from macro/sector drivers to stock-level signals. – A list of candidate variables, the inclusion/exclusion rationale, and the statistical thresholds used. – Monitoring metrics (signal concentration, turnover drivers, exposure to common factors) and escalation protocols for rapid decay. – Periodic revalidation schedules and independent audits of variable selection, data pipelines and code.
How to use this in due diligence
Add a concise item to your DDQ: “Explain, in one sentence each, why the top five variables belong in your model and why you excluded close alternatives.” Follow up with a short failure post-mortem and one trade-level explanation. Tone matters: clear, specific answers beat vague invocations of complexity or secrecy.
A checklist for allocators
– Require an economic rationale for each principal input. – Insist on out-of-sample and regime-aware testing. – Ask for variable-removal and sensitivity results. – Quantify capacity, crowding and implementation costs. – Integrate variable-level risks into portfolio budgets and limits. – Schedule regular revalidation and independent audits.
Why specification matters
Any strategy mined from historical data—hedge funds, quant boutiques, systematic strategies—faces the same risk. Correlation is not causation. Variables that explain past returns may do so only because of transient relationships, data quirks, or mechanical constructions. When regimes change, those relationships can vanish and the model breaks.0
Why specification matters
Any strategy mined from historical data—hedge funds, quant boutiques, systematic strategies—faces the same risk. Correlation is not causation. Variables that explain past returns may do so only because of transient relationships, data quirks, or mechanical constructions. When regimes change, those relationships can vanish and the model breaks.1
