What is a forecast actually worth — and in which regime?
A single averaged CRPS or MAE across a backtest hides the fact that the accuracy was earned across very different market regimes. The number looks stable; the model behind it is not.
It also answers the wrong question. A procurement lead does not care about a pinball loss — they care whether acting on the forecast would have saved money. ForecastLens does not train models; it tells you what your forecasting framework does not.
How it works
Regime-aware evaluation. Accuracy broken down by detected regime — using swappable volatility and CUSUM changepoint detectors — rather than only by calendar time.
Leak-safe backtesting. A causal walk-forward splitter and rolling decision buckets that are provably unable to see future data, with tests that assert it.
Economic decision layer. Forecasts translated into concrete decisions (“buy now vs. wait”) and scored in currency per unit.
Validated metrics, framework-agnostic. CRPS, MASE, WQL and pinball loss, each validated against a reference implementation; adapters normalise darts, neuralforecast or any tabular output into one immutable ForecastResult.
What came out of it
Both headline results come from the package’s synthetic presets, which ship with known ground truth so every claim can be reproduced with zero setup.
CRPS spread across regimes on a structural-break preset — some regimes score ≈0.94, others above 1.4
savings per unit in the supply-shock procurement demo: the model waits 11 days instead of buying at the spike
regime-switching price and demand presets with known ground truth
released on PyPI under Apache 2.0 — pre-alpha, API not yet stable
Stated plainly
- Pre-alpha: the API is not yet stable.
- The relative value score is a deliberately simplified MVP metric — (cost_naive − cost_model) / cost_naive — not the academic Murphy value score, so it can go negative or exceed 1.