Introduction
Mantic Crucible investigates disagreements between forecasting systems by rewarding questions that elicit different predictions [1]. This study examines how research and Meridian Superforecast translate evidence into numerical probability distributions across markets, economics, elections and geopolitics.
For example, one question asked for the average effective federal funds rate in July 2026. The median of Meridian Superforecast’s forecast was 3.63%, matching the recorded monthly average [5]. Crucible scores the full distribution, so matching the median does not by itself earn a perfect score.
We score archived forecasts against recorded outcomes. These question-level losses differ from the tournament’s spot peer scores and question-writer disagreement rewards. The comparison measures distributional accuracy on matching questions; it does not reproduce the official tournament ranking.
CRPS · percentage points
Multiple-choice Brier loss
Study design and question selection
The cohort comprises 50 resolved numerical questions: 14 in financial markets, 16 in macroeconomics and monetary policy, nine in elections, ten in geopolitics and trade, and one on NVIDIA B200 GPU pricing. The 49 core-domain questions reflect Meridian’s subject coverage; GPU pricing is an additional case. Selection is purposive, requiring a numerical resolution, a defined scoring interval and an archived Mantic forecast. This is not a random sample of Crucible or an untouched holdout.
Both systems are scored against the same outcome and numerical answer specification for each question. The reference is Mantic’s archived submission in the corresponding question record, rather than a current community aggregate. The full results identify every question, its historical evidence cutoff, scoring bounds, units and both systems’ losses.
The introduction retains two flagship examples from the original four-case comparison: the federal funds rate and Biogen’s trial statement [2] [5]. These were selected by the lowest Meridian Superforecast-to-Mantic loss ratios in that comparison. Biogen is categorical and is excluded from the 50-question numerical mean. The two detailed figures show the first five question IDs in financial markets and geopolitics respectively; they are illustrations, not domain averages or the basis for the overall score.
Evaluated system and evidence
Meridian Superforecast receives each question, its resolution criteria and a research brief prepared using its historical evidence cutoff. It produces a numerical probability distribution, and we score its final forecast. This study evaluates research and Meridian Superforecast alone, without Meridian’s simulation or final synthesis stages.
Recorded outcomes and archived competitor forecasts are excluded from the evidence supplied to Meridian Superforecast. Market evidence available before the cutoff is permitted. The evaluation is retrospective: forecasts were generated after the questions had resolved. Historical evidence restrictions do not establish that the underlying models were unaware of the outcomes.
Scoring and interpretation
For a numerical forecast with cumulative distribution F and realised value y, range-restricted CRPS is L[a,b] = ∫ₐᵇ (F(x) − 1{x ≥ y})² dx. Lower loss is better. Meridian Superforecast’s distributions retain cumulative probabilities immediately before and at each coordinate, preserving jumps at exact values. Between coordinates, the scorer interpolates from the left coordinate’s post-jump probability to the right coordinate’s pre-jump probability and splits integration intervals at the outcome. Archived Mantic distributions use linear interpolation on their supplied grids. Proper scoring rules provide the framework for assessing probability forecasts [6].
We normalize each question’s loss by its published scoring-grid span: nCRPSᵢ = L[aᵢ,bᵢ] / (bᵢ − aᵢ). The overall score is (1/50) Σᵢ nCRPSᵢ, giving each question equal weight. This unitless score ranges from zero to one. It permits aggregation across different numerical units; raw CRPS values in dollars, counts and percentage points are not averaged together. The score is not percentage accuracy, and its scale depends on the chosen question intervals.
Each question retains its specified open or closed bounds. The score measures error only over the represented interval, excluding the shape of any probability outside it. The full results give each interval and boundary convention. The federal funds example reports native CRPS in percentage points over 0–5; its lower bound is closed and its upper bound open. The Biogen example instead uses half-summed multiclass Brier loss, L = ½ Σₖ (pₖ − yₖ)², where yₖ is one for the realised option and zero otherwise. These individual scores are distinct from the mean normalized CRPS reported for the 50-question numerical cohort.
Research and Meridian Superforecast: financial markets
The financial-market examples cover NVIDIA revenue, copper prices, S&P 500 futures positioning, private-company valuations and Agronomics shares. Each panel compares the two systems on one question, with the same normalization and scale. The displayed questions include both Meridian Superforecast and Mantic advantages; they do not estimate performance across the entire domain.
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Research and Meridian Superforecast: geopolitics and trade
The geopolitical examples concern Taiwan Strait aircraft activity, sanctions, shipping transits, export controls and political prisoners. Meridian Superforecast has lower loss on the BIS Entity List and Venezuelan prisoner-count questions, while Mantic has lower loss on the other three displayed cases. These differences describe probability error; they do not establish which research inputs or forecasting mechanisms caused it.
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Normalized range-restricted CRPS · unitless
Results analysis
Across all 50 questions, Meridian Superforecast’s mean normalized loss is 0.1215 versus Mantic’s 0.1198: an absolute gap of 0.0017 and a relative difference of 1.4%, using Mantic’s mean as the denominator. Meridian Superforecast has lower loss on 20 questions and Mantic on 30, with no ties. The relatively close means coexist with a larger difference in question wins because the mean also reflects the magnitude of each advantage.
The flagship federal funds question illustrates a strong Meridian Superforecast result: native loss is 0.0758 percentage points versus Mantic’s 0.1517 [5]. On NVIDIA B200 pricing, Meridian Superforecast’s median is closer to the recorded $5.00 outcome, yet its distribution has higher native loss: 1.0454 versus 0.6040 USD per GPU-hour [4]. A closer central estimate therefore need not produce a better probability score. Neither example substitutes for the complete cohort comparison.
Limitations and reproducibility
The cohort is selected and includes related events. Fifty questions do not imply fifty independent observations. The small mean gap is descriptive: it establishes neither statistical equivalence nor general superiority, calibration or state-of-the-art status. The results concern Meridian Superforecast, not the complete Meridian system.
Historical evidence cutoffs cannot remove knowledge in model parameters or guarantee historical versions of retrieved pages. Current question records may contain later wording changes, and actual market-probability exposure has not been independently established. The cohort includes forecasts from different evaluated configurations; it is not a controlled comparison of one uniform configuration. Matching questions and scoring rules does not establish identical information access or computational budgets.
The full results publish all question-level losses, normalization intervals and source links. These allow the reported means to be checked. Recomputing the underlying losses requires the exact cumulative distributions and targets; reproducing forecasts also requires research materials and the proprietary implementation. Those materials are not released with this article, and a changing live archive is not an equivalent historical input.
Conclusion
Meridian Superforecast is competitive with Mantic on this selected numerical evaluation, with mean normalized CRPS of 0.1215 against 0.1198. Mantic retains the lower overall loss and wins more individual comparisons. The domain examples show variation across questions; the complete 50-question results provide the basis for assessing the comparison.
Hidden Systems Research will extend these evaluations to additional systems and cohorts as further studies are completed.
References
- Day, B., and Shevlane, T. (2026). A new kind of forecasting tournament. Mantic, 3 April.
- Mantic (2026). Crucible question 74: Biogen CELIA first public top-line statement. Question, forecast and resolution record.
- Mantic (2026). Crucible question 120: Leading AI model at the end of April. Question, forecast and resolution record.
- Mantic (2026). Crucible question 529: NVIDIA B200 SXM advertised GPU-hour price. Question, forecast and resolution record.
- Mantic (2026). Crucible question 566: July monthly federal funds effective rate. Question, forecast and resolution record.
- Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. doi:10.1198/016214506000001437.