Benchmarks/Mantic Crucible

Mantic Crucible

We compare Meridian Superforecast, the superforecasting component of Meridian, with Mantic’s archived forecasts on 50 resolved numerical Crucible questions. Mean normalized CRPS is 0.1215 for Meridian Superforecast and 0.1198 for Mantic, with lower scores indicating smaller probability error. Meridian Superforecast is competitive with Mantic on this selected cohort, recording 1.4% higher mean loss and lower loss on 20 of the 50 questions. See full results.

50 questions · mean normalized CRPS · lower is better
Meridian Superforecast0.1215
Mantic0.1198
Questions with lower error
Meridian Superforecast20Mantic30
Overall loss gap: 1.4% · Meridian Superforecast’s loss is 1.4% higher than Mantic’s.
01

Introduction

Mantic Crucible investigates disagreements between forecasting systems by rewarding questions that elicit different predictions [1]. This study examines how research and Meridian Superforecast translate evidence into numerical probability distributions across markets, economics, elections and geopolitics.

For example, one question asked for the average effective federal funds rate in July 2026. The median of Meridian Superforecast’s forecast was 3.63%, matching the recorded monthly average [5]. Crucible scores the full distribution, so matching the median does not by itself earn a perfect score.

We score archived forecasts against recorded outcomes. These question-level losses differ from the tournament’s spot peer scores and question-writer disagreement rewards. The comparison measures distributional accuracy on matching questions; it does not reproduce the official tournament ranking.

Selected question comparisons · lower is better
Federal funds rate
Meridian Superforecast0.0758
Mantic0.1517

CRPS · percentage points

Biogen trial statement
Meridian Superforecast0.4112
Mantic0.3304

Multiple-choice Brier loss

Fig. 1. Selected examples from the original four-question comparison; separate from the 50-question numerical aggregate.
02

Study design and question selection

The cohort comprises 50 resolved numerical questions: 14 in financial markets, 16 in macroeconomics and monetary policy, nine in elections, ten in geopolitics and trade, and one on NVIDIA B200 GPU pricing. The 49 core-domain questions reflect Meridian’s subject coverage; GPU pricing is an additional case. Selection is purposive, requiring a numerical resolution, a defined scoring interval and an archived Mantic forecast. This is not a random sample of Crucible or an untouched holdout.

Both systems are scored against the same outcome and numerical answer specification for each question. The reference is Mantic’s archived submission in the corresponding question record, rather than a current community aggregate. The full results identify every question, its historical evidence cutoff, scoring bounds, units and both systems’ losses.

The introduction retains two flagship examples from the original four-case comparison: the federal funds rate and Biogen’s trial statement [2] [5]. These were selected by the lowest Meridian Superforecast-to-Mantic loss ratios in that comparison. Biogen is categorical and is excluded from the 50-question numerical mean. The two detailed figures show the first five question IDs in financial markets and geopolitics respectively; they are illustrations, not domain averages or the basis for the overall score.

03

Evaluated system and evidence

Meridian Superforecast receives each question, its resolution criteria and a research brief prepared using its historical evidence cutoff. It produces a numerical probability distribution, and we score its final forecast. This study evaluates research and Meridian Superforecast alone, without Meridian’s simulation or final synthesis stages.

Recorded outcomes and archived competitor forecasts are excluded from the evidence supplied to Meridian Superforecast. Market evidence available before the cutoff is permitted. The evaluation is retrospective: forecasts were generated after the questions had resolved. Historical evidence restrictions do not establish that the underlying models were unaware of the outcomes.

04

Scoring and interpretation

For a numerical forecast with cumulative distribution F and realised value y, range-restricted CRPS is L[a,b] = ∫ₐᵇ (F(x) − 1{x ≥ y})² dx. Lower loss is better. Meridian Superforecast’s distributions retain cumulative probabilities immediately before and at each coordinate, preserving jumps at exact values. Between coordinates, the scorer interpolates from the left coordinate’s post-jump probability to the right coordinate’s pre-jump probability and splits integration intervals at the outcome. Archived Mantic distributions use linear interpolation on their supplied grids. Proper scoring rules provide the framework for assessing probability forecasts [6].

We normalize each question’s loss by its published scoring-grid span: nCRPSᵢ = L[aᵢ,bᵢ] / (bᵢ − aᵢ). The overall score is (1/50) Σᵢ nCRPSᵢ, giving each question equal weight. This unitless score ranges from zero to one. It permits aggregation across different numerical units; raw CRPS values in dollars, counts and percentage points are not averaged together. The score is not percentage accuracy, and its scale depends on the chosen question intervals.

Each question retains its specified open or closed bounds. The score measures error only over the represented interval, excluding the shape of any probability outside it. The full results give each interval and boundary convention. The federal funds example reports native CRPS in percentage points over 0–5; its lower bound is closed and its upper bound open. The Biogen example instead uses half-summed multiclass Brier loss, L = ½ Σₖ (pₖ − yₖ)², where yₖ is one for the realised option and zero otherwise. These individual scores are distinct from the mean normalized CRPS reported for the 50-question numerical cohort.

05

Research and Meridian Superforecast: financial markets

The financial-market examples cover NVIDIA revenue, copper prices, S&P 500 futures positioning, private-company valuations and Agronomics shares. Each panel compares the two systems on one question, with the same normalization and scale. The displayed questions include both Meridian Superforecast and Mantic advantages; they do not estimate performance across the entire domain.

Financial markets · lower is better
Q70. What will NVIDIA's TTM revenue be at the end of Q1 FY2027?
Meridian Superforecast0.0314
Mantic0.0323

Normalized range-restricted CRPS · unitless

Q87. What will the London Metal Exchange (LME) official 3-month copper price be on July 1, 2026, in USD per metric tonne?
Meridian Superforecast0.1160
Mantic0.0639

Normalized range-restricted CRPS · unitless

Q98. What will be the net positions of Non-Commercials in S&P 500 E-Mini Futures
Meridian Superforecast0.0818
Mantic0.0736

Normalized range-restricted CRPS · unitless

Q139. What will be the third-highest valuation reported for any private-company transaction by July 12, 2026?
Meridian Superforecast0.0554
Mantic0.3309

Normalized range-restricted CRPS · unitless

Q144. What will be the share price of Agronomics on the London Stock Exchange at close of trading on 6th of August 2026
Meridian Superforecast0.0137
Mantic0.0170

Normalized range-restricted CRPS · unitless

Fig. 2. 5 individual question comparisons. Range-restricted CRPS divided by each question’s scoring interval; all panels use a 0–1 scale.
06

Research and Meridian Superforecast: geopolitics and trade

The geopolitical examples concern Taiwan Strait aircraft activity, sanctions, shipping transits, export controls and political prisoners. Meridian Superforecast has lower loss on the BIS Entity List and Venezuelan prisoner-count questions, while Mantic has lower loss on the other three displayed cases. These differences describe probability error; they do not establish which research inputs or forecasting mechanisms caused it.

Geopolitics and trade · lower is better
Q101. How many PLA aircraft sorties will Taiwan’s MND report as crossing the Taiwan Strait median line on the busiest 24-hour reporting period from April 25 to August 10, 2026?
Meridian Superforecast0.0900
Mantic0.0461

Normalized range-restricted CRPS · unitless

Q135. How many new individuals and entities will be added to OFAC's SDN List between January 1, 2026 and August 1, 2026?
Meridian Superforecast0.2905
Mantic0.1982

Normalized range-restricted CRPS · unitless

Q161. What will be the total combined weekly transit calls through the Suez Canal, the Cape of Good Hope, and the Bab el-Mandeb Strait for the week ending August 2, 2026?
Meridian Superforecast0.3917
Mantic0.1853

Normalized range-restricted CRPS · unitless

Q247. Net change in entities on the U.S. BIS Entity List via Federal Register rules published between June 1 and August 12, 2026?
Meridian Superforecast0.0243
Mantic0.0423

Normalized range-restricted CRPS · unitless

Q266. How many political prisoners will Foro Penal report are detained in Venezuela, according to the most recent count published on or before August 11, 2026?
Meridian Superforecast0.1535
Mantic0.2295

Normalized range-restricted CRPS · unitless

Fig. 3. 5 individual question comparisons. Range-restricted CRPS divided by each question’s scoring interval; all panels use a 0–1 scale.
07

Results analysis

Across all 50 questions, Meridian Superforecast’s mean normalized loss is 0.1215 versus Mantic’s 0.1198: an absolute gap of 0.0017 and a relative difference of 1.4%, using Mantic’s mean as the denominator. Meridian Superforecast has lower loss on 20 questions and Mantic on 30, with no ties. The relatively close means coexist with a larger difference in question wins because the mean also reflects the magnitude of each advantage.

The flagship federal funds question illustrates a strong Meridian Superforecast result: native loss is 0.0758 percentage points versus Mantic’s 0.1517 [5]. On NVIDIA B200 pricing, Meridian Superforecast’s median is closer to the recorded $5.00 outcome, yet its distribution has higher native loss: 1.0454 versus 0.6040 USD per GPU-hour [4]. A closer central estimate therefore need not produce a better probability score. Neither example substitutes for the complete cohort comparison.

08

Limitations and reproducibility

The cohort is selected and includes related events. Fifty questions do not imply fifty independent observations. The small mean gap is descriptive: it establishes neither statistical equivalence nor general superiority, calibration or state-of-the-art status. The results concern Meridian Superforecast, not the complete Meridian system.

Historical evidence cutoffs cannot remove knowledge in model parameters or guarantee historical versions of retrieved pages. Current question records may contain later wording changes, and actual market-probability exposure has not been independently established. The cohort includes forecasts from different evaluated configurations; it is not a controlled comparison of one uniform configuration. Matching questions and scoring rules does not establish identical information access or computational budgets.

The full results publish all question-level losses, normalization intervals and source links. These allow the reported means to be checked. Recomputing the underlying losses requires the exact cumulative distributions and targets; reproducing forecasts also requires research materials and the proprietary implementation. Those materials are not released with this article, and a changing live archive is not an equivalent historical input.

09

Conclusion

Meridian Superforecast is competitive with Mantic on this selected numerical evaluation, with mean normalized CRPS of 0.1215 against 0.1198. Mantic retains the lower overall loss and wins more individual comparisons. The domain examples show variation across questions; the complete 50-question results provide the basis for assessing the comparison.

Hidden Systems Research will extend these evaluations to additional systems and cohorts as further studies are completed.

10

References

  1. Day, B., and Shevlane, T. (2026). A new kind of forecasting tournament. Mantic, 3 April.
  2. Mantic (2026). Crucible question 74: Biogen CELIA first public top-line statement. Question, forecast and resolution record.
  3. Mantic (2026). Crucible question 120: Leading AI model at the end of April. Question, forecast and resolution record.
  4. Mantic (2026). Crucible question 529: NVIDIA B200 SXM advertised GPU-hour price. Question, forecast and resolution record.
  5. Mantic (2026). Crucible question 566: July monthly federal funds effective rate. Question, forecast and resolution record.
  6. Gneiting, T., and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378. doi:10.1198/016214506000001437.