Benchmarks/ForecastBench

ForecastBench

We evaluate Meridian Superforecast, the superforecasting component of Meridian, against archived ForecastBench submissions on matching questions and outcomes. Across core-domain questions in finance, monetary policy, elections and diplomacy, it achieves 83.3% directional accuracy and a mean Brier loss of 0.1639, ranking 5th of 50 submissions—matching the directional accuracy of Google Nano Forecaster and Torchcast while recording lower probability error on this slice. The study evaluates research and Meridian Superforecast alone, without Meridian’s simulation or final synthesis stages. See full results.

Core domains · Brier loss · lower is better
Vaticinus / vaticinus-3.0-raw0.1423
Anonymous 11 / jforecast-router-v10.1532
Vaticinus / vaticinus-3.00.1539
Ryso AI / Apex0.1629
Meridian Superforecast0.1639Meridian Superforecast · Rank 5th of 50
hliu763 / ForecasterX-Coco-09130.1692
Google Cloud / NanoForecaster-MTV-09130.1702
hliu763 / ForecasterX-Diego-09130.1704
hliu763 / ForecasterX-Butter-09130.1705
Google Cloud / NanoForecaster-ATL-09130.1706
Google Cloud / NanoForecaster-SVL-09130.1707
Afterflow / e12-competition-portfolio-v1-state_research0.1719
Anonymous 33 / Nathan0.1722
Yhat Labs / vani0.1756
Afterflow / e12-competition-portfolio-v1-conservative0.1759
Afterflow / e12-competition-portfolio-v1-state0.1759
Torchcast AI / dimoo-nano0.1762
TRACES / TRACES0.1762
Anonymous 33 / agentic-v10.1763
PathFi / PathFi-F60.1763
Anonymous 24 / ginger-rook0.1768
Anonymous 24 / honeycomb-fox0.1768
Anonymous 24 / mint-badger0.1768
NeuroFusion / new ensemble 10.1768
NeuroFusion / new ensemble 20.1768
NeuroFusion / new ensemble 30.1768
Ryso AI / Frost0.1778
Torchcast AI / hirono-nano0.1779
limeforecast / superpuper0.1781
Torchcast AI / hacipupu-nano0.1781
Ryso AI / Linen0.1785
yarrow / yarrow-kestrel-202609130.1803
Yhat Labs / kaal0.1846
Anonymous 29 / claude-sonnet-50.1868
PolyLedger / polyledger-quant0.1926
Anonymous 19 / ASA ForecastBench Shadow (routed_market_dataset_hybrid)0.1928
Elementary Deduction / Apollo0.1935
Elementary Deduction / Zeus0.1935
DL Solutions / dl-superforecaster-v20.1939
Anonymous 33 / fvb0.1940
Anonymous 22 / ant-batch-pa0.1944
Cenva Intelligence / norma-market-v20.1951
Anonymous 29 / mars-debate-v10.2023
yhat-labs / 8-ball0.2312
Phinomial Labs / phinomial-forecaster0.2357
Anonymous 11 / fs-panel-v10.2454
Anonymous 22 / ant-batch-ra0.2503
PolyLedger / polyledger-core0.2544
Cenva Intelligence / norma-v20.2565
PolyLedger / polyledger-pure0.4108
01

Introduction

ForecastBench evaluates probabilistic predictions on regularly updated questions about future events [1]. This report examines how research and a forecasting ensemble translate evidence into probabilities on a selected set of its questions. We compare both the direction of predictions and the size of their probability errors: a correct majority of outcomes can coexist with substantial losses on a smaller number of confident mistakes.

One question asked whether the Federal Reserve would raise rates by a quarter of a percentage point at its September 2026 meeting. Meridian Superforecast assigned that outcome a 78.8% probability. The question resolved Yes [3].

Core domains · 12 questions · Brier loss · lower is better
Vaticinus83.3% directional accuracy · Rank 1/500.1423
Meridian Superforecast83.3% directional accuracy · Rank 5/500.1639
Google Nano Forecaster83.3% directional accuracy · Rank 7/500.1702
Torchcast AI83.3% directional accuracy · Rank 17/500.1762
Fig. 1. Core-domain forecasts on 12 matched questions. Lower Brier loss is better. Ranks cover all 50 submissions; selected references are shown.
02

Cohort and reference forecasts

The cohort comprises all 36 eligible resolved binary market questions from the 13 September 2026 ForecastBench round: 21 from Kalshi, four from Manifold and eleven from Polymarket [3]. Inclusion required a recorded binary outcome and a resolution date after the forecast day and no later than 1 October 2026. Selection was independent of model performance.

Of the round’s 500 questions, we excluded 250 non-market questions, 208 records marked as unresolved or containing proxy market prices rather than resolved outcomes, and six market questions without resolution records [3]. The resulting cohort covers sports, entertainment, politics, economics and technology; it is not a random sample of the full round. The evidence cutoff for forecasts is 13 September 2026 at 23:59:59 UTC, distinct from the 1 October resolution cutoff used to determine eligibility.

Reference forecasts are matched by source and question identifier, then scored against the same recorded outcomes as Meridian Superforecast. We include 49 archived submissions with complete coverage of all 36 questions. Each model variant is a separate submission. The headline shows all 50 submissions ordered by mean Brier loss on the 12-question core slice, with a common zero-based 0–0.5 scale. The introductory comparison shows Meridian Superforecast, Torchcast AI (dimoo-nano), Google Cloud (NanoForecaster-MTV-0913) and Vaticinus (vaticinus-3.0-raw). Detailed figures also include TRACES (TRACES), Yhat Labs (vani), NeuroFusion (new ensemble 1) and Cenva Intelligence (norma-market-v2). These seven named reference systems provide a fixed comparison roster, with one archived submission per organisation and complete coverage of both cohorts. The same variants are used throughout; Vaticinus is included as the core-domain leader. Rank is determined by Brier loss across all 50 submissions, including Meridian Superforecast, rather than only the displayed references.

03

Evaluation procedure

Meridian Superforecast receives each question, its resolution criteria and a research brief prepared within the historical evidence cutoff. It uses this evidence to assign a probability to the Yes outcome. We score that final probability against the recorded resolution. The study evaluates research and Meridian Superforecast alone, without Meridian’s simulation or final synthesis stages.

Recorded outcomes and archived reference forecasts are excluded from the evidence supplied to Meridian Superforecast. Market evidence available before the cutoff is permitted; the study does not separately measure its contribution.

04

Scoring and interpretation

For forecast probability p and binary outcome y, Brier loss is (p − y)². We report BS = (1/N) Σᵢ₌₁ᴺ (pᵢ − yᵢ)², where N is 12 for the core slice and 36 for the full cohort, giving every question equal weight. The loss ranges from zero to one, with lower values indicating smaller probability error. Assigning 50% throughout gives 0.25. This constant forecast provides a scale reference, rather than a competitive baseline.

A prediction is directionally correct when the realised outcome receives strictly more than 50% probability; an exact 50% forecast does not count as correct. Directional accuracy is the percentage of questions meeting this condition. Unlike this accuracy measure, Brier loss distinguishes a tentative mistake from a confident one.

ForecastBench’s official leaderboard adjusts for question difficulty and reports a Brier Index [2]. The equal-question mean reported here is a local comparison on the stated matched-question cohort, not an official leaderboard score or position.

05

Core-domain results

The core slice includes all 12 questions about financial markets, monetary policy, elections and diplomacy: two Bitcoin price questions, two Federal Reserve decisions, six election outcomes and two diplomatic events. It excludes sports, entertainment, publishing, aerospace and pure mathematics. This is a retrospective subject-based slice, not the twelve best-scoring predictions.

Meridian Superforecast gets 10 of 12 directions correct (83.3%), with mean Brier loss 0.1639. Google Nano Forecaster and Torchcast also achieve 83.3% accuracy, with losses of 0.1702 and 0.1762 respectively; Vaticinus leads the core slice with loss 0.1423 and 83.3% accuracy. Meridian Superforecast ranks 5th of 50. These descriptive results apply to this small domain slice.

Core domains · 12 questions · Brier loss · lower is better
Vaticinus83.3% directional accuracy · Rank 1/500.1423
Meridian Superforecast83.3% directional accuracy · Rank 5/500.1639
Google Nano Forecaster83.3% directional accuracy · Rank 7/500.1702
Yhat Labs83.3% directional accuracy · Rank 14/500.1756
Torchcast AI83.3% directional accuracy · Rank 17/500.1762
TRACES83.3% directional accuracy · Rank 18/500.1762
NeuroFusion83.3% directional accuracy · Rank 21/500.1768
Cenva Intelligence66.7% directional accuracy · Rank 42/500.1951
Fig. 2. Core-domain forecasts on 12 matched questions. Lower Brier loss is better. Ranks cover all 50 submissions; selected references are shown.
06

Full-cohort results

The full cohort includes many questions outside Meridian’s core remit of finance, monetary policy, elections and diplomacy, including sports results, entertainment awards and pure mathematics. This comparison measures performance across a broader range of topics; the core-domain results are reported separately above.

Meridian Superforecast records a mean Brier loss of 0.1927 and 29 correct directions out of 36 (80.6%). Meridian Superforecast ranks 34th of 50 submissions. Torchcast AI’s dimoo-nano scores 0.1359 and Google Cloud’s NanoForecaster-MTV-0913 scores 0.1416; both have 30 correct directions.

Meridian Superforecast’s loss is approximately 22.9% below the constant-50% baseline: (0.25 − 0.1927) / 0.25 ≈ 22.9%. This improvement establishes a comparison with an uninformative forecast on this cohort; the lower losses of the leading references show the remaining gap to those submissions.

Full cohort · 36 questions · Brier loss · lower is better
Torchcast AI83.3% directional accuracy · Rank 4/500.1359
Google Nano Forecaster83.3% directional accuracy · Rank 7/500.1416
NeuroFusion80.6% directional accuracy · Rank 14/500.1440
Yhat Labs83.3% directional accuracy · Rank 20/500.1511
TRACES80.6% directional accuracy · Rank 21/500.1521
Cenva Intelligence69.4% directional accuracy · Rank 33/500.1894
Meridian Superforecast80.6% directional accuracy · Rank 34/500.1927
Vaticinus36.1% directional accuracy · Rank 42/500.2126
Fig. 3. Full-cohort forecasts on 36 matched questions, including many outside Meridian’s core remit, such as sports and entertainment. Lower Brier loss is better.
07

Results analysis

Meridian’s research draws on Nebula, its data-ingestion and market-intelligence layer, alongside external research. Nebula’s coverage is concentrated in financial markets, macroeconomics and geopolitics. Richer domain context is a plausible contributor to the stronger core-domain result; this comparison does not isolate its effect from the forecasting models or other research inputs.

The seven incorrect directions contribute 67.1% of Meridian Superforecast’s total Brier loss. Each assigns less than 50% to a Yes outcome in the dataset. Errors are therefore concentrated in a minority of questions, rather than distributed evenly across the cohort. This explains why 80.6% directional accuracy does not translate into a similarly strong probability score.

For example, the question about a return of If Anyone Builds It to a New York Times bestseller list receives 11.6% Yes probability and resolves Yes, contributing 0.7815 loss. The Russian parliamentary election turnout question, for the 59–62% interval, receives 14% and also resolves Yes, contributing 0.7396. Each error is more than three times the 0.25 loss of a 50% prediction. These cases identify where probability assignment was costly; the scores alone do not determine whether evidence coverage, interpretation or probability aggregation caused the errors.

Torchcast and Google each get only one additional direction correct, while achieving appreciably lower mean loss. The comparison therefore depends on confidence across the full question set, not simply the number of correct outcomes. Establishing a systematic tendency to underpredict Yes outcomes would require broader evaluation; the seven errors here are descriptive evidence, not a calibration estimate.

08

Limitations and reproducibility

These forecasts use historical evidence cutoffs on already resolved questions. A cutoff constrains retrieved evidence but cannot remove knowledge in model parameters or guarantee that a changing web page retains its historical wording. Actual exposure to market probabilities has not been independently established, and matching question identities does not equalise the systems’ information access or computational budgets.

The cohort is small, selected and includes related events. Resolution availability and topic composition can influence its aggregate score. We report descriptive differences without confidence intervals, significance claims or conclusions about general calibration. The reference ordering applies to these submissions on these questions.

The pinned ForecastBench archive supplies question records, resolution labels and reference probabilities [3]. Reproducing the local scores also requires Meridian Superforecast’s question-level probabilities and the scoring conventions above. The underlying research materials and proprietary implementation are not released with this article; the public source archive alone is insufficient to reproduce the system’s forecasts.

09

Conclusion

Meridian Superforecast achieves 83.3% directional accuracy in its core domains and 80.6% across the full cohort. It ranks 5th of 50 submissions on the core slice and 34th of 50 on the full cohort. Its core-domain Brier loss is below Google and Torchcast, but above the leading Vaticinus submission. A small set of confidently underestimated outcomes accounts for most of its error. The result makes probability assignment on those cases a concrete subject for further investigation.

Hidden Systems Research will extend these evaluations to additional systems and cohorts as further studies are completed.

10

References

  1. Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock (2025). ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. International Conference on Learning Representations.
  2. ForecastBench (2026). Leaderboards and scoring methodology. Accessed 21 October 2026.
  3. Forecasting Research Institute (2026). ForecastBench datasets: question sets, resolutions and forecast submissions. Revision 5e184cb7ccf4d577787c8201e08875e2fe070a43.