Introduction
ForecastBench evaluates probabilistic predictions on regularly updated questions about future events [1]. This report examines how research and a forecasting ensemble translate evidence into probabilities on a selected set of its questions. We compare both the direction of predictions and the size of their probability errors: a correct majority of outcomes can coexist with substantial losses on a smaller number of confident mistakes.
One question asked whether the Federal Reserve would raise rates by a quarter of a percentage point at its September 2026 meeting. Meridian Superforecast assigned that outcome a 78.8% probability. The question resolved Yes [3].
Cohort and reference forecasts
The cohort comprises all 36 eligible resolved binary market questions from the 13 September 2026 ForecastBench round: 21 from Kalshi, four from Manifold and eleven from Polymarket [3]. Inclusion required a recorded binary outcome and a resolution date after the forecast day and no later than 1 October 2026. Selection was independent of model performance.
Of the round’s 500 questions, we excluded 250 non-market questions, 208 records marked as unresolved or containing proxy market prices rather than resolved outcomes, and six market questions without resolution records [3]. The resulting cohort covers sports, entertainment, politics, economics and technology; it is not a random sample of the full round. The evidence cutoff for forecasts is 13 September 2026 at 23:59:59 UTC, distinct from the 1 October resolution cutoff used to determine eligibility.
Reference forecasts are matched by source and question identifier, then scored against the same recorded outcomes as Meridian Superforecast. We include 49 archived submissions with complete coverage of all 36 questions. Each model variant is a separate submission. The headline shows all 50 submissions ordered by mean Brier loss on the 12-question core slice, with a common zero-based 0–0.5 scale. The introductory comparison shows Meridian Superforecast, Torchcast AI (dimoo-nano), Google Cloud (NanoForecaster-MTV-0913) and Vaticinus (vaticinus-3.0-raw). Detailed figures also include TRACES (TRACES), Yhat Labs (vani), NeuroFusion (new ensemble 1) and Cenva Intelligence (norma-market-v2). These seven named reference systems provide a fixed comparison roster, with one archived submission per organisation and complete coverage of both cohorts. The same variants are used throughout; Vaticinus is included as the core-domain leader. Rank is determined by Brier loss across all 50 submissions, including Meridian Superforecast, rather than only the displayed references.
Evaluation procedure
Meridian Superforecast receives each question, its resolution criteria and a research brief prepared within the historical evidence cutoff. It uses this evidence to assign a probability to the Yes outcome. We score that final probability against the recorded resolution. The study evaluates research and Meridian Superforecast alone, without Meridian’s simulation or final synthesis stages.
Recorded outcomes and archived reference forecasts are excluded from the evidence supplied to Meridian Superforecast. Market evidence available before the cutoff is permitted; the study does not separately measure its contribution.
Scoring and interpretation
For forecast probability p and binary outcome y, Brier loss is (p − y)². We report BS = (1/N) Σᵢ₌₁ᴺ (pᵢ − yᵢ)², where N is 12 for the core slice and 36 for the full cohort, giving every question equal weight. The loss ranges from zero to one, with lower values indicating smaller probability error. Assigning 50% throughout gives 0.25. This constant forecast provides a scale reference, rather than a competitive baseline.
A prediction is directionally correct when the realised outcome receives strictly more than 50% probability; an exact 50% forecast does not count as correct. Directional accuracy is the percentage of questions meeting this condition. Unlike this accuracy measure, Brier loss distinguishes a tentative mistake from a confident one.
ForecastBench’s official leaderboard adjusts for question difficulty and reports a Brier Index [2]. The equal-question mean reported here is a local comparison on the stated matched-question cohort, not an official leaderboard score or position.
Core-domain results
The core slice includes all 12 questions about financial markets, monetary policy, elections and diplomacy: two Bitcoin price questions, two Federal Reserve decisions, six election outcomes and two diplomatic events. It excludes sports, entertainment, publishing, aerospace and pure mathematics. This is a retrospective subject-based slice, not the twelve best-scoring predictions.
Meridian Superforecast gets 10 of 12 directions correct (83.3%), with mean Brier loss 0.1639. Google Nano Forecaster and Torchcast also achieve 83.3% accuracy, with losses of 0.1702 and 0.1762 respectively; Vaticinus leads the core slice with loss 0.1423 and 83.3% accuracy. Meridian Superforecast ranks 5th of 50. These descriptive results apply to this small domain slice.
Full-cohort results
The full cohort includes many questions outside Meridian’s core remit of finance, monetary policy, elections and diplomacy, including sports results, entertainment awards and pure mathematics. This comparison measures performance across a broader range of topics; the core-domain results are reported separately above.
Meridian Superforecast records a mean Brier loss of 0.1927 and 29 correct directions out of 36 (80.6%). Meridian Superforecast ranks 34th of 50 submissions. Torchcast AI’s dimoo-nano scores 0.1359 and Google Cloud’s NanoForecaster-MTV-0913 scores 0.1416; both have 30 correct directions.
Meridian Superforecast’s loss is approximately 22.9% below the constant-50% baseline: (0.25 − 0.1927) / 0.25 ≈ 22.9%. This improvement establishes a comparison with an uninformative forecast on this cohort; the lower losses of the leading references show the remaining gap to those submissions.
Results analysis
Meridian’s research draws on Nebula, its data-ingestion and market-intelligence layer, alongside external research. Nebula’s coverage is concentrated in financial markets, macroeconomics and geopolitics. Richer domain context is a plausible contributor to the stronger core-domain result; this comparison does not isolate its effect from the forecasting models or other research inputs.
The seven incorrect directions contribute 67.1% of Meridian Superforecast’s total Brier loss. Each assigns less than 50% to a Yes outcome in the dataset. Errors are therefore concentrated in a minority of questions, rather than distributed evenly across the cohort. This explains why 80.6% directional accuracy does not translate into a similarly strong probability score.
For example, the question about a return of If Anyone Builds It to a New York Times bestseller list receives 11.6% Yes probability and resolves Yes, contributing 0.7815 loss. The Russian parliamentary election turnout question, for the 59–62% interval, receives 14% and also resolves Yes, contributing 0.7396. Each error is more than three times the 0.25 loss of a 50% prediction. These cases identify where probability assignment was costly; the scores alone do not determine whether evidence coverage, interpretation or probability aggregation caused the errors.
Torchcast and Google each get only one additional direction correct, while achieving appreciably lower mean loss. The comparison therefore depends on confidence across the full question set, not simply the number of correct outcomes. Establishing a systematic tendency to underpredict Yes outcomes would require broader evaluation; the seven errors here are descriptive evidence, not a calibration estimate.
Limitations and reproducibility
These forecasts use historical evidence cutoffs on already resolved questions. A cutoff constrains retrieved evidence but cannot remove knowledge in model parameters or guarantee that a changing web page retains its historical wording. Actual exposure to market probabilities has not been independently established, and matching question identities does not equalise the systems’ information access or computational budgets.
The cohort is small, selected and includes related events. Resolution availability and topic composition can influence its aggregate score. We report descriptive differences without confidence intervals, significance claims or conclusions about general calibration. The reference ordering applies to these submissions on these questions.
The pinned ForecastBench archive supplies question records, resolution labels and reference probabilities [3]. Reproducing the local scores also requires Meridian Superforecast’s question-level probabilities and the scoring conventions above. The underlying research materials and proprietary implementation are not released with this article; the public source archive alone is insufficient to reproduce the system’s forecasts.
Conclusion
Meridian Superforecast achieves 83.3% directional accuracy in its core domains and 80.6% across the full cohort. It ranks 5th of 50 submissions on the core slice and 34th of 50 on the full cohort. Its core-domain Brier loss is below Google and Torchcast, but above the leading Vaticinus submission. A small set of confidently underestimated outcomes accounts for most of its error. The result makes probability assignment on those cases a concrete subject for further investigation.
Hidden Systems Research will extend these evaluations to additional systems and cohorts as further studies are completed.
References
- Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E. Tetlock (2025). ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. International Conference on Learning Representations.
- ForecastBench (2026). Leaderboards and scoring methodology. Accessed 21 October 2026.
- Forecasting Research Institute (2026). ForecastBench datasets: question sets, resolutions and forecast submissions. Revision 5e184cb7ccf4d577787c8201e08875e2fe070a43.