Calibration and Collective Judgment

A forecast concerns a specified outcome at a specified horizon, conditional on information available when the judgment is made. A binary question requires more than a topic and a deadline. It needs criteria that distinguish occurrence from non-occurrence, an admissible source for resolution and a policy for cases that become ambiguous. An earnings announcement, an analyst's estimate and a subsequent restatement are different events. A forecast cannot be evaluated consistently if these distinctions are settled only after the result is known.

Probability expresses uncertainty about that outcome. It does not measure the fluency of the explanation, the forecaster's enthusiasm or the desirability of the event. A 70% forecast allows for non-occurrence; one such outcome does not establish that the estimate was unreasonable. Assessment requires a collection of forecasts and a rule for evaluating the probabilities assigned before resolution. The collection must also be relevant to the claim: performance on short-horizon earnings questions does not establish equal skill on long-horizon political events.

The term superforecasting emerged from research on sustained performance in forecasting tournaments. In a two-year geopolitical tournament, Mellers and colleagues studied probability training, collaboration and the identification of strong forecasters. Their findings support the value of these interventions within that setting, including improvements in calibration and resolution. They do not imply that exceptional performance is a fixed personal trait that transfers unchanged to every domain, or that copying a reasoning style is sufficient to reproduce the result.

The interpretation of those interventions remains contested. Hauenstein and colleagues (2025)reanalysed tournament data with an item-response model and reported that accounting for additional sources of variation substantially changed estimated effects on latent forecasting ability. This distinction matters: improvement in a measured score and a claim about the psychological ability responsible for it are different conclusions. The reanalysis is a reason to specify the quantity being estimated and the statistical assumptions, rather than to present a single explanation of tournament success as settled.

It is useful to separate three objects of evaluation: the individual judgment, the process that produces it and the aggregate of several judgments. An aggregate may outperform many of its contributors even when no contributor improves. Conversely, a better research process may not improve an aggregate if its outputs receive little weight. A study should identify which object changes and which comparison supports the conclusion, rather than attributing every improvement to the intelligence of the underlying forecasters.

Calibration concerns agreement between forecast probabilities and observed frequencies. Resolution concerns whether forecasts distinguish situations with different outcome rates. Sharpness describes how concentrated the predictive distributions are, without considering the outcomes. These properties are related but not interchangeable. The framework developed by Gneiting, Balabdaoui and Raftery treats concentration as desirable subject to calibration. A confident forecast is not useful merely because it is confident; an uninformative forecast can also exhibit good overall calibration when it reproduces the population's base rate.

For illustration, consider a population of questions in which an event occurs 20% of the time. Always reporting 20% can match that overall frequency while doing nothing to identify which cases are more likely to occur. Forecasts that separate higher-risk from lower-risk cases can provide additional information. Equally, a system can match the overall rate while systematically overstating one group's probabilities and understating another's. The relevant groups and sample sizes should therefore accompany a calibration analysis; an overall diagram does not settle every conditional question.

A scoring rule assigns a numerical value to a forecast and its outcome. A strictly proper rule makes reporting one's actual probability uniquely optimal in expectation, under the rule's assumptions. Gneiting and Raftery provide a general treatment. For binary outcomes, the Brier loss convention used here is the squared difference between probability and outcome, averaged across questions. A realised event is encoded as one and a non-event as zero. Smaller losses indicate better performance under this rule; they are not percentages of correct answers.

Consider four hypothetical outcomes: occurrence, non-occurrence, occurrence and non-occurrence. Forecaster A assigns probabilities of 0.8, 0.2, 0.6 and 0.4; forecaster B assigns 0.6, 0.4, 0.6 and 0.4. A's squared errors are 0.04, 0.04, 0.16 and 0.16, for a mean of 0.10. B's mean is 0.16. Both select the correct side of 50% on every question, but A assigns more probability to the realised outcomes on the first two. Four examples are enough to explain the calculation, not to establish either forecaster's calibration.

A relative skill score adds a reference forecast to the interpretation. Under a common Brier convention, skill is one minus the system's mean loss divided by the reference's mean loss, provided the latter is positive. The two losses must use the same questions, outcome definitions and weighting. A positive value then indicates improvement relative to that reference on that evaluation set. It does not establish performance against a different reference or eliminate uncertainty about how the ordering would change on new questions. The raw losses and reference should remain available alongside the summary.

The target space determines which probabilities can be scored together. Mutually exclusive and exhaustive answers to a multiple-choice question form one distribution. Overlapping events do not: a stock can exceed two different price thresholds on the same date, so their probabilities must not be forced to sum to one. Numerical outcomes require a distribution or explicitly defined summaries such as quantiles. Converting every task into a binary question may simplify presentation while discarding information that the original decision requires. The evaluation should preserve the target it claims to assess.

A defensible forecast begins with a question that can be resolved and evidence that can be assigned to an information cutoff. The outside view uses a reference class of comparable cases to establish a starting point. The inside view considers what makes the current case different. Neither is sufficient without judgment about comparability. A broad reference class may conceal relevant differences, while a narrowly selected class may contain too few cases or reflect knowledge of the outcome the forecaster is trying to predict.

Decomposition can expose hidden assumptions. A question about whether Amazon will announce a particular expansion by a deadline might depend on financing, regulatory decisions and a public announcement. These dependencies can help organise research. They should not be multiplied as though independent unless that assumption is justified. Conditional probabilities need to refer to the same sequence of events, and the decomposition must allow for alternative routes to the outcome. A detailed diagram can still encode an incomplete or inconsistent account of what would make the event occur.

Updating requires a judgment about the information supplied by new evidence. Ten articles repeating one announcement do not necessarily provide ten independent observations. A useful research record distinguishes primary evidence, commentary and repetition, and notes when the same underlying fact supports several arguments. This makes it possible to inspect whether a probability changed because the evidence changed, because its interpretation changed or because the system encountered the same information again in another form.

Evidence can also fail to arrive. The absence of an announcement becomes informative only relative to a model of when it would normally be observed if the event were on track. An uneventful day is not automatically evidence against a forecast. Conversely, a missed statutory deadline may materially change it. The appropriate response depends on the question's timing and observation process. Recording the reason for an update helps distinguish a considered revision from an unexplained movement in the reported number.

Explanations should identify decisive assumptions and uncertainties without being treated as evidence of accuracy in themselves. An articulate rationale can accompany a poor probability, while a well-performing statistical baseline may offer little narrative detail. The useful role of the explanation is to expose the judgment to inspection: what evidence was used, which inference was made and what would lead to revision. Its empirical quality remains a separate question, answered through forecasts recorded before outcomes.

A crowd offers several potentially useful differences: access to evidence, interpretation, domain experience and tolerance for uncertainty. Simply increasing the number of forecasts does not guarantee an increase in independent information. Participants may read the same sources or adopt the same initial estimate. Language-model agents may share training, retrieval results and prompts. The aggregation problem is therefore to combine judgments while accounting for their information and error structure, rather than to treat every additional response as an independent vote.

An unweighted average is an important baseline because it is simple and difficult to overfit. More elaborate weighting can use historical performance or other pre-specified information, but estimating many weights from a small set of resolved questions can fit noise. Development and evaluation questions should be separated, and historical weights should use only outcomes available at the time. Otherwise an ostensibly predictive aggregate can incorporate future knowledge through the procedure used to select its contributors.

Averaging probabilities and combining independent evidence are different operations. Two forecasts of 80% and 60% average to 70%; this does not tell us how much evidence the two forecasters share. Extremisation moves an aggregate away from an uninformative or reference value under a specified transformation. It may improve some aggregates, but a fixed push away from 50% is not a general correction for dependence. Its direction and magnitude should be selected on development data and assessed on held-out questions, with the transformation disclosed.

Forecast age adds another dimension. A recent estimate and an estimate issued before a major announcement do not condition on the same evidence. Satopää and colleagues study aggregation when experts update intermittently, using a dynamic hierarchical model for sparse beliefs. Their work makes the temporal structure explicit. For an operational comparison, the related requirement is to specify how old forecasts remain eligible, how missing updates are handled and whether competing systems are compared at common decision times.

Disagreement should also be inspected before it is averaged away. Contributors may be answering different interpretations of an ambiguous question, using different deadlines or attaching probability to different definitions of success. Reconciliation of the target is distinct from encouraging consensus about its likelihood. Once the target is shared, retaining the distribution of judgments can expose sensitivity to assumptions that a single aggregate hides. That distribution is not, by itself, a confidence interval for the event probability.

A meta-prediction is a judgment about other judgments. Participants may be asked both for their own answer and for the proportion of others who will choose it. This supplies information about perceived consensus, which is distinct from confidence in one's own answer. In Prelec, Seung and McCoy's study, the surprisingly popular rule selects an answer that is more common than respondents predict. The paper establishes its rationale under specified assumptions and examines its performance on knowledge-elicitation tasks.

The transfer to future events requires care. In a knowledge question, some respondents may already know the correct answer. In a forecasting question, the outcome may depend on events that have not yet occurred. Predicting a crowd's belief can still be informative, but the success of a knowledge-aggregation rule does not establish calibrated probabilities for unresolved events. A forecasting study would need to evaluate the additional contribution of meta-predictions against simpler aggregates on the same prospective questions.

The term should also be kept separate from software naming. A service that orchestrates research and combines model outputs is not necessarily eliciting beliefs about other forecasters. Calling that service a meta-forecaster does not establish a connection to the experimental method. The relevant research claim concerns the information actually collected and the rule that uses it. When second-order judgments are absent, ordinary forecast orchestration or aggregation is the more precise description of the operation.

Language models can assist with question interpretation, evidence retrieval, synthesis and probability estimation. These functions should be distinguished when attributing performance. A stronger retrieval system can improve a fixed model's inputs; additional sampling can change the aggregate; better validation can reduce missing or invalid outputs. An end-to-end improvement is useful, but it does not identify which component caused it unless the evaluation isolates those changes.

Halawi and colleagues investigate a system combining retrieval, reasoning and aggregation for binary forecasting. Their comparisons with human forecasts concern the questions, information conditions and evaluation choices in that study. The work supports evaluating complete forecasting procedures rather than assuming that an unaided language-model response represents the system's attainable performance. Its results should not be extended into a claim that all such systems match human expertise across domains or deployment conditions.

Historical evaluation faces a particular difficulty: a model may have encountered the outcome during training. Restricting retrieved documents to an earlier date does not remove information already encoded in its parameters. Prospective evaluation reduces this problem by recording predictions while the relevant outcomes are still unknown. The published ForecastBench work is an example of a benchmark designed around future-event forecasting. Historical and prospective results answer different questions and should be labelled and interpreted accordingly.

Repeating a prompt across agents can estimate variability in that procedure, but it should not be equated with sampling independent human experts. Diversity of wording, persona or model family is not direct evidence of complementary predictive errors. Complementarity can be evaluated by comparing aggregates, contributor errors and performance across question groups. A useful ensemble is one whose combination improves the relevant held-out score, rather than one that merely produces a larger number of distinct rationales.

Finally, forecasting supplied questions differs from discovering which events should be forecast. A system can assign good probabilities to a fixed list while failing to identify an important event outside it. Discovery quality involves coverage and the definition of the candidate event space; probability quality concerns judgments on specified outcomes. The distinction connects this essay to Modelling Human Action and Simulating Society. Evidence for one capability should not be substituted for evaluation of the other.

The fundamental record is a forecast issued at a known time against a versioned question. It should preserve the probability, information cutoff, system configuration and any permitted abstention. Outcome resolution is a later record, with its source and adjudication history. Publication time, event time and resolution time can differ. Keeping them distinct makes it possible to reconstruct what a system could have known and to prevent a later correction from silently rewriting an earlier prediction.

Comparisons should use common eligible questions and explicit submission rules. If a system forecasts only easy questions, its average loss is not directly comparable with a system required to forecast all questions. Coverage and missingness should be reported alongside performance, with a policy for late, failed or invalid submissions established before scoring. Restricting analysis to questions answered by every system may be useful, but the excluded set and resulting change in the target population should remain visible.

Updating creates repeated observations within a question. A daily score and a score taken once at a fixed horizon weight evidence differently. Long-running questions can dominate a pooled daily average unless weighting is specified. Uncertainty estimates should respect dependence among updates and, where relevant, among questions tied to the same event. Treating every timestamp as a new independent event overstates the amount of information supporting a performance comparison.

Baselines should answer the same question under comparable information conditions. Depending on the task, useful references include an empirical base rate, a simple crowd aggregate and an available market forecast. A market price requires an observation convention and matching contract terms; it is not automatically interchangeable with a forecast elicited at another time. Results should identify these choices and show whether the conclusion persists across horizons and domains, rather than relying only on one pooled ordering.

The evaluation report should separate the protocol, observed results and interpretation. Hypothetical examples can explain a scoring rule or demonstrate a presentation, but they provide no evidence of comparative performance. Claims about a particular system require recorded forecasts and resolved outcomes from the stated experiment. The literature reviewed here motivates that evaluation standard; it does not supply missing observations for a new system or establish a ranking among products that have not been compared.

A replication package need not expose confidential source material to make the scoring inspectable. It can publish question definitions, submission timestamps, permitted inputs, forecast values, resolution sources and the procedure that computes each reported quantity, subject to the applicable data rights. Any unavailable evidence should be identified as a limit on independent replication. Versioned artefacts also separate an editorial correction from a changed experiment: revising the explanation of a result should not silently replace the forecasts or outcomes from which that result was calculated.

Even a well-conducted evaluation supports a bounded conclusion. A lower mean loss may coexist with worse performance on a particular domain or rare outcome. Small differences may be unstable across question samples. A change in the information environment can alter the usefulness of historical weights and reference classes. These possibilities should be investigated through pre-specified subgroup analyses and uncertainty estimates, while recognising that extensive exploratory slicing creates additional opportunities to select favourable results.

Predictive performance is also distinct from decision value. The consequences of a false alarm and a missed event depend on the decision, and the same probability can justify different actions under different costs. In equities, a forecast about an announcement is not a forecast of its price impact, and a probability advantage does not by itself establish achievable returns. Evaluation of decisions requires the relevant timing, alternatives and costs in addition to the quality of the probability estimates.

Collective judgment is valuable when combining information improves forecasts that can be independently assessed. Its assessment therefore depends on explicit targets, preserved information boundaries and scoring rules selected before outcomes. Calibration explains how probabilities relate to frequencies; resolution concerns whether they distinguish cases; aggregation concerns how information and error are combined. Keeping these concepts separate allows a research programme to identify what improved, where it improved and what remains uncertain, without asking a single headline score to establish all three.

1. Mellers, B. et al. (2014). Psychological Strategies for Winning a Geopolitical Forecasting Tournament. Psychological Science, 25(5), 1106–1115. doi:10.1177/0956797614524255.
2. Hauenstein, C. E., Thomas, R. P., Illingworth, D. A. and Dougherty, M. R. (2025). Rethinking the Role of Teams and Training in Geopolitical Forecasting: The Effect of Uncontrolled Method Variance on Statistical Conclusions. Psychological Science, 36(1), 3–18.
3. Gneiting, T., Balabdaoui, F. and Raftery, A. E. (2007). Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society: Series B, 69(2), 243–268.
4. Gneiting, T. and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378.
5. Satopää, V. A., Jensen, S. T., Mellers, B. A., Tetlock, P. E. and Ungar, L. H. (2014). Probability aggregation in time-series: Dynamic hierarchical modeling of sparse expert beliefs. Annals of Applied Statistics, 8(2), 1256–1280. doi:10.1214/14-AOAS739.
6. Prelec, D., Seung, H. S. and McCoy, J. (2017). A solution to the single-question crowd wisdom problem. Nature, 541, 532–535. doi:10.1038/nature21054.
7. Halawi, D. et al. (2024). Approaching Human-Level Forecasting with Language Models. arXiv:2402.18563.
8. Karger, E. et al. (2025). ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. International Conference on Learning Representations.