Simulating Society

A simulation makes assumptions executable. An agent-based model represents individual units, assigns them states and decision rules, and allows them to interact within an environment. Its output follows from these choices and their implementation. The units may be people, institutions or deliberately simplified representatives of a population. The appropriate level of representation depends on the question: an institution can be treated as one actor when internal decision making is outside the study, but that simplification becomes consequential when internal disagreement is the mechanism of interest.

Three purposes should be distinguished. Mechanism exploration asks whether a specified process can generate a pattern. Scenario analysis compares outcomes under alternative assumptions or interventions. Forecasting estimates what will occur in an external system. The first two can be useful without establishing the third. A model in which concentrated exposure generates concentrated attention demonstrates a consequence of its assumptions; it does not establish that those assumptions describe a particular market or that its numerical outputs are calibrated forecasts.

Language-model agents make it possible to condition actions on textual histories and interpret events whose meaning is difficult to encode as a small set of rules. In Generative Agents, Park and colleagues combined memory, reflection and planning in an interactive environment containing 25 agents. Their evaluation concerned the believability of individual and emergent behaviour. This is evidence for an architecture capable of coherent interaction, rather than evidence that generated social outcomes have known frequencies in a corresponding real population.

Behavioural plausibility is therefore one level of assessment. Another is whether observed actions, exposure patterns and collective quantities agree with independent data. A third is whether an intervention has the predicted effect outside the model. These questions require different evidence. The actor-level problems of defining actions, reconstructing information and measuring non-action are developed in Modelling Human Action. Here the additional problem is interaction: even a useful individual predictor may produce an inaccurate population when connected through an inappropriate environment.

The first specification is the boundary of the study. It should identify the population, observation period, external inputs and outcome of interest. For example, the spread of an earnings interpretation among equity analysts is a different target from the spread of the same interpretation among all investors. Neither population is equivalent to the subset that posts publicly. A convenient source of observable records should not silently become a definition of the population to which the conclusions apply.

Population construction determines what an agent represents. A named analyst, a sampled participant and a weighted representative of a large group have different interpretations. Multiplying an agent's contribution by a population weight may be reasonable for some aggregate measurements, but it does not automatically reproduce the interactions of the people that agent represents. Compressing many participants into one node changes the number and arrangement of possible relationships. The consequences of that compression need to be examined for the particular process being studied.

Relationships specify possible channels of influence; exposure specifies what actually passes through them. Following an account does not imply reading every message it posts. A public statement does not imply that every agent receives it immediately. The model should distinguish availability, delivery, attention and subsequent action wherever those distinctions matter to the outcome. Giving all agents a shared, complete narrative can remove the information differences that the simulation was supposed to investigate.

Time and scheduling are equally substantive assumptions. In a synchronous model, agents can act from the same prior state before any responses are applied. In a sequential model, later agents may observe earlier actions. These designs can generate different cascades even with identical decision rules. The simulated clock must also have an interpretation: a round of model calls is not inherently an hour of social activity. Activity rates, response delays and opportunities for non-action require their own specification.

A useful reporting reference is the ODD protocol described by Grimm and colleagues. Its overview, design concepts and details organise the description of an agent-based model, including its entities, processes and scheduling. For a language-model simulation, this can be supplemented with model versions, information supplied to agents and action-validation rules. A structured description makes a model inspectable; it does not, by itself, establish that its assumptions are appropriate or its predictions valid.

Li and Tao's position paper argues that agent–environment dynamics, exposure and scheduling deserve explicit treatment in social simulation. This is a methodological argument about what a model must represent, rather than an empirical ranking of simulation systems. It also presents a useful challenge to agent-centred evaluation: improvements in role-playing may leave collective behaviour unchanged if the environment still determines who observes, responds and acts.

One execution reveals one trajectory. Repeated executions reveal how outcomes vary under the implemented sampling procedure. This distinction matters because a visually diverse collection of trajectories can still explore only a narrow set of assumptions. Repeating the same population and network with different generated responses measures a different source of variation from changing the network, the population sample or the rules governing exposure. Reporting these together as a single uncertainty band conceals what was varied.

It is useful to distinguish stochastic variation within the model from uncertainty about the model. The former includes sampled actions and initial states. The latter includes uncertain parameters, missing participants and competing accounts of how influence works. A further concern is misspecification: every tested configuration may omit a relevant mechanism. More repetitions can reduce Monte Carlo error for a chosen configuration, while leaving these other uncertainties largely untouched. Precision within a model should not be confused with accuracy about the external system.

Suppose an event occurs in a fraction of simulated runs. That fraction estimates a probability under the simulation's assumptions and sampling design. Interpreting it as a real-world event probability requires additional justification, including evidence that the relevant outcomes and their frequencies are reproduced on cases not used to construct the model. The distinction remains necessary even when the simulated event is described in natural language and appears realistic. A coherent account of an event is not a frequency estimate derived from observations of that event class.

Replication also has a unit. Agents interacting inside one run are not independent replications of the whole experiment. Multiple runs based on the same historical incident do not create multiple independent incidents. An analysis should distinguish variation across runs, configurations and cases, and state which level supports its uncertainty estimates. Treating thousands of generated messages as thousands of independent tests can produce narrow intervals that describe the volume of simulation output rather than the strength of the empirical evidence.

Computational limits should be included in this account. A fixed budget creates a choice between more runs, more heterogeneous populations and more detailed agent behaviour. There is no general reason to maximise agent count or dialogue length. The appropriate allocation depends on which uncertainty dominates the research question. A convergence check can establish that an estimated model quantity has stabilised with additional runs; sensitivity analysis is needed to establish whether it survives reasonable changes to the assumptions that generated those runs.

Consider a hypothetical study of how an interpretation of NVIDIA's earnings spreads after the release of its results. The question might be whether an early correction changes the proportion of initially exposed participants who later repeat a particular inaccurate claim. This specifies a population, a treatment and an observable outcome. It is more informative than asking whether the overall discussion becomes positive or negative, which can change for reasons unrelated to the claim or its correction.

Construction would begin with information available at a defined cutoff: the release, prior public records and a specified relationship network. Later commentary would be reserved for evaluation. Participants would receive only the material permitted by the exposure model. Their available actions could include repeating, correcting, questioning or ignoring the claim. These categories would need operational definitions that an evaluator can apply consistently. A model-generated explanation of an action would be recorded separately from the action and its effect on the environment.

The comparison could vary the timing of the correction while preserving the initial population and external news sequence. Comparable random inputs can sometimes reduce noise between experimental conditions, but a shared seed alone does not guarantee comparable executions, especially when one condition changes the number or order of model calls. The design should specify what is held fixed and what is allowed to vary. Otherwise an apparent timing effect may also include an unintended difference in exposure, activity or computational effort.

Measurements would include the number of uniquely exposed participants, the proportion repeating the claim and the distribution of time to correction. The denominator matters: a lower repetition rate among all agents may simply reflect lower exposure. Repeated posts by a single prolific agent should not count as repeated adoption by different people. Sentiment, if measured, would be a separate outcome with a stated target and annotation method. Positive language about the stock does not imply endorsement of every claim circulating about it.

This example describes an experimental design, not a finding about NVIDIA or a measured capability of a particular product. Before execution, the study would specify its primary outcome, comparisons, exclusion rules and analysis. After execution, an append-only record of received information, valid actions and state transitions would support reconstruction. A narrative summary could then explain the recorded dynamics. It should not introduce actions or causal relationships that cannot be located in the underlying trace.

Validation requires an external target and a stated purpose. At the individual level, relevant quantities may include participation rates, action frequencies and conditional responses to exposure. At the relational level, they may include who responds to whom and how far information travels. At the aggregate level, they may include concentration, timing and the distribution of collective outcomes. Agreement at one level does not guarantee agreement at the others. An aggregate total can be reproduced by many different configurations of individual behaviour.

Calibration data and evaluation data should play distinct roles. If the activity rate is chosen to reproduce a historical cascade, agreement with that cascade is evidence of fit. Generalisation requires a separate case or period, selected without reference to whether the model performs well on it. Event families may need to remain together when dividing data, because apparently separate examples can share participants, reporting conventions and background information. The split should reflect the transfer claim the study intends to make.

Baselines help identify whether complexity contributes useful information. A simulation can be compared with an empirical activity model, a simpler diffusion rule or a version with restricted textual interpretation. An ablation removes or changes a particular mechanism while preserving the rest of the comparison as closely as possible. If removing detailed personas leaves the relevant outcome unchanged, those personas have not shown incremental value on that target. This does not establish that they are useless for every other task or population.

Predictive agreement also leaves a problem of identification, discussed in Windrum, Fagiolo and Moneta's analysis of empirical validation. Several mechanisms can generate the same observable pattern. A model may match the timing of a discussion because its response delays are realistic, because its exposure rules compensate for unrealistic delays, or because parameters were adjusted until the aggregate curve matched. Discriminating among these explanations requires measurements that constrain intermediate processes. Without them, a fitted mechanism should be described as one compatible account, rather than the uniquely established cause of the observed pattern.

Textual evaluation introduces additional uncertainty. If a model generates messages and another model classifies them, the reported result depends on the classifier's validity. A documented annotation scheme, independently checked examples and estimates of disagreement make that dependency visible. Human judgment can also be inconsistent, so human review is not an automatic substitute for measurement design. The relevant question is whether the evaluation distinguishes the actions and outcomes specified by the study with sufficient reliability for its conclusions.

Validation should permit the simpler model to succeed. If a threshold rule reproduces the relevant timing and participation patterns as well as a language-model population, the additional linguistic detail has not demonstrated value for those outcomes. It may still be valuable for another question, such as interpreting an unfamiliar message, but that contribution needs its own test. Conversely, better performance by the richer model should be examined for differences in available information and computational budget before being attributed to a more faithful representation of human reasoning.

The observation process itself can be tested through a measurement model. For example, simulated private discussion should not be compared directly with a dataset containing only public posts. One option is to restrict both model outputs and observations to comparable channels; another is to specify how latent activity becomes observable and test the resulting predictions. Either choice is stronger than explaining every mismatch through unobserved behaviour after the fact. Missing data should constrain what can be concluded, rather than make the model immune to contrary observations.

Excess activity is a particularly consequential failure. If every agent must speak on every round, the model excludes silence by construction and creates artificial opportunities for reinforcement. Equally, agents who receive identical summaries may agree because their information has been homogenised. Adding more persona descriptions does not necessarily restore differences in knowledge or exposure. Participation, information access and behavioural heterogeneity should be varied separately so that their effects can be examined.

State inconsistency can change the experiment without producing an obvious error. An agent may refer to a future announcement, adopt information it never received or forget a binding earlier action. A fluent continuation can conceal each problem. Checks should therefore operate on event timing, permitted information and valid state transitions, rather than relying solely on the plausibility of the generated prose. Invalid actions and their handling should remain visible in the analysis; quietly regenerating difficult cases can change the distribution being measured.

Sensitivity analysis should address assumptions that could change the substantive conclusion. These include network concentration, response delays, the share of inactive participants and the reliability of influential sources. A conclusion that reverses under small plausible changes is different from one that persists across them. The purpose is not to search for a favourable configuration. It is to establish which assumptions the result depends on and whether those assumptions are constrained by available evidence.

Reproducibility requires both a description and retained artefacts. Provider updates may prevent exact regeneration of language-model outputs even when prompts and nominal model names are unchanged. Preserving inputs, returned outputs, configuration and scoring code permits the recorded experiment to be inspected without implying that an external service will reproduce every token later. A study should distinguish replaying stored observations, repeating its sampling procedure and obtaining the same substantive conclusion from an independent implementation.

An intervention deliberately changes a component of the model: a message arrives earlier, a participant is removed or an exposure rule is altered. The difference between conditions is a causal effect within that specified computational system when the experimental design isolates the intervention appropriately. It is not automatically the effect of making the same change in society. External interpretation depends on whether the mechanisms that transmit the intervention are represented adequately and remain applicable under the change.

Social interventions also create spillovers. A correction delivered to one participant may affect others through subsequent communication. Consequently, a comparison between treated and untreated agents inside one network may not estimate the effect suggested by those labels. The unit of treatment, exposure to others' treatment and outcome window all need definition. Comparing whole simulated networks may answer a different question from comparing individuals within a network; neither design should be interpreted as the other.

Where external intervention data are unavailable, the defensible output is often a conditional statement: under a specified account of exposure and response, a change produces a particular pattern. Such statements can identify measurements worth collecting or mechanisms worth testing. They become stronger when supported by independent observations that constrain those mechanisms. Their value does not require presenting unvalidated model outputs as measured effects in the real world.

Social simulation is most informative when the research question depends on interaction and when the assumptions governing that interaction can be examined. Potential uses include studying the diffusion of market interpretations, testing communication policies and investigating coordination under incomplete information. In each case, the study should explain what would count as failure. If every possible trajectory can be accommodated after the fact, the model offers an interpretation without a discriminating test.

Data access and ethical scope also constrain the inference. Public records may be sufficient to study some observable communication patterns while providing little basis for claims about private beliefs. A representation of an identifiable person should not turn generated psychological attributes into asserted facts. Population-level findings should state whose behaviour is missing and how that absence may affect the result. These are limitations of the research object, not details that can be repaired by making an agent's biography more elaborate.

The resulting standard is an explicit relationship between a question, a model and independent evidence. Richer agents can improve the representation of interpretation and communication, but their contribution must be assessed on the outcomes that matter to the study. Repeated runs expose variation under assumptions; validation establishes where those assumptions support useful inference. Where the output is a probability forecast, the additional requirements of scoring, calibration and comparison are addressed in Calibration and Collective Judgment.

1. Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P. and Bernstein, M. S. (2023). Generative Agents: Interactive Simulacra of Human Behavior. Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. doi:10.1145/3586183.3606763.
2. Grimm, V. et al. (2020). The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism. Journal of Artificial Societies and Social Simulation, 23(2), 7.
3. Li, Y. and Tao, D. (2026). AI Agents Alone Are Not (Yet) Sufficient for Social Simulation. arXiv:2603.00113, version 2. Position paper.
4. Windrum, P., Fagiolo, G. and Moneta, A. (2007). Empirical Validation of Agent-Based Models: Alternatives and Prospects. Journal of Artificial Societies and Social Simulation, 10(2), 8.