Can your silicon sample predict how 18,000 humans will behave?

Whether large language models (LLMs) can predict how human participants behave in surveys and experiments is a consequential open question in social sciences. The evidence so far is mixed, and has methodological issues: Most existing evaluations target experiments that were already published, and plausibly in the models’ training data. Even in cases where the data from these published experiments is outside of the LLMs' training data, reported success rates may still be biased by researcher degrees of freedom. The analytical procedure, evaluation metrics, models, and prompts researchers use impact their findings considerably. A credible evaluation of LLMs' potential for predicting human behavior in social science research requires new data and preregistered methods.

This benchmark study addresses these problems. The target is a megastudy on strengthening trust in climate scientists in the United States, testing 16 novel interventions on 13 outcomes. The human data are held exclusively by the core team, and every prediction must be deposited and time-stamped before any human data are released.

What you get out of it:

• A credible test of your approach. Your method is evaluated on out-of-sample human behavior, against the preregistered human replication reference.
• Co-authorship on the benchmark paper. All participating teams join the author team on the resulting publication.
• A catalogue of the field. Each registered method becomes a structured, comparable entry describing how teams currently approach AI prediction of behavioral experiments (e.g., open- vs. closed-weight models, persona sources, fine-tuning, in-context learning.)
• No single winner. Approaches are scored on a set of metrics from directional agreement to calibration, plus distributional fidelity and stereotyping diagnostics. We expect different approaches to do well on different metrics.

Any AI-based approach is welcome — any models, personas, prompts, fine-tuning, or prior literature — under a few firm ground rules.

• Blinding is absolute. No participating team will have access to any human data from this study before the prediction lock. A signed attestation is part of registration (item I.3 of the form).
• AI-based and automated. Pipelines must be AI-based. Predictions may not be manually made by humans, and teams are not allowed to collect any new human data for this project. Otherwise, any models (open or closed weight), any prompting or persona strategy, any external training data, prior studies, or meta-analytic knowledge may be used — as long as it is declared in the registration form.
• Invited expert teams of up to two. Participation is by invitation, open to teams with a published track record in silicon-sample research. A team is at most two people; an individual may join any number of teams. Requests for larger teams can be made and are evaluated case by case (e.g., when a tool has been developed by more than two people). We encourage invited teams to forward the call to other researchers they consider relevant.
• Multiple submissions are encouraged, up to three entries per tier. A team may enter up to three entries per tier — genuinely different approaches (say, a per-respondent simulation and a direct forecast) or structured variations of one approach. Each is a separate registered entry, and every entry enters all main analyses. If there are good reasons to systematically vary some aspect of an approach, more entries may be granted on request, but only three per tier enter the main analyses. You designate exactly one entry as your primary submission — a robustness analysis reruns the main results on primary entries only, one per team.
• Everything is registered, not everything must be public. The method form, the prediction files, and — strongly encouraged — code and raw model output logs are deposited on Zenodo (a free research archive run by CERN that gives each deposit a permanent DOI and a time-stamp — done through a release on a linked GitHub repository) before the lock date. Teams with proprietary methods may seal parts of their registration under the disclosure policy: locked and verifiable is mandatory, public is not.

We deliberately welcome very different approaches. Submissions fall into one of three tiers, defined by their level of prediction. Tier 1 is the preferred option. Choose the highest tier your approach supports. Tier 2 and Tier 3 are eligible for fewer analyses.

Tier 1 has a minimum pool size: 500 synthetic respondents per intervention and 1,000 in the control condition — the size of the human half every submission is scored against (the benchmark preregistration's precision requirement). Below that floor, effect estimates are noisy for reasons unrelated to the method. Going well beyond it is encouraged — synthetic respondents are cheap, and a larger pool makes your estimates reflect the method rather than a lucky draw. Beyond precision there is no advantage: only point estimates are scored, so a huge pool stabilizes your estimates but cannot buy a better score.

Analysis eligibility by tier

Full metric definitions are in the Evaluation section and the benchmark preregistration.

There are two separate steps, easy to confuse:

• Express interest. Every team fills the short Google interest form to leave contact details and confirm participation. No methodological choices need to be made at this point, it's just to put you on our list of participating teams. Both invited teams and colleagues you forward the call to must complete it.
• Register your predictions. The registration is the time-stamped Zenodo deposit described below, produced from the submission template repository. This registration is your submission to the project, and contains both your predictions and a registration form where you describe your methods.

Before the lock, every team deposits one Zenodo record with two parts: a registration form describing the approach, and the prediction files it produced.

The form. You document your simulation pipeline in a structured form which builds on the GUIDE-LLM reporting checklist^{1Feuerriegel, S., Barrie, C., Crockett, M. J., et al. (2026). A reporting checklist for large language models in behavioural science. Nature Human Behaviour, 1–5. https://doi.org/10.1038/s41562-026-02492-7}, the consensus reporting standard for LLM use in behavioral science. Among other things, you will describe your models, prompts and survey-administration protocol, persona/profile construction, stochasticity and aggregation, post-processing, any learning or retrieval corpora, internal model and prompt selection, and frozen code plus raw output logs. The full form is below, a fillable template (Markdown + machine-readable JSON) ships in the submission template repository.

The predictions. Alongside the form you deposit the prediction file(s) for your tier, a machine-readable metadata.json, and — strongly encouraged — code and raw model output logs. Everything is hashed and time-stamped before the lock; the DOIs and SHA-256 hashes are emailed to the core team by the same deadline. Column schemas, file naming, the metadata.json format, and a self-check validator are in the submission template repository (linked from the materials page).

The deposited, hashed record is final. Once you have emailed the DOI and fingerprints, that snapshot is your locked submission — it cannot be revised, and only the exact file matching the emailed SHA-256 hash is scored. You may keep working in your repository and even publish a newer Zenodo version afterwards, but the core team scores the version whose fingerprint you sent by the deadline. Deposit only when you are ready to lock.

The registration form

The form builds on the GUIDE-LLM reporting checklist and extends it where a reporting checklist alone is not sufficient for, in principle, full replicability of a prediction pipeline. GUIDE-LLM's items on human participants and sensitive-data privacy are omitted, since every submission is fully synthetic. When multiple models serve different pipeline stages, complete the model sections separately per model.

Disclosure policy — proprietary approaches

Some teams — companies especially — cannot or will not publicly disclose their full approach. The benchmark separates two things usually conflated: verifiability (is the registration locked, time-stamped, and auditable?) and publicity (can everyone read it?). Verifiability is non-negotiable; publicity is graded. Each item is deposited as public (in the open Zenodo record), escrowed (in a restricted-access Zenodo record that still carries a DOI, a timestamp, and public SHA-256 content hashes — so the lock is verifiable — but is readable only by the core team and up to two independent auditors under a confidentiality agreement; an embargo with a sunset date, e.g. release upon publication or after 24 months, is encouraged), or withheld (not deposited at all, permitted only for items marked neither ★ nor † in the registration form above). An entry's disclosure class follows from its most restricted item:

A submission is evaluated on every metric its tier supports, on all interventions and outcomes. The scored effects are between-subjects post-treatment contrast, i.e. the intervention-group mean minus the control-group mean.

The exact estimands, metric definitions, and the full scoring pipeline — including how each metric above is computed and reported — are specified in the benchmark preregistration. It is the single authoritative source for how submissions are evaluated.

Participation is by invitation. Once you accept, you build your pipeline with any AI-based approach — any models, personas, prompts, fine-tuning, retrieval, or prior literature — using the intervention texts and the full survey instrument, and constructing your own participant profiles (the benchmark ships no pool), but never any human outcome data. You then register the approach and deposit your predictions before the lock. The roadmap marks which milestones are your team's responsibility (Your task) and which are the core team's (Core team).

Human data collection is already complete (≈18,000 responses); none of these data have been shared, published, or made accessible to any prospective team, and the data stay sealed until after the prediction lock.

Common questions about eligibility, tiers, blinding, and scoring are answered on the FAQ page; file-level questions are covered in the submission template's FAQ.md. If you believe your team should be on the list, or your question is not covered there, head to the contact page.