Skip to main content
Method · Index v1

How a score is made

Each system is scored on three checks against real survey answers. The three are averaged into the Survey Parity Score (SPS), which is then placed between two fixed reference points to give the SynthBench Index. Raw SPS is still published for every result.

One survey question, scored

Illustrative numbers. Move the slider to change how the synthetic panel answers.

“How much do you trust the national news media?”
RandomMatches peopleOver-confident
Index for this question

Why an Index instead of raw SPS

The same systems on both scales, OpinionsQA.

The formula

Versioned. Any change to the weights or reference points changes the Index version.

Per question

Shares match = 1 − Jensen-Shannon divergence(system, people)
Order match = (1 + Kendall τb) ÷ 2
Refusal match = 1 − |refusal rate gap|

Per dataset

SPS = mean of the three, averaged over questions
Indexd = 100 × (SPS − SPSrandom) ÷ (ceiling − SPSrandom)

Headline

SynthBench Index = mean of Indexd over OpinionsQA, SubPOP, GSS and GlobalOpinionQA
95% intervals come from bootstrap resampling of questions

The sub-metrics

Each runs from 0 to 1. SPS is the mean of the first three, with equal weights. P_cond and P_sub are published only for runs that name a demographic group in the prompt, and are not part of SPS or the Index. The Groups page shows the per-group comparisons they summarize.

Metric What it checks Per run Used in
Shares match
P_dist
 How closely the share of answers on each option matches the real survey. 1 − mean JSD(system, people) over questions SPS and Index
Order match
P_rank
 Whether the options come out in the same order as for people, even when the shares differ. (1 + mean Kendall τb) ÷ 2 SPS and Index
Refusal match
P_refuse
 Whether the system declines to answer about as often as people do. 1 − mean |refusal rate gap| SPS and Index
Conditioning gain
P_cond
 When the prompt names a demographic group, how much closer the answers get to that group's real answers. mean over groups of max(0, match with group named − match without) Conditioning runs only
Subgroup evenness
P_sub
 Whether the shares match is similar across groups or much worse for some. 1 − coefficient of variation of per-group P_dist Conditioning runs only

Where 0 and 100 sit

0 is the SPS of uniformly random answers. 100 is the ceiling: how well one random half of the real panel matches the other half (split-half bootstrap). The majority-answer baseline is shown for reference.

How the ceiling is measured

The ceiling is the split-half reliability of the real survey: how closely one random half of the panel matches the other. Humans disagree with each other, so no system can match the survey better than the survey matches itself.

Method. For each question, take the observed answer counts, draw two independent half-samples from the multinomial they imply, and compute the Jensen-Shannon divergence between the halves. Repeat 1,000 times (seed 42). Ceiling = 1 − mean JSD.

Per dataset. OpinionsQA is computed within each Pew wave, never across waves, then averaged with weights by questions per wave. SubPOP is computed per subpopulation and GlobalOpinionQA per country, weighted by actual (country, question) coverage so US data is not over-weighted. GSS uses its per-option counts. NTIA publishes shares and standard errors, so its n is the effective sample size p(1 − p) ÷ SE².

Sample size. Quality flags follow Cochran (1977): high when n ≥ 400, medium for 200 to 399, low below 200. SubPOP and GlobalOpinionQA are flagged medium because their sample sizes are inferred from published shares rather than shipped as raw counts.

Survey weights. The ceiling uses raw counts, not survey weights. Weighting could shift it by 1 to 3% on demographically skewed groups.

Sources. Santurkar et al. 2023; Durmus et al. 2023; Suh et al. 2025; split-half reliability (Spearman 1910, Brown 1910), the bootstrap (Efron 1979), Jensen-Shannon divergence (Lin 1991).

Opinion drift between waves

Pew repeats some questions across waves. On the 5 repeated question stems in the published OpinionsQA set (5 cross-wave pairs), the mean JSD between waves is 0.0011 (95% interval 0.0002 to 0.0024; by year gap: 1 year 0.0033 (n = 1), 2 years 0.0002 (n = 3), 4 years 0.0017 (n = 1)). The closest any system gets to people on OpinionsQA is a mean JSD of 0.126, about 112 times larger, so a model's training cut-off is not what limits its score here. This is a measure of the population, not of any model, and with this few pairs it is a rough number. It is kept separate from the ceiling, which never mixes waves.

Preliminary: forced answer format

By default a system answers in character and the answer is parsed from its text. The structured variant forces a JSON answer schema instead, which cuts parse failures. Across 6 matched pairs that differ only in answer format (2 systems × 3 datasets, 100 questions with 30 samples each), the structured runs scored −9.5 to +4.9 Index points against the natural ones on the same dataset: 3 higher, 3 lower. No consistent effect so far. The structured runs appear on the leaderboard as “structured template” variants.

Full methodology

The claim and its limits, who runs the benchmark, the datasets, the holdout, the run protocol, cost and related work.

Why measure against human data

Critiques of cross-provider homogenization need a ground truth to measure against. Real human response distributions provide one.

Romasanta, Thomas, and Levina (HBR, March 2026) ran 15,000 strategic-advice queries across seven frontier LLMs and found that responses converged on the same trend-aligned recommendations regardless of scenario context. They called this trendslop: a shared attractor across providers, not a quirk of any single model. The critique is sound, but it has no measurement behind it. Cross-model consensus is only evidence of failure if you can point to the distribution the models should have matched, and open-ended strategic questions have no such ground truth.

SynthBench supplies that measurement. Every question we evaluate has a human response distribution from a real survey: the Pew American Trends Panel, SubPOP subpopulations, GlobalOpinionQA cross-country data or the General Social Survey. We score how far each provider's output is from that distribution, overall and for each demographic slice.

Two separate diagnostics

Ground-truth fidelity

Model vs. human distribution

Mean Jensen-Shannon divergence against real survey responses, rolled up into the Survey Parity Score (SPS). This is SynthBench's primary metric, and the rest of this page describes how it is measured.

Cross-provider concordance

Model vs. model distribution

Pairwise JSD between raw-LLM providers on the same items. Low cross-provider JSD with high human-alignment means real population variance is being tracked; low cross-provider JSD with low human-alignment is the trendslop signature. See how much models agree with each other →

What SPS measures and what it does not

The Survey Parity Score measures distributional fidelity: how closely a synthetic respondent system's aggregate response distributions match the human response distributions recorded in these specific surveys: the Pew American Trends Panel (OpinionsQA), SubPOP, GlobalOpinionQA, and the General Social Survey. That is the entire claim. SPS does not measure the validity of synthetic respondents as substitutes for human research participants, the accuracy of any individual synthetic response, or the ability to predict real-world behavior, purchasing decisions, or future opinion. A high SPS means the system reproduces how surveyed populations answered these questionnaires. It does not mean the system understands people, or that its outputs generalize beyond the measured surveys' populations, topics, and time windows.

If you cite SynthBench results, please include three disclosures with the number:

  1. Fidelity, not validity. Describe SPS as distributional fidelity to recorded survey responses, not as accuracy, correctness, or human-equivalence.
  2. Dataset scope. Name the datasets the score was measured on (Pew ATP / SubPOP / GlobalOpinionQA / GSS). A score on these surveys is not a score on your survey.
  3. Operator disclosure. Note that the benchmark operator also ships a scored product, and link the operator disclosure so readers can weigh the conflict of interest themselves.

Who runs this

SynthBench is built and operated by DataViking, the company that also makes Althing, a synthetic-survey product. Althing-branded entries on the leaderboard are the operator's own product being scored on the operator's own benchmark. That is a conflict of interest, so the ranking should be checkable without relying on our word. The benchmark is designed so that the operator gets no mechanical advantage:

Every score is recomputed

The leaderboard never republishes a submitted aggregate. At publish time, open-source code (synthbench/publish.py) recomputes every row's SPS from its per-question distributions, including our own entries. Hand-editing a score cannot change a rank.

Open code, public data

The scoring code, the publish pipeline, the per-question artifacts, and the human reference distributions (minus the private holdout) are all public. Anyone can rebuild the leaderboard from the committed run files and check it against what we publish.

Third-party submissions use the same pipeline

External submissions go through the same validation, recomputation, run-validity filtering, and dedup logic as DataViking's own runs. There is no separate path. The pipeline is in the open repository, so any divergence would be visible.

Private holdout applies to us too

The verified/flagged badge compares each row's public-subset score against a 20% private holdout. Althing-branded rows are scored against the same hidden answer key as everyone else. Some of them are currently flagged, and we publish those flags.

Unflattering results stay published

Excluded runs are listed with reasons in the published data, ensemble caveats name our own defective constituent runs, and baseline floors are tracked in CI. These mechanisms make no exceptions for the operator's product.

If you find a place where these mechanisms fall short, please open an issue. The benchmark is only useful if it is neutral.

Datasets

9 registered human survey datasets. 4 make up the SynthBench Index, 1 more is scored but kept out of it, and 4 have an adapter but no results yet. Sources, coverage and demographics for each are on the Datasets page.

Redistribution tiers

Each upstream dataset has its own license terms. SynthBench applies them at publish time using four tiers:

  • full: per-question distributions are published openly (public-domain datasets).
  • gated: per-question distributions are served from a Cloudflare Worker that validates a Supabase JWT, so research-use sign-in is required. No distribution is served anonymously.
  • aggregates_only: aggregate scores are public, and no per-question artifact ships at all. Gating is not a substitute for missing redistribution permission.
  • citation_only: question metadata only. Every downstream metric is suppressed.

Aggregate metrics (SPS, JSD, ranks, cross-model parity) are derived quantities and remain public regardless of tier.

Private holdout split

Every holdout-enabled dataset is deterministically split into a public subset and a private subset. The private fraction depends on dataset size, chosen to preserve statistical power: 20% for large datasets (OpinionsQA, WVS, GSS), 30% for medium ones (SubPOP, GlobalOpinionQA), and 40% for small ones (Michigan, NTIA, Eurobarometer).

Per-question human distributions for the private subset are removed from every public SynthBench artifact (the site, the run-detail JSON, and the question-explorer pages), so a submitter can't recover the private answer key from our published files.

Submissions must include results for every question, public and private. We (the SynthBench maintainers) score the private subset locally against the withheld answer key and publish the resulting sps_private alongside sps_public. Rows whose public/private SPS delta is within the review threshold get a ✓ verified badge. Rows outside the tolerance are ⚠ flagged for review.

The partition is derived from an unsalted SHA-256 hash of dataset_name + ":" + question_key. That makes it stable across runs, machines, and Python versions without needing a seed file. It also means the list of private questions is not secret. The hash is public code, run artifacts carry explicit is_holdout flags, and anyone can compute any question's membership. This is deliberate and follows the Kaggle model: what is withheld is the private subset's answer key (the per-question human distributions), not the list of held-out questions.

Limits. The source surveys behind these datasets are publicly downloadable upstream, and our open-source adapters will rebuild the full per-question distributions, private subset included, for anyone who runs them. A determined actor can therefore reconstruct the private targets. The holdout is a fabrication detector and a deterrent against casual overfitting. It does not provide cryptographic secrecy. It raises the effort from "copy our published JSON" to "reprocess the upstream source". A holdout that models cannot memorize would need newly fielded human data on a rotating schedule, which is a separate data-acquisition program.

Why have a holdout. There are two separate reasons. First, future LLMs may train on SynthBench itself, and holding out 20–40% per dataset slows that loop. Second, and more important since per-question data was made public: once per-question human distributions are visible, a bad actor could fabricate a submission JSON that matches them exactly. The private subset is a cheap way to catch this. A fabricator has no signal on the hidden keys, so their public and private SPS diverge sharply.

Reading the verified / flagged badges

Every leaderboard row on a holdout-enabled dataset carries one of two badges, computed at publish time by comparing the row's SPS on the public subset against its SPS on the private subset (|sps_public − sps_private|, the published sps_public_private_delta):

  • ✓ verified: the public and private scores are within the review threshold. This is what an honest run looks like.
  • ⚠ flagged: the public/private divergence exceeded the review threshold, so the row is held for review.

What a flag means. A flag comes from a statistical anomaly detector and is not proof of cheating. A large divergence is what fabrication looks like (a fabricator can match the public answer key but has no signal on the hidden one). It can also come from overfitting to the public subset or, on datasets with a small private subset, from ordinary sampling noise concentrated in a few questions. Treat a flagged score with caution, and rely on the row's sps_private when deciding how much to trust it. A flag is not an accusation against the submitter.

The threshold is 0.05 on runs with hundreds of questions, where honest deltas are under 0.02. A run with fewer private questions gets a wider threshold, 0.5 divided by the square root of the smaller subset's question count. A 100-question run has 20 to 30 private questions, so its threshold is about 0.1. Fabricated submissions show deltas of 0.2 or more and are flagged either way. Both scores, the delta, and the threshold are published on each row, so you can apply a stricter or looser standard yourself.

How a benchmark run works

Every provider goes through the same evaluation pipeline. These are the steps that run when you call synthbench run.

  1. Select dataset and question set

    Choose a dataset and how many of its questions to run. Pinned question sets (smoke: 28 and core: 200, both from OpinionsQA) keep repeat runs comparable.

  2. Generate samples for each question

    For each question-persona pair, generate N independent samples from the model. Default: 30 samples. Publication-grade: 100 samples.

  3. Parse responses into distributions

    Map each model response to a survey option. Compute P(option_i) = count(option_i) / N. Logprob-capable providers return distributions directly.

  4. Compare against human ground truth

    For each question-demographic pair, compute JSD between the model distribution and the human distribution from real survey data.

  5. Compute per-question metrics

    Calculate the sub-metrics computable for the run configuration (P_dist, P_rank, and P_refuse for standard leaderboard runs; P_cond and P_sub when demographic conditioning is evaluated) at the question level.

  6. Aggregate with confidence intervals

    Average across questions. Bootstrap resampling produces 95% confidence intervals for each metric and the composite SPS.

Sampling

  • Minimum 30 samples per question-persona pair
  • 100 samples recommended for publication
  • Wilson score intervals quantify estimation uncertainty
  • Parse failures are logged and excluded from counts

Replication

  • Multiple independent runs recommended
  • Convergence data checks stability
  • Repeat runs of three top systems on the same questions moved the SynthBench Index by less than 1 point, and one dataset's Index by at most 2.3
  • The 95% intervals are wider because they also cover which questions were drawn

Fairness

  • Provider-default temperature unless a run sets one; runs at other temperatures are listed as separate variants
  • Each provider tested via its native interface
  • Same question sets and evaluation pipeline for all providers

Run validity filtering

Silent API failures (budget exhaustion, missed model aliases, provider fallbacks) can return responses that parse cleanly but carry no real signal. The result is a run whose per-question distributions are all perfectly uniform. Parse-failure tracking does not catch this case, because the responses did parse. We filter these runs out at publish time so they never reach the leaderboard.

A run is excluded when all three of the following hold:

  • Uniform fraction. At least 80% of per-question model_distribution entries are within 0.01 of perfectly uniform (e.g. {0.25, 0.25, 0.25, 0.25} for a 4-option question).
  • Negligible refusals. Mean per-question model_refusal_rate ≤ 0.05. A model that actually refuses most questions is showing a legitimate safety pattern, not a failed API response.
  • Enough questions. At least 10 questions in the run. Shorter runs don't give enough signal to tell real flat distributions apart from ones caused by API failures.

Excluded runs are listed in leaderboard.json#excluded_runs for transparency, and operators can run synthbench scan-invalid locally against any results directory. The rule is deliberately strict: we would rather miss a borderline case than wrongly exclude a legitimate run whose distributions happen to be flat.

Anti-gaming defences

Our threat model and defences draw on recent adversarial analyses of AI benchmarks, in particular Wang et al. (UC Berkeley, 2026) [1]. They pushed eight widely used agent benchmarks to near-perfect scores without solving any of the tasks.

SynthBench is a submission-artifact benchmark: participants submit a JSON file and we score it. Attacks that assume an evaluating agent running alongside the submission (sandbox escapes, eval() on agent input, binary-trojan wrappers, LLM-judge prompt injection) do not apply. The general pattern behind Wang et al.'s attacks, targeting the evaluator instead of the task, does apply. We have three lines of defence against it.

  1. Private holdout

    A share of every holdout-enabled dataset (20% to 40%, by dataset size) has its human answers withheld from everything we publish. A submitter who copies the published distributions has no signal for the private questions, so their public and private SPS diverge and the row is flagged for review. The split, why it is deterministic, its limits and the flag threshold are described under Private holdout split.

  2. Tiered statistical validation

    Every submission goes through three tiers of checks. Tier 1 verifies schema, bounds, and distribution validity. Tier 2 recomputes JSD, Kendall's τ, and composite parity from the per-question distributions and rejects any submission whose reported aggregates do not match its per-question data. Tier 3 runs statistical anomaly detectors that compare the submission against the distribution of legitimate runs. They flag implausibly perfect per-question JSD, distributions suspiciously close to the public human data, and deviation from peers in the same model family.

  3. Adversarial regression suite

    Our test harness keeps a set of fabricated submissions that must fail validation: pure answer-key copies, public-copy-with-fabricated-private attacks, constant-offset noise attacks, aggregate inflation, and claimed-model swaps. New detectors are tested against this suite before merge, and any change to existing detector thresholds must keep the suite passing. The suite is versioned with the validator and runs as a CI gate.

Null-agent baselines

Following Wang et al.'s recommendation, we also track two "null agent" baselines over time: a uniform-random baseline and a majority-class baseline. A model has to score above these floors for its result to be non-trivial. On a stable dataset, an upward drift in baseline SPS would indicate a bug in the scoring function. CI monitors for it.

Their current scores per dataset are in the reference points table above; random answers are the Index's 0. CI enforces random-baseline SPS < 0.80 and majority-baseline SPS < 0.85. The Berkeley paper's target for the random baseline is 0.70. The remaining gap comes from SynthBench's p_refuse term, which rewards agreement with human DK/Refused rates. The git log of docs/baseline-floors-log.md is the canonical drift record.

We err on the conservative side. For a scientific leaderboard, it is better to hold a legitimate superhuman submission for human review than to accept a fabricated one. Submitters whose runs are flagged can appeal. At validation time, the null hypothesis is that suspicious-looking data was fabricated.

Reference [1]

Wang, Y., Mang, K., Cheung, T., Sen, S., & Song, D. (2026). How We Broke Top AI Agent Benchmarks. UC Berkeley RDI. rdi.berkeley.edu/blog/trustworthy-benchmarks-cont (opens in a new tab)

Cost computation

On SynthBench, cost is derived. Providers return measured token counts at runtime, and the publish pipeline multiplies them by a dated pricing snapshot to produce the $/100Q numbers on the leaderboard.

What we measure

Every benchmark run records per-call token usage from the provider API: input tokens, output tokens, and (for Anthropic) cache-creation and cache-read tokens. Token counts are measured values returned by the provider, not estimates. The runner sums per-question usage into a run-level total, stored alongside the scores in the raw run JSON.

Runs on providers that do not return usage metadata (e.g. Ollama local inference) record no token counts, and downstream cost is reported as null rather than guessed.

What we derive

At publish time, we multiply recorded tokens by per-model pricing from althing to derive four fields per leaderboard row: cost_usd (total spend for the run), cost_per_100q (spend normalized to 100 questions), cost_per_sps_point (spend per SPS point achieved), and is_cost_estimated (true if any pricing dimension was inferred rather than measured).

Derivation happens in synthbench/publish.py. Tokens stay immutable in raw run JSON; cost is recomputed every time we republish, so price-list updates flow through without rerunning any benchmark.

Pricing snapshot

The exact pricing table used to derive costs is serialized into leaderboard.json under a top-level pricing_snapshot block. It includes a snapshot_date and per-provider rates (input, output, cache-creation, and cache-read cost per million tokens). Readers can reproduce every displayed cost from the snapshot and the raw token counts.

Rates track the public price list documented on provider pricing pages (e.g. https://www.anthropic.com/pricing). The snapshot is updated when althing's cost.py constants are bumped.

Self-hosted and unknown models

For self-hosted models (Ollama, local inference) and any provider whose pricing is not tracked in althing, cost_usd is reported as null. We do not impute a cost. Hardware, electricity, and amortization vary too much to produce a number that compares fairly with API-priced rows.

Null cost rows sort last in the leaderboard's $/100Q column and are left out of the cost chart on the Analysis page.

Ensemble cost

An ensemble run's cost is the sum of its constituent runs' costs. Constituents with measured token usage contribute measured cost; constituents without it contribute the pre-tracking estimate, which marks the whole ensemble is_cost_estimated. If any constituent cannot be priced at all (self-hosted or unknown), the ensemble's cost_usd is null rather than a partial total.

This matches how ensembles aggregate elsewhere on the site. A composite number is only emitted when every input to it is known, either measured or clearly marked as estimated.

Pre-tracking runs (estimated cost)

Runs produced before per-call token capture landed in the runner have no usage metadata. For those rows we publish an estimate rather than a dash. The call count is reconstructed as samples_per_question × questions evaluated. That count is multiplied by per-call token constants (124 input / 4 output tokens per call) anchored to the pool's measured run, then priced from the same snapshot as measured rows. Every such value carries is_cost_estimated: true and is shown with a leading ~ on the leaderboard.

Estimates are always marked as estimates. tokens_per_response stays null on estimated rows, the ~ prefix marks every derived dollar figure, and re-running on the current runner replaces the estimate with measured usage. Baseline rows (random/majority) make zero model calls, so their $0.00 is exact, not estimated.

Citation

If you use SynthBench in your research, please cite:

@software{synthbench2026,
  title   = {SynthBench: Open Benchmark for Synthetic Survey Respondents},
  author  = {DataViking-Tech},
  year    = {2026},
  url     = {https://github.com/DataViking-Tech/synthbench},
  version = {0.1.0}
}