Skip to main content

Leaderboard

Full rankings across all datasets and models.

Survey Parity Score (SPS) measures how closely AI-generated survey responses match real human opinion distributions. 1.0 = perfect match.

48 ranked configs · 6 runs excluded · generated 2026-07-18 18:05 UTC · synthbench v0.1.0 · JSON API

Reproduce this yourself

Every number on this page is recomputed from per-question data at publish; the scoring code and public-tier data are open. Don't trust us — re-run it: 

pip install "synthbench[subpop] @ git+https://github.com/DataViking-Tech/SynthBench.git"
synthbench run --provider <yours> --dataset subpop --samples 30
synthbench validate results/<your-run>.json
synthbench compare results/<your-run>.json <any-published-run>.json

run benchmarks your provider against the same public ground truth, validate re-checks the artifact against the integrity rules, and compare runs a paired bootstrap significance test against any published run. Machine-readable data: JSON API · API contract

Default view hides configs with <3 runs on <2 datasets. Toggle “All variants” to see every run.

Leaderboard

Select a column header to sort. Activate a row (Enter) to open its configuration, or use the chevron button to expand details inline.

Legend

✓ verified = public and private-holdout SPS agree within the 0.05 divergence threshold · ⚠ flagged = public/holdout SPS divergence exceeds 0.05, score held for review (a statistical anomaly detector, not proof of cheating) — badge methodology.

~ = estimated cost (run published before token capture; samples × questions × average per-call tokens) · — = not measured for this row · Range % = (SPS − raw-LLM baseline) / (Human Ceiling − raw-LLM baseline), shown only when a raw-LLM baseline and a human-ceiling estimate exist for the row's dataset; "baseline" marks the raw-LLM row itself, and "≤0%" marks rows scoring at or below their raw-LLM baseline.

$/100Q = US dollars per 100 questions at the pricing snapshot · $/resp = average US dollars per API call (ensemble rows have no single underlying call) · tok/resp = average input+output tokens per call · 95% CI = bootstrap confidence interval on SPS.

SPS = Survey Parity Score, the equal-weighted composite of p_dist = 1 − mean(JSD), p_rank = (1 + mean(τ)) / 2, and p_refuse = 1 − mean(|R_model − R_human|) — see the methodology.

Sub-Metric Radar

Top 3 models compared on SPS sub-metrics: distribution accuracy (p_dist), rank correlation (p_rank), and refusal match (p_refuse).

Sub-Metric Radar — top 3 models across SPS components

Radar plot comparing the top 3 models on distribution accuracy (p_dist), rank correlation (p_rank), and refusal match (p_refuse). Larger polygon = better.

Demographic Parity Heatmap

Models × demographic groups, colored by p_dist (distribution similarity — higher = closer match to the conditioned subpopulation). Use the selector to drill into a specific attribute.

Each cell compares a model's synthetic responses against real human respondents in that demographic subgroup (e.g. SubPOP / Pew American Trends Panel ground truth). Higher = closer to how that real subgroup actually answered — conditioning is scored against what real subgroup members said, not against assumptions, and where a model diverges from the real subgroup its score drops.

Coverage flag derived from n_questions: high (≥100) medium (50–99) low (<50)

Cells with n < 30 are suggestive only — too few questions to carry a color-graded conclusion. And with this many cells, a few extremes are expected by chance alone: read patterns across an attribute, not single cells.

SPS by Model

Survey Parity Score per model with 95% confidence intervals. Higher is better.

SPS by Model with 95% confidence intervals

Dot plot of Survey Parity Score per model; horizontal whiskers are 95% CIs. Higher is better.

Per-Metric Breakdown

SPS and component metrics side-by-side per model. All metrics: higher is better.

Per-Metric Breakdown — SPS and component metrics per model

Grouped dot plot of SPS, p_dist, p_rank, and p_refuse for every model. Legend below identifies each metric.

SPS: Survey Parity Score (higher is better) p_dist: Distribution similarity (higher is better) p_rank: Rank preservation (higher is better) p_refuse: Non-refusal rate (higher is better)

Confidence Intervals

95% confidence interval for each model's SPS. Center dot = point estimate, whiskers = CI bounds.

SPS Confidence Intervals per Model

Each row shows a model's point-estimate SPS (dot) and 95% confidence interval (whiskers).