Research Findings
Six experiments that reveal what drives synthetic survey quality — from the levers that matter most to the biases hiding in default model behavior.
Scope of claim: every score below measures distributional fidelity to specific surveys (Pew ATP, SubPOP, GlobalOpinionQA, GSS) — not validity, individual-response accuracy, or real-world behavior prediction. What SPS does and doesn't measure →
What Matters Most
Effect size of each optimization lever, sorted by maximum impact. Ensemble blending delivers +2.7–4.8 SPS points with no additional API calls beyond the three constituent runs — though producing all three costs roughly 3× a single-model run.
Horizontal bar chart of effect sizes for each optimization lever (ensemble, temperature, conditioning, and more), sorted by maximum impact.
The Biggest Lever: Ensemble Blending
For each dataset, ensemble blending of 3 models outperforms the best single leaderboard entry at matched question count by +2.7 to +4.8 SPS points. The blend itself is arithmetic — no new model calls beyond the three constituent runs — but producing all three costs roughly 3× a single-model run.
Bar chart comparing the SPS of the best single model against a 3-model ensemble for each dataset. Ensemble gains range from +2.7 to +4.8 SPS points.
Comparison set: best_single = highest recomputed-SPS non-ensemble, non-baseline leaderboard row (raw or product framework) evaluated on at least as many questions as the ensemble, after per-(model, framework, dataset) dedup; random_baseline = recomputed SPS of the random-baseline run for the same dataset (n in random_baseline_n).
How correlated are the constituents' errors?
Pairwise Pearson r between the constituent runs' per-question error (JSD vs the human distribution) over each ensemble's common question set. The constituents' errors are moderately positively correlated (r 0.22–0.49) — not uncorrelated. The ensemble gain comes from partial, not full, independence of errors.
globalopinionqa
n = 100 questions · mean pairwise r = 0.41
| Constituent | Haiku 4.5 | Gemini Flash Lite | GPT-4o-mini |
|---|---|---|---|
| Haiku 4.5 | — | 0.419 | 0.308 |
| Gemini Flash Lite | 0.419 | — | 0.493 |
| GPT-4o-mini | 0.308 | 0.493 | — |
opinionsqa
n = 684 questions · mean pairwise r = 0.30
| Constituent | Haiku 4.5 | Gemini Flash Lite | GPT-4o-mini |
|---|---|---|---|
| Haiku 4.5 | — | 0.224 | 0.355 |
| Gemini Flash Lite | 0.224 | — | 0.308 |
| GPT-4o-mini | 0.355 | 0.308 | — |
subpop
n = 200 questions · mean pairwise r = 0.32
| Constituent | Haiku 4.5 | Gemini Flash Lite | GPT-4o-mini |
|---|---|---|---|
| Haiku 4.5 | — | 0.282 | 0.260 |
| Gemini Flash Lite | 0.282 | — | 0.425 |
| GPT-4o-mini | 0.260 | 0.425 | — |
Comparison set: For each dataset's published ensemble: pairwise Pearson r between the constituent runs' committed per-question JSD vectors (each run's error magnitude vs the canonical human distribution), over the ensemble's common question set, read from the exact files named in ensemble_sources. Measures whether constituents err on the same questions. Signed per-option residual correlations need the stripped human_distribution fields (#308) and are carried as the ensemble_signed_error_correlation asserted constant.
Cost vs Performance Pareto
Each point is one configuration. Pareto-optimal configurations (no other run is cheaper AND higher-SPS) are highlighted; dominated runs are muted. Lower-left of the frontier is the cost/quality trade you cannot beat.
Scatter plot of LLM cost in USD against Survey Parity Score for each non-baseline configuration. Pareto-optimal points are highlighted; dominated points are muted.
Show data table
| Configuration | Cost (USD) | SPS | Pareto Status |
|---|---|---|---|
| Gemini 2.5 Flash (raw) | $1.61 | 0.829 | Pareto-optimal |
| Llama 3.3 70B (raw) | $0.606 | 0.819 | Pareto-optimal |
| Claude Haiku 4.5 (raw) | $4.92 | 0.815 | Dominated |
| GPT-4o (raw) | $3.50 | 0.813 | Dominated |
| GPT-4o-mini (raw) | $0.718 | 0.813 | Dominated |
| Llama 3.3 70B (raw) | $0.106 | 0.796 | Pareto-optimal |
| Gemini 2.5 Flash (raw) | $0.080 | 0.789 | Pareto-optimal |
| Gemini 2.5 Flash (raw) | $0.071 | 0.783 | Pareto-optimal |
| Claude Sonnet 4 (raw) | $14.77 | 0.782 | Dominated |
| Gemini Flash Lite (raw) | $0.016 | 0.776 | Pareto-optimal |
| Gemini 2.5 Flash (raw) | $0.071 | 0.770 | Dominated |
| GPT-4o-mini (raw) | $0.032 | 0.770 | Dominated |
| Claude Haiku 4.5 (raw) | $0.864 | 0.768 | Dominated |
| Llama 3.3 70B (raw) | $0.027 | 0.762 | Dominated |
| Gemini Flash Lite (raw) | $0.017 | 0.761 | Dominated |
| GPT-4o-mini (raw) | $0.032 | 0.749 | Dominated |
| Claude Sonnet 4.6 (raw) | $0.654 | 0.738 | Dominated |
| Gemini Flash Lite (raw) | $0.042 | 0.738 | Dominated |
| Claude Haiku 4.5 (raw) | $0.216 | 0.726 | Dominated |
Cross-Provider JSD Matrix
Pairwise Jensen-Shannon divergence between each raw-LLM pair, averaged
across shared questions per dataset. Low values (cooler) mean providers
agree with each other — a red flag for HBR's
trendslop hypothesis (cross-model consensus reached on shared errors). High values (warmer) mean
providers genuinely diverge. Ground-truth concordance (model vs. human) stays
a separate axis — see mean_human_jsd below the chart.
Heatmap of pairwise Jensen-Shannon divergence between raw-LLM providers, per dataset. Lower off-diagonal values indicate cross-model agreement; higher values indicate divergence.
Show data table
How the Ensemble Is Built
The biggest lever in the benchmark is also the simplest. Six things worth knowing about how blending turns three good models into one better one.
What we blend
Per-question response distributions from three models — Claude Haiku 4.5, Gemini Flash Lite, and GPT-4o-mini — run through the Althing conditioning framework on the same inputs.
How the blend works
Equal-weight arithmetic average of the three distributions per question. For each option on each question, we take the mean of the three models' probability mass. No learned weights, no router, no post-hoc calibration.
Why it works
The three models' errors are only partially independent — pairwise correlation of per-question errors runs r ≈ 0.2–0.5 (see the error-correlation matrix above), moderate but well short of full correlation. Blending cancels the model-specific share of the error while preserving signal the models agree on — 72–81% of questions improve under blending across datasets. Because the errors are partially shared, the gain is a few SPS points, not a wholesale correction.
What we tested against it
Equal-weight, score-proportional, and inverse-JSD weighting — scores are indistinguishable to three decimals. The blending weights don't matter; the ensemble itself does.
What we didn't beat
Per-question oracle selection (the best single model picked retroactively) adds negligible headroom over equal-weight blending. There is little left to extract from this set of models without adding new ones.
What it costs
The blend itself is pure offline arithmetic — no new API calls beyond the three constituent runs. But it is not free: you have to run all three base models, so producing a 3-model blend costs roughly 3× a single-model run.
Temperature: It Depends on the Model
SPS across temperature settings for each model. Error bars show ±1 std across replicates. Gemini Flash Lite gains +4.5 SPS points from 0.3 to 2.0 (well outside its ±0.003 std), while Claude Haiku 4.5 moves under its own noise band.
Line chart of SPS across temperature settings for each model. Error bars show ±1 standard deviation across replicates.
Demographic Conditioning Reveals Model Bias
How much conditioning on a demographic group shifts model responses (p_cond). Republican conditioning is 2.3x stronger than Democrat — the model already leans progressive by default.
Chart showing how conditioning on different demographic groups shifts model responses (p_cond). Republican conditioning produces a larger shift than Democrat — evidence of a default progressive lean.
Nonresponse Fidelity
How closely each model's explicit-nonresponse mass ("don't know"-style option selections plus refusals) tracks the human survey's. Computed from committed per-question rows on full-tier datasets.
| Provider | Framework | Template | Dataset | Model nonresponse | Human nonresponse | Mean abs gap | n |
|---|---|---|---|---|---|---|---|
| Gemini 2.5 Flash | raw | default | gss | 10.6% | 5.6% | 12.6% | 75 |
| Althing (Haiku 4.5) | product | structured | gss | 6.4% | 5.6% | 9.5% | 75 |
| Althing (Haiku 4.5) | product | default | gss | 6.7% | 5.6% | 9.4% | 75 |
Safety-aligned models over-select explicit nonresponse options on sensitive items
Gemini 2.5 Flash on gss: mean model nonresponse mass 10.6% vs human 5.6% across 75 questions.
| Item | Model nonresponse | Human nonresponse |
|---|---|---|
| GSS_FEPRESCH | 96.7% | 2.2% |
| GSS_NATRACE | 83.3% | 8.8% |
| GSS_POSTLIFE | 100.0% | 27.0% |
| GSS_NATHEAL | 70.0% | 2.4% |
| GSS_NATFARE | 63.3% | 4.6% |
Nonresponse mass concentrates on religion, gender-role, race, and welfare topics; the model selects a legitimate 'don't know'-style option rather than refusing in prose, so the sidestepping is invisible to refusal-rate metrics and only surfaces in the option distribution itself.
Elicitation Mode: Natural vs Schema-Forced
Schema-forced structured capture eliminates parse failures entirely, but it costs distributional fidelity — natural in-character elicitation is load-bearing for human-likeness. Matched pairs differ only in elicitation mode; all scores recomputed from committed per-question rows.
Althing (Haiku 4.5) (product) on gss — 75 questions × 30 samples per arm, refusal detector v2.
| Metric | Natural | Structured | Δ (structured − natural) |
|---|---|---|---|
| SPS | 0.803 | 0.791 | -1.2 pts |
| P_dist | 0.714 | 0.666 | -4.7 pts |
| P_rank | 0.724 | 0.736 | +1.2 pts |
| P_refuse | 0.972 | 0.972 | +0.0 pts |
| Parse failures | 1.6% | 0.0% | -1.6 pts |
Caveat: a single model × single dataset × one matched pair so far — a suggestive first datapoint, not a general law. More pairs are needed before treating the fidelity cost as systematic.
SPS by Topic
Best-per-model performance across the ten topic categories. Baselines and temperature variants hidden; scroll the legend to toggle models.
Grouped bar chart of best-per-model SPS across ten topic categories. Each group is a topic; each bar is a model.
SPS Convergence
How SPS stabilizes as replicate count increases. Showing the ensemble and the three blended single models.
Line chart showing how Survey Parity Score stabilizes as the number of replicates increases. One line per model plus the ensemble.
Temporal Drift Floor
Mean Jensen-Shannon divergence between the same Pew ATP question stems across different waves, grouped by year gap. A separate baseline from the Human Ceiling — drift measures how much real-world opinions shift, not sampling noise.
Temporal Drift Floor ships May 2026
The next publish cycle lifts the full cross-wave JSD table from
baselines.temporal_drift
in leaderboard.json —
mean drift, 95% CI, and per-year-gap pairs are already computed from repeated
Pew ATP stems on OpinionsQA. See the
Temporal Drift Floor definition
on the methodology page.