Datasets

Name: SynthBench Leaderboard
Creator: SynthBench
License: https://opensource.org/licenses/MIT

The real human survey datasets SynthBench validates against — sources, coverage, demographics, and redistribution policy for each.

Datasets

SynthBench validates models against real human survey data across nine registered datasets. Each card shows its redistribution tier — full ships per-question distributions publicly, gated requires a research-use sign-in to reach per-question payloads.

OpinionsQA

gated

The primary ground-truth dataset. Multiple-choice questions covering political, social, and economic attitudes from nationally representative US surveys.

Questions: 1,498 (300 core)
Source: Pew American Trends Panel
Scope: US population
Coverage: 2017–2022 (15 survey waves)
Demographics: 56 groups across 11 attributes

GlobalOpinionQA

gated Phase 2

Cross-cultural attitudes dataset for Phase 2 expansion. Tests whether models capture cultural differences in opinion beyond US demographics.

Questions: 2,556
Source: Pew Global Attitudes
Scope: 138 countries
Coverage: Cross-national
Demographics: Country-level cultural variation

SubPOP

gated

Extended dataset focused on demographic subgroup variation. Tests whether models can differentiate between fine-grained population segments.

Questions: 3,362
Source: 22 US subpopulations
Scope: US demographic subgroups
Coverage: Cross-sectional
Demographics: 11 attributes: age, education, gender, ideology, party, income, religion, attendance, region, marital status, citizenship

GSS

full

General Social Survey cumulative microdata from NORC. The only public-domain adapter that ships per-respondent rows, enabling real-sampling convergence curves alongside the bootstrap baseline.

Source: NORC General Social Survey
Scope: US population
Coverage: 1972–present (cumulative file)
Demographics: Individual-level microdata (microdata-capable)

NTIA Internet Use Supplement

full

NTIA Computer & Internet Use Supplement to the Current Population Survey. U.S. Government public-domain work under 17 USC 105; per-question distributions ship openly.

Source: US Census / NTIA CPS Supplement
Scope: US households
Coverage: Biennial waves, 1994–present
Demographics: Standard CPS demographics

WVS (World Values Survey)

gated

World Values Survey Wave 7. Cross-cultural value orientations across national populations. Research-use license; per-question artifacts gated behind JWT auth.

Source: World Values Survey Association
Scope: Cross-national (Wave 7, 2017–2022)
Coverage: Wave 7 (2017–2022)
Demographics: Country + standard demographic cells

Eurobarometer

gated

Eurobarometer Consumer Modules. EU-wide consumer conditions, digital services, and sustainability attitudes. GESIS redistribution restricted; per-question served from gated origin.

Source: European Commission / GESIS
Scope: EU member states
Coverage: Quarterly (Standard / Flash / Special)
Demographics: Country + EU-harmonized demographic cells

Michigan (Surveys of Consumers)

gated

University of Michigan Surveys of Consumers — consumer sentiment, inflation expectations, and household finance. Academic use terms; per-question gated.

Source: University of Michigan SRC
Scope: US consumers
Coverage: Monthly cadence
Demographics: Standard consumer-survey demographics

Pew Technology

gated

Pew Research Internet & Technology waves: tech adoption, digital privacy, social media, and AI attitudes. Pew redistribution terms; per-question gated.

Source: Pew Research Center Internet & Technology
Scope: US population (American Trends Panel)
Coverage: 10 tech waves (W86–W113)
Demographics: ATP demographic weights

Dataset redistribution policy

Each upstream dataset carries its own license terms. SynthBench honors them at publish time across four tiers:

full — per-question distributions published openly (public-domain datasets).
gated — per-question distributions served from a Cloudflare Worker that validates a Supabase JWT; research-use sign-in required. No distribution ships anonymously.
aggregates_only — aggregate scores public; no per-question artifact ships at all (gating does not substitute for missing redistribution permission).
citation_only — question metadata only; every downstream metric suppressed.

Aggregate metrics (SPS, JSD, ranks, cross-model parity) are derived quantities and remain public regardless of tier.

Per-dataset redistribution policy tier and provenance, used by synthbench publish to gate which fields appear in public artifacts.
Dataset	Tier	Source & license
eurobarometer	gated Per-question human distributions served from authenticated origin; sign-in required (research-use license).	European Commission / GESIS — Eurobarometer Consumer Modules (license)
globalopinionqa	gated Per-question human distributions served from authenticated origin; sign-in required (research-use license).	Durmus et al. 2023, Anthropic — llm_global_opinions (license)
gss	full Per-question human distributions published openly on the static site.	NORC at the University of Chicago — General Social Survey (license)
michigan	gated Per-question human distributions served from authenticated origin; sign-in required (research-use license).	Survey Research Center, University of Michigan — Surveys of Consumers (license)
ntia	full Per-question human distributions published openly on the static site.	NTIA Internet Use Survey (U.S. Government work, 17 USC 105) (license)
opinionsqa	gated Per-question human distributions served from authenticated origin; sign-in required (research-use license).	Santurkar et al., ICML 2023 — Whose Opinions Do LLMs Reflect? (derived from Pew American Trends Panel) (license)
pewtech	gated Per-question human distributions served from authenticated origin; sign-in required (research-use license).	Pew Research Center — Internet & Technology (American Trends Panel) (license)
subpop	gated Per-question human distributions served from authenticated origin; sign-in required (research-use license).	Suh et al., ACL 2025 — SubPOP: Subpopulation-Level Opinion Prediction (license)
wvs	gated Per-question human distributions served from authenticated origin; sign-in required (research-use license).	World Values Survey Association — WVS Wave 7 (2017-2022) (license)

For how these datasets are scored — metrics, baselines, the private holdout split, and anti-gaming defences — see the methodology page.